[MLton] gcc choking on self-compile

[Stephen Weeks]

> Can you describe anything more about the issue you saw?  I suspect I ran
> into something similar.  I was encountering a problem where I was doing a
> make all-no-docs on clean sources (i.e., where there would be two rounds
> of compiling mlton-compile) where the first compile of mlton-compile would
> succeed, but the second compile of mlton-compile would get all the way
> through generating assembly, but then would error with gcc failure of
> insufficient memory.

I'm seeing exactly what you are.  Here's a recent compile.

   Compile C and Assemble starting
      gcc -c -I/tmp/make-16439/src/build/bin/../lib/include -O1 \
          -fno-strict-aliasing -fomit-frame-pointer -w -fno-strength-reduce \
          -fschedule-insns -fschedule-insns2 -malign-functions=5 \
          -malign-jumps=2 -malign-loops=2 -mcpu=pentiumpro -o \
          /tmp/fileang2O3.o /tmp/file2Fsv17.95.c
   Compile C and Assemble raised in 0.01 + 0.00 (0% GC)
MLton raised in 479.82 + 0.00 (0% GC)
call to system failed with Cannot allocate memory:

> I attributed the problem to having upgraded from Mandrake 9.1 to
> Mandrake 10.0 (entailing a kernel 2.4 to kernel 2.6 change), since
> it started immediately after that upgrade.

I see the problem on my 4G machine, which is a RedHat 7.3 with 2.4
kernel, running gcc 2.96.  I haven't changed this machine since I
installed it over two years ago.


After investigating a bit, I have a guess as to the problem.  When we
build MLton with an early version of itself, we use the MLton stubs.
These define MLton.GC.pack as a no-op, which means that MLton does not
free up heap space before compiling C/asm files.  In the case of a
self compile, this is a lot of space, and could explain why the C
compiler can't allocate enough memory.

The best way I can think of to fix this is to give up our invariant
that lib/mlton-stubs/mlton.sml is defined in the standard basis.  We
can give this up (just a little) by defining MLton.GC.pack to be the
MLton.GC.pack provided by the basis.  So that we can compile with
SML/NJ, this also requires adding a stub MLton structure to
lib/mlton-stubs-in-smlnj/ with whatever assumptions we need (in this
case just MLton.GC.pack).