[MLton] Profiling causes excessive memory usage

[Nicholas Kidd]

Hi Matthew,

Thanks for the quick response. With "-drop-pass deepFlatten" I was able to
complete x86 codegen,
after which gcc choked on the generated C file, i.e., this was unable to
complete on my machine
due to insufficient memory:

   Compile and Assemble starting
      gcc -std=gnu99 -c \

-I/Users/nkidd/sw/mlton-20100608-1/usr/local/lib/mlton/targets/self/include
\
          -I/Users/nkidd/sw/mlton-20100608-1/usr/local/lib/mlton/include -O1
\
          -fno-common -fno-strict-aliasing -fomit-frame-pointer -w \
          -I/usr/local/include -I/opt/local/include -I/sw/include -m32 \
          -fno-strength-reduce -fschedule-insns -fschedule-insns2 \
          -malign-functions=5 -malign-jumps=2 -malign-loops=2 -o \
          /var/folders/Ni/NireuwJ4EqeNlBbesdtkj++++TI/-Tmp-/fileRhFG4a.o \
          /var/folders/Ni/NireuwJ4EqeNlBbesdtkj++++TI/-Tmp-/file8ibMpc.82.c

After moving to a linux machine with more memory (5g) I was able to finish
the compilation. I might suggest adding to the TODO list generating
multiple, smaller .c files to keep gcc ;) (I assume that you are not relying
on gcc's optimizer too much at that point in the compilation.)

I'll work on getting the sources to you.

Thanks again,
Nick
Thanks,
Nick

On Tue, Dec 7, 2010 at 10:42 AM, Matthew Fluet <matthew.fluet at gmail.com>wrote:

> On Mon, Dec 6, 2010 at 5:52 PM, Nicholas Kidd <nakidd at gmail.com> wrote:
> > Hi MLton developers,
> >
> > I have a program that MLton compiles just fine without profiling turned
> on;
> > however, when profiling is enabled via -profile {time,alloc,count}, MLton
> > runs out of memory in the deepFlatten phase. I haven't yet tried, but I
> > imagine disabling the deepFlatten pass will allow for compilation to
> > succeed. Is there a better alternative? What would be the best way to
> debug
> > the memory usage? I.e., what information can I gather that would be
> useful
> > in debugging the issue?
>
> There are some known performance issues with the memory usage of
> DeepFlatten, which have never been fully investigated.  See some notes
> at the end of
>  http://mlton.org/DeepFlatten
> Profiling tries to impact the optimizations as little as possibly, but
> in some instances, like your program, things are changed enough that
> it triggers the memory explosion.  Compiling with "-drop-pass
> deepFlatten" is probably the best course of action.  As for
> information useful for debugging the issue, the program that triggers
> the memory usage would be helpful.  It is on a long list of TODOs to
> investigate the RefFlatten and DeepFlatten memory issues.
>
> > Some initial information that might be helpful:
> >
> > :: Machine is a macbook pro 2.8 Ghz with 4GB of memory running OS 10.6.5
> >
> > :: mlton outputs "MLton 20100608 (built Tue Jun  8 13:10:24 EDT 2010 on
> fenrir.cs.rit.edu)"
> >
> > :: I'm using the 32bit version of MLton (though the same behavior was
> observed with the 64bit executable, it just chewed up a lot more memory)
> >
> > :: A `wc -l *.sml` returns 43357
> >
> > :: Successful command line is as follows:
> > mlton -runtime 'gc-summary' -profile no -output athena-profile=no
> -verbose 2 athena.mlb > build-profile=no.txt 2>&1
> >
> > :: Unsuccessful command line:
> > mlton -runtime 'gc-summary' -profile alloc -output athena-profile=alloc
> -verbose 2 athena.mlb > build-profile=alloc.txt 2>&1
> >
> > :: Last few lines of build-profile=alloc.txt
> >      ssa2Simplify starting
> >         typeCheck starting
> >         typeCheck finished in 0.32 + 0.00 (0% GC)
> >         deepFlatten starting
> > Out of memory.  Unable to allocate heap with 1,192,034,304 bytes.
> >
> > That final heap size looks small doesn't it?
>
> A little small, but consistent with MLton's heap resizing policy: it
> can sometimes grow one existing heap so large that it can't allocate
> another (larger) heap.  You can also try compiling with "@MLton
> may-page-heap true", which will allow MLton to write the heap to a
> temporary file, free the heap, and then try to allocate a new heap in
> a less constrained virtual address space.  MacOSX should be able to
> give you about 2.1GiB of heap.  But, if DeepFlatten is seriously over
> allocating, then that might not help.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mlton.org/pipermail/mlton/attachments/20101207/559bfaa0/attachment.htm