[MLton] Profiling causes excessive memory usage

[Matthew Fluet]

On Mon, Dec 6, 2010 at 5:52 PM, Nicholas Kidd <nakidd at gmail.com> wrote:
> Hi MLton developers,
>
> I have a program that MLton compiles just fine without profiling turned on;
> however, when profiling is enabled via -profile {time,alloc,count}, MLton
> runs out of memory in the deepFlatten phase. I haven't yet tried, but I
> imagine disabling the deepFlatten pass will allow for compilation to
> succeed. Is there a better alternative? What would be the best way to debug
> the memory usage? I.e., what information can I gather that would be useful
> in debugging the issue?

There are some known performance issues with the memory usage of
DeepFlatten, which have never been fully investigated.  See some notes
at the end of
  http://mlton.org/DeepFlatten
Profiling tries to impact the optimizations as little as possibly, but
in some instances, like your program, things are changed enough that
it triggers the memory explosion.  Compiling with "-drop-pass
deepFlatten" is probably the best course of action.  As for
information useful for debugging the issue, the program that triggers
the memory usage would be helpful.  It is on a long list of TODOs to
investigate the RefFlatten and DeepFlatten memory issues.

> Some initial information that might be helpful:
>
> :: Machine is a macbook pro 2.8 Ghz with 4GB of memory running OS 10.6.5
>
> :: mlton outputs "MLton 20100608 (built Tue Jun  8 13:10:24 EDT 2010 on fenrir.cs.rit.edu)"
>
> :: I'm using the 32bit version of MLton (though the same behavior was observed with the 64bit executable, it just chewed up a lot more memory)
>
> :: A `wc -l *.sml` returns 43357
>
> :: Successful command line is as follows:
> mlton -runtime 'gc-summary' -profile no -output athena-profile=no -verbose 2 athena.mlb > build-profile=no.txt 2>&1
>
> :: Unsuccessful command line:
> mlton -runtime 'gc-summary' -profile alloc -output athena-profile=alloc -verbose 2 athena.mlb > build-profile=alloc.txt 2>&1
>
> :: Last few lines of build-profile=alloc.txt
>      ssa2Simplify starting
>         typeCheck starting
>         typeCheck finished in 0.32 + 0.00 (0% GC)
>         deepFlatten starting
> Out of memory.  Unable to allocate heap with 1,192,034,304 bytes.
>
> That final heap size looks small doesn't it?

A little small, but consistent with MLton's heap resizing policy: it
can sometimes grow one existing heap so large that it can't allocate
another (larger) heap.  You can also try compiling with "@MLton
may-page-heap true", which will allow MLton to write the heap to a
temporary file, free the heap, and then try to allocate a new heap in
a less constrained virtual address space.  MacOSX should be able to
give you about 2.1GiB of heap.  But, if DeepFlatten is seriously over
allocating, then that might not help.