[MLton-devel] D2 problem

Stephen Weeks MLton@mlton.org
Tue, 23 Apr 2002 20:40:15 -0700


Alain, I just checked in a fix for the out of memory problem you were
seeing.  Now, MLton attempts to use as much of the address space as it
can on Cygwin.  Unfortunately, I still have a problem when running
your test code on Cygwin.  I have included a fragment of the log
below.  The program dies when attempting to do a GC while a
512,000,000 byte array is live.  The error is better :-) in that MLton
has used as large a portion of the address space as it can.
Unfortunately, we are limited by address space restrictions on
Windows/Cygwin.  First, Windows only allows 2G of address space per
process [0, 2G).  Second, before MLton starts running, various
portions of the address space have been reserved, leaving only about
1.1G of contiguous space available.  Because MLton allocates 1.25
times the amount of live data space, which comes to 641 million bytes,
there is only about 480 million available for to space, which is not
enough.

I don't see any easy way to work around the address space limitation
and fragmentation problems on windows with MLton's current approach of
using two large semispaces.  The only way out that I see is to
allocate semispaces as collections of smaller chunks (say 10M or
100M).  There isn't too much code in MLton that relies on the
semispaces being contiguous, so this should be feasible, although it's
not a 1-day fix.  Of course, this also only works if the data fits
into the smaller chunks.  I'm not sure if your actual code uses such
large arrays.  I don't see any solution for that on Windows without
breaking up the array.

Thoughts, anyone?



*** CHECKING PHASE 3, n=3000
***** Phase 2.7 (Allocating 512000000 bytes.)
/tmp/filemV690r.3.s 0: GC_doGC
Starting gc.  bytesRequested = 512000008
Releasing to space.
fromSpace = 1bb40000  toSpace = 1d310000
fromSpace size = 6,205,440  toSpace size = 641,101,824
Shrinking new space at 1d310000 to 640794624 bytes.
Shrinking old space at 1bb40000 to 0 bytes.
Releasing to space.
Finished gc.
time(ms): 40
live(bytes): 633,928 (10.2%)
/tmp/filemV690r.3.s 0: GC_doGC
Starting gc.  bytesRequested = 0
[Requested 611M cannot be satisfied, backing off by 30M (need = 0M).
[Requested 580M cannot be satisfied, backing off by 30M (need = 0M).
[Requested 549M cannot be satisfied, backing off by 30M (need = 0M).
[Requested 519M cannot be satisfied, backing off by 30M (need = 0M).
[Requested 488M cannot be satisfied, backing off by 30M (need = 0M).
fromSpace = 1d310000  toSpace = 43680000
fromSpace size = 640,794,624  toSpace size = 480,579,584
Total Phys. Mem: 427216896
Avail Phys. Mem: 3145728
Total Page File: 1180844032
Avail Page File: 28672
Total Virtual: 2147352576
Avail Virtual: 733569024
0x       0      65536  MEM_FREE PAGE_NOACCESS
0x   10000       4096  MEM_COMMIT PAGE_READWRITE
0x   11000      61440  MEM_FREE PAGE_NOACCESS
0x   20000       4096  MEM_COMMIT PAGE_READWRITE
0x   21000      61440  MEM_FREE PAGE_NOACCESS
0x   30000    2068480  MEM_RESERVE PAGE_NOACCESS
0x  229000       4096  MEM_COMMIT PAGE_NOACCESS
0x  22a000      24576  MEM_COMMIT PAGE_READWRITE
0x  230000      24576  MEM_COMMIT PAGE_READWRITE
0x  236000    1024000  MEM_RESERVE PAGE_READWRITE
0x  330000       4096  MEM_COMMIT PAGE_READWRITE
0x  331000      61440  MEM_RESERVE PAGE_READWRITE
0x  340000      90112  MEM_COMMIT PAGE_READONLY
0x  356000      40960  MEM_FREE PAGE_NOACCESS
0x  360000     147456  MEM_COMMIT PAGE_READONLY
0x  384000      49152  MEM_FREE PAGE_NOACCESS
0x  390000     266240  MEM_COMMIT PAGE_READONLY
0x  3d1000      61440  MEM_FREE PAGE_NOACCESS
0x  3e0000      12288  MEM_COMMIT PAGE_READONLY
0x  3e3000     118784  MEM_FREE PAGE_NOACCESS
0x  400000       4096  MEM_COMMIT PAGE_READONLY
0x  401000     397312  MEM_COMMIT PAGE_EXECUTE_READ
0x  462000      28672  MEM_COMMIT PAGE_WRITECOPY
0x  469000      12288  MEM_COMMIT PAGE_READWRITE
0x  46c000    1413120  MEM_COMMIT PAGE_READONLY
0x  5c5000      45056  MEM_FREE PAGE_NOACCESS
0x  5d0000      12288  MEM_COMMIT PAGE_EXECUTE_READ
0x  5d3000     774144  MEM_RESERVE PAGE_EXECUTE_READ
0x  690000       4096  MEM_COMMIT PAGE_EXECUTE_READ
0x  691000      28672  MEM_RESERVE PAGE_EXECUTE_READ
0x  698000      32768  MEM_FREE PAGE_NOACCESS
0x  6a0000     274432  MEM_COMMIT PAGE_READONLY
0x  6e3000     118784  MEM_FREE PAGE_NOACCESS
0x  700000     245760  MEM_COMMIT PAGE_EXECUTE_READ
0x  73c000    2899968  MEM_RESERVE PAGE_EXECUTE_READ
0x  a00000       4096  MEM_COMMIT PAGE_READWRITE
0x  a01000      61440  MEM_FREE PAGE_NOACCESS
0x  a10000       4096  MEM_COMMIT PAGE_READWRITE
0x  a11000      61440  MEM_FREE PAGE_NOACCESS
0x  a20000       4096  MEM_COMMIT PAGE_READWRITE
0x  a21000     126976  MEM_RESERVE PAGE_READWRITE
0x  a40000       4096  MEM_COMMIT PAGE_READWRITE
0x  a41000  157020160  MEM_FREE PAGE_NOACCESS
0x a000000      49152  MEM_COMMIT PAGE_READWRITE
0x a00c000      16384  MEM_FREE PAGE_NOACCESS
0x a010000       4096  MEM_COMMIT PAGE_READWRITE
0x a011000  268431360  MEM_RESERVE PAGE_READWRITE
0x1a010000      20480  MEM_COMMIT PAGE_READWRITE
0x1a015000      45056  MEM_FREE PAGE_NOACCESS
0x1a020000    2088960  MEM_RESERVE PAGE_NOACCESS
0x1a21e000       4096  MEM_COMMIT PAGE_NOACCESS
0x1a21f000       4096  MEM_COMMIT PAGE_READWRITE
0x1a220000       4096  MEM_COMMIT PAGE_READWRITE
0x1a221000      61440  MEM_FREE PAGE_NOACCESS
0x1a230000    1048576  MEM_COMMIT PAGE_READWRITE
0x1a330000   50200576  MEM_FREE PAGE_NOACCESS
0x1d310000  640794624  MEM_COMMIT PAGE_READWRITE
0x4362c000     307200  MEM_RESERVE PAGE_READWRITE
0x43677000      36864  MEM_FREE PAGE_NOACCESS
0x43680000  480579584  MEM_COMMIT PAGE_READWRITE
0x600d1000   15921152  MEM_FREE PAGE_NOACCESS
0x61000000       4096  MEM_COMMIT PAGE_READONLY
0x61001000     663552  MEM_COMMIT PAGE_EXECUTE_READ
0x610a3000       8192  MEM_COMMIT PAGE_READWRITE
0x610a5000       8192  MEM_COMMIT PAGE_WRITECOPY
0x610a7000      16384  MEM_COMMIT PAGE_READWRITE
0x610ab000       8192  MEM_COMMIT PAGE_WRITECOPY
0x610ad000       4096  MEM_COMMIT PAGE_READWRITE
0x610ae000       4096  MEM_COMMIT PAGE_EXECUTE_READWRITE
0x610af000       4096  MEM_COMMIT PAGE_WRITECOPY
0x610b0000       4096  MEM_COMMIT PAGE_EXECUTE_WRITECOPY
0x610b1000       4096  MEM_COMMIT PAGE_WRITECOPY
0x610b2000       4096  MEM_COMMIT PAGE_EXECUTE_WRITECOPY
0x610b3000       4096  MEM_COMMIT PAGE_READWRITE
0x610b4000       4096  MEM_COMMIT PAGE_EXECUTE_READWRITE
0x610b5000       4096  MEM_COMMIT PAGE_WRITECOPY
0x610b6000       4096  MEM_COMMIT PAGE_EXECUTE_WRITECOPY
0x610b7000       4096  MEM_COMMIT PAGE_WRITECOPY
0x610b8000      12288  MEM_COMMIT PAGE_EXECUTE_WRITECOPY
0x610bb000       4096  MEM_COMMIT PAGE_WRITECOPY
0x610bc000       4096  MEM_COMMIT PAGE_EXECUTE_WRITECOPY
0x610bd000       4096  MEM_COMMIT PAGE_WRITECOPY
0x610be000       4096  MEM_COMMIT PAGE_EXECUTE_WRITECOPY
0x610bf000       4096  MEM_COMMIT PAGE_WRITECOPY
0x610c0000       4096  MEM_COMMIT PAGE_EXECUTE_WRITECOPY
0x610c1000       4096  MEM_COMMIT PAGE_WRITECOPY
0x610c2000       4096  MEM_COMMIT PAGE_EXECUTE_WRITECOPY
0x610c3000       8192  MEM_COMMIT PAGE_READWRITE
0x610c5000     192512  MEM_COMMIT PAGE_WRITECOPY
0x610f4000       4096  MEM_COMMIT PAGE_READWRITE
0x610f5000      12288  MEM_COMMIT PAGE_WRITECOPY
0x610f8000       8192  MEM_COMMIT PAGE_READWRITE
0x610fa000      24576  MEM_COMMIT PAGE_WRITECOPY
0x61100000       4096  MEM_COMMIT PAGE_READWRITE
0x61101000      45056  MEM_COMMIT PAGE_WRITECOPY
0x6110c000    4931584  MEM_FREE PAGE_NOACCESS
0x615c0000      16384  MEM_COMMIT PAGE_READWRITE
0x615c4000    5013504  MEM_RESERVE PAGE_READWRITE
0x61a8c000  271794176  MEM_FREE PAGE_NOACCESS
0x71dc0000       4096  MEM_COMMIT PAGE_READONLY
0x71dc1000      12288  MEM_COMMIT PAGE_EXECUTE_READ
0x71dc4000      16384  MEM_COMMIT PAGE_WRITECOPY
0x71dc8000       8192  MEM_COMMIT PAGE_READONLY
0x71dca000   80699392  MEM_FREE PAGE_NOACCESS
0x76ac0000       4096  MEM_COMMIT PAGE_READONLY
0x76ac1000      86016  MEM_COMMIT PAGE_EXECUTE_READ
0x76ad6000       8192  MEM_COMMIT PAGE_READONLY
0x76ad8000       4096  MEM_COMMIT PAGE_READWRITE
0x76ad9000       4096  MEM_COMMIT PAGE_WRITECOPY
0x76ada000      12288  MEM_COMMIT PAGE_READONLY
0x76add000   17903616  MEM_FREE PAGE_NOACCESS
0x77bf0000       4096  MEM_COMMIT PAGE_READONLY
0x77bf1000       8192  MEM_COMMIT PAGE_EXECUTE_READ
0x77bf3000       4096  MEM_COMMIT PAGE_READONLY
0x77bf4000       4096  MEM_COMMIT PAGE_READWRITE
0x77bf5000       8192  MEM_COMMIT PAGE_READONLY
0x77bf7000    1871872  MEM_FREE PAGE_NOACCESS
0x77dc0000       4096  MEM_COMMIT PAGE_READONLY
0x77dc1000     172032  MEM_COMMIT PAGE_EXECUTE_READ
0x77deb000      40960  MEM_COMMIT PAGE_READONLY
0x77df5000       4096  MEM_COMMIT PAGE_READWRITE
0x77df6000      12288  MEM_COMMIT PAGE_WRITECOPY
0x77df9000       8192  MEM_COMMIT PAGE_READWRITE
0x77dfb000      16384  MEM_COMMIT PAGE_READONLY
0x77dff000      69632  MEM_FREE PAGE_NOACCESS
0x77e10000       4096  MEM_COMMIT PAGE_READONLY
0x77e11000     286720  MEM_COMMIT PAGE_EXECUTE_READ
0x77e57000      24576  MEM_COMMIT PAGE_READONLY
0x77e5d000       4096  MEM_COMMIT PAGE_READWRITE
0x77e5e000      36864  MEM_COMMIT PAGE_READONLY
0x77e67000      36864  MEM_FREE PAGE_NOACCESS
0x77e70000       4096  MEM_COMMIT PAGE_READONLY
0x77e71000     266240  MEM_COMMIT PAGE_EXECUTE_READ
0x77eb2000      36864  MEM_COMMIT PAGE_READONLY
0x77ebb000       4096  MEM_COMMIT PAGE_READWRITE
0x77ebc000      36864  MEM_COMMIT PAGE_READONLY
0x77ec5000      45056  MEM_FREE PAGE_NOACCESS
0x77ed0000       4096  MEM_COMMIT PAGE_READONLY
0x77ed1000     147456  MEM_COMMIT PAGE_EXECUTE_READ
0x77ef5000       4096  MEM_COMMIT PAGE_READONLY
0x77ef6000       4096  MEM_COMMIT PAGE_READWRITE
0x77ef7000      20480  MEM_COMMIT PAGE_READONLY
0x77efc000      16384  MEM_FREE PAGE_NOACCESS
0x77f00000       4096  MEM_COMMIT PAGE_READONLY
0x77f01000     241664  MEM_COMMIT PAGE_EXECUTE_READ
0x77f3c000      36864  MEM_COMMIT PAGE_READONLY
0x77f45000       8192  MEM_COMMIT PAGE_READWRITE
0x77f47000      94208  MEM_COMMIT PAGE_READONLY
0x77f5e000       8192  MEM_FREE PAGE_NOACCESS
0x77f60000       4096  MEM_COMMIT PAGE_READONLY
0x77f61000     245760  MEM_COMMIT PAGE_EXECUTE_READ
0x77f9d000      32768  MEM_COMMIT PAGE_READONLY
0x77fa5000       4096  MEM_COMMIT PAGE_READWRITE
0x77fa6000       4096  MEM_COMMIT PAGE_WRITECOPY
0x77fa7000       8192  MEM_COMMIT PAGE_READWRITE
0x77fa9000       4096  MEM_COMMIT PAGE_WRITECOPY
0x77faa000      81920  MEM_COMMIT PAGE_READONLY
0x77fbe000  123936768  MEM_FREE PAGE_NOACCESS
0x7f5f0000    1048576  MEM_RESERVE PAGE_NOACCESS
0x7f6f0000      24576  MEM_COMMIT PAGE_EXECUTE_READ
0x7f6f6000    1024000  MEM_RESERVE PAGE_EXECUTE_READ
0x7f7f0000    8126464  MEM_FREE PAGE_NOACCESS
0x7ffb0000     147456  MEM_COMMIT PAGE_READONLY
0x7ffd4000      36864  MEM_FREE PAGE_NOACCESS
0x7ffdd000       4096  MEM_COMMIT PAGE_EXECUTE_READWRITE
0x7ffde000       4096  MEM_COMMIT PAGE_EXECUTE_READWRITE
0x7ffdf000       4096  MEM_COMMIT PAGE_EXECUTE_READWRITE
0x7ffe0000       4096  MEM_COMMIT PAGE_READONLY
0x7ffe1000      61440  MEM_RESERVE PAGE_NOACCESS
0x7ffe1000      61440  MEM_RESERVE PAGE_NOACCESS
0x7ffe1000      61440  MEM_RESERVE PAGE_NOACCESS
size=512000008 skip=0 remaining=480547684 s->fromSize=640794624 s->toSize=480579584 headerBytes=8 objectBytes=512000000 header=4000
Out of memory (forward).
Diagnostic: probably a RAM problem.

_______________________________________________
MLton-devel mailing list
MLton-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mlton-devel