[MLton] Crash fread(...) failed (only read 0) (Cannot allocate memory) during deepFlatten with MLton 20070826

[Nicolas Bertolotti]

I finally applied the following patch in order to deal with the page size aspect:
--- mlton/runtime/gc/heap.c        2008-04-11 19:12:58.000000000 +0200
+++ mltonp1/runtime/gc/heap.c      2008-04-22 19:28:35.000000000 +0200
@@ -157,6 +157,7 @@
   assert (isHeapInit (h));
   if (desiredSize < minSize)
     desiredSize = minSize;
+  minSize = align (minSize, s->sysvals.pageSize);
   desiredSize = align (desiredSize, s->sysvals.pageSize);
   assert (0 == h->size and NULL == h->start);
   backoff = (desiredSize - minSize) / 20;
@@ -208,6 +209,9 @@
                uintmaxToCommaString (backoff),
                uintmaxToCommaString (minSize));
     }
+    /* Make sure we always try to allocate at least 'minSize' bytes. */
+    if (h->size > minSize && (h->size - backoff) < minSize)
+      backoff = h->size - minSize;
   }
   h->size = 0;
   return FALSE;

Now, the compilation succeeds ... but it is very slow. A lot of time seems to be spent on GC even while there is about 300 Mb of available memory that could be allocated to get a bigger heap.

Anyway, I am not an expert on the subject and I don't know much about MLton's GC.

I have just applied your latest patch about card cross map and I'll try to give you some feedback.

Nicolas

> -----Original Message-----
> From: Matthew Fluet [mailto:fluet at tti-c.org]
> Sent: Tuesday, April 22, 2008 8:13 PM
> To: Matthew Fluet
> Cc: Nicolas Bertolotti; mlton at mlton.org
> Subject: RE: [MLton] Crash fread(...) failed (only read 0) (Cannot
> allocate memory) during deepFlatten with MLton 20070826
>
> On Tue, 22 Apr 2008, Matthew Fluet wrote:
> > On Tue, 22 Apr 2008, Nicolas Bertolotti wrote:
> >>  There was actually another bug in heap.c. Here is an additional patch
> to
> >>  fix it:
> >>  --- mlton/runtime/gc/heap.c   2008-04-11 19:12:58.000000000 +0200
> >>  +++ mltonp1/runtime/gc/heap.c 2008-04-22 16:24:50.000000000 +0200
> >>  @@ -337,6 +337,7 @@
> >>        }
> >>        GC_diskBack_read (data, curHeapp->start, size);
> >>        GC_diskBack_close (data);
> >>  +      curHeapp->oldGenSize = size;
> >>      } else {
> >>        GC_diskBack_close (data);
> >>        if (s->controls.messages)
> >
> > Yes, I was just coming to that exact conclusion.
> >
> >>  Unfortunately, now, I get the following error:
> >>             deepFlatten starting
> >>  Out of memory.  Unable to allocate 1,686,393,408 bytes.
> >>
> >>  ... which, I guess, is a different problem.
> >
> > It is a different problem, in the sense that you succeeded (at least
> once) in
> > writing the heap to disk, allocating a larger heap, and reading the heap
> from
> > disk.  Getting an out of memory error suggests that even after writing
> the
> > heap to disk and releasing it, the operating system was unable to mmap
> 1.6G.
> > That seems strange, even on a 32bit system, where the virtual address
> space
> > is 4G, though I suppose it could happen with sufficient fragmentation.
> >
> > If you compile with @MLton gc-messages --, you should get a more
> detailed
> > memory listing of what memory is reserved, which might indicate whether
> there
> > is some extra memory being held.
>
> One thing I notice is that when writing the heap to disk, we do not
> release the card/cross maps.  Although with a 1.6G heap, these are only
> 12.8M, if they happen to fall in the middle of the virtual address space,
> then it may not be possible to find the requested 1.6G heap.
>
> You could try the attached patch, which releases the card/cross maps after
> writing the heap to disk.