[MLton] max-heap setting for 64-bit applications

[Wesley W. Terpstra]

On Thu, Dec 10, 2009 at 9:37 PM, Matthew Fluet <matthew.fluet at gmail.com>wrote:

> Note, that while Windows doesn't natively support mremap, Wesley
> implemented a few things that try to mimic mmap/mremap/munamp functionality
> under Windows.  There seems to be some complication with the Windows memory
> system that requires committing in addition to reserving memory, and some
> manner of being able to extend an existing map with space before it and
> after it.  I don't understand it.


When you reserve memory, you get virtual address space that does not overlap
any previous reservations. When you commit (part of) that reservation, you
get memory that can actually be used without segfaulting.

You can freely commit/decommit memory within a reservation, changing the
memory that windows must provide to the application. You >cannot< shrink or
grow a reservation.

While it is possible to reserve+commit in one step, this doesn't ensure that
the reservation is exclusive with any existing reservations. This is what
caused the sporadic windows GC segfaults some years ago, heaps that
overlapped.

With respect to mremap, my main concern was supporting heap growth in a
restricted virtual memory scenario. eg: you have 900MB allocated and need to
grow to 1.2G. This matters when self-compiling MLton on mingw32, for
example. To support this, it was necessary to allow a heap to grow
'in-place'. That's why the mremap implementation starts probing for
available memory before and after the existing heap.

Supposing there is available memory before/after the existing heap, I make
new reservations to expand the heap. The windows.c file describes the
invariant I used for the memory layout. The invariant I choose makes it safe
to free/shrink/grow and be certain the memory was actually allocated to
>this< heap.


> However, I do note that the generic version of mremap
> (<src>/runtime/platform/mremap.c), which is used by the Windows platforms,
> starts off trying to allocate an entirely new region of the desired size and
> copy the existing region into it.


We could revisit this approach in the 64-bit setting. The msdn documentation
makes vague warnings about the cost of making too many reservations, but I
doubt this really matters. From the point of view of a system with a lot of
virtual address space, but not a lot of physical RAM, it might make sense to
try to always expand in-place first.

Of course, even if mremap tries in-place heap growth first, this might not
be possible, and a copy to a new memory location will still thrash.

  (This actually seems to be redundant with the behavior of the garbage
> collector proper.)


Perhaps, but I was trying to implement an mremap compatible to the
specification of mremap, not how MLton uses mremap. If mremap fails to
aquire the desired space, mmap will certainly also fail.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mlton.org/pipermail/mlton/attachments/20091211/3d21a731/attachment.html