[MLton] Address space in windows

[Wesley W. Terpstra]

On Mon, Oct 13, 2008 at 11:18 PM, Matthew Fluet <fluet at tti-c.org> wrote:
> On Mon, 13 Oct 2008, Wesley W. Terpstra wrote:
>> 1. Eliminate the use of VirtualAlloc/Free on cygwin. mmap can acquire
>> just as much memory and presumably cygwin tracks mmap for use in
>> fork().
>
> At one point, there were issues with Cygwin's implementation of mmap, which
> motivated the inclusion of VirtualAlloc/Free.  On the other hand, looking at
> the commit logs, it appears that one does need to use mmap to use Cygwin's
> fork().

Seeing as how fork() is the only reason to use cygwin over mingw...

> I guess you are arguing that SUSv2 munmap() suffices to batch free any
> (contiguous) set of mmap-s; thus, one doesn't need to change the Cygwin
> implementation.

Or any SUSv2 system (bsd, solaris, etc). We can safely create a big
mapping from little ones and it should be completely compatible
AFAICT.

> I'm guessing here, but I would suspect that a potential downside of this
> over a 'native' mmap is that we could end up with a large number of mmap-s,
> as a heap is grown in place little by little.

Yes. I'm not sure what the downside of many small mappings is vs. one
big mapping. Is there a performance penalty? It's all anonymous
memory. Does mmap'ing two pages side-by-side differ from one mmap of
two pages? On windows, yes. On *nix? Isn't this how the C heap is
grown?

> It also misses the opportunity to move the region whole-sale to a completely
> different address
> (more likely to be a possibility in a 64-bit environment). Nonetheless, I
> agree that this would appear to give the right mremap semantics.

We could easily include a moving copy in the fake mremap. eg: try
first to expand in-place, if that fails, try an out-of-place
mmap+memmove.

> One could even attempt this mremap emulation on *BSD/darwin, where no mremap
> exists.

Yes.

> One needs to be careful to use MAP_FIXED in order to get the new mappings in
> exactly the right places.

No. Using MAP_FIXED overwrites existing maps. That would cause
problems. At least cygwin and linux mmap honour the named address (if
it's possible) without MAP_FIXED. If there is another mapping in the
way, it will either fail or pick a different address... we can detect
both.

> Why modify remapHeap in gc/heap.c?  You should do the changes in platform/*,
> adding/modifying the implementation of the the GC_mremap function.

The gc/heap.c has an existing back-off algorithm. I didn't want to
repeat the scan for maximum available size on each back-off attempt.
However, I guess that a simple cache of the largest successful probe
would work. Certainly it's cleaner. I'm refactoring my patch with this
in mind.

> One could also perform a related experiment, where one uses
> 'malloc/realloc/free' in place of 'mmap/mremap/munmap'.  The former are all
> required by POSIX (whereas mremap is only available natively on linux).  One
> issue is that realloc (according to the spec) could move the allocation,
> even if the reallocation is shrinking the allocation.  There are situations
> (e.g., shrinkHeap), where we want to release the unused portion of the heap,
> without moving the heap (because moving the heap requires translating all
> the heap pointers).

Interesting. I notice that translateHeap is free if from == to. So
adding translateHeaps after every shrinkHeap would only cost if an
implementation actually DID move on resize.

I've never seriously used realloc for anything this big. Quick testing
on linux reveals:
0804A008-B7CFFFFC: 2949341172
00001008-08047FFC: 134508532
B7F8D008-BFD76FFC: 132030452
BFD8C008-BFFFFFFC: 2572276
B7D00468-B7DFFFF4: 1047436
B7F5D008-B7F6CFFC: 65524
B7E00008-B7E00FFC: 4084
=> 3219569476
for malloc and
0804A000-B7E36000: 2950610944
00001000-08048000: 134508544
B7FC2000-BF9AC000: 127836160
BF9C1000-C0000000: 6549504
B7F92000-B7FA2000: 65536
=> 3219570688
for mmap. So they both can access more-or-less than same amount of memory.

Unfortunately, realloc from 2048000000U to 2950610944U fails. So it
may be portable, but realloc doesn't work in-place for large memory.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: scan.c
Type: text/x-csrc
Size: 1311 bytes
Desc: not available
Url : http://mlton.org/pipermail/mlton/attachments/20081014/62326eea/scan.c