[MLton] Re: Chat with Wesley Terpstra

[Matthew Fluet]

On Sun, May 2, 2010 at 8:58 AM, Wesley W. Terpstra <wesley at terpstra.ca> wrote:
> On Thu, Apr 29, 2010 at 6:22 PM, Wesley W. Terpstra <wesley at terpstra.ca>
> wrote:
>>>
>>> One thing that you could check is the behavior of this memory leak
>>> program
>>> on x86-mingw (on a machine with >4GiB memory):
>>>
>>>  fun loop (l,w) =
>>>     if (w = 0wx0)
>>>        then let
>>>                val w = List.foldl Word.xorb 0wx0 l
>>>                val _ = print (concat ["w = ", Word.toString w, "\n"])
>>>             in
>>>                loop (l, if w = 0wx0 then 0wxAAAAAAAA else w)
>>>             end
>>>     else loop (w::l, w - 0wx1)
>>>  val _ = loop ([], 0wxAAAAAAAA)
>
> All of these results come from mingw r7454 as uploaded to the Experimental
> page.
>
> I tried it on my machine with =4GB of memory running 64-bit Windows 7, but
> running it as a 32-bit application. With 'may-page-heap' off, it took 20
> seconds. With 'may-page-heap' on, it took 56 seconds. I've attached the
> output of both runs. It doesn't manage to get any more memory with the
> 'may-page-heap', because my windows mremap implementation can get just as
> much memory as paging to disk (it expands in place). I imagine you'd see the
> same behaviour in linux with native mremap?

Actually, mremap can't always grow an existing map as well as a mmap
with a cleared virtual address space (as with paging to disk).  So,
"may-page-heap true" will get to a higher maximum heap size.  A 32-bit
executable on a 64-bit linux with paging to disk can get a
3,984,899,596byte heap before running out of memory.  Without paging
to disk, it can be as low as 2147483648bytes, because that is as large
as mremap will grow an existing 2147481600byte heap.  That is, if you
are able to mmap a heap of significant size, mremap doesn't really
help you get much bigger.  On the other hand, one can jump from a very
small heap to a very large heap with mremap.

> I also tried it on the same machine (4GB, Windows 7, 64-bit) as a 64-bit
> application. With 'may-page-heap' off, it took 10 minutes before I
> terminated it. With 'may-page-heap' on, it ran for over 22 minutes before I
> terminated it. Unfortunately, the stderr file gets truncated when I
> interrupt it.

I'd be surprised that the two may-page-heap variants behaved
differently.  A 64bit application shouldn't ever exhaust virtual
memory address space and require paging to disk.  (And, if it did,
your temp disk space probably isn't big enough.)

> I noticed that the no-paging varient took more than 3.6GB of physical memory
> in windows task manager (enough that my system thrashed). I didn't have task
> manager open for the yes-paging test. Perhaps it is overestimating the
> amount of physical memory and thus not stopping at 50% as it should? Or is
> it the usual last-step resize = up to 2* maximum memory? Whatever is
> happening, it's not ok to push a system to thrashing with default
> parameters.

If you really don't want thrashing, then you need to set a "@MLton
max-heap ??" to your physical memory.  MLton's policy has always been
to use as much memory as the application requires, subject only to the
amount of memory obtainable from the operating system (unless max-heap
sets a lower limit).  MLton will try to size a heap to stay within
physical memory (really, 50% physical memory with the default
ram-slop), but if the live size of the application exceeds that and
the operating system is willing to give over the memory, then MLton
will use it.

> I can also try the 32-bit version under XP (32-bit) if that would useful. I
> expect it will behave just like it did in Windows 7 (32-bit), however. In
> other words, work perfectly.

I agree that it ought not to be any different.