<html><head><style type="text/css"><!-- DIV {margin:0px;} --></style></head><body><div style="font-family:times new roman,new york,times,serif;font-size:12pt"><div>Is GC really sequential in terms of memory access? Certainly the writes<br>to new space are, but not the reads from old space, nor the writes to<br>old space (to set the forwarding pointer).<br><br>I have a vague recollection of Macsyma on the VAX (before even my time),<br>which no doubt used a mark/sweep collector, and they used vadvise to set<br>random behaviour (I thought) during GC.<br><br>With regards to the max-heap argument, it would be really good to have<br>an argument which is the maximum amount of memory MLton can use, or as<br>close as we can come to that. If it tries to grow bigger, even if only<br>temporarily to grow a heap, and can't grow it by any other allowed<br>means, it should
fail with out of memory.<br><br></div><div style="font-family: times new roman,new york,times,serif; font-size: 12pt;"><br><div style="font-family: times new roman,new york,times,serif; font-size: 12pt;"><font size="2" face="Tahoma"><hr size="1"><b><span style="font-weight: bold;">From:</span></b> Wesley W. Terpstra <wesley@terpstra.ca><br><b><span style="font-weight: bold;">To:</span></b> Matthew Fluet <matthew.fluet@gmail.com><br><b><span style="font-weight: bold;">Cc:</span></b> mlton@mlton.org<br><b><span style="font-weight: bold;">Sent:</span></b> Fri, December 11, 2009 6:17:34 PM<br><b><span style="font-weight: bold;">Subject:</span></b> Re: [MLton] max-heap setting for 64-bit applications<br></font><br>
Stepping outside of the current discussion, a matter of practicality: is the paging really that bad? AFAIK, it is a memcpy running between the two heaps which is sequential access. It shouldn't take longer than it would take to copy a 3GB file on the disk.<br>
<br>If it *is* taking longer, then we need to add a hint to the windows VM to tell it that we will be doing sequential access before the memcpy, then flip it back to random-access mode.<br><br>On Fri, Dec 11, 2009 at 9:32 PM, Matthew Fluet <span dir="ltr"><<a rel="nofollow" ymailto="mailto:matthew.fluet@gmail.com" target="_blank" href="mailto:matthew.fluet@gmail.com">matthew.fluet@gmail.com</a>></span> wrote:<br>
<div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Right, and the annoying bit is that the previous heap was so close to<br>
the max heap setting. Perhaps a reasonable heuristic is that if a<br>
desired heap is "close-to" the max-heap size, just round up. Perhaps<br>
0.75 of max heap? In the max-heap 3G setting, this could still leave<br>
you in the situation where you have a 2.25G allocation and a 3G<br>
allocation at the same time to copy. Or 0.55 of max heap; that could<br>
require 1.65G+3G at the time of the copy.<br></blockquote><div><br>I would be against yet another special case in the sizing rules. Any cutoff we pick is going to fail for someone else the same way, while artificially restricting the memory growth for others. His problem would be (mostly) fixed if we flipped windows mremap to only move if growth fails. <br>
<br>I have a higher-level solution proposal: MLton.GC already has hooks that are executed after a GC to implement finalizers. Expose these to the user. If a user knows his application only consumes X memory on an error condition, he can test for this after a GC and terminate with an "Out of Memory" error as desired. <br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im">> It seems like one strategy would be that keepomg the "max-heap"<br>
> setting slightly under half the available physical memory should<br>
> avoid the case where we're already using about 50% and then have<br>
> to create another heap of the same size.<br>
<br>
</div>True. Although, you would really need to use about 50% of the<br>
physical memory that you want the MLton process to have access to,<br>
else you will page via competition with other processes.<br></blockquote><div><br>I think this isn't quite right. You get paging when the working set of the other applications + heap of MLton program > RAM. If his heap stays small, there is no thrashing. Once it gets big, there wilt be thrashing iff the working set of apps + max-heap > RAM. Keeping the max-heap size warm wouldn't help, only cause more thrashing sooner.<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> Going to a single<br>
contiguous heap, interpreting it as two semi-spaces when using a major<br>
copying-collection would be nicer here, because fixed-heap would grab<br>
the whole 3.25G up front.<br></blockquote><div><br>Yes.<br><br><blockquote style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;" class="gmail_quote">the mremap function is described as
using the Linux page table scheme to efficiently change the mapping
between virtual addresses and (physical) memory pages. It's purpose is
to be more efficient than allocating a new map and copying.<br></blockquote><br>If I could ...<br><br></div></div>
</div></div>
<!-- cg4.c201.mail.mud.yahoo.com compressed/chunked Fri Dec 11 14:14:47 PST 2009 -->
</div></body></html>