From matthew.fluet at gmail.com Tue Dec 1 13:29:30 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Tue Dec 1 13:30:09 2009 Subject: [MLton] On "layout.sig, sml" In-Reply-To: <6004794.1243651259632150658.JavaMail.coremail@mailweb> References: <6004794.1243651259632150658.JavaMail.coremail@mailweb> Message-ID: 2009/11/30 Baojian Hua > We've been useing MLton for over 7 years for our research > projects, and recently we're planning to read and hack some source > code of MLton (and hopefully we can contribute in the > future). > It would be great if you could add a link to your project(s) to http://mlton.org/Users. And we start by reading sources in "lib/mlton/basic/", but > the documentation is rare, so we have to send some questions here: > in files "layout.sig, sml", this data structure: > > datatype t = T of {length: int, > tree: tree} > and tree = > Empty > | String of string > | Sequence of t list > | Align of {force: bool, rows: t list} > | Indent of t * int > > Does the "force" mean we must align several line? Looking at layout.sig, the comments note: (* layout the objects on separate lines*) val align: t list -> t (* layout the objects on separate lines, if necessary *) val mayAlign: t list -> t where, "align" corresponds to "Align {force = true, ...}" and "mayAlign" corresponds to "Align {force = false, ...}". So, "force = true" requires that each element be printed and aligned on a separate line. > And in calculating > the length, I'd expect to see this: > > case tree > of Align {rows, ...} => max (rows) (* the longest one *) > | _ => > > but the code is this: > > case tree > of Align {rows, ...} => \Sigma (rows) (* the whole *) > | _ => > The "length" field is really a "size" measure. That is, it measures the total size of all the string elements in the layout (+1 for each space that will appear between Align elements). It does not measure the length of a line of the layout; note, for example, that "length" is not incremented on an Indent. Looking at the "print" function (layout.sml:122) and the "Align" case (layout.sml:168), you see that the length is ignored when force == true; in that case, each layout element is printed on a separate line. When force == false, the length is used to determine if all of the layout elements can be printed on the same line. One subtlety is that if one does: val lay = mayAlign [align [str "A", str "B"], align [str "C", str "D"]] val () = print (lay, TextIO.print) then you will get the output: A B C D Note that the "force = true" on the inner aligns is ignored, because the outer align put all of its inner layout elements on to one line. Hope that helps. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091201/883571a5/attachment.htm From hansel at reactive-systems.com Thu Dec 3 05:40:08 2009 From: hansel at reactive-systems.com (David Hansel) Date: Thu Dec 3 05:40:43 2009 Subject: [MLton] Crashes with 64-bit native code generator on Windows In-Reply-To: References: <4AFA3919.1090003@reactive-systems.com> <162de7480911161107ld408687gd577838b9bccf868@mail.gmail.com> <4B0B3952.2000607@reactive-systems.com> <4B13EC2A.9070600@reactive-systems.com> <4B1404AB.4040802@reactive-systems.com> <4B1436D7.6010303@reactive-systems.com> Message-ID: <4B17BFB8.3090204@reactive-systems.com> Hi, Just in case someone comes across this thread in the future: Matthew Fluet fixed the problem that was causing these crashes in r7368. Many thanks, David Matthew Fluet wrote: > On Mon, Nov 30, 2009 at 4:19 PM, David Hansel > > wrote: > > I tried using "-native-commented 6" but (due to the size of the code > involved) > compilation (in the "outputAssembly" stage) seems to take a VERY > long time. > I also tried "-native-commented 5" with the same result. A setting > of "4" > worked much faster and I have uploaded a file hansel-20091130-1.s > containing > the basic block. > > > That seems to be enough to provide a hint. I think that the issue is > that the function address got placed in %r11, which is a caller save > register. The contents of caller save registers are pushed to memory > immediately before the call instruction, for any register whose content > is live after the call and purged from the register allocation. Of > course, the function address is still live *at* the call instruction, > although it is not live after the call instruction. Small examples seem > to favor %r15 as the register into which the function address is placed, > which is not caller save, and so not susceptible to this issue. It also > fits with small changes near the indirect function call eliminating the > segfault; such changes alter the liveness and used registers and > presumably the function address get stored in a non-caller save > register. If this is indeed the source of the issue, then it is simply > a native amd64 codegen bug (and, possibly, a latent x86 codegen bug as > well) and is independent of the target OS; that is, it is not mingw > specific. > -- ---------------------------------------------------------- David Hansel http://www.reactive-systems.com/ OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~hansel/pgp_public_key.txt ---------------------------------------------------------- From jhr at cs.uchicago.edu Sun Dec 6 09:31:22 2009 From: jhr at cs.uchicago.edu (John Reppy) Date: Sun Dec 6 09:31:33 2009 Subject: [MLton] feature request: hash function for Pointer.t Message-ID: Because MLton doesn't provide a way to dynamically define callbacks, I need to manage the a mapping from C pointers to ML functions. I'd like to use a hash table to do this, but there isn't an easy way to compute hash keys from pointers. I'd like to see a function val hash : t -> word added to the MLTON_POINTER signature. As a more long-term request, it would be nice to have something like the GHC notion of stable pointers that would allow ML values to be passed to C code for later supply as arguments to callbacks. - John From matthew.fluet at gmail.com Sun Dec 6 14:29:07 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Sun Dec 6 14:29:12 2009 Subject: [MLton] feature request: hash function for Pointer.t In-Reply-To: References: Message-ID: On Sun, Dec 6, 2009 at 12:31 PM, John Reppy wrote: > Because MLton doesn't provide a way to dynamically define callbacks, I need > to manage > the a mapping from C pointers to ML functions. I'd like to use a hash > table to do > this, but there isn't an easy way to compute hash keys from pointers. I'd > like > to see a function > > val hash : t -> word > > added to the MLTON_POINTER signature. > If you believe that the C address is unique enough to be a hash key, doesn't val hash = fn p => MLton.Pointer.diff(MLton.Pointer.null, p) suffice as a hashing function? In any case, I think any hashing of a pointer would start with its address, so you can do a mixing function from there if you don't think the distribution of C addresses suffice as a hashing key. Admittedly, this is in terms of Word.word, which defaults to a 32-bit value; MLton.Pointer.diff performs the difference computation at the target pointer size, and then does a low-bits projection to a 32-bit word. Still, that ought to be decent for a hash function. Perhaps all of the MLton.Pointer functions should be in terms of LargeWord.word. Also, val MLton.hash : 'a -> Word32.word is a polymorphic hash function that works with any ML value. > As a more long-term request, it would be nice to have something like the > GHC notion > of stable pointers that would allow ML values to be passed to C code for > later supply > as arguments to callbacks. > Yes, some notion of registering an exported ML value would be desirable. -Matthew -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091206/0f81669c/attachment.html From hansel at reactive-systems.com Wed Dec 9 09:05:02 2009 From: hansel at reactive-systems.com (David Hansel) Date: Wed Dec 9 09:05:44 2009 Subject: [MLton] max-heap setting for 64-bit applications Message-ID: <4B1FD8BE.2090801@reactive-systems.com> Hi all, I have a few questions about the max-heap setting for 64-bit applications built by MLton. The code below is an example that is supposed to allocate more and more memory until it is terminated. While it is running, it outputs roughly how much memory it has allocated so far. I compiled the example with MLton (r7353 or r7369, no difference) on 64-bit Windows 7 as a 64-bit application. The computer has 6GB of RAM, of which 3.5-4.0GB are available to the application. If I run the example with "@MLton max-heap 2950M --" run-time parameters then it keeps allocating memory, the numbers printed by the example being roughly in sync with what Windows' Task Manager shows. Around 3GB it is terminated with the message "Out of memory with max heap size 3,093,299,200", which seems like exactly what should happen. However, if I run the example with "@MLton max-heap 3000M --" then around the the same point where it was was stopped before (the program reports roughly 3.05GB of allocated memory), my system starts thrashing and I can see in the Task Manager that the program has allocated around 4.5GB of memory and is still trying to get more. Is this what you would expect? I thought that the max-heap setting would include the amount of additional memory required for garbage collection. Is that true? Or do I always have to just allow half of the system memory for max-heap to make sure to not allocate too much memory? Thanks, David Example code: fun mk10MBArray () = let val n = 2500000 val a = Array.array (n, 0) fun loop i = if i >= n then () else (Array.update (a, i, i); loop (i+1)) in loop 0; a end fun processArray (a, x : IntInf.int) = let val n = Array.length a fun loop (i, x) = if i >= n then x else loop (i+10000, x + (IntInf.fromInt (Array.sub (a, i)))) in loop (0, x) end fun processMem mls = let val n = length mls val x = IntInf.fromInt 0 fun loop (i, x) = if i >= n then x else let val a = List.nth (mls, i) val x = processArray (a, x) in loop (i+1, x) end in loop (0, IntInf.fromInt 0) end fun allocUntilDeath () = let val timer = Timer.startRealTimer () fun loop (mls, i) = let val t = Timer.checkRealTimer timer val mls = (mk10MBArray ())::mls in processMem mls; print (Time.toString t); print ": "; print (Real.toString i); print " GB\n"; loop (mls, i+0.01) end in loop ([], 0.0) end val () = (allocUntilDeath ()) -- ---------------------------------------------------------- David Hansel http://www.reactive-systems.com/ OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~hansel/pgp_public_key.txt ---------------------------------------------------------- From matthew.fluet at gmail.com Thu Dec 10 12:37:38 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Thu Dec 10 12:38:14 2009 Subject: [MLton] max-heap setting for 64-bit applications In-Reply-To: <4B1FD8BE.2090801@reactive-systems.com> References: <4B1FD8BE.2090801@reactive-systems.com> Message-ID: > The code below is an example that is supposed to allocate more > and more memory until it is terminated. While it is running, it > outputs roughly how much memory it has allocated so far. > You could run with "@MLton gc-messages --" to see the actual heap sizes as the program runs. > If I run the example with "@MLton max-heap 2950M --" run-time > parameters then it keeps allocating memory, the numbers printed > by the example being roughly in sync with what Windows' Task > Manager shows. Around 3GB it is terminated with the message > "Out of memory with max heap size 3,093,299,200", > which seems like exactly what should happen. > You can use "@MLton max-heap 2.95G --" and "@MLton max-heap 3G --" as shorter forms to express gigabyte sized heaps. That is, the {max,fixed}-heap options accept a floating-point number (to express fractional portions of a size) and the "G"/"g" size modifier. > However, if I run the example with "@MLton max-heap 3000M --" > then around the the same point where it was was stopped before > (the program reports roughly 3.05GB of allocated memory), my > system starts thrashing and I can see in the Task Manager > that the program has allocated around 4.5GB of memory and is still > trying to get more. > > Is this what you would expect? I thought that the max-heap setting > would include the amount of additional memory required for garbage > collection. Is that true? Or do I always have to just allow > half of the system memory for max-heap to make sure to not allocate > too much memory? > You should run your program with "@MLton gc-messages --" to be sure, but I believe that the behavior you are seeing is explained by the heap resizing policy. There is very little space overhead in the garbage collector and runtime over that consumed by the ML heap. So, the max-heap option only bounds the size of the ML heap (which includes the card/cross map used for generational garbage collection). MLton will never attempt to create a heap that is larger than the max-heap option. However, MLton starts execution with a small heap and allows it to grow in response to the demands of the live data. After each garbage collection, MLton decides whether to grow or shrink the heap in order to have the heap size approximately 8x the live data at the end of the garbage collection, further modified by the constraints of the available ram, max-heap, etc. MLton will never try to *allocate* a heap that is larger than the max-heap option; however, in order to *obtain* a larger heap than the currently allocated heap, MLton may be required to allocate the new (bigger) heap and copy the data from the current heap over. The idea is that the user would like their program to continue running; suffering a little bit of paging during the time that the two heaps are allocated is preferable to aborting with an out-of-memory error. After the copy, the old heap is discarded and total memory usage is again bounded by max-heap. (Note: This is *not* the policy for the copying collection; that is, during a copying collection, we do not have two heaps of max-heap size. Instead, the semi-space size is fixed so as to be 1/2 of max-heap; or, if the live data is more than 1/2 the max-heap, the runtime will use the mark-compact collection.) Obtaining the larger heap from the old heap actually happens in two ways. For platforms that support it, we first attempt to mremap the existing heap to a larger size. If we are able to expand to at least half-way between the current size and the desired size, we take that as acceptable growth, even if it isn't quite the full desired heap size. If we can't expand, or the platform doesn't support mremap, then we attempt to create a heap of the desired size. This, clearly, results in a second heap that is allocated while the existing heap is still allocated. On a 32-bit system, allocating a second 3G+ heap while one heap of significant size is allocated will fail due to the virtual address space limitations; in that situation, and if the program is running with @MLton may-page-heap true --, MLton will attempt to write out the existing heap to a file, deallocate the existing heap, and now allocate the large heap. We hope that by deallocating the existing heap, there is now sufficient contiguous address space to create the heap of sufficient size. On a 64-bit system, there shouldn't ever be any trouble finding another 3G+ portion of the virtual address space while the existing heap is allocated. I suspect that this is what you are seeing. Note, that while Windows doesn't natively support mremap, Wesley implemented a few things that try to mimic mmap/mremap/munamp functionality under Windows. There seems to be some complication with the Windows memory system that requires committing in addition to reserving memory, and some manner of being able to extend an existing map with space before it and after it. I don't understand it. However, I do note that the generic version of mremap (/runtime/platform/mremap.c), which is used by the Windows platforms, starts off trying to allocate an entirely new region of the desired size and copy the existing region into it. (This actually seems to be redundant with the behavior of the garbage collector proper.) The comment states: "Prefer a moving remap -> it results in less mmapped regions". Since such an allocation is likely to never fail on a 64-bit platform, you will (briefly) have multiple regions allocated at the same time, likely to exceed the max-heap setting. Of course, that "(briefly)" is the time it takes to allocate a 3G region and copy the existing (nearly) 3G region; if you have <6G physical memory, there will be paging. As for why things happen differently with the different max-heap settings, I think that you just experience a slightly different heap growth trajectory. This is a simple example, but suppose I compare running with fixed-heap 31M and fixed-heap 33M. And suppose a very simple heap-size doubling policy, starting with a heap of size 2M. So the first run will have heap sizes: 2M, 4M, 8M, 16M, 31M. In order to transition from 16M to 31M, you might required 46M memory (the old and new heaps both allocated at the same time), but if you have enough RAM, you probably won't notice it, because it isn't too much worse than your desired max-heap. The second run will have heap sizes: 2M, 4M, 8M, 16M, 32M, 33M. Now, to transition from 32M to 33M, you require 65M memory, and that probably becomes noticeable, since it significantly exceeds your desired max-heap. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091210/3fcf9005/attachment.htm From wesley at terpstra.ca Thu Dec 10 16:08:15 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Thu Dec 10 16:08:48 2009 Subject: [MLton] max-heap setting for 64-bit applications In-Reply-To: References: <4B1FD8BE.2090801@reactive-systems.com> Message-ID: <162de7480912101608k3b2f8636i4bcec138ed3685cc@mail.gmail.com> On Thu, Dec 10, 2009 at 9:37 PM, Matthew Fluet wrote: > Note, that while Windows doesn't natively support mremap, Wesley > implemented a few things that try to mimic mmap/mremap/munamp functionality > under Windows. There seems to be some complication with the Windows memory > system that requires committing in addition to reserving memory, and some > manner of being able to extend an existing map with space before it and > after it. I don't understand it. When you reserve memory, you get virtual address space that does not overlap any previous reservations. When you commit (part of) that reservation, you get memory that can actually be used without segfaulting. You can freely commit/decommit memory within a reservation, changing the memory that windows must provide to the application. You >cannot< shrink or grow a reservation. While it is possible to reserve+commit in one step, this doesn't ensure that the reservation is exclusive with any existing reservations. This is what caused the sporadic windows GC segfaults some years ago, heaps that overlapped. With respect to mremap, my main concern was supporting heap growth in a restricted virtual memory scenario. eg: you have 900MB allocated and need to grow to 1.2G. This matters when self-compiling MLton on mingw32, for example. To support this, it was necessary to allow a heap to grow 'in-place'. That's why the mremap implementation starts probing for available memory before and after the existing heap. Supposing there is available memory before/after the existing heap, I make new reservations to expand the heap. The windows.c file describes the invariant I used for the memory layout. The invariant I choose makes it safe to free/shrink/grow and be certain the memory was actually allocated to >this< heap. > However, I do note that the generic version of mremap > (/runtime/platform/mremap.c), which is used by the Windows platforms, > starts off trying to allocate an entirely new region of the desired size and > copy the existing region into it. We could revisit this approach in the 64-bit setting. The msdn documentation makes vague warnings about the cost of making too many reservations, but I doubt this really matters. From the point of view of a system with a lot of virtual address space, but not a lot of physical RAM, it might make sense to try to always expand in-place first. Of course, even if mremap tries in-place heap growth first, this might not be possible, and a copy to a new memory location will still thrash. (This actually seems to be redundant with the behavior of the garbage > collector proper.) Perhaps, but I was trying to implement an mremap compatible to the specification of mremap, not how MLton uses mremap. If mremap fails to aquire the desired space, mmap will certainly also fail. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091211/3d21a731/attachment.html From wesley at terpstra.ca Fri Dec 11 07:42:43 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Fri Dec 11 07:43:17 2009 Subject: [MLton] Versioning and hosting Message-ID: <162de7480912110742q54b65920gdd75377870ab8dd3@mail.gmail.com> I received a FTBFS (fails to build from source) debian bug #560623 [1] today, which turns out to be a patch compatibility bug already fixed by Matthew (r7369). (BTW, what the hell?! patch is as old and stable a unix tool as they come; what genius changed its behaviour?) In the process of fixing this I noticed that svn browsing is broken [2]. Also the MLton server still runs woody and hasn't seen a security update in years. Not to mention downloads and svn from it are extremely slow. If there's interest, we could definitely offer some improved hosting in Germany. Before I upload a new version of the debian package and go through the hassle of getting all the ports to build cleanly again (they are currenly all green! [3]), I thought I'd also address bug #559014[4], which is a complaint about the version number I picked for the MLton snapshot; I called it 20091107. Since MLton doesn't use version numbers, I can't really call it 1.99+r7369 to make it clear it's a prerelease snapshot of version 2.0. However, my current version name is a bad choice because it implies that 20091107 was an "official" release. The best I can propose is 20091211+r7369. Suggestions / complaints? Of course, if those two regressions on osx were fixed, maybe we'd have an official release and the point would be moot. BTW, a colleague of mine reports everything works under Snow Leopard except the regressions world5 and real. A small fix was needed which I'll commit shortly. [1] [2] [3] > [4] -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091211/f285e860/attachment.htm From matthew.fluet at gmail.com Fri Dec 11 11:33:11 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Fri Dec 11 11:33:16 2009 Subject: [MLton] Versioning and hosting In-Reply-To: <162de7480912110742q54b65920gdd75377870ab8dd3@mail.gmail.com> References: <162de7480912110742q54b65920gdd75377870ab8dd3@mail.gmail.com> Message-ID: On Fri, Dec 11, 2009 at 10:42 AM, Wesley W. Terpstra wrote: > I received a FTBFS (fails to build from source) debian bug #560623 [1] > today, which turns out to be a patch compatibility bug already fixed by > Matthew (r7369). (BTW, what the hell?! patch is as old and stable a unix > tool as they come; what genius changed its behaviour?) > Agreed that changing patch's behavior is quite bizarre. > In the process of fixing this I noticed that svn browsing is broken [2]. Just a couple days ago, Stephen disabled viewsvn.cgi because something was causing many copies to be spawned and overload the machine. We wanted to check the apache logs before turning it back on. Before I upload a new version of the debian package and go through the > hassle of getting all the ports to build cleanly again (they are currenly > all green! [3]), I thought I'd also address bug #559014[4], which is a complaint about the version number I picked for the MLton > snapshot; I called it 20091107. > > Since MLton doesn't use version numbers, I can't really call it 1.99+r7369 > to make it clear it's a prerelease snapshot of version 2.0. However, my > current version name is a bad choice because it implies that 20091107 was an > "official" release. The best I can propose is 20091211+r7369. Suggestions / > complaints? > Would something as simple as "20091211pre+r7369" be sufficiently indicative of a pre-release? Of course, if those two regressions on osx were fixed, maybe we'd have an > official release and the point would be moot. BTW, a colleague of mine > reports everything works under Snow Leopard except the regressions world5 > and real. A small fix was needed which I'll commit shortly. > I think I've got the two performance regressions I mentioned last time solved. I only know of the world5 regression failure on Leopard (x86-darwin); I wasn't sure that it was a failure on Snow Leopard (x86-darwin). What did you need to fix? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091211/f487e5e2/attachment-0001.html From matthew.fluet at gmail.com Fri Dec 11 11:44:59 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Fri Dec 11 11:45:04 2009 Subject: [MLton] max-heap setting for 64-bit applications In-Reply-To: <162de7480912101608k3b2f8636i4bcec138ed3685cc@mail.gmail.com> References: <4B1FD8BE.2090801@reactive-systems.com> <162de7480912101608k3b2f8636i4bcec138ed3685cc@mail.gmail.com> Message-ID: On Thu, Dec 10, 2009 at 7:08 PM, Wesley W. Terpstra wrote: > On Thu, Dec 10, 2009 at 9:37 PM, Matthew Fluet wrote: > >> Note, that while Windows doesn't natively support mremap, Wesley >> implemented a few things that try to mimic mmap/mremap/munamp functionality >> under Windows. There seems to be some complication with the Windows memory >> system that requires committing in addition to reserving memory, and some >> manner of being able to extend an existing map with space before it and >> after it. I don't understand it. > > > When you reserve memory, you get virtual address space that does not > overlap any previous reservations. When you commit (part of) that > reservation, you get memory that can actually be used without segfaulting. > That part makes sense; I was simply noting that you go to some lengths to use this facility in an efficient manner and I wasn't familiar enough with that code to change its behavior. Of course, even if mremap tries in-place heap growth first, this might not > be possible, and a copy to a new memory location will still thrash. > Absolutely. (This actually seems to be redundant with the behavior of the garbage >> collector proper.) > > > Perhaps, but I was trying to implement an mremap compatible to the > specification of mremap, not how MLton uses mremap. If mremap fails to > aquire the desired space, mmap will certainly also fail. > Fair enough. Although, and I freely admit that this is not necessarily part of the mremap specification, on Linux, the mremap function is described as using the Linux page table scheme to efficiently change the mapping between virtual addresses and (physical) memory pages. It's purpose is to be more efficient than allocating a new map and copying. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091211/b4e7695a/attachment.htm From hansel at reactive-systems.com Fri Dec 11 10:15:55 2009 From: hansel at reactive-systems.com (David Hansel) Date: Fri Dec 11 12:26:20 2009 Subject: [MLton] max-heap setting for 64-bit applications In-Reply-To: References: <4B1FD8BE.2090801@reactive-systems.com> Message-ID: <4B228C5B.9010506@reactive-systems.com> Hi Matthew and Wesley, Thanks for the detailed description on MLton's memory usage. This does explain what is going on. I ran the example with "gc-messages max-heap 3000M" and I see the following before the trashing starts: [GC: Growing heap at 0x0000000080000000 of size 3,062,628,352 bytes (+ 23,986,176 bytes card/cross map),] [GC: to desired size of 3,121,217,536 bytes and minimum size of 3,070,031,512 bytes.] When running with the "max-heap 2950M" setting, all the other messages are the same but the final "growing" operation does not happen ("Out of memory" is reported). This seems to reflect exactly what Matthew suggested, MLton is trying to grow the heap to its maximum size which causes the thrashing. My question now comes from a practical point of view: how can I (if possible) avoid this from happening to customers using our application? Of course we are trying to make our code as memory efficient as possible but in some cases a scenario like this (where a LOT of memory is being used) can occur. In that case I would prefer it if the application died with an "Out of memory" error as opposed to bringing the system to a standstill. It seems like one strategy would be that keepomg the "max-heap" setting slightly under half the available physical memory should avoid the case where we're already using about 50% and then have to create another heap of the same size. Another observation I've made is that using "fixed-heap" instead of "max-heap" seems to avoid the thrashing (unless I set the size unreasonably high, like 5GB on my 6GB machine running a number of other applications). I had somewhat expected for MLton to immediately allocate the whole heap but (according to the task manager) that did not happen. Are there any other drawbacks about using fixed-heap? Do you have any other suggestions? Thanks and best regards, David Matthew Fluet wrote: > > The code below is an example that is supposed to allocate more > and more memory until it is terminated. While it is running, it > outputs roughly how much memory it has allocated so far. > > > You could run with "@MLton gc-messages --" to see the actual heap sizes > as the program runs. > > > If I run the example with "@MLton max-heap 2950M --" run-time > parameters then it keeps allocating memory, the numbers printed > by the example being roughly in sync with what Windows' Task > Manager shows. Around 3GB it is terminated with the message > "Out of memory with max heap size 3,093,299,200", > which seems like exactly what should happen. > > > You can use "@MLton max-heap 2.95G --" and "@MLton max-heap 3G --" as > shorter forms to express gigabyte sized heaps. That is, the > {max,fixed}-heap options accept a floating-point number (to express > fractional portions of a size) and the "G"/"g" size modifier. > > > However, if I run the example with "@MLton max-heap 3000M --" > then around the the same point where it was was stopped before > (the program reports roughly 3.05GB of allocated memory), my > system starts thrashing and I can see in the Task Manager > that the program has allocated around 4.5GB of memory and is still > trying to get more. > > Is this what you would expect? I thought that the max-heap setting > would include the amount of additional memory required for garbage > collection. Is that true? Or do I always have to just allow > half of the system memory for max-heap to make sure to not allocate > too much memory? > > > You should run your program with "@MLton gc-messages --" to be sure, but > I believe that the behavior you are seeing is explained by the heap > resizing policy. There is very little space overhead in the garbage > collector and runtime over that consumed by the ML heap. So, the > max-heap option only bounds the size of the ML heap (which includes the > card/cross map used for generational garbage collection). MLton will > never attempt to create a heap that is larger than the max-heap option. > > However, MLton starts execution with a small heap and allows it to grow > in response to the demands of the live data. After each garbage > collection, MLton decides whether to grow or shrink the heap in order to > have the heap size approximately 8x the live data at the end of the > garbage collection, further modified by the constraints of the available > ram, max-heap, etc. MLton will never try to *allocate* a heap that is > larger than the max-heap option; however, in order to *obtain* a larger > heap than the currently allocated heap, MLton may be required to > allocate the new (bigger) heap and copy the data from the current heap > over. The idea is that the user would like their program to continue > running; suffering a little bit of paging during the time that the two > heaps are allocated is preferable to aborting with an out-of-memory > error. After the copy, the old heap is discarded and total memory usage > is again bounded by max-heap. (Note: This is *not* the policy for the > copying collection; that is, during a copying collection, we do not have > two heaps of max-heap size. Instead, the semi-space size is fixed so as > to be 1/2 of max-heap; or, if the live data is more than 1/2 the > max-heap, the runtime will use the mark-compact collection.) > > Obtaining the larger heap from the old heap actually happens in two > ways. For platforms that support it, we first attempt to mremap the > existing heap to a larger size. If we are able to expand to at least > half-way between the current size and the desired size, we take that as > acceptable growth, even if it isn't quite the full desired heap size. > If we can't expand, or the platform doesn't support mremap, then we > attempt to create a heap of the desired size. This, clearly, results in > a second heap that is allocated while the existing heap is still > allocated. On a 32-bit system, allocating a second 3G+ heap while one > heap of significant size is allocated will fail due to the virtual > address space limitations; in that situation, and if the program is > running with @MLton may-page-heap true --, MLton will attempt to write > out the existing heap to a file, deallocate the existing heap, and now > allocate the large heap. We hope that by deallocating the existing > heap, there is now sufficient contiguous address space to create the > heap of sufficient size. On a 64-bit system, there shouldn't ever be > any trouble finding another 3G+ portion of the virtual address space > while the existing heap is allocated. I suspect that this is what you > are seeing. > > Note, that while Windows doesn't natively support mremap, Wesley > implemented a few things that try to mimic mmap/mremap/munamp > functionality under Windows. There seems to be some complication with > the Windows memory system that requires committing in addition to > reserving memory, and some manner of being able to extend an existing > map with space before it and after it. I don't understand it. However, > I do note that the generic version of mremap > (/runtime/platform/mremap.c), which is used by the Windows > platforms, starts off trying to allocate an entirely new region of the > desired size and copy the existing region into it. (This actually seems > to be redundant with the behavior of the garbage collector proper.) The > comment states: "Prefer a moving remap -> it results in less mmapped > regions". Since such an allocation is likely to never fail on a 64-bit > platform, you will (briefly) have multiple regions allocated at the same > time, likely to exceed the max-heap setting. Of course, that > "(briefly)" is the time it takes to allocate a 3G region and copy the > existing (nearly) 3G region; if you have <6G physical memory, there will > be paging. > > As for why things happen differently with the different max-heap > settings, I think that you just experience a slightly different heap > growth trajectory. This is a simple example, but suppose I compare > running with fixed-heap 31M and fixed-heap 33M. And suppose a very > simple heap-size doubling policy, starting with a heap of size 2M. So > the first run will have heap sizes: 2M, 4M, 8M, 16M, 31M. In order to > transition from 16M to 31M, you might required 46M memory (the old and > new heaps both allocated at the same time), but if you have enough RAM, > you probably won't notice it, because it isn't too much worse than your > desired max-heap. The second run will have heap sizes: 2M, 4M, 8M, 16M, > 32M, 33M. Now, to transition from 32M to 33M, you require 65M memory, > and that probably becomes noticeable, since it significantly exceeds > your desired max-heap. -- ---------------------------------------------------------- David Hansel http://www.reactive-systems.com/ OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~hansel/pgp_public_key.txt ---------------------------------------------------------- From matthew.fluet at gmail.com Fri Dec 11 12:32:45 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Fri Dec 11 12:32:49 2009 Subject: [MLton] max-heap setting for 64-bit applications In-Reply-To: <4B228C5B.9010506@reactive-systems.com> References: <4B1FD8BE.2090801@reactive-systems.com> <4B228C5B.9010506@reactive-systems.com> Message-ID: On Fri, Dec 11, 2009 at 1:15 PM, David Hansel wrote: > Thanks for the detailed description on MLton's memory usage. ?This > does explain what is going on. ?I ran the example with "gc-messages > max-heap 3000M" and I see the following before the trashing starts: > > [GC: Growing heap at 0x0000000080000000 of size 3,062,628,352 bytes > (+ 23,986,176 bytes card/cross map),] > [GC: ? ?to desired size of 3,121,217,536 bytes and minimum size > of 3,070,031,512 bytes.] > > When running with the "max-heap 2950M" setting, ?all the other > messages are the same but the final "growing" operation does not > happen ("Out of memory" is reported). > > This seems to reflect exactly what Matthew suggested, ?MLton is > trying to grow the heap to its maximum size which causes the > thrashing. Right, and the annoying bit is that the previous heap was so close to the max heap setting.? Perhaps a reasonable heuristic is that if a desired heap is "close-to" the max-heap size, just round up.? Perhaps 0.75 of max heap?? In the max-heap 3G setting, this could still leave you in the situation where you have a 2.25G allocation and a 3G allocation at the same time to copy.? Or 0.55 of max heap; that could require 1.65G+3G at the time of the copy. > It seems like one strategy would be that keepomg the "max-heap" > setting slightly under half the available physical memory should > avoid the case where we're already using about 50% and then have > to create another heap of the same size. True.? Although, you would really need to use about 50% of the physical memory that you want the MLton process to have access to, else you will page via competition with other processes. > Another observation I've made is that using "fixed-heap" instead > of "max-heap" seems to avoid the thrashing (unless I set the > size unreasonably high, ?like 5GB on my 6GB machine running a > number of other applications). ?I had somewhat expected for MLton > to immediately allocate the whole heap but (according to the > task manager) that did not happen. ?Are there any other drawbacks > about using fixed-heap? It would make sense for fixed-heap to grab the whole heap at once. However, because the semi-spaces of the copying garbage collector are two separate heaps (not two halves of one heap), MLton will initially only allocate one of the heaps and won't allocate the secondary heap until the first (major) garbage collection.? If you run with @MLton fixed-heap 3G --, then MLton will allocate a 1.5G heap, wait for that to fill, perform a major copy-collection into another 1.5G heap (now we have two 1.5G heaps, matching our "fixed-heap" setting).? In your example allocating program, where there is almost no garbage, after the major copy-collection, MLton will see a heap at near capacity and attempt to resize it to 3G (the "fixed-heap" setting), in order to have a larger heap for mark-compact collection.? Going from 1.5G to 3G, even when both are allocated for the copy, probably isn't noticeable as thrashing.? If, after the copying collection, the live data was a small fraction of the heap, then MLton stick with copying collection (using a 1.5G heap), but won't deallocate the secondary heap unless keeping both of them exceeds the available RAM. I don't think that there are any drawbacks to using fixed-heap, as long as you are willing to devote that amount of memory to the process.? Especially on a 64-bit system, if you set the fixed-heap to a proportion of ram such that the jump from 1/2-fixed-heap to fixed-heap isn't noticeable as thrashing, then you should have predictable behavior.? On a 32-bit system, it isn't as nice.? Since we now have lots of systems with >4G memory, but only 4G virtual address space, it would be very nice to run processes with fixed-heap 3.25G. Unfortunately, because of the above described semi-space behavior, what end up happening is that a 1.125G heap is allocated and then it can be difficult to allocate the other 1.125G heap in the address space.? And can be even harder (and, indeed, in the absence of a proper mremap, impossible) to grow the 1.125G heap to 3.25G without the runtime dumping the 1.125G heap to disk.? On those systems, you really want to grab the whole 3.25G up front.? Going to a single contiguous heap, interpreting it as two semi-spaces when using a major copying-collection would be nicer here, because fixed-heap would grab the whole 3.25G up front. From wesley at terpstra.ca Fri Dec 11 16:17:34 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Fri Dec 11 16:18:07 2009 Subject: [MLton] max-heap setting for 64-bit applications In-Reply-To: References: <4B1FD8BE.2090801@reactive-systems.com> <4B228C5B.9010506@reactive-systems.com> Message-ID: <162de7480912111617o66774228l32a655ec5776395f@mail.gmail.com> Stepping outside of the current discussion, a matter of practicality: is the paging really that bad? AFAIK, it is a memcpy running between the two heaps which is sequential access. It shouldn't take longer than it would take to copy a 3GB file on the disk. If it *is* taking longer, then we need to add a hint to the windows VM to tell it that we will be doing sequential access before the memcpy, then flip it back to random-access mode. On Fri, Dec 11, 2009 at 9:32 PM, Matthew Fluet wrote: > Right, and the annoying bit is that the previous heap was so close to > the max heap setting. Perhaps a reasonable heuristic is that if a > desired heap is "close-to" the max-heap size, just round up. Perhaps > 0.75 of max heap? In the max-heap 3G setting, this could still leave > you in the situation where you have a 2.25G allocation and a 3G > allocation at the same time to copy. Or 0.55 of max heap; that could > require 1.65G+3G at the time of the copy. > I would be against yet another special case in the sizing rules. Any cutoff we pick is going to fail for someone else the same way, while artificially restricting the memory growth for others. His problem would be (mostly) fixed if we flipped windows mremap to only move if growth fails. I have a higher-level solution proposal: MLton.GC already has hooks that are executed after a GC to implement finalizers. Expose these to the user. If a user knows his application only consumes X memory on an error condition, he can test for this after a GC and terminate with an "Out of Memory" error as desired. > It seems like one strategy would be that keepomg the "max-heap" > > setting slightly under half the available physical memory should > > avoid the case where we're already using about 50% and then have > > to create another heap of the same size. > > True. Although, you would really need to use about 50% of the > physical memory that you want the MLton process to have access to, > else you will page via competition with other processes. > I think this isn't quite right. You get paging when the working set of the other applications + heap of MLton program > RAM. If his heap stays small, there is no thrashing. Once it gets big, there wilt be thrashing iff the working set of apps + max-heap > RAM. Keeping the max-heap size warm wouldn't help, only cause more thrashing sooner. > Going to a single > contiguous heap, interpreting it as two semi-spaces when using a major > copying-collection would be nicer here, because fixed-heap would grab > the whole 3.25G up front. > Yes. the mremap function is described as using the Linux page table scheme to > efficiently change the mapping between virtual addresses and (physical) > memory pages. It's purpose is to be more efficient than allocating a new > map and copying. > If I could ... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091212/890f9aba/attachment.html From henry.cejtin at sbcglobal.net Fri Dec 11 19:56:02 2009 From: henry.cejtin at sbcglobal.net (Henry Cejtin) Date: Fri Dec 11 19:56:37 2009 Subject: [MLton] max-heap setting for 64-bit applications In-Reply-To: <162de7480912111617o66774228l32a655ec5776395f@mail.gmail.com> References: <4B1FD8BE.2090801@reactive-systems.com> <4B228C5B.9010506@reactive-systems.com> <162de7480912111617o66774228l32a655ec5776395f@mail.gmail.com> Message-ID: <613769.26989.qm@web82403.mail.mud.yahoo.com> Is GC really sequential in terms of memory access? Certainly the writes to new space are, but not the reads from old space, nor the writes to old space (to set the forwarding pointer). I have a vague recollection of Macsyma on the VAX (before even my time), which no doubt used a mark/sweep collector, and they used vadvise to set random behaviour (I thought) during GC. With regards to the max-heap argument, it would be really good to have an argument which is the maximum amount of memory MLton can use, or as close as we can come to that. If it tries to grow bigger, even if only temporarily to grow a heap, and can't grow it by any other allowed means, it should fail with out of memory. ________________________________ From: Wesley W. Terpstra To: Matthew Fluet Cc: mlton@mlton.org Sent: Fri, December 11, 2009 6:17:34 PM Subject: Re: [MLton] max-heap setting for 64-bit applications Stepping outside of the current discussion, a matter of practicality: is the paging really that bad? AFAIK, it is a memcpy running between the two heaps which is sequential access. It shouldn't take longer than it would take to copy a 3GB file on the disk. If it *is* taking longer, then we need to add a hint to the windows VM to tell it that we will be doing sequential access before the memcpy, then flip it back to random-access mode. On Fri, Dec 11, 2009 at 9:32 PM, Matthew Fluet wrote: Right, and the annoying bit is that the previous heap was so close to >> >the max heap setting. Perhaps a reasonable heuristic is that if a >>desired heap is "close-to" the max-heap size, just round up. Perhaps >>0.75 of max heap? In the max-heap 3G setting, this could still leave >>you in the situation where you have a 2.25G allocation and a 3G >>allocation at the same time to copy. Or 0.55 of max heap; that could >>require 1.65G+3G at the time of the copy. > I would be against yet another special case in the sizing rules. Any cutoff we pick is going to fail for someone else the same way, while artificially restricting the memory growth for others. His problem would be (mostly) fixed if we flipped windows mremap to only move if growth fails. I have a higher-level solution proposal: MLton.GC already has hooks that are executed after a GC to implement finalizers. Expose these to the user. If a user knows his application only consumes X memory on an error condition, he can test for this after a GC and terminate with an "Out of Memory" error as desired. > It seems like one strategy would be that keepomg the "max-heap" >>> setting slightly under half the available physical memory should >>> avoid the case where we're already using about 50% and then have >>> to create another heap of the same size. > >True. Although, you would really need to use about 50% of the >>physical memory that you want the MLton process to have access to, >>else you will page via competition with other processes. > I think this isn't quite right. You get paging when the working set of the other applications + heap of MLton program > RAM. If his heap stays small, there is no thrashing. Once it gets big, there wilt be thrashing iff the working set of apps + max-heap > RAM. Keeping the max-heap size warm wouldn't help, only cause more thrashing sooner. Going to a single >>contiguous heap, interpreting it as two semi-spaces when using a major >>copying-collection would be nicer here, because fixed-heap would grab >>the whole 3.25G up front. > Yes. the mremap function is described as >using the Linux page table scheme to efficiently change the mapping >between virtual addresses and (physical) memory pages. It's purpose is >to be more efficient than allocating a new map and copying. > If I could ... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091211/a48f953c/attachment.htm From matthew.fluet at gmail.com Fri Dec 11 23:28:21 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Fri Dec 11 23:28:55 2009 Subject: [MLton] Re: [MLton-commit] r7381 In-Reply-To: References: Message-ID: Are you sure this is necessary? John Reppy has been compiling and running MLton under Snow Leopard for a couple of months and only required changing #include to #include (r7231-7232). On Fri, Dec 11, 2009 at 7:52 PM, Wesley Terpstra wrote: > Under Snow Leopard, the ucontext structure is only available if > _XOPEN_SOURCE is defined. > > > ---------------------------------------------------------------------- > > U ? mlton/trunk/runtime/platform/darwin.h > > ---------------------------------------------------------------------- > > Modified: mlton/trunk/runtime/platform/darwin.h > =================================================================== > --- mlton/trunk/runtime/platform/darwin.h ? ? ? 2009-12-11 19:51:40 UTC (rev 7380) > +++ mlton/trunk/runtime/platform/darwin.h ? ? ? 2009-12-12 00:52:19 UTC (rev 7381) > @@ -1,3 +1,5 @@ > +#define _XOPEN_SOURCE > + > ?#include > ?#include > ?#include > > > _______________________________________________ > MLton-commit mailing list > MLton-commit@mlton.org > http://mlton.org/mailman/listinfo/mlton-commit > From matthew.fluet at gmail.com Fri Dec 11 23:48:17 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Fri Dec 11 23:48:51 2009 Subject: [MLton] max-heap setting for 64-bit applications In-Reply-To: <162de7480912111617o66774228l32a655ec5776395f@mail.gmail.com> References: <4B1FD8BE.2090801@reactive-systems.com> <4B228C5B.9010506@reactive-systems.com> <162de7480912111617o66774228l32a655ec5776395f@mail.gmail.com> Message-ID: On Fri, Dec 11, 2009 at 7:17 PM, Wesley W. Terpstra wrote: > Stepping outside of the current discussion, a matter of practicality: is the > paging really that bad? AFAIK, it is a memcpy running between the two heaps > which is sequential access. It shouldn't take longer than it would take to > copy a 3GB file on the disk. > > If it *is* taking longer, then we need to add a hint to the windows VM to > tell it that we will be doing sequential access before the memcpy, then flip > it back to random-access mode. It isn't quite sequential access. See growHeap in /runtime/gc/heap.c. We actually perform the copy from back to front, with a memcpy (presumably doing sequential access) for each 32M chunk. The reason for starting the copy from the back is that we can shrink the old heap as we go. > On Fri, Dec 11, 2009 at 9:32 PM, Matthew Fluet > wrote: >> >> Right, and the annoying bit is that the previous heap was so close to >> the max heap setting.? Perhaps a reasonable heuristic is that if a >> desired heap is "close-to" the max-heap size, just round up.? Perhaps >> 0.75 of max heap?? In the max-heap 3G setting, this could still leave >> you in the situation where you have a 2.25G allocation and a 3G >> allocation at the same time to copy.? Or 0.55 of max heap; that could >> require 1.65G+3G at the time of the copy. > > I would be against yet another special case in the sizing rules. Any cutoff > we pick is going to fail for someone else the same way, while artificially > restricting the memory growth for others. His problem would be (mostly) > fixed if we flipped windows mremap to only move if growth fails. That assumes that the inplace growth can succeed. (Agreed that on amd64, there is a good chance that it will succeed.) If the inplace growth cannot succeed, then runtime will attempt a moving growth (via an explicit create and copy), and we still overcommit memory (relative to the max-heap setting). > I have a higher-level solution proposal: MLton.GC already has hooks that are > executed after a GC to implement finalizers. Expose these to the user. If a > user knows his application only consumes X memory on an error condition, he > can test for this after a GC and terminate with an "Out of Memory" error as > desired. You still need to get out of the GC before the GC signal handler runs. >> the mremap function is described as using the Linux page table scheme to >> efficiently change the mapping between virtual addresses and (physical) >> memory pages.? It's purpose is to be more efficient than allocating a new >> map and copying. > > If I could ... I guess my point is that the way that you indicate that you aren't more efficient than alloc/copy is by not providing mremap. Everything else in the generic implementation with attempting the in place expand is more efficient; it is just the starting off with the alloc/copy that doesn't seem to make sense. From wesley at terpstra.ca Sat Dec 12 05:10:56 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sat Dec 12 05:11:31 2009 Subject: [MLton] Re: [MLton-commit] r7381 In-Reply-To: References: Message-ID: <162de7480912120510o40a9189ay726520b040156853@mail.gmail.com> My colleague reported this error message:/usr/include/ucontext.h:42:2: error: #error ucontext routines are deprecated, and require _XOPEN_SOURCE to be defined" ... then he added the #define and it worked. I don't have a Snow Leopard system, but he got told me that he got this error on the newest svn/HEAD. I'll get him to double-check on Monday and send the real regression log. I notice that the error message explicitly mentions /usr/include/ucontext.h not sys/ucontext.h. As he was bootstrapping from your last experimental upload, he had to change its included darwin.h as the first step. It's possible that if he'd instead changed the installed darwin.h to use sys/ucontext.h it would also have worked. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091212/4fcaf14f/attachment.html From wesley at terpstra.ca Sat Dec 12 05:24:29 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sat Dec 12 05:25:05 2009 Subject: [MLton] max-heap setting for 64-bit applications In-Reply-To: References: <4B1FD8BE.2090801@reactive-systems.com> <4B228C5B.9010506@reactive-systems.com> <162de7480912111617o66774228l32a655ec5776395f@mail.gmail.com> Message-ID: <162de7480912120524m3d27e854x333460efc628fb79@mail.gmail.com> On Sat, Dec 12, 2009 at 8:48 AM, Matthew Fluet wrote: > It isn't quite sequential access. Sorry. You're both wrong. ;) The mremap code does a memcpy, which is 100% sequential. That's the only transition between the two heaps on Windows -> inside mremap. It allocates the new heap, memcopies them, then frees the old heap. The only time the GC touches it is when it fixes up the addresses. > I have a higher-level solution proposal: MLton.GC already has hooks that > are > > executed after a GC to implement finalizers. Expose these to the user. If > a > > user knows his application only consumes X memory on an error condition, > he > > can test for this after a GC and terminate with an "Out of Memory" error > as > > desired. > > You still need to get out of the GC before the GC signal handler runs. > Of course, but we are talking about early detection. Under normal operation his program probably uses >much< less than the total RAM. Stopping at 1GB with an error would certainly be earlier than thrashing set in. Right now we are focusing in on a micro-program which just consumes memory and we're talking about how to optimize it. It's a pathological program! We should instead be supporting David's underlying goal: detecting when his program enters run-away RAM consumption. This happens much much sooner than thrashing. > >> the mremap function is described as using the Linux page table scheme to > >> efficiently change the mapping between virtual addresses and (physical) > >> memory pages. It's purpose is to be more efficient than allocating a > new > >> map and copying. > > > > If I could ... > > I guess my point is that the way that you indicate that you aren't > more efficient than alloc/copy is by not providing mremap. Everything > else in the generic implementation with attempting the in place expand > is more efficient; it is just the starting off with the alloc/copy > that doesn't seem to make sense. > As already mentioned, the design goal for windows mremap was not efficiency, but to use more virtual address space. I think both agree that alloc/copy as the last step makes more sense for 64-bit systems to reduce the likelihood of thrashing. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091212/5c41f361/attachment.htm From cleng at dvs.tu-darmstadt.de Sat Dec 12 05:51:48 2009 From: cleng at dvs.tu-darmstadt.de (Christof Leng) Date: Sat Dec 12 05:52:07 2009 Subject: [MLton] Re: [MLton-commit] r7381 In-Reply-To: <162de7480912120510o40a9189ay726520b040156853@mail.gmail.com> References: <162de7480912120510o40a9189ay726520b040156853@mail.gmail.com> Message-ID: <4B239FF4.2020004@dvs.tu-darmstadt.de> Skipped content of type multipart/alternative-------------- next part -------------- testing real 666c666 < 11.59195518 --- > > 11.59195423 669c669 < 0.9962721467 --- > > 0.9962720871 678c678 < 7.544136047 --- > > 7.544136524 713c713 < 0.001230000518 --- > > 0.001230000402 817c817 < 11.59195518 --- > > 11.59195423 821c821 < ~0.9962721467 --- > > ~0.9962720871 830c830 < ~7.544136047 --- > > ~7.544136524 866c866 < ~0.001230000518 --- > > ~0.001230000402 difference with -type-check true testing world5 1,2c1,4 < success < success --- > > fread (0xbffff5ec, 1, 1, _) failed (only read 0; eof). > > (Unknown error: 0) > > unhandled exception: Fail: child failed > > Nonzero exit status. difference with -type-check true From wesley at terpstra.ca Sun Dec 13 07:13:21 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sun Dec 13 07:13:57 2009 Subject: [MLton] Re: [MLton-commit] r7381 In-Reply-To: <4B239FF4.2020004@dvs.tu-darmstadt.de> References: <162de7480912120510o40a9189ay726520b040156853@mail.gmail.com> <4B239FF4.2020004@dvs.tu-darmstadt.de> Message-ID: <162de7480912130713p681fd0c4r25f861573c145c7d@mail.gmail.com> On Sat, Dec 12, 2009 at 2:51 PM, Christof Leng wrote: > I attached the output of make check for the real and world5 tests. I notice the changes in the real regression you posted are the exact reverse of the changes from real.ok to real.x86-darwin.ok. I guess that means this is a regresion only in the sense that darwin/x86 has regressed into linux/x86-like behaviour. :) From matthew.fluet at gmail.com Mon Dec 14 08:03:42 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Mon Dec 14 08:04:16 2009 Subject: [MLton] Re: [MLton-commit] r7385 In-Reply-To: References: Message-ID: Good to hear. Presumably the warning arose from an older MLton compiling (any) sources on Snow Leopard and the warning in being emitted despite the "gcc -w" invocation. -Matthew On Mon, Dec 14, 2009 at 9:11 AM, Wesley Terpstra wrote: > Comfirmed that this is NOT necessary when building svn/HEAD from a similarly new MLton. > > > ---------------------------------------------------------------------- > > U ? mlton/trunk/runtime/platform/darwin.h > > ---------------------------------------------------------------------- > > Modified: mlton/trunk/runtime/platform/darwin.h > =================================================================== > --- mlton/trunk/runtime/platform/darwin.h ? ? ? 2009-12-12 02:09:51 UTC (rev 7384) > +++ mlton/trunk/runtime/platform/darwin.h ? ? ? 2009-12-14 14:11:34 UTC (rev 7385) > @@ -1,5 +1,3 @@ > -#define _XOPEN_SOURCE > - > ?#include > ?#include > ?#include From matthew.fluet at gmail.com Mon Dec 14 08:11:09 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Mon Dec 14 08:11:43 2009 Subject: [MLton] Re: [MLton-commit] r7386 In-Reply-To: References: Message-ID: On Mon, Dec 14, 2009 at 9:36 AM, Wesley Terpstra wrote: > Snow Leopard real regression no longer differs from normal x86. > > > ---------------------------------------------------------------------- > > D mlton/trunk/regression/real.x86-darwin.ok > > ---------------------------------------------------------------------- > > Deleted: mlton/trunk/regression/real.x86-darwin.ok Weird. I always assumed that the differences in the real regression on darwin were due to differences in the implementation of libm. Because MacOSX doesn't run on any pre-SSE2 Intel chips, the MacOSX delivered libm could be compiled to use SSE2 (but using x87 for the calling convention); seems to be no good reason to go to an x87-only implementation for Snow Leopard. Linux, on the other hand, does need to support x87 only chips, so presumably that libm uses x87 instructions exclusively. From matthew.fluet at gmail.com Mon Dec 14 08:44:28 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Mon Dec 14 08:45:08 2009 Subject: [MLton] max-heap setting for 64-bit applications In-Reply-To: <613769.26989.qm@web82403.mail.mud.yahoo.com> References: <4B1FD8BE.2090801@reactive-systems.com> <4B228C5B.9010506@reactive-systems.com> <162de7480912111617o66774228l32a655ec5776395f@mail.gmail.com> <613769.26989.qm@web82403.mail.mud.yahoo.com> Message-ID: On Fri, Dec 11, 2009 at 10:56 PM, Henry Cejtin wrote: > With? regards? to the max-heap argument, it would be really good to have > an argument which is the maximum amount of memory MLton can use,? or? as > close? as we can come to that.? If it tries to grow bigger, even if only > temporarily to grow a heap, and can't? grow? it? by? any? other? allowed > means, it should fail with out of memory. I think the only way one could use such an argument would be to malloc/mmap that whole space at the beginning of the program and never release or resize it. We could still make use of a smaller heap within that reserved space, and let the OS VMM page out the unused pages. (Or, on Windows, use the reserve/commit distinction. On Posix, use the madvise to preemptively indicate page-out/page-in behavior. I think I recall seeing that some *BSDs allow you to get the reserve/commit distinction that Windows provides.) From wesley at terpstra.ca Mon Dec 14 08:47:26 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Dec 14 08:47:59 2009 Subject: [MLton] Re: [MLton-commit] r7386 In-Reply-To: References: Message-ID: <162de7480912140847p13bd9220y7b5171159e537a89@mail.gmail.com> On Mon, Dec 14, 2009 at 5:11 PM, Matthew Fluet wrote: > ... seems to be no good reason to ... for Snow Leopard. Apple? From matthew.fluet at gmail.com Mon Dec 14 08:50:48 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Mon Dec 14 08:50:53 2009 Subject: [MLton] max-heap setting for 64-bit applications In-Reply-To: <162de7480912120524m3d27e854x333460efc628fb79@mail.gmail.com> References: <4B1FD8BE.2090801@reactive-systems.com> <4B228C5B.9010506@reactive-systems.com> <162de7480912111617o66774228l32a655ec5776395f@mail.gmail.com> <162de7480912120524m3d27e854x333460efc628fb79@mail.gmail.com> Message-ID: On Sat, Dec 12, 2009 at 8:24 AM, Wesley W. Terpstra wrote: > On Sat, Dec 12, 2009 at 8:48 AM, Matthew Fluet > wrote: >> >> It isn't quite sequential access. > > Sorry. You're both wrong. ;) The mremap code does a memcpy, which is 100% > sequential. That's the only transition between the two heaps on Windows -> > inside mremap. It allocates the new heap, memcopies them, then frees the old > heap. The only time the GC touches it is when it fixes up the addresses. But, the Windows specific mremap could still fail --- note that the growHeap function demands significant growth from mremap. If that fails, then it attempts the alloc/copy, but allowing for an alloc of a heap down to the minimum size. (Again, perhaps not extremely likely, but in a program that is using FFI and doing C-side allocations that somewhat fragment the virtual address space, I could imagine a 1.25G heap being unable to be (significantly) grown to a heap of size 2.75G, but the alloc/copy succeeds with a heap of 1.3G.) >> >> the mremap function is described as using the Linux page table scheme >> >> to >> >> efficiently change the mapping between virtual addresses and (physical) >> >> memory pages.? It's purpose is to be more efficient than allocating a >> >> new >> >> map and copying. >> > >> > If I could ... >> >> I guess my point is that the way that you indicate that you aren't >> more efficient than alloc/copy is by not providing mremap. ?Everything >> else in the generic implementation with attempting the in place expand >> is more efficient; it is just the starting off with the alloc/copy >> that doesn't seem to make sense. > > As already mentioned, the design goal for windows mremap was not efficiency, > but to use more virtual address space. I think both agree that alloc/copy as > the last step makes more sense for 64-bit systems to reduce the likelihood > of thrashing. I guess the question is whether the large alloc/copy is really more efficient on even a 32-bit system. From matthew.fluet at gmail.com Mon Dec 14 08:54:40 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Mon Dec 14 08:55:14 2009 Subject: [MLton] Re: [MLton-commit] r7386 In-Reply-To: <162de7480912140847p13bd9220y7b5171159e537a89@mail.gmail.com> References: <162de7480912140847p13bd9220y7b5171159e537a89@mail.gmail.com> Message-ID: On Mon, Dec 14, 2009 at 11:47 AM, Wesley W. Terpstra wrote: > On Mon, Dec 14, 2009 at 5:11 PM, Matthew Fluet wrote: >> ... seems to be no good reason to ... for Snow Leopard. > > Apple? The only thing I could think of, and even this is a stretch, is that having dropped PowerPC with Snow Leopard, they don't need to maintain bit-level compatibility with programs over the two architectures. PowerPC presumably uses proper IEEE-754 arithmetic. But, it seems like that would have been quite hard to establish, given that x86 needed to use x87 calling conventions. And, if one did establish it, keeping IEEE-754 arithmetic on x86/amd64 would be ever so much nicer. From wesley at terpstra.ca Mon Dec 14 09:45:20 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Dec 14 09:45:53 2009 Subject: [MLton] max-heap setting for 64-bit applications In-Reply-To: References: <4B1FD8BE.2090801@reactive-systems.com> <4B228C5B.9010506@reactive-systems.com> <162de7480912111617o66774228l32a655ec5776395f@mail.gmail.com> <162de7480912120524m3d27e854x333460efc628fb79@mail.gmail.com> Message-ID: <162de7480912140945rb4a0ad3x57a5dcf99dad2c92@mail.gmail.com> On Mon, Dec 14, 2009 at 5:50 PM, Matthew Fluet wrote: > On Sat, Dec 12, 2009 at 8:24 AM, Wesley W. Terpstra wrote: >> The mremap code does a memcpy, which is 100% sequential. > But, the Windows specific mremap could still fail --- note that the > growHeap function demands significant growth from mremap. ?If that > fails, then it attempts the alloc/copy, but allowing for an alloc of a > heap down to the minimum size. This could only happen if you fragmentted the entire 16TB of VM. My point was that the only swapping he (or anyone else) saw was sequential, and should be as fast as a disk-backed copy. >> As already mentioned, the design goal for windows mremap was not efficiency, >> but to use more virtual address space. I think both agree that alloc/copy as >> the last step makes more sense for 64-bit systems to reduce the likelihood >> of thrashing. > > I guess the question is whether the large alloc/copy is really more > efficient on even a 32-bit system. I bet it's probably always faster to grow in-place. As there have been no issues with the mremap code since I wrote it, I'm confident enough in it to switch it to the default, committed. It might also be worth writing an improved copy/move code-path that allocates as it goes, thus avoiding swapping. ie: (commit/copy/decommit)s in blocks of 4MB at a time, releasing the source reservation as the last step. Not as amazing as a real mremap which doesn't need to copy at all, but at least it would have the same 'trash or not thrash' behaviour. From fw at deneb.enyo.de Mon Dec 14 09:51:13 2009 From: fw at deneb.enyo.de (Florian Weimer) Date: Mon Dec 14 09:51:23 2009 Subject: [MLton] max-heap setting for 64-bit applications In-Reply-To: (Matthew Fluet's message of "Mon, 14 Dec 2009 11:44:28 -0500") References: <4B1FD8BE.2090801@reactive-systems.com> <4B228C5B.9010506@reactive-systems.com> <162de7480912111617o66774228l32a655ec5776395f@mail.gmail.com> <613769.26989.qm@web82403.mail.mud.yahoo.com> Message-ID: <873a3d1m26.fsf@mid.deneb.enyo.de> * Matthew Fluet: > (Or, on Windows, use the reserve/commit distinction. It's there on Linx, too. > On Posix, use the madvise to preemptively indicate page-out/page-in > behavior. Doesn't work. You have to map with PROT_NONE and then use mprotect to make available the parts you need. This is not a published kernel ABI, so it might break in the future. From matthew.fluet at gmail.com Mon Dec 14 09:53:05 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Mon Dec 14 09:53:45 2009 Subject: [MLton] max-heap setting for 64-bit applications In-Reply-To: <162de7480912140945rb4a0ad3x57a5dcf99dad2c92@mail.gmail.com> References: <4B1FD8BE.2090801@reactive-systems.com> <4B228C5B.9010506@reactive-systems.com> <162de7480912111617o66774228l32a655ec5776395f@mail.gmail.com> <162de7480912120524m3d27e854x333460efc628fb79@mail.gmail.com> <162de7480912140945rb4a0ad3x57a5dcf99dad2c92@mail.gmail.com> Message-ID: On Mon, Dec 14, 2009 at 12:45 PM, Wesley W. Terpstra wrote: > On Mon, Dec 14, 2009 at 5:50 PM, Matthew Fluet wrote: >> On Sat, Dec 12, 2009 at 8:24 AM, Wesley W. Terpstra wrote: >>> The mremap code does a memcpy, which is 100% sequential. >> But, the Windows specific mremap could still fail --- note that the >> growHeap function demands significant growth from mremap. ?If that >> fails, then it attempts the alloc/copy, but allowing for an alloc of a >> heap down to the minimum size. > > This could only happen if you fragmentted the entire 16TB of VM. I was thinking about the 32-bit case, in which case fragmenting the 4G VM isn't terribly difficult. > I bet it's probably always faster to grow in-place. As there have been > no issues with the mremap code since I wrote it, I'm confident enough > in it to switch it to the default, committed. > > It might also be worth writing an improved copy/move code-path that > allocates as it goes, thus avoiding swapping. ie: > (commit/copy/decommit)s in blocks of 4MB at a time, releasing the > source reservation as the last step. Not as amazing as a real mremap > which doesn't need to copy at all, but at least it would have the same > 'trash or not thrash' behaviour. On Windows, yes. Reserve the whole target heap size, and during the copy, commit/decommit as we go. I don't think we can get the same kind of behavior on POSIX. From matthew.fluet at gmail.com Mon Dec 14 09:57:51 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Mon Dec 14 09:58:28 2009 Subject: [MLton] max-heap setting for 64-bit applications In-Reply-To: <873a3d1m26.fsf@mid.deneb.enyo.de> References: <4B1FD8BE.2090801@reactive-systems.com> <4B228C5B.9010506@reactive-systems.com> <162de7480912111617o66774228l32a655ec5776395f@mail.gmail.com> <613769.26989.qm@web82403.mail.mud.yahoo.com> <873a3d1m26.fsf@mid.deneb.enyo.de> Message-ID: On Mon, Dec 14, 2009 at 12:51 PM, Florian Weimer wrote: > * Matthew Fluet: > >> (Or, on Windows, use the reserve/commit distinction. > > It's there on Linx, too. > >> On Posix, use the madvise to preemptively indicate page-out/page-in >> behavior. > > Doesn't work. ?You have to map with PROT_NONE and then use mprotect to > make available the parts you need. > > This is not a published kernel ABI, so it might break in the future. Oh, I certainly understand that under-the-hood any VMM is likely to have the reserve/commit distinction; plus plenty more subtleties: sharing of zero-pages, copy-on-write for fork-exec, etc., etc. But, if it isn't exposed in a user-space ABI, then it isn't "there" for us. Windows, on the other hand, *requires* the user to manage the distinction via the published user-space ABI. From fw at deneb.enyo.de Mon Dec 14 10:15:53 2009 From: fw at deneb.enyo.de (Florian Weimer) Date: Mon Dec 14 10:16:02 2009 Subject: [MLton] max-heap setting for 64-bit applications In-Reply-To: (Matthew Fluet's message of "Mon, 14 Dec 2009 12:57:51 -0500") References: <4B1FD8BE.2090801@reactive-systems.com> <4B228C5B.9010506@reactive-systems.com> <162de7480912111617o66774228l32a655ec5776395f@mail.gmail.com> <613769.26989.qm@web82403.mail.mud.yahoo.com> <873a3d1m26.fsf@mid.deneb.enyo.de> Message-ID: <87ljh5zajq.fsf@mid.deneb.enyo.de> * Matthew Fluet: > Oh, I certainly understand that under-the-hood any VMM is likely to > have the reserve/commit distinction; plus plenty more subtleties: > sharing of zero-pages, copy-on-write for fork-exec, etc., etc. But, > if it isn't exposed in a user-space ABI, then it isn't "there" for us. > Windows, on the other hand, *requires* the user to manage the > distinction via the published user-space ABI. Linux exposes the distinction as well if you set vm.overcommit_memory to 2. There's just no ABI guarantee how to reserve part of the address space without having it to count towards the commit limit. The PROT_NONE business seems to work, but most run-time libraries allocating a single heap block get this wrong. From wesley at terpstra.ca Mon Dec 14 10:50:44 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Dec 14 10:51:17 2009 Subject: [MLton] max-heap setting for 64-bit applications In-Reply-To: References: <4B1FD8BE.2090801@reactive-systems.com> <4B228C5B.9010506@reactive-systems.com> <162de7480912111617o66774228l32a655ec5776395f@mail.gmail.com> <162de7480912120524m3d27e854x333460efc628fb79@mail.gmail.com> <162de7480912140945rb4a0ad3x57a5dcf99dad2c92@mail.gmail.com> Message-ID: <162de7480912141050q71e796b4vf7e2e0992a1e7a89@mail.gmail.com> On Mon, Dec 14, 2009 at 6:53 PM, Matthew Fluet wrote: > I was thinking about the 32-bit case, in which case fragmenting the 4G > VM isn't terribly difficult. Ok, then let me address your original comment again: > But, the Windows specific mremap could still fail --- note that the > growHeap function demands significant growth from mremap. If that > fails, then it attempts the alloc/copy, but allowing for an alloc of a > heap down to the minimum size. This sounds like a bad idea, then. The problem applies equally to all platforms (even linux). So, to be clear, we're talking about this situation: memory is available for the minimum size that memory is NOT large enough to accommodate "significant growth" the memory region contains the current mapping Under this case, AFAICT, every system fails to grow the heap, though it should succeed. Both windows and linux would've been fine if the GC had attempted to use mremap backed off to a lower value. I submit that this is a bug which should be fixed. From wesley at terpstra.ca Tue Dec 15 04:21:05 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Dec 15 04:21:39 2009 Subject: [MLton] target-map considered harmful Message-ID: <162de7480912150421m38f30c2cx75d1bab647307470@mail.gmail.com> So, I've use a lot of MLton cross compilers and have been looking at how to get MLton packages that achieve this. I was hoping in the future to be able to type: apt-get install mlton mlton-mingw32 mlton-mipsel ... and then get a mlton capable of compiling to my native system, mingw32, or mipsel targets. This seems quite easy to achieve, except the annoying target-map file is one file. Adding the needed sml/basis/config/c/xxx/c-types.sml and xxx/*.a files in the mlton-mingw32 package is no problem. However, I can't just overwrite target-map. Couldn't we instead have a personality file inside the target folders? eg: cat self/personality i386 linux cat i586-mingw32msvc/personality i386 mingw From huabj at mail.ustc.edu.cn Fri Dec 18 02:02:57 2009 From: huabj at mail.ustc.edu.cn (Baojian Hua) Date: Fri Dec 18 02:03:44 2009 Subject: [MLton] A Possible "Bug"? Message-ID: <11295644.1278481261130577772.JavaMail.coremail@mailweb> Hi, all, I've just read and modified some webpages that are out-of-date due to some reasons. For instance, I've deleted some sentences on http://mlton.org/SMLNJDeviations related to these code: val [x] = [[]]; val _ = {1::x, "one"::x} for that I've tried it in the latest smlnj-110.71, and it seems that it runs smoothly. However, I happened to run this code: val [[x]] = []; MLton reports a warning message, but smlnj-110.71 reports an error (compiler bug?). I'd like to ask here that which behaves correctly? Thanks in advance. -Paul From henry.cejtin at sbcglobal.net Fri Dec 18 06:15:17 2009 From: henry.cejtin at sbcglobal.net (Henry Cejtin) Date: Fri Dec 18 06:15:50 2009 Subject: [MLton] A Possible "Bug"? In-Reply-To: <11295644.1278481261130577772.JavaMail.coremail@mailweb> References: <11295644.1278481261130577772.JavaMail.coremail@mailweb> Message-ID: <508583.61880.qm@web82407.mail.mud.yahoo.com> I would say that MLton is correct and SML/NJ is wrong. The point is that the left hand side of the binding in val [[x]] = [] is a non-exhastive pattern and the right hand side does not match any of the cases. (The type is correct though.) Hence evaluating this should result in the Bind exception being raised. At least in v110.67 of SML/NJ, that is what happens. Perhaps you are being confused by the fact that the exception raising happens at evaluation time. In MLton, that is when you run the executable, but in SML/NJ, it is when you enter the expression to the read-eval-print loop. Note, if SML/NJ did not issue a warning about the binding not being exhaustive then that would be a bug in their code. ----- Original Message ---- From: Baojian Hua To: mlton Sent: Fri, December 18, 2009 4:02:57 AM Subject: [MLton] A Possible "Bug"? ... However, I happened to run this code: val [[x]] = []; MLton reports a warning message, but smlnj-110.71 reports an error (compiler bug?). I'd like to ask here that which behaves correctly? From matthew.fluet at gmail.com Fri Dec 18 09:02:21 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Fri Dec 18 09:02:56 2009 Subject: [MLton] A Possible "Bug"? In-Reply-To: <508583.61880.qm@web82407.mail.mud.yahoo.com> References: <11295644.1278481261130577772.JavaMail.coremail@mailweb> <508583.61880.qm@web82407.mail.mud.yahoo.com> Message-ID: On Fri, Dec 18, 2009 at 9:15 AM, Henry Cejtin wrote: > I ?would ?say ?that ?MLton is correct and SML/NJ is wrong. Agreed, with the caveat noted below. > The point is > that the left hand side of the binding in > ? ?val [[x]] = [] > is a non-exhaustive pattern and the right hand side does not match any of > the ?cases. ?(The type is correct though.) ?Hence evaluating this should > result in the Bind exception being raised. > > At least in v110.67 of SML/NJ, that is what happens. > > Perhaps you are being confused by the fact that ?the ?exception ?raising > happens ?at ?evaluation ?time. ? In ?MLton, ?that ?is ?when ?you run the > executable, but in SML/NJ, it is when you enter the ?expression ?to ?the > read-eval-print loop. There is a subtle difference in SML/NJ's behavior: Standard ML of New Jersey v110.71 [built: Thu Oct 1 09:13:22 2009] - val [[x]] = []; unexpected exception (bug?) in SML/NJ: Bind [nonexhaustive binding failure] raised at: stdIn:1.5-1.15 ../compiler/MiscUtil/print/ppobj.sml:396.20 ../compiler/TopLevel/interact/evalloop.sml:44.55 - val SOME x = NONE; unexpected exception (bug?) in SML/NJ: Bind [nonexhaustive binding failure] raised at: stdIn:1.5-1.18 ../compiler/MiscUtil/print/ppobj.sml:396.20 ../compiler/TopLevel/interact/evalloop.sml:44.55 - val NONE = SOME 1; uncaught exception Bind [nonexhaustive binding failure] raised at: stdIn:1.5-1.18 The last is what I would expect to see in all cases. The others seem to be a case where the the REPL isn't expecting a Bind exception; it seem to be related to patterns that should be binding variables (i.e., they are entered into the static environment), but then are not bound due to the pattern match failure (i.e, they are not entered into the dynamic environment). > Note, ?if ?SML/NJ ?did ?not ?issue a warning about the binding not being > exhaustive then that would be a bug in their code. Except that the Defn states: "However, this warning should not be given when the binding is a component of a top-level declaration...." MLton gives pattern-match warnings in all cases. > ----- Original Message ---- > From: Baojian Hua > To: mlton > Sent: Fri, December 18, 2009 4:02:57 AM > Subject: [MLton] A Possible "Bug"? > > ... > > However, I happened to run this code: > val [[x]] = []; > > MLton reports a warning message, but smlnj-110.71 reports > an error (compiler bug?). I'd like to ask here that which > behaves correctly? > > _______________________________________________ > MLton mailing list > MLton@mlton.org > http://mlton.org/mailman/listinfo/mlton From huabj at mail.ustc.edu.cn Mon Dec 21 21:47:16 2009 From: huabj at mail.ustc.edu.cn (Baojian Hua) Date: Mon Dec 21 21:48:04 2009 Subject: [MLton] A Possible "Bug"? In-Reply-To: References: <11295644.1278481261130577772.JavaMail.coremail@mailweb> <508583.61880.qm@web82407.mail.mud.yahoo.com> Message-ID: <23980370.1519281261460836756.JavaMail.coremail@mailweb> > -----Original E-mail----- > From: "Matthew Fluet" > Sent Time: 2009-12-19 1:02:21 > To: "Henry Cejtin" > Cc: "Baojian Hua" , mlton > Subject: Re: [MLton] A Possible "Bug"? > > On Fri, Dec 18, 2009 at 9:15 AM, Henry Cejtin > wrote: > > I ?would ?say ?that ?MLton is correct and SML/NJ is wrong. > > Agreed, with the caveat noted below. > > > The point is > > that the left hand side of the binding in > > ? ?val [[x]] = [] > > is a non-exhaustive pattern and the right hand side does not match any of > > the ?cases. ?(The type is correct though.) ?Hence evaluating this should > > result in the Bind exception being raised. > > > > At least in v110.67 of SML/NJ, that is what happens. > > > > Perhaps you are being confused by the fact that ?the ?exception ?raising > > happens ?at ?evaluation ?time. ? In ?MLton, ?that ?is ?when ?you run the > > executable, but in SML/NJ, it is when you enter the ?expression ?to ?the > > read-eval-print loop. > > There is a subtle difference in SML/NJ's behavior: > > Standard ML of New Jersey v110.71 [built: Thu Oct 1 09:13:22 2009] > - val [[x]] = []; > > unexpected exception (bug?) in SML/NJ: Bind [nonexhaustive binding failure] > raised at: stdIn:1.5-1.15 > ../compiler/MiscUtil/print/ppobj.sml:396.20 > ../compiler/TopLevel/interact/evalloop.sml:44.55 > > - val SOME x = NONE; > > unexpected exception (bug?) in SML/NJ: Bind [nonexhaustive binding failure] > raised at: stdIn:1.5-1.18 > ../compiler/MiscUtil/print/ppobj.sml:396.20 > ../compiler/TopLevel/interact/evalloop.sml:44.55 > > - val NONE = SOME 1; > > uncaught exception Bind [nonexhaustive binding failure] > raised at: stdIn:1.5-1.18 > > The last is what I would expect to see in all cases. The others seem > to be a case where the the REPL isn't expecting a Bind exception; it > seem to be related to patterns that should be binding variables (i.e., > they are entered into the static environment), but then are not bound > due to the pattern match failure (i.e, they are not entered into the > dynamic environment). > Yes, I'd tried it. And reading the compiler output: unexpected exception (bug?) in SML/NJ: Bind [nonexhaustive binding failure] it seems that SML/NJ treats it a bug in itself, rather than a Binding exception. (Though I've not read its source code.) > > Note, ?if ?SML/NJ ?did ?not ?issue a warning about the binding not being > > exhaustive then that would be a bug in their code. > > Except that the Defn states: "However, this warning should not be > given when the binding is a component of a top-level declaration...." > MLton gives pattern-match warnings in all cases. > > > > ----- Original Message ---- > > From: Baojian Hua > > To: mlton > > Sent: Fri, December 18, 2009 4:02:57 AM > > Subject: [MLton] A Possible "Bug"? > > > > ... > > > > However, I happened to run this code: > > val [[x]] = []; > > > > MLton reports a warning message, but smlnj-110.71 reports > > an error (compiler bug?). I'd like to ask here that which > > behaves correctly? > > > > _______________________________________________ > > MLton mailing list > > MLton@mlton.org > > http://mlton.org/mailman/listinfo/mlton