From wesley at terpstra.ca Sun Nov 2 07:40:46 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sun Nov 2 07:40:51 2008 Subject: [MLton] Memory layout debugging Message-ID: <162de7480811020740n276e8ecaub1dbd7b46917ec89@mail.gmail.com> Something I would find very useful is to be able to get a graphical picture of the memory used by my application from a specified root. MLton.size is nice, but I was thinking of a similar function that instead created a graphviz dot format file. Nodes in the graph are addresses in the heap, and arcs correspond to objptrs. I bet this would be a very easy picture for 'dot' to render as cycles are not that common in SML programs' memory. This part I am sure I'd have no trouble implementing. To make this picture easier to understand, it would be helpful if objects had their point of allocation or type marked. I know that -profile alloc is able to record the # of allocations at a point in the program. However, I also understand that there may be several points in the program which allocate a given objectType. What sort of sensible information could I provide based on the object type? The other question is how to get this information back to ML. I could just spew the graphviz data to stderr, but this seems crude. I could also allocate a C-side string that I keep growing as the dotfile grows, then convert this string to an ML-string inside the basis library. What's best? I think that a tool like this would not only help the working programmer, but would also help MLton development pick the best places to make representation optimizations. Thoughts? From wesley at terpstra.ca Sun Nov 2 16:46:16 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sun Nov 2 16:46:21 2008 Subject: [MLton] Re: Memory layout debugging In-Reply-To: <162de7480811020740n276e8ecaub1dbd7b46917ec89@mail.gmail.com> References: <162de7480811020740n276e8ecaub1dbd7b46917ec89@mail.gmail.com> Message-ID: <162de7480811021646w692f30l1d6acb74e49bceb9@mail.gmail.com> I've implemented it. Patch attached. It's pretty cool, I think. :-) You use it with MLton.dot (filename, title, obj option). If you pass SOME object, it traces starting only from that root. If you pass NONE, it traces the entire heap. The dot output code adds 4k to the runtime (and hence executable) size. It costs dot about 230* the memory of the dumped core to render it. I've had the best luck outputting svg files. dot seems to work well at that. I suspect I'll have to give up on the cool html-style dot files to reduce dot's memory footprint. On Sun, Nov 2, 2008 at 4:40 PM, Wesley W. Terpstra wrote: > To make this picture easier to understand, it would be helpful if > objects had their point of allocation or type marked. I know that > -profile alloc is able to record the # of allocations at a point in > the program. However, I also understand that there may be several > points in the program which allocate a given objectType. What sort of > sensible information could I provide based on the object type? At the moment I just put the object type as a number below the object. I would like to report something more useful here. > The other question is how to get this information back to ML. I could > just spew the graphviz data to stderr, but this seems crude. I could > also allocate a C-side string that I keep growing as the dotfile > grows, then convert this string to an ML-string inside the basis > library. What's best? I went with writing to a file specified by the user. Keeping it in RAM wouldn't work because the dot code is much larger than the heap. The output format I used isn't the greatest, but it's enough to play around with. Enjoy. -------------- next part -------------- A non-text attachment was scrubbed... Name: dot.patch Type: text/x-patch Size: 20605 bytes Desc: not available Url : http://mlton.org/pipermail/mlton/attachments/20081103/6773bf9b/dot-0001.bin From Nicolas.Bertolotti at mathworks.fr Mon Nov 3 14:04:50 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Mon Nov 3 14:05:27 2008 Subject: [MLton] Compilation crash with FFI on Windows with latest SVN revision (r6960) In-Reply-To: <162de7480810240811s3b412e81sc247d8593de9ba64@mail.gmail.com> References: <8320D98DA9A5C54C926D397795FE7CEA2D98BAB46C@EXCHANGE-UK.ad.mathworks.com> <162de7480810220642s4fad0821oca472b9ffe586b94@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA2D98BAB4D2@EXCHANGE-UK.ad.mathworks.com> <8320D98DA9A5C54C926D397795FE7CEA2D98BAB579@EXCHANGE-UK.ad.mathworks.com> <162de7480810221309u20d61b3fs115d3cd7b83a17f0@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA2D98BAB82E@EXCHANGE-UK.ad.mathworks.com> <162de7480810240811s3b412e81sc247d8593de9ba64@mail.gmail.com> Message-ID: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDB@EXCHANGE-UK.ad.mathworks.com> Case closed ... I finally identified that the issue could also occur with older version of MLton and that it was located in the DLL itself (unfortunately, the debug messages that were sent to stderr from the DLL did not get printed to the screen). It seems that the issue was highly related to the execution speed of the binary and that some improvements in the memory handling and in our code itself have contributed to increase the frequency. Thanks for your help. Nicolas > -----Original Message----- > From: Wesley W. Terpstra [mailto:wesley@terpstra.ca] > Sent: Friday, October 24, 2008 5:11 PM > To: Nicolas Bertolotti > Cc: mlton@mlton.org > Subject: Re: [MLton] Compilation crash with FFI on Windows with latest SVN > revision (r6960) > > On Fri, Oct 24, 2008 at 9:45 AM, Nicolas Bertolotti > wrote: > >> As the old > >> _import didn't have public/external, you would have problems if you > >> tried to import a variable (as opposed to a function) or take the > >> address of functions. The new syntax at least gives you a compile > >> error as opposed to a bug that lurks in wait. =) > > > > So, do you mean that the fact that I was able to import functions from > .c files and from DLLs without specifying a scope was just a matter of > being lucky? > > Lucky in the sense that you didn't do any of the things that would > have caused problems. > > >> Is this code covered by an NDA? Can we see it? > > > > Unfortunately, I can't provide it. Then, I understand that I have to > debug by myself. That is why I am only asking for some useful hints to > investigate it. > > Well, I would try creating my own DLL that proxies calls to the DLL > which hangs. Then I'd stick print statements before passing the call > to the real DLL and after it returns. I'd especially make sure to > print the arguments so I could see if something strange was being > passed. > > With this DLL wrapper, I would compare traces between the broken and > working programs. If the call sequence or parameters is different at > some point before the crash, I would investigate why. > > Since you say the problem is intermittent, I would also be concerned > about garbage collection. Do you only ever call from SML->C or do you > also sometimes call from C->SML? It's my understanding that if you > call from C->SML, any pointers you had into the ML heap (vectors or > strings you received) are invalidated. > > Another thing to check would be to run the program with '@MLton > gc-messages --'. You could then compare the good and bad programs to > see if garbage collection is happening around the hang of the bad > program, but not the good program. > > >> You could also try -codegen c and/or -codegen bytecode. > > The issue does not reproduce with -codegen c. > > I'm not sure what this means. > > > The same code compiled using MLton 2007 works fine. On the other hand, > the code freezes within the DLL itself (when it calls a function for which > I don't even have access to the source code), not at the FFI call. > > If the problem is memory corruption, you might try MLton revision > r6940. The subsequent revision changed how memory is managed under > win32. > > > Then, it may be some kind of latent bug that has been revealed with the > new version of MLton but not a MLton bug. > > This is my personal hope. ;-) From Nicolas.Bertolotti at mathworks.fr Mon Nov 3 14:22:36 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Mon Nov 3 14:23:14 2008 Subject: [MLton] Segmentation fault in 32 bit mode on a 6 GB machine with ram-slop 0.85 Message-ID: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDC@EXCHANGE-UK.ad.mathworks.com> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: image002.gif Type: image/gif Size: 4473 bytes Desc: image002.gif Url : http://mlton.org/pipermail/mlton/attachments/20081103/e30266ca/image002-0001.gif From wesley at terpstra.ca Mon Nov 3 16:10:37 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Nov 3 16:10:41 2008 Subject: [MLton] Segmentation fault in 32 bit mode on a 6 GB machine with ram-slop 0.85 In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDC@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDC@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480811031610k137179d4i37ba3da8aabd0035@mail.gmail.com> On Mon, Nov 3, 2008 at 11:22 PM, Nicolas Bertolotti wrote: > I recently experienced crashes running 32 bit binaries on a 6 GB machine > with ram-slop 0.85. The output with "gc-messages" is the following: What platform is this? > As s->sysvals.ram is set to 4 294 967 295 in that case, sizeofHeapDesired() > may return that value and lead to an attempt to create a heap of size 0 > because align(desiredSize, s->sysvals.pageSize) will then return 0. > > As a matter of fact, after looking at the code, it seems that a number of > other situations may lead to similar issues (for instance, when > sizeofHeapDesired() returns a value that would lead to an overflow when we > add the size of the card/cross map). Yes, the choice of desired heap doesn't respect the actual address layout restrictions that exist for a process. I personally think we need to include some platform-specific information about how big the addressable space is, and cap the physical memory to this limit. Linux has a 3G cap, MinGW a 2G cap, and cygwin a 1.2G cap. Perhaps there is some system call which can inform this choice in a portable way? I pointed out a related problem on Oct 14 : > Another thing I've noticed is that MLton's estimate of the available > RAM can trick it into making bad choices for the heap size. You can > only possibly get 1.2G on cygwin, but MLton thinks it should be able > to get an amount = physical ram. It sets the minSize and desiredSize > both too high and is the remap then fails. However, if it had a less > aggressive minSize, it could have allocated another 300MB. I'm not > sure what to do about this. Perhaps cap the physical RAM based on the > platform? I had hoped to generate some discussion, but it seems no one was listening. :-) From Nicolas.Bertolotti at mathworks.fr Mon Nov 3 17:12:50 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Mon Nov 3 17:13:25 2008 Subject: [MLton] Segmentation fault in 32 bit mode on a 6 GB machine with ram-slop 0.85 In-Reply-To: <162de7480811031610k137179d4i37ba3da8aabd0035@mail.gmail.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDC@EXCHANGE-UK.ad.mathworks.com> <162de7480811031610k137179d4i37ba3da8aabd0035@mail.gmail.com> Message-ID: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDD@EXCHANGE-UK.ad.mathworks.com> > What platform is this? The platform is Linux (64 bit OS but my binary is compiled in 32 bit mode). > Linux has a 3G cap, MinGW a 2G cap, and cygwin a 1.2G cap. Perhaps > there is some system call which can inform this choice in a portable > way? As you can see in the log, before the heap allocation that leads to the crash, the heap size was 3,854,442,496 bytes which seems to indicate that the 3G cap is not an absolute limit (at least on a 64 bit OS). I was surprised too. I had never seen that before on the 4G machines I usually have access to. I thought this was impossible. > I pointed out a related problem on Oct 14 > : > > Another thing I've noticed is that MLton's estimate of the available > > RAM can trick it into making bad choices for the heap size. You can > > only possibly get 1.2G on cygwin, but MLton thinks it should be able > > to get an amount = physical ram. It sets the minSize and desiredSize > > both too high and is the remap then fails. However, if it had a less > > aggressive minSize, it could have allocated another 300MB. I'm not > > sure what to do about this. Perhaps cap the physical RAM based on the > > platform? > > I had hoped to generate some discussion, but it seems no one was > listening. :-) As far as I know, the fact that desiredSize can be too high is not a big issue. After a number of backoffs, mmap/mremap succeeds and the only consequence is that createHeap/remapHeap runs a bit slower. I have never encountered any issue with a too high value for minSize. If there were some, I guess they would have led to some unjustified "Out of memory" crashes and the only ones I have experienced experienced so far were caused by the fact the card/cross map was allocated after the heap itself (which is not the case anymore). Anyway, I spend my time trying to improve memory handling and there are still some open directions. I sent an e-mail about those (see http://mlton.org/pipermail/mlton/2008-September/030358.html). One thing I did not mention is that, in some cases, we may also have some allocations that do succeed but cause the process to swap and slow it down so much that it would have been more efficient to work with a smaller heap. Also, in multi-core context, when 2 processes are running, it is sometimes more efficient to suspend the execution of one process after paging its heap to disk. Then, the second process won't swap and finish the job in a shorter amount of time. Those are just questions for now... Nicolas From wesley at terpstra.ca Mon Nov 3 23:30:03 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Nov 3 23:30:06 2008 Subject: [MLton] Segmentation fault in 32 bit mode on a 6 GB machine with ram-slop 0.85 In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDD@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDC@EXCHANGE-UK.ad.mathworks.com> <162de7480811031610k137179d4i37ba3da8aabd0035@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA385AD21FDD@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480811032330p7dcb5eddicdd7feb43c97a005@mail.gmail.com> On Tue, Nov 4, 2008 at 2:12 AM, Nicolas Bertolotti wrote: >> What platform is this? > The platform is Linux (64 bit OS but my binary is compiled in 32 bit mode). Ahh, the 64bit must explain how you got > 3G. The kernel must not be mapped in the high 1G as usual for 32bit mode. This also explains an open question about why only some systems exhibit problems; machines with large amounts of RAM are probably using a 64bit kernel. >> I pointed out a related problem on Oct 14 >> : >> > Another thing I've noticed is that MLton's estimate of the available >> > RAM can trick it into making bad choices for the heap size. You can >> > only possibly get 1.2G on cygwin, but MLton thinks it should be able >> > to get an amount = physical ram. It sets the minSize and desiredSize >> > both too high and is the remap then fails. However, if it had a less >> > aggressive minSize, it could have allocated another 300MB. I'm not >> > sure what to do about this. Perhaps cap the physical RAM based on the >> > platform? >> >> I had hoped to generate some discussion, but it seems no one was >> listening. :-) > > As far as I know, the fact that desiredSize can be too high is not a big issue. After a number of backoffs, mmap/mremap succeeds and the only consequence is that createHeap/remapHeap runs a bit slower. If the desiredSize never exceeded 3/4G, you wouldn't have had the heap size wrap around like you did. MLton would never have attempted to grow the heap beyond the size of a 32bit int. Two birds with one stone. What I am proposing is to take max(physicalRam * ram-slop, cap) as the limit for desiredSize. For 64bit kernel, 32bit linux, set the cap to say 3.8G to prevent wrapping around the size of an unsigned integer. > Also, in multi-core context, when 2 processes are running, it is sometimes more efficient to suspend the execution of one process after paging its heap to disk. Then, the second process won't swap and finish the job in a shorter amount of time. Managing memory between processes isn't really MLton's job, but the user's. Don't run two programs who need more combined memory than you have. :-) Or set ram-slop 0.5. From wesley at terpstra.ca Mon Nov 3 23:56:54 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Nov 3 23:56:57 2008 Subject: [MLton] Segmentation fault in 32 bit mode on a 6 GB machine with ram-slop 0.85 In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDD@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDC@EXCHANGE-UK.ad.mathworks.com> <162de7480811031610k137179d4i37ba3da8aabd0035@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA385AD21FDD@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480811032356g516f1945nbe8a235db70ea91e@mail.gmail.com> On Tue, Nov 4, 2008 at 2:12 AM, Nicolas Bertolotti wrote: > I have never encountered any issue with a too high value for minSize. If there were some, I guess they would have led to some unjustified "Out of memory" crashes and the only ones I have experienced experienced so far were caused by the fact the card/cross map was allocated after the heap itself (which is not the case anymore). I have had minSize limit the obtainable memory in MinGW. To see these interactions where desiredSize goes crazy you need to have the right physical RAM and the right ram-slop. The problem is that minSize is only a constant fraction of desiredSize, and if desiredSize is too large the backoff gives up too early, failing to grow the heap even though it could have. Another way of addressing this would be to limit how quickly MLton can grow the heap in one GC (to ensure that minSize never exceeds the current size). > Anyway, I spend my time trying to improve memory handling and there are still some open directions. > I sent an e-mail about those (see http://mlton.org/pipermail/mlton/2008-September/030358.html). In that email you state: > Also, there are 2 very different situations which can cause a "createHeap" to fail: > - not enough RAM to satisfy the allocation > - not a big enough logical address range to satisfy the allocation To my knowledge the first case never happens. "RAM" allocation only fails if you are out of address space. This is why MLton's "ask for the physical RAM" approach is necessary. This applies to both windows and linux AFAIK. About the only useful information MLton can get about how much memory it is able to use in a multi-process environment is the resident size. It would be hard to use this information sensibly, though, as you have to first try for a larger amount to determine the resident size you get. From vesa.a.j.k at gmail.com Tue Nov 4 01:49:44 2008 From: vesa.a.j.k at gmail.com (Vesa Karvonen) Date: Tue Nov 4 01:49:52 2008 Subject: [MLton] The bool array/vector performance bug In-Reply-To: References: <9e43b9a0805311019sccc62f8qfd05fbb1f45324e9@mail.gmail.com> Message-ID: <9e43b9a0811040149k67b0b9aayf1693adb4deac34e@mail.gmail.com> On Mon, Jun 2, 2008 at 1:30 AM, Matthew Fluet wrote: > On Sat, 31 May 2008, Vesa Karvonen wrote: >> >> The bool array/vector performance bug (bool array/vector uses 4-bytes per >> element) has been discussed on the list a number of times (you can find >> them by googling for "bool representation", for example). > > It is not clear to me that using 4-bytes per bool (rather than 1-bit or > 1-byte) is best characterized as a "performance bug". (I took that phrase from some of the old e-mails on this subject.) > I suspect that on > most 32-bit systems, accessing a 4-byte element is more efficient than > accessing a 1-byte element. Modern CPUs tend to have instructions like mozx/movsx on the x86 and for a purpose. Accessing 8-bit data is not rare. Such an instruction may be slightly slower than a regular non-extending move instruction, but not by much. > On the other hand, it is certainly the case > that more 1-byte elements will fit in a cache line. Indeed. I would bet that accessing 32-bit elements on a modern 32-bit processor is only faster when it hits the L1 cache and even then the difference is likely to be minimal (if any). I would not be surprised to see that it would actually improve overall performance to essentially always pack booleans as 1 bit elements in memory representations (not just in arrays). It doesn't cost many extra instructions to test/set a bit (when properly optimized) rather than a byte/word and the savings in cache space (and even in time spent doing GC) could be significant. >> - don't allow ML bool in the FFI, but do >> - allow C Bool ("C_Bool.t") in the FFI. > > The C_Bool structure is in the Basis Library implementation, but not > currently exported by c-types.mlb. The C_Bool structure is equal to (the > application of the WordToBool functor to) a WordN structure, where N > corresponds to the target platform's size of the C-type _Bool. I think that for the purpose of writing portable code it would make sense to export C_Bool regardless of whether the ML bool representation is changed. >> I think that this is the pretty much the only sane alternative, because >> the resulting code will be portable (the size of C bool is platform >> dependent) and it allows maximal flexibility on the ML side. > > Portability is a different concern than performance. Given a set of alternatives ways to improve the ML bool representation (the goal), it makes sense to pick an alternative that also allows (or encourages) one to write portable code. Clearly divorcing ML bool from C bool does that. The choice of ML bool representation could then be done freely. It could theoretically even be a compiler switch (-default-type {bool1, bool8, bool16, bool32, bool64}). >> The current approach, using 4-byte bools, is just plain wrong, IMO, >> because a C bool is not specified to be 4-bytes. > > Fair enough, but we don't currently specify that an ML bool is equivalent to > a C _Bool. Rather, we specify that an ML bool is equivalent to a C int32_t > (see http://mlton.org/ForeignFunctionInterfaceTypes). We don't currently > specify that any ML type is equivalent to a C _Bool. That is a good point. > We actually go through some hoops in the elaborator > (/mlton/elaborate/elaborate-core.fun) to compare values coming to/from > C symbols against 0 when they are being used as ML bool. (This could be > simplified by excluding ML bool from the FFI.) Though, this doesn't help > when the bool is coming from an _import or _export and it doesn't help if > the FFI is used with an indirect type. That is, all bets are off if you > have: > > z.c: > void f(int32_t *a, int32_t i) { a[i] = 7; } > > z.sml: > val f = _import "f": bool array -> unit; > val a = Array.tabulate (10, fn () => true); > val () = ... f 3 ... Array.sub (a, 3) ... Indeed. Those are good reasons to exclude (ML) bool from the FFI. >> Using 4-bytes per bool on the ML side is also highly inefficient and >> (which doesn't make it wrong but) makes MLton look bad on some toy >> benchmarks >> (http://shootout.alioth.debian.org/gp4/benchmark.php?test=nsieve&lang=all). > > Changing the benchmark to use Word8.word rather than bool did show a speedup > (which I attribute to packing more elements into a cache line): [...] Indeed. Also note that there is a version of the benchmark using bit arrays (1-bit per element). It runs even faster than the 8-bits per element version. Compare the times for N=9: http://shootout.alioth.debian.org/gp4/fulldata.php?test=nsievebits&p1=mlton-2&p2=mlton-2&p3=mlton-2&p4=mlton-2 http://shootout.alioth.debian.org/gp4/fulldata.php?test=nsieve&p1=mlton-2&p2=mlton-2&p3=mlton-2&p4=mlton-2 > One other item is the handling of primitives that return boolean values. > Some of these primitives are not implemented by the codegens (e.g., the x86 > codegen doesn't implement Word64_{equal,lt}), and so are turned into C > functions (in /mlton/codegen/ssa-to-rssa.fun). (This is why they are > declared in /runtime/gen/basis-ffi.def.) If the C functions do not > return values in the same representation as that of an ML bool, then you > will need to insert some coercion, either a word-size coercion or an > explicit test against 0. Hmm... What would be the best way to fix this? These are somewhat special in that they are really a part of the runtime/compiler and can be changed at will if/when the bool representation is changed and I wouldn't want to add extra overhead to them. > Contrary to some of the comments I made in the threads cited above, getting > hold of platform-dependent types in the compiler proper has proven > expedient. So, if it were necessary to have the size/representation of C > _Bool known by the compiler, then it should be possible to do so; see the > implementation of Control.Target.Size.cint and CType.cint and > /runtime/gen/gen-sizes.c. Thus, another alternative is to simply > establish that ML bool is represented as C _Bool, by making WordSize.bool > pull from Control.Target.Size.cbool. > > Nonetheless, I think that divorcing ML bool from C _Bool is the right > decision. I agree. One can always use C_Bool.t if interfacing with C is the overriding concern. It also allows the widest latitude in choosing the ML bool representation and avoids nasty cases like mutating an ML bool array in C. -Vesa Karvonen From Nicolas.Bertolotti at mathworks.fr Tue Nov 4 02:09:34 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Tue Nov 4 02:10:11 2008 Subject: [MLton] Segmentation fault in 32 bit mode on a 6 GB machine with ram-slop 0.85 In-Reply-To: <162de7480811032330p7dcb5eddicdd7feb43c97a005@mail.gmail.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDC@EXCHANGE-UK.ad.mathworks.com> <162de7480811031610k137179d4i37ba3da8aabd0035@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA385AD21FDD@EXCHANGE-UK.ad.mathworks.com> <162de7480811032330p7dcb5eddicdd7feb43c97a005@mail.gmail.com> Message-ID: <8320D98DA9A5C54C926D397795FE7CEA385AD22096@EXCHANGE-UK.ad.mathworks.com> > Managing memory between processes isn't really MLton's job, but the > user's. Don't run two programs who need more combined memory than you > have. :-) Or set ram-slop 0.5. Yes and no. It is the user's job but MLton could make it easier. I would like to be able to monitor the allocations that are made by my processes and communicate with them in order to change some runtime parameters and/or force one process to page its heap to the disk and sleep. In order to achieve this, I don't have any other solution than to hack MLton. Having a well defined interface for that would be helpful (unfortunately, maybe just for me) From vesa.a.j.k at gmail.com Tue Nov 4 04:15:24 2008 From: vesa.a.j.k at gmail.com (Vesa Karvonen) Date: Tue Nov 4 04:15:28 2008 Subject: [MLton] Segmentation fault in 32 bit mode on a 6 GB machine with ram-slop 0.85 In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA385AD22096@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDC@EXCHANGE-UK.ad.mathworks.com> <162de7480811031610k137179d4i37ba3da8aabd0035@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA385AD21FDD@EXCHANGE-UK.ad.mathworks.com> <162de7480811032330p7dcb5eddicdd7feb43c97a005@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA385AD22096@EXCHANGE-UK.ad.mathworks.com> Message-ID: <9e43b9a0811040415t45e8e445p317947b8e23778de@mail.gmail.com> On Tue, Nov 4, 2008 at 12:09 PM, Nicolas Bertolotti wrote: >> Managing memory between processes isn't really MLton's job, but the >> user's. Don't run two programs who need more combined memory than you >> have. :-) Or set ram-slop 0.5. > > Yes and no. It is the user's job but MLton could make it easier. > > I would like to be able to monitor the allocations that are made by my processes and > communicate with them in order to change some runtime parameters and/or force one > process to page its heap to the disk and sleep. Being able to (better) monitor allocations and change some more runtime parameters programmatically could be useful. However, the ability to communicate with a running MLton process and send commands to it seems rather specialized and potentially dangerous and unsafe. Given interfaces for accessing the allocation information and changing runtime parameters, it shouldn't be too difficult to implement a simple monitoring server in SML. I've written two experimental libraries that could be used for a purpose like that although neither is probably currently a perfect fit for your needs. One is the Program Variable Editor (var-ed) (http://mlton.org/cgi-bin/viewsvn.cgi/mltonlib/trunk/org/mlton/vesak/var-ed/unstable/). Another is the RPC library (rpc-lib) (http://mlton.org/cgi-bin/viewsvn.cgi/mltonlib/trunk/org/mlton/vesak/rpc-lib/unstable/). If you take a look at them, feel free to suggests changes/improvements especially if those could then serve your needs. > In order to achieve this, I don't have any other solution than to hack MLton. Having a > well defined interface for that would be helpful (unfortunately, maybe just for me) I think that adding some more functionality, where possible, to the MLton.GC (http://mlton.org/MLtonGC) structure, such as finding out the heap size, would be a good thing. OTOH, a process monitoring server should be implemented in the SML program rather than by MLton, IMO. -Vesa Karvonen From fluet at tti-c.org Tue Nov 4 09:25:33 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Tue Nov 4 09:28:03 2008 Subject: [MLton] Re: [MLton-commit] r6965 In-Reply-To: References: Message-ID: On Fri, 31 Oct 2008, Ville Laurikari wrote: > Reverted change done in r6964. Mere concatenation of the SML files > won't produce a directly compilable with anything but a trivial flat > MLB structure. I suppose it's better to keep the MLB files in there, > although I fail to see how "-stop sml" is useful with MLB files. It is true that "-stop sml" is probably an out-dated concept. I'm not sure of the history of the "-stop sml" option; it is present in the initial revision of the SVN repository (as imported from the earlier CVS repository). It is also true that "-stop sml" doesn't attempt to reflect the MLB scoping into the resulting SML file. [Indeed, one of the motivations for the MLB system is that there is scoping that cannot be reflected into SML; for example, signatures and functors can only appear at the top-level in SML'97, not within "local .. in .. end" scopes.] On the other hand, prior to the MLB system, we used a "poor man's CM" system -- that is, MLton interprets CM files as nothing more than an ordered list of SML files, albeit with a 'cacheing' behavior so that the files in one CM file are not included multiple times. In that setting, "-stop sml" on a CM file would produce a SML file with the same semantics of the CM file (relative to how MLton interprets a CM file -- neither would reflect SML/NJ's more sophisticated CM semantics). Also, MLton's treatment of a CM file was to implicitly include the Basis Library (and not include the Basis Library in the SML file produced by "-stop sml"). On the other hand, MLton's treatment of an MLB file does not implicitly include the Basis Library (and, does include the Basis Library in the SML file produced by "-stop sml" if "$(SML_LIB)/basis/basis.mlb" is included by the MLB file). If you have a relatively simple project -- one where the scoping introduced by the MLB system isn't used extensively and concatenating all the files in the project does preserve the meaning, then the "-stop f" option will give you the list of MLB and SML files (in their use order); a little grep/sed/xargs/cat piping could be used to produce the SML file with all the constituent files concatenated together. From fluet at tti-c.org Tue Nov 4 19:35:09 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Tue Nov 4 19:37:38 2008 Subject: [MLton] The bool array/vector performance bug In-Reply-To: <9e43b9a0811040149k67b0b9aayf1693adb4deac34e@mail.gmail.com> References: <9e43b9a0805311019sccc62f8qfd05fbb1f45324e9@mail.gmail.com> <9e43b9a0811040149k67b0b9aayf1693adb4deac34e@mail.gmail.com> Message-ID: On Tue, 4 Nov 2008, Vesa Karvonen wrote: > On Mon, Jun 2, 2008 at 1:30 AM, Matthew Fluet wrote: >> On Sat, 31 May 2008, Vesa Karvonen wrote: > > Indeed. I would bet that accessing 32-bit elements on a modern 32-bit > processor is only faster when it hits the L1 cache and even then the > difference is likely to be minimal (if any). I would not be surprised > to see that it would actually improve overall performance to > essentially always pack booleans as 1 bit elements in memory > representations (not just in arrays). It doesn't cost many extra > instructions to test/set a bit (when properly optimized) rather than a > byte/word and the savings in cache space (and even in time spent doing > GC) could be significant. I'm not convinced that 1 bit booleans in non-array objects would save that much on GC time (by saving object space). All objects need to be at least 4byte aligned, so one would need to have an object with more than 4 booleans to be more space efficient with 1 bit booleans than with 8bit booleans. >>> - don't allow ML bool in the FFI, but do >>> - allow C Bool ("C_Bool.t") in the FFI. >> >> The C_Bool structure is in the Basis Library implementation, but not >> currently exported by c-types.mlb. The C_Bool structure is equal to (the >> application of the WordToBool functor to) a WordN structure, where N >> corresponds to the target platform's size of the C-type _Bool. > > I think that for the purpose of writing portable code it would make > sense to export C_Bool regardless of whether the ML bool > representation is changed. Sure, though I don't believe that the C _Bool type is used much in standard headers. > Given a set of alternatives ways to improve the ML bool representation > (the goal), it makes sense to pick an alternative that also allows (or > encourages) one to write portable code. Clearly divorcing ML bool > from C bool does that. The choice of ML bool representation could > then be done freely. It could theoretically even be a compiler switch > (-default-type {bool1, bool8, bool16, bool32, bool64}). Agreed, that it would be good to allow/encourage portable code. >> One other item is the handling of primitives that return boolean values. >> Some of these primitives are not implemented by the codegens (e.g., the x86 >> codegen doesn't implement Word64_{equal,lt}), and so are turned into C >> functions (in /mlton/codegen/ssa-to-rssa.fun). (This is why they are >> declared in /runtime/gen/basis-ffi.def.) If the C functions do not >> return values in the same representation as that of an ML bool, then you >> will need to insert some coercion, either a word-size coercion or an >> explicit test against 0. > > Hmm... What would be the best way to fix this? These are somewhat > special in that they are really a part of the runtime/compiler and can > be changed at will if/when the bool representation is changed and I > wouldn't want to add extra overhead to them. I don't think there is any way to "fix" this other than to make sure that the C functions that implement conditional primitives agree with the compiler on the representation of booleans, or insert coercions. Here are some other thoughts. There is a long-standing tension/debate on whether conditionals should be value-producing or control-flow-producing. MLton currently takes a value-producing view of (most) conditionals (the exception being the checked-arithmetic primitives); that is, the Word32 equality primitive is used like: x_5: bool = Word32_equal (x_2, global_1) Under the control-flow-producing view, it would be used like: if Word32_equal (x_2, global_1) then L1 () else L2 () where the primitive is a transfer (the "if-then-else" being pretty-printing sugar). Note that the two views are equivalent, in the sense that one can always convert a value to control-flow: x_5: bool = Word32_equal (x_2, global_1) case x_5 of true => L1 () | false => L2 () and control-flow to a value: datatype boolz = truez | falsez if Word32_equal (x_2, global_1) then L1 () else L2 () L1 () x_1: boolz = truez L3 (x_1) L2 () x_2: boolz = falsez L3 (x_2) L3 (x_5: boolz) ... Note that the boolz type is not special in any way; indeed, the compiler is free to use any representation it sees fit -- currently, MLton would represent it as 1bit (padded to 8bits when used in an array). I point this out because one could introduce an pass that separates the bool type (as returned by conditional primitives) from the type of the conditional value stored in data structures. That is, introduce the boolz datatype and transform: x_5: bool = Word32_equal (x_2, global_1) ... to z: bool = Word32_equal (x_2, global_1) case z of true => L1 () | false => L2 () L1 () zt: boolz = truez L3 (zt: boolz) L2 () zf: boolz = falsez L3 (zf: boolz) L3 (x_5: boolz) ... for fresh z, L1, zt, L2, zf, L3 and transform case y of true => Lt () | false => Lf () to case y of truez => Lt () | falsez => Lf () and otherwise transform the "bool" type to "boolz" type. One could optimize the transformation; it doesn't make much sense to transform x_5: bool = Word32_equal (x_2, global_1) case x_5 of true => L1 () | false => L2 () when x_5 is otherwise not used (though, the knownCase optimization should clean up after the un-optimized transformation in this case). Note, that you could apply a similar transformation to foreign functions with bool arguments or return; one would require a much more complicated transformation to handle foreign functions with "bool array", "bool vector", or "bool ref" arguments or results. (One would need to do a dataflow analysis to determine which objects need to maintain a representation compatible with C.) So, there is a way of changing the space usage of booleans without needing to change the implementation of the primitives. There is a little overhead here when a 32-bit integer is returned from a C function (implementing a primitive) and then explicitly tested before doing an explicit assignment of 0 or 1 (as an 8bit integer). From fluet at tti-c.org Sun Nov 9 12:55:20 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Sun Nov 9 12:58:19 2008 Subject: [MLton] Segmentation fault in 32 bit mode on a 6 GB machine with ram-slop 0.85 In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDC@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDC@EXCHANGE-UK.ad.mathworks.com> Message-ID: On Mon, 3 Nov 2008, Nicolas Bertolotti wrote: > I recently experienced crashes running 32 bit binaries on a 6 GB machine with ram-slop 0.85. The output with "gc-messages" is the following: > [GC: Found 6,245,584,896 bytes of RAM; using 4,294,967,295 bytes (68.8% of RAM).] You can see that the runtime initialization does not attempt to use more RAM than the virtual address space, even if this means using a smaller ram-slop than specified. However, as the note in <>/runtime/gc/init.c states, we should ensure that s->sysval.ram is aligned by the page size; this wasn't being done when SIZE_MAX was used as the maximum ram. Though, that comment is really describing the test at <>/runtime/gc/garbage-collection.c:29, which chooses to use the mark-compact garbage collector if the current heap is larger than RAM. > ... > [GC: Starting gc #739; requesting 512 nursery bytes and 0 old-gen bytes,] > [GC: heap at 0x0e194000 of size 3,854,442,496 bytes,] > [GC: with nursery of size 215,045,648 bytes (5.6% of heap),] > [GC: and old-gen of size 3,639,396,848 bytes (94.4% of heap),] > [GC: and nursery using 215,045,140 bytes (5.6% of heap, 100.0% of nursery).] > [GC: Creating heap of size 0 bytes (+ 0 bytes card/cross map) cannot be satisfied,] > [GC: backing off by 27,533,312 bytes with minimum size of 3,854,442,496 bytes.] > [GC: Created heap at 0xf77ea000 of size 4,267,433,984 bytes (+ 33,341,440 bytes card/cross map).] > [GC: Starting major Cheney-copy;] > [GC: from heap at 0x0e194000 of size 3,854,442,496 bytes,] > [GC: to heap at 0xf77ea000 of size 4,267,433,984 bytes.] > [GC: Shrinking stack of size 2,088,960 bytes to size 98,560 bytes, using 3,080 bytes.] > Received signal 11 > > As s->sysvals.ram is set to 4 294 967 295 in that case, > sizeofHeapDesired() may return that value and lead to an attempt to > create a heap of size 0 because align(desiredSize, s->sysvals.pageSize) > will then return 0. That seems to be the case. > I am looking for the best way to fix this issue and your help would be > greatly appreciated. > > As a matter of fact, after looking at the code, it seems that a number > of other situations may lead to similar issues (for instance, when > sizeofHeapDesired() returns a value that would lead to an overflow when > we add the size of the card/cross map). > > Any hint ? I think that the best solution is for sizeofHeapDesired to return a size that will not overflow when aligned and will not overflow when combined with the card/cross map size. This can be handled using the architecture virtual address space limit -- irrespective of any OS specific limits. Indeed, I think the whole question of sizing policies that take into account RAM, OS specific factors (like contiguous address space), etc. should be entirely separate from avoiding overflow of size computations. Of course, there are probably a number of places in the runtime where sizes could overflow. I would be particularly worried about things like array allocation and stack growth, which could generate a large request. -Matthew From fluet at tti-c.org Sun Nov 9 12:59:14 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Sun Nov 9 13:01:59 2008 Subject: [MLton] Segmentation fault in 32 bit mode on a 6 GB machine with ram-slop 0.85 In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDD@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDC@EXCHANGE-UK.ad.mathworks.com> <162de7480811031610k137179d4i37ba3da8aabd0035@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA385AD21FDD@EXCHANGE-UK.ad.mathworks.com> Message-ID: On Tue, 4 Nov 2008, Nicolas Bertolotti wrote: > One thing I did not mention is that, in some cases, we may also have > some allocations that do succeed but cause the process to swap and slow > it down so much that it would have been more efficient to work with a > smaller heap. There are different ways that this could arise. One is that that things like a heap's desired size is computed without taking into account the card/cross map, so that the ultimate allocation requests more than the RAM that the ML process is supposed to use. That could probably be handled better in the runtime system. The other situation is that there are other processes in the system that make the RAM that the ML process is supposed to use a bad estimate. The runtime system alone cannot help with that. -Matthew From wesley at terpstra.ca Sun Nov 9 13:26:16 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sun Nov 9 13:26:20 2008 Subject: [MLton] Segmentation fault in 32 bit mode on a 6 GB machine with ram-slop 0.85 In-Reply-To: References: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDC@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480811091326p4e31d130g4bac1d24cf56db30@mail.gmail.com> On Sun, Nov 9, 2008 at 9:55 PM, Matthew Fluet wrote: > I think that the best solution is for sizeofHeapDesired to return a size > that will not overflow when aligned and will not overflow when combined with > the card/cross map size. This can be handled using the architecture virtual > address space limit -- irrespective of any OS specific limits. You mean using ADDRESS_BITS? Seems reasonable. > Indeed, I > think the whole question of sizing policies that take into account RAM, OS > specific factors (like contiguous address space), etc. should be entirely > separate from avoiding overflow of size computations. In fact, I now think it's unnecessary to take any action. As we discussed earlier on googletalk, the phenomenon I saw was not caused by bad decision making in MLton's GC. It seems to me now that no action is required to take contiguous address space in the OS under consideration. The current mremap emulation suffices to grow up to the achievable limit (and/or stop earlier if more is not requried). > Of course, there are probably a number of places in the runtime where sizes > could overflow. I would be particularly worried about things like array > allocation and stack growth, which could generate a large request. You're worried about a single array which almost exceeds 4GB and is then aligned to 4GB? Since it's completely impossible to satisfy these requests, couldn't we just bail out immediately with an out-of-memory error? From fluet at tti-c.org Sun Nov 9 13:31:13 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Sun Nov 9 13:34:10 2008 Subject: [MLton] Segmentation fault in 32 bit mode on a 6 GB machine with ram-slop 0.85 In-Reply-To: <9e43b9a0811040415t45e8e445p317947b8e23778de@mail.gmail.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDC@EXCHANGE-UK.ad.mathworks.com> <162de7480811031610k137179d4i37ba3da8aabd0035@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA385AD21FDD@EXCHANGE-UK.ad.mathworks.com> <162de7480811032330p7dcb5eddicdd7feb43c97a005@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA385AD22096@EXCHANGE-UK.ad.mathworks.com> <9e43b9a0811040415t45e8e445p317947b8e23778de@mail.gmail.com> Message-ID: On Tue, 4 Nov 2008, Vesa Karvonen wrote: > I think that adding some more functionality, where possible, to the > MLton.GC (http://mlton.org/MLtonGC) structure, such as finding out the > heap size, would be a good thing. There are some statistics currently there: http://mlton.org/cgi-bin/viewsvn.cgi/mlton/trunk/basis-library/mlton/gc.sig?rev=6355&view=auto Adding more controls that could be adjusted from the ML side would be a nice addition. However, one would need to read through the runtime system to determine which ratios, controls, etc. could be changed during the run of the program. From fluet at tti-c.org Sun Nov 9 13:41:57 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Sun Nov 9 13:44:41 2008 Subject: [MLton] Segmentation fault in 32 bit mode on a 6 GB machine with ram-slop 0.85 In-Reply-To: <162de7480811091326p4e31d130g4bac1d24cf56db30@mail.gmail.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDC@EXCHANGE-UK.ad.mathworks.com> <162de7480811091326p4e31d130g4bac1d24cf56db30@mail.gmail.com> Message-ID: On Sun, 9 Nov 2008, Wesley W. Terpstra wrote: >> Of course, there are probably a number of places in the runtime where sizes >> could overflow. I would be particularly worried about things like array >> allocation and stack growth, which could generate a large request. > > You're worried about a single array which almost exceeds 4GB and is > then aligned to 4GB? Since it's completely impossible to satisfy these > requests, couldn't we just bail out immediately with an out-of-memory > error? No, I'm worried about a 3.5G heap (which had 3.05G live data at the last major garbage collection, but now has only 2.0G live data) and a 1G array allocation. This allocation should succeed (there is enough memory after the garbage collection); however, at <>/runtime/gc/garbage-collection.c:26, the computation: s->lastMajorStatistics.bytesLive + bytesRequested will overflow, quite possibly allowing a secondary heap of 0.05G to be allocated, then leading to a bogus copying collection (due to the assertion failure at <>/runtime/gc/cheney-copy.c:53) From fluet at tti-c.org Sun Nov 9 13:43:45 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Sun Nov 9 13:46:29 2008 Subject: [MLton] Segmentation fault in 32 bit mode on a 6 GB machine with ram-slop 0.85 In-Reply-To: <162de7480811091326p4e31d130g4bac1d24cf56db30@mail.gmail.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AD21FDC@EXCHANGE-UK.ad.mathworks.com> <162de7480811091326p4e31d130g4bac1d24cf56db30@mail.gmail.com> Message-ID: On Sun, 9 Nov 2008, Wesley W. Terpstra wrote: > On Sun, Nov 9, 2008 at 9:55 PM, Matthew Fluet wrote: >> I think that the best solution is for sizeofHeapDesired to return a size >> that will not overflow when aligned and will not overflow when combined with >> the card/cross map size. This can be handled using the architecture virtual >> address space limit -- irrespective of any OS specific limits. > > You mean using ADDRESS_BITS? Seems reasonable. I prefer the future proof SIZE_MAX, but anything along those lines would work. From fluet at tti-c.org Sun Nov 9 20:59:07 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Sun Nov 9 21:01:52 2008 Subject: [MLton] Re: [MLton-commit] r6978 In-Reply-To: References: Message-ID: On Wed, 5 Nov 2008, Wesley Terpstra wrote: > Ensure that the current thread's stack is traceable by the collector. > It (might be) possible to pass an object to MLton.size that can reach the > current thread. If we don't have a valid return address and an up-to-date > used value for the stack, the collector will segfault tracing the stack. Please test more thoroughly, and ensure that the <>/regression/size.sml (and related) regression tests pass. Currently: [fluet@shadow temp]$ ../build/bin/mlton -type-check true -debug true size.sml invalid transfer: CCall {args = (, global_0), func = {args = (GCState, Objptr (opt_23)), bytesNeeded = None, convention = cdecl, ensuresBytesFree = false, mayGC = false, maySwitchThreads = true, modifiesFrontier = false, prototype = {args = (CPointer, CPointer), res = Some Word64}, readsStackTop = true, return = Word64, symbolScope = private, target = GC_size, writesStackTop = false}, return = Some L_0} ... Rssa.typeCheck [fluet@shadow temp]$ ../build/bin/mlton -type-check false -debug true size.sml [fluet@shadow temp]$ ./size size: gc/invariant.c:10: assertIsObjptrInFromSpace: Assertion `isObjptrInFromSpace (s, *opp)' failed. Aborted [fluet@shadow temp]$ ../build/bin/mlton -type-check false -debug false size.sml [fluet@shadow temp]$ ./size MLton bug: control shouldn't reach here Please send a bug report to MLton@mlton.org. From wesley at terpstra.ca Mon Nov 10 04:24:01 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Nov 10 04:24:04 2008 Subject: [MLton] Re: [MLton-commit] r6978 In-Reply-To: References: Message-ID: <162de7480811100424v62d713cakb95f4a09cad573c6@mail.gmail.com> On Mon, Nov 10, 2008 at 5:59 AM, Matthew Fluet wrote: > Please test more thoroughly, and ensure that the > <>/regression/size.sml (and related) regression tests pass. Sorry about that! I got my patches crossed while trying to commit only the change affecting MLton.size. I still would've messed up the -type-check true, though; I didn't know I had to set writesStackTop and modifiesFrontier. Fixed in r6981. From wesley at terpstra.ca Tue Nov 11 12:09:14 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Nov 11 12:09:20 2008 Subject: [MLton] Re: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> Message-ID: <162de7480811111209x2549aac6y3bf403b14c088c17@mail.gmail.com> On Tue, Nov 11, 2008 at 7:04 PM, Matthew Fluet wrote: > With regards to whether users should actually use GC_getArrayLength, we > don't really advertise it, but neither is it a particularly subtle > operation. And, it is genuinely useful (though, getting the length and > passing it through the FFI along with the array pointer wouldn't been > terribly difficult either). I don't think there is any way to really > distinguish between visible functions in libmlton.a that are available for > use in mlton generated .c and .s files, but not available for user .c code. Well, I personally have always passed the length of arrays with the array through the FFI. This is the usual way C functions work in my experience. I was genuinely surprised to see GC_arrayLength used as I thought GC_* was supposed to be MLton internal. There are some damn scary functions available with the GC_ prefix! It sounds from your later comments that you agree with me, so let's just phase GC_arrayLength out of the examples and FFI documentation. It still works within a DSO since it is a PRIVATE symbol at the moment, so there's no backwards compatibility problem. >> 2. iimport.c uses dlopen/dlclose/dlsym. platform/mingw.c defines >> these... which is a problem since the runtime is statically linked, >> but libdl is dynamically linked. Perhaps these should be public >> libname_ symbols, private MLton_dlopen/close/sym, or both? > > To clarify: the issue is that on *NIX platforms (including MacOSX), > dl{open,close,sym,error,...} are pulled from dynamically linked shared > libraries. That is, they are imported from another DSO. Thus, for these > platforms, the 'external' attribute should be used on the _import. > > MinGW doesn't provide dl{open,close,...} in a (standard) dynamically linked > shared library. Vesa added rudimentary emulation in > /runtime/platform/mingw.c (r4862, 20061127); thus dl{open,close,...} > are pulled from libmlton.a. That is, they are imported from the same DSO. > Thus, for this platform, the 'private' attribute should be used on the > _import. Yes, that's the problem (though they are public in mingw.c, not private). > Personally, I think that MLton should not be attempting to provide > non-essential functionality that is missing on a platform. I agree, with the added provision that if MLton needs these functions itself it implement them, but not export them. A good example being the mingw.{ch} which adds missing functionality as MLton_* (b/c of #define's in mingw.h). Several existing files in platform/*.c don't do this, which IMO is bad (aix.c, float-math.c, hpux.c, mkdir2.c, setenv.putenv.c, and solaris.c). > MLton doesn't > default to including those link options, because they are not essential > functionality (e.g., required by some of the Basis Library implementation).) So why were the dl* functions added to mingw.c in the first place? The only reason I suggested keeping them (and the GC_arrayLength stuff) around was because I thought there was a reason for them that I didn't know. > - MLton shouldn't be providing dl{open,close,...} in any form. If the basis doesn't use them, then I agree and further suggest they be removed from mingw.c. > - GC_getArrayLength isn't advertised, so there is no need to 'bless' it > with a MLton_getArrayLength name. (We should eliminate its use from the > /doc/examples/ffi examples.) Be aware that even the wiki advertises this function at the momeny. Thus, I thought this was (shockingly) supposed to be part of the documented FFI. Phasing it out seems a great solution to me. > It might be a good idea to clone the page (say, as > NextRelease/ForeignFunctionInterfaceSyntax) to make current documentation > available somewhere. Sounds like a plan. Regarding public/private, I can no longer recommend changing the default to public. I recall my reasoning now: external is the safe default on all platforms except MinGW. On all these other platforms it works, and works correctly. If public were the default, it might be more convenient for MinGW, but would in some corner cases break (variables and pointers). Setting the default to external makes it absolutely clear that the code is in error (and dangerous) on MinGW as it fails loudly with a link error. From fluet at tti-c.org Tue Nov 11 15:26:54 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Tue Nov 11 15:29:48 2008 Subject: [MLton] Re: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: <162de7480811111209x2549aac6y3bf403b14c088c17@mail.gmail.com> References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> <162de7480811111209x2549aac6y3bf403b14c088c17@mail.gmail.com> Message-ID: On Tue, 11 Nov 2008, Wesley W. Terpstra wrote: > On Tue, Nov 11, 2008 at 7:04 PM, Matthew Fluet wrote: > > It sounds from your > later comments that you agree with me, so let's just phase > GC_arrayLength out of the examples and FFI documentation. Agreed. I think most of the documentation on the wiki is pulled directly from the SVN repository, so that will be updated when the examples are >> Personally, I think that MLton should not be attempting to provide >> non-essential functionality that is missing on a platform. > > I agree, with the added provision that if MLton needs these functions > itself it implement them, but not export them. I can't forsee a reason why, but, in principle, I agree that the MLton runtime shouldn't export more than necessary. >> MLton doesn't >> default to including those link options, because they are not essential >> functionality (e.g., required by some of the Basis Library implementation).) > > So why were the dl* functions added to mingw.c in the first place? The > only reason I suggested keeping them (and the GC_arrayLength stuff) > around was because I thought there was a reason for them that I didn't > know. Looks like the origin was to get mlnlffi working on MinGW: http://mlton.org/pipermail/mlton/2006-November/029349.html The original issue there (using dlopen(NULL,...) to get a handle on the process local instance of 'malloc' and 'free') is no longer an issue, as 'malloc' and 'free' are _import-ed. There remains an issue that the DynLinkage module as exported by the mlnlffi library expects to find dl{open,close,sym,err,...}. [One needs to use DynLinkage to implement the library handle expected by mlnlffigen -dynamic generated code.] As noted elsewhere in this thread, there are multiple ways to get around this issue: - Use the dlfcn-win32 library when using the mlnlffigen generated code. - Implement a Win32 version of DynLinkage, use MLB path-variables so that the mlnlffi library uses the correct version of DynLinkage. (Effectively, implement dlfcn-win32 in SML.) >> - MLton shouldn't be providing dl{open,close,...} in any form. > > If the basis doesn't use them, then I agree and further suggest they > be removed from mingw.c. Agreed. >> - GC_getArrayLength isn't advertised, so there is no need to 'bless' it >> with a MLton_getArrayLength name. (We should eliminate its use from the >> /doc/examples/ffi examples.) > > Be aware that even the wiki advertises this function at the moment. I think these uses are all pulled directly from the SVN repository examples. From wesley at terpstra.ca Tue Nov 11 15:56:44 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Nov 11 15:56:47 2008 Subject: [MLton] Re: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> <162de7480811111209x2549aac6y3bf403b14c088c17@mail.gmail.com> Message-ID: <162de7480811111556i2d4f30f7o7cc860b5a69381a6@mail.gmail.com> On Wed, Nov 12, 2008 at 12:26 AM, Matthew Fluet wrote: > Agreed. I think most of the documentation on the wiki is pulled directly > from the SVN repository, so that will be updated when the examples are You are right. The wiki changed when I changed the examples. >> I agree, with the added provision that if MLton needs these functions >> itself it implement them, but not export them. > > I can't forsee a reason why, but, in principle, I agree that the MLton > runtime shouldn't export more than necessary. There are two cases this can spell trouble: 1. The OS might add the missing functions later, causing a conflict. (This kept hitting us for the MinGW port.) 2. Other libraries and/or user code might provide them, again conflicting. Private symbols (so that multiple MLton libraries work) prefixed with MLton_ (so they don't conflict with other [system] libraries) are the most durable option. I know macros are ugly/evil, but I think this is a good place for them. >> So why were the dl* functions added to mingw.c in the first place? > Looks like the origin was to get mlnlffi working on MinGW: > http://mlton.org/pipermail/mlton/2006-November/029349.html Using dlfcn-win32 does seem the best approach. I'd like to hear something from Vesa before removing them. From ville at laurikari.net Tue Nov 11 22:09:14 2008 From: ville at laurikari.net (Ville Laurikari) Date: Tue Nov 11 22:09:22 2008 Subject: [MLton] Re: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: <162de7480811111209x2549aac6y3bf403b14c088c17@mail.gmail.com> References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> <162de7480811111209x2549aac6y3bf403b14c088c17@mail.gmail.com> Message-ID: <20081112060914.GA919@laurikari.net> > itself it implement them, but not export them. A good example being > the mingw.{ch} which adds missing functionality as MLton_* (b/c of > #define's in mingw.h). Several existing files in platform/*.c don't do > this, which IMO is bad (aix.c, float-math.c, hpux.c, mkdir2.c, > setenv.putenv.c, and solaris.c). I'll fix at least aix.h and hpux.h, in some schedule, to #define prefixed versions. > Regarding public/private, I can no longer recommend changing the > default to public. I recall my reasoning now: external is the safe > default on all platforms except MinGW. On all these other platforms it > works, and works correctly. If public were the default, it might be > more convenient for MinGW, but would in some corner cases break > (variables and pointers). Setting the default to external makes it > absolutely clear that the code is in error (and dangerous) on MinGW as > it fails loudly with a link error. I understand. At least in our case, backwards compatibility wins. We don't use the corner cases, and are not planning to start using them. We'll probably then need to use a custom version of MLton (with default as public on MinWG) on MinWG for a period of time until we've had a chance to update all live code. We mostly build our own MLton binaries anyway, so it's not a huge problem to do this. -- http://www.iki.fi/vl/ From wesley at terpstra.ca Wed Nov 12 03:26:50 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Nov 12 03:26:54 2008 Subject: [MLton] Tagged ref and array pointers Message-ID: <162de7480811120326v22b0f648gd9eb10e63163b5c@mail.gmail.com> I was debugging what I thought was a space leak when I discovered that MLton doesn't seem to use the lower 2 bits of a pointer to ref/array objects. For example, the program: datatype foo = COOL_ALLOC of int ref | USABLE_ALLOC of int ref val z = ref 7 val y = ref 9 val x = Array.tabulate (10, fn _ => COOL_ALLOC y) val () = Array.update (x, 3, USABLE_ALLOC y) val () = Array.update (x, 4, USABLE_ALLOC z) val () = Array.update (x, 5, COOL_ALLOC y) val () = Array.update (x, 6, COOL_ALLOC z) val () = print (Int.toString (MLton.size x) ^ "\n") val () = MLton.dot ("meh.dot", "meh", SOME x) val out = fn COOL_ALLOC x => print (Int.toString (!x) ^ "c") | USABLE_ALLOC x => print (Int.toString (!x) ^ "f") val () = Array.app out x val () = z := 5 val () = y := 6 val () = Array.app out x val () = print "\n" Produces the memory layout in the attached png file. You can see that it represents the datatype using an intermediate heap object instead of packing the flag into the low bits of the pointer to the ref cell. I've seen the same behaviour with datatypes to a vector. MLton does the right thing other times (tracking the variants of a datatype the low bits of the pointer) for recursive datatypes and a few other cases. Is there a good reason for it not performing this optimization here? -------------- next part -------------- A non-text attachment was scrubbed... Name: meh.png Type: image/png Size: 15900 bytes Desc: not available Url : http://mlton.org/pipermail/mlton/attachments/20081112/926cad0e/meh-0001.png From fluet at tti-c.org Wed Nov 12 08:59:22 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Nov 12 08:59:21 2008 Subject: [MLton] Re: [MLton-commit] r6978 In-Reply-To: <162de7480811100424v62d713cakb95f4a09cad573c6@mail.gmail.com> References: <162de7480811100424v62d713cakb95f4a09cad573c6@mail.gmail.com> Message-ID: On Mon, 10 Nov 2008, Wesley W. Terpstra wrote: > On Mon, Nov 10, 2008 at 5:59 AM, Matthew Fluet wrote: >> Please test more thoroughly, and ensure that the >> <>/regression/size.sml (and related) regression tests pass. > > Sorry about that! I got my patches crossed while trying to commit only > the change affecting MLton.size. I still would've messed up the > -type-check true, though; I didn't know I had to set writesStackTop > and modifiesFrontier. Fixed in r6981. Thanks. BTW, it occurs to me that MLton.share suffers from a similar issue. It traces an object, so would be confused if it encounters the current stack in the heap without the used field being up-to-date. As with MLton.size, it isn't likely to come up in practice, but one can trigger it with: ----- val rt : MLton.Thread.Runnable.t option ref = ref NONE val r1 : int ref = ref 0 val r2 : int ref = ref 0 fun stats () = let (* val () = r1 := MLton.size rt *) val () = MLton.share rt val () = r2 := MLton.size rt in () end fun switcheroo () = MLton.Thread.switch (fn t => let val () = rt := SOME (MLton.Thread.prepare (t, ())) val () = stats () in valOf (!rt) end) val () = switcheroo () val _ = print (concat ["!r1 = ", Int.toString (!r1), "\n"]) val _ = print (concat ["!r2 = ", Int.toString (!r2), "\n"]) ----- The indirection through rt is necessary so that MLton doesn't optimize the representation of the object passed to MLton.size and MLton.share down to a unit object. MLton 20070826 will also bus error if "val () = r := MLton.size rt" is uncommented, demonstrating that MLton.size does crash without establishing the GC invariants. MLton SVN HEAD doesn't bus error if it is uncommented, because MLton.size establishes the GC invariants which are not yet violated at MLton.share. From fluet at tti-c.org Wed Nov 12 09:46:17 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Nov 12 09:46:19 2008 Subject: [MLton] Re: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: <162de7480811111000k5c2fb26do762eef289fbe4338@mail.gmail.com> References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> <20081111170801.GB13363@laurikari.net> <162de7480811111000k5c2fb26do762eef289fbe4338@mail.gmail.com> Message-ID: On Tue, 11 Nov 2008, Wesley W. Terpstra wrote: > On Tue, Nov 11, 2008 at 6:08 PM, Ville Laurikari wrote: >> On Tue, Nov 11, 2008 at 04:15:26PM +0100, Wesley W. Terpstra wrote: >>> I'm also beginning to wonder if perhaps imports should default to >>> public instead of external given the number of people who've run into >>> the __imp__ link error. >> >> I think this is a good idea. We've certainly run into the problem. >> Upgrading to a new version of MLton is problematic, because code >> updated for the new version does not work with the old version. On >> the other hand, code working with the old version does not work with >> the new version. > > I feel your pain, but changing the default to public won't completely > alleviate this. You *should* specify external for any symbols you > import from a DLL if the default is public. Of course, if you only > ever use _import (never _symbol or _address), you should be safe. > > The other problem case is osx/i386. It needs special stub code to > access external functions and variables from a library. If MLton > defaults to public, then any programs on that target which _import > from a system library (functions like cos, sin, etc) will fail to link > unless "external" is specified. > > Thus picking 'public' as the default would make a smooth upgrade for > MinGW, but break darwin/i386. Currently it's a smooth upgrade for > darwin, but not MinGW. For both situations and targets, though, you > really should distinguish public/external. > > One could imagine a default of public for MinGW and external for > darwin, but that seems pretty arcane. It may be arcane, but it can be justified --- this is effectively what previous versions of mlton did, albeit with some implicit help from the platform binutils. For MinGW (and, indeed, all {amd64,x86}-!darwin platforms), we produced assembly code for all functions as we now produce code for 'private' functions. For ELF platforms (compiling without position-independent-code), private/public/external all produce identical assembly sequences. For COEFF platforms, private&public produce identical assembly sequences, but one does need a different sequence to call an 'external' function. However, if MinGW ld notices that a direct call (the assembly sequence for private&public) is being satisfied by a DLL function, it automagically inserts a stub function, that performs the 'external' function call. For Darwin, we produced assembly code for all functions as we now produce assembly code for 'external' functions. However, the MachO linker was happy to patch, at link time, any function symbols that were satisfied by static libraries (i.e., symbols from the same DSO). However, this implicit help only comes with functions -- that is, symbols used in a call instruction (I think). Symbols whose addresses are taken (i.e., via _address and a mov or lea instruction) do not get any implicit help (I think). Nonetheless, it does seem as though 'external' is a better default. I don't know if it is worth adding an 'ffiSymbolScope {error|ignore|warn}' MLB annotation to signal when FFI is used without an explicit symbol scope. From fluet at tti-c.org Wed Nov 12 10:17:28 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Nov 12 10:17:27 2008 Subject: [MLton] symbol scopes Message-ID: I still don't know that I understand the necessity for the difference between public and external for an imported symbol. [Indeed, on ELF, there appears to be no difference.] My intuition is the following: a symbol that is publically exported can be accessed via an 'external' assembly sequence by another DSO; since that sequence works for another DSO, why doesn't it also work for this DSO. Is it just an optimization? Within the DSO, I can get to the public symbol via relative addressing, but from another DSO (with PIC), I need an indirection? The other thing I don't understand is how gcc gets by with just two visibility directives ("default" and "hidden"); how does it make the private/public/external distinction? I guess it is just for Win{32,64} that the EXTERNAL/PUBLIC/PRIVATE macros map to different annotations (though, I think, that is partly due to the fact that a single C declaration needs to serve as both an export directive (for the definition) and an import directive (for other uses)). But, for Darwin, both EXTERNAL and PUBLIC map to the same annotation --- is the MLton x86/amd64 codegen for Darwin simply avoiding a cleanup step that would be performed by the linker (if we treated 'public' as 'external')? From wesley at terpstra.ca Wed Nov 12 10:56:26 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Nov 12 10:56:31 2008 Subject: [MLton] symbol scopes In-Reply-To: References: Message-ID: <162de7480811121056i6a5ba37am65583e0efde5924f@mail.gmail.com> On Wed, Nov 12, 2008 at 7:17 PM, Matthew Fluet wrote: > I still don't know that I understand the necessity for the difference > between public and external for an imported symbol. [Indeed, on ELF, there > appears to be no difference.] That's correct. They're the same on ELF. On darwin, however, you need to output that function call stub to access external functions. One could output it always (MLton used to do this), so in this case it's only an optimization. > My intuition is the following: a symbol that > is publically exported can be accessed via an 'external' assembly sequence > by another DSO; since that sequence works for another DSO, why doesn't it > also work for this DSO. This is the case for every single platform except windows. The problem there is that the address of an imported external symbol is copied by the runtime loader into the local symbol __imp__symname. These __imp__symbols appear in your program when you link against the import library for a DLL. So within the DLL itself, __imp__symname doesn't exist (the import library is a separate thing) and you can't access the symbol this way. Outside of the DLL, you need to look in __imp__symname to find the address of the symbol you want to access. >From this pure point-of-view you would always have to get public/external right for linking to work on windows. However, as a compatibility hack to make *nix-like function calls work, MinGW also adds a local function called 'symname' which calls *__imp__symname to the import libraries. That's why niave code can often get away with just accessing a function without the __imp__ during both static and dynamic linking. However, that function is local and has a different address than __imp__symname, so if you take the address of symname, it is your local proxy function, not *__imp__symname like it should be. Furthermore, that trick only works for functions, not variables. > Is it just an optimization? Within the DSO, I can get to the public symbol > via relative addressing, but from another DSO (with PIC), I need an > indirection? Actually, on ELF the optimization is going from public to private. There is no benefit to public over external. If you try to access a public symbol via relative addressing, the linker will rage out. This is due to a "feature" of ELF where you can override symbols. If I have a library with two public methods "foo" and "bar", with foo calling bar, but you link in my library and provide your own bar, my foo will call your bar. I personally think this is a horrible idea, but it is part of the ELF ABI. For this reason on ELF public lookups cannot be done via relative addressing but must use the PLT. So, no. Except for darwin public/external is not an optimization. On windows you really do need to get it right and on ELF it makes no difference. > The other thing I don't understand is how gcc gets by with just two > visibility directives ("default" and "hidden"); how does it make the > private/public/external distinction? Note that default/hidden are ELF specific. As you note earlier, on ELF public=external in every respect. This is why two suffice. There are also many more attributes than just those two. There is also: internal, protected, extern. However, these don't really matter except as optimizations. > I guess it is just for Win{32,64} that > the EXTERNAL/PUBLIC/PRIVATE macros map to different annotations Correct. > (though, I think, that is partly due to the fact that a single C declaration needs to > serve as both an export directive (for the definition) and an import > directive (for other uses)). No. They really do do different things as I explained above. > But, for Darwin, both EXTERNAL and PUBLIC map > to the same annotation --- is the MLton x86/amd64 codegen for Darwin simply > avoiding a cleanup step that would be performed by the linker (if we treated > 'public' as 'external')? I'm not sure what you're asking me..? Currently the darwin codegen avoids making PLT stub indirections for public symbols, but that's just an optimization. Just a note, I think if you tried to use _import "foo" external for an _export "foo" private, you would have problems on both darwin and ELF. Anyway, it's true that no single operating system (so far) needs all three cases at once, although I wouldn't be so sure for other architectures. These three categories were the simplest framework I could come up with that when used according to the rules would work on every system I've seen. The rules aren't that complicated: 1. match private/public within a DSO 2. import using external from another DSO I also had hoped that the three categories are simple to understand. It's not obvious how they map to each platform until you've really understood the ABI of the target, but I am confident that they do work. I am a bit nervous that an as-yet-unsupported platform might have an ABI that doesn't fit into this categorization, but it's hard to imagine how you could break scope further down than private/public/external. From wesley at terpstra.ca Wed Nov 12 11:10:09 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Nov 12 11:10:14 2008 Subject: [MLton] symbol scopes In-Reply-To: <162de7480811121056i6a5ba37am65583e0efde5924f@mail.gmail.com> References: <162de7480811121056i6a5ba37am65583e0efde5924f@mail.gmail.com> Message-ID: <162de7480811121110s20512a58r387f498933145794@mail.gmail.com> On Wed, Nov 12, 2008 at 7:56 PM, Wesley W. Terpstra wrote: >> I guess it is just for Win{32,64} that >> the EXTERNAL/PUBLIC/PRIVATE macros map to different annotations > > Correct. > >> (though, I think, that is partly due to the fact that a single C declaration needs to >> serve as both an export directive (for the definition) and an import >> directive (for other uses)). > > No. They really do do different things as I explained above. Sorry to double post, but I thought I should elaborate a bit more. No annotation when accessing a symbol means: use _symname. No annotation when defining a symbol means: declare it as _symname. The dllimport annotation when accessing a symbol means: use *__imp__symname Defining a symbol as dllimport has no effect (I personally think gcc should error out). The dllexport annotation when accessing a symbol means: use _symname. The dllexport annotation when defining a symbol means: declare it as _symname and add an entry to the section .drectve which lists functions to export. When a DLL is created, all symbols named in the .drectve are made public. That means: the import library has an __imp__symname for each of them, which will be overwritten by the runtime loader. I speculate it also marks up the DLL to tell the loader what to write into these __imp__ symbols from the import library. MinGW also adds a symname proxy for functions. Now suppose we did what you propose: make dllimport/export the same tag. The definition of a function would be _symname, but uses of that symbol within another object final built into the DSO would be called __imp__symname. However, there is no __imp__symname in the resultin DLL, so you would get a link error. Thus the two tags cannot apply to the same symbol within a DSO. A note: Windows defines the canonical address to be the point of definition, which sounds sane, but is actually not true for most platforms. That's why _symname works even for symbols you will export. I already mentioned this is not true for ELF, where the canonical address is actually the stub that appears in the executable, or the executable's own definition (!!!). Don't believe me? Read the sysv abi and cry. :-( From wesley at terpstra.ca Wed Nov 12 11:29:36 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Nov 12 11:29:44 2008 Subject: [MLton] Re: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> <20081111170801.GB13363@laurikari.net> <162de7480811111000k5c2fb26do762eef289fbe4338@mail.gmail.com> Message-ID: <162de7480811121129h5b6fa629j885223e5fb4bd691@mail.gmail.com> On Wed, Nov 12, 2008 at 6:46 PM, Matthew Fluet wrote: > On Tue, 11 Nov 2008, Wesley W. Terpstra wrote: >> One could imagine a default of public for MinGW and external for >> darwin, but that seems pretty arcane. > > It may be arcane, but it can be justified --- this is effectively what > previous versions of mlton did, albeit with some implicit help from the > platform binutils. It's also what gcc does. The gcc-approach works on every platform MLton supports except win32 for address and variable accesses. This is a definite plus, but since it only hides the details of the win32 case without completely solving them, it's not clear to me that flipping the default for this one platform is a great idea. For compatibility with previous MLton versions it may make sense, but then we really should have some sort of warning for imports with no scope, since the default isn't something simple. > [ a long summary ] Yes, all correct. > For Darwin, we produced assembly code for all functions as we now produce > assembly code for 'external' functions. However, the MachO linker was happy > to patch, at link time, any function symbols that were satisfied by static > libraries (i.e., symbols from the same DSO). > However, this implicit help only comes with functions -- that is, symbols > used in a call instruction (I think). Symbols whose addresses are taken > (i.e., via _address and a mov or lea instruction) do not get any implicit > help (I think). I think you have this confused... You can import any public symbol using PLT-style indirections. Variables, functions, addresses, whatever. It is just that I fixed MLton's handling of addresses and variables on darwin at the same time I implemented symbol scopes. The two changes were more-or-less unrelated. > Nonetheless, it does seem as though 'external' is a better default. Yes, it always works on darwin. > I don't know if it is worth adding an 'ffiSymbolScope {error|ignore|warn}' > MLB annotation to signal when FFI is used without an explicit symbol scope. That sounds like a good idea. From fluet at tti-c.org Wed Nov 12 12:03:39 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Nov 12 12:03:37 2008 Subject: [MLton] symbol scopes In-Reply-To: <162de7480811121056i6a5ba37am65583e0efde5924f@mail.gmail.com> References: <162de7480811121056i6a5ba37am65583e0efde5924f@mail.gmail.com> Message-ID: On Wed, 12 Nov 2008, Wesley W. Terpstra wrote: > On Wed, Nov 12, 2008 at 7:17 PM, Matthew Fluet wrote: >> But, for Darwin, both EXTERNAL and PUBLIC map >> to the same annotation --- is the MLton x86/amd64 codegen for Darwin simply >> avoiding a cleanup step that would be performed by the linker (if we treated >> 'public' as 'external')? > > I'm not sure what you're asking me..? Currently the darwin codegen > avoids making PLT stub indirections for public symbols, but that's > just an optimization. Effectively, I'm asking how gcc handles the following on Darwin: z.h: ----- __attribute__((visibility("default"))) int foo(int); ----- y.c: ----- #include "z.h" int bar(int i) { return (foo(i) + foo(i+1)); } ----- gcc -c y.c gcc main.c y.o z.o --- foo turns out to be 'public' gcc -lz main.c y.o --- foo turns out to be 'extern' (-lz satisfied by z.dylib) It seems to me that at the point that gcc needs to produce object code for y.c, it can't know whether it can avoid making PLT stub indirections, because it doesn't know if foo will turn out to be 'public' or 'extern'. So, it has to use the PLT stub indirections. The only time that gcc could avoid the PLT stub indirections are the cases where there is a use of a 'public' symbol in the same compilation unit as the definition of the 'public' symbol. From fluet at tti-c.org Wed Nov 12 12:20:42 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Nov 12 12:20:41 2008 Subject: [MLton] symbol scopes In-Reply-To: <162de7480811121110s20512a58r387f498933145794@mail.gmail.com> References: <162de7480811121056i6a5ba37am65583e0efde5924f@mail.gmail.com> <162de7480811121110s20512a58r387f498933145794@mail.gmail.com> Message-ID: On Wed, 12 Nov 2008, Wesley W. Terpstra wrote: > > Now suppose we did what you propose: make dllimport/export the same > tag. The definition of a function would be _symname, but uses of that > symbol within another object final built into the DSO would be called > __imp__symname. However, there is no __imp__symname in the resultin > DLL, so you would get a link error. Thus the two tags cannot apply to > the same symbol within a DSO. O.k. Given a time machine, this is what I would change. When building a DLL, I would make it so that there are __imp_symname entries for *both* symbols imported from other DLLs and for symbols exported from this DLL. Then one could use *__imp_symname to access an exported symbol, regardless of whether that symbol is exported from the same DLL or from another DLL. But, I can see that, given this time line, one needs to distinguish between 'public' and 'external' for Windows. But, it also emphasizes the point that 'external' is the right default, because 'public' is almost certainly not the scope one wants. From fluet at tti-c.org Wed Nov 12 12:40:53 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Nov 12 12:40:51 2008 Subject: [MLton] symbol scopes In-Reply-To: <162de7480811121056i6a5ba37am65583e0efde5924f@mail.gmail.com> References: <162de7480811121056i6a5ba37am65583e0efde5924f@mail.gmail.com> Message-ID: On Wed, 12 Nov 2008, Wesley W. Terpstra wrote: > Anyway, it's true that no single operating system (so far) needs all > three cases at once, although I wouldn't be so sure for other > architectures. These three categories were the simplest framework I > could come up with that when used according to the rules would work on > every system I've seen. > > The rules aren't that complicated: > 1. match private/public within a DSO > 2. import using external from another DSO > > I also had hoped that the three categories are simple to understand. > It's not obvious how they map to each platform until you've really > understood the ABI of the target, but I am confident that they do > work. It is clear that you've put a fair bit of thought into it. And, there is certainly a design goal here that allows for identical SML source to be used on different platforms. It is a tough situation: understanding one ABI is hard enough; understanding an abstract ABI that doesn't (directly) correspond to any ABI is harder. From wesley at terpstra.ca Wed Nov 12 14:35:06 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Nov 12 14:35:09 2008 Subject: [MLton] symbol scopes In-Reply-To: References: <162de7480811121056i6a5ba37am65583e0efde5924f@mail.gmail.com> Message-ID: <162de7480811121435y11bb354cye6b01f1d7d00e1b9@mail.gmail.com> On Wed, Nov 12, 2008 at 9:03 PM, Matthew Fluet wrote: > It seems to me that at the point that gcc needs to produce object code for > y.c, it can't know whether it can avoid making PLT stub indirections, > because it doesn't know if foo will turn out to be 'public' or 'extern'. So, > it has to use the PLT stub indirections. The only time that gcc could avoid > the PLT stub indirections are the cases where there is a use of a 'public' > symbol in the same compilation unit as the definition of the 'public' > symbol. That's correct. > O.k. Given a time machine, this is what I would change... You lack ambition. :-) From fluet at tti-c.org Wed Nov 12 16:25:15 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Nov 12 16:25:15 2008 Subject: [MLton] Tagged ref and array pointers In-Reply-To: <162de7480811120326v22b0f648gd9eb10e63163b5c@mail.gmail.com> References: <162de7480811120326v22b0f648gd9eb10e63163b5c@mail.gmail.com> Message-ID: On Wed, 12 Nov 2008, Wesley W. Terpstra wrote: > I was debugging what I thought was a space leak when I discovered that > MLton doesn't seem to use the lower 2 bits of a pointer to ref/array > objects. I don't know if it is that MLton doesn't use the lower 2 bits of a pointer to ref/array objects of if it is that MLton doesn't use the lower 2 bits of a pointer out of a vector; there is a {forceBox = true} when computing the representation of vector elements (that may, or may not, be relevant here). > Produces the memory layout in the attached png file. You can see that > it represents the datatype using an intermediate heap object instead > of packing the flag into the low bits of the pointer to the ref cell. > I've seen the same behaviour with datatypes to a vector. MLton does > the right thing other times (tracking the variants of a datatype the > low bits of the pointer) for recursive datatypes and a few other > cases. Is there a good reason for it not performing this optimization > here? Seems like it would be a valid representation decision. I haven't a clue how to coerce packed-representation.fun into making it. From Nicolas.Bertolotti at mathworks.fr Thu Nov 13 13:32:00 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Thu Nov 13 13:32:49 2008 Subject: [MLton] A few changes proposals for MLton Message-ID: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> Skipped content of type multipart/related-------------- next part -------------- A non-text attachment was scrubbed... Name: 21-prevent-null-heap-size.patch Type: application/octet-stream Size: 1369 bytes Desc: 21-prevent-null-heap-size.patch Url : http://mlton.org/pipermail/mlton/attachments/20081113/518c999d/21-prevent-null-heap-size-0001.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: 01-cross-compile-mlton.patch Type: application/octet-stream Size: 6940 bytes Desc: 01-cross-compile-mlton.patch Url : http://mlton.org/pipermail/mlton/attachments/20081113/518c999d/01-cross-compile-mlton-0001.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: 03-windows-versions.patch Type: application/octet-stream Size: 4176 bytes Desc: 03-windows-versions.patch Url : http://mlton.org/pipermail/mlton/attachments/20081113/518c999d/03-windows-versions-0001.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: 04-windows-paths.patch Type: application/octet-stream Size: 4405 bytes Desc: 04-windows-paths.patch Url : http://mlton.org/pipermail/mlton/attachments/20081113/518c999d/04-windows-paths-0001.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: 05-windows-exit.patch Type: application/octet-stream Size: 701 bytes Desc: 05-windows-exit.patch Url : http://mlton.org/pipermail/mlton/attachments/20081113/518c999d/05-windows-exit-0001.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: 18-fix-big-heap-disk-dump.patch Type: application/octet-stream Size: 778 bytes Desc: 18-fix-big-heap-disk-dump.patch Url : http://mlton.org/pipermail/mlton/attachments/20081113/518c999d/18-fix-big-heap-disk-dump-0001.obj From fluet at tti-c.org Fri Nov 14 15:57:11 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Fri Nov 14 15:57:40 2008 Subject: [MLton] A few changes proposals for MLton In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> Message-ID: On Thu, 13 Nov 2008, Nicolas Bertolotti wrote: > * 18-fix-big-heap-disk-dump.patch > * This patch fixes a crash I could observe when a very large heap > is paged to disk. It seems that in some cases, fwrite() fails to > dump the whole heap at once and we have to split. I encountered this as well on x86-linux; an fwrite (data, size, count, file) with size * count >= 2G immediately returns 0 without writing any data and without setting ferror or errno. (I.e., there is no indication as to why the operation failed.) This doesn't happen on x86-darwin or on amd64-linux (though, I didn't try size * count >= 2^63 (which ought to fail anyways, as the memory system and file system can't take it)). Nor is there a corresponding problem with fread. This is rather annoying, as fwrite is supposed to be the abstraction that removes the need for looping writes. With regards to the patch: --- mlton-r6698.patched/runtime/util/safe.h 2008-09-09 10:54:08.000000000 +0200 +++ mlton-r6698/runtime/util/safe.h 2008-08-13 15:07:57.000000000 +0200 @@ -68,8 +68,21 @@ static inline void fwrite_safe (const void *buf, size_t size, size_t count, FILE *f) { size_t res; - - res = fwrite (buf, size, count, f); + enum { + WRITE_CHUNK_SIZE = 0x2000000, /* 32M */ + }; + + res = 0; + while (res < count) { + size_t n; + n = count - res; + if (n > WRITE_CHUNK_SIZE) + n = WRITE_CHUNK_SIZE; + n = fwrite ((void *) ((char *) buf + res), size, n, f); + if (n == 0) + break; + res += n; + } if (res != count) diee ("fwrite (_, %"PRIuMAX", %"PRIuMAX", _) failed " "(only wrote %"PRIuMAX").\n", This kind of misses the point of the *_safe functions --- they are supposed to signal errors, not mask them. The specification of fwrite indicates that any short object count return value is an error. Also, the code above only handles a large count, not a large size. So, it won't help if one makes the seemingly meaning preserving change (in diskBack.unix.c) from: fwrite_safe (buf, 1, size, f); to fwrite_safe (buf, size, 1, f); I would isolate the extreme call to fwrite_safe as in this alternative patch. Although fread doesn't fail on large reads, it seems prudent to limit extreme I/O; also, it makes the program more responsive to ctrl-C (as control frequently returns to user-mode). -------------- next part -------------- diff --git a/runtime/platform/diskBack.unix.c b/runtime/platform/diskBack.unix.c index f9df82d..95bcea0 100644 --- a/runtime/platform/diskBack.unix.c +++ b/runtime/platform/diskBack.unix.c @@ -34,9 +34,21 @@ typedef struct { void GC_diskBack_read (void *data, pointer buf, size_t size) { FILE *f; + const size_t READ_CHUNK_SIZE = 0x2000000; /* 32M */ + f = ((WriteToDiskData)data)->f; fseek_safe (f, 0, SEEK_SET); - fread_safe (buf, 1, size, f); + /* fread (_, 1, size, _) succeeds + * with size >= 2^31 + * for a 32-bit executable on 64-bit linux. + * Nonetheless, match GC_diskBack_write. + */ + while (size > 0) { + size_t s = min (READ_CHUNK_SIZE, size); + fread_safe (buf, 1, s, f); + buf += s; + size -= s; + } } void GC_diskBack_close (void *data) { @@ -51,8 +63,20 @@ void *GC_diskBack_write (pointer buf, size_t size) { FILE *f; WriteToDiskData d; + const size_t WRITE_CHUNK_SIZE = 0x2000000; /* 32M */ + f = tempFileDes (); - fwrite_safe (buf, 1, size, f); + /* fwrite (_, 1, size, _) fails + * (with no helpful error conditions!) + * with size >= 2^31 + * on x86-linux. + */ + while (size > 0) { + size_t s = min (WRITE_CHUNK_SIZE, size); + fwrite_safe (buf, 1, s, f); + buf += s; + size -= s; + } d = (WriteToDiskData)(malloc_safe (sizeof(*d))); d->f = f; return d; From fluet at tti-c.org Fri Nov 14 15:58:00 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Fri Nov 14 15:58:17 2008 Subject: [MLton] A few changes proposals for MLton In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> Message-ID: On Thu, 13 Nov 2008, Nicolas Bertolotti wrote: > * 03-windows-versions.patch > * This patch extends the setSysname() function that is used in > order to generate the "uname" string on MinGW. It makes it possible > to detect whether we are running Windows XP x64, Vista, Vista 64, > 2003 or 2008. > > > * 04-windows-paths.patch > * This patch is highly related to the request Dan DuVarney has > submitted today about OS.Path. It provides support for UNC paths > (\\foo\bar\...) and handles paths like "\foo" as absolute paths > without a volume specification. > > > * 05-windows-exit.patch > * This patch adds support for Posix.Process.exit() on MinGW. I > guess it has been deactivated on purpose but I don't know why and I > have not experienced any issue with it. I have little to nothing to say with regards to Windows; it seems as though there is a chance to make some global decisions regarding OS.Path and Windows on the other thread. Please weigh in on the discussion, as it is clear that you have some practical experiences that motivate changes. I might be wary of extending the standard to include UNC paths, but it does seem as though one can reasonably take UNC paths to denote paths with volumes of the form "\\\\Server\\SharedFolder" (SML-string syntax). It does, however, appear as though your patch allows either of the Windows separators in UNC volume strings. I don't know if Windows interprets "//Server/SharedFolder/foo" as a UNC path. From fluet at tti-c.org Fri Nov 14 16:02:34 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Fri Nov 14 16:02:52 2008 Subject: [MLton] A few changes proposals for MLton In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> Message-ID: On Thu, 13 Nov 2008, Nicolas Bertolotti wrote: > * 01-cross-compile-mlton.patch > * This patch modifies the MLton Makefiles in order to make it > easier to build a MLton cross compiler by cross compiling the > runtime and only use the target machine to run the needed binaries > (gen-sizes, print-constants etc...). It introduces a Makefile > parameter TARGET_MACHINE that can be used in order to provide the > hostname of the machine that should be used to run those binaries. > The machine is then accessed through "ssh". I don't have any experience with using cross compilers, so if it works for you and doesn't impact things when non cross compiling, then it is probably fine. Two notes: diff -Naur mlton-r6658.orig/runtime/Makefile mlton-r6658/runtime/Makefile --- mlton-r6658.orig/runtime/Makefile 2008-06-16 16:03:08.000000000 +0200 +++ mlton-r6658/runtime/Makefile 2008-06-16 16:42:12.000000000 +0200 @@ -285,7 +296,7 @@ gen/basis-ffi.h gen/basis-ffi.sml: gen/gen-basis-ffi.sml gen/basis-ffi.def mlton -output gen/gen-basis-ffi gen/gen-basis-ffi.sml rm -f gen/basis-ffi.h gen/basis-ffi.sml - cd gen && ./gen-basis-ffi + cd gen && $(TARGET_EXEC_BEGIN) ./gen-basis-ffi $(TARGET_EXEC_FILE) rm -f gen/gen-basis-ffi basis-ffi.h: gen/basis-ffi.h This is (a) broken and (b) unecessary. This will have compiled gen-basis-ffi.sml using the host mlton with no -target option, so the resulting gen-basis-ffi executable will be for the host, but it is being executed on the target machine. Also, gen-basis-ffi reads directly from basis-ffi.def and writes directly to basis-ffi.h and basis-ffi.sml, which aren't being copied to/from the target machine. You probably never encountered this because the derived basis-ffi.{h,sml} files are in the repository. In any case, the gen-basis-ffi program simply textually processes basis-ffi.def, with no target dependencies. Hence, there is no need to execute it on the target machine. diff -Naur mlton-r6658.orig/bin/upgrade-basis mlton-r6658/bin/upgrade-basis --- mlton-r6658.orig/bin/upgrade-basis 2008-06-16 16:08:02.000000000 +0200 +++ mlton-r6658/bin/upgrade-basis 2008-06-16 16:40:47.000000000 +0200 @@ -11,14 +11,15 @@ name=`basename "$0"` usage () { - die "usage: $name " + die "usage: $name " } case "$#" in -3) +4) PATH="$1" ARCH="$2" OS="$3" + TARGET="$4" ;; *) usage @@ -28,7 +29,7 @@ tmp="$$.sml" echo "val () = print \"I work\"" >"$tmp" -if ! mlton "$tmp" 1>&2; then +if ! mlton -target $TARGET "$tmp" 1>&2; then die "Error: cannot upgrade basis because the compiler doesn't work" fi @@ -36,7 +37,7 @@ feature="$1" sml="$2" echo "$feature" >"$tmp" - if ! mlton -stop tc "$tmp" >/dev/null 2>&1; then + if ! mlton -target $TARGET -stop tc "$tmp" >/dev/null 2>&1; then echo "$sml" fi } diff -Naur mlton-r6658.orig/mlton/Makefile mlton-r6658/mlton/Makefile --- mlton-r6658.orig/mlton/Makefile 2008-06-16 16:04:17.000000000 +0200 +++ mlton-r6658/mlton/Makefile 2008-06-16 16:40:47.000000000 +0200 @@ -9,8 +9,8 @@ SRC := $(shell cd .. && pwd) BUILD := $(SRC)/build BIN := $(BUILD)/bin -HOST_ARCH := $(shell "$(SRC)/bin/host-arch") -HOST_OS := $(shell "$(SRC)/bin/host-os") +TARGET_ARCH := $(shell "$(SRC)/bin/host-arch") +TARGET_OS := $(shell "$(SRC)/bin/host-os") LIB := $(BUILD)/lib MLTON := mlton TARGET := self @@ -27,12 +27,12 @@ FLAGS += -default-ann 'warnUnused true' # FLAGS += -type-check true -show-types true else -ifeq (cygwin, $(HOST_OS)) +ifeq (cygwin, $(TARGET_OS)) # The stubs don't work on Cygwin, since they define spawn in terms of # fork, and fork doesn't work on Cygwin. So, make without the stubs. FILE := mlton.cm else -ifeq (mingw, $(HOST_OS)) +ifeq (mingw, $(TARGET_OS)) # Ditto for MinGW. FILE := mlton.cm else @@ -87,7 +87,7 @@ #! Pass $(PATH) to upgrade-basis because it is run via #!/usr/bin/env # bash, which resets the path. $(UP): - "$(SRC)/bin/upgrade-basis" '$(PATH)' "$(HOST_ARCH)" "$(HOST_OS)" >$(UP) + "$(SRC)/bin/upgrade-basis" '$(PATH)' "$(TARGET_ARCH)" "$(TARGET_OS)" "$(TARGET)" >$(UP) mlton.sml: $(SOURCES) rm -f mlton.sml && mlton -stop sml mlton.cm && chmod -w mlton.sml It isn't clear to me why upgrade-basis needs to know the target. In any case, your changes don't seem to completely support cross compiling the compiler. For example, in /Makefile, in the 'world-no-check' target, you don't execute the compiler on the target system, which would be necessary after a make TARGET=xx TARGET_ARCH=xx TARGET_OS=xx TARGET_MACHINE=xx compiler to have both build/lib/mlton-compile and build/lib/world.mlton for the target. It seems to me that your changes support the following workflow: # start from a clean slate make clean # bootstrap; resulting compiler only supports -target self make # cross compile the runtime ## first, get self objects/archives out of the way make -C runtime clean make TARGET=xx TARGET_ARCH=xx TARGET_OS=xx TARGET_MACHINE=xx runtime # there are new target specific files in basis-library/config/c ## this is just a target agnostic copy operation make basis-no-check # get the new target constants make TARGET=xx TARGET_ARCH=xx TARGET_OS=xx TARGET_MACHINE=xx constants # register the new target make TARGET=xx TARGET_ARCH=xx TARGET_OS=xx TARGET_MACHINE=xx targetmap Now, "build/bin/mlton -target new" works, but is a self executable. It wouldn't be wise to do make TARGET=xx TARGET_ARCH=xx TARGET_OS=xx TARGET_MACHINE=xx compiler because that would overwrite build/lib/mlton-compile with a target executable. So, despite the existence of the TARGET variable in the mlton/Makefile, I'm not sure how easy/useful it is to produce a cross-compiled mlton-compile from within the Makefile system. But, I could be wrong. From wesley at terpstra.ca Fri Nov 14 17:25:57 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Fri Nov 14 17:26:02 2008 Subject: [MLton] A few changes proposals for MLton In-Reply-To: References: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480811141725y7651526fu3956545633cf8c8f@mail.gmail.com> On Sat, Nov 15, 2008 at 12:58 AM, Matthew Fluet wrote: > On Thu, 13 Nov 2008, Nicolas Bertolotti wrote: > >> * 03-windows-versions.patch >> * This patch extends the setSysname() function that is used in >> order to generate the "uname" string on MinGW. It makes it possible >> to detect whether we are running Windows XP x64, Vista, Vista 64, >> 2003 or 2008. >> > Clearly an improvement. Should be committed. > * 04-windows-paths.patch >> * This patch is highly related to the request Dan DuVarney has >> submitted today about OS.Path. It provides support for UNC paths >> (\\foo\bar\...) and handles paths like "\foo" as absolute paths >> without a volume specification. >> > Let's wait to hear what happens in the sml-basis thread. You should bring up having UNC volumes as that is currently forbidden by the definition. > * 05-windows-exit.patch >> * This patch adds support for Posix.Process.exit() on MinGW. I >> guess it has been deactivated on purpose but I don't know why and I >> have not experienced any issue with it. >> > To be honest, I have no idea why it's disabled either. I was even more mystified when I found that it was added in r3662 which was supposedly a commit on my behalf. I tracked down the email that included the patch I sent. ... but the changes to stub-mingw.sml are nowhere to be seen. Now would be a good time for Stephen to speak up about where this came from! :-) I don't really see any reason NOT to re-enable it, but as neither of us know why it was disabled in the first place... :-/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20081115/2b2d7451/attachment.htm From vesa.a.j.k at gmail.com Mon Nov 17 06:51:53 2008 From: vesa.a.j.k at gmail.com (Vesa Karvonen) Date: Mon Nov 17 06:51:57 2008 Subject: [MLton] Re: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: <162de7480811170342j1f859f6y1152840165d12046@mail.gmail.com> References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> <162de7480811111209x2549aac6y3bf403b14c088c17@mail.gmail.com> <162de7480811111556i2d4f30f7o7cc860b5a69381a6@mail.gmail.com> <162de7480811170342j1f859f6y1152840165d12046@mail.gmail.com> Message-ID: <9e43b9a0811170651k265d04c4u7f2a937ca420b9d7@mail.gmail.com> On Mon, Nov 17, 2008 at 1:42 PM, Wesley W. Terpstra wrote: > Never heard back from you about this ... > > ---------- Forwarded message ---------- > From: Wesley W. Terpstra > Date: Wed, Nov 12, 2008 at 12:56 AM > Subject: Re: [MLton] Re: [MLton-user] SVN r6941 MLton/MinGW32 and FFI > To: Matthew Fluet > Cc: "mlton@mlton.org" > > > On Wed, Nov 12, 2008 at 12:26 AM, Matthew Fluet wrote: >>> So why were the dl* functions added to mingw.c in the first place? >> Looks like the origin was to get mlnlffi working on MinGW: >> http://mlton.org/pipermail/mlton/2006-November/029349.html Yes, that is why I initially wrote them. It is a minimal implementation and doesn't necessarily work as an absolute complete replacement for dl* functions. > Using dlfcn-win32 does seem the best approach. I'd like to hear > something from Vesa before removing them. Frankly, after a very brief look at the code of dlfcn-win32 (http://code.google.com/p/dlfcn-win32/source/browse/trunk/dlfcn.c), I'm not very impressed with it. I may be mistaken and it might be mostly harmless, but it seems to leak library handles when RTLD_GLOBAL is specified. Specifically, the code seems to first execute LoadLibraryEx, which increments the reference count of the library, then if RTLD_GLOBAL is specified, the library handle is added to the doubly linked list (#1) --- except when the library was already loaded, in which case no matching call to FreeLibrary seems to be made, which means that the handle was leaked. But, it is probably better to use it anyway (at least after all the obvious bugs have been fixed), because then we don't have to maintain a similar library. -Vesa Karvonen #1) I might add here that a singly linked list would work just as well. From wesley at terpstra.ca Mon Nov 17 07:00:08 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Nov 17 07:00:12 2008 Subject: [MLton] Re: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: <9e43b9a0811170651k265d04c4u7f2a937ca420b9d7@mail.gmail.com> References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> <162de7480811111209x2549aac6y3bf403b14c088c17@mail.gmail.com> <162de7480811111556i2d4f30f7o7cc860b5a69381a6@mail.gmail.com> <162de7480811170342j1f859f6y1152840165d12046@mail.gmail.com> <9e43b9a0811170651k265d04c4u7f2a937ca420b9d7@mail.gmail.com> Message-ID: <162de7480811170700w3122b873y173df0268dc4b436@mail.gmail.com> On Mon, Nov 17, 2008 at 3:51 PM, Vesa Karvonen wrote: > >> Looks like the origin was to get mlnlffi working on MinGW: > >> http://mlton.org/pipermail/mlton/2006-November/029349.html > > Yes, that is why I initially wrote them. It is a minimal > implementation and doesn't necessarily work as an absolute complete > replacement for dl* functions. Ok, so we can remove the MLton ones? I think they are broken atm anyway since they get prefix with MLton_ by the mingw.h. > Using dlfcn-win32 does seem the best approach. I'd like to hear > > something from Vesa before removing them. > > Frankly, after a very brief look at the code of dlfcn-win32 > (http://code.google.com/p/dlfcn-win32/source/browse/trunk/dlfcn.c), > I'm not very impressed with it. I may be mistaken and it might be > mostly harmless, but it seems to leak library handles when RTLD_GLOBAL > is specified. Specifically, the code seems to first execute > LoadLibraryEx, which increments the reference count of the library, > then if RTLD_GLOBAL is specified, the library handle is added to the > doubly linked list (#1) --- except when the library was already > loaded, in which case no matching call to FreeLibrary seems to be > made, which means that the handle was leaked. > > But, it is probably better to use it anyway (at least after all the > obvious bugs have been fixed), because then we don't have to maintain > a similar library. Since you are obviously the resident expert, maybe help them it? I was thinking we could just include their (fixed) library in the MLton mingw installer. That way things will just work for users by adding -ldl, but otherwise we don't throw these symbols into your program. This seems the ideal solution to me since as DLL the wiki _import libdl examples will work unchanged with their default external scope. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20081117/10d0eb12/attachment.html From fluet at tti-c.org Mon Nov 17 11:21:21 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Mon Nov 17 11:21:47 2008 Subject: [MLton] A few changes proposals for MLton In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> Message-ID: On Thu, 13 Nov 2008, Nicolas Bertolotti wrote: > * 21-prevent-null-heap-size.patch > * This patch addresses the issue I submitted a few days ago > (Segmentation fault on a 6 GB machine with ram-slop 0.85) --- mlton-r6698.patched/runtime/gc/heap.c 2008-11-03 23:45:47.000000000 +0100 +++ mlton-r6698/runtime/gc/heap.c 2008-11-05 23:53:54.000000000 +0100 @@ -39,6 +39,7 @@ size_t sizeofHeapDesired (GC_state s, size_t liveSize, size_t currentSize) { size_t liveWithMapsSize; size_t res; + size_t maxSize; float withMapsRatio; liveSize = align (liveSize, s->sysvals.pageSize); @@ -82,6 +83,9 @@ res = s->sysvals.ram; } else { /* Required live ratio. */ res = liveSize * s->controls.ratios.markCompact; + /* Avoid overflow */ + if (res < liveSize) + res = SIZE_MAX; /* If the current heap is bigger than res, then shrinking always * sounds like a good idea. However, depending on what pages the * VM keeps around, growing could be very expensive, if it @@ -104,6 +108,16 @@ die ("Out of memory with max heap size %s.", uintmaxToCommaString(s->controls.maxHeap)); } + + /* Avoid overflow. */ + maxSize = align(SIZE_MAX - s->sysvals.pageSize, s->sysvals.pageSize); + maxSize -= sizeofCardMapAndCrossMap(s, maxSize); + if (res > maxSize) + res = maxSize; + if (res < liveSize) + die ("Out of memory with system max heap size %s.", + uintmaxToCommaString(res)); + if (DEBUG_RESIZING) fprintf (stderr, "%s = sizeofHeapDesired (%s, %s)\n", uintmaxToCommaString(res), Looks simple and effective. Equivalent, but perhaps easier to understand would be to replace + maxSize = align(SIZE_MAX - s->sysvals.pageSize, s->sysvals.pageSize); with + maxSize = alignDown(SIZE_MAX, s->sysvals.pageSize); From Nicolas.Bertolotti at mathworks.fr Mon Nov 17 14:46:51 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Mon Nov 17 14:47:28 2008 Subject: [MLton] A few changes proposals for MLton In-Reply-To: References: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> Message-ID: <8320D98DA9A5C54C926D397795FE7CEA385AE94E72@EXCHANGE-UK.ad.mathworks.com> Fine for me (I just didn't know we had this function available). I am also not really sure that the proposed operation: > + maxSize = alignDown(SIZE_MAX, s->sysvals.pageSize); > + maxSize -= sizeofCardMapAndCrossMap(s, maxSize); really returns the exact maximum size for a heap. It is not really an issue but you may think of a better way. It could also be slightly more efficient to compute this value only once at the beginning of the execution. Nicolas > -----Original Message----- > From: Matthew Fluet [mailto:fluet@tti-c.org] > Sent: Monday, November 17, 2008 8:21 PM > To: Nicolas Bertolotti > Cc: mlton@mlton.org > Subject: Re: [MLton] A few changes proposals for MLton > > On Thu, 13 Nov 2008, Nicolas Bertolotti wrote: > > * 21-prevent-null-heap-size.patch > > * This patch addresses the issue I submitted a few days ago > > (Segmentation fault on a 6 GB machine with ram-slop 0.85) > > --- mlton-r6698.patched/runtime/gc/heap.c 2008-11-03 > 23:45:47.000000000 +0100 > +++ mlton-r6698/runtime/gc/heap.c 2008-11-05 23:53:54.000000000 > +0100 > @@ -39,6 +39,7 @@ > size_t sizeofHeapDesired (GC_state s, size_t liveSize, size_t > currentSize) { > size_t liveWithMapsSize; > size_t res; > + size_t maxSize; > float withMapsRatio; > > liveSize = align (liveSize, s->sysvals.pageSize); > @@ -82,6 +83,9 @@ > res = s->sysvals.ram; > } else { /* Required live ratio. */ > res = liveSize * s->controls.ratios.markCompact; > + /* Avoid overflow */ > + if (res < liveSize) > + res = SIZE_MAX; > /* If the current heap is bigger than res, then shrinking always > * sounds like a good idea. However, depending on what pages the > * VM keeps around, growing could be very expensive, if it > @@ -104,6 +108,16 @@ > die ("Out of memory with max heap size %s.", > uintmaxToCommaString(s->controls.maxHeap)); > } > + > + /* Avoid overflow. */ > + maxSize = align(SIZE_MAX - s->sysvals.pageSize, s->sysvals.pageSize); > + maxSize -= sizeofCardMapAndCrossMap(s, maxSize); > + if (res > maxSize) > + res = maxSize; > + if (res < liveSize) > + die ("Out of memory with system max heap size %s.", > + uintmaxToCommaString(res)); > + > if (DEBUG_RESIZING) > fprintf (stderr, "%s = sizeofHeapDesired (%s, %s)\n", > uintmaxToCommaString(res), > > > Looks simple and effective. Equivalent, but perhaps easier to understand > would be to replace > > + maxSize = align(SIZE_MAX - s->sysvals.pageSize, s->sysvals.pageSize); > > with > > + maxSize = alignDown(SIZE_MAX, s->sysvals.pageSize); From Nicolas.Bertolotti at mathworks.fr Mon Nov 17 14:49:25 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Mon Nov 17 14:50:06 2008 Subject: [MLton] A few changes proposals for MLton In-Reply-To: <162de7480811141725y7651526fu3956545633cf8c8f@mail.gmail.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> <162de7480811141725y7651526fu3956545633cf8c8f@mail.gmail.com> Message-ID: <8320D98DA9A5C54C926D397795FE7CEA385AE94E73@EXCHANGE-UK.ad.mathworks.com> Let's wait to hear what happens in the sml-basis thread. You should bring up having UNC volumes as that is currently forbidden by the definition. I am sorry but I don't really understand what you mean with < You should bring up having UNC volumes as that is currently forbidden by the definition > (maybe it is just a problem with my English understanding). * 05-windows-exit.patch * This patch adds support for Posix.Process.exit() on MinGW. I guess it has been deactivated on purpose but I don't know why and I have not experienced any issue with it. To be honest, I have no idea why it's disabled either. I was even more mystified when I found that it was added in r3662 which was supposedly a commit on my behalf. I tracked down the email that included the patch I sent. ... but the changes to stub-mingw.sml are nowhere to be seen. Now would be a good time for Stephen to speak up about where this came from! :-) I don't really see any reason NOT to re-enable it, but as neither of us know why it was disabled in the first place... :-/ Funny !!! Wait and see... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20081117/2325ecbd/attachment.htm From Nicolas.Bertolotti at mathworks.fr Mon Nov 17 15:04:37 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Mon Nov 17 15:05:13 2008 Subject: [MLton] A few changes proposals for MLton In-Reply-To: References: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> Message-ID: <8320D98DA9A5C54C926D397795FE7CEA385AE94E74@EXCHANGE-UK.ad.mathworks.com> > It isn't clear to me why upgrade-basis needs to know the target. In any upgrade-basis generates the MLton structure with a value for Platform.host and OS.host that is based on the target so I guess this has an impact on those values when they are used in the compiler binary itself. Anyway, before I implemented those changes, I remember my cross compiler already worked fine so I am not really sure this is really important. > case, your changes don't seem to completely support cross compiling the > compiler. For example, in /Makefile, in the 'world-no-check' target, > you don't execute the compiler on the target system, which would be > necessary after a > make TARGET=xx TARGET_ARCH=xx TARGET_OS=xx TARGET_MACHINE=xx compiler > to have both build/lib/mlton-compile and build/lib/world.mlton for the > target. Well, I actually use the changes to rebuild a 32 bit version of the compiler (to use less RAM) after building it natively on a 64 bit platform. I do this by running: $ touch mlton/main.sml $ make compiler TARGET=... ... and finally: $ make world-no-check TARGET=... ... (to rebuild the main compiler binary only) or: $ make world libraries tools TARGET=... ... (to rebuild the tools as well) After running: $ make all-no-docs But I do this using an external script so I guess I should add a bit of additional packaging (Makefile modifications) in order to propose an efficient complete cross compilation entry. From wesley at terpstra.ca Mon Nov 17 15:32:49 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Nov 17 15:32:53 2008 Subject: [MLton] A few changes proposals for MLton In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA385AE94E74@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> <8320D98DA9A5C54C926D397795FE7CEA385AE94E74@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480811171532g505e9594l5004ce282c61a62b@mail.gmail.com> On Tue, Nov 18, 2008 at 12:04 AM, Nicolas Bertolotti < Nicolas.Bertolotti@mathworks.fr> > > Well, I actually use the changes to rebuild a 32 bit version of the > compiler (to use less RAM) after building it natively on a 64 bit platform. Just run "i386 make"... The latest svn should build you a 32 bit MLton on your 64 bit machine. No patches or cross-compiling needed. If you have debian, make sure you have gcc-multilib and installed. You might need a few other lib32 packages as well. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20081118/dbfedf27/attachment.html From wesley at terpstra.ca Mon Nov 17 15:44:37 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Nov 17 15:44:41 2008 Subject: [MLton] A few changes proposals for MLton In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA385AE94E73@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> <162de7480811141725y7651526fu3956545633cf8c8f@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA385AE94E73@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480811171544q792f6682y75c8b36915cf0ca6@mail.gmail.com> On Mon, Nov 17, 2008 at 11:49 PM, Nicolas Bertolotti < Nicolas.Bertolotti@mathworks.fr> wrote: > I am sorry but I don't really understand what you mean with ? You should > bring up having UNC volumes as that is currently forbidden by the > definition ? (maybe it is just a problem with my English understanding). > Your patch changes OS.Path.fromString "\\pumpkin\media\foo" to return { vol="\\pumpkin\media", isAbs=true, arcs=["foo"] }. The problem with that is that the standard states "Under Microsoft Windows, the valid volume names have the form "a:", "A:", "b:", "B:", etc". That list pretty clearly doesn't intend to allow \\pumpkin\media as a volume. There was a discussion going on between Andreas and John Reppy (the basis specification editor) about the OS.Path structure. I was proposing that you join the discussion and see if the basis specification could be changed to allow vol="\\pumpkin\media" the way your patch does. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20081118/eeafb5da/attachment.htm From Nicolas.Bertolotti at mathworks.fr Mon Nov 17 23:33:25 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Mon Nov 17 23:34:02 2008 Subject: [MLton] A few changes proposals for MLton In-Reply-To: <162de7480811171532g505e9594l5004ce282c61a62b@mail.gmail.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> <8320D98DA9A5C54C926D397795FE7CEA385AE94E74@EXCHANGE-UK.ad.mathworks.com> <162de7480811171532g505e9594l5004ce282c61a62b@mail.gmail.com> Message-ID: <8320D98DA9A5C54C926D397795FE7CEA385AE94E7B@EXCHANGE-UK.ad.mathworks.com> Well, I also use the patch to build the cross-runtimes for solaris, MinGW, Win64, Linux 32 and Linux 64 (for Linux, I also use Linux -> Linux cross compilers in order to produce some code that is independent from the O.S. the build machine is running). ________________________________ From: Wesley W. Terpstra [mailto:wesley@terpstra.ca] Sent: Tuesday, November 18, 2008 12:33 AM To: Nicolas Bertolotti Cc: Matthew Fluet; mlton@mlton.org Subject: Re: [MLton] A few changes proposals for MLton On Tue, Nov 18, 2008 at 12:04 AM, Nicolas Bertolotti > Well, I actually use the changes to rebuild a 32 bit version of the compiler (to use less RAM) after building it natively on a 64 bit platform. Just run "i386 make"... The latest svn should build you a 32 bit MLton on your 64 bit machine. No patches or cross-compiling needed. If you have debian, make sure you have gcc-multilib and installed. You might need a few other lib32 packages as well. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20081118/5fdbc858/attachment.html From Nicolas.Bertolotti at mathworks.fr Mon Nov 17 23:37:51 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Mon Nov 17 23:38:27 2008 Subject: [MLton] A few changes proposals for MLton In-Reply-To: <162de7480811171544q792f6682y75c8b36915cf0ca6@mail.gmail.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> <162de7480811141725y7651526fu3956545633cf8c8f@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA385AE94E73@EXCHANGE-UK.ad.mathworks.com> <162de7480811171544q792f6682y75c8b36915cf0ca6@mail.gmail.com> Message-ID: <8320D98DA9A5C54C926D397795FE7CEA385AE94E7C@EXCHANGE-UK.ad.mathworks.com> The only reason why I do this is that it is not possible to do a < chdir > to "\\pumpkin". This is how Windows works (too bad) and I'm afraid there is no other way to support UNC paths. I will join the discussion... ________________________________ From: Wesley W. Terpstra [mailto:wesley@terpstra.ca] Sent: Tuesday, November 18, 2008 12:45 AM To: Nicolas Bertolotti Cc: Matthew Fluet; mlton@mlton.org Subject: Re: [MLton] A few changes proposals for MLton On Mon, Nov 17, 2008 at 11:49 PM, Nicolas Bertolotti > wrote: I am sorry but I don't really understand what you mean with < You should bring up having UNC volumes as that is currently forbidden by the definition > (maybe it is just a problem with my English understanding). Your patch changes OS.Path.fromString "\\pumpkin\media\foo" to return { vol="\\pumpkin\media", isAbs=true, arcs=["foo"] }. The problem with that is that the standard states "Under Microsoft Windows, the valid volume names have the form "a:", "A:", "b:", "B:", etc". That list pretty clearly doesn't intend to allow \\pumpkin\media as a volume. There was a discussion going on between Andreas and John Reppy (the basis specification editor) about the OS.Path structure. I was proposing that you join the discussion and see if the basis specification could be changed to allow vol="\\pumpkin\media" the way your patch does. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20081118/baa605d0/attachment.htm From fluet at tti-c.org Tue Nov 18 19:46:48 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Tue Nov 18 19:47:20 2008 Subject: [MLton] A few changes proposals for MLton In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA385AE94E72@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA385AE948FB@EXCHANGE-UK.ad.mathworks.com> <8320D98DA9A5C54C926D397795FE7CEA385AE94E72@EXCHANGE-UK.ad.mathworks.com> Message-ID: On Mon, 17 Nov 2008, Nicolas Bertolotti wrote: > I am also not really sure that the proposed operation: >> + maxSize = alignDown(SIZE_MAX, s->sysvals.pageSize); >> + maxSize -= sizeofCardMapAndCrossMap(s, maxSize); > really returns the exact maximum size for a heap. It is not really an > issue but you may think of a better way. True, your operation yields a maxSize that is slightly smaller than necessary, because it uses a larger than necessary card/cross map size. One could write an invertSizeofCardMapAndCrossMap function such that: heapSize = invertSizeofCardMapAndCrossMap (s, heapWithMapsSize); implies heapSize + sizeofCardMapAndCrossMap (s, heapSize) <= heapWithMapsSize < (heapSize + s->sysvals.pageSize) + sizeofCardMapAndCrossMap (s, heapSize + s->sysvals.pageSize) which yields the largest (up to the pageSize) heapSize such that adding its card/cross map size requires space less than or equal to the input heapWithMapsSize. According to my calculations, your operation yields: maxSize = 4,294,963,200 - 33,554,432 = 4,261,408,768 with sizeofCardMapAndCrossMap(maxSize) = 33,292,288 and maxSize + sizeofCardMapAndCrossMap(maxSize) = 4,294,701,056 However, one could take: maxSize = 4,261,662,720 with sizeofCardMapAndCrossMap(maxSize) = 33,300,480 and maxSize + sizeofCardMapAndCrossMap(maxSize) = 4,294,963,200 In practice, though, either of these values for maxSize is well above what is available in a contiguous mmap allocation (either with mremap or after dumping the heap to disk). > It could also be slightly more efficient to compute this value only once > at the beginning of the execution. Agreed, although sizeofHeapDesired isn't on a critical path, so I doubt that it would have any measurable effect. From ville at laurikari.net Fri Nov 21 06:42:47 2008 From: ville at laurikari.net (Ville Laurikari) Date: Fri Nov 21 06:42:52 2008 Subject: [MLton] Re: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: <162de7480811170700w3122b873y173df0268dc4b436@mail.gmail.com> References: <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> <162de7480811111209x2549aac6y3bf403b14c088c17@mail.gmail.com> <162de7480811111556i2d4f30f7o7cc860b5a69381a6@mail.gmail.com> <162de7480811170342j1f859f6y1152840165d12046@mail.gmail.com> <9e43b9a0811170651k265d04c4u7f2a937ca420b9d7@mail.gmail.com> <162de7480811170700w3122b873y173df0268dc4b436@mail.gmail.com> Message-ID: <20081121144247.GA5370@laurikari.net> On Mon, Nov 17, 2008 at 04:00:08PM +0100, Wesley W. Terpstra wrote: > Ok, so we can remove the MLton ones? I think they are broken atm anyway > since they get prefix with MLton_ by the mingw.h. I can confirm that the current dlopen and friends from mingw.c don't work, exactly because they are prefixed with MLton_. > I was thinking we could just include their (fixed) library in the > MLton mingw installer. This seems like a good idea. -- http://www.iki.fi/vl/ From ville at laurikari.net Mon Nov 24 06:55:57 2008 From: ville at laurikari.net (Ville Laurikari) Date: Mon Nov 24 06:56:02 2008 Subject: [MLton] Including dlfcn-win32 in MLton Message-ID: <20081124145557.GA8029@laurikari.net> Hi, There was discussion a while ago regarding removing the built-in implementation on dlopen() and friends from the MLton runtime (on MinGW), and using dlfcn-win32 instead. An unpatched version of dlfcn-win32 does not work at all with MLton, which Dan DuVarney no doubt has noticed and made a patch: http://code.google.com/p/dlfcn-win32/issues/detail?id=2 With this patch, it at least appears to work. Before, programs would just throw a DynLinkError with an empty messag. I didn't check if it leaks handles or not, but I believe Vesa in that it does leak. Dan, have you had any luck in getting commit access to get your patch included in the official SVN repo? My point is that if we plan to use dlfcn-win32 in MLton, it would be good that the upstream version works as-is. Having to patch it sort of defeats the purpose of not having to maintain our own code for dlopen(). BTW, I could not find an email address for Ramiro Polla, the author of dlfcn-win32. Does anyone have it? -- http://www.iki.fi/vl/ From duvarney at reactive-systems.com Tue Nov 25 15:45:19 2008 From: duvarney at reactive-systems.com (Dan DuVarney) Date: Tue Nov 25 15:45:29 2008 Subject: [MLton] Re: Including dlfcn-win32 in MLton In-Reply-To: <20081124145557.GA8029@laurikari.net> References: <20081124145557.GA8029@laurikari.net> Message-ID: <492C8E0F.6040609@reactive-systems.com> Ville Laurikari wrote: > Dan, have you had any luck in getting commit access to get your patch > included in the official SVN repo I'm afraid I haven't been able to get in touch with Ramiro Polla. If you find his e-mail address, please let me know, too. -Dan -- Dan DuVarney Senior Software Engineer Reactive Systems Inc. duvarney@reactive-systems.com +1 (919) 324-3507 ext. 103 OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~duvarney/pgp_public_key.txt