From terpstra at gmail.com Tue Aug 5 17:58:25 2008 From: terpstra at gmail.com (Wesley W. Terpstra) Date: Tue Aug 5 17:58:30 2008 Subject: [MLton] Re: PIC in amd64 assembly In-Reply-To: References: Message-ID: Since I my last post, I've found a very useful link: http://hackage.haskell.org/trac/ghc/wiki/Commentary/PositionIndependentCode I recommend it to anyone interested in how dynamic PDC/PIC code looks on various platforms. As an aside to Matthew, that page indicates what I suspected about osx. You don't need the indirect symbols except for external symbols. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080806/610751f4/attachment.htm From terpstra at gmail.com Fri Aug 8 11:06:14 2008 From: terpstra at gmail.com (Wesley W. Terpstra) Date: Fri Aug 8 11:06:24 2008 Subject: [MLton] Working: MLton/Win64 toolchain Message-ID: I'm happy to announce that MLton now works perfectly on 64-bit windows using the native codegen. Please give it a spin and report any problems. It has only been tested on Win2003/64 and I would be very interested in hearing reports about XP64 and Vista64. A complete toolchain (no additional software required) can be found at: http://www.dvs.tu-darmstadt.de/staff/terpstra/mlton-20080808.win64.zip If someone could move this to the Experimental page on the wiki, I would appreciate it (I don't have write permissions there). Be warned that the toolchain is a 180MB download. To use the toolchain, uncompress the zip file wherever you like. The contained MLton\bin\mlton.bat should work from the windows shell. If you want to run the included 'msys.bat' to get a UNIX-like environment (make, sed, tar, etc), you must decompress the tarball to a location which does not include spaces in the path name. To rebuild MLton/svn, use 'make all-no-docs' in a checkout with the msys shell. The toolchain includes: * a 20080728 mingw-w64 toolchain * gmp-mpfr mingw64 fork, 20080604 * msys 1.0.11pre, 20070119 * MLton r6698, 20080808 There remain problems with code generated by mingw-w64 gcc. The mingw toolchain included is actually 32 bit, cross-compiling to 64 in order to avoid problems with the native compiled linker. For this reason, I don't advise using the c-codegen. The bytecodegen and amdcodegen are fine as gcc has very little to do. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080808/2fd0a6fe/attachment.htm From Nicolas.Bertolotti at mathworks.fr Mon Aug 11 06:51:18 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Mon Aug 11 06:52:11 2008 Subject: [MLton] Working: MLton/Win64 toolchain In-Reply-To: References: Message-ID: <8320D98DA9A5C54C926D397795FE7CEA28778C16B1@EXCHANGE-UK.ad.mathworks.com> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: essai.tar.gz Type: application/x-gzip Size: 24447 bytes Desc: essai.tar.gz Url : http://mlton.org/pipermail/mlton/attachments/20080811/2e3f31b5/essai.tar-0001.bin From Nicolas.Bertolotti at mathworks.fr Mon Aug 11 07:02:39 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Mon Aug 11 07:03:36 2008 Subject: FW: [MLton] Working: MLton/Win64 toolchain Message-ID: <8320D98DA9A5C54C926D397795FE7CEA28778C16B6@EXCHANGE-UK.ad.mathworks.com> Skipped content of type multipart/alternative-------------- next part -------------- #include struct my_struct { void *p; unsigned char buf[16384]; }; void func(void *p) { } extern void func2(void ** pp) { struct my_struct s; if (pp == NULL) { func(&s.p); } else { s.p = *pp; } fprintf(stderr, "dummy2\n"); } extern void ffi_func() { func2(NULL); fprintf(stderr, "Test OK\n"); } #ifdef C_MAIN int main() { ffi_func(); return 0; } #endif -------------- next part -------------- A non-text attachment was scrubbed... Name: main.sml Type: application/octet-stream Size: 70 bytes Desc: main.sml Url : http://mlton.org/pipermail/mlton/attachments/20080811/aa63f74e/main.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: Makefile Type: application/octet-stream Size: 480 bytes Desc: Makefile Url : http://mlton.org/pipermail/mlton/attachments/20080811/aa63f74e/Makefile.obj From wesley at terpstra.ca Mon Aug 11 14:28:01 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Aug 11 14:28:07 2008 Subject: FW: [MLton] Working: MLton/Win64 toolchain In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA28778C16B6@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA28778C16B6@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480808111428s54218392ueae9f78e8bb0da61@mail.gmail.com> On Mon, Aug 11, 2008 at 4:02 PM, Nicolas Bertolotti < Nicolas.Bertolotti@mathworks.fr> > > I built a cross compiler from Linux to Win64 and tried to port my product > using it. So far, it seems that the pure SML code runs pretty fine on Win64 > but there still a few issues elsewhere. > > 1. Spawn > > I faced some troubles spawning child processes when the arguments contain > some backslashes. Anyway, I am not sure this is related to the port as it > also affects Win32 : > > I could notice that a new cmdEscape() function has been recently added to > fix a common issue with _spawnve() on Windows when the arguments contain > some white spaces. This function duplicates the backslashes which appears to > be incorrect. > I'm looking into this. The fix is not as simple as your proposal. It seems some \s need to be escaped, but not others. It also apparently differs between MinGW and cygwin. :-/ > 2. FFI > > I am now facing a more complex issue with the foreign function interface. > You will find attached a small example (30 lines of C code ; 2 lines of SML > code) which reproduces the issue I am facing. In the example, when > ffi_func() is called from SML code, it crashes (probably some kind of stack > corruption) whereas it runs fine when it is executed from a C main(). > This isn't MLton's fault per-se. If you link your test program with -O1 and -lkernel32, you will see the exact same problem with a pure C program. eg: gcc -Wall -O1 main.c ffi.c -o main.exe -lkernel32 with main.c: void ffi_func(); int main() { ffi_func(); return 0; } I have seen this before. With win64 big allocations on the stack seem flaky. I had to hack around this for gmp as well. Why gcc gets this wrong, I don't know. If you remove -lkernel32 or use -O2, the problem goes away. Probably your best bet is to not use my_struct s as a stack-allocated variable. I've noticed that mingw also links to kernel32 on its own (now?). So you could edit the mlton script to not include -lkernel32, which also seems to fix it for some reason. I'm rebuilding MLton to see whether all regressions pass with -lkernel32 removed from the link line. If it works (and I'll have to check win32 as well), I'll think about removing this from the mlton script by default. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080811/5d5c2708/attachment.htm From wesley at terpstra.ca Mon Aug 11 14:28:32 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Aug 11 14:28:37 2008 Subject: FW: [MLton] Working: MLton/Win64 toolchain In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA28778C16B6@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA28778C16B6@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480808111428r2fcd227eob9743f0d1b862c18@mail.gmail.com> On Mon, Aug 11, 2008 at 4:02 PM, Nicolas Bertolotti < Nicolas.Bertolotti@mathworks.fr> > > I built a cross compiler from Linux to Win64 and tried to port my product > using it. So far, it seems that the pure SML code runs pretty fine on Win64 > but there still a few issues elsewhere. > > 1. Spawn > > I faced some troubles spawning child processes when the arguments contain > some backslashes. Anyway, I am not sure this is related to the port as it > also affects Win32 : > > I could notice that a new cmdEscape() function has been recently added to > fix a common issue with _spawnve() on Windows when the arguments contain > some white spaces. This function duplicates the backslashes which appears to > be incorrect. > I'm looking into this. The fix is not as simple as your proposal. It seems some \s need to be escaped, but not others. It also apparently differs between MinGW and cygwin. :-/ > 2. FFI > > I am now facing a more complex issue with the foreign function interface. > You will find attached a small example (30 lines of C code ; 2 lines of SML > code) which reproduces the issue I am facing. In the example, when > ffi_func() is called from SML code, it crashes (probably some kind of stack > corruption) whereas it runs fine when it is executed from a C main(). > This isn't MLton's fault per-se. If you link your test program with -O1 and -lkernel32, you will see the exact same problem with a pure C program. eg: gcc -Wall -O1 main.c ffi.c -o main.exe -lkernel32 with main.c: void ffi_func(); int main() { ffi_func(); return 0; } I have seen this before. With win64 big allocations on the stack seem flaky. I had to hack around this for gmp as well. Why gcc gets this wrong, I don't know. If you remove -lkernel32 or use -O2, the problem goes away. Probably your best bet is to not use my_struct s as a stack-allocated variable. I've noticed that mingw also links to kernel32 on its own (now?). So you could edit the mlton script to not include -lkernel32, which also seems to fix it for some reason. I'm rebuilding MLton to see whether all regressions pass with -lkernel32 removed from the link line. If it works (and I'll have to check win32 as well), I'll think about removing this from the mlton script by default. PS. I couldn't reply to you directly. Check your mail server... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080811/3f8d0148/attachment.html From wesley at terpstra.ca Mon Aug 11 16:20:18 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Aug 11 16:20:22 2008 Subject: FW: [MLton] Working: MLton/Win64 toolchain In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA28778C16B6@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA28778C16B6@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480808111620g59e5836cl6e4283aa12c587ab@mail.gmail.com> On Mon, Aug 11, 2008 at 4:02 PM, Nicolas Bertolotti < Nicolas.Bertolotti@mathworks.fr> > > I built a cross compiler from Linux to Win64 and tried to port my product > using it. So far, it seems that the pure SML code runs pretty fine on Win64 > but there still a few issues elsewhere. > > 1. Spawn > > I faced some troubles spawning child processes when the arguments contain > some backslashes. Anyway, I am not sure this is related to the port as it > also affects Win32 : > > I could notice that a new cmdEscape() function has been recently added to > fix a common issue with _spawnve() on Windows when the arguments contain > some white spaces. This function duplicates the backslashes which appears to > be incorrect. > I've fixed this in svn. The fix is not as simple as your proposal. Some \s need to be escaped, but not others. It also apparently differs between MinGW and cygwin. :-/ Please give the process.sml in svn/HEAD a whirl. Do you have cygwin also? I would like to know how regression/spawn.sml fairs there. > 2. FFI > > I am now facing a more complex issue with the foreign function interface. > You will find attached a small example (30 lines of C code ; 2 lines of SML > code) which reproduces the issue I am facing. In the example, when > ffi_func() is called from SML code, it crashes (probably some kind of stack > corruption) whereas it runs fine when it is executed from a C main(). > This isn't MLton's fault. If you link your test program with -O1 and -lkernel32, you will see the exact same problem with a pure C program. eg: gcc -Wall -O1 main.c ffi.c -o main.exe -lkernel32 with main.c: void ffi_func(); int main() { ffi_func(); return 0; } I have seen this before. With win64 you can't make big allocations on the stack. I had to hack around this for gmp as well. Why gcc gets this wrong, I don't know. If you remove -lkernel32 or use -O2, the problem goes away. Probably your best bet is to not use my_struct as a stack-allocated variable. I've noticed that mingw (now?) also links to kernel32 on its own. So you could edit the mlton script to not include -lkernel32. I've done a full rebuild and regression test of MLton with -lkernel32 removed, and it worked on my win64 toolchain. If it works for win32 as well, and you can confirm that removing -lkernel32 resolves your FFI problems, I'll probably just remove it in svn too. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080812/7e8ff597/attachment.htm From wesley at terpstra.ca Mon Aug 11 16:27:06 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Aug 11 16:27:40 2008 Subject: FW: [MLton] Working: MLton/Win64 toolchain In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA28778C16E5@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA28778C16B6@EXCHANGE-UK.ad.mathworks.com> <162de7480808111418x1e58f57avb2ec2dfc5096665c@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA28778C16E5@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480808111627g74f458dfw889ddf92862e3db7@mail.gmail.com> First, sorry about the multiple spammed replies! Gmail told me it was failing to send and to come back later. When I came back later and sent the definitive copy of my reply, I found that all my earlier clicks had also succeeded. Oops. On Tue, Aug 12, 2008 at 12:22 AM, Nicolas Bertolotti < Nicolas.Bertolotti@mathworks.fr> wrote: > You can't see it in the sample but the original piece of code on which I > observed the crash is from the "zlib" ? and I thought I would not need to > modify it. > Have you tried removing the -lkernel32? Does that suffice? > You mentioned that you faced a similar issue with GMP. Did you submit a bug > report to the GCC team? > Well, the right people to submit the bug report to are the mingw-w64 people, specifically Kai Tietz. I didn't, because at the time I hacked around the GMP problems gcc was so broken that it hardly seemed a useful report. ;-) Now it's a lot better and maybe he should know about this. For gmp, the problem was alloca (which also allocates on the stack). Fortunately, gmp had a compile-time option to change it's use of alloca to malloc. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080812/e5e70819/attachment.html From Nicolas.Bertolotti at mathworks.fr Mon Aug 11 16:51:29 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Mon Aug 11 16:52:07 2008 Subject: FW: [MLton] Working: MLton/Win64 toolchain In-Reply-To: <162de7480808111627g74f458dfw889ddf92862e3db7@mail.gmail.com> References: <8320D98DA9A5C54C926D397795FE7CEA28778C16B6@EXCHANGE-UK.ad.mathworks.com> <162de7480808111418x1e58f57avb2ec2dfc5096665c@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA28778C16E5@EXCHANGE-UK.ad.mathworks.com> <162de7480808111627g74f458dfw889ddf92862e3db7@mail.gmail.com> Message-ID: <8320D98DA9A5C54C926D397795FE7CEA28778C16E6@EXCHANGE-UK.ad.mathworks.com> Removing -lkernel32 does not solve the issue. But I could workaround the issue anyway following your other recommendation and replacing the local variable with a pointer and a malloc(). I could notice that Kai Tietz had submitted a patch < [patch i386]: x86_64-pc-mingw _alloca and _stkchk may corrupts stack alignment > which seems to be related to the problem you mentionned : http://gcc.gnu.org/ml/gcc-patches/2008-01/msg01350.html It seems that this patch should solve the issue with GMP. As I don't really know how gcc works, I wonder whether it could solve mine as well or if a similar fix should be applied elsewhere. Now I am facing another crash (exit code 128 without any information) in pure SML code so I guess I will come back with additional feedback in a near future. Best regards Nicolas ________________________________ From: Wesley W. Terpstra [mailto:wesley@terpstra.ca] Sent: Tuesday, August 12, 2008 1:27 AM To: Nicolas Bertolotti; mlton@mlton.org Subject: Re: FW: [MLton] Working: MLton/Win64 toolchain First, sorry about the multiple spammed replies! Gmail told me it was failing to send and to come back later. When I came back later and sent the definitive copy of my reply, I found that all my earlier clicks had also succeeded. Oops. On Tue, Aug 12, 2008 at 12:22 AM, Nicolas Bertolotti > wrote: You can't see it in the sample but the original piece of code on which I observed the crash is from the "zlib" ... and I thought I would not need to modify it. Have you tried removing the -lkernel32? Does that suffice? You mentioned that you faced a similar issue with GMP. Did you submit a bug report to the GCC team? Well, the right people to submit the bug report to are the mingw-w64 people, specifically Kai Tietz. I didn't, because at the time I hacked around the GMP problems gcc was so broken that it hardly seemed a useful report. ;-) Now it's a lot better and maybe he should know about this. For gmp, the problem was alloca (which also allocates on the stack). Fortunately, gmp had a compile-time option to change it's use of alloca to malloc. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080812/ff68a25d/attachment.htm From wesley at terpstra.ca Fri Aug 15 12:43:12 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Fri Aug 15 12:43:17 2008 Subject: FW: [MLton] Working: MLton/Win64 toolchain In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA28778C16E6@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA28778C16B6@EXCHANGE-UK.ad.mathworks.com> <162de7480808111418x1e58f57avb2ec2dfc5096665c@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA28778C16E5@EXCHANGE-UK.ad.mathworks.com> <162de7480808111627g74f458dfw889ddf92862e3db7@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA28778C16E6@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480808151243qdb109e7j2c3d31a4816277d3@mail.gmail.com> 2008/8/12 Nicolas Bertolotti > Now I am facing another crash (exit code 128 without any information) in > pure SML code so I guess I will come back with additional feedback in a near > future. > Any word on this crash? I need to see it to fix it. :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080815/85adbbc1/attachment.html From Nicolas.Bertolotti at mathworks.fr Fri Aug 15 14:21:23 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Fri Aug 15 14:21:59 2008 Subject: FW: [MLton] Working: MLton/Win64 toolchain In-Reply-To: <162de7480808151243qdb109e7j2c3d31a4816277d3@mail.gmail.com> References: <8320D98DA9A5C54C926D397795FE7CEA28778C16B6@EXCHANGE-UK.ad.mathworks.com> <162de7480808111418x1e58f57avb2ec2dfc5096665c@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA28778C16E5@EXCHANGE-UK.ad.mathworks.com> <162de7480808111627g74f458dfw889ddf92862e3db7@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA28778C16E6@EXCHANGE-UK.ad.mathworks.com> <162de7480808151243qdb109e7j2c3d31a4816277d3@mail.gmail.com> Message-ID: <8320D98DA9A5C54C926D397795FE7CEA28778C1835@EXCHANGE-UK.ad.mathworks.com> I was actually a problem with GMP that I had forgotten to build with the --enable-alloca=malloc-reentrant option. The remaining issues I am now facing are not related to the Win64 port or even MLton in some cases (scaling issues during refFlatten and deepFlatten ; memory overhead ; call functions in Win64 DLLs) Thanks again for the update. This is already very useful to us. Nicolas ________________________________ From: Wesley W. Terpstra [mailto:wesley@terpstra.ca] Sent: Friday, August 15, 2008 9:43 PM To: Nicolas Bertolotti; mlton@mlton.org Subject: Re: FW: [MLton] Working: MLton/Win64 toolchain 2008/8/12 Nicolas Bertolotti > Now I am facing another crash (exit code 128 without any information) in pure SML code so I guess I will come back with additional feedback in a near future. Any word on this crash? I need to see it to fix it. :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080815/e2ea9773/attachment.htm From fluet at tti-c.org Mon Aug 18 11:22:04 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Mon Aug 18 10:30:02 2008 Subject: [MLton] Re: [MLton-commit] r6687 In-Reply-To: References: Message-ID: On Tue, 5 Aug 2008, Wesley Terpstra wrote: > The MLton pseudoOp global might not always be hidden. This patch adds a new > pseudoOp for the hidden assembler directive. Furthermore, this directive does > not exist on windows and should not be emitted. This commit breaks x86-darwin: "mlton" -target self mllex.mlb /tmp/fileKVIuK1.2.S:127:Unknown pseudo-op: .hidden /tmp/fileKVIuK1.2.S:127:Rest of line ignored. 1st junk character valued 95 (_). /tmp/fileKVIuK1.2.S:559:Unknown pseudo-op: .hidden /tmp/fileKVIuK1.2.S:559:Rest of line ignored. 1st junk character valued 95 (_). /tmp/fileKVIuK1.2.S:667:Unknown pseudo-op: .hidden /tmp/fileKVIuK1.2.S:667:Rest of line ignored. 1st junk character valued 95 (_). ... /tmp/fileKVIuK1.2.S:11619:Unknown pseudo-op: .hidden /tmp/fileKVIuK1.2.S:11619:Rest of line ignored. 1st junk character valued 95 (_). call to system failed with exit status 1: gcc -c -o /tmp/fileamIAwv.o /tmp/fileKVIuK1.2.S make[3]: *** [mllex] Error 1 make[2]: *** [tools] Error 2 make[1]: *** [all-no-docs] Error 2 make: *** [all] Error 2 From fluet at tti-c.org Mon Aug 18 11:31:32 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Mon Aug 18 10:39:27 2008 Subject: [MLton] Re: [MLton-commit] r6687 In-Reply-To: References: Message-ID: On Mon, 18 Aug 2008, Matthew Fluet wrote: > On Tue, 5 Aug 2008, Wesley Terpstra wrote: >> The MLton pseudoOp global might not always be hidden. This patch adds a >> new >> pseudoOp for the hidden assembler directive. Furthermore, this directive >> does >> not exist on windows and should not be emitted. > > This commit breaks x86-darwin: > > "mlton" -target self mllex.mlb > /tmp/fileKVIuK1.2.S:127:Unknown pseudo-op: .hidden > /tmp/fileKVIuK1.2.S:127:Rest of line ignored. 1st junk character valued 95 > (_). > /tmp/fileKVIuK1.2.S:559:Unknown pseudo-op: .hidden > /tmp/fileKVIuK1.2.S:559:Rest of line ignored. 1st junk character valued 95 > (_). > /tmp/fileKVIuK1.2.S:667:Unknown pseudo-op: .hidden > /tmp/fileKVIuK1.2.S:667:Rest of line ignored. 1st junk character valued 95 > (_). > ... > /tmp/fileKVIuK1.2.S:11619:Unknown pseudo-op: .hidden > /tmp/fileKVIuK1.2.S:11619:Rest of line ignored. 1st junk character valued 95 > (_). > call to system failed with exit status 1: > gcc -c -o /tmp/fileamIAwv.o /tmp/fileKVIuK1.2.S > make[3]: *** [mllex] Error 1 > make[2]: *** [tools] Error 2 > make[1]: *** [all-no-docs] Error 2 > make: *** [all] Error 2 *-darwin uses Mach-O object files, not ELF. From fluet at tti-c.org Mon Aug 18 14:21:36 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Mon Aug 18 13:29:34 2008 Subject: [MLton] N-Body benchmark and the refFlatten and the deepFlatten optimizations In-Reply-To: <9e43b9a0807011653p61a76d03p8617be53cf004093@mail.gmail.com> References: <9e43b9a0807011653p61a76d03p8617be53cf004093@mail.gmail.com> Message-ID: On Wed, 2 Jul 2008, Vesa Karvonen wrote: > I'm not sure what exactly causes the optimizations to be missed. I would > assume that refFlatten is trickier to apply and requires a more complex > analysis. The deepFlatten page does not explain the associated analysis > in any way (and I haven't looked at the implementation of deepFlatten). I > find it slightly surprising that the deepFlatten optimization is not > applied in this case. The objects are finite sized and not very large (3 > reals (+ overhead)). Shouldn't this make it safe (for space) to apply > deepFlatten? Of course there are various trade-offs here (between > allocation, copying, sharing), and it is impossible to make optimal > decisions in every case. > > The comments on the refFlatten page led me to think that the problem (not > applying either optimization) might be caused by the code that creates the > system (see the benchmark code) by mapping over a list during which ref > cells are being allocated. So, I manually eliminated the map call. This > obviously changed the results of some analysis, eliminated the allocation > and improved performance considerably from about 32 to about 26 > seconds. If I interpret the generated code correctly, then the > deepFlatten optimization was now applied, but not refFlatten, which > should give even better results by avoiding one more indirection. > > It would seem that improving the analyses of the refFlatten and deepFlatten > optimizations, so that they could be applied in more cases, could have > significant performance benefits. A while back, Vesa and I chatted about this issue, and I thought it would be worth recording the conclusions in the mail archive. For Vesa's code, the interesting bit is how the 'system' value is represented: type body = {pos : Vec3D.t Ref.t, vel : Vec3D.t Ref.t, mass : Real.t} val system : body list = let val f = fn {pos, vel, mass} => {pos = ref pos, vel = ref (vel |* daysPerYear), mass = mass * solarMass} in map f [{pos = {x = 0.0, y = 0.0, z = 0.0}, vel = {x = 0.0, y = 0.0, z = 0.0}, mass = 1.0}, ...] end For the original implementation above, at the end of the SSA2 optimization passes, the 'body list' type is represented by the SSA2 type: list_11 = nil_7 of (unit) | ::_11 of ((real64 ref * real64 ref * real64 ref) * ((real64 * real64 * real64) ref) * real64 * list_11) Thus, a cons cell is represented by an object with 4 fields; the first field is a pointer to a 3-field object (each field of which is a mutable real64); the second field is a pointer to a mutable pointer to a three field object (each field of which is a real64); the third field is a real64; the fourth field is a list_11. Hence, the body record type has been 'inlined' into the specific list type. Also, the deepFlatten pass has 'pushed' the mutability of the vel field into the three components of a Vec3D.t. [MLton initially orders a record type alphabetically by field name and various passes might reverse the order of the entire tuple; so, we know that the field next to the mass field will be the pos field, leaving the other field to be the vel field.] Vesa noted that the line ; #pos a := pos a |+| vel a |* dt ended up allocating a tuple, leading to a high allocation total. Vesa gave an alternate implementation, equivalent to the following: val system : body list = let val f = fn {pos, vel, mass} => {pos = ref pos, vel = ref (vel |* daysPerYear), mass = mass * solarMass} in [f {pos = {x = 0.0, y = 0.0, z = 0.0}, vel = {x = 0.0, y = 0.0, z = 0.0}, mass = 1.0}, ...] end For this implementation, at the end of the SSA2 optimization passes, the 'body list' type is represented by the SSA2 type: list_10 = nil_6 of (unit) | ::_10 of ((real64 ref * real64 ref * real64 ref) * (real64 ref * real64 ref * real64 ref) * real64 * list_10) Thus, a cons cell is represented by an object with 4 fields; the first field is a pointer to a 3-field object (each field of which is a mutable real64); the second field is a pointer to a 3-field object (each field of which is a mutable real64); the third field is a real64; the fourth field is a list_11. Hence, the body record type has been 'inlined' into the specific list type. Also, the deepFlatten pass has 'pushed' the mutability of the vel field into the three components of a Vec3D.t and 'pushed' the mutability of the pos field into the three components of a Vec3D.t. With this implementation, the line ; #pos a := pos a |+| vel a |* dt ends up not allocating anything, simply mutating each of the three components. Both of these SSA2 type representations are established after the deepFlatten pass (but before the refFlatten pass). In both, the refFlatten pass does nothing (with respect to the 'body list' type). This raises a couple of issues, and I don't have a complete understanding of either of them. First, it isn't clear why, in the original implementation, the 'vel' field was deepFlatten-ed but the 'pos' field was not deepFlatten-ed. There doesn't seem to be a significant difference in the way the fields are initialized and used in the program. [That this was an issue didn't occur to me earlier, Vesa and I didn't discuss it, and I don't have anything more to say on it, other than it is an interesting question, and perhaps the best way of achieving the desired allocation behavior.] Second, it wasn't clear why, in the original implementation, the 'pos' field wasn't subject to being refFlatten-ed (given that the 'pos' field wasn't deepFlatten-ed). That is, we might expect the refFlatten pass to take list_11 = nil_7 of (unit) | ::_11 of ((real64 ref * real64 ref * real64 ref) * ((real64 * real64 * real64) ref) * real64 * list_11) to list_11 = nil_7 of (unit) | ::_11 of ((real64 ref * real64 ref * real64 ref) * (real64 * real64 * real64) ref * real64 * list_11) In retrospect, this wouldn't actually change the allocation behavior of the program significantly. It would save one indirection (and one allocation at the allocation of the 'system' value), but it would still require one tuple allocation for each iteration. Nonetheless, it is interesting to note why the reference isn't flattened into the containing data structure. One requirement for the refFlatten optimization is that the 'ref' cell allocation occurs in the same block as the containing object allocation. With regards to the original code above, this means that for each ::_11 allocation, there should be a 'ref' cell allocation. Looking at the SSA2 IL code for the program into the refFlatten pass, we see two ::_11 allocations: L_271 (x_339: list_11, x_338: (real64 * real64 * real64), x_337: (real64 * real64 * real64), x_336: real64, x_335: list_10) x_342: ((real64 * real64 * real64) ref) = (x_337) x_2396: real64 = #2 x_338 x_2397: real64 = #1 x_338 x_2398: real64 = #0 x_338 x_347: real64 = Real64_mul (global_171, x_2398) x_346: real64 = Real64_mul (global_171, x_2397) x_345: real64 = Real64_mul (global_171, x_2396) x_343: (real64 ref * real64 ref * real64 ref) = (x_347, x_346, x_345) x_341: real64 = Real64_mul (x_333, x_336) x_2158: ::_11 of ((real64 ref * real64 ref * real64 ref) * ((real64 * real64 * real64) ref) * real64 * list_11) = ::_11 (x_343, x_342, x_341, x_339) x_340: list_11 = x_2158: list_11 case x_335 of nil_6 => L_273 | ::_3 => L_1326 and L_274 (x_355: list_11, x_354: (real64 ref * real64 ref * real64 ref), x_353: ((real64 * real64 * real64) ref), x_352: real64, x_351: list_11) x_2164: ::_11 of ((real64 ref * real64 ref * real64 ref) * ((real64 * real64 * real64) ref) * real64 * list_11) = ::_11 (x_354, x_353, x_352, x_355) x_356: list_11 = x_2164: list_11 case x_351 of nil_7 => L_276 | ::_11 => L_1327 The first ::_11 allocation is the obvious one, that includes the ref cell allocation (x_342 = (x_337)). The second ::_11 allocation does not include a ref cell allocation, so the 'pos' field is not susceptible to refFlatten-ing. The second ::_11 allocation corresponds to a list reversal that is part of the implementation of map. [A list reversal is used so that map is implemented by two tail-recursive loops, rather than one non-tail-recursive loop.] In general, one can't perform the refFlatten-ing optimization in this case. Indeed, the ref cells are copied from the old list to the new list. Unless we 'know' that the old list is dead, we can't flatten for fear of breaking sharing. (Someone might have a copy of the old list, and we need to see the same references.) We can change the implementation to the following: val system : body list = let val f = fn {pos, vel, mass} => {pos = ref pos, vel = ref (vel |* daysPerYear), mass = mass * solarMass} in foldr (fn (x,l) => (f x)::l) [] [{pos = {x = 0.0, y = 0.0, z = 0.0}, vel = {x = 0.0, y = 0.0, z = 0.0}, mass = 1.0}, ...] end There is still a list reversal as part of the implementation of foldr, but now the input list is reversed. (One can't take this as the implementation of map, because it would apply the mapped function to the elements in the wrong order.) The deepFlatten pass still yields the same list_11 representation for the 'body list' type, but now the SSA2 program into the refFlatten pass has exactly one ::_11 allocation (nearly alpha-equivalent to the L_271 block above). However, the 'pos' field is still not subject to being refFlatten-ed. Another (syntactic) requirement of the refFlatten optimization is that the ref cell updates occur in the same block as the selection of the ref cell from its containing data-structure. This requirement is violated in the SSA2 program: L_389 (x_567: (real64 ref * real64 ref * real64 ref), x_566: ((real64 * real64 * real64) ref), x_565: list_10) x_2456: (real64 * real64 * real64) = #0 x_566 x_2457: real64 = #2 x_567 x_2458: real64 = #1 x_567 x_2459: real64 = #0 x_567 x_576: real64 = Real64_mul (x_2459, global_175) x_575: real64 = Real64_mul (x_2458, global_175) x_573: real64 = Real64_mul (x_2457, global_175) x_2453: real64 = #2 x_2456 x_2454: real64 = #1 x_2456 x_2455: real64 = #0 x_2456 x_571: real64 = Real64_add (x_2455, x_576) x_570: real64 = Real64_add (x_575, x_2454) x_569: real64 = Real64_add (x_573, x_2453) x_568: (real64 * real64 * real64) = (x_571, x_570, x_569) x_566 := x_568 case x_565 of nil_7 => L_391 | ::_11 => L_1357 The 'pos' field reference is x_566, which is passed as a first-class value into the L_389 block, rather than selected from the cons cell within the block. The motivation for this syntactic requirement is to both make it easy to find the containing data structure and to ensure that the flattening of the ref cell into the containing data structure does not violate space safety. Neither of the syntactic requirements are semantic requirements -- there exist programs that violate that the syntactic requirements that could still validly optimized. For example, consider the following common idiom: val r = {a = ref (f x), b = ref (g y)} If the code for g is not inlined (or includes a loop), then the ref cell allocated for the a field will be separated from the allocation of the record, because, in A-normal form, this code looks like: val r = let val fx = f x val ar = ref fx val gy = g y (* point of possible block breaking *) val br = ref gy val r = {a = ar, b = br} in r end One can restore the syntactic requirement in the above code as: val r = let val fx = f x val gy = g y in {a = ref fx, b = ref gy} end The latter requirement is also commonly violated by programs, because, as in the above code, components of a datatype variant are passed as first-class values and, thus, become separated from their selection from the containing data structure. In any case, it seems clear that there are ways of improving the deepFlatten and refFlatten optimizations. Not to mention the fact that there have been other reports of deepFlatten and refFlatten requiring inordinate amounts of memory for large input programs. From wesley at terpstra.ca Tue Aug 19 03:20:17 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Aug 19 03:20:33 2008 Subject: [MLton] Re: r6701 Message-ID: <162de7480808190320g2b97f6b7u598aebaea3e26495@mail.gmail.com> I don't have a i386 with osx, but I gather from this checkin that .hidden doesn't work. Could someone try ".private_extern" instead? If it works, this program: val () = _export "baz": (int -> int) -> unit; (fn x => x +1) when compiled with mlton -default-ann 'allowFFI true' -export-header foo.h -format library foo.sml mlton -default-ann 'allowFFI true' -export-header foo.h -format archive foo.sml should produce libraries with >exactly< these public symbols: nm -g foo.dylib | grep -v 'U ' foo.dylib(single module): 00008520 T _baz 00008868 T _foo_close 0000868c T _foo_open nm -g foo.a | grep -v 'U ' foo.a(foo.a.o): 00007938 T _baz 00007c80 T _foo_close 00007aa4 T _foo_open It should be possible to use both libraries in a C program that only links with '-lfoo -lgmp'. (ie: libmlton.a and libgdtoa.a should be compiled in) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080819/aab23dc0/attachment.html From fluet at tti-c.org Tue Aug 19 10:15:11 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Tue Aug 19 09:23:14 2008 Subject: [MLton] Re: r6701 In-Reply-To: <162de7480808190320g2b97f6b7u598aebaea3e26495@mail.gmail.com> References: <162de7480808190320g2b97f6b7u598aebaea3e26495@mail.gmail.com> Message-ID: On Tue, 19 Aug 2008, Wesley W. Terpstra wrote: > I don't have a i386 with osx, but I gather from this checkin that .hidden > doesn't work. Could someone try ".private_extern" instead? ".private_extern" seems to work for '-format archive'. But, with '-format library', I get: [fluet@fenrir temp]$ ../mlton.git-svn.trunk/build/bin/mlton -default-ann 'allowFFI true' -export-header foo.h -format library -keep g -keep o -verbose 2 foo.sml ... Link starting gcc -o foo.dylib -dynamiclib foo.1.o foo.0.o \ -L/Users/fluet/devel/mlton/mlton.git-svn.trunk/build/lib/self \ -lmlton-pic -lgdtoa-pic -lm -lgmp -L/usr/local/lib -L/opt/local/lib ld: foo.0.o has local relocation entries in non-writable section (__TEXT,__text) /usr/libexec/gcc/i686-apple-darwin8/4.0.1/libtool: internal link edit command failed Link raised in 0.06 + 0.00 (0% GC) MLton raised in 0.23 + 0.00 (0% GC) call to system failed with exit status 1: gcc -o foo.dylib -dynamiclib foo.1.o foo.0.o -L/Users/fluet/devel/mlton/mlton.git-svn.trunk/build/lib/self -lmlton-pic -lgdtoa-pic -lm -lgmp -L/usr/local/lib -L/opt/local/lib From wesley at terpstra.ca Tue Aug 19 09:43:39 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Aug 19 09:43:47 2008 Subject: [MLton] Re: r6701 In-Reply-To: <162de7480808190943k72a3526fp1f7bf386dd62e753@mail.gmail.com> References: <162de7480808190320g2b97f6b7u598aebaea3e26495@mail.gmail.com> <162de7480808190943k72a3526fp1f7bf386dd62e753@mail.gmail.com> Message-ID: <162de7480808190943j48ae8b41t7d0b8ecbb48cec6b@mail.gmail.com> On Tue, Aug 19, 2008 at 7:15 PM, Matthew Fluet wrote: > On Tue, 19 Aug 2008, Wesley W. Terpstra wrote: > >> I don't have a i386 with osx, but I gather from this checkin that .hidden >> doesn't work. Could someone try ".private_extern" instead? >> > > ".private_extern" seems to work for '-format archive'. > But, with '-format library', I get: > > [fluet@fenrir temp]$ ../mlton.git-svn.trunk/build/bin/mlton -default-ann > 'allowFFI true' -export-header foo.h -format library -keep g -keep o > -verbose 2 foo.sml > ... > Link starting > gcc -o foo.dylib -dynamiclib foo.1.o foo.0.o \ > -L/Users/fluet/devel/mlton/mlton.git-svn.trunk/build/lib/self \ > -lmlton-pic -lgdtoa-pic -lm -lgmp -L/usr/local/lib > -L/opt/local/lib > ld: foo.0.o has local relocation entries in non-writable section > (__TEXT,__text) > /usr/libexec/gcc/i686-apple-darwin8/4.0.1/libtool: internal link edit > command failed > Link raised in 0.06 + 0.00 (0% GC) > MLton raised in 0.23 + 0.00 (0% GC) > call to system failed with exit status 1: > gcc -o foo.dylib -dynamiclib foo.1.o foo.0.o > -L/Users/fluet/devel/mlton/mlton.git-svn.trunk/build/lib/self -lmlton-pic > -lgdtoa-pic -lm -lgmp -L/usr/local/lib -L/opt/local/lib That's because the i386 assembly isn't PIC, and osx requires PIC libraries (unlike linux) < http://hackage.haskell.org/trac/ghc/wiki/Commentary/PositionIndependentCode>. Grrrr. A text relocation is where the linker records what locations of the code need to be changed after loading a library. I was hopeful I wouldn't need PIC for i386. :-( -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080819/46ec322c/attachment.htm From fluet at tti-c.org Tue Aug 19 10:42:47 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Tue Aug 19 09:50:48 2008 Subject: [MLton] Re: r6701 In-Reply-To: <162de7480808190943j48ae8b41t7d0b8ecbb48cec6b@mail.gmail.com> References: <162de7480808190320g2b97f6b7u598aebaea3e26495@mail.gmail.com> <162de7480808190943k72a3526fp1f7bf386dd62e753@mail.gmail.com> <162de7480808190943j48ae8b41t7d0b8ecbb48cec6b@mail.gmail.com> Message-ID: On Tue, 19 Aug 2008, Wesley W. Terpstra wrote: > On Tue, Aug 19, 2008 at 7:15 PM, Matthew Fluet wrote: >> On Tue, 19 Aug 2008, Wesley W. Terpstra wrote: >> >>> I don't have a i386 with osx, but I gather from this checkin that .hidden >>> doesn't work. Could someone try ".private_extern" instead? >> >> ".private_extern" seems to work for '-format archive'. >> But, with '-format library', I get: >> >> ld: foo.0.o has local relocation entries in non-writable section (__TEXT,__text) >> /usr/libexec/gcc/i686-apple-darwin8/4.0.1/libtool: internal link edit command failed > > That's because the i386 assembly isn't PIC, and osx requires PIC libraries > (unlike linux) < > http://hackage.haskell.org/trac/ghc/wiki/Commentary/PositionIndependentCode>. > Grrrr. > > A text relocation is where the linker records what locations of the code > need to be changed after loading a library. I was hopeful I wouldn't need > PIC for i386. :-( Well, I just wanted the compiler to build. I don't have any need for MLton generated dynamically linked shared libraries on MacOS X. -Matthew From wesley at terpstra.ca Tue Aug 19 10:22:44 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Aug 19 10:22:53 2008 Subject: [MLton] Re: r6701 In-Reply-To: References: <162de7480808190320g2b97f6b7u598aebaea3e26495@mail.gmail.com> <162de7480808190943k72a3526fp1f7bf386dd62e753@mail.gmail.com> <162de7480808190943j48ae8b41t7d0b8ecbb48cec6b@mail.gmail.com> Message-ID: <162de7480808191022v473ce978vd55df8b476e64618@mail.gmail.com> On Tue, Aug 19, 2008 at 7:42 PM, Matthew Fluet wrote: > ld: foo.0.o has local relocation entries in non-writable section >>> (__TEXT,__text) >>> >> That's because the i386 assembly isn't PIC, and osx requires PIC libraries >> > Well, I just wanted the compiler to build. I don't have any need for MLton > generated dynamically linked shared libraries on MacOS X. Ahh, sorry the '.hidden' broke normal compilation on osx. That's my bad. Thanks for the fix. I'm currently in paper-deadline-before-vacation-pressure-mode, so I haven't gotten to finishing the PIC for amd64 external calls or PIC of any kind for i386. The amd64 stuff seems easy and I'll get to it soon. Like I discussed with you earlier, i386 is quite a bit harder, particularly because %esp is not usable in MLton assembly. Why is %ebp used for the ML stack anyway? Since %esp gets pushed into c_stackP, I'd have thought %esp would be perfectly fine for use as the ML stack too. Would it be safe to push 4 bytes to the ML stack, if %esp were pointing at it? ie, could I do this: call tmp_label tmp_label: pop %eax mov gcState-tmp_label+someOffsetIntoGcState(%eax),%eax to load a value out of the gcState array into %eax? The concern is that I need to be sure that those 4 bytes at %esp are available. I don't really understand how the limit checking of the ML stack works. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080819/c7ccda30/attachment.html From fluet at tti-c.org Tue Aug 19 15:52:38 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Tue Aug 19 15:01:14 2008 Subject: [MLton] RE: card/cross map in heap In-Reply-To: References: <8320D98DA9A5C54C926D397795FE7CEA2875DAC115@EXCHANGE-UK.ad.mathworks.com> <9e43b9a0804300659l406493bhbb0380d7aa2247ee@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA2875EB4D20@EXCHANGE-UK.ad.mathworks.com> <8320D98DA9A5C54C926D397795FE7CEA2875EB4D47@EXCHANGE-UK.ad.mathworks.com> <8320D98DA9A5C54C926D397795FE7CEA2875EB4E67@EXCHANGE-UK.ad.mathworks.com> <8320D98DA9A5C54C926D397795FE7CEA28767C6469@EXCHANGE-UK.ad.mathworks.com> <8320D98DA9A5C54C926D397795FE7CEA28777EAEE3@EXCHANGE-UK.ad.mathworks.com> <8320D98DA9A5C54C926D397795FE7CEA28778304D4@EXCHANGE-UK.ad.mathworks.com> Message-ID: On Thu, 17 Jul 2008, Matthew Fluet wrote: > The ideal solution, especially for a situation like yours, where you are > happy to use lots of memory on a dedicated machine, is to use > @MLton fixed-heap 3.5G -- to grab as large a heap as you can (that > comfortably fits in physical memory) at the begining of the program and never > bother resizing it. As I understand it, resizing is only to 'play nice' with > other processes in the system. > > The problem with fixed-heap, though, is that the runtime starts off trying to > use the Cheney-copy collector (so, it really grabs 1/2 * 3.5G) and it may be > some time before it is forced to use the mark-compact collector, and it is > only at that point that the runtime will try to grab the 3.5G. Since > fixed-heap affects the desiredSize (but not the minSize), you really need to > set fixed-heap to the size that is actually able to be allocated, so that > desiredSize == currentSize, and no resizing occurs. Some thoughts that have occured to me. First, I remarked earlier that we would sometimes like to know "if I unmap this memory, can I mmap this size?" While there is no general way of answering this question, I believe that (for the tight memory situations where it becomes an issue) we have some useful information around. In particular, after paging the heap to disk, if the subsequent createHeap is forced to back off, then we have a reasonable upper bound on the amount of memory we should ever ask for in the future. In particular, because we have paged the heap to disk, we know that we have freed up as much memory as we possible can. If mmap can't satisfy our request in this situation, then we might have gone over the size of a contiguous mapping in our address space. If that is the case, then there is no need to subsequently page the heap to disk and try to allocate a larger heap -- we've already got as large a heap as we can get. (Given that, we may want createHeap to use a finer-grained backoff when used after paging the heap to disk; that would really find the largest sized heap.) Of course, the other reason mmap may fail is that the operating system virtual memory mangager can't currently satisfy our request along with the outstanding requests of other processes. That is, mmap may fail because the MM won't over commit the physical/swap pages. It is possible that a subsequent mmap will succeed, if in the intervening time, other processes have given up memory. That seems to suggest the following policy: - record a mmapMaxHeap statistic. This statistic is updated whenever createHeap is forced to backoff when no extra memory is being used (that is, at the initial createHeap and a createHeap after the heap has been paged to disk). - take mmapMaxHeap into account when resizing the heap. In particular, don't let the desiredSize exceed the mmapMaxHeap; it is better to stick with a heap that is at mmapMaxHeap size than to try paging to disk. There is one exception to this policy. If minSize > mmapMaxHeap, then we should allow the runtime to page the heap to disk and try to mmap the desired memory. This helps handle the case that we saw an mmapMaxHeap because of memory pressure in the system. Technically, it is possible that an inability of mmap to satisfy a minSize request could be a temporary situation, due to memory pressure from other processes. One could always wait and try again in this situation. However, anyone running a program that needs >2.5G memory probably knows better than to run other high-memory processes at the same time and/or provides a decent swap file/partition. So, I suspect that it is fairly safe to assume that mmap failing to satisfy a minSize request corresponds to a hard limit in the virtual address space for a contiguous map, and thus corresponds to a true out-of-memory situation. The second thought is that I wonder if the heap would be easier to predict/control if we used one contiguous heap at all times (that is, even for major copying collections). In particular, it would be nice to have the behavior described above for fixed-heap --- namely, that one could use fixed-heap to grab a large block of memory at the beginning of the program and there would be no subsequent resizing, even if the runtime switched over from major copying collections to major mark-compact collections. I note that the original Samson91 paper (http://mlton.org/References#Sansom91) works with a single contiguous heap. I'm not sure that I understand the advantage of the current implementation, where the secondaryHeap is managed separately, potentially created and released a number of times during the execution of a program. From fluet at tti-c.org Tue Aug 19 17:04:58 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Tue Aug 19 16:12:58 2008 Subject: [MLton] Embedded MLton In-Reply-To: References: Message-ID: On Sun, 20 Jul 2008, Wesley W. Terpstra wrote: > What options can I use to convince MLton to output small executables? > Ideally I'd like to get hello-world to below 100k. With static linking of > all but libc/m it is currently 260k. You can try adjusting the various inlining and code-duplication threshholds. But, that often as much hurts code size, because there is a win in trimming unused code paths after inlining. Also, in a hello-world program, the majority executable will come from the libraries (libmlton and libgmp), not from the individual program. For example, on amd64-linux (with libgmp shared): [fluet@shadow temp]$ cat z.sml val () = print "Hello world!\n" [fluet@shadow temp]$ mlton -keep o z.sml [fluet@shadow temp]$ size z.0.o z.1.o z text data bss dec hex filename 2465 6136 2 8603 219b z.0.o 33411 19 0 33430 8296 z.1.o 122157 7296 10208 139661 2218d z So approx 66% comes from libmlton.a. The garbage collector probably contributes the most from libmlton.a to the executable. > I already tried removing all unused symbols from libgmp.a and libmlton.a. > The benefit was not spectacular, only 40k. libgmp.a already includes each > function in a separate file so static linking drops all the unneeded > methods. Similarly, most of the basis wrapper functions are dropped. Right. You are left with the garbage collector. There are some features that you might be willing to disable for an embedded system (for example, the hash-consing collection is rarely used), but you would need to build a customized libmlton.a for that. From ville at laurikari.net Tue Aug 19 23:14:05 2008 From: ville at laurikari.net (Ville Laurikari) Date: Wed Aug 20 09:22:04 2008 Subject: [MLton] Re: [MLton-commit] r6746 In-Reply-To: References: Message-ID: <20080820061405.GF5449@laurikari.net> Just curious, why should sharing constraints be preferred over where constraints? Is it simply an issue of style/consistency or am I missing something? On Tue, Aug 19, 2008 at 03:13:52PM -0700, Matthew Fluet wrote: > Use sharing constraint rather than where type constraint > ---------------------------------------------------------------------- > > U mlton/trunk/mlton/atoms/con-.sig > > ---------------------------------------------------------------------- > > Modified: mlton/trunk/mlton/atoms/con-.sig > =================================================================== > --- mlton/trunk/mlton/atoms/con-.sig 2008-08-19 22:13:46 UTC (rev 6745) > +++ mlton/trunk/mlton/atoms/con-.sig 2008-08-19 22:13:52 UTC (rev 6746) > @@ -1,4 +1,4 @@ > -(* Copyright (C) 1999-2005 Henry Cejtin, Matthew Fluet, Suresh > +(* Copyright (C) 1999-2005, 2008 Henry Cejtin, Matthew Fluet, Suresh > * Jagannathan, and Stephen Weeks. > * Copyright (C) 1997-2000 NEC Research Institute. > * > @@ -13,7 +13,8 @@ > signature CON = > sig > include ID > - include PRIM_CONS where type con = t > + include PRIM_CONS > + sharing type t = con > > val fromBool: bool -> t > val stats: unit -> Layout.t > > > _______________________________________________ > MLton-commit mailing list > MLton-commit@mlton.org > http://mlton.org/mailman/listinfo/mlton-commit > -- http://www.iki.fi/vl/ From fluet at tti-c.org Wed Aug 20 12:37:23 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Aug 20 11:45:29 2008 Subject: [MLton] Re: [MLton-commit] r6746 In-Reply-To: <20080820061405.GF5449@laurikari.net> References: <20080820061405.GF5449@laurikari.net> Message-ID: In this instance, I don't believe that there is any difference. The MLton sources generally use sharing constraints rather than where constraints, so it was mostly a style issue. I think I was motivated to look for uses of sharing/where in the sources after either: http://mlton.org/pipermail/mlton-user/2008-February/001354.html or http://mlton.org/pipermail/mlton/2008-February/030188.html I tend to find the symmetry in sharing a little more pleasing, along with the fact that you can simultaneously share a number of types. On Wed, 20 Aug 2008, Ville Laurikari wrote: > Just curious, why should sharing constraints be preferred over where > constraints? Is it simply an issue of style/consistency or am I > missing something? > > On Tue, Aug 19, 2008 at 03:13:52PM -0700, Matthew Fluet wrote: >> Use sharing constraint rather than where type constraint >> ---------------------------------------------------------------------- >> >> U mlton/trunk/mlton/atoms/con-.sig >> >> ---------------------------------------------------------------------- >> >> Modified: mlton/trunk/mlton/atoms/con-.sig >> =================================================================== >> --- mlton/trunk/mlton/atoms/con-.sig 2008-08-19 22:13:46 UTC (rev 6745) >> +++ mlton/trunk/mlton/atoms/con-.sig 2008-08-19 22:13:52 UTC (rev 6746) >> @@ -1,4 +1,4 @@ >> -(* Copyright (C) 1999-2005 Henry Cejtin, Matthew Fluet, Suresh >> +(* Copyright (C) 1999-2005, 2008 Henry Cejtin, Matthew Fluet, Suresh >> * Jagannathan, and Stephen Weeks. >> * Copyright (C) 1997-2000 NEC Research Institute. >> * >> @@ -13,7 +13,8 @@ >> signature CON = >> sig >> include ID >> - include PRIM_CONS where type con = t >> + include PRIM_CONS >> + sharing type t = con >> >> val fromBool: bool -> t >> val stats: unit -> Layout.t >> >> >> _______________________________________________ >> MLton-commit mailing list >> MLton-commit@mlton.org >> http://mlton.org/mailman/listinfo/mlton-commit >> > > -- > http://www.iki.fi/vl/ > > _______________________________________________ > MLton mailing list > MLton@mlton.org > http://mlton.org/mailman/listinfo/mlton > From fluet at tti-c.org Wed Aug 20 13:15:47 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Aug 20 12:23:52 2008 Subject: [MLton] Generating C libraries from MLton (patch) In-Reply-To: References: Message-ID: On Thu, 24 Jul 2008, Wesley W. Terpstra wrote: > Since the previous attempt to build C libraries with MLton seems to have > fizzled out, I have taken it over. :-) I strongly suggest that you add a 'doc/examples/ffi-shared' directory and populate it with some sample code (and a Makefile) that demonstrates how to build and use libraries and shared libraries. It would go a long way towards making testing on different platforms easier. The overall architecture seems good, and I know that a lot of commits have already been made. (In retrospect, perhaps this would have been better undertaken on a branch, but it hasn't seemed too disruptive to normal executable compiles.) I would also suggest that commit messages comment directly on the commit in question; some of them seem to dump a lot of information that isn't relevant to the particular commit. The information is good -- just send it directly to the mailing list or put it on the wiki. Another thing that would be helpful would be to document some things on the wiki. Something I would be interested in seeing is pointers/links to other documentation that informs the design/implementation. Particularly with regards to the changes in the codegen, it would be comforting to see an ABI document, and not simply working from the output of 'gcc -S'. From fluet at tti-c.org Wed Aug 20 14:17:01 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Aug 20 13:25:08 2008 Subject: [MLton] Working: MLton/Win64 toolchain In-Reply-To: References: Message-ID: On Fri, 8 Aug 2008, Wesley W. Terpstra wrote: > I'm happy to announce that MLton now works perfectly on 64-bit windows using > the native codegen. Please give it a spin and report any problems. It has > only been tested on Win2003/64 and I would be very interested in hearing > reports about XP64 and Vista64. > > A complete toolchain (no additional software required) can be found at: > http://www.dvs.tu-darmstadt.de/staff/terpstra/mlton-20080808.win64.zip If > someone could move this to the Experimental page on the wiki, I would > appreciate it (I don't have write permissions there). Be warned that the > toolchain is a 180MB download. Posted to the Experimental page. From fluet at tti-c.org Wed Aug 20 17:49:05 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Aug 20 16:57:10 2008 Subject: [MLton] Re: PIC in amd64 assembly In-Reply-To: References: Message-ID: On Mon, 28 Jul 2008, Wesley W. Terpstra wrote: > On a side note, why is it that in the code toAddressMemLoc > (amd64-allocate-registers.fun), when constructing disp, the base and the > index are simply added together if both are immediate? Shouldn't the index > be scaled? An immediate index is scaled at the construction of the MemLoc.t; see, for example, amd64.MemLoc.imm in amd64.fun. From fluet at tti-c.org Wed Aug 20 18:27:36 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Aug 20 17:35:41 2008 Subject: [MLton] Re: PIC in amd64 assembly In-Reply-To: References: Message-ID: On Wed, 6 Aug 2008, Wesley W. Terpstra wrote: > As an aside to Matthew, that page indicates what I suspected about osx. You > don't need the indirect symbols except for external symbols. It seems that you need the indirection for external symbols (that come from shared libraries?), even when generating an executable. For example, changing call _L_printf_stub_0 to call _printf in the output assembly for val p = _import "printf": string * int -> int; val _ = p ("%i\n\000", 10) yields [fluet@fenrir temp]$ mlton z.1.c z.0.S /usr/libexec/gcc/i686-apple-darwin8/4.0.1/ld: /tmp/fileoqR3a0.o has external relocation entries in non-writable section (__TEXT,__text) for symbols: _printf But, for example, changing call _L_Posix_IO_writeChar8Arr_stub_0 to call _Posix_IO_writeChar8Arr links without problems. Thus, it does seem to suggest that we could condition the use of indirect symbols based on the CFunction.SymbolScope.t of the called function. See mkCCallLabel and mkSymbolStubs in x86-generate-transfers.fun. It also reveals _symbol and _address are broken for access to external symbols; for example, consider: val ea = _address "cosf": MLton.Pointer.t; val w = MLton.Pointer.diff (ea, MLton.Pointer.null) val () = (print o concat) ["w = ", Word.toString w, "\n"] On x86-darwin: [fluet@fenrir temp]$ mlton -keep g -default-ann 'allowFFI true' z.sml /usr/libexec/gcc/i686-apple-darwin8/4.0.1/ld: /tmp/fileY93JXr.o has external relocation entries in non-writable section (__TEXT,__text) for symbols: _cosf From wesley at terpstra.ca Wed Aug 20 19:29:21 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Aug 20 19:29:26 2008 Subject: [MLton] Re: PIC in amd64 assembly In-Reply-To: References: Message-ID: <162de7480808201929k6127987ar138b8e8d4754653e@mail.gmail.com> On Thu, Aug 21, 2008 at 3:27 AM, Matthew Fluet wrote: > On Wed, 6 Aug 2008, Wesley W. Terpstra wrote: > >> As an aside to Matthew, that page indicates what I suspected about osx. >> You >> don't need the indirect symbols except for external symbols. >> > > It seems that you need the indirection for external symbols (that come from > shared libraries?), even when generating an executable. For example, > changing > call _L_printf_stub_0 > to > call _printf > in the output assembly for > val p = _import "printf": string * int -> int; > val _ = p ("%i\n\000", 10) > yields > > [fluet@fenrir temp]$ mlton z.1.c z.0.S > /usr/libexec/gcc/i686-apple-darwin8/4.0.1/ld: /tmp/fileoqR3a0.o has > external relocation entries in non-writable section (__TEXT,__text) for > symbols: > _printf > > But, for example, changing > call _L_Posix_IO_writeChar8Arr_stub_0 > to > call _Posix_IO_writeChar8Arr > links without problems. Yep. That's what I was expecting. I believe that i386-osx is the same as windows in this respect. Private/Public both don't need indirect access, but external always does (even for executables). The ghc page indicates that x86_64-darwin is significantly different, more like amd64-linux. I haven't found an official document for either of the darwin platforms, though, so this could be wrong. It also reveals _symbol and _address are broken for access to external > symbols; for example, consider: > val ea = _address "cosf": MLton.Pointer.t; > val w = MLton.Pointer.diff (ea, MLton.Pointer.null) > val () = (print o concat) ["w = ", Word.toString w, "\n"] > > On x86-darwin: > [fluet@fenrir temp]$ mlton -keep g -default-ann 'allowFFI true' z.sml > /usr/libexec/gcc/i686-apple-darwin8/4.0.1/ld: /tmp/fileY93JXr.o has > external relocation entries in non-writable section (__TEXT,__text) for > symbols: > _cosf > Yes. I haven't touched the symbol use in the i386 codegen yet. I expect it also has problems with _address on windows. The changes I made don't just fix things for libraries. On the windows platform (and it seems darwin too) you need more information than the old FFI supplied, even for executables. The thing that I'm still hung up on for i386, though, is how to get the PC. The sysv abi docs for ia32 say to use 'call', but I can't. So again my question: why %ebp instead of %esp? I can't find any x86 instruction other than 'call' to get me the PC, so this is a real problem. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080821/7a20ff67/attachment.htm From fluet at tti-c.org Wed Aug 20 20:55:07 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Aug 20 20:03:11 2008 Subject: [MLton] Re: r6701 In-Reply-To: <162de7480808191022v473ce978vd55df8b476e64618@mail.gmail.com> References: <162de7480808190320g2b97f6b7u598aebaea3e26495@mail.gmail.com> <162de7480808190943k72a3526fp1f7bf386dd62e753@mail.gmail.com> <162de7480808190943j48ae8b41t7d0b8ecbb48cec6b@mail.gmail.com> <162de7480808191022v473ce978vd55df8b476e64618@mail.gmail.com> Message-ID: On Tue, 19 Aug 2008, Wesley W. Terpstra wrote: > I'm currently in paper-deadline-before-vacation-pressure-mode, so I haven't > gotten to finishing the PIC for amd64 external calls or PIC of any kind for > i386. The amd64 stuff seems easy and I'll get to it soon. Like I discussed > with you earlier, i386 is quite a bit harder, particularly because %esp is > not usable in MLton assembly. For Cygwin and MinGW, which don't support sigaltstack, we can't use %esp other than to point to the top of the C stack. Look for uses of reserveEsp in x86-codegen/*. > Why is %ebp used for the ML stack anyway? Since %esp gets pushed into > c_stackP, I'd have thought %esp would be perfectly fine for use as the ML > stack too. See http://mlton.org/pipermail/mlton/2002-May/021982.html for some comments about using %esp. > Would it be safe to push 4 bytes to the ML stack, if %esp were > pointing at it? No. One issue is that the ML stack grows from low addresses to high addresses, so the push and pop instructions don't move it in the right direction. > ie, could I do this: > call tmp_label > tmp_label: pop %eax > mov gcState-tmp_label+someOffsetIntoGcState(%eax),%eax > to load a value out of the gcState array into %eax? The concern is that I > need to be sure that those 4 bytes at %esp are available. I don't really > understand how the limit checking of the ML stack works. I'm not sure exactly what the above code is meant to accomplish. From fluet at tti-c.org Wed Aug 20 21:07:23 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Aug 20 20:15:27 2008 Subject: [MLton] Re: PIC in amd64 assembly In-Reply-To: <162de7480808201929k6127987ar138b8e8d4754653e@mail.gmail.com> References: <162de7480808201929k6127987ar138b8e8d4754653e@mail.gmail.com> Message-ID: On Thu, 21 Aug 2008, Wesley W. Terpstra wrote: > The thing that I'm still hung up on for i386, though, is how to get the PC. > The sysv abi docs for ia32 say to use 'call', but I can't. So again my > question: why %ebp instead of %esp? I can't find any x86 instruction other > than 'call' to get me the PC, so this is a real problem. Well, a normal C call will necessarily use %esp pointing to the C stack, inorder to match the calling convention. If one only needs this trickery to for 'external' symbols and functions, then perhaps the 'internal' C function stub approach would be the easiest. From gbuday at gmail.com Thu Aug 21 08:29:53 2008 From: gbuday at gmail.com (Gergely Buday) Date: Thu Aug 21 08:29:57 2008 Subject: [MLton] unicode branch Message-ID: <90d975d30808210829k78688d83l8642d018608b5ef2@mail.gmail.com> Dear Wesley, what is the state of the art of your unicode branch of mlton? As I see the latest revision is two years old, does it have some basic functionality working? Best Wishes - Gergely From wesley at terpstra.ca Thu Aug 21 17:50:27 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Thu Aug 21 17:50:41 2008 Subject: [MLton] Re: PIC in amd64 assembly In-Reply-To: References: <162de7480808201929k6127987ar138b8e8d4754653e@mail.gmail.com> Message-ID: <162de7480808211750h335723c6nd724a6afd50176fb@mail.gmail.com> On Thu, Aug 21, 2008 at 6:07 AM, Matthew Fluet wrote: > On Thu, 21 Aug 2008, Wesley W. Terpstra wrote: > >> The thing that I'm still hung up on for i386, though, is how to get the >> PC. >> The sysv abi docs for ia32 say to use 'call', but I can't. So again my >> question: why %ebp instead of %esp? I can't find any x86 instruction other >> than 'call' to get me the PC, so this is a real problem. >> > > Well, a normal C call will necessarily use %esp pointing to the C stack, > inorder to match the calling convention. You've also mentioned that the windows port leaves %esp unchanged, because there is no sigaltstack. > If one only needs this trickery to for 'external' symbols and functions, > then perhaps the 'internal' C function stub approach would be the easiest. Unfortunately, this is not the case. You need to use these tricks to access even private symbols... like gcState! This is why I can't just stub it for i386. On amd64 I could get away with %rip relative addressing, but not for i386. Also, windows does not need PIC. So the one case where %esp is usefully available is the one case where I don't need it. So here's the problem: when you have position independent code, your segments get loaded at some arbitrary offset. The differences between your symbols addresses stay the same, but the absolute addresses have changed. That's why %rip relative addressing for data works so well: the program counter is always the same distance away from a local symbol. The solution on i386 is to fill some register, say %ebx with the address of a local symbol and then compute the address of other local symbols relative to this. The local symbol that ELF generally uses for this purpose is the 'global offset table' (whose contents I'll mention in a second). The linker has special syntax for referring to symbols relative to the got: movl %eax,gcState@GOTOFF+0x4(%ebx) This instruction would store %eax into the gcState plus 4 bytes. For this trick to work, %ebx has to point to the global offset table. To bootstrap the process, you need to find out where your segments got loaded to. This is the reason you need the 'call' instruction: it pushes the %eip to the stack. %eip tells you where your text segment is and so you can use a displacement to find a local symbol, eg; call label label: pop %ebx // now %ebx is pointing to the address of label movl %eax, gcState-label+0x4(%ebx) This code would again write eax to the gcState+4. Anyway, since MLton does a lot of symbol accessing, and since 'call' is expensive to get to work (we'd have to save %esp, restore it to the C stack, call, pop, restore %esp, ...), I think the only real option is to load the offset of the GOT into %ebx at MLton_jumpToSML time (while I can still easily use %esp) and make %ebx unavailable for use as a general purpose register. Yes this sucks for performance, but PIC code on i386 is always slower. The choice of %ebx is fairly important, because calls to external functions happen via the procedure lookup table (PLT) as follows: call printf@PLT Whenever you call into the PLT, %ebx must point to the GOT or the dynamic linker stub code will bloom into a beautiful segfault. If you want the address of external symbols, then you actually need to read them from the GOT. The syntax foo@GOTOFF means the difference (foo-got), which is how we can access local symbols. The syntax foo@GOT means the offset into the GOT where the address of foo is stored. eg: movl foo@GOT(%ebx), %eax will load the address of external symbol foo into %eax. The address is filled into the GOT table when the linker loads the shared library. So basically, all I need to know to get this whole mess to work on ELF is how to tell the x86-codegen that it can't touch %ebx when the output format is a library and the platform is ELF or darwin. As I mentioned, windows won't have this problem. I am anticipating some trouble with darwin above and beyond what I've described here. I don't know yet how to solve it and really need to read an ABI document for PIC on darwin. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080822/57c098b1/attachment.html From wesley at terpstra.ca Thu Aug 21 17:55:05 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Thu Aug 21 17:55:09 2008 Subject: [MLton] Re: unicode branch In-Reply-To: <90d975d30808210829k78688d83l8642d018608b5ef2@mail.gmail.com> References: <90d975d30808210829k78688d83l8642d018608b5ef2@mail.gmail.com> Message-ID: <162de7480808211755g2dba14e3gd39f3914ad976c6b@mail.gmail.com> On Thu, Aug 21, 2008 at 5:29 PM, Gergely Buday wrote: > what is the state of the art of your unicode branch of mlton? As I see > the latest revision is two years old, does it have some basic > functionality working? At this point, I'd say it's dead. I didn't have a pressing need for it and so the motivation to bring it to a conclusion wasn't there. :-/ It wasn't entirely clear how Unicode in SML should look. The current MLton/head has WideChar and WideString in it. If for some reason I got motivated to wade through all the relevant specifications again, I would probably start from scratch. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080822/fea9096b/attachment.htm From wesley at terpstra.ca Sun Aug 24 07:52:18 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sun Aug 24 07:52:23 2008 Subject: [MLton] MLTONVERSION for non-release builds Message-ID: <162de7480808240752s47c0fc7bt1ec7236d6f9f876@mail.gmail.com> I have many different MLton toolchains on different platforms and machines. It is often confusing to me which build has which patches applied, especially when I come back to a computer after some time away. Release versions of MLton include a release date as their version information. Builds from svn/HEAD though just report MLTONVERSION. Since I imagine most of the MLton developers work exclusively with svn builds and not release builds, I think it would be helpful if we included the svn revision number the compiler was built from. eg: $ mlton MLton r6673 (built Sun Aug 24 05:10:14 2008 on carrot) I know that this information comes from control-flags.sml. There is a subversion tag "$Rev$" which gets replaced with the file version, but what we really want is the revision of the tree's root. Does anyone know the relevant subverison magic? Also, will this break whatever script creates MLton releases? (ie: does it look for MLTONVERSION) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080824/e3285255/attachment.html From wesley at terpstra.ca Sun Aug 24 09:53:02 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sun Aug 24 09:53:05 2008 Subject: [MLton] Logo / Icon? Message-ID: <162de7480808240953hcab4446x93ff17d4b81ebd45@mail.gmail.com> Is there a MLton icon or logo? I don't think I've ever seen one. I was wondering where I could find one to use in a windows installer / desktop icon. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080824/33241f21/attachment.htm From fluet at tti-c.org Sun Aug 24 20:44:23 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Sun Aug 24 19:52:42 2008 Subject: [MLton] MLTONVERSION for non-release builds In-Reply-To: <162de7480808240752s47c0fc7bt1ec7236d6f9f876@mail.gmail.com> References: <162de7480808240752s47c0fc7bt1ec7236d6f9f876@mail.gmail.com> Message-ID: On Sun, 24 Aug 2008, Wesley W. Terpstra wrote: > I have many different MLton toolchains on different platforms and machines. > It is often confusing to me which build has which patches applied, > especially when I come back to a computer after some time away. Release > versions of MLton include a release date as their version information. > Builds from svn/HEAD though just report MLTONVERSION. Since I imagine most > of the MLton developers work exclusively with svn builds and not release > builds, I think it would be helpful if we included the svn revision number > the compiler was built from. eg: > $ mlton > MLton r6673 (built Sun Aug 24 05:10:14 2008 on carrot) For a while, my workflow has simply been to copy the svn working directory (e.g., mlton.svn.cardmap-in-heap) and to never have anything but a (recent) svn trunk mlton on my path. The price is that I need to give explicit paths to my work-in-progress builds. > I know that this information comes from control-flags.sml. There is a > subversion tag "$Rev$" which gets replaced with the file version, but what > we really want is the revision of the tree's root. Does anyone know the > relevant subverison magic? Also, will this break whatever script creates > MLton releases? (ie: does it look for MLTONVERSION) There is a 'svnversion' command that can be used to produce a compact 'version number' for a working copy. That ought to be useful, as it indicates the range of versions in the working copy and whether there are any uncommitted changes. There is a sed replacement of MLTONVERSION in the root Makefile. Making a source release just uses 'make version; make clean-svn' and a 'tar'. From fluet at tti-c.org Sun Aug 24 21:08:00 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Sun Aug 24 20:16:20 2008 Subject: [MLton] Re: PIC in amd64 assembly In-Reply-To: <162de7480808211750h335723c6nd724a6afd50176fb@mail.gmail.com> References: <162de7480808201929k6127987ar138b8e8d4754653e@mail.gmail.com> <162de7480808211750h335723c6nd724a6afd50176fb@mail.gmail.com> Message-ID: On Fri, 22 Aug 2008, Wesley W. Terpstra wrote: > Anyway, since MLton does a lot of symbol accessing, and since 'call' is > expensive to get to work (we'd have to save %esp, restore it to the C stack, > call, pop, restore %esp, ...), I think the only real option is to load the > offset of the GOT into %ebx at MLton_jumpToSML time (while I can still > easily use %esp) and make %ebx unavailable for use as a general purpose > register. Yes this sucks for performance, but PIC code on i386 is always > slower. Seems reasonable. Since the MLton_jumpToSML code is generated assembly, you can do the 'pop %ebx' without another 'call'. > So basically, all I need to know to get this whole mess to work on ELF is > how to tell the x86-codegen that it can't touch %ebx when the output format > is a library and the platform is ELF or darwin. As I mentioned, windows > won't have this problem. Mimic the uses of 'reserveEsp' in x86-generate-transfers.fun. You'll want to remove %ebx from the list of available registers for intra-procedural (inter-block) transfers (the 'transferRegs' value). You'll also want to assume that %ebx is reserved at the beginning of each block (the 'blockAssumes' function). From wesley at terpstra.ca Sun Aug 24 20:40:55 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sun Aug 24 20:41:29 2008 Subject: [MLton] Re: PIC in amd64 assembly In-Reply-To: References: <162de7480808201929k6127987ar138b8e8d4754653e@mail.gmail.com> <162de7480808211750h335723c6nd724a6afd50176fb@mail.gmail.com> Message-ID: <162de7480808242040y3cc91e60m8d9c8d4f7c8ce226@mail.gmail.com> On Mon, Aug 25, 2008 at 6:08 AM, Matthew Fluet wrote: > On Fri, 22 Aug 2008, Wesley W. Terpstra wrote: > >> Anyway, since MLton does a lot of symbol accessing, and since 'call' is >> expensive to get to work (we'd have to save %esp, restore it to the C >> stack, >> call, pop, restore %esp, ...), I think the only real option is to load the >> offset of the GOT into %ebx at MLton_jumpToSML time (while I can still >> easily use %esp) and make %ebx unavailable for use as a general purpose >> register. Yes this sucks for performance, but PIC code on i386 is always >> slower. >> > > Seems reasonable. Since the MLton_jumpToSML code is generated assembly, you > can do the 'pop %ebx' without another 'call'. The reason I used a 'call' as opposed to 'pop' is because _GLOBAL_OFFSET_TABLE_ is calculated relative to the instruction where it's used. If I did the pop then addl %ebx would need some symbol arithmetic which doesn't fit nicely into the Immediate abstraction. I figured the extra one instruction was a small price to pay. :-) Mimic the uses of 'reserveEsp' in x86-generate-transfers.fun. You'll want to > remove %ebx from the list of available registers for intra-procedural > (inter-block) transfers (the 'transferRegs' value). You'll also want to > assume that %ebx is reserved at the beginning of each block (the > 'blockAssumes' function). > Your reply comes too late. Do an 'svn update'. ;-) But thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080825/5cd6ef8b/attachment.html From wesley at terpstra.ca Sun Aug 24 20:48:46 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sun Aug 24 20:48:50 2008 Subject: [MLton] MLton/MinGW installer on Experimental Message-ID: <162de7480808242048v5d90b3b5h3ad0a776458307ba@mail.gmail.com> I've added some scripts to package/mlton that build an MSI installer for MLton. You'll find this installer on . It's 40MB and bundles MinGW with it. Not much to say really, except that it works and is rather cool to see in action. =) Should hopefully make installing MLton on windows completely painless. That installer doesn't make any start menu entries or desktop icons, though this is easy. I just didn't know what we would want put there, so did nothing for now. Probably some sort of links to documentation or the website should go in the start menu. I'm not sure what a 'MLton' icon should do, except maybe just starting up msys.bat. Speaking of which, I need a MLton icon. ;-) Enjoy. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080825/4d91e7a2/attachment.htm From ville at laurikari.net Sun Aug 24 22:55:29 2008 From: ville at laurikari.net (Ville Laurikari) Date: Sun Aug 24 22:55:35 2008 Subject: [MLton] MLton/MinGW installer on Experimental In-Reply-To: <162de7480808242048v5d90b3b5h3ad0a776458307ba@mail.gmail.com> References: <162de7480808242048v5d90b3b5h3ad0a776458307ba@mail.gmail.com> Message-ID: <20080825055529.GC7457@laurikari.net> On Mon, Aug 25, 2008 at 05:48:46AM +0200, Wesley W. Terpstra wrote: > I've added some scripts to package/mlton that build an MSI installer for > MLton. You'll find this installer on . It's > 40MB and bundles MinGW with it. Not much to say really, except that it works Wow, this is really excellent. It goes a long way into making MLton more accessible to Windows users. I installed it and it works fine. I'd like to make a couple of suggestions to make it even better. It would be good to include a .bat or program to run MLton from outside the msys environment. It would set up the path and whatnot, and then run the mlton (script) with bash. This way people who don't want to get into the shell could just run MLton from a makefile or IDE, without caring about MinGW and bash. The installer could give an option to add this mlton.bat or program in the system PATH, so it could be run directly from anywhere without extra hassle. The machine I installed this on didn't have a MinGW/msys previously. I assume it won't interfere with a separately installed MinGW and msys? > That installer doesn't make any start menu entries or desktop icons, though > this is easy. I just didn't know what we would want put there, so did > nothing for now. Probably some sort of links to documentation or the website > should go in the start menu. Typical start menu items are some subset of these: * Link to start the program * A "README" or "About" document link * Website link * Links to included documents * Uninstall link > I'm not sure what a 'MLton' icon should do, > except maybe just starting up msys.bat. Speaking of which, I need a MLton > icon. ;-) Because MLton isn't really useful as a program that you just run, I think a desktop icon is not needed at all. Perhaps a start menu link to start the bundled msys shell would be a good idea, though. -- http://www.iki.fi/vl/ From wesley at terpstra.ca Mon Aug 25 15:45:25 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Aug 25 15:45:29 2008 Subject: [MLton] MLton/MinGW installer on Experimental In-Reply-To: <20080825055529.GC7457@laurikari.net> References: <162de7480808242048v5d90b3b5h3ad0a776458307ba@mail.gmail.com> <20080825055529.GC7457@laurikari.net> Message-ID: <162de7480808251545y4612f7f3mfa0c184c45011bdf@mail.gmail.com> On Mon, Aug 25, 2008 at 7:55 AM, Ville Laurikari wrote: > It would be good to include a .bat or program to run MLton from > outside the msys environment. It would set up the path and whatnot, > and then run the mlton (script) with bash. This way people who don't > want to get into the shell could just run MLton from a makefile or > IDE, without caring about MinGW and bash. > > The installer could give an option to add this mlton.bat or program in > the system PATH, so it could be run directly from anywhere without > extra hassle. I've done both. The new installer (r6811) has a mlton.bat stuck in the path (along with gcc, ld, make, etc). Typical start menu items are some subset of these: > * Link to start the program > * A "README" or "About" document link > * Website link > * Links to included documents > * Uninstall link I did all of these except the Uninstall link, since I read somewhere that this is forbidden by the "windows logo", whatever that means. At any rate, apparently it's not something nice programs do. Because MLton isn't really useful as a program that you just run, I > think a desktop icon is not needed at all. Perhaps a start menu link > to start the bundled msys shell would be a good idea, though. The MSYS shell has a link from the Start Menu. I also associated MLton to the extensions .sml and .mlb. They show up with a ... very artistic ... logo and double-clicking on them will automatically launch MLton to compile them. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20080826/9bd4a79a/attachment.html From fluet at tti-c.org Sat Aug 30 16:08:24 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Sat Aug 30 15:17:03 2008 Subject: [MLton] Re: i386-darwin external _symbols In-Reply-To: <162de7480808231800h766edb57q13c4d9901436bc6b@mail.gmail.com> References: <162de7480808231800h766edb57q13c4d9901436bc6b@mail.gmail.com> Message-ID: On Sun, 24 Aug 2008, Wesley W. Terpstra wrote: > Hey. I've been working on a patch to generate symbol stubs for darwin-i386. > I can't test it, as I don't have an i386 mac. Also, I'm not sure how to get > the assembly I generate to end up in the output. The function > makeDarwinNonLazySymbolPointers needs to be called somewhere to output the > symbols at the bottom of each assembler file. > > Once the patch works, the following should compile (as an executable): > val f = _import "cos" external: real -> real; > val g = _import * : MLton.Pointer.t -> real -> real; > val h = _address "sin" external: MLton.Pointer.t; > val () = print (Real.toString (f 4.0) ^ "\n") > val () = print (Real.toString (g h 4.0) ^ "\n") > As far as I understand it, this will not work currently. That's what my > patch is attempting to fix. > > If you could take a whack on the attached patch when you have some spare > time, I'd appreciate it. I've committed your patch plus a few more that gets external functions and symbols working on x86-darwin. The above code works with the native codegen, but not with the C codegen --- due to the fact that 'sin' is declared as a function in and then declared as an untyped extern (defaulting to int; would it be better to declare _address symbols as 'extern void'?) in the generated C code. A somewhat better example than the above might be: val lgamma = _import "lgamma" external: Real64.real -> Real64.real; val signgamAddr = _address "signgam" external: MLton.Pointer.t; val signgam = fn () => MLton.Pointer.getInt32 (signgamAddr, 0) val () = (print o concat) ["lgamma(10.0) = ", Real64.toString (lgamma 10.0), "; ", "signgam = ", Int32.toString (signgam ()), "\n"] which works with the native and C codegens. From fluet at tti-c.org Sun Aug 31 20:10:30 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Sun Aug 31 19:19:13 2008 Subject: [MLton] symbol visibility Message-ID: Commit r6680 adds and commit r6785 modifies the CFunction.SymbolScope.t datatype. When emitting a c-prototype, the symbol scope is translated thus: val symbolScope = case symbolScope of SymbolScope.External => "IMPORTED " | SymbolScope.Private => "INTERNAL " | SymbolScope.Public => "EXPORTED " which is then expanded by cpp via the "export.h" header. Is there a good reason to have a different names (i.e., External for SymbolScope.t, but "IMPORTED" for the macro)?