From wesley at terpstra.ca Fri Jul 3 06:53:17 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Fri Jul 3 06:53:51 2009 Subject: [MLton] test-{create,spawn} fail(?) on MinGW Message-ID: <162de7480907030653y7c286fbbq85bc234f55e31f09@mail.gmail.com> I was running a regression of HEAD when I noticed: testing test-create > 6c6 > < exit_status: W_EXITSTATUS 7 > --- > > exit_status: W_SIGNALED 7 > difference with -type-check true > testing test-spawn > 6c6 > < exit_status: W_EXITSTATUS 7 > --- > > exit_status: W_SIGNALED 7 > 8c8,9 > < exit_status: W_SIGNALED 9 > --- > > unhandled exception: SysErr: Unknown error: kill [nosys] > > Nonzero exit status. > difference with -type-check true The program test-spawn.exe went into an infinite loop and ate 100% CPU. I'm looking into fixing this, so please don't make a release yet! :) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090703/a0149e36/attachment.html From wesley at terpstra.ca Fri Jul 3 15:55:15 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Fri Jul 3 15:55:49 2009 Subject: [MLton] Windows 7 x64 vs MLton r7999 Message-ID: <162de7480907031555u7136bbfcn578a82bb2eb267f4@mail.gmail.com> Since there's a MLton release coming up, I thought it would be a good idea to check out how/if MLton works on Windows 7 (RC1) which is due out in a few months. The good news is that it mostly works (!). Some caveats: * A self-compile of MLton trips over the use of "mlton -stop f" in mlton/Makefile to determine dependencies. The problem is that this (correctly) includes paths like c:/program files (x86)/mlton/lib/mlton/sml/basis/build/sources.mlb. GNU make cannot handle paths with spaces in their names. I imagine this is also a problem with Windows XP, but I have a German WinXP so it is called "C:/Programme/MLton/..." which doesn't trigger this problem. One solution might be to make a -stop f that doesn't include anything pulled from $(SML_LIB). This corresponds to the gcc "-MM" option which is actually what we want in the context where -stop f is used in the Makefile. Another option would be to simply remove the -stop f from mlton/Makefile (I did this to get the build done, and it worked after removing the offending line). Suggestions? * msys uses an rxvt.exe which does not work under Windows7. Using msys -norxvt fixed this. I'll either modify the msys.bat to disable rxvt.exe or find a fixed rxvt.exe. * Windows7 complains vehemently about installing MLton-7999.msi because it's from an "unknown publisher". I need to learn how to sign MSI files; this might require acquiring an official MLton signing certificate. * Windows7 appears to treat some file extensions as dangerous. If you try to create a file with the extension ".exe" or ".bat" in the install location of MLton, you will get permission denied errors. This doesn't seem to be a problem in the c:/Documents and Settings/Username/... path area. This means that you can't build ML programs in your msys home directory, nor the examples in the documentation directory. I think the first case is a problem, and I am looking into moving the home directory to c:/Documents and Settings/Username/MLton. The second case seems quite reasonable to me; we can't compile examples in /usr/share/doc/examples either. * I noticed that we don't strip executables created by MLton. On windows this can reduce the file size from 322k to 232k for hello-world. I can't imagine that the C debugging symbols are very useful anyway. Perhaps we should strip by default? * All regressions except the test-{spawn,create} pass. * I'm investigating including a Win64 toolchain in the MSI with a MLton cross-target. Then users could build 64-bit applications as simply as '-target win64'. The main downside seems to be the increased msi file size. I propose we provide two packages: a win32 only version and a win32/64 version. Both would default to outputting win32 and be win32 hosted. On a final note, I'll mention that Windows7 impressed me. I think it's quite likely most people will be upgrading to it. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090704/bed222f4/attachment.htm From wesley at terpstra.ca Sat Jul 4 09:29:47 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sat Jul 4 09:30:21 2009 Subject: [MLton] Re: Windows 7 x64 vs MLton r7999 In-Reply-To: <162de7480907031555u7136bbfcn578a82bb2eb267f4@mail.gmail.com> References: <162de7480907031555u7136bbfcn578a82bb2eb267f4@mail.gmail.com> Message-ID: <162de7480907040929v1d4f2306v7e6dc607b57b441e@mail.gmail.com> Some progress! On Sat, Jul 4, 2009 at 12:55 AM, Wesley W. Terpstra wrote: > * A self-compile of MLton trips over the use of "mlton -stop f" in mlton/Makefile Still unresolved. * msys uses an rxvt.exe which does not work under Windows7. Using msys > -norxvt fixed this. I'll either modify the msys.bat to disable rxvt.exe or > find a fixed rxvt.exe. It works just fine without rxvt.exe, so I've gone ahead with that. This file was always giving us warnings anyway (it violates some windows logo requirement or another). > * Windows7 complains vehemently about installing MLton-7999.msi because > it's from an "unknown publisher". I need to learn how to sign MSI files; > this might require acquiring an official MLton signing certificate. A code signing certificate costs 200EUR! I think we'll just let the user click 'yes install anyway'. :P > * Windows7 appears to treat some file extensions as dangerous. Fixed. > * I noticed that we don't strip executables created by MLton. I've made the .bat file auto-strip, since that's what non-unix savy folks will be using. The shell script mlton does not strip. * I'm investigating including a Win64 toolchain in the MSI with a MLton > cross-target. I've succeeded in including this into one package. After trimming fat, it's only 10MB bigger, so I think it makes sense to include both win32/64 in the same package. The only regressions compared to mingw32 are the failing MLton.size tests. They fail because the regression script can't tell the difference between win32/64. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090704/508d8404/attachment.html From wesley at terpstra.ca Sat Jul 4 12:32:47 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sat Jul 4 12:33:22 2009 Subject: [MLton] Re: test-{create,spawn} fail(?) on MinGW In-Reply-To: <162de7480907030653y7c286fbbq85bc234f55e31f09@mail.gmail.com> References: <162de7480907030653y7c286fbbq85bc234f55e31f09@mail.gmail.com> Message-ID: <162de7480907041232x4d80713bjedb6177e4c460520@mail.gmail.com> On Fri, Jul 3, 2009 at 3:53 PM, Wesley W. Terpstra wrote: > I was running a regression of HEAD when I noticed: testing test-create/spawn fail I've fixed this for MinGW. My fix may have impacted the cygwin test-create regression for the useWindowsProcess case. I made the TerminateProcess call set the high bit on the exit status. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090704/5d62ab26/attachment.htm From fluet at tti-c.org Mon Jul 6 14:40:11 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Mon Jul 6 14:40:15 2009 Subject: [MLton] test-{create,spawn} fail(?) on MinGW In-Reply-To: <162de7480907030653y7c286fbbq85bc234f55e31f09@mail.gmail.com> References: <162de7480907030653y7c286fbbq85bc234f55e31f09@mail.gmail.com> Message-ID: On Fri, 3 Jul 2009, Wesley W. Terpstra wrote: > I was running a regression of HEAD when I noticed: > > testing test-create >> 6c6 >> < exit_status: W_EXITSTATUS 7 >> --- >>> exit_status: W_SIGNALED 7 >> difference with -type-check true >> testing test-spawn >> 6c6 >> < exit_status: W_EXITSTATUS 7 >> --- >>> exit_status: W_SIGNALED 7 >> 8c8,9 >> < exit_status: W_SIGNALED 9 >> --- >>> unhandled exception: SysErr: Unknown error: kill [nosys] >>> Nonzero exit status. >> difference with -type-check true Cygwin (with use-mmap false) has a similar diff: testing test-create 6c6 < exit_status: W_EXITSTATUS 7 --- > exit_status: W_SIGNALED 7 8c8 < exit_status: W_SIGNALED 9 --- > exit_status: W_EXITED difference with -type-check true I was happy to ignore it, since it is clearly due to a difference in interpretation of exit codes when using GetExitCodeProcess. One could go through the effort of searching the Cygwin source for how they translate from a GetExitCodeProcess exit code to a Posix style exit code, but I wasn't sure it was worth the effort. From fluet at tti-c.org Mon Jul 6 15:09:20 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Mon Jul 6 15:09:22 2009 Subject: [MLton] Windows 7 x64 vs MLton r7999 In-Reply-To: <162de7480907031555u7136bbfcn578a82bb2eb267f4@mail.gmail.com> References: <162de7480907031555u7136bbfcn578a82bb2eb267f4@mail.gmail.com> Message-ID: On Sat, 4 Jul 2009, Wesley W. Terpstra wrote: > * A self-compile of MLton trips over the use of "mlton -stop f" in > mlton/Makefile to determine dependencies. The problem is that this > (correctly) includes paths like c:/program files > (x86)/mlton/lib/mlton/sml/basis/build/sources.mlb. GNU make cannot handle > paths with spaces in their names. I imagine this is also a problem with > Windows XP, but I have a German WinXP so it is called > "C:/Programme/MLton/..." which doesn't trigger this problem. One solution > might be to make a -stop f that doesn't include anything pulled from > $(SML_LIB). This corresponds to the gcc "-MM" option which is actually what > we want in the context where -stop f is used in the Makefile. Another option > would be to simply remove the -stop f from mlton/Makefile (I did this to get > the build done, and it worked after removing the offending line). > Suggestions? This issue of paths with spaces would appear to affect all platforms, where one might install MLton at a location with spaces in the path. (MinGW is the only platform where by default the install location is a path with spaces). Does piping the output of mlton -stop f through sed to escape spaces suffice on MinGW? It seems to on other platforms. However, there are a number of different uses of 'mlton -stop f' in the mlton/Makefile, and it isn't quite as simple as using a Makefile function to abstract the piping through sed. An auxilary script in /bin might suffice. From wesley at terpstra.ca Mon Jul 6 15:11:13 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Jul 6 15:11:47 2009 Subject: [MLton] test-{create,spawn} fail(?) on MinGW In-Reply-To: References: <162de7480907030653y7c286fbbq85bc234f55e31f09@mail.gmail.com> Message-ID: <162de7480907061511v54e76f74h137b737fea4ef843@mail.gmail.com> On Mon, Jul 6, 2009 at 11:40 PM, Matthew Fluet wrote: > Cygwin (with use-mmap false) has a similar diff: > I was happy to ignore it, since it is clearly due to a difference in > interpretation of exit codes when using GetExitCodeProcess. One could go > through the effort of searching the Cygwin source for how they translate > from a GetExitCodeProcess exit code to a Posix style exit code, but I wasn't > sure it was worth the effort. > I'm not even sure that's how it works on cygwin. I have vague recollections of cygwin using a pipe mechanism to convey signal information. TerminateProcess doesn't inter-operate with signal(); it's like kill -9 where you get to pick the exit status. One could imagine emulating unix behaviour with CreateRemoteThread combined with DLL injection / VirtualAllocEx / WriteProcessMemory. Windows APIs are sick. Anyway, I think the important thing is that you can now kill a spawned process under MinGW, something that was clearly missing. I'm not sure my hack to convey 'killed by signal' was a good idea, but it gets the job done where MLton is the parent process (spawning, killing, and reaping). -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090707/c166d2f7/attachment.html From fluet at tti-c.org Mon Jul 6 15:14:40 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Mon Jul 6 15:14:42 2009 Subject: [MLton] Re: Windows 7 x64 vs MLton r7999 In-Reply-To: <162de7480907040929v1d4f2306v7e6dc607b57b441e@mail.gmail.com> References: <162de7480907031555u7136bbfcn578a82bb2eb267f4@mail.gmail.com> <162de7480907040929v1d4f2306v7e6dc607b57b441e@mail.gmail.com> Message-ID: On Sat, 4 Jul 2009, Wesley W. Terpstra wrote: > I've succeeded in including this into one package. After trimming fat, it's > only 10MB bigger, so I think it makes sense to include both win32/64 in the > same package. Fair enough. > The only regressions compared to mingw32 are the failing MLton.size tests. > They fail because the regression script can't tell the difference between > win32/64. Aren't the two platforms appearing as amd64-mingw and x86-mingw? You should be able to add appropriate platform specific regression .ok files for the different platforms. From fluet at tti-c.org Mon Jul 6 15:25:20 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Mon Jul 6 15:25:22 2009 Subject: [MLton] Re: test-{create,spawn} fail(?) on MinGW In-Reply-To: <162de7480907041232x4d80713bjedb6177e4c460520@mail.gmail.com> References: <162de7480907030653y7c286fbbq85bc234f55e31f09@mail.gmail.com> <162de7480907041232x4d80713bjedb6177e4c460520@mail.gmail.com> Message-ID: On Sat, 4 Jul 2009, Wesley W. Terpstra wrote: > On Fri, Jul 3, 2009 at 3:53 PM, Wesley W. Terpstra wrote: > >> I was running a regression of HEAD when I noticed: > > testing test-create/spawn fail > > > I've fixed this for MinGW. > > My fix may have impacted the cygwin test-create regression for the > useWindowsProcess case. I made the TerminateProcess call set the high bit on > the exit status. test-create behaves the same with r7179 and r7208. My guess is that the bit is too high to be noticed by the Cygwin/Posix WIFEXITED/etc macros (used by Process.fromStatus' to convert from an integer to the Posix.Process.exit_status datatype). From wesley at terpstra.ca Mon Jul 6 15:44:36 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Jul 6 15:53:07 2009 Subject: [MLton] Re: test-{create,spawn} fail(?) on MinGW In-Reply-To: References: <162de7480907030653y7c286fbbq85bc234f55e31f09@mail.gmail.com> <162de7480907041232x4d80713bjedb6177e4c460520@mail.gmail.com> Message-ID: <162de7480907061544h3f8e6be4l63fb4eed76a9e608@mail.gmail.com> On Tue, Jul 7, 2009 at 12:25 AM, Matthew Fluet wrote: > test-create behaves the same with r7179 and r7208. My guess is that the > bit is too high to be noticed by the Cygwin/Posix WIFEXITED/etc macros (used > by Process.fromStatus' to convert from an integer to the > Posix.Process.exit_status datatype). > Good, I choose that bit for this exact reason. I assumed it was working on cygwin and didn't want to break it. From what you said earlier, it didn't work anyway, so no harm done. :) I'm not sure it's worth trying to fix these two tests for "use-mmap false", which seems a rather obsolete feature. Use cygwin as unix, or else go with mingw for "native" windows. On a side-note: win64 is working surprisingly well! It seems there has been real progress made by the mingw-w64 project since I last tried it. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090707/86578247/attachment.htm From wesley at terpstra.ca Mon Jul 6 16:09:48 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Jul 6 16:10:21 2009 Subject: [MLton] Windows 7 x64 vs MLton r7999 In-Reply-To: References: <162de7480907031555u7136bbfcn578a82bb2eb267f4@mail.gmail.com> Message-ID: <162de7480907061609n3fe60df3wdfa3c786d0d2d54d@mail.gmail.com> On Tue, Jul 7, 2009 at 12:09 AM, Matthew Fluet wrote: > This issue of paths with spaces would appear to affect all platforms, where > one might install MLton at a location with spaces in the path. (MinGW is the > only platform where by default the install location is a path with spaces). The problem is GNU make. > Does piping the output of mlton -stop f through sed to escape spaces > suffice on MinGW? It seems to on other platforms. However, there are a > number of different uses of 'mlton -stop f' in the mlton/Makefile, and it > isn't quite as simple as using a Makefile function to abstract the piping > through sed. An auxilary script in /bin might suffice. > Changing to: $(shell if [ -r $(FILE) ]; then mlton -stop f $(FILE) | sed 's/ /\\ /'; fi) did indeed work! I didn't think make supported backslash escapes. It doesn't supports "s. Aren't the two platforms appearing as amd64-mingw and x86-mingw? You should > be able to add appropriate platform specific regression .ok files for the > different platforms. > If I modify bin/platform to report amd64, the regressions run clean. The problem is that 'uname -m' reports i686 on both win32 and win64. I don't think there is a good way to tell the difference since msys/uname are 32-bit apps in both cases. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090707/8c8b4335/attachment.html From wesley at terpstra.ca Mon Jul 6 16:27:47 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Jul 6 16:28:21 2009 Subject: [MLton] Windows 7 x64 vs MLton r7999 In-Reply-To: <162de7480907061609n3fe60df3wdfa3c786d0d2d54d@mail.gmail.com> References: <162de7480907031555u7136bbfcn578a82bb2eb267f4@mail.gmail.com> <162de7480907061609n3fe60df3wdfa3c786d0d2d54d@mail.gmail.com> Message-ID: <162de7480907061627w72f1ed91w504486b30c8b847f@mail.gmail.com> On Tue, Jul 7, 2009 at 1:09 AM, Wesley W. Terpstra wrote: > If I modify bin/platform to report amd64, the regressions run clean. The > problem is that 'uname -m' reports i686 on both win32 and win64. I don't > think there is a good way to tell the difference since msys/uname are 32-bit > apps in both cases. I lie! Here's how to reliably detect the system arch/os: mlton -target XXX -verbose 2 | grep "target arch" | cut -d" " -f6 | tr "[A-Z]" "[a-z]" I'm only half joking. The regression script is going to use mlton -target XXX anyway. Why use bin/platform? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090707/b116fedd/attachment.htm From wesley at terpstra.ca Tue Jul 7 04:08:12 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Jul 7 04:08:48 2009 Subject: [MLton] PackWord to/from nonsense Message-ID: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> As I'm sure everyone has run into at some time or another, the PackWordX API is flawed: *val* bytesPerElem > *:* int > *val* isBigEndian > *:* bool > *val* subVec > *:* Word8Vector.vector *** int *->* LargeWord.word > *val* subVecX > *:* Word8Vector.vector *** int *->* LargeWord.word > *val* subArr > *:* Word8Array.array *** int *->* LargeWord.word > *val* subArrX > *:* Word8Array.array *** int *->* LargeWord.word > *val* update > *:* Word8Array.array *** int *** LargeWord.word > *->* unit > where instead it should read something like: *type* word > *val* bytesPerElem > *:* int > *val* isBigEndian > *:* bool > *val* subVec > *:* Word8Vector.vector *** int *->* word > ***val* subArr > *:* Word8Array.array *** int *->* word > ***val* update > *:* Word8Array.array *** int *** word *->* unit In our networking code, I worked around this by using _prim "Word8Array_subWordX" if MLton is used. This avoids the two C calls casting in and out of a 64-bit word for every word written into the data stream. I recently ran into trouble on a 64-bit machine because SeqIndex.int is not int, and I got a PrimApp error. As a stop-gap measure, I'm open to suggestions of an Int/Word type that must match SeqIndex. It would be nice to have 'unsafe' versions without the LargeWord baggage available somewhere, so _prim isn't needed. Armed with 'unsafe' PackWord, it would be easy to implement faster string/Word8Array copies, as discussed beforre. I'll also note that PackWord represents yet another case where the basis library expects MLton to optimize fromLarge o toLarge to nothing. I've been getting increasingly annoyed by the costs I pay to convert between types. I really liked Vesa's suggestion of {to/from}Fixed for the INTEGER signature. Combining that with the optimization to turn x_1227: word32 = Word8Vector_subWord32 (x_1072, x_1074) x_1226: word64 = WordU32_extdToWord64 (x_1227) x_1225: word32 = WordU64_extdToWord32 (x_1226) into x_1225:Word = x_1227 I think we would be able to achieve 0-cost conversions in almost all the cases where it is safe. If that conversion optimization were placed before commonArg and knownCase I think Int8.fromFixed o Int8.toFixed would even become a no-op with overflow checking: x_1 = ... x_2 = WordU8_sextdToWord64 x_1 x_3 = WordU64_sextdToWord8 x_2 (* from iwconv0 bounds checking: *) x_4 = WordU8_sextdToWord64 x_3 x_5 = Word64_eq (x_2, x_4) raise Overflow exception if x_5 is false First, comes the new optimization: x_3 = x_1 Then comes commonArg/commSubexp x_4 and x_3 are replaced by x_2 and x_1 respectively Then comes knownCase: Word64_eq (x_2, x_2) is never false -> exception never raised Am I correct in this assessment? If so, that's a pretty serious speed-up: 5 C calls and a potential branch turned into a no-op. Compared to 4 conversion in/out of an IntInf, things look even better! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090707/351a26ad/attachment-0001.html From wesley at terpstra.ca Tue Jul 7 05:16:32 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Jul 7 05:17:06 2009 Subject: [MLton] Re: PackWord to/from nonsense In-Reply-To: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> Message-ID: <162de7480907070516p6da9ee80kc6cc81ed051f3eb@mail.gmail.com> On Tue, Jul 7, 2009 at 1:08 PM, Wesley W. Terpstra wrote: > I really liked Vesa's suggestion of {to/from}Fixed for the INTEGER > signature. > Could we maybe get these {to/from}Fixed in a MLton.IntX structure? And perhaps petition John Reppy to have them added to the basis. I'm implementing > x_1227: word32 = Word8Vector_subWord32 (x_1072, x_1074) > x_1226: word64 = WordU32_extdToWord64 (x_1227) > x_1225: word32 = WordU64_extdToWord32 (x_1226) > into > x_1225:Word = x_1227 > at the moment. Since working on the LLVM codegen, I now have the tools to get the job done. =) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090707/3d9e4eb7/attachment.htm From wesley at terpstra.ca Tue Jul 7 08:15:11 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Jul 7 08:15:48 2009 Subject: [MLton] Re: PackWord to/from nonsense In-Reply-To: <162de7480907070516p6da9ee80kc6cc81ed051f3eb@mail.gmail.com> References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> <162de7480907070516p6da9ee80kc6cc81ed051f3eb@mail.gmail.com> Message-ID: <162de7480907070815w56981701jc2957fd9446d9983@mail.gmail.com> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: conversion.patch Type: text/x-patch Size: 7661 bytes Desc: not available Url : http://mlton.org/pipermail/mlton/attachments/20090707/6aa7cdf1/conversion.bin From wesley at terpstra.ca Tue Jul 7 14:59:37 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Jul 7 15:00:13 2009 Subject: [MLton] Re: PackWord to/from nonsense In-Reply-To: <162de7480907070815w56981701jc2957fd9446d9983@mail.gmail.com> References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> <162de7480907070516p6da9ee80kc6cc81ed051f3eb@mail.gmail.com> <162de7480907070815w56981701jc2957fd9446d9983@mail.gmail.com> Message-ID: <162de7480907071459s2752ac5cp559a08ed2d9888d6@mail.gmail.com> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: conversion.patch Type: text/x-patch Size: 7365 bytes Desc: not available Url : http://mlton.org/pipermail/mlton/attachments/20090707/13548872/conversion.bin From fluet at tti-c.org Tue Jul 7 17:39:10 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Tue Jul 7 17:39:14 2009 Subject: [MLton] Re: test-{create,spawn} fail(?) on MinGW In-Reply-To: <162de7480907061544h3f8e6be4l63fb4eed76a9e608@mail.gmail.com> References: <162de7480907030653y7c286fbbq85bc234f55e31f09@mail.gmail.com> <162de7480907041232x4d80713bjedb6177e4c460520@mail.gmail.com> <162de7480907061544h3f8e6be4l63fb4eed76a9e608@mail.gmail.com> Message-ID: On Tue, 7 Jul 2009, Wesley W. Terpstra wrote: > On Tue, Jul 7, 2009 at 12:25 AM, Matthew Fluet wrote: > >> test-create behaves the same with r7179 and r7208. My guess is that the >> bit is too high to be noticed by the Cygwin/Posix WIFEXITED/etc macros (used >> by Process.fromStatus' to convert from an integer to the >> Posix.Process.exit_status datatype). > > Good, I choose that bit for this exact reason. I assumed it was working on > cygwin and didn't want to break it. From what you said earlier, it didn't > work anyway, so no harm done. :) Agreed. > I'm not sure it's worth trying to fix these two tests for "use-mmap false", > which seems a rather obsolete feature. Use cygwin as unix, or else go with > mingw for "native" windows. I agree with the "use cygwin as unix". Indeed, in the upcoming Cygwin 1.7, one gets a warning upon any use of a Windows path. I'm thinking that after the MLton release, the Path module should drop all of the "Windows hacks" for Cygwin. > On a side-note: win64 is working surprisingly well! It seems there has been > real progress made by the mingw-w64 project since I last tried it. Nice to hear. From fluet at tti-c.org Tue Jul 7 17:48:44 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Tue Jul 7 17:48:47 2009 Subject: [MLton] Windows 7 x64 vs MLton r7999 In-Reply-To: <162de7480907061609n3fe60df3wdfa3c786d0d2d54d@mail.gmail.com> References: <162de7480907031555u7136bbfcn578a82bb2eb267f4@mail.gmail.com> <162de7480907061609n3fe60df3wdfa3c786d0d2d54d@mail.gmail.com> Message-ID: On Tue, 7 Jul 2009, Wesley W. Terpstra wrote: > On Tue, Jul 7, 2009 at 12:09 AM, Matthew Fluet wrote: >> Aren't the two platforms appearing as amd64-mingw and x86-mingw? You should >> be able to add appropriate platform specific regression .ok files for the >> different platforms. > > If I modify bin/platform to report amd64, the regressions run clean. The > problem is that 'uname -m' reports i686 on both win32 and win64. I don't > think there is a good way to tell the difference since msys/uname are 32-bit > apps in both cases. Ah, I understand now. It is a situtation of the host platform supporting multiple target platforms. From fluet at tti-c.org Tue Jul 7 17:51:33 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Tue Jul 7 17:51:37 2009 Subject: [MLton] Windows 7 x64 vs MLton r7999 In-Reply-To: <162de7480907061627w72f1ed91w504486b30c8b847f@mail.gmail.com> References: <162de7480907031555u7136bbfcn578a82bb2eb267f4@mail.gmail.com> <162de7480907061609n3fe60df3wdfa3c786d0d2d54d@mail.gmail.com> <162de7480907061627w72f1ed91w504486b30c8b847f@mail.gmail.com> Message-ID: On Tue, 7 Jul 2009, Wesley W. Terpstra wrote: > On Tue, Jul 7, 2009 at 1:09 AM, Wesley W. Terpstra wrote: >> If I modify bin/platform to report amd64, the regressions run clean. The >> problem is that 'uname -m' reports i686 on both win32 and win64. I don't >> think there is a good way to tell the difference since msys/uname are 32-bit >> apps in both cases. > > I lie! Here's how to reliably detect the system arch/os: > > mlton -target XXX -verbose 2 | grep "target arch" | cut -d" " -f6 | tr > "[A-Z]" "[a-z]" > > I'm only half joking. The regression script is going to use mlton -target > XXX anyway. Why use bin/platform? Well, I don't think this works when cross compiling and running the regressions. Also, the number of regressions that "fail" when picking the wrong *.ok file is fairly small, and it is easy to eyeball the ones that are not really failures. From wesley at terpstra.ca Wed Jul 8 04:30:46 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Jul 8 04:31:22 2009 Subject: [MLton] Re: PackWord to/from nonsense In-Reply-To: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> Message-ID: <162de7480907080430h75fff5d1ld910a41ada35ef4c@mail.gmail.com> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: fixed-int.patch Type: text/x-patch Size: 6181 bytes Desc: not available Url : http://mlton.org/pipermail/mlton/attachments/20090708/d6f8fb42/fixed-int-0001.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: conversion.patch Type: text/x-patch Size: 8204 bytes Desc: not available Url : http://mlton.org/pipermail/mlton/attachments/20090708/d6f8fb42/conversion-0001.bin From wesley at terpstra.ca Wed Jul 8 05:38:08 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Jul 8 05:38:42 2009 Subject: [MLton] Re: PackWord to/from nonsense In-Reply-To: <162de7480907080430h75fff5d1ld910a41ada35ef4c@mail.gmail.com> References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> <162de7480907080430h75fff5d1ld910a41ada35ef4c@mail.gmail.com> Message-ID: <162de7480907080538h6b4fedfdye7f9ba6b8cc9412d@mail.gmail.com> On Wed, Jul 8, 2009 at 1:30 PM, Wesley W. Terpstra wrote: > I am considering whether it might make sense for IntInf to be a datatype: > datatype t = FixedInt of FixedInt.t | PrimIntInf of PrimIntInf.t > Converting toLarge just wraps a FixedInt conversion. Any LargeInt > arithmetic completes the conversion to PrimIntInf. By deferring the packing > of the final tagged int32 / GMP type, the optimizer would be able to see a > conversion to/from FixedInt, allowing the usual 'free' casting. I'm not sure > how good MLton would be at recognizing that operations on IntInf are always > of the PrimIntInf type. I think the only place it might not be able to > optimize this away is on recursive calls? > > I'm going to try hacking this as an experiment and see if it negatively > impacts the benchmark directory. If it doesn't work out, then the > fixed-int.patch would at least tide MLton-specific applications over. > I had an even better approach: type int = FixedInt.int option * int Whenever a calculation is performed, the FixedInt is ignored and set NONE in the result. I'm quite confident that MLton would have been able to optimize the unused parts away in nearly all cases. If you take an IntInf.int and operate on it, you ignore the FixedInt part so it is dropped as an argument. Similarly, if you are just using IntInf for a conversion, the int part is not needed and dropped. Unfortunately, this doesn't quite work because IntInf.int has to be an eqtype. Is there any way to instruct MLton to ignore the FixedInt part of the tuple when comparing these types? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090708/a515ad8e/attachment.htm From wesley at terpstra.ca Wed Jul 8 07:02:19 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Jul 8 07:02:53 2009 Subject: [MLton] _pos aka __FILE__ __LINE__ etc Message-ID: <162de7480907080702j41c6bd8drda7bad68b55c1a8f@mail.gmail.com> Would anyone else like a _pos language extension that evaluates to the current file, line, and column? For use in something like: assert (_pos, "Bad mojo", x < j) It could return "foo.sml:4.2" or maybe ("foo.sml", 4, 2) or even ("/home/terpstra/BubbleStorm/demo/foo.sml", 4, 2). -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090708/edccc9f4/attachment.html From fluet at tti-c.org Wed Jul 8 08:26:37 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Jul 8 08:26:41 2009 Subject: [MLton] PackWord to/from nonsense In-Reply-To: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> Message-ID: On Tue, 7 Jul 2009, Wesley W. Terpstra wrote: > In our networking code, I worked around this by using _prim > "Word8Array_subWordX" if MLton is used. This avoids the two C calls casting > in and out of a 64-bit word for every word written into the data stream. A number of 64-bit operations can (and should) be implemented by the native x86 codegen, to avoid the C calls. This should help even in the presence of conversion optimizations. > I > recently ran into trouble on a 64-bit machine because SeqIndex.int is not > int, and I got a PrimApp error. As a stop-gap measure, I'm open to > suggestions of an Int/Word type that must match SeqIndex. You can use the same technique that the Basis Library uses. There is an (undocumented) MLB path variable SEQINDEX_INT which expands to either "int32" or "int64", depending on the size of indices of the target platform. You can nicely package it up in a .mlb file as follows: ** seqindex.mlb local $(SML_LIB)/basis/basis.mlb in seqindex-$(SEQINDEX_INT).sml end ** seqindex-int32.sml structure SeqIndex = Int32 ** seqindex-int64.sml structure SeqIndex = Int64 > It would be nice to have 'unsafe' versions without the LargeWord baggage > available somewhere, so _prim isn't needed. Armed with 'unsafe' PackWord, it > would be easy to implement faster string/Word8Array copies, as discussed > beforre. I'm not sure why you call them "unsafe" versions. Your proposed PACK_WORD signature (with the "type word" specification) wouldn't be unsafe in any way. > I'll also note that PackWord represents yet another case where the basis > library expects MLton to optimize fromLarge o toLarge to nothing. > ... > If that conversion optimization were placed before commonArg and knownCase I > think Int8.fromFixed o Int8.toFixed would even become a no-op with overflow > checking: > > x_1 = ... > x_2 = WordU8_sextdToWord64 x_1 > x_3 = WordU64_sextdToWord8 x_2 > (* from iwconv0 bounds checking: *) > x_4 = WordU8_sextdToWord64 x_3 > x_5 = Word64_eq (x_2, x_4) > raise Overflow exception if x_5 is false > > First, comes the new optimization: > x_3 = x_1 > Then comes commonArg/commSubexp > x_4 and x_3 are replaced by x_2 and x_1 respectively > Then comes knownCase: > Word64_eq (x_2, x_2) is never false -> exception never raised > > Am I correct in this assessment? In general, yes, conversion optimization should be a win. However, the "clean-up" optimizations aren't commonArg and knownCase. The SSA shrinker (ssa/shrink.fun) will perform the necessary simplifications: * copy propagation of x_3 = x_1 (replace all uses of x_3 by x_1 and eliminate the x_3 variable) * prim-app folding of Word64_eq (x_2, x_2) to true * case simplification of a manifest discriminant knownCase handles case simplification when the discriminant is only manifest on some of the incoming edges. That is, the SSA shrinker will get: L_1: x_10 = true case x_10 of true => L_11 | false => L_12 while knownCase will get: L_1(): x_10 = true L_4(x_10) L_2(): x_20 = false L_4(x_20) L_3(): x_30 = Word64_eq (x_1, x_2) L_4(x_30) L_4(x_40): case x_40 of true => L_11 | false => L_12 transforming it to: L_1(): x_10 = true L_11() L_2(): x_20 = false L_12() L_3(): x_30 = Word64_eq (x_1, x_2) L_4(x_30) L_4(x_40): case x_40 of true => L_11 | false => L_12 It is likely that then the SSA shrinker will be able to eliminate the use of x_10 and x_20 as unused variables, perform the jump chaining to replace transfers to L_1 by L_11 and L_2 by L_12, and combine the L_3 and L_4 blocks (assuming that now L_3 is the only predecessor of L_4). > If so, that's a pretty serious speed-up: 5 > C calls and a potential branch turned into a no-op. Compared to 4 conversion > in/out of an IntInf, things look even better! From fluet at tti-c.org Wed Jul 8 08:32:09 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Jul 8 08:32:11 2009 Subject: [MLton] Re: PackWord to/from nonsense In-Reply-To: <162de7480907070516p6da9ee80kc6cc81ed051f3eb@mail.gmail.com> References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> <162de7480907070516p6da9ee80kc6cc81ed051f3eb@mail.gmail.com> Message-ID: On Tue, 7 Jul 2009, Wesley W. Terpstra wrote: > On Tue, Jul 7, 2009 at 1:08 PM, Wesley W. Terpstra wrote: >> I really liked Vesa's suggestion of {to/from}Fixed for the INTEGER >> signature. > > Could we maybe get these {to/from}Fixed in a MLton.IntX structure? Adding a MLTON_INTEGER signature and MLton.Int structures (and MLton.Position, MLton.FixedInt, MLton.LargeInt structures?) would be reasonable. We have MLton_REAL and MLTON_WORD signatures. > And perhaps petition John Reppy to have them added to the basis. I think that future revisions of the Basis Library are highly unlikely. From fluet at tti-c.org Wed Jul 8 08:38:15 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Jul 8 08:38:18 2009 Subject: [MLton] Re: PackWord to/from nonsense In-Reply-To: <162de7480907080538h6b4fedfdye7f9ba6b8cc9412d@mail.gmail.com> References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> <162de7480907080430h75fff5d1ld910a41ada35ef4c@mail.gmail.com> <162de7480907080538h6b4fedfdye7f9ba6b8cc9412d@mail.gmail.com> Message-ID: On Wed, 8 Jul 2009, Wesley W. Terpstra wrote: > I had an even better approach: > type int = FixedInt.int option * int > Whenever a calculation is performed, the FixedInt is ignored and set NONE in > the result. > > I'm quite confident that MLton would have been able to optimize the unused > parts away in nearly all cases. If you take an IntInf.int and operate on it, > you ignore the FixedInt part so it is dropped as an argument. Similarly, if > you are just using IntInf for a conversion, the int part is not needed and > dropped. I think you are right about MLton being able to drop the useless component of the tuple. > Unfortunately, this doesn't quite work because IntInf.int has to be an > eqtype. Is there any way to instruct MLton to ignore the FixedInt part of > the tuple when comparing these types? No, unless you bake in to the compiler as a primitive type. From fluet at tti-c.org Wed Jul 8 10:00:15 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Jul 8 10:00:21 2009 Subject: [MLton] Re: PackWord to/from nonsense In-Reply-To: <162de7480907070815w56981701jc2957fd9446d9983@mail.gmail.com> References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> <162de7480907070516p6da9ee80kc6cc81ed051f3eb@mail.gmail.com> <162de7480907070815w56981701jc2957fd9446d9983@mail.gmail.com> Message-ID: On Tue, 7 Jul 2009, Wesley W. Terpstra wrote: > On Tue, Jul 7, 2009 at 2:16 PM, Wesley W. Terpstra wrote: > >> I'm implementing >> >>> x_1227: word32 = Word8Vector_subWord32 (x_1072, x_1074) >>> x_1226: word64 = WordU32_extdToWord64 (x_1227) >>> x_1225: word32 = WordU64_extdToWord32 (x_1226) >>> into >>> x_1225:Word = x_1227 >>> >> at the moment. Since working on the LLVM codegen, I now have the tools to >> get the job done. > > I've attached my optimization pass. As it's my first, I'd appreciate > feedback. I'm now working on a gigantic test-conversion.sml regression to > make certain this optimization has no effect on output. Commenting on the third conversion.patch; overall, it looks very good. A nice pass with clear benefits and a simple implementation. Index: mlton/ssa/combine-conversions.fun =================================================================== --- mlton/ssa/combine-conversions.fun (revision 0) +++ mlton/ssa/combine-conversions.fun (revision 0) @@ -0,0 +1,149 @@ +(* Copyright (C) 2009 Wesley W. Tersptra. + * Copyright (C) 1999-2008 Henry Cejtin, Matthew Fluet, Suresh + * Jagannathan, and Stephen Weeks. + * Copyright (C) 1997-2000 NEC Research Institute. No need to give outdated copyright on a new file unless it is a derivative work of an existing file (in which case it inherits the existing file's plus the current year). I don't think this file is a derivative of any existing optimization pass (except in the most obvious manner of coding style), so I think 2009 suffices. +(* + * This pass looks for nested calls to (signed) extension/truncation. + * + * It processes each block in order: in depth-first order (only because the default order of blocks in a function is arbitrary, and this optimization requires visiting defs before uses) + * If the statement is not a PrimApp Word_extdToWords, skip it. + * After processing a conversion, it tags the Var for subsequent use. + * When inspecting a conversion, check if the Var operated on is also the + * result of a conversion. If it is, try to combine the two operations. + * Repeatedly simplify until hitting either a non-conversion Var or a + * case where the conversions cause an effect. + * + * The optimization rules are very simple: + * x_1 = ... + * x_2 = Word_extdToWord (x_1, {signed1}) + * x_3 = Word_extdToWord (x_2, {signed2}) + * + * W1 = width(x_1), W2 = width(x_2), W3 = width(x_3) I think the detailed comments are great. I might suggest matching the prim-app data constructors: x_1 = ... x_2 = Word_extdToWord (W1, W2, {signed1}) x_1 x_3 = Word_extdToWord (W2, W3, {signed2}) x_2 + * If W1=W2, then there is no conversions before x_1. + * This is guaranteed because W2=W3 will always trigger optimization. + * + * Case W1 <= W3 <= W2: + * x_3 = Word_extdToWord (x_1, {signed1}) + * Case W1 < W2 < W3 AND (NOT signed1 OR signed2): + * x_3 = Word_extdToWord (x_1, {signed1}) + * Case W1 = W2 < W3 + * do nothing; there are no conversions past W1 and x_2 = x_1. + * + * Case W3 <= W2 <= W1: ] + * x_3 = Word_extdToWord (x_1, {whatever}) ] W3 <= W1 && W3 <= W2 + * Case W3 <= W1 <= W2: ] just clip x1 + * x_3 = Word_extdToWord (x_1, {whatever}) ] + * + * Case W2 < W1 <= W3: unoptimized ] W2 < W1 && W2 < W3 + * Case W2 < W3 <= W1: unoptimized ] has side-effect: truncation + * + * Case W1 < W2 < W3 AND signed1 AND (NOT signed2): unoptimized + * ... each conversion affects the result separately + *) + +val { get, set, ... } = + Property.getSet (Var.plist, Property.initConst NONE) You might use Property.getSetOnce, since the property doesn't change after being set. (It will also assert if you attempt to set a property more than once, which would indicate a bug). +fun rules x3 (conversion as ((W2, W3, {signed=s2}), args, targs)) = Since the rules only apply to conversions, args must be a singleton and targs must be empty. Might be clearer to use the single arg and omit targs. (This does require a little more work at the point where the primapp is reconstructed.) + let + val x2 = Vector.sub (args, 0) + val { equals, <, <=, ... } = Relation.compare WordSize.compare + + fun stop () = set (x3, SOME conversion) + fun loop ((W1, W2, {signed=s1}), args, targs) = + rules x3 ((W1, W3, {signed=s1}), args, targs) + in + case get x2 of + NONE => stop () + | SOME (prev as ((W1, _, {signed=s1}), _, _)) => + if W1 <= W3 andalso W3 <= W2 then loop prev else + if W1 < W2 andalso W2 < W3 andalso (not s1 orelse s2) + then loop prev else + if W3 <= W1 andalso W3 <= W2 then loop prev else + (* If W2=W3, we never reach here *) + stop () + end + +val markStatement = fn + Statement.T { exp as Exp.PrimApp { args, prim, targs }, + ty = ty, + var = SOME v } => + (case Prim.name prim of + Prim.Name.Word_extdToWord a => rules v (a, args, targs) + | _ => ()) + | stmt => () See http://mlton.org/SyntacticConventions for some general coding style rules. I always find a fn with multiple branches with multiple lines very hard to read. I would use fun markStatement stmt = case stmt of Statement.T {exp as Exp.PrimApp {args, prim, targs}, ty = ty, var = SOME v} => (case Prim.name prim of Prim.Name.Word_extdToWord a => rules v (a, args, targs) | _ => ()) | _ => () +fun combine program = + let + val Program.T { datatypes, functions, globals, main } = program We always run the SSA shrinker at the end of optimizing each function, to perform the "clean-up" optimizations discussed in the previous e-mail. To prep the shrinker, seed it with the global variables: val shrink = shrinkFunction {globals = globals} + val functions = + List.map Because the order of functions in an SSA program doesn't matter, List.revMap avoids an extra reversal. + (functions, fn f => + let + (* Traverse blocks in dfs order, marking their statements *) + fun markBlock (Block.T {statements, ... }) = + (Vector.foreach (statements, markStatement); fn () => ()) + val () = Function.dfs (f, markBlock) + + (* Map the statements using the marks *) + val {args, blocks, mayInline, name, raises, returns, start} = + Function.dest f + fun mapBlock block = + let + val Block.T {args, label, statements, transfer} = block + in + Block.T {args = args, + label = label, + statements = Vector.map (statements, mapStatement), + transfer = transfer} + end + val f = + Function.new {args = args, + blocks = Vector.map (blocks, mapBlock), + mayInline = mayInline, + name = name, + raises = raises, + returns = returns, + start = start} Now that we have a new function, run the shrinker. The shrinker will clear the function: val f = shrink f + + (* Clear the mess we've made on the variables *) + val () = Function.clear f + in + f + end) However, the shrinker will not clear globals and the property will automatically be set for any global to which get was applied. val () = Vector.foreach (globals, Statement.clear) Alternatively, use Program.clearTop, though you haven't added properties to datatypes or function names, so it is a little overkill. + in + Program.T { datatypes = datatypes, + functions = functions, + globals = globals, + main = main } + end + +end I think your current Function.dfs (for the analysis) followed by Vector.map (for the transformation) is probably sufficiently efficient (and certainly the clearest), but here are some alternative structures: * Because you have sufficient information for the transformation when visiting a block in DFS order, you could accumulate transformed blocks into a Block.t list ref during the traversal. This probably isn't more efficient, since you are not changing the number of blocks; a Vector.map is likely to be more efficient than a bunch of ref updates followd by a Vector.fromList. * In the mapBlock and mapStatement, if the transformation is sufficiently rare, then it can sometimes be better to first check if the block needs to be transformed at all. If not, then simply return the existing block, avoiding the allocation (and delayed GC cost of reclaiming the discarded block). Of course, the SSA shrinker does completely rewrite the program, so it is probably a wash. Index: mlton/ssa/simplify.fun =================================================================== --- mlton/ssa/simplify.fun (revision 7211) +++ mlton/ssa/simplify.fun (working copy) @@ -15,6 +15,7 @@ structure CommonArg = CommonArg (S) structure CommonBlock = CommonBlock (S) structure CommonSubexp = CommonSubexp (S) +structure CombineConversions = CombineConversions (S) structure ConstantPropagation = ConstantPropagation (S) structure Contify = Contify (S) structure Flatten = Flatten (S) @@ -77,6 +78,7 @@ {name = "localRef", doit = LocalRef.eliminate} :: {name = "flatten", doit = Flatten.flatten} :: {name = "localFlatten3", doit = LocalFlatten.flatten} :: + {name = "combineConversions", doit = CombineConversions.combine} :: {name = "commonArg", doit = CommonArg.eliminate} :: {name = "commonSubexp", doit = CommonSubexp.eliminate} :: {name = "commonBlock", doit = CommonBlock.eliminate} :: @@ -183,6 +185,7 @@ val passGens = inlinePassGen :: (List.map([("addProfile", Profile.addProfile), + ("combineConversions", CombineConversions.combine), ("commonArg", CommonArg.eliminate), ("commonBlock", CommonBlock.eliminate), ("commonSubexp", CommonSubexp.eliminate), As mentioned in the previous e-mail, commonArg doesn't really apply to the result of the combineConversions optimization. combineConversions should come before commonSubexp for the reasons you've noted earlier. I would have liked to see combineConversions come before some inlining, since it has the effect of shrinking the size of functions using conversions. I wonder if we shouldn't have an earlier round of commonSubexp (and thereby able to move combineConversions earlier) after contify2 and before inlineNonRecursive. From wesley at terpstra.ca Wed Jul 8 08:36:47 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Jul 8 10:08:16 2009 Subject: [MLton] PackWord to/from nonsense In-Reply-To: References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> Message-ID: <162de7480907080836v7cdd9662ha61eebd50fc2988c@mail.gmail.com> On Wed, Jul 8, 2009 at 5:26 PM, Matthew Fluet wrote: > A number of 64-bit operations can (and should) be implemented by the native > x86 codegen, to avoid the C calls. This should help even in the presence of > conversion optimizations. True. However, my combineConversions pass eliminates nearly all the cases that matter. ;) Improving the x86 codegen also won't be able to eliminate the Overflow tests in Int conversions the way my pass does. > You can use the same technique that the Basis Library uses. There is an > (undocumented) MLB path variable SEQINDEX_INT which expands to either > "int32" or "int64", depending on the size of indices of the target platform. Sneaky. At this point, though, (assuming my optimization pass is ok to commit) the only benefit I get from using _prim is the lack of bounds checking. I'm not sure why you call them "unsafe" versions. Your proposed PACK_WORD > signature (with the "type word" specification) wouldn't be unsafe in any > way. I meant PackWord versions without bounds checking. I would still really like exposed this somewhere. The Unsafe library, for example. In general, yes, conversion optimization should be a win. However, the > "clean-up" optimizations aren't commonArg and knownCase. The SSA shrinker > (ssa/shrink.fun) will perform the necessary simplifications: > * copy propagation of x_3 = x_1 (replace all uses of x_3 by x_1 and > eliminate the x_3 variable) > * prim-app folding of Word64_eq (x_2, x_2) to true > * case simplification of a manifest discriminant Ok, well those passes definitely do the job from the experiments I've done. > knownCase handles case simplification when the discriminant is only > manifest on some of the incoming edges. Alright. Is my optimization pass in the wrong spot then? I put it right before commonSubexp. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090708/b966cca1/attachment.htm From fluet at tti-c.org Wed Jul 8 10:13:57 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Jul 8 10:14:00 2009 Subject: [MLton] PackWord to/from nonsense In-Reply-To: <162de7480907080836v7cdd9662ha61eebd50fc2988c@mail.gmail.com> References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> <162de7480907080836v7cdd9662ha61eebd50fc2988c@mail.gmail.com> Message-ID: On Wed, 8 Jul 2009, Wesley W. Terpstra wrote: > On Wed, Jul 8, 2009 at 5:26 PM, Matthew Fluet wrote: >> A number of 64-bit operations can (and should) be implemented by the native >> x86 codegen, to avoid the C calls. This should help even in the presence of >> conversion optimizations. > > > True. However, my combineConversions pass eliminates nearly all the cases > that matter. ;) Improving the x86 codegen also won't be able to eliminate > the Overflow tests in Int conversions the way my pass does. No, but for the cases where the Overflow test couldn't be eliminated, it would still be nice to avoid bouncing through C calls. > I'm not sure why you call them "unsafe" versions. Your proposed PACK_WORD >> signature (with the "type word" specification) wouldn't be unsafe in any >> way. > > I meant PackWord versions without bounds checking. I would still really like > exposed this somewhere. The Unsafe library, for example. O.k. Alternatively, someone could write a bounds check elimination pass. ;-) > In general, yes, conversion optimization should be a win. However, the >> "clean-up" optimizations aren't commonArg and knownCase. The SSA shrinker >> (ssa/shrink.fun) will perform the necessary simplifications: >> * copy propagation of x_3 = x_1 (replace all uses of x_3 by x_1 and >> eliminate the x_3 variable) >> * prim-app folding of Word64_eq (x_2, x_2) to true >> * case simplification of a manifest discriminant > > Ok, well those passes definitely do the job from the experiments I've done. I think what you are seeing is the use of the SSA shrinker at the end of every optimization pass. Basically, you would see the clean up after the next pass, regardless of what that pass was. >> knownCase handles case simplification when the discriminant is only >> manifest on some of the incoming edges. > > Alright. Is my optimization pass in the wrong spot then? I put it right > before commonSubexp. See comments in other message. From wesley at terpstra.ca Wed Jul 8 12:23:53 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Jul 8 12:24:27 2009 Subject: [MLton] Re: PackWord to/from nonsense In-Reply-To: References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> <162de7480907070516p6da9ee80kc6cc81ed051f3eb@mail.gmail.com> <162de7480907070815w56981701jc2957fd9446d9983@mail.gmail.com> Message-ID: <162de7480907081223w59cb4890v35e7827b1270378a@mail.gmail.com> Thank you very much for the detailed suggestions and feedback! On Wed, Jul 8, 2009 at 7:00 PM, Matthew Fluet wrote: > +fun rules x3 (conversion as ((W2, W3, {signed=s2}), args, targs)) = > > Since the rules only apply to conversions, args must be a singleton and > targs must be empty. Might be clearer to use the single arg and omit targs. > (This does require a little more work at the point where the primapp is > reconstructed.) I thought the targs was the type of the things in args? So targs must have one element in it. + val functions = + List.map > Because the order of functions in an SSA program doesn't matter, > List.revMap avoids an extra reversal. I know, but it's much easier to read a 'diff -u' of the pre.ssa and post.ssa this way. However, I guess I'm done debugging it this way now. However, the shrinker will not clear globals and the property will > automatically be set for any global to which get was applied. > val () = Vector.foreach (globals, Statement.clear) The shrinker is marking the globals? I don't think I am doing so. Should I also have run this pass on the globals? I wasn't clear on what computations (if any) globals can do. I think your current Function.dfs (for the analysis) followed by Vector.map > (for the transformation) is probably sufficiently efficient (and certainly > the clearest), but here are some alternative structures: Yes, I saw those tricks already when reading other optimization passes for tips. How expensive is Function.dominatorTree? I had a much simpler version of the code where I used Tree.map to pre-order rewrite variables and then folded the tree into a list. I decided that the algorithm involved was probably prohibitively expensive. It would be very nice if there was a dfsMap function or something that let you rewrite blocks at finish. As mentioned in the previous e-mail, commonArg doesn't really apply to the > result of the combineConversions optimization. combineConversions should > come before commonSubexp for the reasons you've noted earlier. > I would have liked to see combineConversions come before some inlining, > since it has the effect of shrinking the size of functions using > conversions. It doesn't shrink the code that much, and if you don't inline first you might miss an opportunity to simplify the code. > I wonder if we shouldn't have an earlier round of commonSubexp (and > thereby able to move combineConversions earlier) after contify2 and before > inlineNonRecursive. > When I compiled MLton with this optimization it affected a grand total of five conversions. So I think you're overestimating the impact on code size. The optimization has the most impact on tight loops involving conversions (typical for serialization code like I'm working on). Even there the code-size isn't greatly reduced. I meant PackWord versions without bounds checking. I would still really like >> exposed this somewhere. The Unsafe library, for example. >> > > O.k. Alternatively, someone could write a bounds check elimination pass. > ;-) > How hard is this to do? For it to be useful it would have to work with loops. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090708/1d6810ec/attachment.html From fluet at tti-c.org Thu Jul 9 15:53:34 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Thu Jul 9 15:53:38 2009 Subject: [MLton] Re: PackWord to/from nonsense In-Reply-To: <162de7480907081223w59cb4890v35e7827b1270378a@mail.gmail.com> References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> <162de7480907070516p6da9ee80kc6cc81ed051f3eb@mail.gmail.com> <162de7480907070815w56981701jc2957fd9446d9983@mail.gmail.com> <162de7480907081223w59cb4890v35e7827b1270378a@mail.gmail.com> Message-ID: On Wed, 8 Jul 2009, Wesley W. Terpstra wrote: > Thank you very much for the detailed suggestions and feedback! > > On Wed, Jul 8, 2009 at 7:00 PM, Matthew Fluet wrote: >> +fun rules x3 (conversion as ((W2, W3, {signed=s2}), args, targs)) = >> >> Since the rules only apply to conversions, args must be a singleton and >> targs must be empty. Might be clearer to use the single arg and omit targs. >> (This does require a little more work at the point where the primapp is >> reconstructed.) > > I thought the targs was the type of the things in args? So targs must have > one element in it. The "targs" element of the Exp.PrimApp constructor is "type arguments" (not "type of arguments"). It specifies the type at which a polymorphic primitive is being used. E.g., the Array_sub primitive requires a type argument in order to determine the type of the array and the type of the result. In contrast, the Word_extdToWord primitive (technically, family of primitives as instantiated by the src/dst word sizes and signedness) are not polymorphic, hence, have no type arguments. > + val functions = + List.map >> Because the order of functions in an SSA program doesn't matter, >> List.revMap avoids an extra reversal. > > I know, but it's much easier to read a 'diff -u' of the pre.ssa and post.ssa > this way. However, I guess I'm done debugging it this way now. Good point. A lot of optimizations do sufficient "damage" to the control-flow graph that a diff isn't particularly helpful. > However, the shrinker will not clear globals and the property will >> automatically be set for any global to which get was applied. >> val () = Vector.foreach (globals, Statement.clear) > > The shrinker is marking the globals? I don't think I am doing so. Should I > also have run this pass on the globals? I wasn't clear on what computations > (if any) globals can do. The shrinker is adding properties to globals, so they should be cleared. The combine conversions property is also being added to globals, since the init property is added to any property list when you "get" it without "set"ing it. In your case, if there is a Word_extdToWord applied to a global, then you will "get" the property from the global (adding and returning the default NONE). Globals can do any pure computations. On the other hand, most globals are constants, there are probably few applications of Word_extdToWord to a global that isn't immediately constant folded. > I think your current Function.dfs (for the analysis) followed by Vector.map >> (for the transformation) is probably sufficiently efficient (and certainly >> the clearest), but here are some alternative structures: > > Yes, I saw those tricks already when reading other optimization passes for > tips. > > How expensive is Function.dominatorTree? I had a much simpler version of the > code where I used Tree.map to pre-order rewrite variables and then folded > the tree into a list. I decided that the algorithm involved was probably > prohibitively expensive. It would be very nice if there was a dfsMap > function or something that let you rewrite blocks at finish. Function.dominatorTree is more expensive than Function.dfs; both are more expensive than a simple Vector.map. I don't think that the dominator tree gives you any more useful structure than the dfs; you just need to visit defs before uses. >> As mentioned in the previous e-mail, commonArg doesn't really apply to the >> result of the combineConversions optimization. combineConversions should >> come before commonSubexp for the reasons you've noted earlier. > >> I would have liked to see combineConversions come before some inlining, >> since it has the effect of shrinking the size of functions using >> conversions. > > It doesn't shrink the code that much, and if you don't inline first you > might miss an opportunity to simplify the code. Sure; there is always a phase ordering problem. But, I think that combineConversions would be a fairly cheap optimization and could be applied more than once. >> I wonder if we shouldn't have an earlier round of commonSubexp (and >> thereby able to move combineConversions earlier) after contify2 and before >> inlineNonRecursive. > > When I compiled MLton with this optimization it affected a grand total of > five conversions. So I think you're overestimating the impact on code size. > The optimization has the most impact on tight loops involving conversions > (typical for serialization code like I'm working on). Even there the > code-size isn't greatly reduced. Interesting. I guess it is true that most of the conversions would be quite explicit: WordN.fromLargeWord o WordM.toLargeWord. BTW, what were the results from the benchmarks? That is good data to include in the initial commit of the combineConversions optimization. >>> I meant PackWord versions without bounds checking. I would still really like >>> exposed this somewhere. The Unsafe library, for example. >>> >> O.k. Alternatively, someone could write a bounds check elimination pass. >> ;-) >> > How hard is this to do? For it to be useful it would have to work with > loops. There is a rich literature on bounds check elimination. Actually, the redundantTests optimization is able to eliminate some array bounds checks. From wesley at terpstra.ca Thu Jul 9 03:40:01 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Thu Jul 9 15:54:15 2009 Subject: [MLton] Re: PackWord to/from nonsense In-Reply-To: <162de7480907081223w59cb4890v35e7827b1270378a@mail.gmail.com> References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> <162de7480907070516p6da9ee80kc6cc81ed051f3eb@mail.gmail.com> <162de7480907070815w56981701jc2957fd9446d9983@mail.gmail.com> <162de7480907081223w59cb4890v35e7827b1270378a@mail.gmail.com> Message-ID: <162de7480907090340o6639fe5cy1eb230c45f90f857@mail.gmail.com> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: conversion.patch Type: text/x-patch Size: 8218 bytes Desc: not available Url : http://mlton.org/pipermail/mlton/attachments/20090709/18e0ce68/conversion.bin From fluet at tti-c.org Thu Jul 9 16:18:27 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Thu Jul 9 16:18:31 2009 Subject: [MLton] Re: PackWord to/from nonsense In-Reply-To: <162de7480907090340o6639fe5cy1eb230c45f90f857@mail.gmail.com> References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> <162de7480907070516p6da9ee80kc6cc81ed051f3eb@mail.gmail.com> <162de7480907070815w56981701jc2957fd9446d9983@mail.gmail.com> <162de7480907081223w59cb4890v35e7827b1270378a@mail.gmail.com> <162de7480907090340o6639fe5cy1eb230c45f90f857@mail.gmail.com> Message-ID: On Thu, 9 Jul 2009, Wesley W. Terpstra wrote: > On Wed, Jul 8, 2009 at 9:23 PM, Wesley W. Terpstra wrote: > >> The shrinker is marking the globals? I don't think I am doing so. Should I >> also have run this pass on the globals? I wasn't clear on what computations >> (if any) globals can do. > > This is the only open question relevant to the patch being "done" or not. I doubt that applying the transformation to the globals would have any effect. > I've attached the new (final?) version of the optimization. Ok to commit? Just a couple of comments: Index: ssa/combine-conversions.fun =================================================================== --- ssa/combine-conversions.fun (revision 0) +++ ssa/combine-conversions.fun (revision 0) + val f = shrink f + (* Clear the mess we've made on the variables *) + val () = Function.clear f You don't need the "Function.clear f"; the "shrink f" will clear all properties from the function. Index: ssa/sources.mlb =================================================================== --- ssa/sources.mlb (revision 7211) +++ ssa/sources.mlb (working copy) @@ -54,6 +54,8 @@ global.fun multi.sig multi.fun + combine-conversions.sig + combine-conversions.fun constant-propagation.sig constant-propagation.fun contify.sig Also add to sources.cm to support SML/NJ build. Index: ssa/combine-conversions.sig =================================================================== --- ssa/combine-conversions.sig (revision 0) +++ ssa/combine-conversions.sig (revision 0) @@ -0,0 +1,21 @@ +(* Copyright (C) 2009 Wesley W. Tersptra. + * Copyright (C) 1999-2008 Henry Cejtin, Matthew Fluet, Suresh + * Jagannathan, and Stephen Weeks. + * Copyright (C) 1997-2000 NEC Research Institute. Excess copyright. Beyond that, it looks good to go. From fluet at tti-c.org Thu Jul 9 17:37:16 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Thu Jul 9 17:37:22 2009 Subject: [MLton] Re: PackWord to/from nonsense In-Reply-To: <162de7480907080430h75fff5d1ld910a41ada35ef4c@mail.gmail.com> References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> <162de7480907080430h75fff5d1ld910a41ada35ef4c@mail.gmail.com> Message-ID: On Wed, 8 Jul 2009, Wesley W. Terpstra wrote: > It would be ideal if I could get this conversion code to work also on > Int.{to,from}Large, but there are two problems: 1) packing an IntInf > involves shifting and tagging 2) FixedInt doesn't fit into IntInf without > using GMP's representation. Another option is to make the (unsafe) IntInf conversions into primitives: IntInf_truncToWord of WordSize.t | Word_extdToIntInf of WordSize.t * {signed: bool} Like the (unsafe) Word conversions, the IntInf_truncToWord does not check for Overflow; that check would be left (in some fashion) to the Basis Library implementation. Your conversion code should work with fun rules x3 (conversion as ((W2: WordSize.t option, W3: WordSize.t option, {signed=s2}), x2)) = let val { equals, <, <=, ... } = Relation.compare (fn (NONE, NONE) => EQUAL | (NONE, SOME _) => GREATER | (SOME _, NONE) => LESS | (SOME w1, SOME w3) => WordSize.compare (w1, w3)) That is, represent a width as a WordSize.t option, with NONE for the infinite width (IntInf). The downside of this implementation is that the shifting and tagging (small int inf) or vector creation (big int inf) for extending to an int inf and untagging and shifting or vector extraction for truncating an int inf that currently resides in basis-library/integer/int-inf0.sml would need to be implemented in the compiler (perhaps expanding the IntInf_truncToWord and IntInf_extdToWord in the combine conversions pass, since they wouldn't be needed after). It isn't terribly complicated code, but it is a little more verbose to write in SSA, rather than in SML. From fluet at tti-c.org Thu Jul 9 17:55:25 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Thu Jul 9 17:55:28 2009 Subject: [MLton] Windows 7 x64 vs MLton r7999 In-Reply-To: <162de7480907031555u7136bbfcn578a82bb2eb267f4@mail.gmail.com> References: <162de7480907031555u7136bbfcn578a82bb2eb267f4@mail.gmail.com> Message-ID: On Sat, 4 Jul 2009, Wesley W. Terpstra wrote: > * msys uses an rxvt.exe which does not work under Windows7. Using msys > -norxvt fixed this. I'll either modify the msys.bat to disable rxvt.exe or > find a fixed rxvt.exe. What purpose does msys.bat serve in the context of a MLton-NNNN.msi distribution? If the user has mingw/msys installed, then won't it come with a msys.bat? If the user doesn't have mingw/msys, then won't (the MLton) msys.bat not find any of the necessary executables? From wesley at terpstra.ca Fri Jul 10 04:06:57 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Fri Jul 10 04:07:32 2009 Subject: [MLton] Re: PackWord to/from nonsense In-Reply-To: References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> <162de7480907080430h75fff5d1ld910a41ada35ef4c@mail.gmail.com> Message-ID: <162de7480907100406o4b2bda14tb753ba046bae009a@mail.gmail.com> On Fri, Jul 10, 2009 at 2:37 AM, Matthew Fluet wrote: > On Wed, 8 Jul 2009, Wesley W. Terpstra wrote: > >> It would be ideal if I could get this conversion code to work also on >> Int.{to,from}Large, but there are two problems: 1) packing an IntInf >> involves shifting and tagging 2) FixedInt doesn't fit into IntInf without >> using GMP's representation. >> > > Another option is to make the (unsafe) IntInf conversions into primitives: > IntInf_truncToWord of WordSize.t > | Word_extdToIntInf of WordSize.t * {signed: bool} That's a great idea! Most conversions done in existing code are IntX.{to/from}Large so I expect this would improve the performance of the patch a lot. > The downside of this implementation is that the shifting and tagging (small > int inf) or vector creation (big int inf) for extending to an int inf and > untagging and shifting or vector extraction for truncating an int inf that > currently resides in basis-library/integer/int-inf0.sml would need to be > implemented in the compiler (perhaps expanding the IntInf_truncToWord and > IntInf_extdToWord in the combine conversions pass, since they wouldn't be > needed after). It isn't terribly complicated code, but it is a little more > verbose to write in SSA, rather than in SML. > Would there be a negative performance impact since this code would not be inlined any more? I am not concerned with "verbose" typing on my part. ;) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090710/a98d8918/attachment.htm From wesley at terpstra.ca Fri Jul 10 04:09:46 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Fri Jul 10 04:10:21 2009 Subject: [MLton] Windows 7 x64 vs MLton r7999 In-Reply-To: References: <162de7480907031555u7136bbfcn578a82bb2eb267f4@mail.gmail.com> Message-ID: <162de7480907100409j2767b47fu80a9dd2f9af308ab@mail.gmail.com> On Fri, Jul 10, 2009 at 2:55 AM, Matthew Fluet wrote: > On Sat, 4 Jul 2009, Wesley W. Terpstra wrote: > >> * msys uses an rxvt.exe which does not work under Windows7. Using msys >> -norxvt fixed this. I'll either modify the msys.bat to disable rxvt.exe or >> find a fixed rxvt.exe. >> > > What purpose does msys.bat serve in the context of a MLton-NNNN.msi > distribution? It works from the windows shell (cmd.exe), the double-click compiling of mlb/sml files in explorer / elsewhere, and the way MLton gets invoked from other windows apps (like eclipse). ie: It works in all the places where you are using MLton as a native windows compiler and not from some unix-like shell. The people in my office use mlton.bat almost exclusively. "I don't need that UNIX shit!" (obviously not my opinion) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090710/92431185/attachment.html From fluet at tti-c.org Fri Jul 10 04:44:54 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Fri Jul 10 04:44:58 2009 Subject: [MLton] Windows 7 x64 vs MLton r7999 In-Reply-To: <162de7480907100409j2767b47fu80a9dd2f9af308ab@mail.gmail.com> References: <162de7480907031555u7136bbfcn578a82bb2eb267f4@mail.gmail.com> <162de7480907100409j2767b47fu80a9dd2f9af308ab@mail.gmail.com> Message-ID: On Fri, 10 Jul 2009, Wesley W. Terpstra wrote: > On Fri, Jul 10, 2009 at 2:55 AM, Matthew Fluet wrote: > >> On Sat, 4 Jul 2009, Wesley W. Terpstra wrote: >> >>> * msys uses an rxvt.exe which does not work under Windows7. Using msys >>> -norxvt fixed this. I'll either modify the msys.bat to disable rxvt.exe or >>> find a fixed rxvt.exe. >>> >> >> What purpose does msys.bat serve in the context of a MLton-NNNN.msi >> distribution? > > > It works from the windows shell (cmd.exe), the double-click compiling of > mlb/sml files in explorer / elsewhere, and the way MLton gets invoked from > other windows apps (like eclipse). > > ie: It works in all the places where you are using MLton as a native windows > compiler and not from some unix-like shell. The people in my office use > mlton.bat almost exclusively. "I don't need that UNIX shit!" (obviously not > my opinion) I was asking about msys.bat (not mlton.bat), which seems precisely to launch up /bin/sh (that UNIX thing ;). From wesley at terpstra.ca Fri Jul 10 05:34:19 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Fri Jul 10 05:34:53 2009 Subject: [MLton] Windows 7 x64 vs MLton r7999 In-Reply-To: References: <162de7480907031555u7136bbfcn578a82bb2eb267f4@mail.gmail.com> <162de7480907100409j2767b47fu80a9dd2f9af308ab@mail.gmail.com> Message-ID: <162de7480907100534p1e96c6bdk675d22c28739fe7c@mail.gmail.com> On Fri, Jul 10, 2009 at 1:44 PM, Matthew Fluet wrote: > I was asking about msys.bat (not mlton.bat), which seems precisely to > launch up /bin/sh (that UNIX thing ;). > Ahhh. I bundle our own msys.bat because the one included by default is buggy. It segfaults on all 64bit machines. It's also patched to not use rxvt.exe which won't load on Windows 7. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090710/ecb088af/attachment.htm From vesa.a.j.k at gmail.com Sat Jul 11 04:44:11 2009 From: vesa.a.j.k at gmail.com (Vesa Karvonen) Date: Sat Jul 11 04:44:15 2009 Subject: [MLton] _pos aka __FILE__ __LINE__ etc In-Reply-To: <162de7480907080702j41c6bd8drda7bad68b55c1a8f@mail.gmail.com> References: <162de7480907080702j41c6bd8drda7bad68b55c1a8f@mail.gmail.com> Message-ID: <9e43b9a0907110444h49ba1ad2kac8acef267e2ca28@mail.gmail.com> On Wed, Jul 8, 2009 at 5:02 PM, Wesley W. Terpstra wrote: > Would anyone else like a _pos language extension that evaluates to the > current file, line, and column? For use in something like: assert (_pos, > "Bad mojo", x < j) > > It could return "foo.sml:4.2" or maybe ("foo.sml", 4, 2) or even > ("/home/terpstra/BubbleStorm/demo/foo.sml", 4, 2). Not that I wouldn't like to have a robust method of implementing assert that reports a useful source location like this, but there are some issues. First of all, having to explicitly pass _pos to assert is cumbersome. Second, the _pos token would not be portable. The code would simply not compile with other ML compilers that do not support the extension. With the current MLton, you can get a stack trace (exception history) by calling MLton.Exn.history. You could even use the exception history to implement an assert function that reports an approximate caller source location. However, I think that it is usually more helpful to see the whole trace. Perhaps the main problem with using exception history is that the current implementation of MLton.Exn.history is rather costly and can be prohibitive. -Vesa Karvonen From fluet at tti-c.org Sat Jul 11 09:59:37 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Sat Jul 11 09:59:40 2009 Subject: [MLton] Re: [MLton-commit] r7212 In-Reply-To: References: Message-ID: On Fri, 10 Jul 2009, Wesley Terpstra wrote: > Added a new pass supporting elimination / combination of type conversions. > It is able to eliminate costly conversions through LargeWord, but not > IntInf. > IntInf conversions tag variables before conversion which blocks the > analysis. Very nice commit message. It would also be worthwhile to add a short comment to doc/commitlog. Also, it would be good to document the pass at http://mlton.org/SSASimplify. > I ran the benchmark suite three times and only 'checksum' has a > significant > and reproducible change: > > MLton0 -- mlton -drop-pass combineConversions > MLton1 -- mlton > run time ratio > benchmark MLton1 > checksum 0.45 > size > benchmark MLton0 MLton1 > checksum 187,726 186,254 > compile time > benchmark MLton0 MLton1 > checksum 4.52 4.61 > run time > benchmark MLton0 MLton1 > checksum 36.84 16.43 > > > ... which is not terribly surprising since it (and md5sum) are the only > tests which make use of type conversions and md5sum is dominated by md5. A very nice speedup, even if it is only on the one benchmark. From fluet at tti-c.org Sat Jul 11 10:04:53 2009 From: fluet at tti-c.org (Matthew Fluet) Date: Sat Jul 11 10:04:56 2009 Subject: [MLton] Re: PackWord to/from nonsense In-Reply-To: <162de7480907100406o4b2bda14tb753ba046bae009a@mail.gmail.com> References: <162de7480907070408jbbf1011wf456a2dda052d07a@mail.gmail.com> <162de7480907080430h75fff5d1ld910a41ada35ef4c@mail.gmail.com> <162de7480907100406o4b2bda14tb753ba046bae009a@mail.gmail.com> Message-ID: On Fri, 10 Jul 2009, Wesley W. Terpstra wrote: > On Fri, Jul 10, 2009 at 2:37 AM, Matthew Fluet wrote: >> The downside of this implementation is that the shifting and tagging (small >> int inf) or vector creation (big int inf) for extending to an int inf and >> untagging and shifting or vector extraction for truncating an int inf that >> currently resides in basis-library/integer/int-inf0.sml would need to be >> implemented in the compiler (perhaps expanding the IntInf_truncToWord and >> IntInf_extdToWord in the combine conversions pass, since they wouldn't be >> needed after). It isn't terribly complicated code, but it is a little more >> verbose to write in SSA, rather than in SML. > > Would there be a negative performance impact since this code would not be > inlined any more? I am not concerned with "verbose" typing on my part. ;) On the contrary --- the primitives would be counted (by default) as unit cost for the purposes of inline threshholds and so might be inlined in places where the original SML code might not have. It isn't really a question of the IntInf conversions being inlined (they are quite small compared to the thressholds), but enabling the functions into which the conversions are inlined to themselves be inlined. I think this is a reasonable way to go, but I would like to tackle it after the release; introducing the new primitives will touch a lot more of the compiler and the Basis Library implementation. Not that I think there are any bugs in the combineConversions pass, but, if there were, it is easily disabled by a '-drop-pass combineConversions'; not so with introducing and expanding the new primitives. From savev at ccs.neu.edu Sat Jul 18 20:25:47 2009 From: savev at ccs.neu.edu (Stefan Savev) Date: Sat Jul 18 20:32:00 2009 Subject: [MLton] serialize/deserialize from MLton Message-ID: <13521242.33901247973947390.JavaMail.root@zimbra> Hi, I saw the functions serialize/deserialize from the MLton extension module are commmented. Do they work? Any chance? Thanks, Stefan From Nicolas.Bertolotti at mathworks.fr Tue Jul 21 01:20:45 2009 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Tue Jul 21 01:21:39 2009 Subject: [MLton] Problem with recent implementation of gettimeofday() on MinGW Message-ID: <8320D98DA9A5C54C926D397795FE7CEA6428225801@EXCHANGE-UK.ad.mathworks.com> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 2748 bytes Desc: image002.jpg Url : http://mlton.org/pipermail/mlton/attachments/20090721/655368a4/image002.jpg From wesley at terpstra.ca Tue Jul 21 05:34:23 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Jul 21 05:35:01 2009 Subject: [MLton] Problem with recent implementation of gettimeofday() on MinGW In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA6428225801@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA6428225801@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480907210534k17047aeehb2a9be553f5c2c87@mail.gmail.com> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: horribly-bloated-gettimeofday.patch Type: text/x-patch Size: 5162 bytes Desc: not available Url : http://mlton.org/pipermail/mlton/attachments/20090721/647f6b0a/horribly-bloated-gettimeofday-0001.bin From Nicolas.Bertolotti at mathworks.fr Tue Jul 21 06:04:31 2009 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Tue Jul 21 06:05:08 2009 Subject: [MLton] Problem with recent implementation of gettimeofday() on MinGW References: <8320D98DA9A5C54C926D397795FE7CEA6428225801@EXCHANGE-UK.ad.mathworks.com> <162de7480907210534k17047aeehb2a9be553f5c2c87@mail.gmail.com> Message-ID: <8320D98DA9A5C54C926D397795FE7CEA642822587C@EXCHANGE-UK.ad.mathworks.com> (Forgot to do < Reply all >) From: Nicolas Bertolotti Sent: Tuesday, July 21, 2009 3:01 PM To: 'Wesley W. Terpstra' Subject: RE: [MLton] Problem with recent implementation of gettimeofday() on MinGW I have observed this in practice. I have a binary which ran for 12783 seconds and calls Time.now() once at the beginning of its execution and once at the end of its execution. Both calls return the same value which, I guess, is caused by the face the SML side implementation of Time.now() manages to return the value from the previous call if the new call returns a smaller value. I could not identify the exact location of the overflow by looking at the code. I suspect the instruction: nowMicroSeconds.QuadPart += 1000000 * deltaCounter.QuadPart / frequency.QuadPart; But, in my point of view, it could only lead to a failure if the call occurs after (0xFFFFFFFFFFFFFFFFULL / 10000ULL) / 0xA6E78230ULL = 658767 seconds. From: Wesley W. Terpstra [mailto:wesley@terpstra.ca] Sent: Tuesday, July 21, 2009 2:34 PM To: Nicolas Bertolotti Cc: mlton@mlton.org Subject: Re: [MLton] Problem with recent implementation of gettimeofday() on MinGW On Tue, Jul 21, 2009 at 10:20 AM, Nicolas Bertolotti > wrote: It seems to me that the new implementation of gettimeofday() (the one in the release candidate) is buggy. If a long running process calls gettimeofday() at the beginning of its execution and then calls it again after a few hours, it won't detect the wrap-around in PerformanceCounter and will not properly rebase the clock. Have you observed this in practice? If yes, see if the attached patch helps. I'm at work, so can't test it beyond seeing if it compiles, but I believe it is anal enough to catch all possible wrap-arounds. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090721/7120ad2c/attachment.html From wesley at terpstra.ca Tue Jul 21 07:20:03 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Jul 21 07:20:38 2009 Subject: [MLton] Problem with recent implementation of gettimeofday() on MinGW In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA642822587B@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA6428225801@EXCHANGE-UK.ad.mathworks.com> <162de7480907210534k17047aeehb2a9be553f5c2c87@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA642822587B@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480907210720g3a1abbb3x1d1f7c1e219687e9@mail.gmail.com> On Tue, Jul 21, 2009 at 3:01 PM, Nicolas Bertolotti < Nicolas.Bertolotti@mathworks.fr> wrote: > I have a binary which ran for 12783 seconds and calls Time.now() once at > the beginning of its execution and once at the end of its execution. > > > > I could not identify the exact location of the overflow by looking at the > code. I suspect the instruction: > > nowMicroSeconds.QuadPart += > > 1000000 * deltaCounter.QuadPart / frequency.QuadPart; > > But, in my point of view, it could only lead to a failure if the call > occurs after (0xFFFFFFFFFFFFFFFFULL / 10000ULL) / 0xA6E78230ULL = 658767 > seconds. > Actually, it would be 2^63/10^6/0xa6e78230 = 3294 seconds. That corresponds to what you saw. I'll note that your clock's frequency is significantly higher than mine! My patch uses a double there now which definitely has enough precision. Even with your faster clock rate, the QueryPerformanceCounter won't wrap around for 209 years, so the overly anal check comparing the high resolution timer to the low resolution timer is unnecessary... Still, perhaps other systems have an even faster clock. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090721/ad18f4c0/attachment.htm From Nicolas.Bertolotti at mathworks.fr Tue Jul 21 08:08:13 2009 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Tue Jul 21 08:08:50 2009 Subject: [MLton] Problem with recent implementation of gettimeofday() on MinGW In-Reply-To: <162de7480907210720g3a1abbb3x1d1f7c1e219687e9@mail.gmail.com> References: <8320D98DA9A5C54C926D397795FE7CEA6428225801@EXCHANGE-UK.ad.mathworks.com> <162de7480907210534k17047aeehb2a9be553f5c2c87@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA642822587B@EXCHANGE-UK.ad.mathworks.com> <162de7480907210720g3a1abbb3x1d1f7c1e219687e9@mail.gmail.com> Message-ID: <8320D98DA9A5C54C926D397795FE7CEA64282258CB@EXCHANGE-UK.ad.mathworks.com> Right, my mistake. Then, the only thing that we actually need is to use the "double" type. The other change you propose would make it possible to go over the 209 year limit. I won't be there to see it being reached so it's fine for me :) Thanks for your help From: Wesley W. Terpstra [mailto:wesley@terpstra.ca] Sent: Tuesday, July 21, 2009 4:20 PM To: Nicolas Bertolotti Cc: mlton@mlton.org Subject: Re: [MLton] Problem with recent implementation of gettimeofday() on MinGW On Tue, Jul 21, 2009 at 3:01 PM, Nicolas Bertolotti > wrote: I have a binary which ran for 12783 seconds and calls Time.now() once at the beginning of its execution and once at the end of its execution. I could not identify the exact location of the overflow by looking at the code. I suspect the instruction: nowMicroSeconds.QuadPart += 1000000 * deltaCounter.QuadPart / frequency.QuadPart; But, in my point of view, it could only lead to a failure if the call occurs after (0xFFFFFFFFFFFFFFFFULL / 10000ULL) / 0xA6E78230ULL = 658767 seconds. Actually, it would be 2^63/10^6/0xa6e78230 = 3294 seconds. That corresponds to what you saw. I'll note that your clock's frequency is significantly higher than mine! My patch uses a double there now which definitely has enough precision. Even with your faster clock rate, the QueryPerformanceCounter won't wrap around for 209 years, so the overly anal check comparing the high resolution timer to the low resolution timer is unnecessary... Still, perhaps other systems have an even faster clock. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090721/5eeaff64/attachment.html From wesley at terpstra.ca Tue Jul 21 16:35:12 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Jul 21 16:35:45 2009 Subject: [MLton] Problem with recent implementation of gettimeofday() on MinGW In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA64282258CB@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA6428225801@EXCHANGE-UK.ad.mathworks.com> <162de7480907210534k17047aeehb2a9be553f5c2c87@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA642822587B@EXCHANGE-UK.ad.mathworks.com> <162de7480907210720g3a1abbb3x1d1f7c1e219687e9@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA64282258CB@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480907211635s3b5c0f48n4118c0bd5f8776e2@mail.gmail.com> On Tue, Jul 21, 2009 at 5:08 PM, Nicolas Bertolotti < Nicolas.Bertolotti@mathworks.fr> > > Then, the only thing that we actually need is to use the ?double? type. > I've committed this fix and some conditional code tho handle the case that someone has an even faster performance timer than yours. The gettimeofday should now be good until unix time wraps around. The internal representation is good for some 500,000 years or 100 years for a single run of a program. Both are larger than the time till the unix clock wraps around, so gettimeofday will break due to the API first. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090722/095fa217/attachment.htm From vesa.a.j.k at gmail.com Wed Jul 22 02:55:56 2009 From: vesa.a.j.k at gmail.com (Vesa Karvonen) Date: Wed Jul 22 02:56:30 2009 Subject: [MLton] serialize/deserialize from MLton In-Reply-To: <13521242.33901247973947390.JavaMail.root@zimbra> References: <13521242.33901247973947390.JavaMail.root@zimbra> Message-ID: <9e43b9a0907220255s5dbd6921w475064e4eee3c9b3@mail.gmail.com> On Sun, Jul 19, 2009 at 6:25 AM, Stefan Savev wrote: > Hi, > I saw the functions serialize/deserialize from the > MLton extension module are commmented. > Do they work? Any chance? I'm afraid the serialization primitives have been disabled for a long time already. This question pops up every now and then. For a previous discussion, see the thread starting with the message http://mlton.org/pipermail/mlton-user/2007-October/001200.html . Depending on your needs, a combinator library for serialization might suffice. See the page http://mlton.org/Serialization for some pointers. -Vesa Karvonen From mlton at mlton.org Wed Jul 22 18:36:25 2009 From: mlton at mlton.org (VIAGRA Inc.) Date: Wed Jul 22 18:36:38 2009 Subject: [MLton] RE: Pharmacy Online Sale 82% OFF! Message-ID: <20090723153625.3135.qmail@hienvn> An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090723/499f13a2/attachment.html -------------- next part -------------- New from WebMD: Dear mlton@mlton.org!. Sign-up today! You are subscribed as mlton@mlton.org. View and manage your WebMD newsletter preferences. Subscribe to more newsletters. Change/update your email address. WebMD Privacy Policy WebMD Office of Privacy 1175 Peachtree Street, Suite 2400, Atlanta, GA 30361 ? 2009 WebMD, LLC. All rights reserved. From savev at ccs.neu.edu Sat Jul 25 16:02:17 2009 From: savev at ccs.neu.edu (Stefan Savev) Date: Sat Jul 25 16:02:35 2009 Subject: [MLton] 64bit numbers Message-ID: <32580478.96011248562937191.JavaMail.root@zimbra> Do you guys support 64 bit ints and how? Thanks, Stefan From wesley at terpstra.ca Sat Jul 25 16:09:08 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sat Jul 25 16:09:44 2009 Subject: [MLton] 64bit numbers In-Reply-To: <32580478.96011248562937191.JavaMail.root@zimbra> References: <32580478.96011248562937191.JavaMail.root@zimbra> Message-ID: <162de7480907251609k60bb3967ib77b38384727c5bd@mail.gmail.com> On Sun, Jul 26, 2009 at 1:02 AM, Stefan Savev wrote: > Do you guys support 64 bit ints and how? structure Int64 ... see http://mlton.org/basis/integer.html Also, you can change the default 'int' type to int64 using mlton -default-type 'int64'. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090726/c8f4f10f/attachment.htm From savev at ccs.neu.edu Mon Jul 27 11:57:54 2009 From: savev at ccs.neu.edu (Stefan Savev) Date: Mon Jul 27 11:58:04 2009 Subject: [MLton] hash table Message-ID: <6606462.107621248721074830.JavaMail.root@zimbra> Hi, I've been looking for a hash table in the basis library, but I did not find one. Do you have? Thanks, Stefan From gbuday at gmail.com Mon Jul 27 13:54:28 2009 From: gbuday at gmail.com (Gergely Buday) Date: Mon Jul 27 13:54:34 2009 Subject: [MLton] hash table In-Reply-To: <6606462.107621248721074830.JavaMail.root@zimbra> References: <6606462.107621248721074830.JavaMail.root@zimbra> Message-ID: <90d975d30907271354r6aef08ceoc58cb5952e3ddf7e@mail.gmail.com> 2009/7/27 Stefan Savev : > I've been looking for a hash table in the basis library, > but I did not find one. > > Do you have? There is no such thing in the Basis Library since you can implement one using the language. Remember, Basis Library functionality is for those things that can not be implemented in SML per se or the implementation would not be fast. Standard ML of New Jersey has a structure called HashTable whose signature is in theory this: http://www.smlnj.org//doc/smlnj-lib/Manual/hash-table.html but my actual signature in my sml/nj 110.68 is a bit different. - Gergely