From wesley at terpstra.ca Sun Nov 1 11:46:37 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sun Nov 1 11:47:15 2009 Subject: [MLton] Possible bug in MLton-7230.msi for MinGW and 64 bit questions In-Reply-To: <4AECF5BC.6050400@reactive-systems.com> References: <4AECF5BC.6050400@reactive-systems.com> Message-ID: <162de7480911011146h268a85a7v115b1b944e99442e@mail.gmail.com> On Sun, Nov 1, 2009 at 3:43 AM, David Hansel wrote: > whereas with the 7230 version, ?the constants from > MLton\lib\mlton\sml\mlnlffi-lib\memory\platform\rtld-flags.mingw.sml > are used which are defined as > ? val RTLD_GLOBAL = 0wx2 > ? val RTLD_LAZY ? = 0wx0 > ? val RTLD_LOCAL ?= 0wx4 > ? val RTLD_NOW ? ?= 0wx0 These match the dlfcn we got from http://code.google.com/p/dlfcn-win32/downloads/list So the newer version of MLton is correct. Inspecting the code for dlfcn reveals that only the RTLD_GLOBAL flag has any effect at all. All other flags are ignored on windows. With the old (incorrect flags) "global=true,lazy=true" -> 0x101, which means: do nothing. That's because the actual flag for global is 0x2. The new MLton gets it correct and uses 0x2 for the flags you provided. The end result of this is that the new MLton is actually honouring the global=true option whereas the old version was silently ignoring it. > After changing the constants in rtld-flags.mingw.sml to the values from the previous > version, ?the crashes went away. ?Our code calls DynLinkage.open_lib with > parameters "global=true" and "lazy=true", ?so the values of the constants > definitely come into play. Why are you setting "global=true"? Do you load subsequent libraries that depend on earlier loaded libraries? >?Should we be using different parameters or is this a bug in MLton? A mixture of both. The *older* MLton had a bug, which is resolved in the newer MLton builds. To reproduce the behaviour you had before, switch to "global=false". From wesley at terpstra.ca Sun Nov 1 16:06:53 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sun Nov 1 16:07:32 2009 Subject: [MLton] MLton.hash deeply flawed Message-ID: <162de7480911011606v6bf45c29v518f45b8620dda7c@mail.gmail.com> Strings that differ in only a few places don't get unique hash values from MLton.hash. In a program where I tried to use MLton.hash I had 38 collisions out of 8325 distinct input strings. Not good. val x = "klahjflaskjflaksjfgklajsglkasjglaksjglaksjglaksgjaklsgaslkgjaslgkjas" val y = "klahjflbskjflaksjfgklajsglkasjglaksjglaksjglaksgjaklsgaslkgjaslgkjaS" val () = print (Word32.toString (MLton.hash x) ^ "\n") val () = print (Word32.toString (MLton.hash y) ^ "\n") From hansel at reactive-systems.com Sun Nov 1 17:44:58 2009 From: hansel at reactive-systems.com (David Hansel) Date: Sun Nov 1 17:45:47 2009 Subject: [MLton] Possible bug in MLton-7230.msi for MinGW and 64 bit questions In-Reply-To: <162de7480911011146h268a85a7v115b1b944e99442e@mail.gmail.com> References: <4AECF5BC.6050400@reactive-systems.com> <162de7480911011146h268a85a7v115b1b944e99442e@mail.gmail.com> Message-ID: <4AEE399A.3040300@reactive-systems.com> Hi Wesley, > [...] > Why are you setting "global=true"? Do you load subsequent libraries > that depend on earlier loaded libraries? This part of our code is very old and I do not know why the setting was chosen. Changing the calls to "global=false" does indeed fix the crashes with the original constants in place. Thank you very much for your help. > [...] >> Also, from the web site it sounds like the MLton-7230.msi version supports >> generating Windows 64 bit executables, is that true? If so, how do we tell >> MLton to generate a 64 bit executable? > > mlton -target x86_64-w64-mingw32 your-mlb-file-here.mlb Calling mlton -target x86_64-w64-mingw32 -verbose 2 test.sml for a "Hello World" style test.sml gives me the following error: openIn c:\program files (x86)\mlton\lib\mlton\x86_64-w64-mingw32\sizes: The system cannot find the file specified. Do I need some additional files or settings? One other question: Are FFI calls supported in the Win64 executables produced by MLton? Are there any special considerations regarding 64 bit FFI? Thanks again, David -- ---------------------------------------------------------- David Hansel http://www.reactive-systems.com/ OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~hansel/pgp_public_key.txt ---------------------------------------------------------- From wesley at terpstra.ca Sun Nov 1 19:35:53 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sun Nov 1 19:36:29 2009 Subject: [MLton] Possible bug in MLton-7230.msi for MinGW and 64 bit questions In-Reply-To: <4AEE399A.3040300@reactive-systems.com> References: <4AECF5BC.6050400@reactive-systems.com> <162de7480911011146h268a85a7v115b1b944e99442e@mail.gmail.com> <4AEE399A.3040300@reactive-systems.com> Message-ID: <162de7480911011935i1db348f0v629cd33321055987@mail.gmail.com> On Mon, Nov 2, 2009 at 2:44 AM, David Hansel wrote: >>> Also, ?from the web site it sounds like the MLton-7230.msi version supports >>> generating Windows 64 bit executables, ?is that true? ?If so, ?how do we tell >>> MLton to generate a 64 bit executable? >> >> mlton -target x86_64-w64-mingw32 your-mlb-file-here.mlb > > Calling mlton -target x86_64-w64-mingw32 -verbose 2 test.sml > for a "Hello World" style test.sml gives me the following error: > > openIn c:\program files (x86)\mlton\lib\mlton\x86_64-w64-mingw32\sizes: > The system cannot find the file specified. You need to install the 64-bit pieces from the msi. When it asks you complete/typical/custom, select complete. It sounds like you selected typical which does not install the 64-bit toolchain. > One other question: ?Are FFI calls supported in the Win64 executables > produced by MLton? Yes >?Are there any special considerations regarding 64 bit FFI? No. From hansel at reactive-systems.com Sun Nov 1 20:02:47 2009 From: hansel at reactive-systems.com (David Hansel) Date: Sun Nov 1 20:03:20 2009 Subject: [MLton] Possible bug in MLton-7230.msi for MinGW and 64 bit questions In-Reply-To: <162de7480911011935i1db348f0v629cd33321055987@mail.gmail.com> References: <4AECF5BC.6050400@reactive-systems.com> <162de7480911011146h268a85a7v115b1b944e99442e@mail.gmail.com> <4AEE399A.3040300@reactive-systems.com> <162de7480911011935i1db348f0v629cd33321055987@mail.gmail.com> Message-ID: <4AEE59E7.5060709@reactive-systems.com> Hi again, Sorry for the trivial questions so far. Is there any 64 bit specific documentation that I am missing? Regarding the 64 bit FFI support, I am currently getting a linker error whenever I attempt to use FFI functions. The error is: c:/program files (x86)/mlton/bin/../lib/gcc/x86_64-w64-mingw32/4.4.1/../../../.. /x86_64-w64-mingw32/bin/ld.exe: cannot find -ldl (this is of course caused by the "-link-opt -ldl" parameter to MLton but if I leave that out then I get missing symbols). I did install the full 64 bit support. Is there anything else (more or less obvious) that I should have done? Thanks, David On 11/1/2009 10:35 PM, Wesley W. Terpstra wrote: > On Mon, Nov 2, 2009 at 2:44 AM, David Hansel > wrote: >>>> Also, from the web site it sounds like the MLton-7230.msi version supports >>>> generating Windows 64 bit executables, is that true? If so, how do we tell >>>> MLton to generate a 64 bit executable? >>> mlton -target x86_64-w64-mingw32 your-mlb-file-here.mlb >> Calling mlton -target x86_64-w64-mingw32 -verbose 2 test.sml >> for a "Hello World" style test.sml gives me the following error: >> >> openIn c:\program files (x86)\mlton\lib\mlton\x86_64-w64-mingw32\sizes: >> The system cannot find the file specified. > > You need to install the 64-bit pieces from the msi. When it asks you > complete/typical/custom, select complete. It sounds like you selected > typical which does not install the 64-bit toolchain. > >> One other question: Are FFI calls supported in the Win64 executables >> produced by MLton? > > Yes > >> Are there any special considerations regarding 64 bit FFI? > > No. -- ---------------------------------------------------------- David Hansel Chief Technology Officer -- Reactive Systems, Inc. http://www.reactive-systems.com/ (919) 324-3507 ext. 102 -- hansel@reactive-systems.com OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~hansel/pgp_public_key.txt ---------------------------------------------------------- From wesley at terpstra.ca Sun Nov 1 20:09:57 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sun Nov 1 20:10:33 2009 Subject: [MLton] Possible bug in MLton-7230.msi for MinGW and 64 bit questions In-Reply-To: <4AEE59E7.5060709@reactive-systems.com> References: <4AECF5BC.6050400@reactive-systems.com> <162de7480911011146h268a85a7v115b1b944e99442e@mail.gmail.com> <4AEE399A.3040300@reactive-systems.com> <162de7480911011935i1db348f0v629cd33321055987@mail.gmail.com> <4AEE59E7.5060709@reactive-systems.com> Message-ID: <162de7480911012009u17483fa3v320802310ee65761@mail.gmail.com> On Mon, Nov 2, 2009 at 5:02 AM, David Hansel wrote: > Sorry for the trivial questions so far. ?Is there any 64 bit specific > documentation that I am missing? There is no 64-bit specific documentation. It's exactly the same as 32 bit, though. > c:/program files (x86)/mlton/bin/../lib/gcc/x86_64-w64-mingw32/4.4.1/../../../.. > /x86_64-w64-mingw32/bin/ld.exe: cannot find -ldl > (this is of course caused by the "-link-opt -ldl" parameter to MLton > but if I leave that out then I get missing symbols). The problem is that the dlfcn library from google is a 32-bit library. You need a 64-bit library. This is a general problem; all libraries you use must also be 64-bit. Try the newest msi from . I included a 64-bit dlfcn in that build. It might not work, though. Please report back if it does (or does not) work for you. However, any other libraries you use you will have to find 64-bit versions of yourself. From hansel at reactive-systems.com Mon Nov 2 18:00:20 2009 From: hansel at reactive-systems.com (David Hansel) Date: Mon Nov 2 18:00:52 2009 Subject: [MLton] Possible bug in MLton-7230.msi for MinGW and 64 bit questions In-Reply-To: <162de7480911012009u17483fa3v320802310ee65761@mail.gmail.com> References: <4AECF5BC.6050400@reactive-systems.com> <162de7480911011146h268a85a7v115b1b944e99442e@mail.gmail.com> <4AEE399A.3040300@reactive-systems.com> <162de7480911011935i1db348f0v629cd33321055987@mail.gmail.com> <4AEE59E7.5060709@reactive-systems.com> <162de7480911012009u17483fa3v320802310ee65761@mail.gmail.com> Message-ID: <4AEF8EB4.9080301@reactive-systems.com> Hi again On 11/1/2009 11:09 PM, Wesley W. Terpstra wrote: > On Mon, Nov 2, 2009 at 5:02 AM, David Hansel > wrote: >> Sorry for the trivial questions so far. Is there any 64 bit specific >> documentation that I am missing? > > There is no 64-bit specific documentation. It's exactly the same as 32 > bit, though. > >> c:/program files (x86)/mlton/bin/../lib/gcc/x86_64-w64-mingw32/4.4.1/../../../.. >> /x86_64-w64-mingw32/bin/ld.exe: cannot find -ldl >> (this is of course caused by the "-link-opt -ldl" parameter to MLton >> but if I leave that out then I get missing symbols). > > The problem is that the dlfcn library from google is a 32-bit library. > You need a 64-bit library. This is a general problem; all libraries > you use must also be 64-bit. > > Try the newest msi from . I included a > 64-bit dlfcn in that build. It might not work, though. Please report > back if it does (or does not) work for you. > > However, any other libraries you use you will have to find 64-bit > versions of yourself. Thanks for providing the 64-bit dlfcn. We are aware of the 32-bit vs. 64-bit library incompatibilities (in fact the need to work together with a 64-bit 3rd party library is partially driving our 64-bit development). We were able to compile our code as a 64-bit application and some initial testing shows the FFI calls appear to work fine too (no problems with the 64-bit dlfcn so far). While most of our ML code appears to work fine, we do have a problem with one (rather large) module. When compiled using the native code-generator, the resulting executable will crash on startup. However, when compiled with the "C" code-generator the executable works without problems. The other parts of our code work fine for both native and "C" code-generators. We will try to break up the module to see if we can narrow down the offending code. Do you have any other suggestions on how to debug a problem like this? Are there any (rough) estimates about the runtime performance penalty for using the C code generator? Thanks again, David -- ---------------------------------------------------------- David Hansel http://www.reactive-systems.com/ OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~hansel/pgp_public_key.txt ---------------------------------------------------------- From wesley at terpstra.ca Mon Nov 2 18:28:44 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Nov 2 18:29:23 2009 Subject: [MLton] Possible bug in MLton-7230.msi for MinGW and 64 bit questions In-Reply-To: <4AEF8EB4.9080301@reactive-systems.com> References: <4AECF5BC.6050400@reactive-systems.com> <162de7480911011146h268a85a7v115b1b944e99442e@mail.gmail.com> <4AEE399A.3040300@reactive-systems.com> <162de7480911011935i1db348f0v629cd33321055987@mail.gmail.com> <4AEE59E7.5060709@reactive-systems.com> <162de7480911012009u17483fa3v320802310ee65761@mail.gmail.com> <4AEF8EB4.9080301@reactive-systems.com> Message-ID: <162de7480911021828m3d186f4fh4652d5f030261547@mail.gmail.com> On Tue, Nov 3, 2009 at 3:00 AM, David Hansel wrote: > While most of our ML code appears to work fine, ?we do have a > problem with one (rather large) module. ?When compiled using the > native code-generator, the resulting executable will crash on startup. That could be caused by a lot of things. > However, ?when compiled with the "C" code-generator the executable > works without problems. ?The other parts of our code work fine for > both native and "C" code-generators. ?We will try to break up the > module to see if we can narrow down the offending code. If you narrow it down to something that looks like a MLton bug, please post a test-case demonstrating the problem. Win64 does have a "special" ABI, so it is possible there is an incorrect corner case in MLton. >?Do you have any other suggestions on how to debug a problem like this? Supposing it's not MLton, I'd look closely at your C types. The sizes of of variables in C may have changed in a way that causes them to no longer match up to the 32-bit assumptions made on the SML side of your code. >?Are there any (rough) estimates about the runtime performance penalty for > using the C code generator? This depends heavily on the application. Sometimes the C-codegen can even be faster. Sometimes it can be much slower. From wesley at terpstra.ca Mon Nov 2 19:40:19 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Nov 2 19:41:17 2009 Subject: [MLton] MLton/MinGW installer improved Message-ID: <162de7480911021940x408a04d4o4f411c207f90d5bf@mail.gmail.com> I've just finished a fairly major revision of the installer for MLton/MinGW. There are several improvements: * Supports upgrades; previously you had to de-install before upgrading * Uses the same toolchain for both 32 and 64 bits (from the mingw-w64 project) * Keeps the complete toolchain, i.e. c++, fortran, and java compilers (previously I only kept gcc) * Reduced installer size to 28MB (despite containing some 340MB!) * Obviously, it also includes the newest MLton from svn/HEAD There are two versions of the installer, an msi and an exe. The trade-off is that the msi is bigger (59MB vs 28MB), but the exe requires 340MB of temporary space during the install (released upon completion). From nitralime at googlemail.com Tue Nov 3 01:24:45 2009 From: nitralime at googlemail.com (nitralime) Date: Tue Nov 3 01:25:29 2009 Subject: [MLton] MLton/MinGW installer improved In-Reply-To: <162de7480911021940x408a04d4o4f411c207f90d5bf@mail.gmail.com> References: <162de7480911021940x408a04d4o4f411c207f90d5bf@mail.gmail.com> Message-ID: <53e0cac60911030124p2c70ff17s63756c5ff37f6e3f@mail.gmail.com> Hi, Where is this new MLton/MinGW installer? Regrads Nik On Tue, Nov 3, 2009 at 4:40 AM, Wesley W. Terpstra wrote: > I've just finished a fairly major revision of the installer for > MLton/MinGW. There are several improvements: > * Supports upgrades; previously you had to de-install before upgrading > * Uses the same toolchain for both 32 and 64 bits (from the mingw-w64 > project) > * Keeps the complete toolchain, i.e. c++, fortran, and java > compilers (previously I only kept gcc) > * Reduced installer size to 28MB (despite containing some 340MB!) > * Obviously, it also includes the newest MLton from svn/HEAD > > There are two versions of the installer, an msi and an exe. The > trade-off is that the msi is bigger (59MB vs 28MB), but the exe > requires 340MB of temporary space during the install (released upon > completion). > > _______________________________________________ > MLton mailing list > MLton@mlton.org > http://mlton.org/mailman/listinfo/mlton > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091103/c4409d3e/attachment.htm From ville at laurikari.net Tue Nov 3 01:27:25 2009 From: ville at laurikari.net (Ville Laurikari) Date: Tue Nov 3 01:27:50 2009 Subject: [MLton] MLton/MinGW installer improved In-Reply-To: <53e0cac60911030124p2c70ff17s63756c5ff37f6e3f@mail.gmail.com> References: <162de7480911021940x408a04d4o4f411c207f90d5bf@mail.gmail.com> <53e0cac60911030124p2c70ff17s63756c5ff37f6e3f@mail.gmail.com> Message-ID: <20091103092725.GA6042@laurikari.net> Experimental releases are normally posted here: http://mlton.org/Experimental On Tue, Nov 03, 2009 at 10:24:45AM +0100, nitralime wrote: > Hi, > > Where is this new MLton/MinGW installer? -- http://hashedbits.com/ From nitralime at googlemail.com Tue Nov 3 01:30:03 2009 From: nitralime at googlemail.com (nitralime) Date: Tue Nov 3 01:31:07 2009 Subject: [MLton] MLton/MinGW installer improved In-Reply-To: <20091103092725.GA6042@laurikari.net> References: <162de7480911021940x408a04d4o4f411c207f90d5bf@mail.gmail.com> <53e0cac60911030124p2c70ff17s63756c5ff37f6e3f@mail.gmail.com> <20091103092725.GA6042@laurikari.net> Message-ID: <53e0cac60911030130v633970eesbf8fdd14be3f6d54@mail.gmail.com> But I didn't find it there (and hence the question)! Regards Nik On Tue, Nov 3, 2009 at 10:27 AM, Ville Laurikari wrote: > Experimental releases are normally posted here: > http://mlton.org/Experimental > > On Tue, Nov 03, 2009 at 10:24:45AM +0100, nitralime wrote: > > Hi, > > > > Where is this new MLton/MinGW installer? > > -- > http://hashedbits.com/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091103/d8e1c36e/attachment.html From ville at laurikari.net Tue Nov 3 01:37:48 2009 From: ville at laurikari.net (Ville Laurikari) Date: Tue Nov 3 01:40:57 2009 Subject: [MLton] MLton/MinGW installer improved In-Reply-To: <53e0cac60911030130v633970eesbf8fdd14be3f6d54@mail.gmail.com> References: <162de7480911021940x408a04d4o4f411c207f90d5bf@mail.gmail.com> <53e0cac60911030124p2c70ff17s63756c5ff37f6e3f@mail.gmail.com> <20091103092725.GA6042@laurikari.net> <53e0cac60911030130v633970eesbf8fdd14be3f6d54@mail.gmail.com> Message-ID: <20091103093748.GB6042@laurikari.net> On Tue, Nov 03, 2009 at 10:30:03AM +0100, nitralime wrote: > But I didn't find it there (and hence the question)! I see links to MLton-20091102.msi and MLton-20091102.exe. They are posted in misleading place under 200908XX, maybe that's what was throwing you off? -- http://hashedbits.com/ From nitralime at googlemail.com Tue Nov 3 02:09:44 2009 From: nitralime at googlemail.com (nitralime) Date: Tue Nov 3 02:10:22 2009 Subject: [MLton] MLton/MinGW installer improved In-Reply-To: <20091103093748.GB6042@laurikari.net> References: <162de7480911021940x408a04d4o4f411c207f90d5bf@mail.gmail.com> <53e0cac60911030124p2c70ff17s63756c5ff37f6e3f@mail.gmail.com> <20091103092725.GA6042@laurikari.net> <53e0cac60911030130v633970eesbf8fdd14be3f6d54@mail.gmail.com> <20091103093748.GB6042@laurikari.net> Message-ID: <53e0cac60911030209j59d4ebe1ma454b92dd33a0c20@mail.gmail.com> My fault! I found it now. Thank you! Regards Nik On Tue, Nov 3, 2009 at 10:37 AM, Ville Laurikari wrote: > On Tue, Nov 03, 2009 at 10:30:03AM +0100, nitralime wrote: > > But I didn't find it there (and hence the question)! > > I see links to MLton-20091102.msi and MLton-20091102.exe. They are > posted in misleading place under 200908XX, maybe that's what was > throwing you off? > > -- > http://hashedbits.com/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091103/1f389a5f/attachment.htm From wesley at terpstra.ca Tue Nov 3 08:11:51 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Nov 3 08:12:26 2009 Subject: [MLton] Re: MLton/MinGW installer improved In-Reply-To: <162de7480911021940x408a04d4o4f411c207f90d5bf@mail.gmail.com> References: <162de7480911021940x408a04d4o4f411c207f90d5bf@mail.gmail.com> Message-ID: <162de7480911030811j4f054fc0lf047927c19c2e5c5@mail.gmail.com> Now I get to eat crow. On Tue, Nov 3, 2009 at 4:40 AM, Wesley W. Terpstra wrote: > MLton/MinGW. There are several improvements: > ?* Supports upgrades; previously you had to de-install before upgrading ... there was a problem here, now fixed in the 20091103 version on the Experimental page. From hansel at reactive-systems.com Tue Nov 10 20:10:01 2009 From: hansel at reactive-systems.com (David Hansel) Date: Tue Nov 10 20:10:44 2009 Subject: [MLton] Crashes with 64-bit native code generator on Windows Message-ID: <4AFA3919.1090003@reactive-systems.com> Hi all, We are still trying to figure out why our code crashes (brings up a Windows error message box saying that the application was terminated) when compiled with the native 64-bit codegen on Windows. We were able to break down the code a bit but unfortunately not enough to produce a small enough example that could be shared here. Removing more (mostly unrelated) code makes the crash go away. In our testing we have confirmed the following: - The crashes NEVER occur before the first FFI call. Removing all FFI calls makes the application work without crashes (as well as possible without the functionality that would be provided by the FFI calls). - Sometimes the crashes occur on the first FFI call, sometimes some time after the call (within ML code), sometimes on the second or third FFI call. This changes randomly depending on which code we include. - The crashes are not caused by our user code in the FFI functions. We have removed all code from the bodies of those functions, leaving only a simple return statement. - Our FFI DLLs do not have entry functions that would be called when the DLL is loaded. - The crashes do not occur if MLton's C code generator is used We were able to create an example that only uses a single FFI call and crashes on the first call to that function. I have consolidated the (partially ml-nlffigen generated) code and listed it below. Please let me know if you find any problem in the code. Please don't mind the useless conversions in the "cp" function within "foo". In the real code they are partially within some compatibility wrapper code and removing them completely makes the crash go away. I can not see why these conversions should cause a crash. The code below does causes the crash when called from within our (large) code. It does not produce a crash when called within a small example. As mentioned, this is the only FFI call that is actually called by the code. We do have to include another function making an FFI call in order to make the crash happen. However, that call is never executed before the crash. It could be executed some time later. If that second FFI code is not present, the crash does not happen. My question basically is this: do you have any suggestions on how to debug this any further? Any MLton command-line options for debugging? Are there any optimization passes that we should try to disable? Do you know of any caveats that we might have missed when creating our DLLs? Any suggestions are welcome. Best regards, David C code: ------- __declspec(dllexport) int __stdcall foo(const char *p, int f, const void *buf, int sz) { return 1; } ML code: -------- structure F_foo = struct val lib = DynLinkage.open_lib {name = "bar.dll", lazy=true, global=false} val h = DynLinkage.lib_symbol (lib, "foo") val callop = _import * : CMemory.addr -> CMemory.cc_addr * CMemory.cc_sint * CMemory.cc_addr * CMemory.cc_sint -> CMemory.cc_sint; fun mkcall a (x1, x2, x3, x4) = C_Int.Cvt.c_sint (CMemory.unwrap_sint (callop a (CMemory.wrap_addr (C_Int.reveal (C_Int.Ptr.inject' x1)), CMemory.wrap_sint (C_Int.Cvt.ml_sint x2), CMemory.wrap_addr (C_Int.reveal x3), CMemory.wrap_sint (C_Int.Cvt.ml_sint x4)))) fun f' (x1 : C_Int.ro C_Int.uchar_obj C_Int.ptr', x2 : MLRep.Int.Signed.int, x3 : C_Int.voidptr, x4 : MLRep.Int.Signed.int) : MLRep.Int.Signed.int = C_Int.Cvt.ml_sint (C_Int.call (C_Int.mk_fptr (mkcall, DynLinkage.addr h), (x1, C_Int.Cvt.c_sint x2, x3, C_Int.Cvt.c_sint x4))) end fun foo (pp : string, f : bool, c : Word8.word vector) : bool = let val _ = print "foo-start\n" val sz = Vector.length c val buf = C.alloc' C.S.uchar (Word.fromInt sz) fun cp (i, p) = if i >= sz then () else (C.Set.uchar' (C.Ptr.|*! p, Word8.fromLargeInt (Word32.toLargeInt (Word32.fromLargeInt (Word8.toLargeInt (Vector.sub (c, i)))))); cp (i+1, C.Ptr.|+! C.S.uchar (p, 1))) in cp (0, buf); F_foo.f' (C.Ptr.null', if f then 1 else 0, C.Ptr.inject' buf, Int32.fromInt sz); C.free' buf; print "foo-end\n"; true end -- ---------------------------------------------------------- David Hansel http://www.reactive-systems.com/ OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~hansel/pgp_public_key.txt ---------------------------------------------------------- From wesley at terpstra.ca Mon Nov 16 11:07:08 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Nov 16 11:07:46 2009 Subject: [MLton] Crashes with 64-bit native code generator on Windows In-Reply-To: <4AFA3919.1090003@reactive-systems.com> References: <4AFA3919.1090003@reactive-systems.com> Message-ID: <162de7480911161107ld408687gd577838b9bccf868@mail.gmail.com> Sorry for the slow reply. On Wed, Nov 11, 2009 at 5:10 AM, David Hansel wrote: > The code below does causes the crash when called from within our (large) > code. ?It does not produce a crash when called within a small example. > As mentioned, ?this is the only FFI call that is actually called by > the code. ?We do have to include another function making an FFI call > in order to make the crash happen. ?However, ?that call is never executed > before the crash. ?It could be executed some time later. ?If that second > FFI code is not present, ?the crash does not happen. I went ahead and tried to build it. I compiled the C (bar.c) program using: x86_64-w64-mingw32-gcc -Wall -O2 -o bar.dll -shared -Wl,--out-implib,bar.a -Wl,--output-def,bar.def bar.c I wrote the following baz.mlb file: $(SML_LIB)/basis/basis.mlb $(SML_LIB)/mlnlffi-lib/mlnlffi-lib.mlb $(SML_LIB)/mlnlffi-lib/memory/memory.mlb $(SML_LIB)/mlnlffi-lib/internals/c-int.mlb ann "allowFFI true" in baz.sml end Then I compiled with: mlton -target x86_64-w64-mingw32 -link-opt -ldl -verbose 1 baz.mlb The resulting program worked. Are you using similar compile options? In the time since your last post have you perhaps found a more complete crash example? > My question basically is this: ?do you have any suggestions on how to > debug this any further? ?Any MLton command-line options for debugging? Well, there's -debug true, but gdb under 64-bit windows is so flakey I wouldn't bother trying that. In fact, the MLton.msi doesn't include the debug version of the runtime (it is over 200MB due to the windows debugging format), so you would need to build MLton from source to get the debug library. I doubt it would help you, though. > Are there any optimization passes that we should try to disable? I doubt this is an optimization problem. > Do you know of any caveats that we might have missed when creating our > DLLs? Ok, here are the things I can think of from the top of my head: 0) You're loading a 32-bit dll instead of a 64-bit one. Double check. 1) Windows might require a stack alignment that doesn't match the amd64 FFI codegen. Your program happens to end up with bad alignment, and my programs have just never been unlucky. You could declare a volatile local 64-bit variable and printf it's address in the C code. See if the offset of this variable fails to be 64-bit aligned (only) in the failing programs. 2) The __stdcall is confusing gcc. There is only one calling convention under win64. Try specifying nothing. However, I am guessing blind! Without a way to reproduce this I can't really help. I've used the FFI quite heavily under win64 in one of our recent projects without problems, so FFI definitely works most of the time. It's possible you've found a corner case, which can often be an alignment problem. Is the program really too secret to release the buggy part of its source code? MLton is free. ;) From hansel at reactive-systems.com Mon Nov 23 17:39:30 2009 From: hansel at reactive-systems.com (David Hansel) Date: Mon Nov 23 17:40:15 2009 Subject: [MLton] Crashes with 64-bit native code generator on Windows In-Reply-To: <162de7480911161107ld408687gd577838b9bccf868@mail.gmail.com> References: <4AFA3919.1090003@reactive-systems.com> <162de7480911161107ld408687gd577838b9bccf868@mail.gmail.com> Message-ID: <4B0B3952.2000607@reactive-systems.com> Hello Wesley, Wesley W. Terpstra wrote: > Sorry for the slow reply. > > On Wed, Nov 11, 2009 at 5:10 AM, David Hansel > wrote: >> The code below does causes the crash when called from within our (large) >> code. It does not produce a crash when called within a small example. >> As mentioned, this is the only FFI call that is actually called by >> the code. We do have to include another function making an FFI call >> in order to make the crash happen. However, that call is never executed >> before the crash. It could be executed some time later. If that second >> FFI code is not present, the crash does not happen. > > I went ahead and tried to build it. > [...] > The resulting program worked. Are you using similar compile options? We are using Microsoft Visual C++ to create our DLL's so (just in case there was some obscure compiler setting that we were missing) I gave it a try and compiled our DLL with gcc -- which didn't change anything. Our MLton command line is: mlton @MLton gc-summary hash-cons 1.0 -- -target x86_64-w64-mingw32 -codegen native -profile no -profile-stack false -const 'Exn.keepHistory false' -drop-pass deepFlatten -link-opt -ldl -output foo.exe -verbose 2 foo.mlb I do not know exactly what the '-drop-pass deepFlatten' does but it was put in by Stephen Weeks back in 2006 when he assisted us in making our code compile with MLton. If I remember correctly there was a compiler performace issue. However, as you said before, optimization settings are probably not the problem here. > In the time since your last post have you perhaps found a more > complete crash example? Unfortunately no. I have been trying but the crash goes away anytime I cut down the code some more to produce a smaller example. >> My question basically is this: do you have any suggestions on how to >> debug this any further? Any MLton command-line options for debugging? > > Well, there's -debug true, but gdb under 64-bit windows is so flakey I > wouldn't bother trying that. In fact, the MLton.msi doesn't include > the debug version of the runtime (it is over 200MB due to the windows > debugging format), so you would need to build MLton from source to get > the debug library. I doubt it would help you, though. That's unfortunate. >> Are there any optimization passes that we should try to disable? > > I doubt this is an optimization problem. > >> Do you know of any caveats that we might have missed when creating our >> DLLs? > > Ok, here are the things I can think of from the top of my head: > 0) You're loading a 32-bit dll instead of a 64-bit one. Double check. Double- and triple-checked that. > 1) Windows might require a stack alignment that doesn't match the > amd64 FFI codegen. Your program happens to end up with bad alignment, > and my programs have just never been unlucky. You could declare a > volatile local 64-bit variable and printf it's address in the C code. > See if the offset of this variable fails to be 64-bit aligned (only) > in the failing programs. An alignment problem or something similar is what I suspect, too. Creating a local variable won't help because the process dies even before the first time it enters the code in the DLL, so any printf in there will not happen before the crash. > 2) The __stdcall is confusing gcc. There is only one calling > convention under win64. Try specifying nothing. I've tried with and without. No difference. > However, I am guessing blind! Without a way to reproduce this I can't > really help. I've used the FFI quite heavily under win64 in one of our > recent projects without problems, so FFI definitely works most of the > time. It's possible you've found a corner case, which can often be an > alignment problem. > Is the program really too secret to release the buggy part of its > source code? MLton is free. ;) It's good to hear that the FFI has been tested in win64. I completely understand about the guessing, we do have the same problems with our customer (our product not working with their code, can't send the code). Unfortunately since this is a commercial application and we do have to include a large part of our code to make the crash happen I can definitely not post the code to the list. If we can't figure this out otherwise we might be able to set up an NDA with you so we could send the code to you in private. One thing I can make available is the executable that actually experiences the crash as well as the MLton-produced assembly code. I don't know what kind of debugging tools you have available and whether that would be any help. Please let me know. There are two observations that I have made since my last post. They may or may not be related to the actual problem but I thought I'd mention them anyways: I was looking into what could be causing the problem and came across file MLton/lib/mlton/sml/mlnlffi-lib/memory/linkage-libdl.sml which is of course used by the FFI. I wasn't completely sure what the "era" deal in that code is, so I changed the body of function "get" to just "f()", resolving the FFI function's address before every call. After that change, all crashes were gone. Furthermore, changing the body of "get" to just "a" does NOT fix the crashes. That looked good so I added some "print" statements in "get" to see whether there is a problem with the address not being resolved properly. Unfortunately, just adding the "print" statements also made the crashes go away. In fact, just adding 'print "";' at the beginning of "get" eliminates the crashes. Interestingly, this eliminates the crashes completely. With other changes in our code I was able to eliminate some instances of the crashes but new ones would pop up at other places. I suspect that the proximity of this code to the actual FFI calls might play a role in that. I gave the "Debugging Tools for Windows" debugger a try and loaded the crashing executable there. With that, I was able to track the crash in our simplest example to the following assembly code: 00000000`0054b2c8 4c897df8 mov qword ptr [rbp-8],r15 00000000`0054b2cc 48892d3d408000 mov qword ptr [rsim4c_mlton!MLton_main+0x402901 (00000000`0080403d)],rbp 00000000`0054b2d3 4c892526408000 mov qword ptr [rsim4c_mlton!MLton_main+0x4028ea (00000000`00804026)],r12 00000000`0054b2da ff15683d8000 call qword ptr [rsim4c_mlton!MLton_main+0x40262c (00000000`00803d68)] ds:00000000`00d4f048=0000000000000000 Note the "=0" address at the end. The crash happens because the result address of the indirect call is 0, which could be some hint but I don't know how to look into this any further. Do you have a suggestion how to track this back to MLton's assembly output or even to the original ML code? Best regards, David -- ---------------------------------------------------------- David Hansel http://www.reactive-systems.com/ OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~hansel/pgp_public_key.txt ---------------------------------------------------------- From hansel at reactive-systems.com Mon Nov 30 08:00:42 2009 From: hansel at reactive-systems.com (David Hansel) Date: Mon Nov 30 08:01:28 2009 Subject: [MLton] Crashes with 64-bit native code generator on Windows In-Reply-To: <4B0B3952.2000607@reactive-systems.com> References: <4AFA3919.1090003@reactive-systems.com> <162de7480911161107ld408687gd577838b9bccf868@mail.gmail.com> <4B0B3952.2000607@reactive-systems.com> Message-ID: <4B13EC2A.9070600@reactive-systems.com> Hi again, I have a few more observations regarding the crashes that might ring a bell for some of the MLton developers involved in the native code-generators and/or FFI interface: 1) In my previous post I included a disassembly of the location where the crash happens. With some creative grep-ing I was able to find the location of that code within the assembly code that MLton produces for our program: _L_176132: movq (_c_stackP+0x0)(%rip),%rsp movq 0x40(%rbp),%r14 movq 0x0(%r14),%r13 movq 0x0(%r13),%r11 movl %r15d,%r9d xorq %r8,%r8 movl $0x0,%r15d movl %r15d,%edx xorq %rcx,%rcx subq $0x20,%rsp addq $0x40,%rbp leaq (_L_176133+0x0)(%rip),%r15 movq %r15,0xFFFFFFFFFFFFFFF8(%rbp) movq %rbp,(_gcState+0x10)(%rip) movq %r12,(_gcState+0x0)(%rip) call *(_applyFFTempFun+0x0)(%rip) <<------- CRASH addq $0x20,%rsp movq (_gcState+0x0)(%rip),%r12 movq (_gcState+0x10)(%rip),%rbp jmp _L_176133 I was able to reproduce this with several examples for which the crash occurs (all of which unfortunately include a large part of our code so I can not make them available here). The crash always occurs in the "*applyFFTempFun" call and always because applyFFTempFun is NULL. I am reasonably sure that this actually is the location of the crash and not just some similar looking code because if I comment out the "call" statement in the assembly code and then compile the executable from the assembly code, the crash goes away (the target of the FFI call in question is an empty function that does not do anything so it is not surprising that skipping the call does not produce other problems). Also, if I introduce an infinite loop right before the call, the compiled program hangs right before where I would usually observe the crash. 2) As I mentioned before, if I compile the program from the SML code and just insert a "print" statement in function "get" within MLton/lib/mlton/sml/mlnlffi-lib/memory/linkage-libdl.sml, the crash also does not occur. Interestingly, the MLton-produced assembly code for that version (only change is the "print" statement) does not contain ANY calls to "applyFFTempFun". 3) Looking at the MLton source code (amd64-generate-transfers.fun), I can see that calls to "applyFFTempFun" seem to be inserted for "Indirect" FFI calls. I do not know enough about the code generator or the FFI interface to make much sense out of this. However, I can see that the MLTon-produced code with the crash only contains a call to "applyFFTempFun" (which I assume is created in line 1566 of file amd64-generate-transfers.fun) but never any code that would set the value of "applyFFTempFun" (which I assume should be created in line 1183 of file amd64-generate-transfers.fun). Given these observations, does anyone have any suggestions about MLton debugging options or other ways to shed more light on what might be going wrong here? Thanks, David From matthew.fluet at gmail.com Mon Nov 30 08:59:07 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Mon Nov 30 08:59:43 2009 Subject: [MLton] Crashes with 64-bit native code generator on Windows In-Reply-To: <4B0B3952.2000607@reactive-systems.com> References: <4AFA3919.1090003@reactive-systems.com> <162de7480911161107ld408687gd577838b9bccf868@mail.gmail.com> <4B0B3952.2000607@reactive-systems.com> Message-ID: On Mon, Nov 23, 2009 at 8:39 PM, David Hansel wrote: > I was looking into what could be causing the problem and came across > file MLton/lib/mlton/sml/mlnlffi-lib/memory/linkage-libdl.sml which > is of course used by the FFI. I wasn't completely sure what the "era" > deal in that code is, so I changed the body of function "get" to just > "f()", resolving the FFI function's address before every call. The "era" is supposed to invalidated dynamically loaded library addresses if the executable is started up again after saving the world (MLton.World.save). Because it is a new executable invocation, the dynamically linked library needs to be reloaded and it might end up at a different address. This isn't actually done in the linkage-libdl.sml code; see the commented out "Cleaner.addNew" application. I don't recall why it is disabled. In any case, unless you are saving and loading worlds, it shouldn't affect your code. > After > that change, all crashes were gone. Furthermore, changing the body > of "get" to just "a" does NOT fix the crashes. That looked good > so I added some "print" statements in "get" to see whether there is > a problem with the address not being resolved properly. Unfortunately, > just adding the "print" statements also made the crashes go away. In > fact, just adding 'print "";' at the beginning of "get" eliminates > the crashes. Interestingly, this eliminates the crashes completely. > With other changes in our code I was able to eliminate some instances > of the crashes but new ones would pop up at other places. I suspect > that the proximity of this code to the actual FFI calls might play > a role in that. > This, and your next email, suggest that it is a bug with the native codegen. The probable role that the proximity of the "print" call plays is that there will be an C function call invoked by the "print", which "resets" the register allocator. Without the "print" call, there is a wider scope over which the register allocator is able to work, and, apparently, is mistakenly dropping a def. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091130/6f7da90d/attachment.htm From matthew.fluet at gmail.com Mon Nov 30 09:14:33 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Mon Nov 30 09:14:41 2009 Subject: [MLton] Crashes with 64-bit native code generator on Windows In-Reply-To: <4B13EC2A.9070600@reactive-systems.com> References: <4AFA3919.1090003@reactive-systems.com> <162de7480911161107ld408687gd577838b9bccf868@mail.gmail.com> <4B0B3952.2000607@reactive-systems.com> <4B13EC2A.9070600@reactive-systems.com> Message-ID: On Mon, Nov 30, 2009 at 11:00 AM, David Hansel wrote: > 1) In my previous post I included a disassembly of the location where the > crash happens. With some creative grep-ing I was able to find the location > of that code within the assembly code that MLton produces for our program: > > I was able to reproduce this with several examples for which the crash > occurs (all of which unfortunately include a large part of our code so > I can not make them available here). The crash always occurs in the > "*applyFFTempFun" call and always because applyFFTempFun is NULL. > I agree that that seems to pinpoint the source of the crash. > 2) As I mentioned before, if I compile the program from the SML > code and just insert a "print" statement in function "get" within > MLton/lib/mlton/sml/mlnlffi-lib/memory/linkage-libdl.sml, the crash > also does not occur. Interestingly, the MLton-produced assembly code > for that version (only change is the "print" statement) does not contain > ANY calls to "applyFFTempFun". > More evidence. Although, in this case, I suspect that there is still an indirect call in the assembly code. It simply doesn't go through the temporary variable --- gets allocated and stays in a register. 3) Looking at the MLton source code (amd64-generate-transfers.fun), I can > see > that calls to "applyFFTempFun" seem to be inserted for "Indirect" FFI > calls. > I do not know enough about the code generator or the FFI interface to make > much sense out of this. > However, I can see that the MLTon-produced code with the crash only > contains > a call to "applyFFTempFun" (which I assume is created in line 1566 of file > amd64-generate-transfers.fun) but never any code that would set the value > of "applyFFTempFun" (which I assume should be created in line 1183 of file > amd64-generate-transfers.fun). > > Given these observations, does anyone have any suggestions about MLton > debugging options or other ways to shed more light on what might be going > wrong here? > Sounds like a bug in the amd64 codegen simplifier and/or register allocator. It seems that somewhere along the line, the definition of the applyFFTempFun variable is being dropped, but the use in the indirect call is being retained. When the register allocator comes along, when it doesn't locally find the def point of applyFFTempFun, it has to fetch the value from the (uninitialized) variable. Could you compile with "-native-commented 3 -native-split 0 -keep g" and post the basic block that has the call through applyFFTempFun? It will be pretty noisy, but should shed some light on what the native codegen is doing (wrong). -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091130/85dfebb7/attachment.html From hansel at reactive-systems.com Mon Nov 30 09:45:15 2009 From: hansel at reactive-systems.com (David Hansel) Date: Mon Nov 30 09:46:05 2009 Subject: [MLton] Crashes with 64-bit native code generator on Windows In-Reply-To: References: <4AFA3919.1090003@reactive-systems.com> <162de7480911161107ld408687gd577838b9bccf868@mail.gmail.com> <4B0B3952.2000607@reactive-systems.com> <4B13EC2A.9070600@reactive-systems.com> Message-ID: <4B1404AB.4040802@reactive-systems.com> Hi Matthew, Matthew Fluet wrote: > [...] > Sounds like a bug in the amd64 codegen simplifier and/or register > allocator. It seems that somewhere along the line, the definition of > the applyFFTempFun variable is being dropped, but the use in the > indirect call is being retained. When the register allocator comes > along, when it doesn't locally find the def point of applyFFTempFun, it > has to fetch the value from the (uninitialized) variable. > > Could you compile with "-native-commented 3 -native-split 0 -keep g" and > post the basic block that has the call through applyFFTempFun? It will > be pretty noisy, but should shed some light on what the native codegen > is doing (wrong). See the code below. It should match up with the code I posted before. >From what I can tell it does look like MLton puts the target address for applyFFTempFun into a register but then later does the indirect call via the memory location. Please let me know if you need any more context or other debugging information. It does seem like you are on the right track. Thanks! David /* Live: (SW64(24): ExnStack, SW32(40): Word32, SP(64): Objptr (opt_1516), SP(48): Objptr (opt_36)) */ /* begin: RP(0): Objptr (opt_22) = OP (SP(64): Objptr (opt_1516), 0): Objptr (opt_22) */ /* end: RP(0): Objptr (opt_22) = OP (SP(64): Objptr (opt_1516), 0): Objptr (opt_22) */ /* begin: RQ(0): CPointer = OQ (RP(0): Objptr (opt_22), 0): CPointer */ /* end: RQ(0): CPointer = OQ (RP(0): Objptr (opt_22), 0): CPointer */ /* CCall {args = (RQ(0): CPointer, NULL, 0x0, NULL, SW32(40): Word32), frameInfo = Some {frameLayoutsIndex = 1072}, func = {args = (CPointer, CPointer, Word32, CPointer, Word32), bytesNeeded = None, convention = cdecl, ensuresBytesFree = false, mayGC = true, maySwitchThreads = false, modifiesFrontier = true, prototype = {args = (CPointer, Int32, CPointer, Int32), res = Some Int32}, readsStackTop = true, return = Word32, symbolScope = external, target = <*>, writesStackTop = true}, return = Some L_176133} */ /* begin ccall: cdecl <*> */ /* CCALL cdecl <*>(MEM{Heap}[(MEM{Heap}[(MEM{Stack}[(MEM{GCStateHold}[((_gcState+0x10))+(0x0)])+(0x40)])+(0x0)])+(0x0)], $0x0, $0x0, $0x0, MEM{Stack}[(MEM{GCStateHold}[((_gcState+0x10))+(0x0)])+(0x28)]) */ /* ************************************************************ */ /* Cache: caches: MEM{StaticNonTemp}[(_c_stackP)+(0x0)] -> %rsp (reserved) */ movq (_c_stackP+0x0)(%rip),%rsp /* ************************************************************ */ /* movq MEM{Heap}[(MEM{Heap}[(MEM{Stack}[(MEM{GCStateHold}[((_gcState+0x10))+(0x0)])+(0x40)])+(0x0)])+(0x0)],MEM{CArg}[(_applyFFTempFun)+(0x0)] */ movq 0x40(%rbp),%r14 movq 0x0(%r14),%r13 movq 0x0(%r13),%r11 /* ************************************************************ */ /* movzlq MEM{Stack}[(MEM{GCStateHold}[((_gcState+0x10))+(0x0)])+(0x28)],MEM{CArg}[(_applyFFTempRegArg)+(0x0)] */ movl %r15d,%r9d /* ************************************************************ */ /* Cache: caches: MEM{CArg}[(_applyFFTempRegArg)+(0x0)] -> %r9 (reserved) */ /* ************************************************************ */ /* movq $0x0,MEM{CArg}[(_applyFFTempRegArg)+(0x8)] */ xorq %r8,%r8 /* ************************************************************ */ /* Cache: caches: MEM{CArg}[(_applyFFTempRegArg)+(0x8)] -> %r8 (reserved) */ /* ************************************************************ */ /* movzlq $0x0,MEM{CArg}[(_applyFFTempRegArg)+(0x10)] */ movl $0x0,%r15d movl %r15d,%edx /* ************************************************************ */ /* Cache: caches: MEM{CArg}[(_applyFFTempRegArg)+(0x10)] -> %rdx (reserved) */ /* ************************************************************ */ /* movq $0x0,MEM{CArg}[(_applyFFTempRegArg)+(0x18)] */ xorq %rcx,%rcx /* ************************************************************ */ /* Cache: caches: MEM{CArg}[(_applyFFTempRegArg)+(0x18)] -> %rcx (reserved) */ /* ************************************************************ */ /* subq $0x20,MEM{StaticNonTemp}[(_c_stackP)+(0x0)] */ subq $0x20,%rsp /* ************************************************************ */ /* Force: commit_memlocs: commit_classes: remove_memlocs: remove_classes: dead_memlocs: dead_classes: */ /* ************************************************************ */ /* addq $0x40,MEM{GCStateHold}[((_gcState+0x10))+(0x0)] */ addq $0x40,%rbp /* ************************************************************ */ /* leaq MEM{Code}[(_L_176133)+(0x0)],MEM{Stack}[(MEM{GCStateHold}[((_gcState+0x10))+(0x0)])+(0xFFFFFFFFFFFFFFF8)] */ leaq (_L_176133+0x0)(%rip),%r15 movq %r15,0xFFFFFFFFFFFFFFF8(%rbp) movq %rbp,(_gcState+0x10)(%rip) /* ************************************************************ */ /* Force: commit_memlocs: MEM{Stack}[(MEM{GCStateHold}[((_gcState+0x10))+(0x0)])+(0xFFFFFFFFFFFFFFF8)] commit_classes: remove_memlocs: remove_classes: dead_memlocs: dead_classes: */ /* ************************************************************ */ /* Force: commit_memlocs: commit_classes: GCStateVolatile GCState CStatic Globals Stack Heap Code CStack remove_memlocs: remove_classes: dead_memlocs: dead_classes: */ /* ************************************************************ */ /* Force: commit_memlocs: commit_classes: GCStateVolatile GCStateHold GCState Globals Stack Heap remove_memlocs: remove_classes: dead_memlocs: dead_classes: */ movq %r12,(_gcState+0x0)(%rip) /* ************************************************************ */ /* CCall */ /* ************************************************************ */ /* call *MEM{CArg}[(_applyFFTempFun)+(0x0)] */ call *(_applyFFTempFun+0x0)(%rip) /* ************************************************************ */ /* XmmUnreserve: registers: */ /* ************************************************************ */ /* Unreserve: registers: %rcx %rdx %r8 %r9 */ /* ************************************************************ */ /* Force: commit_memlocs: commit_classes: remove_memlocs: remove_classes: dead_memlocs: dead_classes: GCStateVolatile GCStateHold GCState Globals Stack Heap */ /* ************************************************************ */ /* Return: [(%eax,MEM{StaticTemp}[(_cReturnTemp)+(0x0)])] */ /* ************************************************************ */ /* addq $0x20,MEM{StaticNonTemp}[(_c_stackP)+(0x0)] */ addq $0x20,%rsp /* ************************************************************ */ /* Unreserve: registers: %rsp */ /* ************************************************************ */ /* Cache: caches: MEM{GCStateHold}[((_gcState+0x0))+(0x0)] -> %r12 (reserved) MEM{GCStateHold}[((_gcState+0x10))+(0x0)] -> %rbp (reserved) */ movq (_gcState+0x0)(%rip),%r12 movq (_gcState+0x10)(%rip),%rbp /* ************************************************************ */ /* XmmCache: caches: */ /* ************************************************************ */ /* Cache: caches: MEM{StaticTemp}[(_cReturnTemp)+(0x0)] -> %eax (reserved) */ /* ************************************************************ */ /* Force: commit_memlocs: commit_classes: GCStateVolatile GCState CStatic Globals Stack Heap Code CStack remove_memlocs: remove_classes: dead_memlocs: dead_classes: */ /* ************************************************************ */ /* jmp _L_176133 */ jmp _L_176133 From matthew.fluet at gmail.com Mon Nov 30 10:44:40 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Mon Nov 30 10:45:18 2009 Subject: [MLton] Crashes with 64-bit native code generator on Windows In-Reply-To: <4B1404AB.4040802@reactive-systems.com> References: <4AFA3919.1090003@reactive-systems.com> <162de7480911161107ld408687gd577838b9bccf868@mail.gmail.com> <4B0B3952.2000607@reactive-systems.com> <4B13EC2A.9070600@reactive-systems.com> <4B1404AB.4040802@reactive-systems.com> Message-ID: On Mon, Nov 30, 2009 at 12:45 PM, David Hansel wrote: > See the code below. It should match up with the code I posted before. > From what I can tell it does look like MLton puts the target address for > applyFFTempFun into a register but then later does the indirect call via > the memory location. > Yes, it looks like it gets dropped into %r11, but not used from that location. > Please let me know if you need any more context or other debugging > information. It does seem like you are on the right track. > We need to find out when the codegen loses track of the fact that %r11 has applyFFTempFun. Could you compile with "-native-commented 6"? That's the most debugging information that we can get from a precompiled MLton binary. It produces a *lot* of debugging information (in the form of comments in the assembly). Rather than posting to the mailing list, I suggest posting the basic block to http://mlton.org/TemporaryUpload. -Matthew -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091130/7888612b/attachment.htm From hansel at reactive-systems.com Mon Nov 30 13:19:19 2009 From: hansel at reactive-systems.com (David Hansel) Date: Mon Nov 30 13:20:11 2009 Subject: [MLton] Crashes with 64-bit native code generator on Windows In-Reply-To: References: <4AFA3919.1090003@reactive-systems.com> <162de7480911161107ld408687gd577838b9bccf868@mail.gmail.com> <4B0B3952.2000607@reactive-systems.com> <4B13EC2A.9070600@reactive-systems.com> <4B1404AB.4040802@reactive-systems.com> Message-ID: <4B1436D7.6010303@reactive-systems.com> Hello Matthew, I tried using "-native-commented 6" but (due to the size of the code involved) compilation (in the "outputAssembly" stage) seems to take a VERY long time. I also tried "-native-commented 5" with the same result. A setting of "4" worked much faster and I have uploaded a file hansel-20091130-1.s containing the basic block. I will try the others again but the "4" setting produced about 1200 .s output files and with the "6" setting MLton produced the first (.0.s) output file and then I stopped it after about an hour of not producing any more output. I will try again but unless the process gets much faster after the second file I don't think we'll get to output file #913 in any reasonable amount of time. Are there ways to restrict the additional output to specific parts of the code? Unfortunately, I can't cut down the code much without the problem going away. David Matthew Fluet wrote: > On Mon, Nov 30, 2009 at 12:45 PM, David Hansel > > wrote: > > See the code below. It should match up with the code I posted before. > From what I can tell it does look like MLton puts the target address for > applyFFTempFun into a register but then later does the indirect call via > the memory location. > > > Yes, it looks like it gets dropped into %r11, but not used from that > location. > > > Please let me know if you need any more context or other debugging > information. It does seem like you are on the right track. > > > We need to find out when the codegen loses track of the fact that %r11 > has applyFFTempFun. Could you compile with "-native-commented 6"? > That's the most debugging information that we can get from a precompiled > MLton binary. It produces a *lot* of debugging information (in the form > of comments in the assembly). Rather than posting to the mailing list, > I suggest posting the basic block to http://mlton.org/TemporaryUpload. > > -Matthew -- ---------------------------------------------------------- David Hansel Chief Technology Officer -- Reactive Systems, Inc. http://www.reactive-systems.com/ (919) 324-3507 ext. 102 -- hansel@reactive-systems.com OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~hansel/pgp_public_key.txt ---------------------------------------------------------- From huabj at mail.ustc.edu.cn Mon Nov 30 17:49:10 2009 From: huabj at mail.ustc.edu.cn (Baojian Hua) Date: Mon Nov 30 17:49:49 2009 Subject: [MLton] On "layout.sig, sml" Message-ID: <6004794.1243651259632150658.JavaMail.coremail@mailweb> hi, We've been useing MLton for over 7 years for our research projects, and recently we're planning to read and hack some source code of MLton (and hopefully we can contribute in the future). And we start by reading sources in "lib/mlton/basic/", but the documentation is rare, so we have to send some questions here: in files "layout.sig, sml", this data structure: datatype t = T of {length: int, tree: tree} and tree = Empty | String of string | Sequence of t list | Align of {force: bool, rows: t list} | Indent of t * int Does the "force" mean we must align several line? And in calculating the length, I'd expect to see this: case tree of Align {rows, ...} => max (rows) (* the longest one *) | _ => but the code is this: case tree of Align {rows, ...} => \Sigma (rows) (* the whole *) | _ => Thanks. -Paul From matthew.fluet at gmail.com Mon Nov 30 18:24:01 2009 From: matthew.fluet at gmail.com (Matthew Fluet) Date: Mon Nov 30 18:24:36 2009 Subject: [MLton] Crashes with 64-bit native code generator on Windows In-Reply-To: <4B1436D7.6010303@reactive-systems.com> References: <4AFA3919.1090003@reactive-systems.com> <162de7480911161107ld408687gd577838b9bccf868@mail.gmail.com> <4B0B3952.2000607@reactive-systems.com> <4B13EC2A.9070600@reactive-systems.com> <4B1404AB.4040802@reactive-systems.com> <4B1436D7.6010303@reactive-systems.com> Message-ID: On Mon, Nov 30, 2009 at 4:19 PM, David Hansel wrote: > I tried using "-native-commented 6" but (due to the size of the code > involved) > compilation (in the "outputAssembly" stage) seems to take a VERY long time. > I also tried "-native-commented 5" with the same result. A setting of "4" > worked much faster and I have uploaded a file hansel-20091130-1.s > containing > the basic block. > That seems to be enough to provide a hint. I think that the issue is that the function address got placed in %r11, which is a caller save register. The contents of caller save registers are pushed to memory immediately before the call instruction, for any register whose content is live after the call and purged from the register allocation. Of course, the function address is still live *at* the call instruction, although it is not live after the call instruction. Small examples seem to favor %r15 as the register into which the function address is placed, which is not caller save, and so not susceptible to this issue. It also fits with small changes near the indirect function call eliminating the segfault; such changes alter the liveness and used registers and presumably the function address get stored in a non-caller save register. If this is indeed the source of the issue, then it is simply a native amd64 codegen bug (and, possibly, a latent x86 codegen bug as well) and is independent of the target OS; that is, it is not mingw specific. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20091130/68e1f273/attachment.html