From duvarney at reactive-systems.com Fri Nov 7 10:34:50 2008 From: duvarney at reactive-systems.com (Dan DuVarney) Date: Fri Nov 7 10:35:12 2008 Subject: [MLton-user] Strange type problem in OS.Process Message-ID: <49148A4A.10305@reactive-systems.com> Greetings all: I am trying to use the MinGW / SVN r6941 version of MLton. I installed it using the .msi installer and created a version of msys which uses the same binaries and libraries provided by the .msi file. Hello world compiles and runs fine. However, MLton insists that type OS.Process.status is not an equality type, which causes many programs to fail to compile. Below I have appended a small program, and the output of MLton --verbose 3. I am doing this on a centrino pro x86 processor running Windows XP professional. Does anyone know if the error I am getting is likely due to some mis-configuration on my part, or is it an actual MLton bug? Before this I tried the 20070826 release, but I kept running into the known bug in which the attempt to page the heap to disk fails. I'd be grateful for any suggestions on what to try next. Thanks in advance, Dan ---- systest.sml ----------------------------------------------------------------- val cmd = "/bin/ls" val pr = TextIO.print fun prLn x = (pr x; pr "\n") val (OS.Process.status) = let val status = OS.Process.system cmd in if status = OS.Process.failure then pr (cmd ^ " failed") else pr (cmd ^ " succeeded"); status end --- output of mlton -verbose 3 systest.sml ---------------------------------------- MLton starting MLton MLTONVERSION (built Thu Oct 16 02:32:53 2008 on orange) created this file on Fri Nov 07 13:27:34 2008. Do not edit this file. Flag settings: align: 4 atMLtons: (systest, @MLton, --) chunk: chunk per function closureConvertGlobalize: true closureConvertShrink: true codegen: x86 contifyIntoMain: false debug: false defaultChar: char8 defaultWideChar: widechar32 defaultInt: int32 defaultReal: real64 defaultWord: word32 diag passes: [] drop passes: [] elaborate allowConstant (default): false elaborate allowConstant (enabled): true elaborate allowFFI (default): false elaborate allowFFI (enabled): true elaborate allowPrim (default): false elaborate allowPrim (enabled): true elaborate allowOverload (default): false elaborate allowOverload (enabled): true elaborate allowRebindEquals (default): false elaborate allowRebindEquals (enabled): true elaborate deadCode (default): false elaborate deadCode (enabled): true elaborate forceUsed (default): false elaborate forceUsed (enabled): true elaborate ffiStr (default): elaborate ffiStr (enabled): true elaborate nonexhaustiveExnMatch (default): default elaborate nonexhaustiveExnMatch (enabled): true elaborate nonexhaustiveMatch (default): warn elaborate nonexhaustiveMatch (enabled): true elaborate redundantMatch (default): warn elaborate redundantMatch (enabled): true elaborate sequenceNonUnit (default): ignore elaborate sequenceNonUnit (enabled): true elaborate warnUnused (default): false elaborate warnUnused (enabled): true elaborate only: false emit main: true export header: None exn history: false generated output format: executable gc check: Limit indentation: 3 inlineIntoMain: true inlineLeafA: {loops = true, repeat = true, size = Some 20} inlineLeafB: {loops = true, repeat = true, size = Some 40} inlineNonRec: {small = 60, product = 320} input file: systest keep CoreML: false keep def use: true keep dot: false keep Machine: false keep passes: [] keep RSSA: false keep SSA: false keep SSA2: false keep SXML: false keep XML: false extra_: true lib dir: c:\msys\1.0\mingw\lib\mlton lib target dir: c:\msys\1.0\mingw\lib\mlton\self loop passes: 1 mark cards: true max function size: 10000 mlb path vars: [{var = MLTON_ROOT, path = $(LIB_MLTON_DIR)/sml}, {var = SML_LIB, path = $(LIB_MLTON_DIR)/sml}] native commented: 0 native live stack: false native optimize: 1 native move hoist: true native copy prop: true native copy prop cutoff: 1000 native cutoff: 100 native live transfer: 8 native shuffle: true native ieee fp: false native split: Some 20000 optimizationPasses: [, , , ] polyvariance: Some {hofo = true, rounds = 2, small = 30, product = 300} prefer abs paths: false prof passes: [] profile: None profile branch: false profile C: [] profile IL: ProfileSource profile include/exclude: [(Seq [Star [.], Or [Seq [Seq [[$], [(], [S], [M], [L], [_], [L], [I], [B], [)]]]], Star [.]], false)] profile raise: false profile stack: false profile val: false show basis: None show def-use: None show types: true target: self target arch: X86 target OS: MinGW type check: false verbosity: Detail warn unrecognized annotation: true zone cut depth: 100 Compile SML starting pre codegen starting parseAndElaborate starting Error: systest.sml 10.17. Function applied to incorrect argument. expects: [] * [] but got: [] * [] in: = (status, OS.Process.failure) Error: systest.sml 6.6. Undefined constructor: OS.Process.status. parseAndElaborate finished in 1.64 + 2.28 (58% GC) parseAndElaborate stats core-ml size = 37,590,060 bytes property list numPeeks = 6133856 property list maxLength = 3 property list average position = 0.264 hash set numPeeks = 87333 hash set average position = 0.973 pre codegen raised in 1.95 + 2.28 (54% GC) Compile SML raised in 1.95 + 2.28 (54% GC) MLton raised in 1.95 + 2.28 (54% GC) compilation aborted: parseAndElaborate reported errors -- Dan DuVarney Senior Software Engineer Reactive Systems Inc. duvarney@reactive-systems.com +1 (919) 324-3507 ext. 103 OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~duvarney/pgp_public_key.txt From wesley at terpstra.ca Fri Nov 7 10:41:42 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Fri Nov 7 10:41:46 2008 Subject: [MLton-user] Strange type problem in OS.Process In-Reply-To: <49148A4A.10305@reactive-systems.com> References: <49148A4A.10305@reactive-systems.com> Message-ID: <162de7480811071041q3f7a7edbwe8ec93702c1633b5@mail.gmail.com> On Fri, Nov 7, 2008 at 7:34 PM, Dan DuVarney wrote: > I am trying to use the MinGW / SVN r6941 version of MLton. > I installed it using the .msi installer and created a version of msys which > uses the same binaries and libraries provided by the .msi file. I wouldn't advise replacing the binaries and libraries the msi installed unless you have a good reason. > Hello world compiles and runs fine. However, MLton insists that type > OS.Process.status is not an equality type, which causes many programs > to fail to compile. http://mlton.org/basis/os-process.html lists it as 'type status', not an equality 'eqtype'. So I'd say this is a bug in your program, not MLton. Perhaps you want to use Posix.Process.fromStatus ? From wesley at terpstra.ca Fri Nov 7 11:20:40 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Fri Nov 7 11:20:43 2008 Subject: [MLton-user] Strange type problem in OS.Process In-Reply-To: <49148A4A.10305@reactive-systems.com> References: <49148A4A.10305@reactive-systems.com> Message-ID: <162de7480811071120lc704210nec5d3b56fb75f061@mail.gmail.com> A couple more quick bits to add: > val cmd = "/bin/ls" /bin/ls will fail on MinGW. I am guessing you want c:/path/to/msys/bin/ls. What you see as /bin doesn't exist for windows programs (like the output of MLton). '/' is remapped by bash when you execute things from the MinGW shell. Your string constant above doesn't get run by bash, but by cmd.exe. > if status = OS.Process.failure then If this is really all you want to do, "if OS.Process.isSuccess status ..." is your best bet. If you really do want to compare status values in your larger program, then Posix.Process.fromStatus will get you there. From fluet at tti-c.org Fri Nov 7 11:18:47 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Fri Nov 7 11:21:23 2008 Subject: [MLton-user] Strange type problem in OS.Process In-Reply-To: <162de7480811071041q3f7a7edbwe8ec93702c1633b5@mail.gmail.com> References: <49148A4A.10305@reactive-systems.com> <162de7480811071041q3f7a7edbwe8ec93702c1633b5@mail.gmail.com> Message-ID: On Fri, 7 Nov 2008, Wesley W. Terpstra wrote: > On Fri, Nov 7, 2008 at 7:34 PM, Dan DuVarney >> Hello world compiles and runs fine. However, MLton insists that type >> OS.Process.status is not an equality type, which causes many programs >> to fail to compile. > > http://mlton.org/basis/os-process.html lists it as 'type status', not > an equality 'eqtype'. So I'd say this is a bug in your program, not > MLton. Perhaps you want to use Posix.Process.fromStatus ? Correct -- the Basis Library specifies that OS.Process.status is an abstract, non-equality type. Prior to SVN r6425 (20080302), MLton's implementation of the Basis Library erroneously exposed OS.Process.status as an equality type. You can use OS.Process.isSuccess for a simple test. From duvarney at reactive-systems.com Fri Nov 7 11:20:43 2008 From: duvarney at reactive-systems.com (Dan DuVarney) Date: Fri Nov 7 11:24:16 2008 Subject: [MLton-user] Strange type problem in OS.Process In-Reply-To: <162de7480811071041q3f7a7edbwe8ec93702c1633b5@mail.gmail.com> References: <49148A4A.10305@reactive-systems.com> <162de7480811071041q3f7a7edbwe8ec93702c1633b5@mail.gmail.com> Message-ID: <4914950B.2060805@reactive-systems.com> Yes, you are correct. The problem originated in some old code and I incorrectly assumed that it was legal code. This appears to be difference between SML/NJ and MLton. In SML/NJ 100.65, OS.Process.status is an equality type. Thanks for the help. -Dan Wesley W. Terpstra wrote: > On Fri, Nov 7, 2008 at 7:34 PM, Dan DuVarney > wrote: > >> Hello world compiles and runs fine. However, MLton insists that type >> OS.Process.status is not an equality type, which causes many programs >> to fail to compile. >> > > http://mlton.org/basis/os-process.html lists it as 'type status', not > an equality 'eqtype'. So I'd say this is a bug in your program, not > MLton. Perhaps you want to use Posix.Process.fromStatus ? > -- Dan DuVarney Senior Software Engineer Reactive Systems Inc. duvarney@reactive-systems.com +1 (919) 324-3507 ext. 103 OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~duvarney/pgp_public_key.txt From duvarney at reactive-systems.com Fri Nov 7 13:07:13 2008 From: duvarney at reactive-systems.com (Dan DuVarney) Date: Fri Nov 7 13:08:58 2008 Subject: [MLton-user] Strange type problem in OS.Process In-Reply-To: <162de7480811071120lc704210nec5d3b56fb75f061@mail.gmail.com> References: <49148A4A.10305@reactive-systems.com> <162de7480811071120lc704210nec5d3b56fb75f061@mail.gmail.com> Message-ID: <4914AE01.1080509@reactive-systems.com> Wesley W. Terpstra wrote: > A couple more quick bits to add: > >> val cmd = "/bin/ls" >> > > /bin/ls will fail on MinGW. I am guessing you want > c:/path/to/msys/bin/ls. What you see as /bin doesn't exist for windows > programs (like the output of MLton). '/' is remapped by bash when you > execute things from the MinGW shell. Your string constant above > doesn't get run by bash, but by cmd.exe. > Thanks for the advice. The code was actually not something I wanted to do, it was just intended to reproduce the error. I guess it's a good thing I never got to actually execute the program :-) >> if status = OS.Process.failure then >> > > If this is really all you want to do, "if OS.Process.isSuccess status > ..." is your best bet. If you really do want to compare status values > in your larger program, then Posix.Process.fromStatus will get you > there. > The main thing I want to do is maintain portability of a large pre-existing code base between SML/NJ and MLton. Since the SML/NJ basis library we are using does not provide OS.Process.isSuccess, it looks to me like a fairly unconvoluted way of portably checking exit status is (Posix.Process.fromStatus x) <> (Posix.Process.fromStatus OS.Process.success) Thanks again for the help. Dan -- Dan DuVarney Senior Software Engineer Reactive Systems Inc. duvarney@reactive-systems.com +1 (919) 324-3507 ext. 103 OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~duvarney/pgp_public_key.txt From vesa.a.j.k at gmail.com Sun Nov 9 03:40:20 2008 From: vesa.a.j.k at gmail.com (Vesa Karvonen) Date: Sun Nov 9 03:40:23 2008 Subject: [MLton-user] Strange type problem in OS.Process In-Reply-To: <4914AE01.1080509@reactive-systems.com> References: <49148A4A.10305@reactive-systems.com> <162de7480811071120lc704210nec5d3b56fb75f061@mail.gmail.com> <4914AE01.1080509@reactive-systems.com> Message-ID: <9e43b9a0811090340r43615b1bm8022368266d83bf@mail.gmail.com> On Fri, Nov 7, 2008 at 11:07 PM, Dan DuVarney wrote: [...] > The main thing I want to do is maintain portability of a large > pre-existing code base between SML/NJ and MLton. Since the > SML/NJ basis library we are using does not provide > OS.Process.isSuccess, it looks to me like a fairly unconvoluted > way of portably checking exit status is > > (Posix.Process.fromStatus x) <> (Posix.Process.fromStatus > OS.Process.success) Which version of SML/NJ are you actually using? Digging a bit in SML/NJ's repository, it would seem that OS.Process.isSuccess was added to SML/NJ's Basis library in late 2003: http://smlnj-gforge.cs.uchicago.edu/scm/viewvc.php/sml/trunk/system/Basis/Implementation/OS/os-process.sig?root=smlnj&r1=651&r2=1426 Rather than working around issues that were fixed literally 5 years ago, I would rather recommend porting your code to the latest SML/NJ, which does provide OS.Process.isSuccess. -Vesa Karvonen From duvarney at reactive-systems.com Mon Nov 10 16:43:50 2008 From: duvarney at reactive-systems.com (Dan DuVarney) Date: Mon Nov 10 16:44:16 2008 Subject: [MLton-user] SVN r6941 MLton/MinGW32 and FFI Message-ID: <4918D546.1040407@reactive-systems.com> Greetings: Has anyone else had problems with using the C FFI with SVN r6941 MLton under MinGW32? I have been unsuccessful in getting any program containing FFI calls to work. It looks as if a step which generates a dll wrapper is being skipped. For example, in /doc/examples/ffi I am observing the following behavior: bash-3.1$ make test_quot mlton -default-ann 'allowFFI true' test_quot.sml c_quot.o C:\DOCUME~1\Dan\LOCALS~1\Temp\fileT2zfil.o:fake:(.text+0x2fbe): undefined reference to `__imp__c_quot' C:\DOCUME~1\Dan\LOCALS~1\Temp\fileT2zfil.o:fake:(.text+0x35f0): undefined reference to `__imp__call_sml_quot' collect2: ld returned 1 exit status call to system failed with signal 1: gcc -o test_quot C:\DOCUME~1\Dan\LOCALS~1\Temp\filekfJCQj.o C:\DOCUME~1\Dan\LOCALS~1\Temp\fileT2zfil.o c_quot.o -Lc:\program files\mlton\lib\mlton\self -lmlton -lgdtoa -lm -lgmp -lws2_32 -lkernel32 -lpsapi -lnetapi32 -lwinmm -Wl,--enable-stdcall-fixup -- Dan DuVarney Senior Software Engineer Reactive Systems Inc. duvarney@reactive-systems.com +1 (919) 324-3507 ext. 103 OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~duvarney/pgp_public_key.txt From wesley at terpstra.ca Tue Nov 11 01:00:31 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Nov 11 01:00:35 2008 Subject: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: <4918D546.1040407@reactive-systems.com> References: <4918D546.1040407@reactive-systems.com> Message-ID: <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> On Tue, Nov 11, 2008 at 1:43 AM, Dan DuVarney wrote: > For example, in /doc/examples/ffi I am observing the following behavior: > > bash-3.1$ make test_quot > mlton -default-ann 'allowFFI true' test_quot.sml c_quot.o > C:\DOCUME~1\Dan\LOCALS~1\Temp\fileT2zfil.o:fake:(.text+0x2fbe): > undefined reference to `__imp__c_quot' > C:\DOCUME~1\Dan\LOCALS~1\Temp\fileT2zfil.o:fake:(.text+0x35f0): > undefined reference to `__imp__call_sml_quot' I guess that you _export the function call_sml_quot, and then import it back? Or that call_sml_quot comes from a library you are linking with statically (not dynamically via a DLL)? See http://mlton.org/LibrarySupport , suggestions to improve the documentation are welcome. The short answer: wherever you are using _import "call_sml_quot" you probably need _import "call_sml_quot" private or public. Which depends on what you're doing. From fluet at tti-c.org Tue Nov 11 05:36:05 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Tue Nov 11 05:38:56 2008 Subject: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> Message-ID: On Tue, 11 Nov 2008, Wesley W. Terpstra wrote: > On Tue, Nov 11, 2008 at 1:43 AM, Dan DuVarney > wrote: >> For example, in /doc/examples/ffi I am observing the following behavior: >> >> bash-3.1$ make test_quot >> mlton -default-ann 'allowFFI true' test_quot.sml c_quot.o >> C:\DOCUME~1\Dan\LOCALS~1\Temp\fileT2zfil.o:fake:(.text+0x2fbe): >> undefined reference to `__imp__c_quot' >> C:\DOCUME~1\Dan\LOCALS~1\Temp\fileT2zfil.o:fake:(.text+0x35f0): >> undefined reference to `__imp__call_sml_quot' > > I guess that you _export the function call_sml_quot, and then import > it back? Or that call_sml_quot comes from a library you are linking > with statically (not dynamically via a DLL)? It is one of the examples from /doc/examples/ffi. These examples really should work 'out of the box'. > See http://mlton.org/LibrarySupport , suggestions to improve the > documentation are welcome. If the symbol scopes can't be given sensible defaults for executables that match the behavior of previous versions, then they really need to be documented as well at: http://mlton.org/ForeignFunctionInterfaceSyntax From duvarney at reactive-systems.com Tue Nov 11 06:41:43 2008 From: duvarney at reactive-systems.com (Dan DuVarney) Date: Tue Nov 11 06:42:10 2008 Subject: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> Message-ID: <491999A7.6080101@reactive-systems.com> Thanks for the pointer to the LibrarySupport documenation page. It was quite useful. I discovered the problem with the distributed ffi examples. They are missing symbol scopes which cause link-time errors on MinGW32. It looks like the default scope is "external." When I added "public" scopes to import.sml (see below), I was able to get "import" to compile and run. Someone might want to check to see if this is a problem on other OSes, as I don't have access to a linux machine. If I can get SVN access, I'd be happy to check-in the updated example code (although I'm unsure of the effect of changing the scope to "public" on other OSes). One question occurs to me. Is there a compiler option to change the default scope of foreign functions and data? That could also be used to get this example to work. -Dan --------------------------------- import.sml ----------------------------------------- (* main.sml *) (* Declare ffi to be implemented by calling the C function ffi. *) val ffi = _import "ffi" public: real array * int ref * char ref * int -> char; open Array val size = 10 val a = tabulate (size, fn i => real i) val ri = ref 0 val rc = ref #"0" val n = 17 (* Call the C function *) val c = ffi (a, ri, rc, n) val (nGet, nSet) = _symbol "FFI_INT" public: (unit -> int) * (int -> unit); val _ = print (concat [Int.toString (nGet ()), "\n"]) val _ = print (if c = #"c" andalso !ri = 45 andalso !rc = c then "success\n" else "fail\n") -- Dan DuVarney Senior Software Engineer Reactive Systems Inc. duvarney@reactive-systems.com +1 (919) 324-3507 ext. 103 OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~duvarney/pgp_public_key.txt From wesley at terpstra.ca Tue Nov 11 07:15:26 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Nov 11 07:15:31 2008 Subject: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> Message-ID: <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> On Tue, Nov 11, 2008 at 2:36 PM, Matthew Fluet wrote: > It is one of the examples from /doc/examples/ffi. > These examples really should work 'out of the box'. I didn't know about these files. I've started updating them to use symbol scopes. I've noticed a couple of things: 1. ffi-import.c uses GC_getArrayLength from "platform.h"! Is this really supposed to be user-accessible? If yes, should it be a private symbol only accessible within the DSO, a public symbol (in which case it needs a libname_ prefix), or both? 2. iimport.c uses dlopen/dlclose/dlsym. platform/mingw.c defines these... which is a problem since the runtime is statically linked, but libdl is dynamically linked. Perhaps these should be public libname_ symbols, private MLton_dlopen/close/sym, or both? What would make most sense to me: * MLton_{dlopen,dlclose,dlsym,getArrayLength} are private symbols available within the DSO * GC_getArrayLength is also a private alias for compatibility * libname_getArrayLength are public symbols available to users of a DSO. Perhaps the dl* symbols too. >> See http://mlton.org/LibrarySupport , suggestions to improve the >> documentation are welcome. > > If the symbol scopes can't be given sensible defaults for executables that > match the behavior of previous versions, then they really need to be > documented as well at: > http://mlton.org/ForeignFunctionInterfaceSyntax I agree. However, that page corresponds to the release version of MLton, which doesn't have symbol scopes. I'm also beginning to wonder if perhaps imports should default to public instead of external given the number of people who've run into the __imp__ link error. The plus side is that public would make anything that worked before continue to work (MinGW stubs would see to that). The downside is that any cases that silently failed before will continue to do so. AFAIK, the only silently failing case is: you get a different _address that doesn't compare to equality with what other parts of the program see for functions. _import'ing a variable as public from a DLL will just flat-out fail at link time, so it's not silent. From wesley at terpstra.ca Tue Nov 11 07:28:32 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Nov 11 07:28:36 2008 Subject: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: <491999A7.6080101@reactive-systems.com> References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <491999A7.6080101@reactive-systems.com> Message-ID: <162de7480811110728n6fd155f2h2eb22f5d69952069@mail.gmail.com> On Tue, Nov 11, 2008 at 3:41 PM, Dan DuVarney wrote: > It looks like the default scope is "external." When I added > "public" scopes to import.sml (see below), I was able to get "import" to > compile and run. Someone might want to check to see if this is a problem > on other OSes, as I don't have access to a linux machine. It will work there too. external=public on linux. > One question occurs to me. Is there a compiler option to change the > default scope of foreign functions and data? That could also be used to > get this example to work. There is no compiler switch. As a program might import both public and external symbols, changing the default for the entire program might not work as expected. From ville at laurikari.net Tue Nov 11 09:08:01 2008 From: ville at laurikari.net (Ville Laurikari) Date: Tue Nov 11 09:08:06 2008 Subject: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> Message-ID: <20081111170801.GB13363@laurikari.net> On Tue, Nov 11, 2008 at 04:15:26PM +0100, Wesley W. Terpstra wrote: > I'm also beginning to wonder if perhaps imports should default to > public instead of external given the number of people who've run into > the __imp__ link error. I think this is a good idea. We've certainly run into the problem. Upgrading to a new version of MLton is problematic, because code updated for the new version does not work with the old version. On the other hand, code working with the old version does not work with the new version. This means that as soon as we commit code which uses explicit symbol scoping, builds start failing an all hosts which don't have an updated MLton version yet (quite a few). If there is any way to keep MLton backwards compatible, it would help a lot as people would be able to upgrade when they feel like it. By the way, it occured to me if nlffigen should be modified in some way. The trunk version does not include an explicit scope when it generates code that uses _address. Will this be a problem? I suppose importing variables may not work in some circumstances (we import only functions at the moment...) -- http://www.iki.fi/vl/ From wesley at terpstra.ca Tue Nov 11 10:00:31 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Nov 11 10:00:35 2008 Subject: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: <20081111170801.GB13363@laurikari.net> References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> <20081111170801.GB13363@laurikari.net> Message-ID: <162de7480811111000k5c2fb26do762eef289fbe4338@mail.gmail.com> On Tue, Nov 11, 2008 at 6:08 PM, Ville Laurikari wrote: > On Tue, Nov 11, 2008 at 04:15:26PM +0100, Wesley W. Terpstra wrote: >> I'm also beginning to wonder if perhaps imports should default to >> public instead of external given the number of people who've run into >> the __imp__ link error. > > I think this is a good idea. We've certainly run into the problem. > Upgrading to a new version of MLton is problematic, because code > updated for the new version does not work with the old version. On > the other hand, code working with the old version does not work with > the new version. I feel your pain, but changing the default to public won't completely alleviate this. You *should* specify external for any symbols you import from a DLL if the default is public. Of course, if you only ever use _import (never _symbol or _address), you should be safe. The other problem case is osx/i386. It needs special stub code to access external functions and variables from a library. If MLton defaults to public, then any programs on that target which _import from a system library (functions like cos, sin, etc) will fail to link unless "external" is specified. Thus picking 'public' as the default would make a smooth upgrade for MinGW, but break darwin/i386. Currently it's a smooth upgrade for darwin, but not MinGW. For both situations and targets, though, you really should distinguish public/external. One could imagine a default of public for MinGW and external for darwin, but that seems pretty arcane. > By the way, it occured to me if nlffigen should be modified in some > way. The trunk version does not include an explicit scope when it > generates code that uses _address. Will this be a problem? It depends, ANSI C requires that all pointers to the same function compare for equality. Taking _address of an external function as though it were public will violate this. However, it does seem the uncommon program which depends on this, and I imagine gcc on MinGW must expect programmers to get this wrong and thus does not make bad optimization decisions because of it. Finally, _import is 100% "safe" since you never see the address and MinGW makes automagic stub functions to proxy the call to __imp__fname. At any rate, it's hard to imagine how mlnlffigen could make a sane choice. It doesn't know if you are statically or dynamically linking to the library which corresponds to the header file it processes. At the moment it picks nothing, aka external, which makes sense if you are importing a dynamic library. > I suppose importing variables may not work in some circumstances (we > import only functions at the moment...) I guessed as much, or the old MLton wouldn't work for you. =) At any rate, the attached patch would change the default to public. -------------- next part -------------- A non-text attachment was scrubbed... Name: default-public.patch Type: text/x-patch Size: 1645 bytes Desc: not available Url : http://mlton.org/pipermail/mlton-user/attachments/20081111/72227f36/default-public.bin From fluet at tti-c.org Tue Nov 11 10:04:43 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Tue Nov 11 10:07:37 2008 Subject: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> Message-ID: On Tue, 11 Nov 2008, Wesley W. Terpstra wrote: > On Tue, Nov 11, 2008 at 2:36 PM, Matthew Fluet wrote: >> It is one of the examples from /doc/examples/ffi. >> These examples really should work 'out of the box'. > > I didn't know about these files. I've started updating them to use > symbol scopes. I've noticed a couple of things: > > 1. ffi-import.c uses GC_getArrayLength from "platform.h"! Is this > really supposed to be user-accessible? If yes, should it be a private > symbol only accessible within the DSO, a public symbol (in which case > it needs a libname_ prefix), or both? It is a function defined in libmlton.a, so I would guess that it should be a private symbol, only accessible within the DSO. (Just like all the other runtime functions.) For libraries, I think the current setup is good -- the only visible symbols are those explicitly 'public' exported (and the auto-generated 'libname_open' and 'libname_close' functions). With regards to whether users should actually use GC_getArrayLength, we don't really advertise it, but neither is it a particularly subtle operation. And, it is genuinely useful (though, getting the length and passing it through the FFI along with the array pointer wouldn't been terribly difficult either). I don't think there is any way to really distinguish between visible functions in libmlton.a that are available for use in mlton generated .c and .s files, but not available for user .c code. > 2. iimport.c uses dlopen/dlclose/dlsym. platform/mingw.c defines > these... which is a problem since the runtime is statically linked, > but libdl is dynamically linked. Perhaps these should be public > libname_ symbols, private MLton_dlopen/close/sym, or both? To clarify: the issue is that on *NIX platforms (including MacOSX), dl{open,close,sym,error,...} are pulled from dynamically linked shared libraries. That is, they are imported from another DSO. Thus, for these platforms, the 'external' attribute should be used on the _import. MinGW doesn't provide dl{open,close,...} in a (standard) dynamically linked shared library. Vesa added rudimentary emulation in /runtime/platform/mingw.c (r4862, 20061127); thus dl{open,close,...} are pulled from libmlton.a. That is, they are imported from the same DSO. Thus, for this platform, the 'private' attribute should be used on the _import. Personally, I think that MLton should not be attempting to provide non-essential functionality that is missing on a platform. My suspicion is that 99% of all MLton compiled SML programs will not need dl{open,close,...} support. Of the remaining 1%, most will probably be on *NIX platforms, where dl{open,close,...} are provided by standard system libraries. (You can see from the /doc/examples/ffi/Makefile iimport target that some *NIX platforms require extra link options to find the library; MLton doesn't default to including those link options, because they are not essential functionality (e.g., required by some of the Basis Library implementation).) That leaves a very small number of programs that are on Windows (via MinGW, since, apparently, Cygwin provides dl{open,close,...}). If they are MinGW exclusive, they can use the Win32 specific {Load,Free}Library/GetProcAddress functions. The remaining (infinitesmal) programs are ones that require dlopen-ing libraries and are trying to stay source compatible for *NIX and Windows/MinGW. For those situations, there are more robust solutions than MLton emulating dl{open,close,...} for MinGW. One is to use MLB path-variables to provide a compatibility module: dynlink.mlb: local dyn-link.sig platform/dyn-link.$(TARGET_ARCH)-$(TARGET_OS).mlb in signature DYN_LINK structure DynLink end platform/dynlink.x86-linux.mlb: dynlink.dlfcn.mlb platform/dyn-link.dlfcn.mlb: dyn-link.dlfnc.sml platform/dyn-link.dlfcn.sml: structure DynLink :> DYN_LINK = struct type hndl = MLton.Pointer.t type mode = Word32.word val dlopen = _import "dlopen" external : string * mode -> hndl; ... end platform/dynlink.x86-mingw.mlb: dynlink.win32.mlb platform/dynlink.win32.mlb: dynlink.win32.sml platform/dynlink.win32.sml: structure DynLink :> DYN_LINK = struct type hndl = MLton.Pointer.t val loadLibrary = _import "LoadLibrary" stdcall external : string -> hndl; ... end Alternatively, use the GNU libtool libtldl library, whose express purpose is to provide a uniform interface to various system specific dynamic library mechanisms. Assuming that libtldl is available as a dynamically linked shared library (i.e., a different DSO than the program) on all the platforms of interest, then its functions may all be _import-ed with 'external' scope. > What would make most sense to me: > * MLton_{dlopen,dlclose,dlsym,getArrayLength} are private symbols > available within the DSO > * GC_getArrayLength is also a private alias for compatibility > * libname_getArrayLength are public symbols available to users of a > DSO. Perhaps the dl* symbols too. I (respectfully) disagree on all counts ;-) - MLton shouldn't be providing dl{open,close,...} in any form. - GC_getArrayLength isn't advertised, so there is no need to 'bless' it with a MLton_getArrayLength name. (We should eliminate its use from the /doc/examples/ffi examples.) - No functions should be public symbols available to users of a DSO other than those specified by the user. If a function needs the length of an array, arrange the interface to pass the length along with the array pointer. >>> See http://mlton.org/LibrarySupport , suggestions to improve the >>> documentation are welcome. >> >> If the symbol scopes can't be given sensible defaults for executables that >> match the behavior of previous versions, then they really need to be >> documented as well at: >> http://mlton.org/ForeignFunctionInterfaceSyntax > > I agree. However, that page corresponds to the release version of > MLton, which doesn't have symbol scopes. It might be a good idea to clone the page (say, as NextRelease/ForeignFunctionInterfaceSyntax) to make current documentation available somewhere. Also, it makes it easier to switch in when the time comes. From wesley at terpstra.ca Tue Nov 11 12:09:14 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Nov 11 12:09:21 2008 Subject: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> Message-ID: <162de7480811111209x2549aac6y3bf403b14c088c17@mail.gmail.com> On Tue, Nov 11, 2008 at 7:04 PM, Matthew Fluet wrote: > With regards to whether users should actually use GC_getArrayLength, we > don't really advertise it, but neither is it a particularly subtle > operation. And, it is genuinely useful (though, getting the length and > passing it through the FFI along with the array pointer wouldn't been > terribly difficult either). I don't think there is any way to really > distinguish between visible functions in libmlton.a that are available for > use in mlton generated .c and .s files, but not available for user .c code. Well, I personally have always passed the length of arrays with the array through the FFI. This is the usual way C functions work in my experience. I was genuinely surprised to see GC_arrayLength used as I thought GC_* was supposed to be MLton internal. There are some damn scary functions available with the GC_ prefix! It sounds from your later comments that you agree with me, so let's just phase GC_arrayLength out of the examples and FFI documentation. It still works within a DSO since it is a PRIVATE symbol at the moment, so there's no backwards compatibility problem. >> 2. iimport.c uses dlopen/dlclose/dlsym. platform/mingw.c defines >> these... which is a problem since the runtime is statically linked, >> but libdl is dynamically linked. Perhaps these should be public >> libname_ symbols, private MLton_dlopen/close/sym, or both? > > To clarify: the issue is that on *NIX platforms (including MacOSX), > dl{open,close,sym,error,...} are pulled from dynamically linked shared > libraries. That is, they are imported from another DSO. Thus, for these > platforms, the 'external' attribute should be used on the _import. > > MinGW doesn't provide dl{open,close,...} in a (standard) dynamically linked > shared library. Vesa added rudimentary emulation in > /runtime/platform/mingw.c (r4862, 20061127); thus dl{open,close,...} > are pulled from libmlton.a. That is, they are imported from the same DSO. > Thus, for this platform, the 'private' attribute should be used on the > _import. Yes, that's the problem (though they are public in mingw.c, not private). > Personally, I think that MLton should not be attempting to provide > non-essential functionality that is missing on a platform. I agree, with the added provision that if MLton needs these functions itself it implement them, but not export them. A good example being the mingw.{ch} which adds missing functionality as MLton_* (b/c of #define's in mingw.h). Several existing files in platform/*.c don't do this, which IMO is bad (aix.c, float-math.c, hpux.c, mkdir2.c, setenv.putenv.c, and solaris.c). > MLton doesn't > default to including those link options, because they are not essential > functionality (e.g., required by some of the Basis Library implementation).) So why were the dl* functions added to mingw.c in the first place? The only reason I suggested keeping them (and the GC_arrayLength stuff) around was because I thought there was a reason for them that I didn't know. > - MLton shouldn't be providing dl{open,close,...} in any form. If the basis doesn't use them, then I agree and further suggest they be removed from mingw.c. > - GC_getArrayLength isn't advertised, so there is no need to 'bless' it > with a MLton_getArrayLength name. (We should eliminate its use from the > /doc/examples/ffi examples.) Be aware that even the wiki advertises this function at the momeny. Thus, I thought this was (shockingly) supposed to be part of the documented FFI. Phasing it out seems a great solution to me. > It might be a good idea to clone the page (say, as > NextRelease/ForeignFunctionInterfaceSyntax) to make current documentation > available somewhere. Sounds like a plan. Regarding public/private, I can no longer recommend changing the default to public. I recall my reasoning now: external is the safe default on all platforms except MinGW. On all these other platforms it works, and works correctly. If public were the default, it might be more convenient for MinGW, but would in some corner cases break (variables and pointers). Setting the default to external makes it absolutely clear that the code is in error (and dangerous) on MinGW as it fails loudly with a link error. From duvarney at reactive-systems.com Tue Nov 11 12:40:45 2008 From: duvarney at reactive-systems.com (Dan DuVarney) Date: Tue Nov 11 12:40:42 2008 Subject: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> Message-ID: <4919EDCD.6080901@reactive-systems.com> Some good news: After some work, I've been able to get the "iimport" example (which uses makes FFI calls to "dlopen") to work. The steps required were: 1. I downloaded and installed a dlfcn library for MinGW from Google code. 2. Added "public" keywords to the _import directives in iimport.sml. 3. Add to Makefile: -target-lib-opt mingw -ldl 4. Since we didn't have a libm.dll, created a bogus one for testing purposes. 5. After much pain, I discovered and fixed a bug in the dlfcn library. The implementation of dlerror() was returning a pointer to a null character on a clean exit instead of a null pointer. After these steps, the "iimport" example works correctly (although my "cos" function returns x/2 instead of cos(x)). I am now updating linkage-libdl.sml. Hopefully we will be up and running with MLton once again. -Dan -- Dan DuVarney Senior Software Engineer Reactive Systems Inc. duvarney@reactive-systems.com +1 (919) 324-3507 ext. 103 OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~duvarney/pgp_public_key.txt From wesley at terpstra.ca Tue Nov 11 13:11:05 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Nov 11 13:11:09 2008 Subject: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: <4919EDCD.6080901@reactive-systems.com> References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> <4919EDCD.6080901@reactive-systems.com> Message-ID: <162de7480811111311j23663bfaufe8186138eeb467d@mail.gmail.com> On Tue, Nov 11, 2008 at 9:40 PM, Dan DuVarney wrote: > 1. I downloaded and installed a dlfcn library for MinGW from Google code. I would suggest you download the dlfcn-win32-shared-r8.tar.bz2 version. As it's shared, you won't need to change the _import from external to public; external will be correct. > 5. After much pain, I discovered and fixed a bug in the dlfcn library. The > implementation of dlerror() was returning a pointer to a null > character on a > clean exit instead of a null pointer. Please forward your fix to the google folk so that it is included in their future releases. > I am now updating linkage-libdl.sml. Hopefully we will be up and running > with MLton once again. Hopefully, with the above two items done, linkage-libdl.sml doesn't need modification. From fluet at tti-c.org Tue Nov 11 16:14:16 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Tue Nov 11 16:17:14 2008 Subject: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: <20081111170801.GB13363@laurikari.net> References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> <20081111170801.GB13363@laurikari.net> Message-ID: On Tue, 11 Nov 2008, Ville Laurikari wrote: > By the way, it occured to me if nlffigen should be modified in some > way. The trunk version does not include an explicit scope when it > generates code that uses _address. Will this be a problem? I > suppose importing variables may not work in some circumstances (we > import only functions at the moment...) Updating mlnlffigen would be fairly trivial, though I think we need to change one of the command line options. Currently, there is a command line option '-linkage {dynamic|static}'. With '-linkage dynamic', mlnlffigen generates code to get the address of a symbol like: val h = LibH.libh "sinh" where LibH.libh : string -> unit -> CMemory.addr; typically, one would implement LibH.libh using the DynLinkage module provided by the mlnlffi library (which in turn used dl{open,close,...} or the platform equivalent). With '-linkage static', mlnlffigen generates code to get the address of a symbol like: fun h () = _address "sinh" : CMemory.addr; The use of the term 'static' is a bit ambiguous --- in the absence of symbol scopes, this is the code we would use whether we were pulling sinh from libm.a (i.e., link against an archive) or from libm.so (i.e., link against shared object). As I understand it, if the symbol is coming from an archive, then 'public' or 'private' is the correct scope (but, I don't understand the rules for using one or the other); if the symbol is coming from a shared object, then 'external' is the correct scope. So, the '-linkage' option should probably be refined to {dynamic|archive|shared} to distinguish between using the dynamic linking loader (with dynamic), using 'public' (with archive), and using 'external' (with shared). From wesley at terpstra.ca Tue Nov 11 16:41:40 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Tue Nov 11 16:41:44 2008 Subject: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> <20081111170801.GB13363@laurikari.net> Message-ID: <162de7480811111641w2eb358afq8741ba83ba6c1061@mail.gmail.com> On Wed, Nov 12, 2008 at 1:14 AM, Matthew Fluet wrote: > As I understand it, if the symbol is coming from an > archive, then 'public' or 'private' is the correct scope (but, I don't > understand the rules for using one or the other); if the symbol is coming > from a shared object, then 'external' is the correct scope. Within a DSO symbols exported as private should be imported as private. Public as public. Imports and exports need to agree within the DSO, that's all. I suppose you could have an archive of private symbols (in fact, libmlton.a is such an archive), then _import private would be appropriate. This is not the common case, though. Your typical scenario for mlnlffi -linkage archive as I understand it is a library that's packaged as an archive. It's public interface (in the header) is almost surely public in the symbol table too. > So, the '-linkage' option should probably be refined to > {dynamic|archive|shared} to distinguish between using the dynamic linking > loader (with dynamic), using 'public' (with archive), and using 'external' > (with shared). Sounds convenient. From duvarney at reactive-systems.com Tue Nov 11 20:34:16 2008 From: duvarney at reactive-systems.com (Dan DuVarney) Date: Tue Nov 11 20:34:31 2008 Subject: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: <162de7480811111311j23663bfaufe8186138eeb467d@mail.gmail.com> References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> <4919EDCD.6080901@reactive-systems.com> <162de7480811111311j23663bfaufe8186138eeb467d@mail.gmail.com> Message-ID: <491A5CC8.2070507@reactive-systems.com> Wesley W. Terpstra wrote: > On Tue, Nov 11, 2008 at 9:40 PM, Dan DuVarney > wrote: > >> 1. I downloaded and installed a dlfcn library for MinGW from Google code. >> > > I would suggest you download the dlfcn-win32-shared-r8.tar.bz2 > version. As it's shared, you won't need to change the _import from > external to public; external will be correct. > Good point. I built a shared version of libdll, and the code now works without the need to add any "public" keywords. No changes are required to linkage-libdl.sml, and only a "MinGW" case needs to be added to the dll filename expression in the iimport.sml example. -Dan -- Dan DuVarney Senior Software Engineer Reactive Systems Inc. duvarney@reactive-systems.com +1 (919) 324-3507 ext. 103 OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~duvarney/pgp_public_key.txt From ville at laurikari.net Tue Nov 11 22:09:14 2008 From: ville at laurikari.net (Ville Laurikari) Date: Tue Nov 11 22:09:21 2008 Subject: [MLton-user] SVN r6941 MLton/MinGW32 and FFI In-Reply-To: <162de7480811111209x2549aac6y3bf403b14c088c17@mail.gmail.com> References: <4918D546.1040407@reactive-systems.com> <162de7480811110100k81964a5mf7c99d24cf8ac368@mail.gmail.com> <162de7480811110715q93d4904kfa50e4e780eddfad@mail.gmail.com> <162de7480811111209x2549aac6y3bf403b14c088c17@mail.gmail.com> Message-ID: <20081112060914.GA919@laurikari.net> > itself it implement them, but not export them. A good example being > the mingw.{ch} which adds missing functionality as MLton_* (b/c of > #define's in mingw.h). Several existing files in platform/*.c don't do > this, which IMO is bad (aix.c, float-math.c, hpux.c, mkdir2.c, > setenv.putenv.c, and solaris.c). I'll fix at least aix.h and hpux.h, in some schedule, to #define prefixed versions. > Regarding public/private, I can no longer recommend changing the > default to public. I recall my reasoning now: external is the safe > default on all platforms except MinGW. On all these other platforms it > works, and works correctly. If public were the default, it might be > more convenient for MinGW, but would in some corner cases break > (variables and pointers). Setting the default to external makes it > absolutely clear that the code is in error (and dangerous) on MinGW as > it fails loudly with a link error. I understand. At least in our case, backwards compatibility wins. We don't use the corner cases, and are not planning to start using them. We'll probably then need to use a custom version of MLton (with default as public on MinWG) on MinWG for a period of time until we've had a chance to update all live code. We mostly build our own MLton binaries anyway, so it's not a huge problem to do this. -- http://www.iki.fi/vl/ From duvarney at reactive-systems.com Thu Nov 13 08:59:48 2008 From: duvarney at reactive-systems.com (Dan DuVarney) Date: Thu Nov 13 09:00:06 2008 Subject: [MLton-user] MLton OS.Path under Windows Message-ID: <491C5D04.2020803@reactive-systems.com> Greetings: I have found that the MLton interpretation of Windows paths breaks a lot of code that works under SML/NJ with MinGW. From reading the source code, I see that MLton does not consider a path to be absolute unless it has both a root and a volume. I'm inclined to think this is a mistake for the following reasons: 1. It differs from the 2002 Basis library standard, which clearly states that both "\\a\\b" and "A:\\a\\b" are absolute paths on Windows (note: these strings are SML literals, so double backslash represents a single backslash character). 2. Under MinGW and Cygwin, a rooted path without a volume is not ambiguous. The volume and arcs of the root are determined by /etc/fstab. Furthermore, I am observing behavior under SVN r6941 MLton/MinGW32: 1. (OS.Path.toString { isAbs = true, vol = "", arcs = ["home", "joe"] }) raises a Path exception. Shouldn't this return the path "\\home\\joe", which would be considered to be a MLton relative path? Or is MLton requiring that all absolute paths include a volume? 2. Similarly, (OS.Path.mkCanonical "\\home\\joe") raises a Path exception. Why not simply return the input unchanged? On a positive note, except for this path handling issue, all the code I have thrown at MLton so far is compiling and running. I would appreciate some clarification on whether the current OS.Path behavior is considered to be correct as-is, or is likely to change in the near future, so that I can decide how to handle paths in an implementation-independent fashion. Thanks, -Dan -- Dan DuVarney Senior Software Engineer Reactive Systems Inc. duvarney@reactive-systems.com +1 (919) 324-3507 ext. 103 OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~duvarney/pgp_public_key.txt From wesley at terpstra.ca Thu Nov 13 10:50:01 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Thu Nov 13 10:50:07 2008 Subject: [MLton-user] MLton OS.Path under Windows In-Reply-To: <491C5D04.2020803@reactive-systems.com> References: <491C5D04.2020803@reactive-systems.com> Message-ID: <162de7480811131050v2ddd061dqf3f1612c9d7a352d@mail.gmail.com> My comments are restricted to MinGW. On Thu, Nov 13, 2008 at 5:59 PM, Dan DuVarney wrote: > I have found that the MLton interpretation of Windows paths breaks a lot > of code that works under SML/NJ with MinGW. From reading the source > code, I see that MLton does not consider a path to be absolute unless it > has both a root and a volume. I don't believe the code does this. The definition of fromString sets isAbs=true for "\\bar". > 1. It differs from the 2002 Basis library standard, which clearly states > that both "\\a\\b" and "A:\\a\\b" are absolute paths on Windows (note: > these strings are SML literals, so double backslash represents a single > backslash character). Try it for yourself. Those both set isAbs=true. > 2. Under MinGW and Cygwin, a rooted path without a volume is not > ambiguous. The volume and arcs of the root are determined by /etc/fstab. No. This is a common misunderstanding. Under MinGW /etc (probably) does not exist. You are being lied to by the msys shell. Write a program that lists files in /etc to see for yourself that a MinGW program doesn't see this directory and it only comes from the bash shell in the msys window. MinGW programs (except for the msys shell) behave exactly like windows native programs with regards to the filesystem. There is no unix-like '/'. So even though MLton complies with the spec and calls "/foo" absolute, you're not going to convince me that it isn't ambiguous. Write a program that lists files in "/". Now run it from two different volumes and see two different results. > 1. (OS.Path.toString { isAbs = true, vol = "", arcs = ["home", "joe"] }) > raises a Path exception. Shouldn't this return the path "\\home\\joe", > which would be considered to be a MLton relative path? Or is MLton > requiring that all absolute paths include a volume? >From the standard I quote: > toString: The exception Path is raised if validVolume{isAbs, vol} is false, > validVolume: Under Microsoft Windows, the valid volume names have the form "a:", "A:", "b:", "B:", etc. and, if isAbs = false, also "" I agree that the standard sucks because it contradicts itself here. Earlier it purports that "\\a\\b" isAbs, but now it forbids isAbs=false + volume="". This directly contradicts that toString o fromString is the identity, but that claim is wrong anyway since "foo/bar" -> "foo\bar". > 2. Similarly, (OS.Path.mkCanonical "\\home\\joe") raises a Path > exception. Why not simply return the input unchanged? I think this is a consequence of toString as above. Anyway, it probably makes sense to allow volume="" and isAbs=true in defiance of the standard, but we probably need some standard wranglers to weigh in on the topic first. I still believe that the standard should've said that "\\foo\\bar" is not absolute on windows (since it IS ambiguous and depends on your current directory), but I lost this fight long ago. Instead we have a standard that says in some places yes and some places no. From rossberg at mpi-sws.mpg.de Thu Nov 13 13:32:12 2008 From: rossberg at mpi-sws.mpg.de (Andreas Rossberg) Date: Thu Nov 13 13:32:43 2008 Subject: [MLton-user] MLton OS.Path under Windows In-Reply-To: <162de7480811131050v2ddd061dqf3f1612c9d7a352d@mail.gmail.com> References: <491C5D04.2020803@reactive-systems.com> <162de7480811131050v2ddd061dqf3f1612c9d7a352d@mail.gmail.com> Message-ID: "Wesley W. Terpstra" wrote: > > So even though MLton complies with the spec and calls "/foo" absolute, > you're not going to convince me that it isn't ambiguous. Write a > program that lists files in "/". Now run it from two different volumes > and see two different results. > >> 1. (OS.Path.toString { isAbs = true, vol = "", arcs = ["home", "joe"] }) >> raises a Path exception. Shouldn't this return the path "\\home\\joe", >> which would be considered to be a MLton relative path? Or is MLton >> requiring that all absolute paths include a volume? > >>From the standard I quote: >> toString: The exception Path is raised if validVolume{isAbs, vol} is >> false, >> validVolume: Under Microsoft Windows, the valid volume names have the >> form "a:", "A:", "b:", "B:", etc. and, if isAbs = false, also "" > > I agree that the standard sucks because it contradicts itself here. > Earlier it purports that "\\a\\b" isAbs, but now it forbids > isAbs=false + volume="". This directly contradicts that toString o > fromString is the identity, but that claim is wrong anyway since > "foo/bar" -> "foo\bar". The spec of OS.Path is probably the most inconsistent part of the standard. I have a page-long list of issues, including these ones. The definition of canonical paths and the treatment of empty arcs are particularly off. Even the examples explicitly given contradict the specification (e.g. for cases as simple as "/"). However, I think that -- apart from the "bug" about validVolume{isAbs=true, vol=""} -- the standard is consistent about treating "absoluteness" and presence of volumes independently. Whether that's a useful definition is another question. The idea probably was that you want to be able to somehow qualify paths like "bla\\bla" and "\\bla\\bla" differently, which is useful (under Windows, drives easily get mapped to different letters at different times, so you often want to make your paths absolute for a drive, but not "too absolute"). The reason that validVolume does not reflect this might simply be an oversight - an earlier version of the spec may have had a different treatment of volumes. - Andreas From duvarney at reactive-systems.com Thu Nov 13 18:37:42 2008 From: duvarney at reactive-systems.com (Dan DuVarney) Date: Thu Nov 13 18:37:40 2008 Subject: [MLton-user] OS.Path Message-ID: <491CE476.8010200@reactive-systems.com> Hello again: This message isn't going to be attached to the ongoing "path" thread, but I am still waiting for the digest to arrive and wanted to follow up on what I said earlier. First, forget what I said about absolute paths being treated as if they were relative. I was confused when I wrote that. Second, the source of my confusion was that, when I read the specification for the OS.Path, I failed to notice the requirement that validVolume { isAbs = true, vol = "" } should return false under Windows. This requirement leads to the other behavior I described, such as OS.Path.toString (OS.Path.fromString "\\usr\\local") raising an exception. Once again, MLton's behavior is correct. It is SML/NJ which is not complying with the standard. I added this issue to the StandardMLDeviations wiki page. -Dan -- Dan DuVarney Senior Software Engineer Reactive Systems Inc. duvarney@reactive-systems.com +1 (919) 324-3507 ext. 103 OpenPGP (GnuPG) public key file: http://www.reactive-systems.com/~duvarney/pgp_public_key.txt From jhr at cs.uchicago.edu Fri Nov 14 01:01:11 2008 From: jhr at cs.uchicago.edu (John Reppy) Date: Fri Nov 14 01:01:36 2008 Subject: [MLton-user] Re: MLton OS.Path under Windows [MLton-user Digest, Vol 27, Issue 6] In-Reply-To: References: Message-ID: <364FD5E6-5DAE-434C-A3EF-441C2B62B190@cs.uchicago.edu> Looking at the OS.Path specification, there is clearly an inconsistency in the definition of absolute paths on Windows. The confusion arises because Windows has the notion of a path that is absolute relative to the current volume (but not strictly absolute, since changing the volume changes its meaning). I'm not sure what the correct fix is. The path of least resistance might be to change the specification of toString to read The exception Path is raised if vol <> "" and validVolume{isAbs, vol} is false. Alternatively, we could redefine the notion of absolute path on Windows to require a valid volume. - John From rossberg at mpi-sws.mpg.de Fri Nov 14 04:54:19 2008 From: rossberg at mpi-sws.mpg.de (Andreas Rossberg) Date: Fri Nov 14 04:54:35 2008 Subject: [MLton-user] Re: MLton OS.Path under Windows [MLton-user Digest, Vol 27, Issue 6] In-Reply-To: <364FD5E6-5DAE-434C-A3EF-441C2B62B190@cs.uchicago.edu> References: <364FD5E6-5DAE-434C-A3EF-441C2B62B190@cs.uchicago.edu> Message-ID: <3E019801-9B69-4722-8EF4-D77D8133C100@mpi-sws.mpg.de> On Nov 14, 2008, at 10.01h, John Reppy wrote: > Looking at the OS.Path specification, there is clearly an > inconsistency in the definition of > absolute paths on Windows. The confusion arises because Windows > has the notion of a path > that is absolute relative to the current volume (but not strictly > absolute, since changing > the volume changes its meaning). I'm not sure what the correct fix > is. The path of least > resistance might be to change the specification of toString to read > > The exception Path is raised if vol <> "" and validVolume{isAbs, > vol} is false. That would still allow fromString to deliver results that are not considered "valid". For paths that are perfectly sensible. I think the simpler and more coherent fix is to simply change validVolume to allow empty volumes unconditionally on Windows. > Alternatively, we could redefine the notion of absolute path on > Windows to require a > valid volume. I fear that this would require quite a few (incompatible) changes with the potential to break existing code in subtle ways. Since we are at it, as I mentioned earlier (on the MLton list) I have a number of other issues with inconsistencies in OS.Path. If you are interested, I'd be happy to post them. Best, - Andreas From jhr at cs.uchicago.edu Fri Nov 14 05:08:15 2008 From: jhr at cs.uchicago.edu (John Reppy) Date: Fri Nov 14 05:08:38 2008 Subject: [MLton-user] Re: MLton OS.Path under Windows [MLton-user Digest, Vol 27, Issue 6] In-Reply-To: <3E019801-9B69-4722-8EF4-D77D8133C100@mpi-sws.mpg.de> References: <364FD5E6-5DAE-434C-A3EF-441C2B62B190@cs.uchicago.edu> <3E019801-9B69-4722-8EF4-D77D8133C100@mpi-sws.mpg.de> Message-ID: <4403AA15-BF1E-4EF3-B3C8-3E0797467452@cs.uchicago.edu> On Nov 14, 2008, at 12:54 PM, Andreas Rossberg wrote: > On Nov 14, 2008, at 10.01h, John Reppy wrote: > >> Looking at the OS.Path specification, there is clearly an >> inconsistency in the definition of >> absolute paths on Windows. The confusion arises because Windows >> has the notion of a path >> that is absolute relative to the current volume (but not strictly >> absolute, since changing >> the volume changes its meaning). I'm not sure what the correct fix >> is. The path of least >> resistance might be to change the specification of toString to read >> >> The exception Path is raised if vol <> "" and validVolume{isAbs, >> vol} is false. > > That would still allow fromString to deliver results that are not > considered "valid". For paths that are perfectly sensible. I'm not sure what you are driving at? Can you give some examples? > I think the simpler and more coherent fix is to simply change > validVolume to allow empty volumes unconditionally on Windows. > >> Alternatively, we could redefine the notion of absolute path on >> Windows to require a >> valid volume. > > I fear that this would require quite a few (incompatible) changes > with the potential to break existing code in subtle ways. > > Since we are at it, as I mentioned earlier (on the MLton list) I > have a number of other issues with inconsistencies in OS.Path. If > you are interested, I'd be happy to post them. Please do so. - John From henry.cejtin at sbcglobal.net Fri Nov 14 06:03:03 2008 From: henry.cejtin at sbcglobal.net (Henry Cejtin) Date: Fri Nov 14 06:03:35 2008 Subject: [MLton-user] Re: MLton OS.Path under Windows [MLton-user Digest, Vol 27, Issue 6] Message-ID: <930650.90342.qm@web82404.mail.mud.yahoo.com> I don't use Windows, so take what I saw with a grain of salt, but my understanding was that the only point of a path being absolute that it did not depend on the context of the running program. Hence it would seem that the only thing that makes sense is to require that on Windows an absolute path requires a valid volume. I.e., the question is does it ever make sense to treat paths which start with a \ but have no volume any differently than paths that do not start with a \. From wesley at terpstra.ca Fri Nov 14 07:00:57 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Fri Nov 14 07:01:02 2008 Subject: [MLton-user] Re: MLton OS.Path under Windows [MLton-user Digest, Vol 27, Issue 6] In-Reply-To: <930650.90342.qm@web82404.mail.mud.yahoo.com> References: <930650.90342.qm@web82404.mail.mud.yahoo.com> Message-ID: <162de7480811140700t11ed1569pac3da7449d1d0f6b@mail.gmail.com> On Fri, Nov 14, 2008 at 3:03 PM, Henry Cejtin wrote: > I.e., the question is does it ever make sense to treat paths which start with > a \ but have no volume any differently than paths that do not start with a \. In my opinion, no. The path \bar is not absolute in any useful sense, only in the sense that this is the definition in the basis specification. However, as Andreas points out, changing this could impact on existing SML code for windows. Andreas Rossberg wrote: > I think the simpler and more coherent fix is to simply change > validVolume to allow empty volumes unconditionally on Windows. This seems the appropriate stop-gap measure to me as well. From rossberg at mpi-sws.mpg.de Fri Nov 14 07:15:05 2008 From: rossberg at mpi-sws.mpg.de (Andreas Rossberg) Date: Fri Nov 14 07:15:21 2008 Subject: [Sml-basis-discuss] [MLton-user] Re: MLton OS.Path under Windows [MLton-user Digest, Vol 27, Issue 6] In-Reply-To: <162de7480811140700t11ed1569pac3da7449d1d0f6b@mail.gmail.com> References: <930650.90342.qm@web82404.mail.mud.yahoo.com> <162de7480811140700t11ed1569pac3da7449d1d0f6b@mail.gmail.com> Message-ID: <1BFAC8DE-8071-4C50-B7D5-BEDADFC7213F@mpi-sws.mpg.de> On Nov 14, 2008, at 16.00h, Wesley W. Terpstra wrote: > > In my opinion, no. The path \bar is not absolute in any useful sense, > only in the sense that this is the definition in the basis > specification. Note again that, if you considered "\bar" as relative, then there would be no way to distinguish the paths "bar" and "\bar" in the triple representation - they would both map to {isAbs=false, vol="", arc=["bar']}. I think the fallacy here is assuming that the isAbs flag has any semantic meaning. No, it is a purely syntactic piece of information about the presence of the root separator. Once you realise that, the spec makes sense (up to validVolume). Maybe another choice of name would have been better, though. - Andreas From rossberg at mpi-sws.mpg.de Fri Nov 14 08:53:40 2008 From: rossberg at mpi-sws.mpg.de (Andreas Rossberg) Date: Fri Nov 14 08:53:58 2008 Subject: [Sml-basis-discuss] [MLton-user] Re: MLton OS.Path under Windows [MLton-user Digest, Vol 27, Issue 6] In-Reply-To: <4403AA15-BF1E-4EF3-B3C8-3E0797467452@cs.uchicago.edu> References: <364FD5E6-5DAE-434C-A3EF-441C2B62B190@cs.uchicago.edu> <3E019801-9B69-4722-8EF4-D77D8133C100@mpi-sws.mpg.de> <4403AA15-BF1E-4EF3-B3C8-3E0797467452@cs.uchicago.edu> Message-ID: <3252BCBF-0513-4FB1-9E80-1F1B16538B68@mpi-sws.mpg.de> On Nov 14, 2008, at 14.08h, John Reppy wrote: > > On Nov 14, 2008, at 12:54 PM, Andreas Rossberg wrote: > >> On Nov 14, 2008, at 10.01h, John Reppy wrote: >> >>> Looking at the OS.Path specification, there is clearly an >>> inconsistency in the definition of >>> absolute paths on Windows. The confusion arises because Windows >>> has the notion of a path >>> that is absolute relative to the current volume (but not strictly >>> absolute, since changing >>> the volume changes its meaning). I'm not sure what the correct fix >>> is. The path of least >>> resistance might be to change the specification of toString to read >>> >>> The exception Path is raised if vol <> "" and validVolume{isAbs, >>> vol} is false. >> >> That would still allow fromString to deliver results that are not >> considered "valid". For paths that are perfectly sensible. > > I'm not sure what you are driving at? Can you give some examples? Well, fromString "\\bar" = {isAbs=true, vol=false, arcs=["bar"]}, but validVolume{isAbs=true, vol=""} = false. Likewise for the root path "\ \". These certainly are meaningful paths, so it's weird that the validVolume function somehow considers them invalid. >> Since we are at it, as I mentioned earlier (on the MLton list) I >> have a number of other issues with inconsistencies in OS.Path. If >> you are interested, I'd be happy to post them. > > Please do so. Here is my list. Note that about half of the problems can easily be resolved by interpreting "/" as containing no arcs. (I would guess that the motivation for letting it have an empty arc was the desire to treat the root separator like any other arc separator (in particular, a trailing separator indicates an empty arc). Unfortunately, the root separator is quite different in nature, and behaves more like a volume.) Best, - Andreas Empty arcs: - The path "/" is treated as having an empty arc. This introduces a special case of a valid initial empty arc, and consequently, a number of issues with canonical paths and invertibility of string conversion (see below). It also arguably is inconsistent with the treatment of "", which has no arcs. Proposal: Define "/" to have no arcs. Consider initial empty arcs illegal under all circumstances. - Related, what is #arcs(fromString "A:"), e.g. on Windows, [] or [""]? Proposal: Following the previous point, choose []. Canonical paths: - The definition of canonical paths rules out empty arcs, but then lists "/" as a canonical path (which supposedly has an empty arc). Proposal: Resolved by interpreting "/" to have no arcs. - On the other hand, the spec as is actually makes "" a canonical path (it contains no arcs). This does not seem to be intended, as witnessed by the examples and the definition of mkCanonical. Proposal: Explicitly rule out "" from canonical paths. - Since "/" and "/." denote the same path, not both should be canonical. (That also is inconsistent with defining "." as canonical, but not "") Proposal: Only allow "." in canonical paths that otherwise empty. In particular, don't treat "/." as canonical. [All systems I tried actually treat "/." as non-canonical] String conversion: - toString o fromString can only be the identity up to the choice of arc separators (e.g. on Windows, where both #"/" and #"\\" are allowed). Proposal: Weaken the respective statement. - fromString o toString is not the identity for absolute paths with no arcs, e.g.: {isAbs=true, vol="", arcs=[]} -> {isAbs=true, vol="", arcs=[""]} Proposal: Resolved by interpreting "/" to have no arcs. - validVolume on Windows allows empty volumes only for relative paths. This treats basic paths like "\\d\\e" or even "\\" as "invalid", and furthermore causes violation of the requirement for toString o fromString to be the identity. Proposal: Allow empty volumes unconditionally on Windows. JoinDirFile: - Is defined as just "extending" a path with an arc, but this violates the stated principle that canonical paths should be preserved (namely in the cases of "." and "/."). Proposal: Specify that for dir=".", file is returned. The case "/." is dealt with by not considering it canonical (see above). [All systems I tried construct non-canonical paths in these cases.] - Since "/" is supposed to have an empty arc, the spec as stated would imply joinDirFile{dir="/", file="a"} = "//a", which is non-sensical. Proposal: Resolved by interpreting "/" to have no arcs. - The description also does not cover the case where dir has no arcs and the file is the empty arc (the logical result is not representable as a string, since it would have an initial empty arc). Proposal: In order to behave like the inverse of splitDirFile, specify that the special case of joining a dir without arcs with the empty arc returns the dir. Absolute/relative paths: - According to the spec, mkRelative{path="/a", relativeTo="/"} has to deliver "../a", because "/" is canonical but defined to contain an arc. Proposal: Resolved by interpreting "/" to have no arcs. - It would be *very* useful if mkRelative worked for relative paths, too. Proposal: Instead of raising Path, specify that if both arguments are relative paths, resolution proceeds analogously to two absolute paths, essentially treating "." as the root. Here are examples: path relativeTo mkRelative{path, relativeTo} "a" "" "a" "a" "." "a" "a" "a" "." "a/b/c" "a" "b/c" "a" "b/c" "../../a" "a" "./a" "." "a/" "" "a/" - The spec of mkRelative is inadequate for case-insensitive file systems, because relativeTo is brought into canonical form, but not path. Proposal: At least make clear that "common" prefix has to be interpreted up to equivalence classes of file names on a given system. Or simply require path to be made canonical, too. - Similary, the sample implementation for concat suggests that volumes must be equal even with respect to case. Again, this is inadequate for case-insensitive file systems. Proposal: Add a respective comment. - Typos mkAbsolute: s/mkCanonical (concat (abs, p))/mkCanonical (concat(relativeTo, path))/ mkRelative: s/but have different roots/but have different volumes/ concat: s/A implementation/An implementation/ From seanmcl at gmail.com Sat Nov 29 11:17:40 2008 From: seanmcl at gmail.com (Sean McLaughlin) Date: Sat Nov 29 11:18:16 2008 Subject: [MLton-user] Conditional compilation Message-ID: <33129095-D356-457D-8C0E-17A075B0A539@gmail.com> Hello, This is a feature request. Or at least a request for a discussion. I have a fairly large (for a single person) software project that does a number of related tasks. Many of these tasks are considerably improved by using external C packages for which I wrote ffi code. Now that others are trying to compile the software, I have a problem. If they don't have the relevant C library, for instance CPLEX, a proprietary linear programming solver, the code won't compile. Now, the code using CPLEX may never be needed by a particular user, or they may be equally well served by using a free alternative such as GLPK, for which I also have ffi stubs. My current solution is to use #include a file that has the names of the relevant libraries, and each of my mlb files are preprocessed using cpp to generate the actual mlb file that will be compiled. This is a somewhat ugly solution. If MLton supported some kind of preprocessing directives, as you do for the single variable SMLNJ_VERSION. According to http://mlton.org/CompilationManager , no other symbols are defined. If only I could define my own symbols, I'd be able to avoid the gcc step, which would be a considerable improvement. Would it be very difficult to allow a user to define his own preprocessing variables? Thanks, Sean From ville at laurikari.net Sat Nov 29 22:42:51 2008 From: ville at laurikari.net (Ville Laurikari) Date: Sat Nov 29 22:42:57 2008 Subject: [MLton-user] Conditional compilation In-Reply-To: <33129095-D356-457D-8C0E-17A075B0A539@gmail.com> References: <33129095-D356-457D-8C0E-17A075B0A539@gmail.com> Message-ID: <20081130064251.GA27295@laurikari.net> On Sat, Nov 29, 2008 at 02:17:40PM -0500, Sean McLaughlin wrote: > My current solution is to use #include a file that has the names of > the relevant libraries, and each of my > mlb files are preprocessed using cpp to generate the actual mlb file > that will be compiled. This is a somewhat ugly solution. If MLton The usual way to deal with this is to use MLB path variables. The path map file can be set with -mlb-path-map. With recent MLton versions, you can also set individual path variables with -mlb-path-var. In your case, you should divide the different variants of linear programming support (none, cplex, glpk) into separate libraries which all export the same interface. These could be, for example, in a layout like this: lp/none.mlb lp/cplex.mlb lp/glpk.mlb Then you can define a path variable, say LP, to choose the appropriate linear programming library for your application: lp/$(LP).mlb -- Ville