From wesley at terpstra.ca Thu Oct 2 03:20:17 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Thu Oct 2 03:20:39 2008 Subject: [MLton] MLton import headers Message-ID: <162de7480810020320s28ffc3d7y42dd4b30abfa6f80@mail.gmail.com> Currently MLton outputs C import headers (ironically with -export-header). I think it might also be useful to output an ML import file. One of the problems with using a library is that you need to know how you are linking against it. Often programmers can get away without knowing, but this breaks down on some combinations of operating system, architecture, and definitions. One of the tasks the C import header is to make the details transparent to the user. When using a MLton generated library from another MLton program, there is the same problem. _import "foo" public: ...; is appropriate for a static link, while _import "foo" external: ...; is appropriate for a dynamic link. I was thinking of an output file something like: signature M1 = sig val m1_open : int * string vector -> unit; val m1_close : unit -> unit; end structure STATIC_LINK_M1 :> M1 = struct val m1_open = _import "m1_open" public; val m1_close = _import "m1_close" public; end structure DYNAMIC_LINK_M1 :> M1 = struct val m1_open = _import "m1_open" external; val m1_close = _import "m1_close" external; end I have intentionally left out PART_OF_M1 because you're better off not using the FFI in this case. Thoughts? From vesa.a.j.k at gmail.com Fri Oct 3 01:01:46 2008 From: vesa.a.j.k at gmail.com (Vesa Karvonen) Date: Fri Oct 3 01:02:22 2008 Subject: [MLton] MLton import headers In-Reply-To: <162de7480810020320s28ffc3d7y42dd4b30abfa6f80@mail.gmail.com> References: <162de7480810020320s28ffc3d7y42dd4b30abfa6f80@mail.gmail.com> Message-ID: <9e43b9a0810030101h450144c1pd086454c34b73b87@mail.gmail.com> On Thu, Oct 2, 2008 at 1:20 PM, Wesley W. Terpstra wrote: [...] > One of the problems with using a library is that you need to know how > you are linking against it. Often programmers can get away without > knowing, but this breaks down on some combinations of operating > system, architecture, and definitions. In C and C++, and particularly on Windows, one often uses macros in library headers and source files to insert appropriate compiler specific declarations to mark symbols that are either exported from the currently compiled DLL or imported from an external DLL. These macros are often defined in a configuration header of some sort and one can then easily switch between various linkage options (static/dynamic library, import/export symbols). > When using a MLton generated library from another MLton program, there > is the same problem. _import "foo" public: ...; is appropriate for a > static link, while _import "foo" external: ...; is appropriate for a > dynamic link. I was thinking of an output file something like: > > signature M1 = > sig > val m1_open : int * string vector -> unit; > val m1_close : unit -> unit; > end > > structure STATIC_LINK_M1 :> M1 = > struct > val m1_open = _import "m1_open" public; > val m1_close = _import "m1_close" public; > end > > structure DYNAMIC_LINK_M1 :> M1 = > struct > val m1_open = _import "m1_open" external; > val m1_close = _import "m1_close" external; > end > > I have intentionally left out PART_OF_M1 because you're better off not > using the FFI in this case. > > Thoughts? One thing that comes to mind from this is that it would be nice to be able to switch between static and dynamic linking easily without using a complicated mechanism at the source level. In fact, I think that ideally code that uses a library should really be exactly the same regardless of whether the library is linked statically or (unoptionally) dynamically. I'm not saying that it isn't possible with the above format (you need to bind one of the modules, STATIC_LINK_M1 or DYNAMIC_LINK_M1, to a module that you use in the client code), but rather that you might want to consider this from that perspective. One alternative that comes to mind would be to specify as a command-line option whether one wishes to link statically or dynamically and MLton would then generate only one set of imports and the module name would be the same (e.g. just M1) in either case. This import file generation step could be called as a part of the build script (for the library) and the resulting module would then be used to call the library. Alternatively, the module could be selected in a MLB file based on a path variable and the library would have pregenerated import files for both cases (static/dynamic). In either case, the SML client code would be exactly the same regardless of whether the library is linked statically or dynamically. -Vesa Karvonen From wesley at terpstra.ca Fri Oct 3 06:04:22 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Fri Oct 3 06:04:26 2008 Subject: [MLton] MLton import headers In-Reply-To: <9e43b9a0810030101h450144c1pd086454c34b73b87@mail.gmail.com> References: <162de7480810020320s28ffc3d7y42dd4b30abfa6f80@mail.gmail.com> <9e43b9a0810030101h450144c1pd086454c34b73b87@mail.gmail.com> Message-ID: <162de7480810030604p31ff4e0ayfee829183b4ff4c5@mail.gmail.com> On Fri, Oct 3, 2008 at 10:01 AM, Vesa Karvonen > In C and C++, and particularly on Windows, one often uses macros in > library headers and source files to insert appropriate compiler > specific declarations to mark symbols ... Yes, this is what MLton/svn does in the generated C import header. However, in some cases a programmer still needs to specify how he links a library. The MLton import headers do this with a macro: #define STATIC_LINK_FOOBAR #include > One thing that comes to mind from this is that it would be nice to be > able to switch between static and dynamic linking easily without using > a complicated mechanism at the source level. In fact, I think that > ideally code that uses a library should really be exactly the same > regardless of whether the library is linked statically or > (unoptionally) dynamically. I'm not saying that it isn't possible with > the above format (you need to bind one of the modules, STATIC_LINK_M1 > or DYNAMIC_LINK_M1, to a module that you use in the client code), but > rather that you might want to consider this from that perspective. I agree this would be nice, but unfortunately it isn't. The assembly/C generated by MLton to use a library must know whether those symbols are imported from a dynamically or statically linked library. MLton does not know this. When we pass '-lfoo' to the linker, it picks some library out of paths we know nothing about. > One alternative that comes to mind would be to specify as a > command-line option whether one wishes to link statically or > dynamically and MLton would then generate only one set of imports and > the module name would be the same (e.g. just M1) in either case. I am not sure where these options are to be passed, but it sounds like you mean at library creation. What I think you've overlooked is that the same library can be linked three different ways. Consider a PIC archive library foo. During compilation of C that will be included in the final PIC library you have access to private symbols and shouldn't import symbols that will be in the final foo product. A shared library built against foo will statically link foo in. This means it no longer has access to private symbols, but shouldn't be importing the symbols either as they will be in the same final DSO. MLton import headers call this PART_OF_LIBNAME. A user of the resulting shared library wants to call functions that were in foo. He has no access to private symbols and needs to import the public ones. MLton import headers call this STATIC_LINK_LIBNAME Each of these three uses are distinct and if you try to coflate any of them I can present you with a platform where your solution breaks. MLton import headers call this DYNAMIC_LINK_LIBNAME. While it's certainly true that for many libraries there is a sane default, sometimes there isn't. In MLton/svn terminology, we have four output formats. The generated header access modes are described below: executable -- only ever makes sense to use PART_OF_LIBNAME linkage archive -- currently defaults to STATIC_LINK_LIBNAME, but if C is included into the resulting library, PART_OF_LIBNAME should be defined to override this default. library -- currently defaults to DYNAMIC_LINK_LIBNAME, but override to PART_OF_LIBNAME makes sense. libarchive -- the case I detailed above, the import header has no default and you have to specify the linkage. I imagine we would output ML import files in the same way, eg: default M1 = STATIC_LINK_M1 for archive, M1 = DYNAMIC_LINK_M1 for library. No default for libarchive. Happily, there is no PART_OF_M1 case. At any rate, this is what I am proposing: an automatic M1 that "does the right thing" when it can, but includes both options in case you need them. Another problem on my mind, is that it's fairly common (on linux) for libraries to be shipped as both dynamic (PIC .so) and static (non-PIC archive), with only one header file. Sometimes important libraries also include a PIC archive. This works because happily on ELF systems public/external end up being the same. I'm not sure how to support this model and at the same time also support platforms like windows where public/external are critically different. From fluet at tti-c.org Fri Oct 3 13:21:54 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Fri Oct 3 13:22:31 2008 Subject: [MLton] MLton import headers In-Reply-To: <162de7480810020320s28ffc3d7y42dd4b30abfa6f80@mail.gmail.com> References: <162de7480810020320s28ffc3d7y42dd4b30abfa6f80@mail.gmail.com> Message-ID: On Thu, 2 Oct 2008, Wesley W. Terpstra wrote: > Thoughts? The whole set of shared-library/visibility issues seems fairly opaque (a reflection on the ABI itself, not its realization in MLton). From some of the scenarios that Wesley has described, it often seems to be the case that one has to be very clear on how a program/library links to other libraries, how a library is generated depending on its future use, etc. Things seem sufficiently complex that it is unclear whether a complicated set of implicit defaults is enough to shield a programmer completely (or even mostly) from the details of which they need to be aware. If that is the case, then it is often simpler to allow/force someone to be explicit up front, since that may be easier than trying to back out of the implicit defaults when the need arises. Also, picking up a theme similar to one that Vesa raised, is it so difficult to set the visibility of an imported library at configure/build time of a target. For example, something like: ---foo_import.src--- structure Foo = struct val libopen = _import "foo_open" FOO_SCOPE: int * string vector -> unit; val libclose = _import "foo_close" FOO_SCOPE: unit -> unit; end ------ ---Makefile--- FOO_SCOPE=public foo_import.sml: foo_import_src sed 's|FOO_SCOPE|$(FOO_SCOPE)|' < foo_import.scc > foo_import.sml ------ where the 'FOO_SCOPE=public' in the Makefile could either have been determined at configure time (depending on the availability of libfoo.a or libfoo.so) or left blank to be set at the 'make' invocation. Finally, importing a MLton library (either static or dynamic) into another MLton library or executable seems to be a fairly obscure usage. Independent of the fact that MLton is a whole-program compiler (which benefits from exposing all of the SML code code the compiler at once), I don't know of any high-level language implementation (e.g., OCaml, GHC) that prefers to import language-native libraries as though they were language-independent system libraries. [One might argue that CLR/.NET is an exception, but, really, the 'language-native library' in that instance happens to be .NET assemblies.] It is almost certainly a win to have all high-level language code sharing the same instance of the runtime/GC/etc. From vesa.a.j.k at gmail.com Fri Oct 3 14:28:26 2008 From: vesa.a.j.k at gmail.com (Vesa Karvonen) Date: Fri Oct 3 14:28:29 2008 Subject: [MLton] MLton import headers In-Reply-To: References: <162de7480810020320s28ffc3d7y42dd4b30abfa6f80@mail.gmail.com> Message-ID: <9e43b9a0810031428v1ecf7232r8c14612f1a51dbf@mail.gmail.com> On Fri, Oct 3, 2008 at 11:21 PM, Matthew Fluet wrote: > Finally, importing a MLton library (either static or dynamic) into another > MLton library or executable seems to be a fairly obscure usage. [...] I could certainly imagine some more or less plausible practical reasons to do so, such as cutting compilation time. Although, given the limited set of types available for interfacing with such libraries, many such uses could just as well be written by compiling the MLton "library" as a regular program that is spawned by the host program as a separate process and communicated with via some form of IPC. BTW, according to Armstrong's book this is a preferred way of using foreign language libraries in Erlang. > It is almost certainly a win to have all high-level language code sharing > the same instance of the runtime/GC/etc. Yes, if you are directly linking to the code, then I would tend to agree (modulo exceptional practical concerns). Isolating parts of a program to different processes, OTOH, may have practical advantages such as fault isolation (bug in a subsystem cannot crash the entire program) that can be more valuable than efficiency. -Vesa Karvonen From wesley at terpstra.ca Fri Oct 3 16:55:22 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Fri Oct 3 16:55:24 2008 Subject: [MLton] MLton import headers In-Reply-To: References: <162de7480810020320s28ffc3d7y42dd4b30abfa6f80@mail.gmail.com> Message-ID: <162de7480810031655v90e8efaoc346af8fe8779920@mail.gmail.com> On Fri, Oct 3, 2008 at 10:21 PM, Matthew Fluet wrote: > Things seem > sufficiently complex that it is unclear whether a complicated set of > implicit defaults is enough to shield a programmer completely (or even > mostly) from the details of which they need to be aware. If that is the > case, then it is often simpler to allow/force someone to be explicit up > front, since that may be easier than trying to back out of the implicit > defaults when the need arises. So you would propose that I always require the user to #define PART_OF_X / STATIC_LINK_X / DYNAMIC_LINK_X before #include'ing the header? > ---foo_import.src--- > structure Foo = struct > val libopen = _import "foo_open" FOO_SCOPE: int * string vector -> unit; > val libclose = _import "foo_close" FOO_SCOPE: unit -> unit; > end > ------ > ---Makefile--- > FOO_SCOPE=public > > foo_import.sml: foo_import_src > sed 's|FOO_SCOPE|$(FOO_SCOPE)|' < foo_import.scc > foo_import.sml Well, I thought it would be nicer to have structure FooStatic = struct val libopen = _import "foo_open" public: int * string vector -> unit; val libclose = _import "foo_close" public: unit -> unit; end structure FooDynamic = struct val libopen = _import "foo_open" public: int * string vector -> unit; val libclose = _import "foo_close" public: unit -> unit; end structure Foo = FooStatic (* or whatever a sane default is *) ... my way you can use functors and/or the basis system to do what you want without having to drop out to make. > Finally, importing a MLton library (either static or dynamic) into another > MLton library or executable seems to be a fairly obscure usage. I was aiming for completeness. Perhaps it's not necessary then to output an ML import header; if you really need one, you can write it yourself. While I agree it's pretty bizarre to have two MLton libraries at once, I think this could happen if you have pure C programs/libraries that in turn use ML libraries. From fluet at tti-c.org Sat Oct 4 07:50:19 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Sat Oct 4 07:51:11 2008 Subject: [MLton] MLton import headers In-Reply-To: <162de7480810031655v90e8efaoc346af8fe8779920@mail.gmail.com> References: <162de7480810020320s28ffc3d7y42dd4b30abfa6f80@mail.gmail.com> <162de7480810031655v90e8efaoc346af8fe8779920@mail.gmail.com> Message-ID: On Sat, 4 Oct 2008, Wesley W. Terpstra wrote: > On Fri, Oct 3, 2008 at 10:21 PM, Matthew Fluet wrote: >> Things seem >> sufficiently complex that it is unclear whether a complicated set of >> implicit defaults is enough to shield a programmer completely (or even >> mostly) from the details of which they need to be aware. If that is the >> case, then it is often simpler to allow/force someone to be explicit up >> front, since that may be easier than trying to back out of the implicit >> defaults when the need arises. > > So you would propose that I always require the user to > #define PART_OF_X / STATIC_LINK_X / DYNAMIC_LINK_X before #include'ing > the header? Not necessarily. What is 'best/common' practice? Doing a 'grep -r "(visibility(" /usr/include/*' (on an x86-linux) turns up some headers that use it, but none appear to demand a #define PART_OF/STATIC_LINK/DYNAMIC_LINK in order to control the visibility. >> ---foo_import.src--- >> structure Foo = struct >> val libopen = _import "foo_open" FOO_SCOPE: int * string vector -> unit; >> val libclose = _import "foo_close" FOO_SCOPE: unit -> unit; >> end >> ------ >> ---Makefile--- >> FOO_SCOPE=public >> >> foo_import.sml: foo_import_src >> sed 's|FOO_SCOPE|$(FOO_SCOPE)|' < foo_import.scc > foo_import.sml > > Well, I thought it would be nicer to have > structure FooStatic = struct > val libopen = _import "foo_open" public: int * string vector -> unit; > val libclose = _import "foo_close" public: unit -> unit; > end > structure FooDynamic = struct > val libopen = _import "foo_open" public: int * string vector -> unit; > val libclose = _import "foo_close" public: unit -> unit; > end > structure Foo = FooStatic (* or whatever a sane default is *) > > ... my way you can use functors and/or the basis system to do what you > want without having to drop out to make. I don't see how functors and/or the ml basis sytem help here. Neither allow for control-flow or conditional compilation. In your example above, there is no way to change Foo from FooStatic to FooDynamic without changing the source file. As Vesa noted, you can use MLB variables to get a poor-man's form of conditional compilation, but then you have to (in the Makefile) load an appropriate mlb-path-map. From wesley at terpstra.ca Sat Oct 4 09:37:31 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sat Oct 4 09:37:35 2008 Subject: [MLton] MLton import headers In-Reply-To: References: <162de7480810020320s28ffc3d7y42dd4b30abfa6f80@mail.gmail.com> <162de7480810031655v90e8efaoc346af8fe8779920@mail.gmail.com> Message-ID: <162de7480810040937u2d576017u5dc3be18e5112464@mail.gmail.com> On Sat, Oct 4, 2008 at 4:50 PM, Matthew Fluet wrote: >> So you would propose that I always require the user to >> #define PART_OF_X / STATIC_LINK_X / DYNAMIC_LINK_X before #include'ing >> the header? > > Not necessarily. What is 'best/common' practice? Doing a > 'grep -r "(visibility(" /usr/include/*' (on an x86-linux) turns up some > headers that use it, but none appear to demand a > #define PART_OF/STATIC_LINK/DYNAMIC_LINK in order to control the visibility. On linux you can ignore the difference of STATIC_LINK and DYNAMIC_LINK. Thus most projects do. Libraries that must also work as a dll on windows are the only ones which have to care. All library projects provide some form of 'PART_OF', but it differs on a library-to-library basis. The most common approach is to provide a public header and one or more private headers. Examples of this are gdtoa, gmp, and sqlite3. Some libraries also use a macro. For example, gmp uses __GMP_WITHIN_GMP. gmp also distinguishes static/dynamic linkage with __GMP_LIBGMP_DLL. My approach follows gmp very closely. gmp sets GMP_LIBGMP_DLL at library creation time depending on if it is a static or dynamic library. It also has this to say: AC_MSG_ERROR([cannot build both static and DLL, since gmp.h is different for each. Use "--disable-static --enable-shared" to build just a DLL.]) I also set STATIC/DYNAMIC_LINK to some default in our export header based on static/dynamic output. I've just allowed the user to override it. Very few libraries can compile to a PIC archive. glibc is one example, but it clearly has no need to be portable to a non-ELF platform. This is the only case where I currently leave no default. There is definitely no 'best practice' for this type of library. The current MLton approach is: the export header is exactly the same for all output formats, except that the default PART_OF/STATIC_LINK/DYNAMIC_LINK differs. The user can be explicit and override this default. On an ELF system you can use the export headers for both static/dynamic, because EXERNAL/PUBLIC are identical on that target. Thus you can do like most linux programs and supply one header that works for both static and dynamic libraries. My personal opinion is that removing the default and requiring the user to always specify the linkage would be surprising when compared to how one typically uses C libraries. There are only two cases where the default is 'wrong'. 1) PIC archives, which are a corner case very few projects need. In this case the header has no default and forces you to pay attention. 2) You are compiling a library including ML code and C code. In this case we require: #define PART_OF_XYZ #include "xyz.h" inside your C files instead of: #include "xyz.h" #include "xyz-private.h" This doesn't seem particularly onerous or strange to me. You're cooperating with MLton to build a library in this case, so it's fair for us to have a (not-so-uncommon) convention you need to be aware of. > I don't see how functors and/or the ml basis sytem help here. Neither allow > for control-flow or conditional compilation. I meant to use the mlb-path-map approach. You can write your library-dependent code as a functor and then bind it to one of the two structures in a file chosen depending on a path variable. At any rate, I no longer think the ML export header is necessary. The only situtation it seems reasonable for an ML program to use an ML library via FFI is if there was a pure C library between them, in which case you probably shouldn't be using the internal ML library anyway, but rather the C library's wrappers. What I do think we need is a new annotation, 'defaultImport public/external'. This way your 'prim.sml' that does all the _import/_address/_symbol'ing can be easily switched between static/dynamic import. From fluet at tti-c.org Sat Oct 4 15:02:24 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Sat Oct 4 15:03:03 2008 Subject: [MLton] MLton import headers In-Reply-To: <162de7480810040937u2d576017u5dc3be18e5112464@mail.gmail.com> References: <162de7480810020320s28ffc3d7y42dd4b30abfa6f80@mail.gmail.com> <162de7480810031655v90e8efaoc346af8fe8779920@mail.gmail.com> <162de7480810040937u2d576017u5dc3be18e5112464@mail.gmail.com> Message-ID: On Sat, 4 Oct 2008, Wesley W. Terpstra wrote: > My personal opinion is that removing the default and requiring the > user to always specify the linkage would be surprising when compared > to how one typically uses C libraries. There are only two cases where > the default is 'wrong'. 1) PIC archives, which are a corner case very > few projects need. In this case the header has no default and forces > you to pay attention. 2) You are compiling a library including ML code > and C code. In this case we require: > #define PART_OF_XYZ > #include "xyz.h" > inside your C files instead of: > #include "xyz.h" > #include "xyz-private.h" > This doesn't seem particularly onerous or strange to me. You're > cooperating with MLton to build a library in this case, so it's fair > for us to have a (not-so-uncommon) convention you need to be aware of. That seems like a reasonable rationale. If I understand things correctly, though, one advantage of the #include "xyz.h" #include "xyz-private.h" approach is that is textually separates the public (in the sense of the documented portions of the library API available for use in other projects) functions from the private (in the sense of the undocumented portions of the library API) functions. That is, clients of "xyz.h" can't even see the prototypes of the private functions. On the other hand, if the platform supports the right visibility attributes, then although a client might be able to see the prototype of a private function in a (combined) "xyz.h" header, they would fail at link time, because the function wouldn't be visible in the binary library. I guess you acheive this effect in the (combined) "xyz.h" by using #define MLLIB_PRIVATE(x) when not PART_OF_XYZ to hide the private functions. > What I do think we need is a new annotation, 'defaultImport > public/external'. This way your 'prim.sml' that does all the > _import/_address/_symbol'ing can be easily switched between > static/dynamic import. This shifts the point of change from the 'prim.sml' file to the 'xyz.mlb' file. While one could use "-default-ann 'defaultImport public'", this could have undesirable effects if you import from two different libraries that demand different linking, since one global default doesn't apply. Anyways, it doesn't appear that there are clear-cut solutions. But, I tend to agree that an auto-generated SML import header is a sufficiently rare corner case that it doesn't really demand extra compiler support. -Matthew From wesley at terpstra.ca Sat Oct 4 16:05:15 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sat Oct 4 16:05:21 2008 Subject: [MLton] MLton import headers In-Reply-To: References: <162de7480810020320s28ffc3d7y42dd4b30abfa6f80@mail.gmail.com> <162de7480810031655v90e8efaoc346af8fe8779920@mail.gmail.com> <162de7480810040937u2d576017u5dc3be18e5112464@mail.gmail.com> Message-ID: <162de7480810041605i29fe5fbfw8fe0615482ae274e@mail.gmail.com> On Sun, Oct 5, 2008 at 12:02 AM, Matthew Fluet wrote: > That seems like a reasonable rationale. If I understand things correctly, > though, one advantage of the > #include "xyz.h" > #include "xyz-private.h" > approach is that is textually separates the public (in the sense of the > documented portions of the library API available for use in other projects) > functions from the private (in the sense of the undocumented portions of the > library API) functions. That is, clients of "xyz.h" can't even see the > prototypes of the private functions. Correct. On the flip-side, having only one file is convenient. While I agree that header files can serve a documentation purpose, I don't think that's true for us. An automatically generated header is full of extra-trash and lacks comments. The header doesn't declare the symbols to C and output library doesn't expose them to the linker. If you want to document your API, do a better job than an automatic header file. >> What I do think we need is a new annotation, 'defaultImport >> public/external'. This way your 'prim.sml' that does all the >> _import/_address/_symbol'ing can be easily switched between >> static/dynamic import. > > This shifts the point of change from the 'prim.sml' file to the 'xyz.mlb' > file. That's exactly what I wanted: one place to change. Some of my projects have prim.sml files that run over 1000 lines. These aren't auto-generated, so there's no possiible autogenerated ML import header to have in this case. An annotation could solve this neatly, without trying to 'sed' the file, which I think is a pretty crude and fragile approach. From wesley at terpstra.ca Wed Oct 8 06:26:43 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Oct 8 06:26:46 2008 Subject: [MLton] Code being dropped that shouldn't Message-ID: <162de7480810080626x6ac88adeg876e4e8db443acdf@mail.gmail.com> I've been trying to track down the last bug in the library regression test, but I think now it's not a bug in the test at all. Try an i386 compiler (windows and linux both show the same behaviour). Run: ./library-test -debug true -keep g -codegen c -profile time Expected (wrong) output: check starting up libm5 starting up libm4 starting up libm3 starting up libm2 starting up libm1 starting up m1 pointer test complete. m2 pointer test complete. m3 pointer test complete. m4 pointer test complete. m5 pointer test complete. check pointer test complete. check exits profiling is on profiling is on profiling is on profiling is on profiling is on The problem is that the atExit code for libm1-5 doesn't get run. However, just adding a print statement to mlton/exit.sml:60: (* Return to 'lib_open'. *) val () = Primitive.MLton.Thread.returnToC () (* Enter from 'lib_close'. *) val () = print "lib_close invoked\n" val _ = exiting := true val () = let open Cleaner in clean atExit end (* Return to 'lib_close'. *) val () = Primitive.MLton.Thread.returnToC () ... changes the output to check starting up libm5 starting up libm4 starting up libm3 starting up libm2 starting up libm1 starting up m1 pointer test complete. m2 pointer test complete. m3 pointer test complete. m4 pointer test complete. m5 pointer test complete. check pointer test complete. lib_close invoked lib_close invoked lib_close invoked lib_close invoked lib_close invoked libm1 exits libm2 exits libm3 exits libm4 exits libm5 exits check exits So it seems that the library suffix doesn't get run until the print statement is added. Tracing the execution of m5_close with gdb confirms this. I know there's some sort of basis-specific code elimination. Is it possible that librarySuffix is getting hit by this? From fluet at tti-c.org Thu Oct 9 20:00:13 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Thu Oct 9 20:01:12 2008 Subject: [MLton] Code being dropped that shouldn't In-Reply-To: <162de7480810080626x6ac88adeg876e4e8db443acdf@mail.gmail.com> References: <162de7480810080626x6ac88adeg876e4e8db443acdf@mail.gmail.com> Message-ID: On Wed, 8 Oct 2008, Wesley W. Terpstra wrote: > I've been trying to track down the last bug in the library regression > test, but I think now it's not a bug in the test at all. Try an i386 > compiler (windows and linux both show the same behaviour). Run: > ./library-test -debug true -keep g -codegen c -profile time ... > I know there's some sort of basis-specific code > elimination. Is it possible that librarySuffix is getting hit by this? No. Turns out that it was a (long-standing) bug with 'MLton_callFromC' in c-main.h. In the library test, there is a point when the call-stack for libm5 looks like: m5_open() libm5.sml (* top-level SML-code, via m5_open trampoline *) libm5confirmC() libm5smlFn{Private,Public}() (* _export-ed from libm5.sml *) MLton_callFromC() libm5.sml (* libm5smlFn{Private,Public}, via MLton_callFromC trampoline *) The _export-ed 'libm5smlFn{Private,Public}' functions terminate the 'MLton_callFromC' trampoline by setting the global 'returnToC' to 'TRUE'. This returns to the top-level SML-code from libm5.sml, executed via the 'm5_open' trampoline. However, the 'returnToC' global is set to 'TRUE', so the trampoline terminates at the first inter-chunk transfer, without completing the top-level SML-code from libm5.sml; in particular, it doesn't execute to the 'suffixArchiveOrLibrary' and doesn't suspend the ML stack at the point just before the 'clean atExit'. Rather, it suspends the ML stack much earlier in the execution, and when the top-level C-code (as executed by check.sml) executes 'm5_close', it starts a new trampoline (setting 'returnToC' to 'FALSE') and resumes the ML code somewhere shortly after the call to 'libm5confirmC()', at which point it executes to the first 'Primitive.MLton.Thread.returnToC' in 'suffixArchiveOrLibrary', returns to 'm5_close'. That explains why the 'clean atExit' never executed. The reasons that this erroneous behavior doesn't occur with all of the C-codegen library tests is that it depends upon how the program is chunkify-ed. This is simply the coarse grouping of RSSA IL functions into larger 'chunks' to create larger/fewer .c files. However, it is a sized-based grouping, so small changes to the RSSA IL (such as the extra code inserted for time profiling) can change the grouping and cause an inter-chunk transfer to manifest the bug. One can cause the bug to manifest in the other C-codegen tests by compiling with '-chunkify func', which uses the finest chunking (increasing the number of inter-chunk transfers). The fix is trivial; set 'returnToC' to 'FALSE' after completing a 'MLton_callFromC' trampoline: Modified: mlton/trunk/include/c-main.h =================================================================== --- mlton/trunk/include/c-main.h 2008-10-09 21:35:20 UTC (rev 6918) +++ mlton/trunk/include/c-main.h 2008-10-10 02:22:13 UTC (rev 6919) @@ -39,6 +39,7 @@ do { \ cont=(*(struct cont(*)(void))cont.nextChunk)(); \ } while (not returnToC); \ + returnToC = FALSE; \ s->atomicState += 1; \ GC_switchToThread (s, GC_getSavedThread (s), 0); \ s->atomicState -= 1; \ From vesa.a.j.k at gmail.com Mon Oct 13 01:22:56 2008 From: vesa.a.j.k at gmail.com (Vesa Karvonen) Date: Mon Oct 13 01:23:34 2008 Subject: [MLton] MLton import headers In-Reply-To: <9e43b9a0810031428v1ecf7232r8c14612f1a51dbf@mail.gmail.com> References: <162de7480810020320s28ffc3d7y42dd4b30abfa6f80@mail.gmail.com> <9e43b9a0810031428v1ecf7232r8c14612f1a51dbf@mail.gmail.com> Message-ID: <9e43b9a0810130122s4aba6a9fi7140d38bcd1dada5@mail.gmail.com> On Sat, Oct 4, 2008 at 12:28 AM, Vesa Karvonen wrote: > On Fri, Oct 3, 2008 at 11:21 PM, Matthew Fluet wrote: >> Finally, importing a MLton library (either static or dynamic) into another >> MLton library or executable seems to be a fairly obscure usage. [...] > > I could certainly imagine some more or less plausible practical > reasons to do so, such as cutting compilation time. Although, given > the limited set of types available for interfacing with such > libraries, many such uses could just as well be written by compiling > the MLton "library" as a regular program that is spawned by the host > program as a separate process and communicated with via some form of > IPC. BTW, according to Armstrong's book this is a preferred way of > using foreign language libraries in Erlang. Continuing on that thought, I just committed an experimental (and somewhat incomplete with respect to what I want it to do) RPC (Remove Procedure Call) library to mltonlib (http://mlton.org/cgi-bin/viewsvn.cgi/mltonlib/trunk/org/mlton/vesak/rpc-lib/unstable/) that allows one to do that fairly easily. There is also simple example of using the library. To try the example, first build it then start the server and finally run the client. -Vesa Karvonen From wesley at terpstra.ca Mon Oct 13 05:17:35 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Oct 13 05:17:58 2008 Subject: [MLton] Address space in windows Message-ID: <162de7480810130517u3ccd0ea6w3979e4215c97236b@mail.gmail.com> I've recently took a look into the address space situation on windows (after cygwin self-compile failed due to it). My machine has 2GB of RAM running on WinXP. MLton/cygwin (use-mmap) can allocate at most 800MB of RAM and MinGW at most 1GB. As MLton needs contiguous address space, I wrote a quick program (attached) to see how big the memory chunks available are. Under cygwin, both mmap and VirtualAlloc provide the exact same available memory. The mmap version is (noticeably) slower, but otherwise identical output: 18660000-60000000: 1201274880 61200000-77DA0000: 381288448 77FE0000-7C800000: 75628544 7C9D0000-7F6F0000: 47316992 60050000-61000000: 16449536 7F7F0000-7FFB0000: 8126464 77EF0000-77FC0000: 851968 003F0000-00400000: 65536 Under MinGW, there is more address space available: 00480000-77BE0000: 2004221952 77C40000-7C800000: 79429632 7C9D0000-7F6F0000: 47316992 7F7F0000-7FFB0000: 8126464 I also tried turning on "large address space" support by adding /3GB to my boot.ini and compiling with --large-address-space. This changes win32 to use 3GB of address space for user processes (normally win32 limits you to 2GB address space regardless of your installed RAM). Unfortunately, MLton cannot take advantage of this because /3GB only adds: 7FFF0000-BFFB0000: 1073479680 to both cygwin and mingw. They have something mapped at the 2GB barrier point, and thus this extra 1G cannot be used contiguously. Therefore, it seems the best we can hope for is 1.2G and a bit under 2.0G heaps under cygwin/mingw respectively. Compared to 800MB and 1GB there is still considerable room for improvement. The reason we get stuck on these smaller limits is because you can't resize to use the full size of the heap while keeping the old heap in memory. I believe this is the purpose of the "dump RAM to disk" trick. However, this trick isn't needed on linux because we use mremap. I propose the following three changes: 1. Eliminate the use of VirtualAlloc/Free on cygwin. mmap can acquire just as much memory and presumably cygwin tracks mmap for use in fork(). 2. Eliminate the tricky reserve before commit mapping done on MinGW (it helps, but not as much as step 3). 3. Implement our own portable mremap. According to SUSv2: "The function munmap() removes any mappings for those entire pages containing any part of the address space of the process starting at addr and continuing for len bytes." ie: If we map pages directly before or after an existing mmap(), one munmap will release both mappings. To grow the map, try adding the new space with a mapping at the end of the existing map. If it fails, binary search to find the maximum expansion we can achieve. Then try to grow the map at the front of the existing map for the remaining required space. If this succeeds, memmove the buffer. I imagine doing this by modifying the remapHeap implementation in gc/heap.c. I'm testing out this idea now, but have no results yet. Comments? From fluet at tti-c.org Mon Oct 13 14:18:33 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Mon Oct 13 14:19:45 2008 Subject: [MLton] Address space in windows In-Reply-To: <162de7480810130517u3ccd0ea6w3979e4215c97236b@mail.gmail.com> References: <162de7480810130517u3ccd0ea6w3979e4215c97236b@mail.gmail.com> Message-ID: On Mon, 13 Oct 2008, Wesley W. Terpstra wrote: > I wrote a quick program (attached) to see how big the memory > chunks available are. Missing... (though, not necessarily important). > I propose the following three changes: > 1. Eliminate the use of VirtualAlloc/Free on cygwin. mmap can acquire > just as much memory and presumably cygwin tracks mmap for use in > fork(). At one point, there were issues with Cygwin's implementation of mmap, which motivated the inclusion of VirtualAlloc/Free. On the other hand, looking at the commit logs, it appears that one does need to use mmap to use Cygwin's fork(). > 3. Implement our own portable mremap. > > According to SUSv2: "The function munmap() removes any mappings for > those entire pages containing any part of the address space of the > process starting at addr and continuing for len bytes." I guess you are arguing that SUSv2 munmap() suffices to batch free any (contiguous) set of mmap-s; thus, one doesn't need to change the Cygwin implementation. > ie: If we map pages directly before or after an existing mmap(), one > munmap will release both mappings. To grow the map, try adding the new > space with a mapping at the end of the existing map. If it fails, > binary search to find the maximum expansion we can achieve. Then try > to grow the map at the front of the existing map for the remaining > required space. If this succeeds, memmove the buffer. I'm guessing here, but I would suspect that a potential downside of this over a 'native' mmap is that we could end up with a large number of mmap-s, as a heap is grown in place little by little. It also misses the opportunity to move the region whole-sale to a completely different address (more likely to be a possibility in a 64-bit environment). Nonetheless, I agree that this would appear to give the right mremap semantics. One could even attempt this mremap emulation on *BSD/darwin, where no mremap exists. One needs to be careful to use MAP_FIXED in order to get the new mappings in exactly the right places. > I imagine doing this by modifying the remapHeap implementation in > gc/heap.c. I'm testing out this idea now, but have no results yet. Why modify remapHeap in gc/heap.c? You should do the changes in platform/*, adding/modifying the implementation of the the GC_mremap function. One could also perform a related experiment, where one uses 'malloc/realloc/free' in place of 'mmap/mremap/munmap'. The former are all required by POSIX (whereas mremap is only available natively on linux). One issue is that realloc (according to the spec) could move the allocation, even if the reallocation is shrinking the allocation. There are situations (e.g., shrinkHeap), where we want to release the unused portion of the heap, without moving the heap (because moving the heap requires translating all the heap pointers). From vesa.a.j.k at gmail.com Mon Oct 13 16:31:10 2008 From: vesa.a.j.k at gmail.com (Vesa Karvonen) Date: Mon Oct 13 16:31:14 2008 Subject: [MLton] Phantom typing of sockets broken in MLton Message-ID: <9e43b9a0810131631u402c7273hbc375537b223b8cb@mail.gmail.com> Just a quick note before going to bed. Phantom typing of sockets seems broken in MLton. Here is a small example that should not pass type checking (but does in MLton): val socket = INetSock.TCP.socket () val () = Socket.listen (socket, 1) val _ = Socket.recvVec (socket, 1) Both SML/NJ and Poly/ML reject this example. -Vesa Karvonen From wesley at terpstra.ca Mon Oct 13 21:01:29 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Oct 13 21:01:33 2008 Subject: [MLton] Address space in windows In-Reply-To: References: <162de7480810130517u3ccd0ea6w3979e4215c97236b@mail.gmail.com> Message-ID: <162de7480810132101l26a0c12escacc5eeae9656bba@mail.gmail.com> On Mon, Oct 13, 2008 at 11:18 PM, Matthew Fluet wrote: > On Mon, 13 Oct 2008, Wesley W. Terpstra wrote: >> 1. Eliminate the use of VirtualAlloc/Free on cygwin. mmap can acquire >> just as much memory and presumably cygwin tracks mmap for use in >> fork(). > > At one point, there were issues with Cygwin's implementation of mmap, which > motivated the inclusion of VirtualAlloc/Free. On the other hand, looking at > the commit logs, it appears that one does need to use mmap to use Cygwin's > fork(). Seeing as how fork() is the only reason to use cygwin over mingw... > I guess you are arguing that SUSv2 munmap() suffices to batch free any > (contiguous) set of mmap-s; thus, one doesn't need to change the Cygwin > implementation. Or any SUSv2 system (bsd, solaris, etc). We can safely create a big mapping from little ones and it should be completely compatible AFAICT. > I'm guessing here, but I would suspect that a potential downside of this > over a 'native' mmap is that we could end up with a large number of mmap-s, > as a heap is grown in place little by little. Yes. I'm not sure what the downside of many small mappings is vs. one big mapping. Is there a performance penalty? It's all anonymous memory. Does mmap'ing two pages side-by-side differ from one mmap of two pages? On windows, yes. On *nix? Isn't this how the C heap is grown? > It also misses the opportunity to move the region whole-sale to a completely > different address > (more likely to be a possibility in a 64-bit environment). Nonetheless, I > agree that this would appear to give the right mremap semantics. We could easily include a moving copy in the fake mremap. eg: try first to expand in-place, if that fails, try an out-of-place mmap+memmove. > One could even attempt this mremap emulation on *BSD/darwin, where no mremap > exists. Yes. > One needs to be careful to use MAP_FIXED in order to get the new mappings in > exactly the right places. No. Using MAP_FIXED overwrites existing maps. That would cause problems. At least cygwin and linux mmap honour the named address (if it's possible) without MAP_FIXED. If there is another mapping in the way, it will either fail or pick a different address... we can detect both. > Why modify remapHeap in gc/heap.c? You should do the changes in platform/*, > adding/modifying the implementation of the the GC_mremap function. The gc/heap.c has an existing back-off algorithm. I didn't want to repeat the scan for maximum available size on each back-off attempt. However, I guess that a simple cache of the largest successful probe would work. Certainly it's cleaner. I'm refactoring my patch with this in mind. > One could also perform a related experiment, where one uses > 'malloc/realloc/free' in place of 'mmap/mremap/munmap'. The former are all > required by POSIX (whereas mremap is only available natively on linux). One > issue is that realloc (according to the spec) could move the allocation, > even if the reallocation is shrinking the allocation. There are situations > (e.g., shrinkHeap), where we want to release the unused portion of the heap, > without moving the heap (because moving the heap requires translating all > the heap pointers). Interesting. I notice that translateHeap is free if from == to. So adding translateHeaps after every shrinkHeap would only cost if an implementation actually DID move on resize. I've never seriously used realloc for anything this big. Quick testing on linux reveals: 0804A008-B7CFFFFC: 2949341172 00001008-08047FFC: 134508532 B7F8D008-BFD76FFC: 132030452 BFD8C008-BFFFFFFC: 2572276 B7D00468-B7DFFFF4: 1047436 B7F5D008-B7F6CFFC: 65524 B7E00008-B7E00FFC: 4084 => 3219569476 for malloc and 0804A000-B7E36000: 2950610944 00001000-08048000: 134508544 B7FC2000-BF9AC000: 127836160 BF9C1000-C0000000: 6549504 B7F92000-B7FA2000: 65536 => 3219570688 for mmap. So they both can access more-or-less than same amount of memory. Unfortunately, realloc from 2048000000U to 2950610944U fails. So it may be portable, but realloc doesn't work in-place for large memory. -------------- next part -------------- A non-text attachment was scrubbed... Name: scan.c Type: text/x-csrc Size: 1311 bytes Desc: not available Url : http://mlton.org/pipermail/mlton/attachments/20081014/62326eea/scan.c From wesley at terpstra.ca Wed Oct 15 07:48:53 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Oct 15 07:49:00 2008 Subject: [MLton] page-heap or may-page-heap? Message-ID: <162de7480810150748q66ee8670mb917b2359a8c11fa@mail.gmail.com> The documentation and error messages refer to the @MLton option may-page-heap. However, the actual runtime accepts 'page-heap' not may-page-heap. Which is intended? From fluet at tti-c.org Wed Oct 15 08:04:35 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Oct 15 08:05:52 2008 Subject: [MLton] Phantom typing of sockets broken in MLton In-Reply-To: <9e43b9a0810131631u402c7273hbc375537b223b8cb@mail.gmail.com> References: <9e43b9a0810131631u402c7273hbc375537b223b8cb@mail.gmail.com> Message-ID: On Tue, 14 Oct 2008, Vesa Karvonen wrote: > Just a quick note before going to bed. Phantom typing of sockets > seems broken in MLton. Here is a small example that should not pass > type checking (but does in MLton): > > val socket = INetSock.TCP.socket () > val () = Socket.listen (socket, 1) > val _ = Socket.recvVec (socket, 1) > > Both SML/NJ and Poly/ML reject this example. I think the issue is that basis-library/net/socket.{sig,sml} has structure Socket : SOCKET_EXTRA = ... and basis-library/libs/basis-extra/basis.{sig,sml} has signature BASIS_EXTRA = sig ... end ... where type ('a, 'b) Socket.sock = ('a, 'b) Socket.sock ... structure BasisExtra :> BASIS_EXTRA The non-opaque signature constraint on Socket allows the phantomness of the type-arguments to ('a, 'b) Socket.sock to leak out as a transparent type, and the where type... propagates the transparent type out to user code. Using Socket :> SOCKET_EXTRA leads to lots of type errors in the Basis Library; Deleting where type ('a, 'b) Socket.sock = ('a, 'b) Socket.sock doesn't lead to type errors in the Basis Library itself, but does lose the type equivalence between the visible Socket.sock and the Socket.sock type mentioned in INetSock and UnixSock. From fluet at tti-c.org Wed Oct 15 08:15:44 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Wed Oct 15 08:17:13 2008 Subject: [MLton] page-heap or may-page-heap? In-Reply-To: <162de7480810150748q66ee8670mb917b2359a8c11fa@mail.gmail.com> References: <162de7480810150748q66ee8670mb917b2359a8c11fa@mail.gmail.com> Message-ID: On Wed, 15 Oct 2008, Wesley W. Terpstra wrote: > The documentation and error messages refer to the @MLton option > may-page-heap. However, the actual runtime accepts 'page-heap' not > may-page-heap. Which is intended? >From the commit log, it looks like 'may-page-heap' was intended. From wesley at terpstra.ca Wed Oct 15 11:51:04 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Oct 15 11:51:09 2008 Subject: [MLton] Address space in windows In-Reply-To: <162de7480810132101l26a0c12escacc5eeae9656bba@mail.gmail.com> References: <162de7480810130517u3ccd0ea6w3979e4215c97236b@mail.gmail.com> <162de7480810132101l26a0c12escacc5eeae9656bba@mail.gmail.com> Message-ID: <162de7480810151151i3fceb9a5r55f854790050f7f8@mail.gmail.com> On Tue, Oct 14, 2008 at 6:01 AM, Wesley W. Terpstra wrote: >>> 1. Eliminate the use of VirtualAlloc/Free on cygwin I have elected not to do this. While it is true that mmap and VA can grab the same memory, cygwin likes to re-use munmap'd memory ASAP (probably because it just decommits it like us). However, this means that if MLton ever shrinks it's heap all it takes is one allocation by cygwin to make it impossible for us to recover the memory we gave up. When using VA, this fragmentation doesn't happen because cygwin doesn't know about our decommitted memory. Both use-mmap and the default (VA) acquire > 1.1G of RAM now without problems on cygwin. On MinGW, we easily grab >1.9GB of memory. As a side note, the reason cygwin memory is smaller is due to the unfortunate position of the cygwin DLLs at address: 61000000-61200000 r-xs 61054740 8CBD:AB20 1125899906875461 /usr/bin/cygwin1.dll The limit to 2G on MinGW comes from the windows system DLLs which are loaded around 7FFF0000. Another thing I've noticed is that MLton's estimate of the available RAM can trick it into making bad choices for the heap size. You can only possibly get 1.2G on cygwin, but MLton thinks it should be able to get an amount = physical ram. It sets the minSize and desiredSize both too high and is the remap then fails. However, if it had a less aggressive minSize, it could have allocated another 300MB. I'm not sure what to do about this. Perhaps cap the physical RAM based on the platform? > Yes. I'm not sure what the downside of many small mappings is vs. one > big mapping. Is there a performance penalty? It's all anonymous > memory. Does mmap'ing two pages side-by-side differ from one mmap of > two pages? On windows, yes. On *nix? Isn't this how the C heap is > grown? These questions all remain open. > We could easily include a moving copy in the fake mremap. eg: try > first to expand in-place, if that fails, try an out-of-place > mmap+memmove. My patch now does this, so "fragmentation" with many concatenated maps only happens if you really need mremap to do it. From vesa.a.j.k at gmail.com Wed Oct 15 12:42:14 2008 From: vesa.a.j.k at gmail.com (Vesa Karvonen) Date: Wed Oct 15 12:42:18 2008 Subject: [MLton] Phantom typing of sockets broken in MLton In-Reply-To: References: <9e43b9a0810131631u402c7273hbc375537b223b8cb@mail.gmail.com> Message-ID: <9e43b9a0810151242rce15d91yd8134537c141d780@mail.gmail.com> On Wed, Oct 15, 2008 at 6:04 PM, Matthew Fluet wrote: [...] > Using Socket :> SOCKET_EXTRA leads to lots of type errors in the Basis > Library; Deleting where type ('a, 'b) Socket.sock = ('a, 'b) Socket.sock > doesn't lead to type errors in the Basis Library itself, but does lose the > type equivalence between the visible Socket.sock and the Socket.sock type > mentioned in INetSock and UnixSock. Yeah, I think that the easiest way to fix this is to expose some more stuff in the SOCKET_EXTRA signature. Below is a patch proposal (which I haven't yet fully verified to be correct). -Vesa Karvonen Index: net/socket.sml =================================================================== --- net/socket.sml (revision 6920) +++ net/socket.sml (working copy) @@ -5,7 +5,7 @@ * See the file MLton-LICENSE for details. *) -structure Socket : SOCKET_EXTRA = +structure Socket :> SOCKET_EXTRA = struct structure Prim = PrimitiveFFI.Socket @@ -15,6 +15,8 @@ structure Sock = Net.Sock type sock = Sock.t +val fromRep = Sock.fromRep +val toRep = Sock.toRep val sockToWord = C_Sock.castToSysWord o Sock.toRep val wordToSock = Sock.fromRep o C_Sock.castFromSysWord val sockToFD = PrePosix.FileDesc.fromRep o Sock.toRep @@ -64,6 +66,7 @@ structure SOCK = struct type sock_type = SockType.t + val toRep = SockType.toRep val stream = SockType.fromRep Prim.SOCK.STREAM val dgram = SockType.fromRep Prim.SOCK.DGRAM val names : (string * sock_type) list = @@ -80,6 +83,7 @@ SOME (_, st) => SOME st | NONE => NONE end +structure SOCKExtra = SOCK structure CtlExtra = struct Index: net/generic-sock.sml =================================================================== --- net/generic-sock.sml (revision 6920) +++ net/generic-sock.sml (working copy) @@ -12,19 +12,19 @@ structure PESC = PE.SysCall fun socket' (af, st, p) = - (Net.Sock.fromRep o PESC.simpleResult) + (Socket.fromRep o PESC.simpleResult) (fn () => Prim.socket (Net.AddrFamily.toRep af, - Net.SockType.toRep st, + Socket.SOCKExtra.toRep st, C_Int.fromInt p)) fun socketPair' (af, st, p) = let val a : C_Sock.t array = Array.array (2, C_Sock.fromInt 0) - val get = fn i => Net.Sock.fromRep (Array.sub (a, i)) + val get = fn i => Socket.fromRep (Array.sub (a, i)) in PESC.syscall (fn () => (Prim.socketPair (Net.AddrFamily.toRep af, - Net.SockType.toRep st, + Socket.SOCKExtra.toRep st, C_Int.fromInt p, a), fn _ => (get 0, get 1))) Index: net/socket.sig =================================================================== --- net/socket.sig (revision 6920) +++ net/socket.sig (working copy) @@ -165,14 +165,21 @@ signature SOCKET_EXTRA = sig include SOCKET + val fromRep : C_Sock.t -> ('af, 'sock_type) sock + val toRep : ('af, 'sock_type) sock -> C_Sock.t val sockToWord: ('af, 'sock_type) sock -> SysWord.word val wordToSock: SysWord.word -> ('af, 'sock_type) sock val sockToFD: ('af, 'sock_type) sock -> Posix.FileSys.file_desc val fdToSock: Posix.FileSys.file_desc -> ('af, 'sock_type) sock - type pre_sock_addr + type pre_sock_addr = Word8.word array val unpackSockAddr: 'af sock_addr -> Word8.word vector val newSockAddr: unit -> (pre_sock_addr * C_Socklen.t ref * (unit -> 'af sock_addr)) + structure SOCKExtra: + sig + val toRep : SOCK.sock_type -> C_Sock.t + end + structure CtlExtra: sig type level = C_Int.int From vesa.a.j.k at gmail.com Thu Oct 16 00:05:50 2008 From: vesa.a.j.k at gmail.com (Vesa Karvonen) Date: Thu Oct 16 00:05:54 2008 Subject: [MLton] Attempt to build MLton with 20070826 release fails with "CreateFile failed with error 80" Message-ID: <9e43b9a0810160005x721a7e83wcb2b6c9fe94bbc25@mail.gmail.com> I just tried to build MLton from SVN with the 20070826 release under Cygwin, but it fails with "CreateFile failed with error 80". Looking at the SVN logs for windows.c (rev 6624) in the runtime it seems that this is a known problem that was fixed some time ago (indeed I can see temporary files created in the root of the drive). Is there a way around this issue? I suppose I could try building via SML/NJ. Does someone have a more recent build of MLton for Cygwin? -Vesa Karvonen From wesley at terpstra.ca Thu Oct 16 05:58:29 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Thu Oct 16 05:58:43 2008 Subject: [MLton] Attempt to build MLton with 20070826 release fails with "CreateFile failed with error 80" In-Reply-To: <9e43b9a0810160005x721a7e83wcb2b6c9fe94bbc25@mail.gmail.com> References: <9e43b9a0810160005x721a7e83wcb2b6c9fe94bbc25@mail.gmail.com> Message-ID: <162de7480810160558q34253828v4542ee7e666cdb16@mail.gmail.com> On Thu, Oct 16, 2008 at 9:05 AM, Vesa Karvonen wrote: > I just tried to build MLton from SVN with the 20070826 release under > Cygwin, but it fails with "CreateFile failed with error 80". I worked around it by cross-compiling from MinGW. Your other option is to download the build I jst put on Experimental. It also has enough address space to do a self-compile. Please report any strange segfaults you see. I ironed out a lot of problems already with the mremap code, but it's quite possible I missed some. From vesa.a.j.k at gmail.com Thu Oct 16 10:00:56 2008 From: vesa.a.j.k at gmail.com (Vesa Karvonen) Date: Thu Oct 16 10:00:59 2008 Subject: [MLton] Attempt to build MLton with 20070826 release fails with "CreateFile failed with error 80" In-Reply-To: <162de7480810160558q34253828v4542ee7e666cdb16@mail.gmail.com> References: <9e43b9a0810160005x721a7e83wcb2b6c9fe94bbc25@mail.gmail.com> <162de7480810160558q34253828v4542ee7e666cdb16@mail.gmail.com> Message-ID: <9e43b9a0810161000n77349fe0pcf44c78b782b33a0@mail.gmail.com> On Thu, Oct 16, 2008 at 3:58 PM, Wesley W. Terpstra wrote: > On Thu, Oct 16, 2008 at 9:05 AM, Vesa Karvonen wrote: >> I just tried to build MLton from SVN with the 20070826 release under >> Cygwin, but it fails with "CreateFile failed with error 80". > > I worked around it by cross-compiling from MinGW. Your other option is > to download the build I jst put on Experimental. It also has enough > address space to do a self-compile. Thanks! BTW, I tried building with nj-mlton. While I could build nj-mlton itself with SML/NJ, building MLton with nj-mlton failed (to allocate enough memory it seems). > Please report any strange segfaults you see. I ironed out a lot of > problems already with the mremap code, but it's quite possible I > missed some. Will do. -Vesa Karvonen From wesley at terpstra.ca Thu Oct 16 13:28:57 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Thu Oct 16 13:29:00 2008 Subject: [MLton] Library support mostly complete Message-ID: <162de7480810161328j63ce079ck5a4394e2fb5a880f@mail.gmail.com> The script regression/library/library-all passes on all of these platforms: cygwin i386 darwin ppc/i386/amd64 linux i386/amd64 mingw i386/amd64 I don't have access to solaris / aix / freebsd, so I'm looking for volunteers to compile a svn/head MLton and run the script on those platforms. I've written some prototype documentation on mlton.org/LibrarySupport. Where else should it be documented? What else needs to be covered? I have a patch to add a 'defaultImport' and 'defaultExport' annotation, but haven't decided whether or not to include it. So hopefully library support is no longer an issue blocking release of a new version. From vesa.a.j.k at gmail.com Tue Oct 21 23:47:03 2008 From: vesa.a.j.k at gmail.com (Vesa Karvonen) Date: Tue Oct 21 23:47:06 2008 Subject: [MLton] Library support mostly complete In-Reply-To: <162de7480810161328j63ce079ck5a4394e2fb5a880f@mail.gmail.com> References: <162de7480810161328j63ce079ck5a4394e2fb5a880f@mail.gmail.com> Message-ID: <9e43b9a0810212347y5329d8a4i294931400a835b61@mail.gmail.com> On Thu, Oct 16, 2008 at 11:28 PM, Wesley W. Terpstra wrote: [...new library support...] > I've written some prototype documentation on mlton.org/LibrarySupport. > Where else should it be documented? What else needs to be covered? I took a brief look at the documentation and it seemed fine to me. One thing that might further help would be to include a full example including build files (MLBs, Makefile/build script --- not just the C and MLton code snippets) so that one can see an example usage of the commands (options) required to compile and link libraries and see what files are produced as a result (this would be most useful for newbie and casual MLton users). FYI, I've been meaning to try out the new library support for some time now, but I've been too busy. I've been thinking about trying to build a SML utility library (based on pickling) that would allow one to call arbitrary SML functions (including higher-order and possibly polymorphic functions, but with explicit typing) defined in a separately compiled MLton library. -Vesa Karvonen From ville at laurikari.net Wed Oct 22 01:24:45 2008 From: ville at laurikari.net (Ville Laurikari) Date: Wed Oct 22 01:25:34 2008 Subject: [MLton] Library support mostly complete In-Reply-To: <162de7480810161328j63ce079ck5a4394e2fb5a880f@mail.gmail.com> References: <162de7480810161328j63ce079ck5a4394e2fb5a880f@mail.gmail.com> Message-ID: <20081022082445.GA30130@laurikari.net> On Thu, Oct 16, 2008 at 10:28:57PM +0200, Wesley W. Terpstra wrote: > I don't have access to solaris / aix / freebsd, so I'm looking for > volunteers to compile a svn/head MLton and run the script on those > platforms. GCC does not implement the visibility options on AIX and complains loudly when trying to use them. On AIX, hiding symbols in shared objects is done at link time by supplying a list of exported symbols in an "export file" to the linker. There's also similar "import file" mechanism to identify the external symbols to resolve at run time. Luckily, GCC normally uses the system ld (and not GNU ld) on AIX, so it should be possible to get things to work by using "-Wl,something" with gcc when linking the shared libs. MLton would have to generate export files, and the runtime should also have the export file. Perhaps the runtime export file could be created automatically with a hack involving the C preprocessor and a small program or script to parse it. There are the additional small problems of "-debug true" being hardcoded to stabs+ which is not supported on AIX, and an incorrect shared library suffix. These are easy. Note that I will have very limited time to work on this, so it's not going to get done any time soon (by me at least). I'll try HP-UX next. -- http://www.iki.fi/vl/ From wesley at terpstra.ca Wed Oct 22 04:09:00 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Oct 22 04:09:09 2008 Subject: [MLton] Library support mostly complete In-Reply-To: <20081022082445.GA30130@laurikari.net> References: <162de7480810161328j63ce079ck5a4394e2fb5a880f@mail.gmail.com> <20081022082445.GA30130@laurikari.net> Message-ID: <162de7480810220409v8f73f66u3889aaac2b0800d3@mail.gmail.com> Thanks for the report. On Wed, Oct 22, 2008 at 10:24 AM, Ville Laurikari wrote: > GCC does not implement the visibility options on AIX and complains > loudly when trying to use them. Are you using gcc > 4.0? What is the error it reports? > On AIX, hiding symbols in shared > objects is done at link time by supplying a list of exported symbols > in an "export file" to the linker. There's also similar "import file" > mechanism to identify the external symbols to resolve at run time. Is AIX not ELF? Why didn't we need an import file when building executables? > There are the additional small problems of "-debug true" being > hardcoded to stabs+ which is not supported on AIX, and an incorrect > shared library suffix. These are easy. What the correct options for debug on AIX? I guess these options should be moved into the mlton-script. What is the extension for a shared library? From wesley at terpstra.ca Wed Oct 22 04:32:50 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Oct 22 04:33:02 2008 Subject: [MLton] Library support mostly complete In-Reply-To: <162de7480810220409v8f73f66u3889aaac2b0800d3@mail.gmail.com> References: <162de7480810161328j63ce079ck5a4394e2fb5a880f@mail.gmail.com> <20081022082445.GA30130@laurikari.net> <162de7480810220409v8f73f66u3889aaac2b0800d3@mail.gmail.com> Message-ID: <162de7480810220432l1ba2111aife47ed4a01a2844d@mail.gmail.com> On Wed, Oct 22, 2008 at 1:09 PM, Wesley W. Terpstra wrote: > Is AIX not ELF? Ok. I found some documentation. It's XCOFF. You're quite right that this platform will require some tweaking. Is there a public AIX compile farm anywhere? From ville at laurikari.net Wed Oct 22 04:48:21 2008 From: ville at laurikari.net (Ville Laurikari) Date: Wed Oct 22 04:48:27 2008 Subject: [MLton] Library support mostly complete In-Reply-To: <162de7480810220432l1ba2111aife47ed4a01a2844d@mail.gmail.com> References: <162de7480810161328j63ce079ck5a4394e2fb5a880f@mail.gmail.com> <20081022082445.GA30130@laurikari.net> <162de7480810220409v8f73f66u3889aaac2b0800d3@mail.gmail.com> <162de7480810220432l1ba2111aife47ed4a01a2844d@mail.gmail.com> Message-ID: <20081022114821.GA4156@laurikari.net> On Wed, Oct 22, 2008 at 01:32:50PM +0200, Wesley W. Terpstra wrote: > Ok. I found some documentation. It's XCOFF. You're quite right that > this platform will require some tweaking. Is there a public AIX > compile farm anywhere? Quick googling found http://www.polarhome.com/, they apparently have some sort of an AIX box. It appears to be quite old, though. I have access to AIX at work, so I haven't looked very hard. In any case, I wouldn't treat lack of library support as a release blocker for MLton. -- Ville From ville at laurikari.net Wed Oct 22 05:06:59 2008 From: ville at laurikari.net (Ville Laurikari) Date: Wed Oct 22 05:07:04 2008 Subject: [MLton] Library support mostly complete In-Reply-To: <162de7480810220409v8f73f66u3889aaac2b0800d3@mail.gmail.com> References: <162de7480810161328j63ce079ck5a4394e2fb5a880f@mail.gmail.com> <20081022082445.GA30130@laurikari.net> <162de7480810220409v8f73f66u3889aaac2b0800d3@mail.gmail.com> Message-ID: <20081022120659.GB4156@laurikari.net> On Wed, Oct 22, 2008 at 01:09:00PM +0200, Wesley W. Terpstra wrote: > Are you using gcc > 4.0? What is the error it reports? GCC 4.2.2, it says: gc/int-inf.c: In function 'initIntInf': gc/int-inf.c:423: warning: visibility attribute not supported in this configuration; ignored ad nauseaum. > Why didn't we need an import file when building executables? It appears that importing external symbols works largely automatically with both the IBM and GCC toolchains. Perhaps there are cases where one needs to create import files explicitly, but I'm not familiar enough with the system to tell what these cases could be. > What is the extension for a shared library? On AIX, all libraries are packaged into regular archives files which bear the .a suffix. The .a archives then contain objects or shared objects. So, a shared library is typically created by building the .so with "gcc -shared" and then wrapping into a .a using "ar". -- Ville From Nicolas.Bertolotti at mathworks.fr Wed Oct 22 05:43:17 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Wed Oct 22 05:43:54 2008 Subject: [MLton] Compilation crash with FFI on Windows with latest SVN revision (r6960) Message-ID: <8320D98DA9A5C54C926D397795FE7CEA2D98BAB46C@EXCHANGE-UK.ad.mathworks.com> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 4473 bytes Desc: image001.gif Url : http://mlton.org/pipermail/mlton/attachments/20081022/f7c23b74/image001-0001.gif From wesley at terpstra.ca Wed Oct 22 06:42:12 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Oct 22 06:42:17 2008 Subject: [MLton] Compilation crash with FFI on Windows with latest SVN revision (r6960) In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA2D98BAB46C@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA2D98BAB46C@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480810220642s4fad0821oca472b9ffe586b94@mail.gmail.com> On Wed, Oct 22, 2008 at 2:43 PM, Nicolas Bertolotti wrote: > It seems that the latest SVN revision prefixes all symbols with ? __imp ? > during code generation. That's correct. > This includes FFI functions which do not use the stdcall convention which > does not appear to be correct (simple C function defined in an external C > file that is given to MLton together with the SML sources). You are using the wrong symbol scope. Please change your _import "somefn" : ... to _import "somefn" public : ... The problem you see is because _import default to external, which means it needs __imp__ to be fetched from a DLL. See http://mlton.org/LibrarySupport for some (bad) documentation. > Anyway, this is just a tree in the forest (I'm not sure this expression > makes sense in english) because the resulting binary simply freezes. Can you provide more details? The linking problem you describe should be unrelated to the freeze. What was the programming doing when it froze? > Please note that this regression is quite recent as the binary used to run > almost fine using revision 6698 (I know it is the one that introduced an > effective Win64 support but I am really talking about Win32 binaries). > > ? and when I say "almost", this is because we experience some sporadic > failures (the binary freezes) during FFI calls on some machines. If it froze before, it's quite possible that recent changes just made this sporadic behavoiur ... more reliable. ;-) From Nicolas.Bertolotti at mathworks.fr Wed Oct 22 07:13:43 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Wed Oct 22 07:14:20 2008 Subject: [MLton] Compilation crash with FFI on Windows with latest SVN revision (r6960) In-Reply-To: <162de7480810220642s4fad0821oca472b9ffe586b94@mail.gmail.com> References: <8320D98DA9A5C54C926D397795FE7CEA2D98BAB46C@EXCHANGE-UK.ad.mathworks.com> <162de7480810220642s4fad0821oca472b9ffe586b94@mail.gmail.com> Message-ID: <8320D98DA9A5C54C926D397795FE7CEA2D98BAB4D2@EXCHANGE-UK.ad.mathworks.com> I have never heard about those "public" and "external" scopes. Have they been introduced recently ? The same piece of code used to build and run fine with MLton 2007. That's why I am asking (in my point of view, it can be considered as a regression). I am going to perform some additional tests and let you know. Thanks Nicolas > -----Original Message----- > From: Wesley W. Terpstra [mailto:wesley@terpstra.ca] > Sent: Wednesday, October 22, 2008 3:42 PM > To: Nicolas Bertolotti > Cc: mlton@mlton.org > Subject: Re: [MLton] Compilation crash with FFI on Windows with latest SVN > revision (r6960) > > On Wed, Oct 22, 2008 at 2:43 PM, Nicolas Bertolotti > wrote: > > It seems that the latest SVN revision prefixes all symbols with < __imp > > > > during code generation. > > That's correct. > > > This includes FFI functions which do not use the stdcall convention > which > > does not appear to be correct (simple C function defined in an external > C > > file that is given to MLton together with the SML sources). > > You are using the wrong symbol scope. Please change your _import > "somefn" : ... to _import "somefn" public : ... The problem you see is > because _import default to external, which means it needs __imp__ to > be fetched from a DLL. See http://mlton.org/LibrarySupport for some > (bad) documentation. > > > Anyway, this is just a tree in the forest (I'm not sure this expression > > makes sense in english) because the resulting binary simply freezes. > > Can you provide more details? The linking problem you describe should > be unrelated to the freeze. What was the programming doing when it > froze? > > > Please note that this regression is quite recent as the binary used to > run > > almost fine using revision 6698 (I know it is the one that introduced an > > effective Win64 support but I am really talking about Win32 binaries). > > > > ... and when I say "almost", this is because we experience some sporadic > > failures (the binary freezes) during FFI calls on some machines. > > If it froze before, it's quite possible that recent changes just made > this sporadic behavoiur ... more reliable. ;-) From Nicolas.Bertolotti at mathworks.fr Wed Oct 22 11:01:06 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Wed Oct 22 11:01:54 2008 Subject: [MLton] Compilation crash with FFI on Windows with latest SVN revision (r6960) In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA2D98BAB4D2@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA2D98BAB46C@EXCHANGE-UK.ad.mathworks.com> <162de7480810220642s4fad0821oca472b9ffe586b94@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA2D98BAB4D2@EXCHANGE-UK.ad.mathworks.com> Message-ID: <8320D98DA9A5C54C926D397795FE7CEA2D98BAB579@EXCHANGE-UK.ad.mathworks.com> Good news first ... I modified my code to use the "public" scope where this was needed and removed my patch in x86-generate-transfers.fun. Now, the code runs (almost) fine using MLton r6960. I am still surprised that "public" is not the default as the new behavior introduces an incompatibility with the existing official release. I think that most users generally simply want to write a small piece of C code and compile the .c file together with their SML files where they import the corresponding function. I am afraid that the new default behavior is not as intuitive as the previous one. Now, my binary still freezes at the end of its execution under some circumstances. The issue occurs when it calls, from the SML code, a function that is included in a Windows DLL. I added some "print" and the last message I get is the one I put just before the call to the import. The C function itself, which also contains a print at the very beginning, never prints anything so I guess it is not even called. In my big application, the DLL publishes 2 functions. The call to the first one is well executed and the binary freezes when it tries to call the second one. I tried to reproduce the problem on a small sample but it unfortunately didn't work. Anyway, you will find it as an attachment. It basically shows how the code is designed. If the sample did reproduce the issue, it would print the message "Calling foo2()..." and then freeze. I am afraid there are a number of things the big application does which cause it to go through other parts of the x86 code generator and generate some kind of memory or stack corruption that can not be easily reproduced with a small sample. Thanks for your help Nicolas > -----Original Message----- > From: mlton-bounces@mlton.org [mailto:mlton-bounces@mlton.org] On Behalf > Of Nicolas Bertolotti > Sent: Wednesday, October 22, 2008 4:14 PM > To: Wesley W. Terpstra > Cc: mlton@mlton.org > Subject: RE: [MLton] Compilation crash with FFI on Windows with latest SVN > revision (r6960) > > I have never heard about those "public" and "external" scopes. Have they > been introduced recently ? > > The same piece of code used to build and run fine with MLton 2007. That's > why I am asking (in my point of view, it can be considered as a > regression). > > I am going to perform some additional tests and let you know. > > Thanks > > Nicolas > > > -----Original Message----- > > From: Wesley W. Terpstra [mailto:wesley@terpstra.ca] > > Sent: Wednesday, October 22, 2008 3:42 PM > > To: Nicolas Bertolotti > > Cc: mlton@mlton.org > > Subject: Re: [MLton] Compilation crash with FFI on Windows with latest > SVN > > revision (r6960) > > > > On Wed, Oct 22, 2008 at 2:43 PM, Nicolas Bertolotti > > wrote: > > > It seems that the latest SVN revision prefixes all symbols with < > __imp > > > > > > during code generation. > > > > That's correct. > > > > > This includes FFI functions which do not use the stdcall convention > > which > > > does not appear to be correct (simple C function defined in an > external > > C > > > file that is given to MLton together with the SML sources). > > > > You are using the wrong symbol scope. Please change your _import > > "somefn" : ... to _import "somefn" public : ... The problem you see is > > because _import default to external, which means it needs __imp__ to > > be fetched from a DLL. See http://mlton.org/LibrarySupport for some > > (bad) documentation. > > > > > Anyway, this is just a tree in the forest (I'm not sure this > expression > > > makes sense in english) because the resulting binary simply freezes. > > > > Can you provide more details? The linking problem you describe should > > be unrelated to the freeze. What was the programming doing when it > > froze? > > > > > Please note that this regression is quite recent as the binary used to > > run > > > almost fine using revision 6698 (I know it is the one that introduced > an > > > effective Win64 support but I am really talking about Win32 binaries). > > > > > > ... and when I say "almost", this is because we experience some > sporadic > > > failures (the binary freezes) during FFI calls on some machines. > > > > If it froze before, it's quite possible that recent changes just made > > this sporadic behavoiur ... more reliable. ;-) > > _______________________________________________ > MLton mailing list > MLton@mlton.org > http://mlton.org/mailman/listinfo/mlton -------------- next part -------------- A non-text attachment was scrubbed... Name: essai.zip Type: application/x-zip-compressed Size: 1736 bytes Desc: essai.zip Url : http://mlton.org/pipermail/mlton/attachments/20081022/4e47a582/essai.bin From wesley at terpstra.ca Wed Oct 22 13:09:42 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Oct 22 13:09:47 2008 Subject: [MLton] Compilation crash with FFI on Windows with latest SVN revision (r6960) In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA2D98BAB579@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA2D98BAB46C@EXCHANGE-UK.ad.mathworks.com> <162de7480810220642s4fad0821oca472b9ffe586b94@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA2D98BAB4D2@EXCHANGE-UK.ad.mathworks.com> <8320D98DA9A5C54C926D397795FE7CEA2D98BAB579@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480810221309u20d61b3fs115d3cd7b83a17f0@mail.gmail.com> On Wed, Oct 22, 2008 at 8:01 PM, Nicolas Bertolotti wrote: > Now, the code runs (almost) fine using MLton r6960. I am still surprised that "public" is not the default as the new behavior introduces an incompatibility with the existing official release. Well, picking 'public' as the default would break _import from libraries. Either way something that used to "work" won't. As the old _import didn't have public/external, you would have problems if you tried to import a variable (as opposed to a function) or take the address of functions. The new syntax at least gives you a compile error as opposed to a bug that lurks in wait. =) > I think that most users generally simply want to write a small piece of C code and compile the .c file together with their SML files where they import the corresponding function. I am afraid that the new default behavior is not as intuitive as the previous one. Maybe. It's also pretty usual to do _import "sqlite3_open" or similar, though. > The issue occurs when it calls, from the SML code, a function that is included in a Windows DLL. Is this function maybe an stdcall function? Try _import "fn_in_a_dll" external stdcall : ... > In my big application, the DLL publishes 2 functions. The call to the first one is well executed and the binary freezes when it tries to call the second one. What do you mean freezes? It pegs the CPU at 100%? Crashes? Simply stops? > I tried to reproduce the problem on a small sample but it unfortunately didn't work. Anyway, you will find it as an attachment. It basically shows how the code is designed. If the sample did reproduce the issue, it would print the message "Calling foo2()..." and then freeze. I need to be able to debug the freeze in order to fix it... You could also try to debug the program yourself and set a break-point at the function call. It's pretty much impossible to diagnose from the information you've provided so far. Is this code covered by an NDA? Can we see it? > I am afraid there are a number of things the big application does which cause it to go through other parts of the x86 code generator and generate some kind of memory or stack corruption that can not be easily reproduced with a small sample. You could also try -codegen c and/or -codegen bytecode. From rrnewton at gmail.com Thu Oct 23 08:37:47 2008 From: rrnewton at gmail.com (Ryan Newton) Date: Thu Oct 23 08:38:21 2008 Subject: [MLton] Semantics of giving a .c file to mlton? Message-ID: This is perhaps an undocumented bit of functionality, but I frequently enjoy passing .c files to mlton directly, rather than compiling them to .o files first. Consider the two small files attached below. They will compile as follows: mlton -default-ann 'allowFFI true' export.sml export.c I was (probably erroneously) under the impression that I didn't need to do -export-header, and #include the resulting file if I was passing in .c files directly (mlton could do that for me). Well, the above compiles fine and runs fine on a 32 bit platform: Running ML program... Inside C program... calling back to ML Got ML array: 0xb7fd2f98 Set first element But on 64 bit platforms it the call back into ML *completes* but returns a bogus address, then writing to that bogus address segfaults. Running ML program... Inside C program... calling back to ML Got ML array: 0xffffffff910996a8 Segmentation fault Exporting the header and including it in the .c file fixes the problem on 64-bit machines. I can do it that way but it would be awfully nice to officially be able to do what I happen to get away with on 32-bit platforms. Is this all as intended? Thanks, -Ryan export.sml ======================================================= val cfun = _import "cfun" : unit -> unit; val _ = (_export "ARRAYALLOC_CHAR" : (int -> char array) -> unit;) (fn len => Array.array(len,#"_")) val _ = print "Running ML program...\n" val _ = cfun() export.c ======================================================= #include //#include "export.h" void cfun () { printf("Inside C program... calling back to ML\n"); char* arr = ARRAYALLOC_CHAR(50); printf("Got ML array: %p \n", arr); arr[0] = 65; printf("Set first element\n"); } From fluet at tti-c.org Thu Oct 23 09:34:47 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Thu Oct 23 09:36:52 2008 Subject: [MLton] Semantics of giving a .c file to mlton? In-Reply-To: References: Message-ID: On Thu, 23 Oct 2008, Ryan Newton wrote: > This is perhaps an undocumented bit of functionality, but I frequently > enjoy passing .c files to mlton directly, rather than compiling them > to .o files first. Yes, that is supported. > Consider the two small files attached below. They will compile as follows: > > mlton -default-ann 'allowFFI true' export.sml export.c > > I was (probably erroneously) under the impression that I didn't need > to do -export-header, and #include the resulting file if I was passing > in .c files directly (mlton could do that for me). Well, the above > compiles fine and runs fine on a 32 bit platform: > > Running ML program... > Inside C program... calling back to ML > Got ML array: 0xb7fd2f98 > Set first element > > But on 64 bit platforms it the call back into ML *completes* but > returns a bogus address, then writing to that bogus address segfaults. > > Running ML program... > Inside C program... calling back to ML > Got ML array: 0xffffffff910996a8 > Segmentation fault > > Exporting the header and including it in the .c file fixes the problem > on 64-bit machines. I can do it that way but it would be awfully nice > to officially be able to do what I happen to get away with on 32-bit > platforms. > > Is this all as intended? When MLton invokes gcc on a .c file, it does so with '-w' to supress all warnings. This is because MLton invokes gcc on generated .c files, which, while mostly warning free, would have some spurious warnings that we don't want to bother the user. However, this means that your export.c file is being compiled with '-w', and so you are not getting the warning about the implicit declaration of ARRAYALLOC_CHAR. In the absence of a declaration, gcc introduces the implicit declaration: int ARRAYALLOC_CHAR(...); assuming that ARRAYALLOC_CHAR returns int (and admitting any number and types of arguments to ARRAYALLOC_CHAR). It happens to work for a 32-bit platform because the C calling convention for returning (a 32-bit) void* is the same for returning (a 32-bit) int. It fails to work on for a 64-bit platform because the C calling convention for returning (a 64-bit) void* is different from returning a (32-bit) int. From fluet at tti-c.org Thu Oct 23 09:51:56 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Thu Oct 23 09:54:01 2008 Subject: [MLton] Semantics of giving a .c file to mlton? In-Reply-To: References: Message-ID: On Thu, 23 Oct 2008, Matthew Fluet wrote: > On Thu, 23 Oct 2008, Ryan Newton wrote: >> This is perhaps an undocumented bit of functionality, but I frequently >> enjoy passing .c files to mlton directly, rather than compiling them >> to .o files first. > > Yes, that is supported. Though, actually, I should note that the real reason for supporting passing .c files to mlton directly was to support debugging/resuming the codegens. That is, one needs to pass appropriate -I flags to gcc to compile the .c and .S files produced by the C and native codegens. Since mlton knows the right -I flags, it was useful to be able to invoke mlton on just a collection of .c files (such as might have been produced by a previous invocation of mlton -keep g -stop g) in order to get the right flags. A useful side-effect is that mlton will accept any .c files on the command line, and compile and link them into the final executable. >> I was (probably erroneously) under the impression that I didn't need >> to do -export-header, and #include the resulting file if I was passing >> in .c files directly (mlton could do that for me). I don't think it is appropriate for mlton to implicitly change the meaning of a c file given on the command line (say, by implicitly #include-ing another file). From rrnewton at gmail.com Thu Oct 23 12:58:36 2008 From: rrnewton at gmail.com (Ryan Newton) Date: Thu Oct 23 14:03:51 2008 Subject: [MLton] Semantics of giving a .c file to mlton? In-Reply-To: References: Message-ID: Ah, thanks! That explanation makes sense, but I would not have guessed that gcc had such an odd convention (assumes an arbitrary return type and continues compiling?!). In my situation, I didn't want to have to invoke mlton twice. (Once to generate the header.) That would be pretty painful (has to typecheck the whole app twice -- several extra seconds). However, one thing I haven't tried is making a single call to mlton, with an -export-header flag, AND a .c file that depends on the header. Presumably, the ordering is such that this is safe? This is probably what I should be doing. -Ryan On Thu, Oct 23, 2008 at 12:51 PM, Matthew Fluet wrote: > On Thu, 23 Oct 2008, Matthew Fluet wrote: >> >> On Thu, 23 Oct 2008, Ryan Newton wrote: >>> >>> This is perhaps an undocumented bit of functionality, but I frequently >>> enjoy passing .c files to mlton directly, rather than compiling them >>> to .o files first. >> >> Yes, that is supported. > > Though, actually, I should note that the real reason for supporting passing > .c files to mlton directly was to support debugging/resuming the codegens. > That is, one needs to pass appropriate -I flags to gcc to compile the .c > and .S files produced by the C and native codegens. Since mlton knows the > right -I flags, it was useful to be able to invoke mlton on just a > collection of .c files (such as might have been produced by a previous > invocation of mlton -keep g -stop g) in order to get the right flags. > > A useful side-effect is that mlton will accept any .c files on the command > line, and compile and link them into the final executable. > >>> I was (probably erroneously) under the impression that I didn't need >>> to do -export-header, and #include the resulting file if I was passing >>> in .c files directly (mlton could do that for me). > > I don't think it is appropriate for mlton to implicitly change the meaning > of a c file given on the command line (say, by implicitly #include-ing > another file). > > From fluet at tti-c.org Thu Oct 23 14:30:51 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Thu Oct 23 14:33:05 2008 Subject: [MLton] Semantics of giving a .c file to mlton? In-Reply-To: References: Message-ID: On Thu, 23 Oct 2008, Ryan Newton wrote: > Ah, thanks! That explanation makes sense, but I would not have > guessed that gcc had such an odd convention (assumes an arbitrary > return type and continues compiling?!). The behavior is mandated by the C standard. > In my situation, I didn't want to have to invoke mlton twice. (Once > to generate the header.) That would be pretty painful (has to > typecheck the whole app twice -- several extra seconds). The export header should only depend on the static '_export' expressions in your program. (And the order in which they are type checked, so be careful if you rearrange files within .mlb files.) That is, if you have isolated your exported functions into a small number of .sml files, you could reasonably do the following in a Makefile: export.h: export1.sml export2.sml export3.sml mlton -stop tc -export-header export.h prog.mlb prog: prog.mlb $(shell mlton -stop f prog.mlb) export.h extra.c mlton prog.mlb extra.c Of course, if you are in a Makefile, then you might as well compile the extra.c file by itself: export.h: export1.sml export2.sml export3.sml mlton -stop tc -export-header export.h prog.mlb extra.o: export.h extra.c gcc -Wall -c extra.c prog: prog.mlb $(shell mlton -stop f prog.mlb) extra.o mlton prog.mlb extra.o > However, one thing I haven't tried is making a single call to mlton, > with an -export-header flag, AND a .c file that depends on the header. > Presumably, the ordering is such that this is safe? This is probably > what I should be doing. That should work, since the export header will be written at the end of type checking and be there before the .c file is compiled. -Matthew From Nicolas.Bertolotti at mathworks.fr Fri Oct 24 00:45:08 2008 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Fri Oct 24 00:45:55 2008 Subject: [MLton] Compilation crash with FFI on Windows with latest SVN revision (r6960) In-Reply-To: <162de7480810221309u20d61b3fs115d3cd7b83a17f0@mail.gmail.com> References: <8320D98DA9A5C54C926D397795FE7CEA2D98BAB46C@EXCHANGE-UK.ad.mathworks.com> <162de7480810220642s4fad0821oca472b9ffe586b94@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA2D98BAB4D2@EXCHANGE-UK.ad.mathworks.com> <8320D98DA9A5C54C926D397795FE7CEA2D98BAB579@EXCHANGE-UK.ad.mathworks.com> <162de7480810221309u20d61b3fs115d3cd7b83a17f0@mail.gmail.com> Message-ID: <8320D98DA9A5C54C926D397795FE7CEA2D98BAB82E@EXCHANGE-UK.ad.mathworks.com> > Well, picking 'public' as the default would break _import from > libraries. Either way something that used to "work" won't. As the old > _import didn't have public/external, you would have problems if you > tried to import a variable (as opposed to a function) or take the > address of functions. The new syntax at least gives you a compile > error as opposed to a bug that lurks in wait. =) So, do you mean that the fact that I was able to import functions from .c files and from DLLs without specifying a scope was just a matter of being lucky? > What do you mean freezes? It pegs the CPU at 100%? Crashes? Simply stops? It just enters a dead lock (not CPU resources used but the process is still running). > I need to be able to debug the freeze in order to fix it... You could > also try to debug the program yourself and set a break-point at the > function call. It's pretty much impossible to diagnose from the > information you've provided so far. > > Is this code covered by an NDA? Can we see it? Unfortunately, I can't provide it. Then, I understand that I have to debug by myself. That is why I am only asking for some useful hints to investigate it. > > > I am afraid there are a number of things the big application does which > cause it to go through other parts of the x86 code generator and generate > some kind of memory or stack corruption that can not be easily reproduced > with a small sample. > > You could also try -codegen c and/or -codegen bytecode. The issue does not reproduce with -codegen c. The same code compiled using MLton 2007 works fine. On the other hand, the code freezes within the DLL itself (when it calls a function for which I don't even have access to the source code), not at the FFI call. Then, it may be some kind of latent bug that has been revealed with the new version of MLton but not a MLton bug. Still investigating... Thanks for your help. Nicolas From wesley at terpstra.ca Fri Oct 24 08:11:15 2008 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Fri Oct 24 08:11:21 2008 Subject: [MLton] Compilation crash with FFI on Windows with latest SVN revision (r6960) In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA2D98BAB82E@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA2D98BAB46C@EXCHANGE-UK.ad.mathworks.com> <162de7480810220642s4fad0821oca472b9ffe586b94@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA2D98BAB4D2@EXCHANGE-UK.ad.mathworks.com> <8320D98DA9A5C54C926D397795FE7CEA2D98BAB579@EXCHANGE-UK.ad.mathworks.com> <162de7480810221309u20d61b3fs115d3cd7b83a17f0@mail.gmail.com> <8320D98DA9A5C54C926D397795FE7CEA2D98BAB82E@EXCHANGE-UK.ad.mathworks.com> Message-ID: <162de7480810240811s3b412e81sc247d8593de9ba64@mail.gmail.com> On Fri, Oct 24, 2008 at 9:45 AM, Nicolas Bertolotti wrote: >> As the old >> _import didn't have public/external, you would have problems if you >> tried to import a variable (as opposed to a function) or take the >> address of functions. The new syntax at least gives you a compile >> error as opposed to a bug that lurks in wait. =) > > So, do you mean that the fact that I was able to import functions from .c files and from DLLs without specifying a scope was just a matter of being lucky? Lucky in the sense that you didn't do any of the things that would have caused problems. >> Is this code covered by an NDA? Can we see it? > > Unfortunately, I can't provide it. Then, I understand that I have to debug by myself. That is why I am only asking for some useful hints to investigate it. Well, I would try creating my own DLL that proxies calls to the DLL which hangs. Then I'd stick print statements before passing the call to the real DLL and after it returns. I'd especially make sure to print the arguments so I could see if something strange was being passed. With this DLL wrapper, I would compare traces between the broken and working programs. If the call sequence or parameters is different at some point before the crash, I would investigate why. Since you say the problem is intermittent, I would also be concerned about garbage collection. Do you only ever call from SML->C or do you also sometimes call from C->SML? It's my understanding that if you call from C->SML, any pointers you had into the ML heap (vectors or strings you received) are invalidated. Another thing to check would be to run the program with '@MLton gc-messages --'. You could then compare the good and bad programs to see if garbage collection is happening around the hang of the bad program, but not the good program. >> You could also try -codegen c and/or -codegen bytecode. > The issue does not reproduce with -codegen c. I'm not sure what this means. > The same code compiled using MLton 2007 works fine. On the other hand, the code freezes within the DLL itself (when it calls a function for which I don't even have access to the source code), not at the FFI call. If the problem is memory corruption, you might try MLton revision r6940. The subsequent revision changed how memory is managed under win32. > Then, it may be some kind of latent bug that has been revealed with the new version of MLton but not a MLton bug. This is my personal hope. ;-) From rrnewton at gmail.com Fri Oct 24 13:50:08 2008 From: rrnewton at gmail.com (Ryan Newton) Date: Fri Oct 24 13:50:49 2008 Subject: [MLton] Performance of Real.toInt Message-ID: Hello, My compiler uses either SML or C as a backend. There's one application right now where SML/MLton is getting crushed. It's a computer vision app where the core computation involves updating histograms -- picking a location in some space (inexact) and then discretizing (truncating a floating point number). The truncation in the source language happens by converting the floats to ints. Under MLton I generate code like this: (Real64.toInt IEEEReal.TO_ZERO (var_tmpsmp_77)) But it performs very poorly. I haven't researched this, but if I had to guess, I'd bet this is because mlton is implementing some more semantically meaningful notion than C casts. Nevertheless, is there any inexpensive way to ape the behavior one gets from (int)x in C? -Ryan From vesa.a.j.k at gmail.com Sun Oct 26 01:17:26 2008 From: vesa.a.j.k at gmail.com (Vesa Karvonen) Date: Sun Oct 26 01:17:32 2008 Subject: [MLton] Performance of Real.toInt In-Reply-To: References: Message-ID: <9e43b9a0810260217i60e02011s892ca6bd709d50f2@mail.gmail.com> On Fri, Oct 24, 2008 at 10:50 PM, Ryan Newton wrote: > [...] It's a > computer vision app where the core computation involves updating > histograms -- picking a location in some space (inexact) and then > discretizing (truncating a floating point number). > > The truncation in the source language happens by converting the floats > to ints. Just to clarify, is the final result of "truncating a floating point number" a floating point number or an integer? > Under MLton I generate code like this: > > (Real64.toInt IEEEReal.TO_ZERO (var_tmpsmp_77)) > > But it performs very poorly. I haven't researched this, but if I had > to guess, I'd bet this is because mlton is implementing some more > semantically meaningful notion than C casts. An excellent guess! > Nevertheless, is there > any inexpensive way to ape the behavior one gets from (int)x in C? Have you peeked into the real/real.sml source file in MLton's basis library implementation? The implementation of Real.toInt uses a family of toIntUnsafe functions, that do not set the rounding mode or check that the floating point number is in the range of the integer type. One could perhaps extend the MLton.Real structure (http://mlton.org/MLtonReal) to expose those functions. You could then implement the conversion in terms of the unsafe functions. -Vesa Karvonen From fluet at tti-c.org Mon Oct 27 08:48:02 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Mon Oct 27 08:50:34 2008 Subject: [MLton] Performance of Real.toInt In-Reply-To: <9e43b9a0810260217i60e02011s892ca6bd709d50f2@mail.gmail.com> References: <9e43b9a0810260217i60e02011s892ca6bd709d50f2@mail.gmail.com> Message-ID: On Sun, 26 Oct 2008, Vesa Karvonen wrote: > On Fri, Oct 24, 2008 at 10:50 PM, Ryan Newton wrote: >> Under MLton I generate code like this: >> >> (Real64.toInt IEEEReal.TO_ZERO (var_tmpsmp_77)) >> >> But it performs very poorly. I haven't researched this, but if I had >> to guess, I'd bet this is because mlton is implementing some more >> semantically meaningful notion than C casts. > > An excellent guess! > >> Nevertheless, is there >> any inexpensive way to ape the behavior one gets from (int)x in C? > > Have you peeked into the real/real.sml source file in MLton's basis > library implementation? The implementation of Real.toInt uses a > family of toIntUnsafe functions, that do not set the rounding mode > or check that the floating point number is in the range of the integer > type. One could perhaps extend the MLton.Real structure > (http://mlton.org/MLtonReal) to expose those functions. You could > then implement the conversion in terms of the unsafe functions. As Vesa noted, SML's Real.toInt function does a lot more range checking than C's (int)d cast. In SML, there are at least two floating-point comparisons (performing the range check), a rounding mode set, a floating-point round, a rounding mode (re)set, and a floating-point to int coercion (the toIntUnsafe). If you are using the C codegen, then toIntUnsafe is implemented by a C cast; the semantics of a C cast is to convert with truncation (TO_ZERO) semantics. If you are using the x86 codegen, then toIntUnsafe is implemented by the 'fist' instruction; the semantics of the 'fist' instruction is to convert with the current rounding mode. If you are using the amd64 codegen, then toIntUnsafe is implemented by the 'cvt{s,d}2si{l,q}' instruction; the semantics of the 'cvt{s,d}2si{l,q}' instruction is to convert with truncation (TO_ZERO) semantics. Since the implmentations of toIntUnsafe do not always obey the current rounding mode, the SML implementation first does a floating-point round (under an appropriate rounding mode); thus, all of the toIntUnsafe implementations behave the same. But, it also means that the toIntUnsafe primitives are only well defined when the floating-point value is an integer; on non-integeral floating-point values, the different codegens could return different results. Note: on x86 with the C-codegen, the C cast actually generates another set/reset of the rounding mode, because gcc wants to use the 'fist' instruction, but with truncation (TO_ZERO) semantics (rather than the current rounding mode). This may also be the case on other architectures. If you are exclusively using the C-codegen, the exposing the toIntUnsafe functions in the MLton.Real structure would have the behavior of a C-cast. (It will still be a little slower, because the cast will occur in a non-inlined function; we don't inline some of the floating-point operations, because gcc will constant fold without obeying possible changes in the rounding mode. Though, given the explaination above, since C's cast always ignores the current rounding mode and uses truncation semantics, then it may be acceptable to inline.) If you wanted something a little more well-defined, you could expose in MLton.Real the composition of Primitive.Real.round with Primitive.Real.toIntUnsafe. That would first do a floating-point round to integer (under the current rounding mode), followed by a coercion to int (which, because the input will be an integral floating-point, will be well-defined for all implementations). However, this would be slightly different from a C-cast, since the default floating-point rounding mode is TO_NEAREST (at least on x86 and amd64, and possibly specified by C99 and/or IEEE754), not TO_ZERO. So, lots of choices, but nothing jumps out as a clear winner. From rrnewton at gmail.com Tue Oct 28 09:02:24 2008 From: rrnewton at gmail.com (Ryan Newton) Date: Tue Oct 28 10:40:25 2008 Subject: [MLton] Performance of Real.toInt In-Reply-To: References: <9e43b9a0810260217i60e02011s892ca6bd709d50f2@mail.gmail.com> Message-ID: Matthew, thanks for the comprehensive response. I feel a little bad, because when I looked at the numeric ranges involved I was able to fix this application to use integer ops only ;). It doesn't fix the performance divot, but avoids it. By the way, doesn't the FIST instruction raise a numeric exception on overflow? For the x86 backend, could the hardware do some of the work to avoid some of the range checking in real.sml? On Mon, Oct 27, 2008 at 12:48 PM, Matthew Fluet wrote: > On Sun, 26 Oct 2008, Vesa Karvonen wrote: >> >> On Fri, Oct 24, 2008 at 10:50 PM, Ryan Newton wrote: >>> >>> Under MLton I generate code like this: >>> >>> (Real64.toInt IEEEReal.TO_ZERO (var_tmpsmp_77)) >>> >>> But it performs very poorly. I haven't researched this, but if I had >>> to guess, I'd bet this is because mlton is implementing some more >>> semantically meaningful notion than C casts. >> >> An excellent guess! >> >>> Nevertheless, is there >>> any inexpensive way to ape the behavior one gets from (int)x in C? >> >> Have you peeked into the real/real.sml source file in MLton's basis >> library implementation? The implementation of Real.toInt uses a >> family of toIntUnsafe functions, that do not set the rounding mode >> or check that the floating point number is in the range of the integer >> type. One could perhaps extend the MLton.Real structure >> (http://mlton.org/MLtonReal) to expose those functions. You could >> then implement the conversion in terms of the unsafe functions. > > As Vesa noted, SML's Real.toInt function does a lot more range checking than > C's (int)d cast. In SML, there are at least two floating-point comparisons > (performing the range check), a rounding mode set, a floating-point round, a > rounding mode (re)set, and a floating-point to int coercion (the > toIntUnsafe). > > If you are using the C codegen, then toIntUnsafe is implemented by a C > cast; the semantics of a C cast is to convert with truncation (TO_ZERO) > semantics. If you are using the x86 codegen, then toIntUnsafe is > implemented by the 'fist' instruction; the semantics of the 'fist' > instruction is to convert with the current rounding mode. If you are using > the amd64 codegen, then toIntUnsafe is implemented by the > 'cvt{s,d}2si{l,q}' instruction; the semantics of the 'cvt{s,d}2si{l,q}' > instruction is to convert with truncation (TO_ZERO) semantics. Since the > implmentations of toIntUnsafe do not always obey the current rounding > mode, the SML implementation first does a floating-point round (under an > appropriate rounding mode); thus, all of the toIntUnsafe implementations > behave the same. But, it also means that the toIntUnsafe primitives are > only well defined when the floating-point value is an integer; on > non-integeral floating-point values, the different codegens could return > different results. > > Note: on x86 with the C-codegen, the C cast actually generates another > set/reset of the rounding mode, because gcc wants to use the 'fist' > instruction, but with truncation (TO_ZERO) semantics (rather than the > current rounding mode). This may also be the case on other architectures. > > If you are exclusively using the C-codegen, the exposing the toIntUnsafe > functions in the MLton.Real structure would have the behavior of a C-cast. > (It will still be a little slower, because the cast will occur in a > non-inlined function; we don't inline some of the floating-point operations, > because gcc will constant fold without obeying possible changes in the > rounding mode. Though, given the explaination above, since C's cast always > ignores the current rounding mode and uses truncation semantics, then it may > be acceptable to inline.) > > If you wanted something a little more well-defined, you could expose in > MLton.Real the composition of Primitive.Real.round with > Primitive.Real.toIntUnsafe. That would first do a floating-point > round to integer (under the current rounding mode), followed by a coercion > to int (which, because the input will be an integral floating-point, will be > well-defined for all implementations). However, this would be slightly > different from a C-cast, since the default floating-point rounding mode is > TO_NEAREST (at least on x86 and amd64, and possibly specified by C99 and/or > IEEE754), not TO_ZERO. > > So, lots of choices, but nothing jumps out as a clear winner. > > From fluet at tti-c.org Tue Oct 28 10:46:27 2008 From: fluet at tti-c.org (Matthew Fluet) Date: Tue Oct 28 10:48:50 2008 Subject: [MLton] Performance of Real.toInt In-Reply-To: References: <9e43b9a0810260217i60e02011s892ca6bd709d50f2@mail.gmail.com> Message-ID: On Tue, 28 Oct 2008, Ryan Newton wrote: > By the way, doesn't the FIST instruction raise a numeric exception on > overflow? For the x86 backend, could the hardware do some of the work > to avoid some of the range checking in real.sml? The native codegens aren't set-up to handle numeric exceptions (other than checking integral arithmetic overflow). Also, we try as much as possible to avoid special codepaths in the Basis Library that are sensitive to the codegen. > On Mon, Oct 27, 2008 at 12:48 PM, Matthew Fluet wrote: >> On Sun, 26 Oct 2008, Vesa Karvonen wrote: >>> >>> On Fri, Oct 24, 2008 at 10:50 PM, Ryan Newton wrote: >>>> >>>> Under MLton I generate code like this: >>>> >>>> (Real64.toInt IEEEReal.TO_ZERO (var_tmpsmp_77)) >>>> >>>> But it performs very poorly. I haven't researched this, but if I had >>>> to guess, I'd bet this is because mlton is implementing some more >>>> semantically meaningful notion than C casts. >>> >>> An excellent guess! >>> >>>> Nevertheless, is there >>>> any inexpensive way to ape the behavior one gets from (int)x in C? >>> >>> Have you peeked into the real/real.sml source file in MLton's basis >>> library implementation? The implementation of Real.toInt uses a >>> family of toIntUnsafe functions, that do not set the rounding mode >>> or check that the floating point number is in the range of the integer >>> type. One could perhaps extend the MLton.Real structure >>> (http://mlton.org/MLtonReal) to expose those functions. You could >>> then implement the conversion in terms of the unsafe functions. >> >> As Vesa noted, SML's Real.toInt function does a lot more range checking than >> C's (int)d cast. In SML, there are at least two floating-point comparisons >> (performing the range check), a rounding mode set, a floating-point round, a >> rounding mode (re)set, and a floating-point to int coercion (the >> toIntUnsafe). >> >> If you are using the C codegen, then toIntUnsafe is implemented by a C >> cast; the semantics of a C cast is to convert with truncation (TO_ZERO) >> semantics. If you are using the x86 codegen, then toIntUnsafe is >> implemented by the 'fist' instruction; the semantics of the 'fist' >> instruction is to convert with the current rounding mode. If you are using >> the amd64 codegen, then toIntUnsafe is implemented by the >> 'cvt{s,d}2si{l,q}' instruction; the semantics of the 'cvt{s,d}2si{l,q}' >> instruction is to convert with truncation (TO_ZERO) semantics. Since the >> implmentations of toIntUnsafe do not always obey the current rounding >> mode, the SML implementation first does a floating-point round (under an >> appropriate rounding mode); thus, all of the toIntUnsafe implementations >> behave the same. But, it also means that the toIntUnsafe primitives are >> only well defined when the floating-point value is an integer; on >> non-integeral floating-point values, the different codegens could return >> different results. >> >> Note: on x86 with the C-codegen, the C cast actually generates another >> set/reset of the rounding mode, because gcc wants to use the 'fist' >> instruction, but with truncation (TO_ZERO) semantics (rather than the >> current rounding mode). This may also be the case on other architectures. >> >> If you are exclusively using the C-codegen, the exposing the toIntUnsafe >> functions in the MLton.Real structure would have the behavior of a C-cast. >> (It will still be a little slower, because the cast will occur in a >> non-inlined function; we don't inline some of the floating-point operations, >> because gcc will constant fold without obeying possible changes in the >> rounding mode. Though, given the explaination above, since C's cast always >> ignores the current rounding mode and uses truncation semantics, then it may >> be acceptable to inline.) >> >> If you wanted something a little more well-defined, you could expose in >> MLton.Real the composition of Primitive.Real.round with >> Primitive.Real.toIntUnsafe. That would first do a floating-point >> round to integer (under the current rounding mode), followed by a coercion >> to int (which, because the input will be an integral floating-point, will be >> well-defined for all implementations). However, this would be slightly >> different from a C-cast, since the default floating-point rounding mode is >> TO_NEAREST (at least on x86 and amd64, and possibly specified by C99 and/or >> IEEE754), not TO_ZERO. >> >> So, lots of choices, but nothing jumps out as a clear winner. >> >> > > _______________________________________________ > MLton mailing list > MLton@mlton.org > http://mlton.org/mailman/listinfo/mlton >