[MLton] Crashes with 64-bit native code generator on Windows

[David Hansel]

Hi all,

We are still trying to figure out why our code crashes (brings up a
Windows error message box saying that the application was terminated) when
compiled with the native 64-bit codegen on Windows.  We were able to break
down the code a bit but unfortunately not enough to produce a small enough
example that could be shared here.  Removing more (mostly unrelated) code
makes the crash go away.  In our testing we have confirmed the following:

- The crashes NEVER occur before the first FFI call.  Removing all FFI
  calls makes the application work without crashes (as well as possible
  without the functionality that would be provided by the FFI calls).
- Sometimes the crashes occur on the first FFI call, sometimes some
  time after the call (within ML code),  sometimes on the second or third
  FFI call.  This changes randomly depending on which code we include.
- The crashes are not caused by our user code in the FFI functions.  We
  have removed all code from the bodies of those functions,  leaving only
  a simple return statement.
- Our FFI DLLs do not have entry functions that would be called
  when the DLL is loaded.
- The crashes do not occur if MLton's C code generator is used

We were able to create an example that only uses a single FFI call
and crashes on the first call to that function.  I have consolidated
the (partially ml-nlffigen generated) code and listed it below.  Please
let me know if you find any problem in the code.  Please don't mind the
useless conversions in the "cp" function within "foo".  In the real
code they are partially within some compatibility wrapper code and
removing them completely makes the crash go away.  I can not see
why these conversions should cause a crash.
The code below does causes the crash when called from within our (large)
code.  It does not produce a crash when called within a small example.
As mentioned,  this is the only FFI call that is actually called by
the code.  We do have to include another function making an FFI call
in order to make the crash happen.  However,  that call is never executed
before the crash.  It could be executed some time later.  If that second
FFI code is not present,  the crash does not happen.

My question basically is this:  do you have any suggestions on how to
debug this any further?  Any MLton command-line options for debugging?
Are there any optimization passes that we should try to disable?
Do you know of any caveats that we might have missed when creating our
DLLs?

Any suggestions are welcome.

Best regards,

David


C code:
-------

__declspec(dllexport) int __stdcall foo(const char *p, int f, const void *buf, int sz)
{
  return 1;
}


ML code:
--------

structure F_foo =
struct
   val lib = DynLinkage.open_lib {name = "bar.dll", lazy=true, global=false}
   val h   = DynLinkage.lib_symbol (lib, "foo")

   val callop =
       _import * :
       CMemory.addr ->
       CMemory.cc_addr * CMemory.cc_sint * CMemory.cc_addr * CMemory.cc_sint -> CMemory.cc_sint;

   fun mkcall a (x1, x2, x3, x4) =
       C_Int.Cvt.c_sint
           (CMemory.unwrap_sint
                (callop
                     a
                     (CMemory.wrap_addr (C_Int.reveal (C_Int.Ptr.inject' x1)),
                      CMemory.wrap_sint (C_Int.Cvt.ml_sint x2),
                      CMemory.wrap_addr (C_Int.reveal x3),
                      CMemory.wrap_sint (C_Int.Cvt.ml_sint x4))))

   fun f' (x1 : C_Int.ro C_Int.uchar_obj C_Int.ptr',
           x2 : MLRep.Int.Signed.int,
           x3 : C_Int.voidptr,
           x4 : MLRep.Int.Signed.int) : MLRep.Int.Signed.int =
       C_Int.Cvt.ml_sint
           (C_Int.call (C_Int.mk_fptr (mkcall, DynLinkage.addr h),
                        (x1, C_Int.Cvt.c_sint x2, x3, C_Int.Cvt.c_sint x4)))
end


fun foo (pp : string, f : bool, c : Word8.word vector) : bool =
    let val _   = print "foo-start\n"
        val sz  = Vector.length c
        val buf = C.alloc' C.S.uchar (Word.fromInt sz)

    fun cp (i, p) =
        if   i >= sz
        then ()
        else (C.Set.uchar' (C.Ptr.|*! p, Word8.fromLargeInt (Word32.toLargeInt (Word32.fromLargeInt (Word8.toLargeInt (Vector.sub (c, i))))));
              cp (i+1, C.Ptr.|+! C.S.uchar (p, 1)))
    in
        cp (0, buf);
        F_foo.f' (C.Ptr.null', if f then 1 else 0, C.Ptr.inject' buf, Int32.fromInt sz);
        C.free' buf;
        print "foo-end\n";
        true
    end

-- 
  ----------------------------------------------------------
  David Hansel
  http://www.reactive-systems.com/
  OpenPGP (GnuPG) public key file:
  http://www.reactive-systems.com/~hansel/pgp_public_key.txt
  ----------------------------------------------------------