[MLton] Crashes with 64-bit native code generator on Windows

[Wesley W. Terpstra]

Sorry for the slow reply.

On Wed, Nov 11, 2009 at 5:10 AM, David Hansel
<hansel at reactive-systems.com> wrote:
> The code below does causes the crash when called from within our (large)
> code.  It does not produce a crash when called within a small example.
> As mentioned,  this is the only FFI call that is actually called by
> the code.  We do have to include another function making an FFI call
> in order to make the crash happen.  However,  that call is never executed
> before the crash.  It could be executed some time later.  If that second
> FFI code is not present,  the crash does not happen.

I went ahead and tried to build it. I compiled the C (bar.c) program using:
x86_64-w64-mingw32-gcc -Wall -O2 -o bar.dll -shared
-Wl,--out-implib,bar.a -Wl,--output-def,bar.def bar.c
I wrote the following baz.mlb file:
$(SML_LIB)/basis/basis.mlb
$(SML_LIB)/mlnlffi-lib/mlnlffi-lib.mlb
$(SML_LIB)/mlnlffi-lib/memory/memory.mlb
$(SML_LIB)/mlnlffi-lib/internals/c-int.mlb
ann
   "allowFFI true"
in
   baz.sml
end

Then I compiled with:
mlton -target x86_64-w64-mingw32 -link-opt -ldl -verbose 1 baz.mlb

The resulting program worked. Are you using similar compile options?
In the time since your last post have you perhaps found a more
complete crash example?

> My question basically is this:  do you have any suggestions on how to
> debug this any further?  Any MLton command-line options for debugging?

Well, there's -debug true, but gdb under 64-bit windows is so flakey I
wouldn't bother trying that. In fact, the MLton.msi doesn't include
the debug version of the runtime (it is over 200MB due to the windows
debugging format), so you would need to build MLton from source to get
the debug library. I doubt it would help you, though.

> Are there any optimization passes that we should try to disable?

I doubt this is an optimization problem.

> Do you know of any caveats that we might have missed when creating our
> DLLs?

Ok, here are the things I can think of from the top of my head:
0) You're loading a 32-bit dll instead of a 64-bit one. Double check.
1) Windows might require a stack alignment that doesn't match the
amd64 FFI codegen. Your program happens to end up with bad alignment,
and my programs have just never been unlucky. You could declare a
volatile local 64-bit variable and printf it's address in the C code.
See if the offset of this variable fails to be 64-bit aligned (only)
in the failing programs.
2) The __stdcall is confusing gcc. There is only one calling
convention under win64. Try specifying nothing.

However, I am guessing blind! Without a way to reproduce this I can't
really help. I've used the FFI quite heavily under win64 in one of our
recent projects without problems, so FFI definitely works most of the
time. It's possible you've found a corner case, which can often be an
alignment problem.

Is the program really too secret to release the buggy part of its
source code? MLton is free. ;)