[MLton] Exporting closures to C code

[skaller]

On Sun, 2005-09-25 at 22:03 +0200, Florian Weimer wrote:
> Most C libraries which use callback functions provide some means to
> pass a user-supplied pointer to the callback function.  This could be
> used to invoke MLton functions from C code (passing the closure
> explicitly), even if these functions are not exported.
> 
> How hard would that be to implement?

There have been a few answers to this, so here are my comments.
First --- I know NOTHING about the MLton side of it, so take
note of other people's comments.

However, the MLton side of it is largely irrelevant
as you will see in a minute.

For my Felix language I have a wrapper generator 
called 'flxcc' which does all that I describe below automatically.

Here is the pattern you must use. 

First, you have to distinguish two routines:

Routine 1 is the callback establisher.
Routine 2 is the actual callback.

The callback establisher is some routine which is used
to save the callback, this routine is part of the API you
are wrapping. These always exist on GUI platforms.
In some other applications, there is no establisher function,
you do it manually, by storing the callback function pointer
into a struct. The point, however, is that there is
ALWAYS an establisher. 

The establisher function has a signature like this:

T establisher(
  .... void *client_data, ... 
  callback_type_t callback ...
)

The callbacks themselves have a signature like this:

V callback( ... void *client_data ... )

which type is named 'callback_type_t' above in the
establisher.

Flxcc *recognizes* establishers automatically, by
the presence of a void * and a function accepting
a void* in the argument list .. this works 99% of the time.

You MUST provide wrappers for BOTH the callback type and
establisher!!

For each establisher, you need ONE C function.

This function is an MLton function written in C,
which accepts an MLton function closure as an argument.
This MLton function closure is registered as a callback
in the MLton way, and then it calls the C API establisher
you are wrapping, passing:

(a) the MLton closure is passed as the client_data
(b) A callback wrapper function (described next) is passed
   as the callback

Now you need to build the callback wrapper. You need
ONE callback wrapper FOR EACH C CALLBACK TYPE.

You do NOT need one for each callback. The same wrapper
is used for ALL callbacks of the same type.

The callback wrapper is a C function with the signature
required by the API you're wrapping, it is NOT an MLton
function. This function accepts the client_data* argument
and casts that value into an MLTon function closure,
then uses the MLton API to execute the closure.

If it's a function, it also has to convert the MLton
returned value to something C can understand.
You will ALSO need to convert any C values to the
MLton format to call the MLton function closure.

Note: Felix allows you to create a closure, and
then bind the arguments into it in two distinct
operations (technically, the closure is created
by the C++ constructor, and the arguments
are bound in by the call() method).

If MLton does not support this, you can work around
it by using a curried function.

The pattern I have described here has these properties:

(a) there is no other way to do it
(b) It can be automated by a wrapper generator
(c) It CANNOT be automated, only supported, by MLton
(d) It doesn't allow for deleting unused callbacks

Point (d) is a design problem with no trivial solutions.
This is because C is brain dead .. function pointers
never need to be deleted. The deletion of the 'client data'
in C API's is up to the client .. so too, here, unregistering
the closures so they can be garbage collected has to be
handled additionally, in a way that cannot be automated.
You have to know when the callback cannot be called.

The 'flxcc' solution uses the FrontC/CIL parser as a front
end to parse the CAPI in question, and generates all of
the above stuff automatically, whenever it detects callback
establishers. Flxcc has processed the whole of GTK .. believe me
you do NOT want to do all those wrappers entirely by hand!!

Anyhow: bottom line is: the MLton FFI is largely irrelevant
to the pattern. You need to provide

(a) a MLton callable wrapper for every C establisher
(b) a C callable wrapper for every callback type

and use the MLton C API to arrange the appropriate type
conversions, casts, root registrations, etc.

You will also need a 'callback forgetter' function
in many applications, which unregisters the callback
when it cannot be called from the C side.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net