packages (was Re: [MLton] syntax error for "_address")

[Matthew Fluet]

>> Our philosophy is to integrate with existing tools and build processes
>> as much as possible.  Producing our own replacement for make or
>> existing packaging systems would take a lot of effort, would likely
>> produce worse results (than the better ones anyway), and would make
>> tools harder for people to use than the ones they know.  So, Matthew's
>> answer to your question is the right one: MLBs live within whatever
>> existing mechanism (e.g. make) you have for expressing requirements,
>> handling platform dependencies, and building applications.
>> 
>> MLBs exist to solve a problem that makes sense solely in the context
>> of SML (namely, namespace control at the module level) and solve the
>> problem in the spirit of SML, with a clean static and dynamic
>> semantics.  Other components necessarily sit on top of MLBs to make
>> real-world things happen, but to complicate MLBs to handle such issues
>> would lead to a worse overall system with fewer clearly understood
>> components.
>
> I think that this is a fine philosophy (in fact, it is the one that I
> follow for Moby), but the fact that mlton is a whole program compiler
> means that I have no way to use the underlying system tools to package
> SML libraries.  I don't know what the right solution is, but I think
> that there needs to be a mechanism for packaging both SML and C
> code together in a library.  Anyway, I asked the question because I'm
> trying to build some graphics libraries that I plan to use in a bunch
> of applications and I wanted to know what the "best practice" was.
> It sounds like this is still an open problem.

If you only use _import, the situation appears to be a little easier.  You 
just need to ensure that the top-level invocation of mlton has the right 
flags.  In this situation, I think using a Makefile include to get 
additional flags for the compilation make sense.

The difficulty is with _export, although part of that difficulty is that 
we currently have a compilation model for _export that requires it to be 
keyed to the whole program.  If you are packaging SML and C in a manner 
that requires _exports, then it might be worth considering moving to a 
different compilation strategy for _export.  In particular, it would be 
nice if John could generate his _exports in a way that the generated .h 
and .c files would not be keyed to the whole program.

What I have in mind is switching the implementation of
/basis-library/mlton/thread.sml's register from

    val register: int * (unit -> unit) -> unit =
       let
          val exports = Array.array (Primitive.FFI.numExports, fn () =>
                                     raise Fail "undefined export")

to something like

    val register: string * (unit -> unit) -> unit =
       let
          val exports : {hash: word, name: string} HashSet.t =
             HashSet.new {hash = #hash}

Now John can generate (and compile) his .h and .c files in isolation 
(i.e., before shipping code), because the registration string (i.e., the 
exported symbol name) won't change and should not conflict with any other 
symbols (either from other _exports or from other linked code).

This has a minor performance penalty on calling an _exported function, but 
it appears to ease the integration burden.

> BTW, what does MLB do about the diamond problem?  E.g., if I have a
> library A that B and C depend on, and then I write an application
> that uses both B and C, do I get two copies of A or does it share?

There will be one copy of A that is shared between B and C.