[MLton] Separate compilation

[Stephen Weeks]

> True separate compilation is pretty much incompatible with MLton's
> compilation strategy.  From the MLton User's Guide:
...
> In particular, the compilation transformations of monomorphisation
> and defunctionalization (which is different from defunctorization)
> are impossible without having the entire program.  Furthermore,
> these transformations are pretty much the first things that MLton
> does when compiling, so all the subsequent optimizations depend
> critically upon them being performed.

I agree -- much of what happens in MLton after the front end relies on
having the whole program.  So, it would be very difficult to reuse.
Therefore, if we want a byte-code compiler with separate compilation
we should reuse only the front end, and compile either CoreML or XML
to a new target IL designed specifically for our byte-code compiler.
My guess is that an IL with a universal type is the way to go.

It would also be possible reuse much of MLton to build a whole-program
byte-code compiler, and hook in where the other code generators (C,
nativ) currently hook in.  The (very reasonable) question about such
an approach is what it really buys you, since one of the main benefits
of a byte-code compiler is rapid development, and even turning off all
the SSA optimization will still leave a pretty big compile time for
>100K line programs like MLton.

> I think most of the developers would argue that development without SML/NJ
> wouldn't be too bad (assuming a sufficiently fast machine with enough
> memory).  At least for me, a fair amount of my development time is spent
> fixing type errors, and the new front-end can be quite a bit faster than
> SML/NJ in type-checking large programs.

I almost exclusively use the new front end for type checking MLton.
However, I still find SML/NJ invaluable for doing quick testing.
That's the one niche that I would like to see a byte-code compiler
fill.  I would think that the approach of separate compilation to a
universally typed IL would work well for that niche.

> Now, as fast as MLton's front-end is right now, there exists the
> possibility for implementing "partial separate compilation" which could
> lex, parse, elaborate, and type-check a compilation unit and save the
> resulting intermediate form to disk.

Yes, I hope to see something like that after the next release.  I'm
not sure whether it will be explcit, via command line arguments
(-{load,save}-basis) or implicit and somehow connected to mlb files.