[MLton] Eliminating library elaboration time

[Matthew Fluet]

On Sat, 10 Nov 2007, Vesa Karvonen wrote:
> The Basis library elaboration overhead seems to be a big issue to some
> people.  So, I was wondering whether we could mostly eliminate the
> overhead through the save/load world functionality of MLton.  Googling
> revealed that, indeed, MLton used to do that, but the feature was
> removed:
>
>  http://mlton.org/pipermail/mlton/2003-November/024667.html

It was further removed with the implementation and integration of the ML 
Basis system.  That is, after that point, even the SML/NJ build doesn't 
pre-parse and elaborate the basis library.

> Now, I think that it we should reintroduce the feature and generalize
> it.  Here is what I was thinking.  We would add two options:
>
>  -with-precompiled-basis <path.mlb>
>  -precompile-to <path>
>
> The first option is used to specify all the ML Basis files that you
> wish to precompile (pre-elaboration).  The effect would be the same as
> if your top-level MLB file would start with
>
>  local <path.mlb> in end
>
> for each <path.mlb> that you specified.  The second option would
> specify where MLton saves the world after precompiling or loads at
> startup.

That's probably the most expedient method.  Though, the Basis Library now 
also depends upon some compile time information; for example, the 
'-default-type tyN' option actually sets an ML Basis variable to load 
particular files in the Basis Library.  So, you'd need to check the 
up-to-dateness of files with respect to the expanded path (though, I 
suppose one would do that anyways).  You would just have this oddity that 
compiling with '-default-type intinf' could launch a recompile and dump of 
the world.

> The mlton script would check if the path does in fact
> exists.  If it doesn't, it runs MLton with a default world.  After
> MLton starts, it checks that all the bases specified on the
> command-line have been precompiled and that the precompiled bases are,
> in fact, up to date by checking file dates.  If the check fails, then
> MLton precompiles the bases, saves the world to the file named by the
> -precompile-to option and exists.  The mlton script then restarts
> mlton (this avoids issues on some platforms).  Otherwise mlton
> proceeds to compile as usual.

That seems like a lot to ask of the mlton script.  But, it seems like the 
best way to get the behavior you want.

> So, what do you think?  Would a scheme like this work and eliminate
> (Basis) library elaboration time?

I think this scheme would work; while it might have some unusual corner 
cases, I suspect that it would do the right thing for most standard usage 
scenarios.

But, I also think that the compile time of the Basis Library is noise when 
compiling programs of interest (e.g., *real* programs, not Jon's 100 line 
raytracer).  It is not where I would spend my implementation effort.