[MLton] caching elaboration

[Matthew Fluet]

> > I'll note that the proposal seems to point towards a scheme where
> > elaboration prompts lexing/parsing.
>
> I don't see why (but you have a much better understanding of this
> stuff right now).

As you note, the compiler currently parses the _entire_ program before
starting any elaboration.  Before the MLBs integration, the compile of a
CM file went through:

fun parseAndElaborateFiles (fs: File.t list, E: Env.t, lookupConstant):
Decs.t =
   Control.pass
   {name = "parseAndElaborate",
    suffix = "core-ml",
    style = Control.ML,
    thunk = fn () => (List.fold
		      (fs, Decs.empty, fn (f, ds) =>
		       Decs.append
		       (ds, elaborateProg (lexAndParseFile f,
					   E,
					   lookupConstant)))),
    display = displayDecs}

Now, the problem is that elaboration of a MLB Basdec requires non-trivial
manipulation of the environment.  So, I don't think it makes sense to try
to co-routine it in such a way that it returns to the main/compile.fun
functions between every file (in the implicit way that the List.fold above
does).  The real question is what should the Basdec datatype look like for
a source file:

 | SMLFile of File.t
 | SMLFile of File.t * Ast.Program.t
 | SMLFile of File.t * Ast.Program.t Promise.t

(The same question applies to the variant for an imported mlb file, but I
assume .mlb files are changing less rapidly than source files.) I've
currently opted for the second one.  The first one would explicitly
require doing lexing and parsing from the elaborator, while the third one
would implicitly prompt the lexing and parsing.

> I think this is minimally invasive and doesn't require much code
> reorganization.  So, I don't see that it points either way for whether
> elaboration prompts lexing/parsing.

I disagree.  It think it _must_ point to the elaborator prompting
lexing/parsing.  If we lex/parse the _entire_ program before starting
elaboration (as we currently do), then we'll never see any benefit in your
common case, because we'll invariably change one file, which will
invalidate the assumptions _before_ type-checking even begins.  We want to
do as much elaboration (and type-checking) as possible before encountering
the changed file.

> > In particular, it will no longer be the case that all elaboration
> > type-errors will occur after all syntax errors.
>
> Hmmm.  This definitely changed at some point since the last release.

Yes, it changed with the MLBs integration.  If you want to change back to
the old behavior, then I really think the only thing to do is to change
the type of elaborate from:

val elaborateMLB: Ast.Basdec.t * {addPrim: Env.t -> Decs.t} ->
                  Env.t * (Decs.t * bool) vector

to

val elaborateMLB: String.t * {addPrim: Env.t -> Decs.t} ->
                  Env.t * (Decs.t * bool) vector

and trigger lexing/parsing from elaboration.