[MLton-user] re: Generic pickler and MLB files

[Vesa Karvonen]

I got an email with some questions on my generics library in Finnish.  I
took the liberty of translating the questions to English and answering
them here, because the questions and answers may be useful to others.

> What is the correct way to take Generic.Pickle to use with MLB-files?
> The only combination I've gotten to work looks like this:
>
>  $(MLTON_LIB)/com/ssh/generic/unstable/lib.mlb
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/generic.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/type-info.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/type-hash.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/hash.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/uniplate.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/pretty.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/eq.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/some.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/pickle.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/seq.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/read.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/reduce.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/transform.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/fmap.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/size.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/ord.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/shrink.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/close-pretty-with-extra.sml
>  $(MLTON_LIB)/com/ssh/generic/unstable/with/reg-basis-exns.sml
>
> There is probably something extra here.

Yes, the above includes most of the generics and also registers a bunch of
standard exceptions (reg-basis-exns.sml) and also makes it so that values
of the product type are printed in infix notation
(close-pretty-with-extra.sml).  If one just wants to use just the Pickle
generic, then the following should do:

   (* First we need the generics library itself: *)
   $(MLTON_LIB)/com/ssh/generic/unstable/lib.mlb

   (* Then we begin the definition of the Generic module: *)
   $(MLTON_LIB)/com/ssh/generic/unstable/with/generic.sml

   (* Then we extends the Generic module with the desired combination: *)
   $(MLTON_LIB)/com/ssh/generic/unstable/with/type-info.sml
   $(MLTON_LIB)/com/ssh/generic/unstable/with/type-hash.sml
   $(MLTON_LIB)/com/ssh/generic/unstable/with/hash.sml
   $(MLTON_LIB)/com/ssh/generic/unstable/with/eq.sml
   $(MLTON_LIB)/com/ssh/generic/unstable/with/some.sml
   $(MLTON_LIB)/com/ssh/generic/unstable/with/pickle.sml

   (* Then we "close" the combination to make it convenient to use: *)
   $(MLTON_LIB)/com/ssh/generic/unstable/with/close.sml

   (* Optionally, we can add support for some standard type constructors
      (e.g. option, order, tuples, ...): *)

   $(MLTON_LIB)/com/ssh/generic/unstable/with/extra.sml

So, how does one know how to write the required combination as above?  A
short answer could be:

   A proper combination is a topological ordering of the graph whose root
   vertices are the desired generics and whose edges are specified by the
   WITH_?_DOM signatures of the generics.

However, it probably helps to understand a bit about how the generics
library works.  Each generic, like the Pickle generic, is implemented by a
functor, that extends an existing combination of generics.  The with
-files, one for each generic, like

   $(MLTON_LIB)/com/ssh/generic/unstable/with/hash.sml
   http://mlton.org/cgi-bin/viewsvn.cgi/*checkout*/mltonlib/trunk/com/ssh/generic/unstable/with/hash.sml?rev=6071

instantiate the functor (and perform a bit of plumbing).  In the case of
the Pickle generic, the functor is named WithPickle.  Looking at the
export file of the generics library

  http://mlton.org/cgi-bin/viewsvn.cgi/*checkout*/mltonlib/trunk/com/ssh/generic/unstable/public/export.sml?rev=6464

one can see that the WithPickle functor has the "signature"

   functor WithPickle (Arg : WITH_PICKLE_DOM) : PICKLE_CASES

This means that WithPickle takes as an argument a module with the
signature WITH_PICKLE_DOM and produces a module with the signature
PICKLE_CASES.  Looking at the signature file of the Pickle generic

   http://mlton.org/cgi-bin/viewsvn.cgi/*checkout*/mltonlib/trunk/com/ssh/generic/unstable/public/value/pickle.sig?rev=6389

we can see the WITH_PICKLE_DOM signature

   signature WITH_PICKLE_DOM = sig
      include CASES EQ HASH SOME TYPE_HASH TYPE_INFO
      sharing Open.Rep = EqRep = HashRep = SomeRep = TypeHashRep = TypeInfoRep
   end

>From this one can see that to instantiate the WithPickle functor, one
first needs to produce a combination of generics that includes the Eq
(EQ), Hash (HASH), Some (SOME), TypeHash (TYPE_HASH), and TypeInfo
(TYPE_INFO) generics.  Coincidentally, in this case, that is the whole set
of generics needed.  However, in order to know the topological ordering,
one still needs to peek into the WITH_?_DOM signatures of the required
generics.  For example, looking at

   http://mlton.org/cgi-bin/viewsvn.cgi/*checkout*/mltonlib/trunk/com/ssh/generic/unstable/public/value/hash.sig?rev=6378

we can see the WITH_HASH_DOM signature

   signature WITH_HASH_DOM = sig
      include CASES TYPE_HASH TYPE_INFO
      sharing Open.Rep = TypeHashRep = TypeInfoRep
   end

and it tells us that TypeHash and TypeInfo must be in the combination
before Hash.

(Unfortunately, the MLB system is not expressive enough to specify MLB
files that would automatically compute a topological ordering (or
dependencies) allowing one to just specify which generics are actually
wanted (and not their dependencies).  So, one need to do it by hand or
write a tool to do it.

> I tried to use some extras from the with/ directory on top of the
> lib-with-default.mlb, but it didn't seem to work.

It can be made to work, but one needs to (re)close the combination and
observe the topological ordering requirements as discussed above.  Looking
at lib-with-default.mlb, one can see that it defines the combination as
follows:

   with/generic.sml
   with/eq.sml
   with/type-hash.sml
   with/type-info.sml
   with/hash.sml
   with/ord.sml
   with/pretty.sml
   with/read.sml
   with/close-pretty-with-extra.sml

Comparing to the previous minimal combination for the Pickle generic, one
can see that only the Some generic is missing.  So, to extend the default
with Pickle, one could write the following in an MLB file:

   (* First the default combination: *)
   $(MLTON_LIB)/com/ssh/generic/unstable/lib-with-default.mlb

   (* Then extend it with Some and then Pickle: *)
   $(MLTON_LIB)/com/ssh/generic/unstable/with/some.sml
   $(MLTON_LIB)/com/ssh/generic/unstable/with/pickle.sml

   (* Finally (re)close the combination for use: *)
   $(MLTON_LIB)/com/ssh/generic/unstable/with/close-pretty-with-extra.sml

Now, one may wonder what is the close-pretty-with-extra.sml file doing.
It is, in effect, a convenience for the following frequently desired
combination:

   $(MLTON_LIB)/com/ssh/generic/unstable/with/close.sml
   $(MLTON_LIB)/com/ssh/generic/unstable/with/extra.sml
   $(MLTON_LIB)/com/ssh/generic/unstable/with/infix-product.sml

Another thing that one might find puzzling is that starting with
lib-with-default.mlb and extending it with the Pickle generic, the
combination was first started (in lib-with-default.mlb by including
with/generic.sml), then closed (in lib-with-default.mlb by including
with/close-pretty-with-extra.sml), and then closed again (in our extended
combination by including with/close-pretty-with-extra.sml) (without
starting a combination).  The answer to the puzzle is that closing a
combination does not close it in the sense of making it impossible to
extend it later.  Effectively, closing a combination just produces a set
of combinators, specified by the CLOSED_CASES signature, that build type
representations having just the closed combination of generics.

> It seems that type representations made with lib-with-default.mlb and
> with the above combination are incompatible.  Is it possible that the
> type of type representations changes depending on which "with/" -files
> from the generics library have been taken into use?

[To clarify, "above combination" refers to the previously mentioned "only
combination" of generics that includes almost all the generics.]

Yes, they are incompatible and, yes, the type of the type representation
depends on which with -files are included and in which order.  The
functors for generics are generative.  That is, each time a functor like
WithPickle is instantiated, it creates a new type for the part of the type
representation that corresponds just to the specific generic, like the
Pickling generic.

> This causes problems if one wants to write modular code in such a way
> that modules export type representations of their own types.

Yes, it is quite true that a weakness of the approach is that the
combination must be specified explicitly and this has an effect on program
organization.  This is mentioned in my paper and in the README file of the
generics library.

However, it is not difficult to avoid (or solve) the problem although it
requires extra linguistic measures (i.e. programming conventions /
idioms).

The simplest basic approach (although not the only one) to avoiding the
problem is to organize the program in such a way that only a single
combination of generics is ever produced.  The simplest programming
convention to make this happen is to make it so that the combination is
specified in the context of the application being specified.  This means
that libraries using generics need to be parameterized with respect to the
actual combination of generics, which is only known in the context of a
complete application, to use.  There are a number of ways to achieve such
a parameterization.

One way to parameterize libraries with respect to a combination of
generics is functorize the libraries and only instantiate them in the
final application.  For example, the unit-test library is currently
functorized.  This works, but can be tedious (although programming in
fully functorial style definitely has a number advantages; e.g. it helps
to ensure that each module can be tested in isolation).

Perhaps a more practical technique is to have the parameterization in the
build files.  With MLton's MLB system, one can use MLB path variables to
implement such parameterization.  The idea is that, in an MLB file for a
library, one refers to the MLB file for the generics via a MLB path
variable.  For example, a plausible convention would be to use the path
variable "GENERICS" to refer to the MLB file that defines the Generic
module.  In an MLB file, you would then specify

  $(GENERICS).mlb

to get the generics.  For example, in an application whose libraries just
uses the Pickle generic, the file $(GENERIC).mlb could look like this:

   $(MLTON_LIB)/com/ssh/generic/unstable/lib.mlb
   $(MLTON_LIB)/com/ssh/generic/unstable/with/generic.sml
   $(MLTON_LIB)/com/ssh/generic/unstable/with/type-info.sml
   $(MLTON_LIB)/com/ssh/generic/unstable/with/type-hash.sml
   $(MLTON_LIB)/com/ssh/generic/unstable/with/hash.sml
   $(MLTON_LIB)/com/ssh/generic/unstable/with/eq.sml
   $(MLTON_LIB)/com/ssh/generic/unstable/with/some.sml
   $(MLTON_LIB)/com/ssh/generic/unstable/with/pickle.sml
   $(MLTON_LIB)/com/ssh/generic/unstable/with/close.sml
   $(MLTON_LIB)/com/ssh/generic/unstable/with/extra.sml

When the application is compiled, one must then simply specify the MLB
path variable GENERIC using either the -mlb-path-map or the -mlb-path-var
(recently added) command line switch.

I'll probably delete the lib-with-default.* files and change the libraries to
use the convention (using MLB path variable GENERICS) discussed here.

> How should one use the pickling generic?

I suppose what the real question here is:

   How one should write a library that uses the Pickle generic?

I believe the previous discussion already answers the question, but for
concreteness, here is a complete example:

<-- application.mlb ---
local
   $(GENERICS).mlb
   library.mlb
in
   main.sml
end
--- application.mlb -->

<-- main.sml ---
local open Generic in
   val () = Library.doStuff (list int) [1, 2, 3]
end
--- main.sml -->

<-- generics.mlb ---
$(MLTON_LIB)/com/ssh/generic/unstable/lib.mlb
$(MLTON_LIB)/com/ssh/generic/unstable/with/generic.sml
$(MLTON_LIB)/com/ssh/generic/unstable/with/type-info.sml
$(MLTON_LIB)/com/ssh/generic/unstable/with/type-hash.sml
$(MLTON_LIB)/com/ssh/generic/unstable/with/hash.sml
$(MLTON_LIB)/com/ssh/generic/unstable/with/eq.sml
$(MLTON_LIB)/com/ssh/generic/unstable/with/some.sml
$(MLTON_LIB)/com/ssh/generic/unstable/with/pickle.sml
$(MLTON_LIB)/com/ssh/generic/unstable/with/close.sml
$(MLTON_LIB)/com/ssh/generic/unstable/with/extra.sml
--- generics.mlb -->

<-- library.mlb ---
local
   $(MLTON_LIB)/com/ssh/extended-basis/unstable/basis.mlb
   $(GENERICS).mlb
in
   library.sml
end
--- library.mlb -->

<-- library.sml ---
structure Library :> sig
   val doStuff : ('a, 'x) Generic.Open.Rep.t -> 'a Effect.t
end = struct
   open Cvt Generic

   fun doStuff t x =
       (print "Hex bytes:"
      ; String.app
           (fn c => prints [" ", P#L#"0"2 (I#x (ord c))])
           (Generic.pickle t x)
      ; print "\n")
end
--- library.sml -->

<-- mlb-path-map ---
GENERICS generics
MLTON_LIB *** NOTE: path to mltonlib root; you'll have to change this line ***
--- mlb-path-map -->

To try the above, just create files with the indicated names and contents
to a single directory and compile with:

  mlton -mlb-path-map mlb-path-map application.mlb

Running the produced application should produce the following output (at
least with the current implementation of the Pickle generic):

  Hex bytes: 00 03 01 01 01 02 01 03

-Vesa Karvonen