[MLton] callcc and space use

[Daniel Spoonhower]

After some investigation, I think I have an answer to my problem.  In 
the end, it seems the problem had nothing to do with callcc, but with 
part of the analysis used in the reference flattening optimization 
(http://mlton.org/RefFlatten).

As the documentation says, to be safe for space, this optimization must 
ensure that it does not extend the lifetime of any objects of unbounded 
size.  I don't claim to totally understand this code, but I believe that 
the analysis isn't sufficiently conservative.  First, it doesn't seem to 
consider vectors to be objects of unbounded size.  Second, though 
individual instances of a datatype may have bounded size, an instance of 
a *recursive* datatype may refer to an unbounded amount of heap space. 
For example, flattening a reference into a tuple that contains a list 
may extend the lifetime of the entire list.

I attached a patch that fixes the leaks that I've observed.  Can someone 
take a look and tell me if I'm missing something?

Thanks,

--djs
-------------- next part --------------
diff --git a/mlton/ssa/ref-flatten.fun b/mlton/ssa/ref-flatten.fun
index cce824a..3f27084 100644
--- a/mlton/ssa/ref-flatten.fun
+++ b/mlton/ssa/ref-flatten.fun
@@ -717,6 +717,7 @@ fun flatten (program as Program.T {datatypes, functions, globals, main}) =
                                | Thread => Size.makeTop s
                                | Weak t => dependsOn t
                                | Word _ => ()
+                           val () = if Type.isVector t then Size.makeTop s else ()
                         in
                            s
                         end))
@@ -725,7 +726,17 @@ fun flatten (program as Program.T {datatypes, functions, globals, main}) =
          (datatypes, fn Datatype.T {cons, tycon} =>
           let
              val s = tyconSize tycon
-             fun dependsOn (t: Type.t): unit = Size.<= (typeSize t, s)
+             fun dependsOn (t: Type.t): unit = 
+                 let 
+                   datatype z = datatype Type.dest
+                   val () = case Type.dest t of 
+                              Datatype tycon' => if Tycon.equals (tycon, tycon')
+                                                 then Size.makeTop s
+                                                 else ()
+                            | _ => ()
+                 in
+                   Size.<= (typeSize t, s)
+                 end
              val () = Vector.foreach (cons, fn {args, ...} =>
                                       Prod.foreach (args, dependsOn))
           in