[MLton] bug in refFlatten

Matthew Fluet fluet@cs.cornell.edu
Fri, 15 Apr 2005 19:46:58 -0400 (EDT)

> > There appears to be a subtle bug in the refFlatten pass.  The pass seems
> > to be sensitive to the ordering of functions in the SSA2 program
> > representation.  While it would be o.k. (though, less than ideal) to get
> > different results depending on the order of functions, in this situation, 
> > the semantics of the resulting program changes.
> > 
> > The bug is exhibitted by ref-flatten.sml from the regression suite.  To 
> > trigger the bug, write a simple SSA2 that simply reverses the order of 
> > functions in the program representation.  Run this pass immediately before 
> > the refFlatten pass.  The program prints 13 instead of 44.

I've further narrowed the bug down to the foreachObject loop; or, at 
least, if I reverse the order of functions before the loop and restore it 
afterwards, then the bug is triggered.

      (* Try to flatten each ref. *)
      val () =
	 (fn (var, args, obj as Obj {flat, ...}) =>

I think it is related to the processing of arguments:

	     (* Check that all arguments that are represented by flattening them
	      * into the object are available as an explicit allocation.
	     val () =
		(args, fn a =>
		 case Value.deFlat {inner = varValue a, outer = obj} of
		    NONE => ()
		  | SOME (Obj {flat, ...}) =>
		       case ! (varInfo a) of
			  Flattenable _ => ()
			| Unflattenable => flat := NotFlat)

I note that Value.deFlat returns the inner object, and only succeeds if
the inner object's flat field is Offset.  Hence, if we process an argument
with an object with an Unknown flat field, and then later process the
argument's object as an object in its own right, we may at that later
point in time promote the flat field to Offset.  But, because the object
is used as an argument, if we promote the flat field to an Offset of the
object of which it is an argument, then we really should promote it all
the way to NotFlat -- as would have happened if we processed the object as 
an object first and then as an argument.