[MLton] Re: segfault/assertion with mark-compact gc

[Matthew Fluet]

On Wed, May 12, 2010 at 11:02 AM, Matthew Fluet <matthew.fluet at gmail.com> wrote:
> I was trying to bootstrap SVN HEAD on Ubuntu 10.04 (running in VMWare
> Fusion), using 20091212~svn-r7383 from the Debian/Ubuntu packages as
> the baseline compiler.  This resulted in a segmentation fault
> somewhere early in the "toMachine" pass.  However, compiling
> /mlton/mlton.mlb directly from the command-line (i.e., not through
> make), did successfully compile.  But, bootstrapping with that
> compiler again resulted in a segmentation fault (but, somewhere else
> in the SSA optimization passes).  After a bit of tweaking with
> gc-messages and using the debug runtime, it seems that a mark-compact
> gc corrupts the heap, resulting in an "assert (1 == (header &
> GC_VALID_HEADER_MASK))".
>
> Has anyone else observed this?  I can't reproduce the error on
> x86-darwin, even if I tweak the runtime to always use the mark-compact
> gc.  However, I can regularly reproduce it in this x86-linux virtual
> machine.

Looks like it was due to a memcpy with overlapping src/dst.  This is
technically undefined behavior, but in older versions of gcc (and/or
libc), it worked out so long as dst <= src.  Overlapping moves can
occur in the compact phase of the mark compact GC, though they are not
extremely common.  One relatively large garbage object near the
beginning of the heap will mean that all of the (small) compacted
objects further down in the heap can be compacted without overlapping
themselves.

Changing the memcpy to a memmove solves the problem.  I'm auditing the
other uses of memcpy to determine if any others should be changed to
memmove.