[MLton-user] ieee-fp flag

[Matthew Fluet]

>  Is there a description anywhere of exactly how MLton will deviate from 
> the IEEE floating point standard when the -ieee-fp flag is enabled?  I 
> couldn't find any details on the internet.

With '-ieee-fp true' option, the native codegen acts similar to the 
behavior of gcc with the '-ffloat-store' option.  That is, floating-point 
values are never cached in floating-point registers.  For each operation, 
floating-point values are fetched from memory and the result is 
immediately written back to memory.

This is the well known problem of the x86 floating-point unit using 80bit 
precision for all operations.  If one performs a sequence of 
floating-point operations, leaving intermediate results on the 
floating-point stack, then one is actually computing the intermediate 
results at 80bit precision, not the 64bit precision required of the
SML Real64.real type (and of the C double type).

So, with '-ieee-fp false' (the default), MLton will deviate from the IEEE 
floating point standard by virtue of the fact that computations on 64-bit 
floating point values may actually be performed at 80bit precision. 
(MLton uses the floating-point hardware for arithmetic, comparisons, sqrt, 
trig functions, exp, ln, log10 on the x86.)  If and when a result is 
rounded to 64bits depends on the vagaries of the register allocator and 
when spilling is required by calling conventions, as the rounding will 
only occur when a floating-point value is written to memory.

This computing at 80bit precision also affects the SML Real32.real type 
(and the C float type).

It is possible to set the x86 processor in a mode that forces computations 
to occur at 64bit (or 32bit precision, I believe), but that can have 
undesirable consequences when calling into system libraries.  FreeBSD sets 
the processor to this mode by default; Linux does not.  Also, note that 
even in this mode, Real32.real computations are being done at 64bit 
precision, so one can still get excess precision artifacts.

AFAIK, this is a problem that plagues most x86 compilers.
   http://gcc.gnu.org/bugzilla/show_bug.cgi?id=323

On amd64, MLton uses the SSE extensions, so all floating-point arithmetic, 
comparisions, and sqrt are done in hardware at the required IEEE 
precision.  (On the other hand, the remaining floating-point operations 
must call out to the system math library.)