Real.maxFinite bug

Matthew Fluet fluet@CS.Cornell.EDU
Tue, 2 Oct 2001 15:31:05 -0400 (EDT)


> Anyways, please take a look at Real.c and see if you have any suggestions for
> Real_class, Real_qequal, and Real_isNormal, which are still implemented by hand
> instead of calling math.h functions.  They may be susceptible to similar bugs.

It wasn't clear to me from the start of this discussion why exactly
maxFinite, minPos, and minNormalPos are _computed_ values?  In particular,
what justifies the correctness of 

      local
         fun min (p: real -> bool): real =
            let
               fun loop (x: real): real =
                  let
                     val y = x / 2.0
                  in
                     if p y
                        then loop y
                     else x
                  end
            in loop 1.0
            end
      in
         val minNormalPos = min isNormal
         val minPos = min isPositive
      end

where minNormalPos and minPos will be of the form 1/(2^n) for some n (and
that's assuming infinite precision arithmetic).  It seems that this is
sensitive to the binary encoding of floating-point numbers; and if you're
depending on IEEE, then just construct the value as a pair of words (or a
packed real) and cast it.  maxFinite doesn't have the exact same issue; 
the resulting value will be of the form a_n*2^n + a_{n-1}*2^{n-1} + ... +
a_1*2 + a_0*2^0 + a_{-1}*2^{-1} + ...  where a_i \in {0,1} and there is an
I < n such that a_i = 1 for i > I and a_i = 0 for I <= I. But this again
assumes that the binary encoding of the maximum finite number is of this
form.