benchmarking Poly/ML

[David Matthews]

These figures are interesting but I'm not sure how useful they are.  I'm
not particularly surprised by the figures.  I don't know anything about
the programs you've used but if they are affected by whether an
implementation provides unsafe Array.sub and update then they are
probably not typical ML programs.  Were they translated from C or some
other language?  My feeling is that programs, such as Isabelle, written
from scratch in ML where ML is the language of choice, behave
significantly differently to more "conventional" programs.  They put
much more emphasis on the garbage collector, closure optimisations and
redundant tuple removal.  The sort of benchmarks I've seen in the past
have tended to have tight inner loops which test the efficiency of
low-level code generation and the removal of constant expressions from
loops.  This kind of optimisation is crucial for the performance of
languages such as C and Fortran but is of less concern to the ML
programmer.  Looking at the behaviour of ML compilers on such programs
can provide some useful insights but my feeling is that it is more
important to focus on the immediate needs of the ML programmer.  For
example, although Poly/ML provides floating point operations these are
all implemented in terms of calls to the run-time system.  There has
simply never been a demand for them to be incorporated into the
code-generator.  Similarly, assignment and array update involve calls to
the RTS, at least in PolyML 3.3X.  ML programs tend to be written
applicatively so this has not been an issue.  Some ML implementations
and most "conventional" languages provide fixed precision integer rather
than arbitrary precision.  It will always be more efficient to run
programs on a fixed precision implementation, provided the precision is
sufficient, but allowing the ML programmer to ignore the problems of
overflow seems to be a definite gain.

Thanks for letting me have those figures.  It might be useful to have a
look at the benchmark programs themselves some time.  Now that Poly/ML
is open source someone may be tempted to look at areas in which the code
generation could be improved, particularly floating point.
David.

On Tuesday, October 17, 2000 8:58 PM, Stephen Weeks
[SMTP:sweeks@intertrust.com] wrote:
> 
> > Those benchmarks are interesting!
> > 
> > Why don't you try Poly/ML?  It is a free implementation of Standard
ML and
> > in my experience it greatly outperforms SML/NJ (50% faster).
> 
> I went ahead and ran Poly/ML on the same examples.  Here are the
running times.
> Note that I had to use a safe Array sub and update for the Poly/ML
version since
> I didn't know how to use the unsafe ones.  I'm sure that hurt Poly/ML
some, but
> nowhere near enough to explain the slowdown.
> 
>               OCAML  MLton  SML/NJ  Poly/ML
> holes           1.8    3.5     3.9     17.5
> fov             1.5    1.8     3.2     24.6
> intercyl        1.6    2.0     4.3     32.5
> snowgoon        2.9    3.2     5.1     65.2
> dice            3.9    5.1     8.8     72.5
> golf            1.5    2.5     3.1     22.1
> cone-fractal    3.7    4.4     6.5     82.7
> large           4.3    3.5     6.7     11.1
> pipe            5.4    4.7     7.9     74.6
> chess          16.0   15.9    21.6    271.5
> fractal        12.2   13.8    45.4    109.4
> geom-mean       3.6    4.3     7.1     47.8