[MLton] benchmarks vs last release

Stephen Weeks MLton@mlton.org
Fri, 13 Aug 2004 12:48:04 -0700


To make sure there are no problems with our upcoming release, I ran
the benchmarks to compare the performance of our last release to our
current internal version.  The complete numbers are below.  The upshot
is:

1. In compile time, there are no significant changes.

2. In code size, there are some significant improvements and no
   significant worsenings. I suspect the improvements are mostly due
   to some mistake in 20040227 that caused too much library code to be
   included. 

3. In run time, most benchmarks are roughly the same.  There are some 
   nice improvements, some small slowdowns, and no show stoppers.
   Here are the number of benchmarks with run time ratios in various 
   ranges.

	# bench	run-time ratio
	-------	--------------
	3	[0.6, 0.7)
	2	[0.7, 0.8)
	5	[0.8, 0.9)
	28	[0.9. 1.1)
	4	[1.1, 1.2)

Here's all the numbers.

MLton0 -- mlton 20040227
MLton1 -- mlton HEAD

run time ratio
benchmark         MLton1
barnes-hut          0.94
boyer               1.09
checksum            0.68
count-graphs        0.87
DLXSimulator        1.00
fft                 1.01
fib                 0.98
flat-array          0.64
hamlet              1.05
imp-for             1.16
knuth-bendix        1.04
lexgen              0.81
life                0.88
logic               1.00
mandelbrot          0.94
matrix-multiply     0.73
md5                 0.68
merge               0.98
mlyacc              1.07
model-elimination   1.04
mpuz                1.09
nucleic             1.11
output1             0.75
peek                1.18
psdes-random        0.93
ratio-regions       0.98
ray                 0.95
raytrace            1.03
simple              1.03
smith-normal-form   1.00
tailfib             1.19
tak                 0.99
tensor              0.95
tsp                 0.80
tyan                1.02
vector-concat       0.94
vector-rev          1.01
vliw                0.99
wc-input1           1.01
wc-scanStream       0.99
zebra               0.83
zern                0.99

size
benchmark            MLton0    MLton1
barnes-hut          143,701   144,233
boyer               226,477   142,603
checksum            116,365    60,333
count-graphs        133,901    70,935
DLXSimulator        176,853   173,033
fft                 134,777    70,083
fib                 114,701    52,311
flat-array          114,673    52,339
hamlet            1,339,351 1,295,313
imp-for             114,685    52,159
knuth-bendix        152,549   147,873
lexgen              235,930   240,950
life                134,149    70,227
logic               174,485   113,191
mandelbrot          114,785    52,251
matrix-multiply     117,357    53,487
md5                 123,529   115,533
merge               116,117    53,875
mlyacc              552,670   549,366
model-elimination   671,086   674,124
mpuz                118,877    56,387
nucleic             265,193   206,327
output1             133,271   126,535
peek                124,857   120,965
psdes-random        115,101    53,031
ratio-regions       142,969    81,251
ray                 222,819   218,513
raytrace            325,759   301,955
simple              283,637   279,228
smith-normal-form   219,741   226,029
tailfib             114,813    51,975
tak                 114,753    52,387
tensor              145,996   142,456
tsp                 125,313   123,725
tyan                182,665   183,449
vector-concat       116,417    53,623
vector-rev          115,765    52,927
vliw                433,564   431,946
wc-input1           152,353   148,029
wc-scanStream       150,925   148,745
zebra               184,609   162,441
zern                148,770   128,776

compile time
benchmark         MLton0 MLton1
barnes-hut          6.05   6.76
boyer               8.07   7.47
checksum            4.64   4.84
count-graphs        5.33   5.53
DLXSimulator        7.05   7.77
fft                 5.16   5.27
fib                 4.61   4.80
flat-array          4.64   4.81
hamlet             45.53  45.57
imp-for             4.64   4.83
knuth-bendix        5.69   6.30
lexgen              8.12   9.03
life                5.17   5.31
logic               6.15   6.71
mandelbrot          4.65   4.85
matrix-multiply     4.73   4.89
md5                 4.93   5.48
merge               4.69   4.84
mlyacc             22.78  22.62
model-elimination  20.87  21.87
mpuz                4.79   4.96
nucleic            61.99  62.30
output1             4.96   5.50
peek                4.86   5.42
psdes-random        4.65   4.88
ratio-regions       5.79   5.98
ray                 7.29   8.06
raytrace           11.18  11.40
simple              9.20   9.65
smith-normal-form   8.86   8.32
tailfib             4.62   4.81
tak                 4.62   4.79
tensor              6.44   6.96
tsp                 5.15   5.79
tyan                6.91   7.66
vector-concat       4.68   4.87
vector-rev          4.66   4.83
vliw               14.73  15.82
wc-input1           5.52   6.24
wc-scanStream       5.57   6.30
zebra               6.79   7.11
zern                5.23   5.48

run time
benchmark         MLton0 MLton1
barnes-hut         31.24  29.45
boyer              27.62  30.00
checksum           99.28  67.03
count-graphs       31.27  27.08
DLXSimulator       70.79  71.10
fft                40.49  40.72
fib                37.67  37.03
flat-array         58.54  37.35
hamlet             28.66  30.14
imp-for            30.74  35.58
knuth-bendix       25.77  26.72
lexgen             38.15  30.91
life               13.52  11.88
logic              32.97  32.88
mandelbrot         36.29  34.08
matrix-multiply    11.01   8.01
md5                42.49  28.93
merge              55.18  53.99
mlyacc             28.82  30.80
model-elimination  46.65  48.58
mpuz               26.68  28.99
nucleic            25.06  27.92
output1            13.21   9.88
peek               21.19  25.00
psdes-random       29.78  27.81
ratio-regions      45.09  44.22
ray                15.44  14.60
raytrace           21.83  22.40
simple             30.48  31.27
smith-normal-form  19.95  20.04
tailfib            24.64  29.39
tak                17.02  16.77
tensor             28.07  26.63
tsp                37.80  30.18
tyan               34.87  35.60
vector-concat      66.67  62.47
vector-rev         75.02  75.72
vliw               31.45  31.27
wc-input1          40.52  40.89
wc-scanStream      23.81  23.48
zebra              39.93  33.05
zern               38.77  38.24