[MLton] benchmarks vs last release
Stephen Weeks
MLton@mlton.org
Fri, 13 Aug 2004 12:48:04 -0700
To make sure there are no problems with our upcoming release, I ran
the benchmarks to compare the performance of our last release to our
current internal version. The complete numbers are below. The upshot
is:
1. In compile time, there are no significant changes.
2. In code size, there are some significant improvements and no
significant worsenings. I suspect the improvements are mostly due
to some mistake in 20040227 that caused too much library code to be
included.
3. In run time, most benchmarks are roughly the same. There are some
nice improvements, some small slowdowns, and no show stoppers.
Here are the number of benchmarks with run time ratios in various
ranges.
# bench run-time ratio
------- --------------
3 [0.6, 0.7)
2 [0.7, 0.8)
5 [0.8, 0.9)
28 [0.9. 1.1)
4 [1.1, 1.2)
Here's all the numbers.
MLton0 -- mlton 20040227
MLton1 -- mlton HEAD
run time ratio
benchmark MLton1
barnes-hut 0.94
boyer 1.09
checksum 0.68
count-graphs 0.87
DLXSimulator 1.00
fft 1.01
fib 0.98
flat-array 0.64
hamlet 1.05
imp-for 1.16
knuth-bendix 1.04
lexgen 0.81
life 0.88
logic 1.00
mandelbrot 0.94
matrix-multiply 0.73
md5 0.68
merge 0.98
mlyacc 1.07
model-elimination 1.04
mpuz 1.09
nucleic 1.11
output1 0.75
peek 1.18
psdes-random 0.93
ratio-regions 0.98
ray 0.95
raytrace 1.03
simple 1.03
smith-normal-form 1.00
tailfib 1.19
tak 0.99
tensor 0.95
tsp 0.80
tyan 1.02
vector-concat 0.94
vector-rev 1.01
vliw 0.99
wc-input1 1.01
wc-scanStream 0.99
zebra 0.83
zern 0.99
size
benchmark MLton0 MLton1
barnes-hut 143,701 144,233
boyer 226,477 142,603
checksum 116,365 60,333
count-graphs 133,901 70,935
DLXSimulator 176,853 173,033
fft 134,777 70,083
fib 114,701 52,311
flat-array 114,673 52,339
hamlet 1,339,351 1,295,313
imp-for 114,685 52,159
knuth-bendix 152,549 147,873
lexgen 235,930 240,950
life 134,149 70,227
logic 174,485 113,191
mandelbrot 114,785 52,251
matrix-multiply 117,357 53,487
md5 123,529 115,533
merge 116,117 53,875
mlyacc 552,670 549,366
model-elimination 671,086 674,124
mpuz 118,877 56,387
nucleic 265,193 206,327
output1 133,271 126,535
peek 124,857 120,965
psdes-random 115,101 53,031
ratio-regions 142,969 81,251
ray 222,819 218,513
raytrace 325,759 301,955
simple 283,637 279,228
smith-normal-form 219,741 226,029
tailfib 114,813 51,975
tak 114,753 52,387
tensor 145,996 142,456
tsp 125,313 123,725
tyan 182,665 183,449
vector-concat 116,417 53,623
vector-rev 115,765 52,927
vliw 433,564 431,946
wc-input1 152,353 148,029
wc-scanStream 150,925 148,745
zebra 184,609 162,441
zern 148,770 128,776
compile time
benchmark MLton0 MLton1
barnes-hut 6.05 6.76
boyer 8.07 7.47
checksum 4.64 4.84
count-graphs 5.33 5.53
DLXSimulator 7.05 7.77
fft 5.16 5.27
fib 4.61 4.80
flat-array 4.64 4.81
hamlet 45.53 45.57
imp-for 4.64 4.83
knuth-bendix 5.69 6.30
lexgen 8.12 9.03
life 5.17 5.31
logic 6.15 6.71
mandelbrot 4.65 4.85
matrix-multiply 4.73 4.89
md5 4.93 5.48
merge 4.69 4.84
mlyacc 22.78 22.62
model-elimination 20.87 21.87
mpuz 4.79 4.96
nucleic 61.99 62.30
output1 4.96 5.50
peek 4.86 5.42
psdes-random 4.65 4.88
ratio-regions 5.79 5.98
ray 7.29 8.06
raytrace 11.18 11.40
simple 9.20 9.65
smith-normal-form 8.86 8.32
tailfib 4.62 4.81
tak 4.62 4.79
tensor 6.44 6.96
tsp 5.15 5.79
tyan 6.91 7.66
vector-concat 4.68 4.87
vector-rev 4.66 4.83
vliw 14.73 15.82
wc-input1 5.52 6.24
wc-scanStream 5.57 6.30
zebra 6.79 7.11
zern 5.23 5.48
run time
benchmark MLton0 MLton1
barnes-hut 31.24 29.45
boyer 27.62 30.00
checksum 99.28 67.03
count-graphs 31.27 27.08
DLXSimulator 70.79 71.10
fft 40.49 40.72
fib 37.67 37.03
flat-array 58.54 37.35
hamlet 28.66 30.14
imp-for 30.74 35.58
knuth-bendix 25.77 26.72
lexgen 38.15 30.91
life 13.52 11.88
logic 32.97 32.88
mandelbrot 36.29 34.08
matrix-multiply 11.01 8.01
md5 42.49 28.93
merge 55.18 53.99
mlyacc 28.82 30.80
model-elimination 46.65 48.58
mpuz 26.68 28.99
nucleic 25.06 27.92
output1 13.21 9.88
peek 21.19 25.00
psdes-random 29.78 27.81
ratio-regions 45.09 44.22
ray 15.44 14.60
raytrace 21.83 22.40
simple 30.48 31.27
smith-normal-form 19.95 20.04
tailfib 24.64 29.39
tak 17.02 16.77
tensor 28.07 26.63
tsp 37.80 30.18
tyan 34.87 35.60
vector-concat 66.67 62.47
vector-rev 75.02 75.72
vliw 31.45 31.27
wc-input1 40.52 40.89
wc-scanStream 23.81 23.48
zebra 39.93 33.05
zern 38.77 38.24