benchmarks

Matthew Fluet Matthew Fluet <fluet@CS.Cornell.EDU>
Mon, 5 Nov 2001 14:09:56 -0500 (EST)


Here are the results of benchmarking the 20011006 release versus the new
SSA passes.  Obviously, a little ways to go, but I just wanted to try out
the SSA passes on another set of programs.  All of the "Out of memory"
failures can be attributed to non-tail loops that should be turned into
tail loops.  Compile times are a bit high, but I think that's reasonable
since I'm running removeUnused after every pass, which in turn in
introducing lots of extra wrapper blocks that aren't being shrunk.

MLton0 -- mlton-stable 
MLton1 -- mlton -drop-pass removeUnused1CPS -drop-pass leafInlineCPS
-drop-pass raiseToJump1CPS -drop-pass contify1CPS -drop-pass
localFlatten1CPS -drop-pass uselessCPS -drop-pass removeUnused2CPS
-drop-pass unusedArgs1CPS -drop-pass simplifyTypesCPS -drop-pass
polyEqualCPS -drop-pass contify2CPS -drop-pass inlineCPS -drop-pass
localFlatten2CPS -drop-pass removeUnused3CPS -drop-pass raiseToJump2CPS
-drop-pass contify3CPS -drop-pass unusedArgs2CPS -drop-pass
introduceLoopsCPS -drop-pass loopInvariantCPS -drop-pass flattenCPS
-drop-pass localFlatten3CPS -drop-pass commonSubexpCPS -drop-pass
commonBlockCPS -drop-pass redundantTestsCPS -drop-pass redundantCPS
-drop-pass unusedArgs3CPS -drop-pass removeUnused4CPS
compile time
benchmark         MLton0 MLton1
barnes-hut           2.2    8.0
checksum             0.6    1.1
count-graphs         1.6    3.8
DLXSimulator         3.8   13.0
fft                  1.1    3.8
fib                  0.6    0.9
hamlet              41.5  105.1
knuth-bendix         2.0    4.2
lexgen               4.7   14.8
life                 1.2    2.7
logic                5.5   14.0
mandelbrot           0.6    1.1
matrix-multiply      0.6    1.3
md5                  1.5    3.3
merge                0.6    1.1
mlyacc              19.1   44.8
mpuz                 0.8    1.7
nucleic              2.9    5.2
peek                 0.9    2.6
psdes-random         0.6    1.2
ratio-regions        2.3    8.0
ray                  3.0   10.3
raytrace             8.1   24.1
simple               5.8   20.1
smith-normal-form    7.4   16.6
tailfib              0.6    0.9
tak                  0.6    0.9
tensor               2.6    9.5
tsp                  1.4    3.8
tyan                 3.4   12.9
vector-concat        0.6    1.2
vector-rev           0.7    1.2
vliw                11.0   36.1
wc-input1            1.4    4.0
wc-scanStream        1.5    4.6
zebra                8.3   10.4
zern                 0.9    2.5
run time
benchmark         MLton0 MLton1
barnes-hut           3.9   66.3
checksum             3.2      *  -- Out of memory
count-graphs         4.7   74.2
DLXSimulator        15.1   75.6
fft                  7.8  143.9
fib                  3.4    7.6
hamlet               8.1   89.8
knuth-bendix         6.5   32.2
lexgen              10.5  253.4
life                 7.9   13.3
logic               25.7   46.9
mandelbrot           6.7  474.9
matrix-multiply      5.2  141.0
md5                  3.3  121.4
merge               49.0  116.9
mlyacc               9.4  102.5
mpuz                 4.6   57.9
nucleic              6.8  144.0
peek                 3.4   35.1
psdes-random         3.4      *  -- Out of memory
ratio-regions        8.2  395.2
ray                  3.8   89.7
raytrace             4.6  195.0
simple               6.0   79.1
smith-normal-form    0.9      *  -- IntInf/flattening bug
tailfib             16.4      *  -- Out of memory
tak                  7.9  103.8
tensor               7.1 1824.7
tsp                  9.0  165.3
tyan                19.5  153.8
vector-concat        5.7  404.0
vector-rev           4.1  209.8
vliw                 6.2   66.7
wc-input1            2.2   55.9
wc-scanStream        3.6  131.3
zebra                2.1    9.1
zern                33.8 1199.9
run time ratio
benchmark       MLton1
barnes-hut        16.8
count-graphs      15.6
DLXSimulator       5.0
fft               18.4
fib                2.2
hamlet            11.1
knuth-bendix       4.9
lexgen            24.2
life               1.7
logic              1.8
mandelbrot        71.0
matrix-multiply   27.3
md5               37.0
merge              2.4
mlyacc            10.9
mpuz              12.5
nucleic           21.2
peek              10.2
ratio-regions     48.1
ray               23.4
raytrace          42.8
simple            13.2
tak               13.2
tensor           255.8
tsp               18.3
tyan               7.9
vector-concat     71.2
vector-rev        50.8
vliw              10.8
wc-input1         25.1
wc-scanStream     36.6
zebra              4.4
zern              35.5
size
benchmark          MLton0    MLton1
barnes-hut         59,793   278,920
checksum           20,917    39,864
count-graphs       40,461   126,856
DLXSimulator       78,237   419,896
fft                29,441   126,836
fib                20,909    34,520
hamlet            945,328 2,939,955
knuth-bendix       59,710   125,273
lexgen            122,061   462,920
life               38,565    87,896
logic             147,501   436,984
mandelbrot         20,901    40,512
matrix-multiply    21,309    45,528
md5                34,038   113,345
merge              21,885    38,408
mlyacc            409,501 1,388,752
mpuz               26,645    59,040
nucleic            60,653   180,352
peek               28,542    86,841
psdes-random       21,901    43,688
ratio-regions      41,893   268,568
ray                66,688   331,059
raytrace          159,381   791,056
simple            146,913   681,644
smith-normal-form 141,053   346,756
tailfib            20,637    34,888
tak                20,957    35,624
tensor             62,516   323,843
tsp                33,774   128,385
tyan               77,054   436,169
vector-concat      21,557    42,384
vector-rev         21,389    40,928
vliw              261,417 1,193,180
wc-input1          39,222   133,561
wc-scanStream      41,614   155,153
zebra             103,502   253,049
zern               26,504    80,619