benchmarking

Stephen Weeks MLton@sourcelight.com
Tue, 3 Jul 2001 15:33:31 -0700


Here are some numbers similar to what's going to go on the performance page,
except that old mlton (20000906) won't be there.  But it's nice to see how much
better new MLton is than the old -- uniformly across the board!  It's also nice
to see how much better we are than the new Kit (3.9.0) and the new SML/NJ
(110.33), with the exception of a few times when NJ is better.

run time ratio (to MLton 20010708)
			old
bench              kit mlton mosml  nj
barnes-hut         3.5   1.3  12.3  0.8
checksum             *   1.7     *  3.0
count-graphs       6.4   1.4  21.1  1.5
fft                  *   1.8  24.9  1.0
fib                1.0   1.3   4.7  1.1
hamlet               *     *     *  1.4
knuth-bendix       4.7   1.1  12.1  2.1
lexgen             2.4   2.8   5.6  1.6
life               6.9   1.9  19.8  1.0
logic                *   1.4   3.4  0.6
mandelbrot         4.4   1.7  33.0  1.4
matrix-multiply   10.7   2.0  37.2  3.4
mlyacc               *   1.8   7.6  1.7
mpuz               8.4   2.2  47.7  3.0
nucleic              *   1.2  15.6  0.7
peek               6.9   4.1  31.8  2.1
psdes-random       8.4   1.7     *  2.3
ratio-regions     12.0   2.6  46.8  5.1
ray                4.4   1.9  11.4  0.8
raytrace             *   1.6     *  2.3
simple             2.3   1.3  12.7  1.4
smith-normal-form    *   1.1     * 95.6
tak                1.6   1.6   7.4  1.3
tensor               *   1.7     *  5.8
tsp                2.5   1.5  14.5  1.6
vector-concat      9.5   4.1  16.9  5.9
vector-rev        10.1   3.4  32.3 19.2
vliw               2.6   1.6   7.7  1.3
wc-input1          0.0   4.4     *  6.8
wc-scanStream      0.0   2.5     *  1.5
zebra             12.2   1.8  23.6  7.0
zern                 *   1.4     *  1.5

compile time
bench            kit mlton old-mlton mosml    nj
barnes-hut         8.2   2.5       5.6   0.8   1.7
checksum             *   0.7       1.1     *   0.2
count-graphs       2.6   1.7       3.9   0.2   1.1
fft                2.1   1.5       2.9   0.2   0.9
fib                1.0   0.6       1.0   0.1   0.2
hamlet               *  52.8         *     * 100.4
knuth-bendix       5.4   2.3       6.3   0.4   2.0
lexgen            10.2   5.8      16.6   0.8   4.8
life               2.9   1.4       2.9   0.2   0.7
logic              6.7   7.5      20.3   0.4   2.0
mandelbrot         1.1   0.7       1.1   0.1   0.2
matrix-multiply    1.2   0.7       1.2   0.1   0.3
mlyacc            60.1  19.6     156.0   7.4  24.2
mpuz               1.3   0.9       1.6   0.1   0.4
nucleic           28.0   4.3       6.6   2.1   2.9
peek               1.0   1.1       2.4   0.1   0.2
psdes-random       1.1   0.7       1.2     *   0.3
ratio-regions      5.2   3.1       6.7   0.4   2.1
ray                3.8   3.6      10.0   0.3   1.1
raytrace             *  10.3      28.4     *   6.7
simple            14.5   7.5      39.8   0.9   4.5
smith-normal-form    *   8.1      70.5     *   3.5
tak                1.0   0.6       1.0   0.1   0.2
tensor               *   3.1       7.7     *   3.5
tsp                2.7   1.8       3.4   0.3   0.8
vector-concat      1.0   0.7       1.2   0.1   0.2
vector-rev         1.0   0.7       1.1   0.1   0.2
vliw              37.2  12.5      64.1   3.0  18.2
wc-input1          1.0   1.7       3.2   0.1   0.3
wc-scanStream      1.1   1.8       3.3   0.1   0.3
zebra              2.9   4.8      16.1   0.1   0.8
zern                 *   1.1       2.1     *   0.7

size
                                     old
bench                 kit   mlton   mlton   mosml        nj
barnes-hut        179,964  41,194  56,472  94,990   331,768
checksum                *  18,121  21,190       *   332,504
count-graphs      109,996  37,305  43,766  84,575   355,376
fft               107,348  31,921  32,730  84,095   332,808
fib                68,700  17,905  20,774  79,878   310,968
hamlet                  * 979,200       *       * 1,263,816
knuth-bendix      115,348  56,410  76,500  88,439   321,504
lexgen            227,444 128,601 168,044 104,883   390,136
life              100,380  34,593  38,062  83,390   305,120
logic             136,180 145,577 151,158  87,252   331,744
mandelbrot        101,660  17,753  19,598  81,341   311,992
matrix-multiply   118,380  18,673  21,958  81,879   338,632
mlyacc            525,580 421,737 595,532 148,286   700,456
mpuz               89,564  22,961  25,334  82,383   320,184
nucleic           234,188  55,033  48,306 207,154   354,288
peek               77,500  26,074  41,324  81,618   311,016
psdes-random       84,636  19,065  22,374       *   313,016
ratio-regions     111,356  54,425  55,174  87,485   335,856
ray               123,420  70,808 100,476  89,860   384,072
raytrace                * 178,237 215,473       *   510,056
simple            194,092 166,265 201,080  94,397   641,056
smith-normal-form       * 139,153 153,440       *   483,400
tak                68,468  17,817  19,806  79,928   306,872
tensor                  *  50,521  61,468       *   342,048
tsp               115,732  34,394  42,866  86,140   322,552
vector-concat      77,756  18,569  21,878  80,191   317,128
vector-rev         77,956  18,713  21,582  80,073   317,128
vliw              418,268 260,745 736,036 135,386   618,576
wc-input1         144,636  38,330  49,157  86,900   311,992
wc-scanStream     145,084  40,882  49,501  87,076   313,016
zebra              85,516 106,234 135,996  83,419   310,256
zern                    *  24,880  25,554       *   326,664