This page compares the performance of a number of SML compilers on a range of benchmarks.
This page compares the following SML compiler versions.
There are tables for run time, code size, and compile time.
Setup
All benchmarks were compiled and run on a 2.6 GHz Core i7-5600U with 16G of
RAM. The benchmarks were compiled with the default settings for all
the compilers, except for Moscow ML, which was passed the
-orthodox -standalone -toplevel switches. The Poly/ML executables
were produced using polyc.
The SML/NJ executables were produced by wrapping the entire program in
a local declaration whose body performs an SMLofNJ.exportFn.
For more details, or if you want to run the benchmarks yourself,
please see the benchmark directory of our
Sources.
All of the benchmarks are available for download from this page. Some
of the benchmarks were obtained from the SML/NJ benchmark suite. Some
of the benchmarks expect certain input files to exist in the
DATA subdirectory.
Run-time ratio
The following table gives the ratio of the run time of each benchmark when compiled by another compiler to the run time when compiled by MLton. That is, the larger the number, the slower the generated code runs. A number larger than one indicates that the corresponding compiler produces code that runs more slowly than MLton. A * in an entry means the compiler failed to compile the benchmark or that the benchmark failed to run.
| benchmark | MLton | ML-Kit | MosML | Poly/ML | SML/NJ |
|---|---|---|---|---|---|
1.00 |
10.11 |
19.36 |
2.98 |
1.24 |
|
1.00 |
* |
7.87 |
1.22 |
1.75 |
|
1.00 |
30.79 |
* |
10.94 |
9.08 |
|
1.00 |
6.51 |
40.42 |
2.34 |
2.32 |
|
1.00 |
0.97 |
* |
0.60 |
* |
|
1.00 |
0.50 |
11.50 |
0.42 |
0.42 |
|
1.00 |
7.35 |
81.51 |
4.03 |
1.19 |
|
1.00 |
1.41 |
10.94 |
1.25 |
1.17 |
|
1.00 |
7.19 |
68.33 |
5.28 |
13.16 |
|
1.00 |
4.97 |
22.85 |
1.58 |
* |
|
1.00 |
4.99 |
57.84 |
3.34 |
4.67 |
|
1.00 |
* |
18.43 |
3.18 |
3.06 |
|
1.00 |
2.76 |
7.94 |
3.19 |
* |
|
1.00 |
1.80 |
20.19 |
0.89 |
1.50 |
|
1.00 |
5.10 |
11.06 |
1.15 |
1.27 |
|
1.00 |
3.50 |
25.52 |
1.33 |
1.28 |
|
1.00 |
29.40 |
183.02 |
7.41 |
15.19 |
|
1.00 |
95.18 |
* |
32.61 |
47.47 |
|
1.00 |
1.42 |
* |
0.74 |
3.24 |
|
1.00 |
1.83 |
8.45 |
0.84 |
* |
|
1.00 |
4.03 |
12.42 |
1.70 |
2.25 |
|
1.00 |
3.73 |
57.44 |
2.05 |
3.22 |
|
1.00 |
3.96 |
* |
1.73 |
1.20 |
|
1.00 |
6.26 |
30.85 |
7.82 |
5.99 |
|
1.00 |
9.37 |
44.78 |
2.18 |
2.15 |
|
1.00 |
* |
* |
2.79 |
3.59 |
|
1.00 |
5.68 |
165.56 |
3.92 |
37.52 |
|
1.00 |
12.05 |
25.08 |
8.73 |
1.75 |
|
1.00 |
* |
* |
2.11 |
3.33 |
|
1.00 |
2.95 |
24.03 |
3.67 |
1.93 |
|
1.00 |
* |
* |
1.04 |
* |
|
1.00 |
1.88 |
28.01 |
0.70 |
2.67 |
|
1.00 |
1.58 |
23.57 |
0.90 |
1.04 |
|
1.00 |
1.69 |
15.90 |
1.57 |
2.01 |
|
1.00 |
* |
* |
* |
2.07 |
|
1.00 |
2.19 |
66.76 |
3.27 |
1.48 |
|
1.00 |
* |
19.43 |
1.08 |
1.03 |
|
1.00 |
13.85 |
* |
1.80 |
12.48 |
|
1.00 |
* |
* |
* |
13.92 |
|
1.00 |
7.88 |
68.85 |
9.39 |
68.80 |
|
1.00 |
2.46 |
15.39 |
1.43 |
1.55 |
|
1.00 |
6.00 |
* |
29.25 |
9.54 |
|
1.00 |
80.43 |
* |
19.45 |
8.71 |
|
1.00 |
4.62 |
35.56 |
1.68 |
9.97 |
|
1.00 |
* |
* |
* |
1.60 |
Note: for SML/NJ, the
smith-normal-form.sml
benchmark was killed after running for over 51,000 seconds.
Code size
The following table gives the code size of each benchmark in bytes.
The size for MLton and the ML Kit is the sum of text and data for the
standalone executable as reported by size. The size for Moscow
ML is the size in bytes of the executable a.out. The size for
Poly/ML is the difference in size of the database before the session
start and after the commit. The size for SML/NJ is the size of the
heap file created by exportFn and does not include the size of
the SML/NJ runtime system (approximately 100K). A * in an entry means
that the compiler failed to compile the benchmark.
| benchmark | MLton | ML-Kit | MosML | Poly/ML | SML/NJ |
|---|---|---|---|---|---|
180,788 |
810,267 |
199,503 |
148,120 |
402,480 |
|
250,246 |
* |
248,018 |
196,984 |
496,664 |
|
122,422 |
225,274 |
* |
106,088 |
406,560 |
|
151,878 |
250,126 |
187,048 |
144,032 |
428,136 |
|
223,073 |
827,483 |
* |
272,664 |
* |
|
122,350 |
87,586 |
181,415 |
106,072 |
380,928 |
|
145,008 |
237,230 |
186,228 |
131,400 |
418,896 |
|
122,310 |
87,402 |
181,312 |
106,088 |
380,928 |
|
121,958 |
104,102 |
181,464 |
106,072 |
394,256 |
|
1,503,849 |
2,280,691 |
407,219 |
2,249,504 |
* |
|
122,078 |
89,346 |
181,470 |
106,088 |
381,952 |
|
193,145 |
* |
192,659 |
161,080 |
400,408 |
|
308,296 |
826,819 |
213,128 |
268,272 |
* |
|
141,862 |
721,419 |
186,463 |
118,552 |
384,024 |
|
211,086 |
782,667 |
188,908 |
198,408 |
409,624 |
|
122,086 |
700,075 |
183,037 |
106,104 |
386,048 |
|
124,398 |
280,006 |
184,328 |
110,232 |
416,784 |
|
150,497 |
271,794 |
* |
122,624 |
399,416 |
|
123,846 |
100,858 |
181,542 |
106,136 |
381,960 |
|
678,920 |
1,233,587 |
263,721 |
576,728 |
* |
|
846,779 |
1,432,283 |
297,108 |
777,664 |
985,304 |
|
124,126 |
229,078 |
184,440 |
114,584 |
392,232 |
|
298,038 |
507,186 |
* |
475,808 |
456,744 |
|
157,973 |
699,003 |
181,680 |
118,800 |
380,928 |
|
156,401 |
201,138 |
183,438 |
110,456 |
385,072 |
|
126,486 |
106,166 |
* |
106,088 |
393,256 |
|
150,174 |
265,694 |
190,088 |
184,536 |
414,760 |
|
260,863 |
736,795 |
195,064 |
198,976 |
512,160 |
|
384,905 |
* |
* |
446,424 |
623,824 |
|
365,578 |
895,139 |
197,765 |
1,051,952 |
708,696 |
|
286,474 |
* |
* |
262,616 |
547,984 |
|
119,102 |
140,626 |
183,249 |
106,088 |
390,160 |
|
122,110 |
87,890 |
181,369 |
106,072 |
381,952 |
|
122,246 |
87,402 |
181,349 |
106,088 |
376,832 |
|
186,545 |
* |
* |
* |
421,984 |
|
163,033 |
722,571 |
188,634 |
126,984 |
393,264 |
|
235,449 |
* |
195,401 |
184,816 |
478,296 |
|
123,790 |
104,398 |
* |
106,200 |
394,256 |
|
123,846 |
* |
* |
* |
405,552 |
|
122,982 |
104,614 |
181,534 |
106,072 |
394,256 |
|
538,074 |
1,182,851 |
249,884 |
580,792 |
749,752 |
|
186,152 |
699,459 |
191,347 |
127,200 |
386,048 |
|
196,232 |
700,131 |
191,539 |
127,232 |
387,072 |
|
230,433 |
128,354 |
186,322 |
127,048 |
390,184 |
|
156,902 |
* |
* |
* |
453,768 |
Compile time
The following table gives the compile time of each benchmark in seconds. A * in an entry means that the compiler failed to compile the benchmark.
| benchmark | MLton | ML-Kit | MosML | Poly/ML | SML/NJ |
|---|---|---|---|---|---|
2.70 |
0.89 |
0.15 |
0.29 |
0.20 |
|
2.87 |
* |
0.14 |
0.20 |
0.41 |
|
2.21 |
0.24 |
* |
0.07 |
0.05 |
|
2.28 |
0.34 |
0.04 |
0.11 |
0.21 |
|
2.93 |
1.01 |
* |
0.27 |
* |
|
2.23 |
0.20 |
0.01 |
0.07 |
0.04 |
|
2.35 |
0.28 |
0.03 |
0.09 |
0.10 |
|
2.16 |
0.19 |
0.01 |
0.07 |
0.04 |
|
2.16 |
0.20 |
0.01 |
0.07 |
0.04 |
|
12.28 |
19.25 |
23.75 |
6.44 |
* |
|
2.14 |
0.20 |
0.01 |
0.08 |
0.04 |
|
2.48 |
* |
0.08 |
0.14 |
0.23 |
|
3.31 |
0.75 |
0.15 |
0.22 |
* |
|
2.25 |
0.32 |
0.03 |
0.09 |
0.10 |
|
2.72 |
0.57 |
0.07 |
0.17 |
0.21 |
|
2.14 |
0.24 |
0.01 |
0.07 |
0.04 |
|
2.14 |
0.24 |
0.01 |
0.08 |
0.05 |
|
2.31 |
0.39 |
* |
0.12 |
0.27 |
|
2.15 |
0.21 |
0.01 |
0.07 |
0.04 |
|
7.07 |
4.53 |
2.05 |
0.80 |
* |
|
6.78 |
4.76 |
1.20 |
1.65 |
4.78 |
|
2.14 |
0.28 |
0.02 |
0.08 |
0.07 |
|
3.96 |
2.12 |
* |
0.37 |
0.49 |
|
2.30 |
0.22 |
0.01 |
0.07 |
0.04 |
|
2.26 |
0.20 |
0.01 |
0.07 |
0.04 |
|
2.12 |
0.22 |
* |
9.83 |
12.55 |
|
2.59 |
0.47 |
0.07 |
0.16 |
0.24 |
|
2.95 |
0.46 |
0.05 |
0.17 |
0.14 |
|
3.93 |
* |
* |
0.45 |
0.74 |
|
3.42 |
1.23 |
0.30 |
0.32 |
0.53 |
|
3.23 |
* |
* |
0.15 |
0.32 |
|
2.25 |
0.28 |
0.01 |
0.08 |
0.05 |
|
2.24 |
0.21 |
0.01 |
0.08 |
0.05 |
|
2.23 |
0.20 |
0.01 |
0.08 |
0.05 |
|
2.73 |
* |
* |
* |
0.44 |
|
2.42 |
0.38 |
0.05 |
0.11 |
0.11 |
|
2.93 |
* |
0.10 |
0.27 |
0.31 |
|
2.23 |
0.22 |
* |
0.07 |
0.04 |
|
2.18 |
* |
* |
* |
0.04 |
|
2.23 |
0.22 |
0.01 |
0.08 |
0.05 |
|
5.25 |
2.93 |
0.63 |
0.94 |
1.85 |
|
2.46 |
0.24 |
0.01 |
0.08 |
0.05 |
|
2.61 |
0.25 |
0.01 |
0.08 |
0.05 |
|
2.99 |
0.35 |
0.03 |
0.09 |
0.11 |
|
2.31 |
* |
* |
* |
0.11 |