Performance

This page compares the performance of a number of SML compilers on a range of benchmarks. For a performance comparison of many different languages, including MLton, see the Computer Language Shootout.

This page compares the following SML compiler versions.

MLton 20051202
ML Kit 4.1.4
Moscow ML 2.00
Poly/ML 4.1.3
SML/NJ 110.57

There are tables for run time, code size, and compile time.

Setup

All benchmarks were compiled and run on a 2.6 GHz Celeron with 2G of RAM. The benchmarks were compiled with the default settings for all the compilers, except for Moscow ML, which was passed the -orthodox -standalone -toplevel switches. The Poly/ML executables were produced by use-ing the file, followed by a PolyML.commit. The SML/NJ executables were produced by wrapping the entire program in a local declaration whose body performs an SMLofNJ.exportFn.

For more details, or if you want to run the benchmarks yourself, please see the benchmark directory of our Sources.

All of the benchmarks are available for download from this page. Some of the benchmarks were obtained from the SML/NJ benchmark suite. Some of the benchmarks expect certain input files to exist in the DATA subdirectory.

Run-time ratio

The following table gives the ratio of the run time of each benchmark when compiled by another compiler to the run time when compiled by MLton. That is, the larger the number, the slower the generated code runs. A number larger than one indicates that the corresponding compiler produces code that runs more slowly than MLton. A * in an entry means the compiler failed to compile the benchmark or that the benchmark failed to run.

benchmark	MLton	ML-Kit	MosML	Poly/ML	SML/NJ
barnes-hut.sml	1.0	*	*	*	1.6
boyer.sml	1.0	*	10.1	1.9	3.1
checksum.sml	1.0	*	*	*	*
count-graphs.sml	1.0	7.3	60.7	4.2	3.8
DLXSimulator.sml	1.0	*	*	*	*
fft.sml	1.0	1.2	*	24.2	0.8
fib.sml	1.0	0.9	5.0	1.2	1.3
flat-array.sml	1.0	2.2	35.0	1041.6	13.4
hamlet.sml	1.0	*	*	*	3.1
imp-for.sml	1.0	2.8	63.0	5.1	5.6
knuth-bendix.sml	1.0	*	19.8	4.8	4.6
lexgen.sml	1.0	2.5	5.0	1.7	1.5
life.sml	1.0	1.7	30.6	7.7	1.4
logic.sml	1.0	*	9.4	1.2	2.1
mandelbrot.sml	1.0	4.2	34.0	51.1	1.3
matrix-multiply.sml	1.0	8.3	42.5	13.2	5.3
md5.sml	1.0	*	*	*	*
merge.sml	1.0	*	*	1.1	7.9
mlyacc.sml	1.0	1.5	8.2	1.2	2.2
model-elimination.sml	1.0	*	*	*	2.6
mpuz.sml	1.0	2.3	78.2	4.6	4.1
nucleic.sml	1.0	*	*	23.5	0.8
output1.sml	1.0	30.7	61.4	16.2	14.4
peek.sml	1.0	15.2	176.9	17.9	11.3
psdes-random.sml	1.0	5.0	*	*	2.7
ratio-regions.sml	1.0	2.0	34.7	2.1	5.4
ray.sml	1.0	*	14.8	22.3	0.8
raytrace.sml	1.0	*	*	*	3.3
simple.sml	1.0	1.7	19.3	7.3	2.4
smith-normal-form.sml	1.0	*	*	*	>1000
tailfib.sml	1.0	1.0	51.9	3.2	1.4
tak.sml	1.0	1.2	17.0	1.3	2.0
tensor.sml	1.0	*	*	*	7.4
tsp.sml	1.0	3.4	31.8	*	17.7
tyan.sml	1.0	*	15.7	1.0	1.6
vector-concat.sml	1.0	1.2	20.4	2.0	20.4
vector-rev.sml	1.0	2.2	41.9	2.3	152.4
vliw.sml	1.0	*	*	*	2.5
wc-input1.sml	1.0	11.1	*	7.5	17.2
wc-scanStream.sml	1.0	22.1	*	203.7	11.5
zebra.sml	1.0	3.9	30.2	3.4	8.5
zern.sml	1.0	*	*	*	2.6

Note: for SML/NJ, the smith-normal-form.sml benchmark was killed after running for over 25,000 seconds.

Code size

The following table gives the code size of each benchmark in bytes. The size for MLton and the ML Kit is the sum of text and data for the standalone executable as reported by size. The size for Moscow ML is the size in bytes of the executable a.out. The size for Poly/ML is the difference in size of the database before the session start and after the commit. The size for SML/NJ is the size of the heap file created by exportFn and does not include the size of the SML/NJ runtime system (approximately 100K). A * in an entry means that the compiler failed to compile the benchmark.

benchmark	MLton	ML-Kit	MosML	Poly/ML	SML/NJ
barnes-hut.sml	103,231	*	*	*	433,216
boyer.sml	138,518	163,204	116,300	122,880	526,376
checksum.sml	52,794	*	*	*	*
count-graphs.sml	66,838	84,124	84,613	98,304	454,776
DLXSimulator.sml	129,398	*	*	*	*
fft.sml	64,797	80,240	84,046	65,536	434,256
fib.sml	47,738	18,588	79,892	49,152	415,488
flat-array.sml	47,762	23,820	80,034	49,152	410,680
hamlet.sml	1,256,813	*	*	*	1,412,360
imp-for.sml	47,626	19,372	80,040	57,344	400,424
knuth-bendix.sml	109,126	93,400	88,439	180,224	431,144
lexgen.sml	203,559	208,332	104,883	196,608	501,824
life.sml	66,130	78,084	83,390	65,536	414,760
logic.sml	106,614	116,880	87,251	114,688	440,360
mandelbrot.sml	47,690	77,004	81,340	57,344	404,520
matrix-multiply.sml	49,181	87,016	82,417	57,344	435,256
md5.sml	77,646	*	*	*	*
merge.sml	49,318	24,296	80,090	49,152	400,432
mlyacc.sml	507,431	473,748	148,286	2,850,816	820,336
model-elimination.sml	638,084	*	*	*	1,009,880
mpuz.sml	50,594	73,232	82,382	81,920	408,616
nucleic.sml	199,181	258,552	*	221,184	487,480
output1.sml	80,720	63,336	80,187	49,152	399,400
peek.sml	76,302	62,092	81,621	57,344	403,544
psdes-random.sml	48,402	25,196	*	*	421,944
ratio-regions.sml	73,914	95,924	87,482	73,728	443,448
ray.sml	183,243	108,848	89,859	147,456	493,712
raytrace.sml	265,332	*	*	*	636,112
simple.sml	222,914	192,032	94,396	475,136	756,840
smith-normal-form.sml	181,686	*	*	131,072	558,224
tailfib.sml	47,434	18,804	79,943	57,344	399,400
tak.sml	47,818	18,580	79,908	57,344	411,392
tensor.sml	97,677	*	*	*	450,672
tsp.sml	82,190	97,716	86,146	*	425,024
tyan.sml	134,910	137,800	91,586	196,608	477,272
vector-concat.sml	49,018	23,924	80,194	49,152	410,680
vector-rev.sml	48,246	24,104	80,078	57,344	410,680
vliw.sml	393,762	*	*	*	731,304
wc-input1.sml	101,850	129,212	85,771	49,152	404,520
wc-scanStream.sml	109,106	129,708	85,947	49,152	405,544
zebra.sml	141,146	41,532	83,422	90,112	419,896
zern.sml	91,087	*	*	*	479,384

Compile time

The following table gives the compile time of each benchmark in seconds. A * in an entry means that the compiler failed to compile the benchmark.

benchmark	MLton	ML-Kit	MosML	Poly/ML	SML/NJ
barnes-hut.sml	8.28	*	*	*	1.37
boyer.sml	8.14	8.99	0.39	0.12	3.20
checksum.sml	5.45	*	*	*	*
count-graphs.sml	6.12	2.06	0.14	0.05	0.90
DLXSimulator.sml	9.81	*	*	*	*
fft.sml	5.95	1.32	0.11	0.05	0.69
fib.sml	5.45	0.60	0.05	0.02	0.22
flat-array.sml	5.33	0.61	0.04	0.01	0.25
hamlet.sml	85.70	*	*	*	88.87
imp-for.sml	5.37	0.73	0.05	0.01	0.25
knuth-bendix.sml	7.09	4.11	0.19	0.12	1.60
lexgen.sml	11.02	7.21	0.40	0.26	3.63
life.sml	5.84	2.16	0.10	0.04	0.64
logic.sml	7.02	4.82	0.22	0.09	1.68
mandelbrot.sml	5.41	0.75	0.06	0.02	0.29
matrix-multiply.sml	5.39	0.77	0.06	0.01	0.30
md5.sml	6.01	*	*	*	*
merge.sml	5.41	0.62	0.06	0.02	0.26
mlyacc.sml	24.70	40.69	3.35	1.08	18.04
model-elimination.sml	25.04	*	*	*	28.79
mpuz.sml	5.41	1.07	0.07	0.03	0.45
nucleic.sml	14.24	24.79	*	0.36	2.78
output1.sml	6.05	0.68	0.05	0.01	0.23
peek.sml	6.04	0.70	0.05	0.02	0.25
psdes-random.sml	5.39	0.75	*	*	64.13
ratio-regions.sml	6.63	4.02	0.21	0.11	1.50
ray.sml	9.51	3.02	0.15	0.08	1.03
raytrace.sml	13.92	*	*	*	5.08
simple.sml	11.40	13.19	0.43	0.21	3.76
smith-normal-form.sml	8.90	*	*	0.10	2.25
tailfib.sml	5.35	0.64	0.05	0.02	0.24
tak.sml	5.36	0.62	0.05	0.01	0.22
tensor.sml	8.75	*	*	*	2.81
tsp.sml	6.50	1.93	0.15	*	0.66
tyan.sml	8.86	6.25	0.30	0.17	2.28
vector-concat.sml	5.52	0.68	0.05	0.01	0.25
vector-rev.sml	5.33	0.64	0.05	0.02	0.26
vliw.sml	18.28	*	*	*	13.12
wc-input1.sml	6.85	0.68	0.07	0.02	0.27
wc-scanStream.sml	7.07	0.69	0.06	0.02	0.29
zebra.sml	8.57	2.30	0.09	0.04	0.78
zern.sml	6.20	*	*	*	0.65