[MLton-devel] Porting from Moscow ML

[Matthew Fluet]

> * make it run for something closer to a minute or less.  Actually
>   probably less is important, since I assume the other compilers will
>   be slower.

I'd actually argue against this.  I think that benchmarks that are running
for under a minute (even if they are iterated until over a minute of
execution time), are too susceptible to random variations.  Point in case,
there was a benchmark (I think count-graphs) that came out about 20%
slower under the basis-2002 branch than under the basis-1997 branch.
After looking at it for a while, I discovered that under basis-2002 the
exit function wasn't inlined, while the "main" code was pretty much
identical.  The extra time in basis-2002 was entirely due to the extra
cost of exiting the program, which was run multiple times.

Anyways, my point is that a benchmark should be influenced by the cost of
what the program is "really" doing, not the incidentals of statup and
shutdown cost (unless the program is really just used in that type of
situation).

Probably the best solution would be to have a set of "big" benchmarks that
run for 10+ minutes, which could be excluded when just doing a quick set
of benchmarks. 



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
MLton-devel mailing list
MLton-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mlton-devel