[MLton-devel] Porting from Moscow ML
Wed, 25 Sep 2002 12:37:12 -0700
> I'd actually argue against this. I think that benchmarks that are running
> for under a minute (even if they are iterated until over a minute of
> execution time), are too susceptible to random variations.
I agree with you, except that I think that the randomness only starts
to hurt under 10 seconds. I think a minute is fine.
> there was a benchmark (I think count-graphs) that came out about 20%
> slower under the basis-2002 branch than under the basis-1997 branch.
> After looking at it for a while, I discovered that under basis-2002 the
> exit function wasn't inlined, while the "main" code was pretty much
> identical. The extra time in basis-2002 was entirely due to the extra
> cost of exiting the program, which was run multiple times.
We have way too many benchmarks (count-graphs included) that run for
only a couple of seconds (with MLton).
> Anyways, my point is that a benchmark should be influenced by the cost of
> what the program is "really" doing, not the incidentals of statup and
> shutdown cost (unless the program is really just used in that type of
> Probably the best solution would be to have a set of "big" benchmarks that
> run for 10+ minutes, which could be excluded when just doing a quick set
> of benchmarks.
The problem with a benchmark that runs for 10 minutes under MLton is
that it probably runs for over an hour with Moscow ML, and will take
(at least) a few hours to complete. I'd like to keep it so that the
all the benchmarks can be run overnight.
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
MLton-devel mailing list