[MLton-devel] Porting from Moscow ML

[Stephen Weeks]

> I'd actually argue against this.  I think that benchmarks that are running
> for under a minute (even if they are iterated until over a minute of
> execution time), are too susceptible to random variations.  

I agree with you, except that I think that the randomness only starts
to hurt under 10 seconds.  I think a minute is fine.

> there was a benchmark (I think count-graphs) that came out about 20%
> slower under the basis-2002 branch than under the basis-1997 branch.
> After looking at it for a while, I discovered that under basis-2002 the
> exit function wasn't inlined, while the "main" code was pretty much
> identical.  The extra time in basis-2002 was entirely due to the extra
> cost of exiting the program, which was run multiple times.

We have way too many benchmarks (count-graphs included) that run for
only a couple of seconds (with MLton).  

> Anyways, my point is that a benchmark should be influenced by the cost of
> what the program is "really" doing, not the incidentals of statup and
> shutdown cost (unless the program is really just used in that type of
> situation).

Agreed.

> Probably the best solution would be to have a set of "big" benchmarks that
> run for 10+ minutes, which could be excluded when just doing a quick set
> of benchmarks. 

The problem with a benchmark that runs for 10 minutes under MLton is
that it probably runs for over an hour with Moscow ML, and will take
(at least) a few hours to complete.  I'd like to keep it so that the
all the benchmarks can be run overnight.


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
MLton-devel mailing list
MLton-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mlton-devel