shootout scorecard

[Stephen Weeks]

I thought a bit about your scoring system, and invented a new formula :-) that
better captures the information I care about, which is comparing the good
compilers.  The problem with your current formula, as I see it, is that the
performance of the bad compilers has a tremendous impact on the scoring.  For
example, if a new compiler shows up that is worse than the worst compiler by a
factor of two, then everybody's score is compressed into the top half of the
5-10 range.  The effect is that the scores do not distinguish much between the
good compilers.

To address, this, I propose the following.  The score for compiler C is

10 * (best compiler CPU time / C's CPU time)

This gives a number between 0 and 10, but will compress the bad compilers
(e.g. giving all the compilers more than 10X slower than the best a score less
than 1).

I would be interested to see how this formula would affect things.

My 2 cents.

Thanks for keeping up the site.