shootout scorecard

[Doug Bagley]

Stephen Weeks wrote:
> Hi Doug.  I think there is something wrong in the way that your scorecard is
> computed.

Possibly ... 

> In particular, mlton's average CPU is 6, while ocamlb's is 10.

That's the average CPU Rank on the language summary pages.  The rank
is the distance from the top position on the tests.  Then I just take
the arithmetic mean.  That number is fairly meaningless, but I wanted
something to allow me to colorize the ranks that were probably in need
of improvement.  

> Further mlton beats ocamlb in all but two of the benchmarks (reversefile,
> wordfreq), yet is only slightly ahead of it in the scoring.  Am I misleading
> myself?

Well, the scoring algorithm (as described on the bottom of the craps
page) may reflect which language is faster, but it doesn't reflect at
all how much faster.

For CPU (for example), the score on the scorecard is relative to the
highest and lowest scores.  For instance if language A takes 1 second,
language B takes 2 seconds, and language C takes 200 seconds, then the
base score is 200 + 1 = 201.  The scores for each language are calculated
by subtracting the CPU from the base, then normalizing to between 0 and
10.  So the score for A is 10 * (201 - 1) / 201 = 9.95.  score for B is
9.90, and the score for C is 0.05.  These scores are then multiplied
by the weight of each test, and then summed over all tests.

I've been thinking about replacing this algorithm ... I just haven't
spent the time on it yet.

Cheers,
Doug