[MLton] latest benchmarks

[Matthew Fluet]

Jesper Louis Andersen wrote:
> I've attached a benchmark run for x86-FreeBSD on my machine. 

Thanks.  Those numbers look very good.  count-graphs and vliw seem to be 
the biggest regressions; they are also the outliers on the amd64 
benchmarks.  I looked at count-graphs a little bit yesterday, and as 
best I could determine, there isn't a single culprit; it seems that the 
changes to the Basis Library (primarily using coercions to handle 
changing the default and c integer sizes) just produces many more small 
functions that have the effect of not being sufficiently inlined early 
in the optimizer.  I'm experimenting with tweaking the initial 
leafInline optimization to execute more than once, so that more of these 
small functions get inlined.

 > An interesting
> thing from the benchmark-run of the recent trunk:
> 
> testing real
> 666c666
> < 11.59195423
> ---
>> 11.59195518
> 817c817
> < 11.59195423
> ---
>> 11.59195518
> 
> I note that
> 
> http://mlton.org/pages/PortStatus/attachments/x86-freebsd.log
> 
> has the regression as well.

Yeah, we generally ignore low bit floating point differences. 
Especially on the x86, it can be due to many different factors.
Feel free to check in a 'real.x86-freebsd.ok' file if you want to quell 
the message in the future.

> 
> On 6/21/07, Matthew Fluet <fluet at tti-c.org> wrote:
>>
>> Jesper Louis Andersen wrote:
>> > Yup, that was indeed the problem. With -codegen x86 it runs. I can add
>> the
>> > benchmark results besides yours in the same format in SVN if you like
>> it,
>> > or I can post them here.
>>
>> I "reset" the http://www.mlton.org/PortStatus wiki page.
>> I think it makes sense to accumulate logs and benchmarks there.
>>
>> > It will probably take some time to run anyway.
>>
>> Definitely.  It's multiple hours with 8 different compilers.
>>
>>
>> > On 6/21/07, Matthew Fluet <fluet at tti-c.org> wrote:
>> >>
>> >> Jesper Louis Andersen wrote:
>> >> > Hmm, it failed the -codegen native as well as -codegen amd64. I'll
>> >> > investigate further.
>> >>
>> >> For mlton-20051205 (on x86 or x86_64), -codegen native implies using
>> the
>> >> x86 codegen.
>> >>
>> >> For mlton.svn.trunk, -codegen native has been eliminated.
>> >> On x86, -codegen x86 implies using the x86 codegen.
>> >> On amd64, -codegen amd64 implies using the amd64 codegen.
>> >>
>> >> It would be easy to restore the -codegen native option on
>> >> mlton.svn.trunk and choose the appropriate native codegen for the
>> >> architecture.
>> >>
>> >> > On 6/21/07, Jesper Louis Andersen <jesper.louis.andersen at gmail.com>
>> >> wrote:
>> >> >>
>> >> >> I have a 32bit FreeBSD compile ready. Benchmarks to trickle in
>> >> tomorrow
>> >> >> when I
>> >> >> get it to crunch while at work (It takes a fair amount of time for
>> it
>> >> to
>> >> >> finish and I don't
>> >> >> want to mess too much with the laptop while it processes the
>> >> benchmarks).
>> >> >>
>> >> >> There are also a few tweaks needed to run on 64-bit FreeBSD. I hope
>> to
>> >> be
>> >> >> able to
>> >> >> look into them around the 1st of July.
>> >> >>
>> >> >> On 6/20/07, Matthew Fluet <fluet at tti-c.org> wrote:
>> >> >> >
>> >> >> > I've merged the x86_64 branch into trunk.  Since the previous
>> >> >> > announcement of the experimental release, there were only two
>> minor
>> >> >> bugs
>> >> >> > reported:
>> >> >> >   1) Bug with -align 8 on x86_64
>> >> >> >   2) Inconsistent behavior with -const 
>> 'MLton.detectOverflowfalse'
>> >> >> > These have both been fixed, and I'm pretty happy with the 
>> state of
>> >> the
>> >> >> > x86_64 port.
>> >> >> >
>> >> >> > I ran the benchmark suite to compare the last public release to
>> the
>> >> >> > current trunk.  It is a bit of an apples-to-oranges comparison,
>> >> since
>> >> I
>> >> >> > ran the benchmarks on an AMD Opteron (64-bit) system.  So, the
>> >> 20051205
>> >> >> > compiler (and its resulting executables) are running in 32-bit
>> mode,
>> >> >> > while the trunk compiler (and its resulting executables) are
>> running
>> >> in
>> >> >> > 64-bit mode.
>> >> >> >
>> >> >> > [BTW, it would be nice if someone could run a corresponding
>> >> benchmark
>> >> >> > suite on a 32-bit system, for a more apples-to-apples 
>> comparison.]
>> >> >> >
>> >> >> > You can see all of the results at:
>> >> >> >
>> >> >> >
>> >> >>
>> >>
>> http://mlton.org/cgi-bin/viewsvn.cgi/*checkout*/mlton/trunk/doc/x86_64-port-notes/bench-20070619.txt?rev=5659 
>>
>> >>
>> >> >>
>> >> >> >
>> >> >> > Some of the highlights:
>> >> >> >
>> >> >> > * Benchmarks were run on a uni-core, dual-processor AMD Opteron
>> >> >> 2.0GHz ,
>> >> >> > 8GB Memory, Fedora Core 6 machine (with gcc version 4.1.1 and
>> linux
>> >> >> > version 2.6.20 (x86_64)).
>> >> >> >
>> >> >> > * compile time and code size is up across the board on trunk vs
>> >> >> > 20051205.  I suspect that part of the code size increase can be
>> >> >> > attributed to the comparison of 32-bit executables to 64-bit
>> >> >> > executables.  Any 64-bit operation requires an additional 8bit
>> >> >> > instruction prefix (as do 32-bit ops that touch the extended
>> >> register
>> >> >> > set).  Compile time is probably partly explained by the bigger
>> Basis
>> >> >> > Library implementation (increasing elaboration time and carrying
>> >> more
>> >> >> > code through early optimizations), and partly by the fact that 
>> the
>> >> >> trunk
>> >> >> > compiler is executing a little slower than the 20051205 compiler.
>> >> >> >
>> >> >> > * recent versions of gcc are doing fairly well with the C
>> >> code.  (Note
>> >> >> > that using -codegen c with 20051205 uses the version of gcc on 
>> the
>> >> host
>> >> >> > machine.)  Indeed, the flat-array.sml benchmark needs to be
>> revised,
>> >> as
>> >> >> > gcc recognizes that the inner loop is pure (Overflow exceptions
>> are
>> >> >> > handled within the loop) and unused.  The SSA{,2} optimizer 
>> should
>> >> also
>> >> >> > discover that the loop may be optimized, but that is another
>> issue.
>> >> >> > GCC also does fairly well on the checksum benchmark with 
>> 20051205,
>> >> >> > though it does horribly on the checksum benchmark with trunk.
>> >> >> > I suspect that the later behavior is due to the fact that on
>> x86_64,
>> >> >> > sequences (arrays/vectors) are indexed by 64-bit integers in the
>> >> >> > primitive operations (sub, update, etc), but indexed by 32-bit
>> >> integers
>> >> >> > in the user code (Array.sub, Array.update, etc. since Int.int
>> >> >> > corresponds to Int32.int ).  Hence, there are quite a few 64/32
>> >> >> > conversions going on.
>> >> >> >
>> >> >> > * I note that with both native codegens and C codegens, with both
>> >> >> > 20051205 and trunk, that -align 8 often has a positive impact on
>> >> >> > runtime, and rarely has a significant negative impact.  This
>> >> might be
>> >> >> > due to the Opteron memory system.  Aligned reads probably help
>> most
>> >> on
>> >> >> > Real64 intensive benchmarks.
>> >> >> >
>> >> >> > * The amd64 codegen is doing alright as compared to the x86
>> >> codegen.  I
>> >> >> > see at most a factor of 2 slowdown, and a few speedups.  Again,
>> I'm
>> >> not
>> >> >> > sure what real conclusions can be drawn.  Some slowdowns are 
>> going
>> >> >> to be
>> >> >> > due to the changes to the runtime and Basis Library since
>> 20051205;
>> >> to
>> >> >> > isolate those, I need a comparison of 20051205 to trunk on a
>> 32-bit
>> >> >> > system.  Some slowdowns are probably going to be due to the
>> sequence
>> >> >> > indexing discussed above.
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > MLton mailing list
>> >> >> > MLton at mlton.org
>> >> >> > http://mlton.org/mailman/listinfo/mlton
>> >> >> >
>> >> >> >
>> >> >>
>> >> >
>> >> >
>> >> >
>> >>
>> ------------------------------------------------------------------------
>> >> >
>> >> > _______________________________________________
>> >> > MLton mailing list
>> >> > MLton at mlton.org
>> >> > http://mlton.org/mailman/listinfo/mlton
>> >>
>> >>
>> >
>> >
>> > 
>> ------------------------------------------------------------------------
>> >
>> > _______________________________________________
>> > MLton mailing list
>> > MLton at mlton.org
>> > http://mlton.org/mailman/listinfo/mlton
>>
>>
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> MLton mailing list
> MLton at mlton.org
> http://mlton.org/mailman/listinfo/mlton