benchmarking confusion

[Henry Cejtin]

How big is the vector?  If it is large enough, then you can get pimped by
caching effects.  The caches are physically tagged (i.e., it is the physical
address, not the virtual address), so if you are unlucky, you can have an
adjacent area of your address space with all of the pages in that having
physical addresses which are congruent modulo the L2 cache size.  Thus they
will all compete for the same L2 cache slot, and the L2 cache is only something
like 4-way associative.  Thus you are save up until 4 * 4K = 16K bytes, but
larger than that and you can thrash even though you the size of the array is
less than the size of the L2 cache.
The result of all of this is that the time would vary based on what physical
memory you get.  Rico and I have played games to see if this effect is real,
and it is.  There were some discussions on `page coloring' in the Linux kernel
group which is an attempt to avoid this, but nothing was actually implemented.
Could this explain things?