[LLVMdev] [LNT] Question about results reliability in LNT infrustructure

[Jakob Stoklund Olesen]

On Jun 30, 2013, at 6:02 PM, Chris Matthews <chris.matthews at apple.com> wrote:

> This is probably another area where a bit of dynamic behavior could help.  When we find a regressions, kick off some runs to bisect back to where it manifests. This is what we would be doing manually anyway.  We could just search back with the set of regressing benchmarks, meaning the whole suite does not have to be run (unless it is a global regression).
> 
> There are situations where we see commit which make things slower then faster again, but so far those seem to be from experimental features being switched on then off.

This is an interesting paper: http://people.cs.umass.edu/~emery/pubs/stabilizer-asplos13.pdf

"However, caches and branch predictors make performance dependent on machine-specific parameters and the exact layout of code, stack frames, and heap objects. A single binary constitutes just one sample from the space of program layouts, regardless of the number of runs. Since compiler optimizations and code changes also alter layout, it is currently impossible to distinguish the impact of an optimization from that of its layout effects."

"We find that the performance impact of -O3 over -O2 optimizations is indistinguishable from random noise.”

Thanks,
/jakob