[LLVMdev] Why is the default LNT aggregation function min instead of mean

[Chris Matthews]

I think the idea with min is that it would the the ideal fastest run.  The other runs were ‘slowed' by system noise or something else.


On Jan 16, 2014, at 5:03 PM, Tobias Grosser <tobias at grosser.es> wrote:

> Hi,
> 
> I am currently investigating how to ensure that LNT only shows relevant performance regressions for the -O3 performance tests I am running.
> 
> One question that came up here is why the default aggregate function for LNT is 'min' instead of 'mean'. This looks a little surprising from the statistical point, but also from looking at my test results picking 'min' seems to be an inferior choice.
> 
> For all test runs I have looked at, picking mean largely reduces the run-over-run changes reported due to noise.
> 
> See this run e.g:
> 
> If we use the median, we just get just one change reported:
> 
> http://llvm.org/perf/db_default/v4/nts/20661?num_comparison_runs=10&test_filter=&test_min_value_filter=&aggregation_fn=median&compare_to=20659&submit=Update
> 
> If you use min, we get eight reports one claiming over 100% performance
> reduction for a case that is really just pure noise. I am planning to look into using better statistical methods. However, as a start, could we switch the default to 'mean'?
> 
> Cheers,
> Tobias