[PATCH] [LNT] Use Mann-Whitney U test to identify changes

[Tobias Grosser]

On 01/05/2014 23:27, Yi Kong wrote:
> This patch adds Mann-Whitney U tests to identify changes, as suggested by Tobias and Anton. User is able to configure the desired confidence level.

Hi Yi Kong,

thanks for this nice patch. I looked into it briefly by setting up an 
LNT server and adding a couple of the -O3 nightly test results to it. It 
seems at least with the default 0.5 confidence level this does not 
reduce the noise at all. Just switching aggregative function from 
minimum to mean helps here a lot more (any idea why?). Did you play with 
the confidence level and got an idea which level would be useful?
My very brief experiments showed that a value of 0.999 or even 0.9999 is 
something that gets us below the noise level. I verified this by looking 
at subsequent runs where the commits itself really just where 
documentation commits. Those commits should not show any noise. Even 
with those high confidence requirements, certain performance regressions 
such as r205965 can still be spotted. For me, this is already useful as 
we can really ask for extremely low noise answers,
which will help to at least catch the very clear performance 
regressions. (Compared to today, where even those are hidden in the 
reporting noise)

I would like to play with this a little bit more. Do you think it is 
possible to print the p value in the Run-Over-Run Changes Details?

Also, it may make sense to investigate this on another machine. I use 5 
identical but different machines. It may be interesting to see if runs 
on a same machine are more reliable and could get away with a lower 
confident interval. Did you do any experiments? Maybe with a higher run 
number 20?

Cheers,
Tobias