[PATCH] [LNT] Use Mann-Whitney U test to identify changes

[Yi Kong]

On 02/05/14 10:44, Renato Golin wrote:
> On 2 May 2014 10:32, Yi Kong <Yi.Kong at arm.com> wrote:
>> By looking at Mann-Whitney test, seems it requires data group to be
>> distributed in the same way, so your suggestion probably won't work.
>> Maybe we need to ask Anton to explain.
>
> What I understood is that the MWW doesn't expect the data to be
> generated with any order or around any mean value (non-parametric), so
> this shouldn't be a problem.

Non-parametric means we don't assume the distribution of the samples,
i.e. they are not required to be normally distributed. There are some
sophisticated algorithms designed to deal with data streams. I think we
can discuss that further after implemented the sliding window for
simpler(minimum, median) tests.

>> I do have a local installation on my desktop, but it is way too noisy
>> compared to ones Tobias is running. Thus the results are not very
>> representative. I'm trying to set up an ARM Highbank board at the moment.
>
> This is why he needed the confidence level so high. ;) I think this
> should be a parameter per buildbot, but I'm not sure where one would
> set that up.

Maybe we can just leave for users to decide. Let's first see if one
value works well across different machines.

>> Also can you share the results you are getting from your ARM runs? I am
>> not sure what magnitude of noise you are seeing.
>
> http://llvm.org/perf/db_default/v4/nts/machine/10
>
> cheers,
> --renato
>

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2548782