[PATCH] [LNT] Use Mann-Whitney U test to identify changes

Yi Kong Yi.Kong at arm.com
Fri May 2 02:32:55 PDT 2014


On 02/05/14 10:07, Renato Golin wrote:
> On 1 May 2014 22:27, Yi Kong <Yi.Kong at arm.com> wrote:
>> This patch adds Mann-Whitney U tests to identify changes, as suggested by Tobias and Anton. User is able to configure the desired confidence level.
>
> Hi Yi,
>
>  From what I understood, this test will only check for disjoint groups,
> so larger noise levels will still trigger differences. Also, if the
> number of runs is less than 4, this test will not be ran, so ARM
> benchmarks won't "benefit" from the check.
>
> For runs with less than 4 data points, you could consider the last N
> runs as one group, and the last N-1 runs + the current as another and
> run the same check.

By looking at Mann-Whitney test, seems it requires data group to be
distributed in the same way, so your suggestion probably won't work.
Maybe we need to ask Anton to explain.

> Have you used it on a perf installation locally? What's the impact?

I do have a local installation on my desktop, but it is way too noisy
compared to ones Tobias is running. Thus the results are not very
representative. I'm trying to set up an ARM Highbank board at the moment.

Also can you share the results you are getting from your ARM runs? I am
not sure what magnitude of noise you are seeing.

> cheers,
> --renato

Regards,
Yi Kong

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2548782





More information about the llvm-commits mailing list