[LNT] r207898 - Use Mann-Whitney U test to identify changes

[Yi Kong]

Updated again. If SciPy is installed, it will use normal approximation
to compute MWU. Otherwise it will show an error page in large problem
sizes.

On Wed, 2014-05-07 at 17:16 +0100, Hal Finkel wrote:
> ----- Original Message -----
> > From: "Yi Kong" <Yi.Kong at arm.com>
> > To: "Hal Finkel" <hfinkel at anl.gov>
> > Cc: "LLVM Commits" <llvm-commits at cs.uiuc.edu>, "Renato Golin" <renato.golin at linaro.org>, "Anton Korobeynikov"
> > <anton at korobeynikov.info>, "Chris Matthews" <chris.matthews at apple.com>
> > Sent: Wednesday, May 7, 2014 11:06:36 AM
> > Subject: Re: [LNT] r207898 - Use Mann-Whitney U test to identify changes
> >
> > Updated. If the sample size if greater than 20, Mann-Whitney U test
> > won't be performed.
>
> Sorry, I was not clear...
>
> +        # Use Mann-Whitney U test to test null hypothesis that result is
>
> +        # unchanged.
>
> +        if len(self.samples) >= 4 and len(self.samples) <= 20 and\
>
> +        len(self.prev_samples) >= 4 and len(self.prev_samples) <= 20:
>
> +            same = stats.mannwhitneyu(self.samples, self.prev_samples, self.confidence_lv)
>
> +            if same:
>
> +                return UNCHANGED_PASS
>
> This is going to be very confusing; in theory, this means that increasing the number of samples will give you better results until you hit 20, and then suddenly you'll get meaningless answers. I had meant that it should produce an *error* if you even try to run such a configuration.
>
>  -Hal
>
> >
> > On Wed, 2014-05-07 at 16:12 +0100, Hal Finkel wrote:
> > > ----- Original Message -----
> > > > From: "Yi Kong" <Yi.Kong at arm.com>
> > > > To: "Anton Korobeynikov" <anton at korobeynikov.info>, "Chris
> > > > Matthews" <chris.matthews at apple.com>
> > > > Cc: "Hal Finkel" <hfinkel at anl.gov>, "LLVM Commits"
> > > > <llvm-commits at cs.uiuc.edu>, "Renato Golin"
> > > > <renato.golin at linaro.org>
> > > > Sent: Wednesday, May 7, 2014 10:02:11 AM
> > > > Subject: Re: [LNT] r207898 - Use Mann-Whitney U test to identify
> > > > changes
> > > >
> > > > I've updated the patch to perform Mann-Whitney U test using
> > > > significance
> > > > table. We still need SciPy when sample size is large(> 20).
> > >
> > > Why do you still require SciPy? That is a large package to pull in
> > > for one rarely-used function. Are there other parts of SciPy that
> > > you anticipate us using in the future? If not, then there are two
> > > options (which I think are both reasonable):
> > >
> > >  1. Limit the number of repeat samples taken to 20 (running the
> > >  test suite more than 20 times per revision seems unlikely in
> > >  practice).
> > >  2. Implement the normal approximation to the U-value calculation
> > >  in the code. From the description on the wikipedia page, the
> > >  algorithm seems pretty simple.
> > >
> > >  -Hal
> > >
> > > >
> > > > On Wed, 2014-05-07 at 09:12 +0100, Anton Korobeynikov wrote:
> > > > > > If we are using significance table, we can no longer
> > > > > > calculate p
> > > > > > values,
> > > > > > right? Is there any algorithm to calculate p value for all
> > > > > > sample
> > > > > > sizes?
> > > > > We would just need to fix the threshold (say, 0.05 or 0.1 or
> > > > > 0.01).
> > > > > Also, we can have like 2 or 3 tables for various thresholds.
> > > > > This
> > > > > should be enough for all the practical purposes.
> > > > >
> > > >
> > > >
> > > >
> > >
> >
>


-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2548782
-------------- next part --------------
A non-text attachment was scrubbed...
Name: correct_mw_dyn_import.diff
Type: text/x-patch
Size: 12631 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20140508/2f7cdbfb/attachment.bin>