[LNT] r207898 - Use Mann-Whitney U test to identify changes

[Yi Kong]

Updated. If the sample size if greater than 20, Mann-Whitney U test
won't be performed.

On Wed, 2014-05-07 at 16:12 +0100, Hal Finkel wrote:
> ----- Original Message -----
> > From: "Yi Kong" <Yi.Kong at arm.com>
> > To: "Anton Korobeynikov" <anton at korobeynikov.info>, "Chris Matthews" <chris.matthews at apple.com>
> > Cc: "Hal Finkel" <hfinkel at anl.gov>, "LLVM Commits" <llvm-commits at cs.uiuc.edu>, "Renato Golin"
> > <renato.golin at linaro.org>
> > Sent: Wednesday, May 7, 2014 10:02:11 AM
> > Subject: Re: [LNT] r207898 - Use Mann-Whitney U test to identify changes
> > 
> > I've updated the patch to perform Mann-Whitney U test using
> > significance
> > table. We still need SciPy when sample size is large(> 20).
> 
> Why do you still require SciPy? That is a large package to pull in for one rarely-used function. Are there other parts of SciPy that you anticipate us using in the future? If not, then there are two options (which I think are both reasonable):
> 
>  1. Limit the number of repeat samples taken to 20 (running the test suite more than 20 times per revision seems unlikely in practice).
>  2. Implement the normal approximation to the U-value calculation in the code. From the description on the wikipedia page, the algorithm seems pretty simple.
> 
>  -Hal
> 
> > 
> > On Wed, 2014-05-07 at 09:12 +0100, Anton Korobeynikov wrote:
> > > > If we are using significance table, we can no longer calculate p
> > > > values,
> > > > right? Is there any algorithm to calculate p value for all sample
> > > > sizes?
> > > We would just need to fix the threshold (say, 0.05 or 0.1 or 0.01).
> > > Also, we can have like 2 or 3 tables for various thresholds. This
> > > should be enough for all the practical purposes.
> > > 
> > 
> > 
> > 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: correct_mw_20.diff
Type: text/x-patch
Size: 11444 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20140507/e30a08d3/attachment.bin>