[LNT] r207898 - Use Mann-Whitney U test to identify changes

[Yi Kong]

That's odd, I can cleanly apply the diff. Git format patch is
attached, hopefully
that will work.

Cheers,
Yi Kong

On Thu, 2014-05-08 at 19:09 +0100, Chris Matthews wrote:
> I get a “malformed patch” warning with patch and git.   Can you resend
> the patch?
>
> On May 8, 2014, at 2:01 AM, Yi Kong <Yi.Kong at arm.com> wrote:
>
> > Updated again. If SciPy is installed, it will use normal
> > approximation
> > to compute MWU. Otherwise it will show an error page in large
> > problem
> > sizes.
> >
> > On Wed, 2014-05-07 at 17:16 +0100, Hal Finkel wrote:
> > > ----- Original Message -----
> > > > From: "Yi Kong" <Yi.Kong at arm.com>
> > > > To: "Hal Finkel" <hfinkel at anl.gov>
> > > > Cc: "LLVM Commits" <llvm-commits at cs.uiuc.edu>, "Renato Golin"
> > > > <renato.golin at linaro.org>, "Anton Korobeynikov"
> > > > <anton at korobeynikov.info>, "Chris Matthews"
> > > > <chris.matthews at apple.com>
> > > > Sent: Wednesday, May 7, 2014 11:06:36 AM
> > > > Subject: Re: [LNT] r207898 - Use Mann-Whitney U test to identify
> > > > changes
> > > >
> > > > Updated. If the sample size if greater than 20, Mann-Whitney U
> > > > test
> > > > won't be performed.
> > >
> > > Sorry, I was not clear...
> > >
> > > +        # Use Mann-Whitney U test to test null hypothesis that
> > > result is
> > >
> > > +        # unchanged.
> > >
> > > +        if len(self.samples) >= 4 and len(self.samples) <= 20 and
> > > \
> > >
> > > +        len(self.prev_samples) >= 4 and len(self.prev_samples) <=
> > > 20:
> > >
> > > +            same = stats.mannwhitneyu(self.samples,
> > > self.prev_samples, self.confidence_lv)
> > >
> > > +            if same:
> > >
> > > +                return UNCHANGED_PASS
> > >
> > > This is going to be very confusing; in theory, this means that
> > > increasing the number of samples will give you better results
> > > until you hit 20, and then suddenly you'll get meaningless
> > > answers. I had meant that it should produce an *error* if you even
> > > try to run such a configuration.
> > >
> > > -Hal
> > >
> > > >
> > > > On Wed, 2014-05-07 at 16:12 +0100, Hal Finkel wrote:
> > > > > ----- Original Message -----
> > > > > > From: "Yi Kong" <Yi.Kong at arm.com>
> > > > > > To: "Anton Korobeynikov" <anton at korobeynikov.info>, "Chris
> > > > > > Matthews" <chris.matthews at apple.com>
> > > > > > Cc: "Hal Finkel" <hfinkel at anl.gov>, "LLVM Commits"
> > > > > > <llvm-commits at cs.uiuc.edu>, "Renato Golin"
> > > > > > <renato.golin at linaro.org>
> > > > > > Sent: Wednesday, May 7, 2014 10:02:11 AM
> > > > > > Subject: Re: [LNT] r207898 - Use Mann-Whitney U test to
> > > > > > identify
> > > > > > changes
> > > > > >
> > > > > > I've updated the patch to perform Mann-Whitney U test using
> > > > > > significance
> > > > > > table. We still need SciPy when sample size is large(> 20).
> > > > >
> > > > > Why do you still require SciPy? That is a large package to
> > > > > pull in
> > > > > for one rarely-used function. Are there other parts of SciPy
> > > > > that
> > > > > you anticipate us using in the future? If not, then there are
> > > > > two
> > > > > options (which I think are both reasonable):
> > > > >
> > > > > 1. Limit the number of repeat samples taken to 20 (running the
> > > > > test suite more than 20 times per revision seems unlikely in
> > > > > practice).
> > > > > 2. Implement the normal approximation to the U-value
> > > > > calculation
> > > > > in the code. From the description on the wikipedia page, the
> > > > > algorithm seems pretty simple.
> > > > >
> > > > > -Hal
> > > > >
> > > > > >
> > > > > > On Wed, 2014-05-07 at 09:12 +0100, Anton Korobeynikov wrote:
> > > > > > > > If we are using significance table, we can no longer
> > > > > > > > calculate p
> > > > > > > > values,
> > > > > > > > right? Is there any algorithm to calculate p value for
> > > > > > > > all
> > > > > > > > sample
> > > > > > > > sizes?
> > > > > > > We would just need to fix the threshold (say, 0.05 or 0.1
> > > > > > > or
> > > > > > > 0.01).
> > > > > > > Also, we can have like 2 or 3 tables for various
> > > > > > > thresholds.
> > > > > > > This
> > > > > > > should be enough for all the practical purposes.
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Use-Mann-Whitney-U-test-to-identify-changes-2.patch
Type: text/x-patch
Size: 13239 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20140508/f77efa17/attachment.bin>