[LLVMdev] Proposal: change LNT’s regression detection algorithm and how it is used to reduce false positives

[Renato Golin]

We also need to consider multiple binaries, like shared libraries that are
compiled, for instance, libc++.

Maybe a simple binary diff + ldd + binary diff on all deps would work...

Cheers,
Renato
On 21 May 2015 7:24 pm, "Chris Matthews" <chris.matthews at apple.com> wrote:

> I agree this is a great idea.  I think it needs to be fleshed out a little
> though.
>
> It would still be wise to run the regression detection algorithm, because
> the test suite changes and the machines change, and the algorithm is not
> perfect yet.  It would be a valuable source of information though.
>
> This is not a small change to how LNT works, so I think some due diligence
> is necessary.  Is clang *really* that deterministic, especially over
> successive revs?  I know it is supposed to be.  Does anyone have any data
> to show this is going to be an effective approach?  It seems like there are
> benchmarks in the test-suite which use __DATE__ and __TIME__ in them. I
> assume that will be a problem?
>
> > On May 21, 2015, at 1:43 AM, Renato Golin <renato.golin at linaro.org>
> wrote:
> >
> > On 20 May 2015 at 23:31, Sean Silva <chisophugis at gmail.com> wrote:
> >> In the last 10,000 revisions of LLVM+Clang, only 10 revisions actually
> >> caused the binary of MultiSource/Benchmarks/BitBench/five11 to change.
> So if
> >> just store a hash of the binary in the database, we should be able to
> pool
> >> all samples we have collected while the binary is the the same as it
> >> currently is, which will let us use significantly more datapoints for
> the
> >> reference.
> >
> > +1
> >
> >
> >> Also, we can trivially eliminate running the regression detection
> algorithm
> >> if the binary hasn't changed.
> >
> > +2!
> >
> > --renato
> > _______________________________________________
> > LLVM Developers mailing list
> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150521/1603eb9b/attachment.html>