[LLVMdev] Proposal: change LNT’s regression detection algorithm and how it is used to reduce false positives

[Chris Matthews]

I agree this is a great idea.  I think it needs to be fleshed out a little though.

It would still be wise to run the regression detection algorithm, because the test suite changes and the machines change, and the algorithm is not perfect yet.  It would be a valuable source of information though.

This is not a small change to how LNT works, so I think some due diligence is necessary.  Is clang *really* that deterministic, especially over successive revs?  I know it is supposed to be.  Does anyone have any data to show this is going to be an effective approach?  It seems like there are benchmarks in the test-suite which use __DATE__ and __TIME__ in them. I assume that will be a problem?

> On May 21, 2015, at 1:43 AM, Renato Golin <renato.golin at linaro.org> wrote:
> 
> On 20 May 2015 at 23:31, Sean Silva <chisophugis at gmail.com> wrote:
>> In the last 10,000 revisions of LLVM+Clang, only 10 revisions actually
>> caused the binary of MultiSource/Benchmarks/BitBench/five11 to change. So if
>> just store a hash of the binary in the database, we should be able to pool
>> all samples we have collected while the binary is the the same as it
>> currently is, which will let us use significantly more datapoints for the
>> reference.
> 
> +1
> 
> 
>> Also, we can trivially eliminate running the regression detection algorithm
>> if the binary hasn't changed.
> 
> +2!
> 
> --renato
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev