[LLVMdev] Proposal: change LNT’s regression detection algorithm and how it is used to reduce false positives

[Sean Silva]

On Wed, May 27, 2015 at 4:09 PM, Chris Matthews <chris.matthews at apple.com>
wrote:

> Lets try this on the whole test suite?
>

I started this as a drill-down on a single benchmark, so I've just written
a little bit of Python for the build logic, and grown it to a little list
of hardcoded benchmarks.

Is there a way to programmatically build and access all the binaries in the
test suite?

-- Sean Silva


>
>
>
> On May 26, 2015, at 7:05 PM, Sean Silva <chisophugis at gmail.com> wrote:
>
> Update: in that same block of 10,000 LLVM/Clang revisions, this the number
> of distinct SHA1 hashes for the binaries of the following benchmarks:
>
> 7 MultiSource/Applications/aha/aha
> 2 MultiSource/Benchmarks/BitBench/drop3/drop3
> 10 MultiSource/Benchmarks/BitBench/five11/five11
> 7 MultiSource/Benchmarks/BitBench/uudecode/uudecode
> 3 MultiSource/Benchmarks/BitBench/uuencode/uuencode
> 5 MultiSource/Benchmarks/Trimaran/enc-rc4/rc4
> 11 SingleSource/Benchmarks/BenchmarkGame/n-body
> 2 SingleSource/Benchmarks/Shootout/ackermann
>
> Let me know if there are any specific benchmarks you would like me to test.
>
> -- Sean Silva
>
>
> On Wed, May 20, 2015 at 3:31 PM, Sean Silva <chisophugis at gmail.com> wrote:
>
>> I found an interesting datapoint:
>>
>> In the last 10,000 revisions of LLVM+Clang, only 10 revisions actually
>> caused the binary of MultiSource/Benchmarks/BitBench/five11 to change. So
>> if just store a hash of the binary in the database, we should be able to
>> pool all samples we have collected while the binary is the the same as it
>> currently is, which will let us use significantly more datapoints for the
>> reference.
>>
>> Also, we can trivially eliminate running the regression detection
>> algorithm if the binary hasn't changed.
>>
>> -- Sean Silva
>>
>> On Mon, May 18, 2015 at 9:02 PM, Chris Matthews <chris.matthews at apple.com
>> > wrote:
>>
>>> The reruns flag already does that.  It helps a bit, but only as long as
>>> the the benchmark is flagged as regressed.
>>>
>>>
>>> On May 18, 2015, at 8:28 PM, Sean Silva <chisophugis at gmail.com> wrote:
>>>
>>>
>>>
>>> On Mon, May 18, 2015 at 11:24 AM, Mikhail Zolotukhin <
>>> mzolotukhin at apple.com> wrote:
>>>
>>>> Hi Chris and others!
>>>>
>>>> I totally support any work in this direction.
>>>>
>>>> In the current state LNT’s regression detection system is too noisy,
>>>> which makes it almost impossible to use in some cases. If after each run a
>>>> developer gets a dozen of ‘regressions’, none of which happens to be real,
>>>> he/she won’t care about such reports after a while. We clearly need to
>>>> filter out as much noise as we can - and as it turns out even simplest
>>>> techniques could help here. For example, the technique I used (which you
>>>> mentioned earlier) takes ~15 lines of code to implement and filters out
>>>> almost all noise in our internal data-sets. It’d be really cool to have
>>>> something more scientifically-proven though:)
>>>>
>>>> One thing to add from me - I think we should try to do our best in
>>>> assumption that we don’t have enough samples. Of course, the more data we
>>>> have - the better, but in many cases we can’t (or we don’t want) to
>>>> increase number os samples, since it dramatically increases testing time.
>>>>
>>>
>>> Why not just start out with only a few samples, then collect more for
>>> benchmarks that appear to have changed?
>>>
>>> -- Sean Silva
>>>
>>>
>>>> That’s not to discourage anyone from increasing number of samples, or
>>>> adding techniques relying on a significant number of samples, but rather to
>>>> try mining as many ‘samples’ as possible from the data we have - e.g. I
>>>> absolutely agree with your idea to pass more than 1 previous run.
>>>>
>>>> Thanks,
>>>> Michael
>>>>
>>>> _______________________________________________
>>> LLVM Developers mailing list
>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>
>>>
>>>
>>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150527/38092ac9/attachment.html>