[LLVMdev] Proposal: change LNT’s regression detection algorithm and how it is used to reduce false positives

[Sean Silva]

On Mon, May 18, 2015 at 11:24 AM, Mikhail Zolotukhin <mzolotukhin at apple.com>
wrote:

> Hi Chris and others!
>
> I totally support any work in this direction.
>
> In the current state LNT’s regression detection system is too noisy, which
> makes it almost impossible to use in some cases. If after each run a
> developer gets a dozen of ‘regressions’, none of which happens to be real,
> he/she won’t care about such reports after a while. We clearly need to
> filter out as much noise as we can - and as it turns out even simplest
> techniques could help here. For example, the technique I used (which you
> mentioned earlier) takes ~15 lines of code to implement and filters out
> almost all noise in our internal data-sets. It’d be really cool to have
> something more scientifically-proven though:)
>
> One thing to add from me - I think we should try to do our best in
> assumption that we don’t have enough samples. Of course, the more data we
> have - the better, but in many cases we can’t (or we don’t want) to
> increase number os samples, since it dramatically increases testing time.
>

Why not just start out with only a few samples, then collect more for
benchmarks that appear to have changed?

-- Sean Silva


> That’s not to discourage anyone from increasing number of samples, or
> adding techniques relying on a significant number of samples, but rather to
> try mining as many ‘samples’ as possible from the data we have - e.g. I
> absolutely agree with your idea to pass more than 1 previous run.
>
> Thanks,
> Michael
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150518/3ba578ef/attachment.html>