[LLVMdev] [LNT] Question about results reliability in LNT infrustructure

[Renato Golin]

On 30 June 2013 20:08, Anton Korobeynikov <anton at korobeynikov.info> wrote:

> > Getting 10 samples at different commits will give you similar accuracy if
> > behaviour doesn't change, and you can rely on 10-point blocks before and
> > after each change to have the same result.
> Right. But this way you will have 10-commits delay. So, you will need
> 3-4 additional test runs to pinpoint the offending commit in the worst
> case.
>

Well, 10 was an example, but yes, you'll always have N commit-groups delay.
My assumption is that some (say 5) commit-groups delay is not a critical
issue if it happens once in a while, as opposed to having to examine every
hike on a range of several dozen commits.



> This is why I proposed something like moving averages.
> Moving average will "smooth" the result. So, only really big changes
> will be caught by it.
>

Absolutely. Smoothing is bad, but it's better than what we have, and at
least it would catch big regressions. Today, not even the big ones are
being caught.

You don't have to throw away the original data-points, you just run a
moving average to pinpoint big changes, where the confidence that
regression occurred is high. In parallel, you can still use the same
data-points to do more refined analysis, and even cross-reference multiple
analysis' data to give you even more confidence.

Anton and David, I could not agree with you more on what's necessary to
have a good analysis, I just wished we had something cruder but sooner
while we develop the perfect statistical model. I believe Chris is doing
that now. So, whatever is wrong with his analysis, let's just wait and see
how it turns out, and how we can improve further. For now, anything will be
an improvement.

cheers,
--renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130701/e5ce354c/attachment.html>