[LLVMdev] [LNT] Question about results reliability in LNT infrustructure

[Renato Golin]

On 27 June 2013 17:05, Tobias Grosser <tobias at grosser.es> wrote:

> We are looking for a good way/value to show the reliability of individual
> results in the UI. Do you have some experience, what a good measure of the
> reliability of test results is?
>

Hi Tobi,

I had a look at this a while ago, but never got around to actually work on
it. My idea was to never use point-changes as indication of
progress/regressions, unless there was a significant change (2/3 sigma).
What we should do is to compare the current moving-average with the past
moving averages (of K runs) with both last-avg and the (N-K)th
moving-average (to make sure previous values included in the current moving
average are not toning it down/up), and keep the biggest difference as the
final result.

We should also compare the current mov-avg with M non-overlapping mov-avgs
before, and calculate if we're monotonically increasing, decreasing or if
there is a difference of 2/3 sigma between the current mov-avg (N) and the
(N-M)th mov-avg. That would give us an idea on the trends of each test.

cheers,
--renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130627/34b38916/attachment.html>