[LLVMdev] [LNT] Question about results reliability in LNT infrustructure

Renato Golin renato.golin at linaro.org
Fri Jun 28 02:43:19 PDT 2013


On 28 June 2013 10:28, David Tweed <david.tweed at arm.com> wrote:

> (Inicidentally, responding to the earlier email below, I think you don't
> really want to compare moving averages but use some statistical test to
> quantify if the separation between the set of points within the "earlier
> window" are statistically significantly higher than the "later window"; all
> moving averages do is smear out useful information which can be useful if
> you've just got far too many data points, but otherwise it doesn't really
> help.
>

When your data is explicitly grouped, I'd agree with you. But all I can see
from my results are hardware and OS flukes in the millisecond range, with
no distinct modal signal from them. Chris said he knows of some, I haven't
looked deep enough, so I trust his judgement. What I don't want is to be
treating noise groups as signal, that's all.

I think we probably need a few different approaches, depending on the
benchmark, with moving averages being the simplest, and why I suggested we
implement it first. Sometimes, smoothing the line is all you need... ;)

cheers,
--renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130628/10ca5b66/attachment.html>


More information about the llvm-dev mailing list