[LLVMdev] [LNT] Question about results reliability in LNT infrustructure

[James Courtier-Dutton]

On Jun 30, 2013 8:12 PM, "Anton Korobeynikov" <anton at korobeynikov.info>
wrote:
>
> > Getting 10 samples at different commits will give you similar accuracy
if
> > behaviour doesn't change, and you can rely on 10-point blocks before
and > after each change to have the same result.
> Right. But this way you will have 10-commits delay. So, you will need
> 3-4 additional test runs to pinpoint the offending commit in the worst
> case.
>
> > This is why I proposed something like moving averages.
> Moving average will "smooth" the result. So, only really big changes
> will be caught by it.
>

Like any result in statistics, the result should be quoted together with a
+/- figure derived from the statistical method used. Generally, low sample
size means high +/-.

Another option is to take a deterministic approach to measurement. The code
should executive the same cpu  instructions every time it is run, so some
method to measure just these instructions should be attempted. Maybe
processing qemu logs when llvm is run inside qemu might give a possible
solution?

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130701/f9971a44/attachment.html>