[www] r176209 - Add LNT statistics project

[David Blaikie]

On Thu, Feb 28, 2013 at 3:26 AM, Renato Golin <renato.golin at linaro.org> wrote:
> On 28 February 2013 11:20, David Tweed <david.tweed at arm.com> wrote:
>>
>> Hi Renato, sounds like a useful project to have. However, since unlike
>> correctness regressions there's less time pressure, there are other
>> statistical measures with greater resolving power you could use if you
>> don't
>> need to only consider the latest sample, eg, you can use change point
>> detection tests between 5 samples and the next five samples. (Of course
>> this
>> will still only produce estimates of when there was a big perf change in
>> one
>> go, moving averages may be the way to deal with the "0.5% here, 0.5%, ...,
>> soon it adds up to real slowdowns".
>
>
> Hi David,
>
> I agree that this is just the starting point. Once we have the
> infrastructure in place we can go to great lengths to spot regressions.
>
> My main focus now is to get rid of the 3000% regression/improvement reports
> for sub-second benchmarks. Another approach would also be to make all
> sub-second benchmarks run for longer on the fastest machine, to say at least
> 10s, so that the noise would naturally become less important.

If I were doing this that's probably where I would want to head.
Essentially gut the existing test suite running infrastructure (this
needs to be done sooner or later anyway because the whole thing is
terribly crufty) & make it smart enough to choose the number of
executions dynamically to get compensate for the noise to some level
of confidence (& presumably have some cutoff threshold where the test
would be considered just too noisy to be worthwhile).

> All ideas are welcome, feel free to add things to that project on the doc.
> :D
>
> cheers,
> --renato
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>