[PATCH] [LNT] Aggregate most recent runs

[Renato Golin]

Hi Yi,

I like the idea, but Anton and Chris had some reservations. I'd very
much like to see this going in, even if just optional and not even
default, so that I can turn my test-suite bot back on reporting
performances. I'm not very good at Python, so others will have to
review your code. :)

Thanks!
--renato

On 8 May 2014 11:17, Yi Kong <Yi.Kong at arm.com> wrote:
> Hi all,
>
> After previously discussion, I've implemented aggregation across most
> recent runs. This method significantly reduces noise when applied in
> combination of Mann-Whitney U test. It replaces standard deviation
> estimation feature, as it clearly performs better.
>
> On a Cortex-A9 board running one iteration per run, regressions can be
> easily spotted when the sliding window is greater than 8 runs and
> aggregated by median, however changes are usually only significant
> enough after 6 consecutive runs. On x86 machines with 10 iterations, the
> improvement is not very obvious.
>
> Cheers,
> Yi Kong