[PATCH] [LNT] Aggregate most recent runs

[Chris Matthews]

Yes, I’d like to test this on a few of our data sets.  I just want to make sure we are not going to miss regressions.

On May 9, 2014, at 1:05 PM, Renato Golin <renato.golin at linaro.org> wrote:

> Hi Yi,
> 
> I like the idea, but Anton and Chris had some reservations. I'd very
> much like to see this going in, even if just optional and not even
> default, so that I can turn my test-suite bot back on reporting
> performances. I'm not very good at Python, so others will have to
> review your code. :)
> 
> Thanks!
> --renato
> 
> On 8 May 2014 11:17, Yi Kong <Yi.Kong at arm.com> wrote:
>> Hi all,
>> 
>> After previously discussion, I've implemented aggregation across most
>> recent runs. This method significantly reduces noise when applied in
>> combination of Mann-Whitney U test. It replaces standard deviation
>> estimation feature, as it clearly performs better.
>> 
>> On a Cortex-A9 board running one iteration per run, regressions can be
>> easily spotted when the sliding window is greater than 8 runs and
>> aggregated by median, however changes are usually only significant
>> enough after 6 consecutive runs. On x86 machines with 10 iterations, the
>> improvement is not very obvious.
>> 
>> Cheers,
>> Yi Kong