[PATCH] [LNT] Aggregate most recent runs

[Yi Kong]

In theory, if you set the window size to N, it's at least as good as
running N iterations. However this makes bisecting the source of
problem more convenient, as we can move the window in a much smaller
step. I'm really interested in is after how many runs, the change of
samples will be reliably significant. If it's less than N, it should
let us to discover regressions sooner. Therefore running less
iterations and more frequently will produce better results.

On 9 May 2014 21:57, Chris Matthews <chris.matthews at apple.com> wrote:
> Yes, I’d like to test this on a few of our data sets.  I just want to make sure we are not going to miss regressions.
>
> On May 9, 2014, at 1:05 PM, Renato Golin <renato.golin at linaro.org> wrote:
>
>> Hi Yi,
>>
>> I like the idea, but Anton and Chris had some reservations. I'd very
>> much like to see this going in, even if just optional and not even
>> default, so that I can turn my test-suite bot back on reporting
>> performances. I'm not very good at Python, so others will have to
>> review your code. :)
>>
>> Thanks!
>> --renato
>>
>> On 8 May 2014 11:17, Yi Kong <Yi.Kong at arm.com> wrote:
>>> Hi all,
>>>
>>> After previously discussion, I've implemented aggregation across most
>>> recent runs. This method significantly reduces noise when applied in
>>> combination of Mann-Whitney U test. It replaces standard deviation
>>> estimation feature, as it clearly performs better.
>>>
>>> On a Cortex-A9 board running one iteration per run, regressions can be
>>> easily spotted when the sliding window is greater than 8 runs and
>>> aggregated by median, however changes are usually only significant
>>> enough after 6 consecutive runs. On x86 machines with 10 iterations, the
>>> improvement is not very obvious.
>>>
>>> Cheers,
>>> Yi Kong
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits