[PATCH] [LNT] Aggregate most recent runs
chris.matthews at apple.com
Fri May 9 13:57:57 PDT 2014
Yes, I’d like to test this on a few of our data sets. I just want to make sure we are not going to miss regressions.
On May 9, 2014, at 1:05 PM, Renato Golin <renato.golin at linaro.org> wrote:
> Hi Yi,
> I like the idea, but Anton and Chris had some reservations. I'd very
> much like to see this going in, even if just optional and not even
> default, so that I can turn my test-suite bot back on reporting
> performances. I'm not very good at Python, so others will have to
> review your code. :)
> On 8 May 2014 11:17, Yi Kong <Yi.Kong at arm.com> wrote:
>> Hi all,
>> After previously discussion, I've implemented aggregation across most
>> recent runs. This method significantly reduces noise when applied in
>> combination of Mann-Whitney U test. It replaces standard deviation
>> estimation feature, as it clearly performs better.
>> On a Cortex-A9 board running one iteration per run, regressions can be
>> easily spotted when the sliding window is greater than 8 runs and
>> aggregated by median, however changes are usually only significant
>> enough after 6 consecutive runs. On x86 machines with 10 iterations, the
>> improvement is not very obvious.
>> Yi Kong
More information about the llvm-commits