[LLVMdev] Dev Meeting BOF: Performance Tracking

[Renato Golin]

On 5 August 2014 15:41, Chad Rosier <mcrosier at codeaurora.org> wrote:
> I agree.  IIRC, there's functionality to set a baseline run to compare against.
> Unfortunately, I think this is too coarse.  It would be great if the golden
> standard could be set on a per benchmark basis.  Thus, upward trending
> benchmarks can have their standard updated while other benchmarks remain
> static.

Having multiple "golden standards" showing as a coloured line would
give the visual impression of mostly the highest score, no matter
which release that was. Programatically, it'd also allow us to enquire
about the "best golden standard" and always compare against it. I
think the historical values are important to show a graph of the
progress of releases, as well as the current revision, so you know how
that fluctuated in the past few years as well as in the past few
weeks.


> Would it be useful to detect upwards trends as well?  Per my comment above,
> it would be great to update the golden standard so we're always moving in the
> right direction.

Upwards trends are nice to know, but the "current standard" can be the
highest average of a set of N points since the last golden standard,
and for that we don't explicitly need to be tracking upwards trends.
If the last moving average is NOT the current standard, than we must
have had detected a downwards slope since then.


> Could we create a minimal test-suite that includes only benchmarks that
> are known to have little variance and run times greater than some decided
> upon threshold?  With that in place we could begin the performance
> tracking (and hopefully adoption) sooner.

That's done. I haven't tested yet because of the failures in Perf.

In the beginning, we could start with the set of
gloden/current/previous standards for the benchmark-specific results,
not the whole test-suite. As we progress towards more stability, we
can implement that for all, but still allow configurations to only
warn (user/admin) of the restricted set, to avoid extra noise on noisy
targets (like ARM).

cheers,
--renato