[llvm-dev] [RFC] LNT Feature: tracking performance changes

[Kristof Beyls via llvm-dev]

Sounds like a great plan :)

> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
> Chris Matthews via llvm-dev
> Sent: 22 September 2015 00:16
> To: llvm-dev
> Subject: [llvm-dev] [RFC] LNT Feature: tracking performance changes
> 
> The regression can be associated with a separate bugzilla
> report, and provides a way of linking between the bug report and the raw
> live performance results. Changes in performance will also be able to be
> annotated in the performance graphs by referring to the regression
> information.

I'm assuming that the implementation will also allow to link to other bug tracking systems, not just bugzilla? That'd be great as we're also using LNT internally to track performance and we're using a different bug tracker internally.

> My intent is that with a small amount of user input it will be possible
> to find/track/verify performance changes.  Use of this feature will be
> optional, and probably off by default.

I don't see yet why this needs to be off by default, but I'm assuming I'll understand once a few more details are described.

> I plan to make 4 changes to the LNT interface: the triage page, a
> regression list page, a regression details page, and changes to the
> graph page to add regression annotations.

Our use of LNT to track performance so far has been to manually inspect the daily report page - making sure that every single day has been inspected to try and not miss performance deltas. This has been error-prone indeed.
After this lands, should we aim to write up some "best practice to do performance tracking with LNT"-documentation?
 
> Under the hood, I will be making some other big changes, including,
> authentication and user accounts (so that only known users can edit
> regression details), form validation, adding a mechanism for large
> offline computations, and replacing the guts of the graphing system to
> load data dynamically.

Really nice - hopefully this will also alleviate some of the stability and performance issues we've seen on llvm.org/perf.

> I will be developing some heuristics for combining changes. For instance
> several benchmarks that all regress at the same run are probably caused
> by the same change. Likewise, changes of the same benchmark on different
> architecture or optimization levels.
> 
> Even if you won’t be using the tracking directly these features should
> speed up the rest of the LNT interface.
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev