[llvm-dev] [RFC] LNT Feature: tracking performance changes

[Chris Matthews via llvm-dev]

I’d like to propose a new LNT feature that I plan to start working on immediately: formal performance change tracking.

Motivation: LNT produces a lot of data, but it is not very actionable.  Unless you are watching the nightly reports, or checking LNT regularly, performance changes can happen and be missed.  Oh no!  Once commits have been in tree for a long time, it is harder to fix them even if they cause a negative performance impact. Finding performance changes early is important.

Solution: ease the burden of finding and verifying performance changes, with a new set of interfaces to assist the user.  I’ll build a new area in LNT for tracking performance changes that are identified as significant based on our current techniques for finding performance changes.  Changes will be shown in a “triage page” for review and be grouped together into “regressions” using heuristics. The database entry for a regression will be used to track its current status and resolution. The regression can be associated with a separate bugzilla report, and provides a way of linking between the bug report and the raw live performance results. Changes in performance will also be able to be annotated in the performance graphs by referring to the regression information.

My intent is that with a small amount of user input it will be possible to find/track/verify performance changes.  Use of this feature will be optional, and probably off by default.

I plan to make 4 changes to the LNT interface: the triage page, a regression list page, a regression details page, and changes to the graph page to add regression annotations.

Under the hood, I will be making some other big changes, including, authentication and user accounts (so that only known users can edit regression details), form validation, adding a mechanism for large offline computations, and replacing the guts of the graphing system to load data dynamically.

I will be developing some heuristics for combining changes. For instance several benchmarks that all regress at the same run are probably caused by the same change. Likewise, changes of the same benchmark on different architecture or optimization levels.

Even if you won’t be using the tracking directly these features should speed up the rest of the LNT interface.