[llvm-dev] [RFC] LNT Feature: tracking performance changes

[Hal Finkel via llvm-dev]

----- Original Message -----
> From: "Chris Matthews via llvm-dev" <llvm-dev at lists.llvm.org>
> To: "llvm-dev" <llvm-dev at lists.llvm.org>
> Sent: Monday, September 21, 2015 6:15:44 PM
> Subject: [llvm-dev] [RFC] LNT Feature: tracking performance changes
> 
> I’d like to propose a new LNT feature that I plan to start working on
> immediately: formal performance change tracking.
> 
> Motivation: LNT produces a lot of data, but it is not very
> actionable.  Unless you are watching the nightly reports, or
> checking LNT regularly, performance changes can happen and be
> missed.  Oh no!  Once commits have been in tree for a long time, it
> is harder to fix them even if they cause a negative performance
> impact. Finding performance changes early is important.
> 
> Solution: ease the burden of finding and verifying performance
> changes, with a new set of interfaces to assist the user.  I’ll
> build a new area in LNT for tracking performance changes that are
> identified as significant based on our current techniques for
> finding performance changes.  Changes will be shown in a “triage
> page” for review and be grouped together into “regressions” using
> heuristics. The database entry for a regression will be used to
> track its current status and resolution. The regression can be
> associated with a separate bugzilla report, and provides a way of
> linking between the bug report and the raw live performance results.
> Changes in performance will also be able to be annotated in the
> performance graphs by referring to the regression information.
> 
> My intent is that with a small amount of user input it will be
> possible to find/track/verify performance changes.  Use of this
> feature will be optional, and probably off by default.
> 
> I plan to make 4 changes to the LNT interface: the triage page, a
> regression list page, a regression details page, and changes to the
> graph page to add regression annotations.
> 
> Under the hood, I will be making some other big changes, including,
> authentication and user accounts (so that only known users can edit
> regression details), form validation, adding a mechanism for large
> offline computations, and replacing the guts of the graphing system
> to load data dynamically.
> 
> I will be developing some heuristics for combining changes. For
> instance several benchmarks that all regress at the same run are
> probably caused by the same change. Likewise, changes of the same
> benchmark on different architecture or optimization levels.
> 
> Even if you won’t be using the tracking directly these features
> should speed up the rest of the LNT interface.

This all sounds really nice, and linking the performance regressions to specific bug reports sounds useful. However, I think that unless we're able to e-mail commit authors who cause regressions like we do for buildbot failures, as well as others who are architecture performance experts on the relevant systems, this won't see the use it needs.

 -Hal

> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory