[llvm-dev] LLVM compile-time regression tracking?

[Florian Hahn via llvm-dev]

Hi,

> On 9 Feb 2020, at 03:28, Mehdi AMINI via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> Hi,
> 
> 
> On Sat, Feb 8, 2020 at 5:17 AM Nikita Popov via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> Hi,
> 
> Does the LLVM project perform any kind of tracking for commit-by-commit compile-time changes? It looks like LNT only tracks run-time performance (and to be honest I wasn't able to make heads or tails of the results even for that -- the interface was pretty unintuitive to me.)
> 
> At some point Apple was feeding compile-time data to LNT, but it seems like the jobs aren't active anymore: http://green.lab.llvm.org/green/view/Compile%20Time/ <http://green.lab.llvm.org/green/view/Compile%20Time/>
Yes, it looks like the matrix view is broken. One of the relevant jobs is http://104.154.54.203/db_default/v4/nts/machine/1353 <http://104.154.54.203/db_default/v4/nts/machine/1353> which has a fairly recent submission http://104.154.54.203/db_default/v4/nts/131539 <http://104.154.54.203/db_default/v4/nts/131539> 

But it looks like we need to adjust the ‘Run order’. It is currently using git commit hashes, which messes up the ordering of submissions. And submissions by the bot seem to be very irregular, due to various failures.



> While it is "normal" that each new LLVM release regresses compile-time by 2-3%, LLVM 10 seems to be going for a new record here, with optimized builds regressing by around 5-15%, and that's already after mitigating the largest regression: https://perf.rust-lang.org/compare.html?start=58b834344fc7b9185e7a50db1ff24e5eb07dae5e&end=f9c17b2d9bd4ca61bb0eb8b226dd61aaf254156c <https://perf.rust-lang.org/compare.html?start=58b834344fc7b9185e7a50db1ff24e5eb07dae5e&end=f9c17b2d9bd4ca61bb0eb8b226dd61aaf254156c>
> 
> It does not seem like these regressions are specific to Rust either, see for example https://bugs.llvm.org/show_bug.cgi?id=44408 <https://bugs.llvm.org/show_bug.cgi?id=44408>.
> 
> The Rust project tracks compile-time results for each commit (see link above), and thanks to that can quickly spot and revert changes that negatively affect performance. I think it would be really great if something similar would exist for LLVM...
> 

The Rust tracking looks awesome! It’s great that we get feedback from there, thanks for keeping an eye on that.

In theory, the bots on green dragon should submit CTMark numbers regularly for CTMark on a few configurations (although not for every commit), but as mentioned earlier there are a few issues that crept in. I’ll try to reach out to the right people to have a look.

Cheers,
Florian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200213/50dd696a/attachment-0001.html>