[all-commits] [llvm/llvm-project] dcf9ad: [Github][RFC] Add workflow to diff codegen on llvm...

[Luke Lau via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: dcf9adde9375a9843d90cf8df6d2e0a0e498b0ed
      https://github.com/llvm/llvm-project/commit/dcf9adde9375a9843d90cf8df6d2e0a0e498b0ed
  Author: Luke Lau <luke at igalia.com>
  Date:   2026-06-10 (Wed, 10 Jun 2026)

  Changed paths:
    M .github/workflows/issue-write.yml
    A .github/workflows/test-suite.yml
    A .github/workflows/test-suite/aarch64.cmake
    A .github/workflows/test-suite/configure-and-build.sh
    A .github/workflows/test-suite/llvm.cmake
    A .github/workflows/test-suite/riscv64.cmake
    A .github/workflows/test-suite/x86_64.cmake

  Log Message:
  -----------
  [Github][RFC] Add workflow to diff codegen on llvm-test-suite (#190010)

A common task when reviewing PRs in the LLVM subproject is checking out
the PR locally, building it, running it on some benchmarks e.g.
llvm-test-suite, and comparing the codegen against some known version.

The process is fairly laborious so this PR adds a GitHub workflow to
automate it. It's triggered by commenting "/test-suite" on a PR. The
workflow will kick off, build clang with the head and base of the PR,
build the benchmarks in llvm-test-suite for several configurations with
each version of clang, compute the diff in the output assembly via the
[tdiff.py
script](https://github.com/llvm/llvm-test-suite/blob/main/utils/tdiff.py),
and then report back with the diffs in a comment.

Here's an example on my fork where you can see the diff of a codegen
change in the RISC-V backend:
https://github.com/lukel97/llvm-project/pull/7#issuecomment-4283315532

At the moment it's very simple but could be fleshed out later. Currently
it builds llvm-test-suite for handful of common configurations, some
cross-compiled:

- `-target aarch64-linux-gnu -march=armv9-a -O3`
- `-target riscv64-linux-gnu -march=rva23u64 -O3`
- `-target x86_64-linux-gnu -O3`

We could eventually extend this to accept arbitrary targets and flags in
the comment. It would also be nice to support LTO diffs in future but we
will need to add some extra support in llvm-test-suite to extract the
asm during the link step.

It also just comments a link to download the codegen diffs, but could
eventually also include some output from the ./utils/compare.py script
about e.g. changes in code size or statistics. For now, those results
are just uploaded as an artifact.

In terms of worker resources, running it on the free GitHub hosted
workers is good enough. Building Clang takes a while, over an hour, but
building the test-suite only takes around 10 minutes. But we could stick
it on something beefier if we wanted the feedback to be faster.

This workflow requires the PR to be mergeable, as it wants to get the
diff of the "mergeability" commit that GitHub generates for each PR.
That way the diff is always between the latest version of the base
branch and the PR, not the base branch at the time the PR was created.



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications