[llvm] [Github][RFC] Add workflow to diff codegen on llvm-test-suite (PR #190010)
Luke Lau via llvm-commits
llvm-commits at lists.llvm.org
Wed Apr 1 20:43:28 PDT 2026
https://github.com/lukel97 updated https://github.com/llvm/llvm-project/pull/190010
>From f208b306fd80f2b0b3240741b44ab639d923b785 Mon Sep 17 00:00:00 2001
From: Luke Lau <luke at igalia.com>
Date: Thu, 2 Apr 2026 00:13:05 +0800
Subject: [PATCH 1/4] [Github][RFC] Add workflow to diff codegen on
llvm-test-suite
A common task when reviewing PRs in the LLVM subproject is checking out the PR locally, building it, running it on some benchmarks e.g. llvm-test-suite, and comparing the codegen against some known version.
This is quite laborious though so this PR adds a GitHub workflow to automate the process. It's triggered by commenting "/test-suite" on a PR. The workflow will kick off, build clang with the head and base of the PR, build the benchmarks in llvm-test-suite for several configurations with each version of clang, compute the diff in the output assembly via the tdiff.py script, and then report back with the diffs in a comment.
Here's an example where you can see the diff of a codegen change in the RISC-V backend on my fork: https://github.com/lukel97/llvm-project/pull/5#issuecomment-4171850051
At the moment it's very simple but could be fleshed out later. Currently it builds llvm-test-suite for handful of common configurations, some cross-compiled:
- `-target aarch64-linux-gnu -march=armv9-a -O3`
- `-target riscv64-linux-gnu -march=rva23u64 -O3`
- `-target x86_64-linux-gnu -O3`
We could eventually extend this to accept arbitrary targets and flags in the comment.
It also just comments a link to download the codegen diffs, but could eventually also include some output from the ./utils/compare.py script about e.g. changes in code size or statistics. For now, those results are just uploaded as an artifact.
In terms of worker resources, running it on the free GitHub hosted workers is good enough. Building Clang takes a while, over an hour, but building the test-suite only takes around 10 minutes. But we could stick it on something beefier if we wanted the feedback to be faster.
This workflow requires the PR to be mergeable, as it wants to get the diff of the "mergeability" commit that GitHub generates for each PR. That way the diff is always between the latest version of the base branch and the PR, not the base branch at the time the PR was created.
---
.github/workflows/test-suite.ll | 148 ++++++++++++++++++++++++++++++++
1 file changed, 148 insertions(+)
create mode 100644 .github/workflows/test-suite.ll
diff --git a/.github/workflows/test-suite.ll b/.github/workflows/test-suite.ll
new file mode 100644
index 0000000000000..0d9c81fc74083
--- /dev/null
+++ b/.github/workflows/test-suite.ll
@@ -0,0 +1,148 @@
+# When /test-suite is commented on a PR, checks out the PR, builds clang and
+# then the test-suite in several configurations. It then checks out the base of
+# the PR, builds clang and the test-suite again, and then uploads the diff of
+# the codegen.
+
+name: Diff test-suite codegen
+
+on:
+ issue_comment:
+ types:
+ - created
+
+jobs:
+ test-suite:
+ name: Build and diff
+ runs-on: ubuntu-24.04
+ permissions:
+ issues: write
+ if: >-
+ !startswith(github.event.comment.body, '<!--IGNORE-->') &&
+ github.event.issue.pull_request && contains(github.event.comment.body, '/test-suite')
+ steps:
+ - id: get-pr
+ uses: actions/github-script at ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
+ with:
+ script: |
+ const { data: pr } = await github.rest.pulls.get({
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ pull_number: context.payload.issue.number
+ })
+ if (!pr.mergeable)
+ await github.rest.issues.createComment({
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ body: "Can't diff PR, PR isn't mergeable"
+ })
+ return pr
+ - if: ${{ !fromJSON(steps.get-pr.outputs.result).mergeable }}
+ run: exit 1
+ - uses: actions/github-script at ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
+ with:
+ script: |
+ github.rest.reactions.createForIssueComment({
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ comment_id: context.payload.comment.id,
+ content: '+1'
+ })
+ - uses: actions/checkout at 8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
+ with:
+ ref: ${{ fromJSON(steps.get-pr.outputs.result).merge_commit_sha }}
+ repository: ${{ fromJSON(steps.get-pr.outputs.result).head.repo.full_name }}
+ fetch-depth: 2
+ - run: |
+ echo "HEAD_SHA=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ echo "BASE_SHA=$(git rev-parse HEAD^)" >> $GITHUB_ENV
+ - uses: actions/checkout at 8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
+ with:
+ repository: llvm/llvm-test-suite
+ path: llvm-test-suite
+ - name: Install system dependencies
+ run: |
+ sudo apt-get update
+ sudo apt-get install -y cmake ninja-build libc6-dev-{arm64,riscv64}-cross libgcc-14-dev-{arm64,riscv64}-cross libstdc++-14-dev-{arm64,riscv64}-cross
+ - name: Configure Clang
+ run: cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD='AArch64;X86;RISCV' -DLLVM_ENABLE_PROJECTS='clang;lld' -DLLVM_APPEND_VC_REV=OFF llvm -GNinja
+ - name: Build Clang @ head
+ run: ninja -C build
+ - name: Configure and build test-suite @ head
+ run: |
+ cat << EOF > rva23u64.cmake
+ set(CMAKE_SYSTEM_NAME Linux)
+ set(CMAKE_C_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang)
+ set(CMAKE_CXX_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang++)
+ set(CMAKE_C_COMPILER_TARGET riscv64-linux-gnu)
+ set(CMAKE_CXX_COMPILER_TARGET riscv64-linux-gnu)
+ set(CMAKE_C_FLAGS_INIT "-march=rva23u64 -save-temps=obj")
+ set(CMAKE_CXX_FLAGS_INIT "-march=rva23u64 -save-temps=obj")
+ set(CMAKE_SYSTEM_PROCESSOR riscv64)
+ set(CMAKE_LINKER_TYPE LLD)
+ EOF
+ cat << EOF > armv9-a.cmake
+ set(CMAKE_SYSTEM_NAME Linux)
+ set(CMAKE_C_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang)
+ set(CMAKE_CXX_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang++)
+ set(CMAKE_C_COMPILER_TARGET aarch64-linux-gnu)
+ set(CMAKE_CXX_COMPILER_TARGET aarch64-linux-gnu)
+ set(CMAKE_C_FLAGS_INIT "-march=armv9-a -save-temps=obj")
+ set(CMAKE_CXX_FLAGS_INIT "-march=armv9-a -save-temps=obj")
+ set(CMAKE_SYSTEM_PROCESSOR arm64)
+ set(CMAKE_LINKER_TYPE LLD)
+ EOF
+ cat << EOF > x86_64.cmake
+ set(CMAKE_C_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang)
+ set(CMAKE_CXX_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang++)
+ set(CMAKE_C_FLAGS_INIT "-save-temps=obj")
+ set(CMAKE_CXX_FLAGS_INIT "-save-temps=obj")
+ set(CMAKE_LINKER_TYPE LLD)
+ EOF
+ build_llvm_test_suite () {
+ cmake -B build.$1 -C cmake/caches/O3.cmake --toolchain $2 -DTEST_SUITE_BENCHMARKING_ONLY=ON -DTEST_SUITE_RUN_BENCHMARKS=OFF -GNinja
+ ninja -C build.$1
+ $GITHUB_WORKSPACE/build/bin/llvm-lit build.$1 -o results.$1.json
+ }
+ build_llvm_test_suite rva23u64-O3-b rva23u64.cmake
+ build_llvm_test_suite armv9-a-O3-b armv9-a.cmake
+ build_llvm_test_suite x86_64-O3-b x86_64.cmake
+ working-directory: llvm-test-suite
+ - name: Build test-suite @ base
+ run: git checkout $BASE_SHA && ninja -C build
+ - name: Configure and build test-suite @ base
+ run: |
+ build_llvm_test_suite () {
+ cmake -B build.$1 -C cmake/caches/O3.cmake --toolchain $2 -DTEST_SUITE_BENCHMARKING_ONLY=ON -DTEST_SUITE_RUN_BENCHMARKS=OFF -GNinja
+ ninja -C build.$1
+ $GITHUB_WORKSPACE/build/bin/llvm-lit build.$1 -o results.$1.json
+ }
+ build_llvm_test_suite rva23u64-O3-a rva23u64.cmake
+ build_llvm_test_suite armv9-a-O3-a armv9-a.cmake
+ build_llvm_test_suite x86_64-O3-a x86_64.cmake
+ working-directory: llvm-test-suite
+ - run: |
+ mkdir diffs
+ ./utils/tdiff.py -a build.rva23u64-O3-a -b build.rva23u64-O3-b -s all > diffs/rva23u64-O3.diff || true
+ ./utils/tdiff.py -a build.armv9-a-O3-a -b build.armv9-a-O3-b -s all > diffs/armv9-a-O3.diff || true
+ ./utils/tdiff.py -a build.x86_64-O3-a -b build.x86_64-O3-b -s all > diffs/x86_64-O3.diff || true
+ working-directory: llvm-test-suite
+ - uses: actions/upload-artifact at bbbca2ddaa5d8feaa63e36b76fdaad77386f024f #v7.0.0
+ id: upload-diffs
+ with:
+ name: diffs
+ path: llvm-test-suite/diffs
+ - uses: actions/upload-artifact at bbbca2ddaa5d8feaa63e36b76fdaad77386f024f #v7.0.0
+ with:
+ name: results
+ path: llvm-test-suite/results*.json
+ - uses: actions/github-script at ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
+ env:
+ DIFF_URL: ${{ steps.upload-diffs.outputs.artifact-url }}
+ with:
+ script: |
+ github.rest.issues.createComment({
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ issue_number: context.issue.number,
+ body: `test-suite diff from ${process.env.BASE_SHA}...${process.env.HEAD_SHA}: ${process.env.DIFF_URL}`
+ })
>From ba5f906c3b42df714931ba8515a519783dd3be86 Mon Sep 17 00:00:00 2001
From: Luke Lau <luke at igalia.com>
Date: Thu, 2 Apr 2026 10:55:14 +0800
Subject: [PATCH 2/4] Address review comments
* .ll -> .yml (doh)
* Put cmake files in separate location
* Add bash script for configuring and building llvm-test-suite
---
.../{test-suite.ll => test-suite.yml} | 52 +++----------------
.github/workflows/test-suite/aarch64.cmake | 9 ++++
.../test-suite/configure-and-build.sh | 12 +++++
.github/workflows/test-suite/riscv64.cmake | 9 ++++
.github/workflows/test-suite/x86_64.cmake | 5 ++
5 files changed, 42 insertions(+), 45 deletions(-)
rename .github/workflows/{test-suite.ll => test-suite.yml} (64%)
create mode 100644 .github/workflows/test-suite/aarch64.cmake
create mode 100755 .github/workflows/test-suite/configure-and-build.sh
create mode 100644 .github/workflows/test-suite/riscv64.cmake
create mode 100644 .github/workflows/test-suite/x86_64.cmake
diff --git a/.github/workflows/test-suite.ll b/.github/workflows/test-suite.yml
similarity index 64%
rename from .github/workflows/test-suite.ll
rename to .github/workflows/test-suite.yml
index 0d9c81fc74083..3cd3946a80161 100644
--- a/.github/workflows/test-suite.ll
+++ b/.github/workflows/test-suite.yml
@@ -55,6 +55,7 @@ jobs:
- run: |
echo "HEAD_SHA=$(git rev-parse HEAD)" >> $GITHUB_ENV
echo "BASE_SHA=$(git rev-parse HEAD^)" >> $GITHUB_ENV
+ echo "$GITHUB_WORKSPACE/.github/workflows/test-suite" >> $GITHUB_PATH
- uses: actions/checkout at 8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
with:
repository: llvm/llvm-test-suite
@@ -69,56 +70,17 @@ jobs:
run: ninja -C build
- name: Configure and build test-suite @ head
run: |
- cat << EOF > rva23u64.cmake
- set(CMAKE_SYSTEM_NAME Linux)
- set(CMAKE_C_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang)
- set(CMAKE_CXX_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang++)
- set(CMAKE_C_COMPILER_TARGET riscv64-linux-gnu)
- set(CMAKE_CXX_COMPILER_TARGET riscv64-linux-gnu)
- set(CMAKE_C_FLAGS_INIT "-march=rva23u64 -save-temps=obj")
- set(CMAKE_CXX_FLAGS_INIT "-march=rva23u64 -save-temps=obj")
- set(CMAKE_SYSTEM_PROCESSOR riscv64)
- set(CMAKE_LINKER_TYPE LLD)
- EOF
- cat << EOF > armv9-a.cmake
- set(CMAKE_SYSTEM_NAME Linux)
- set(CMAKE_C_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang)
- set(CMAKE_CXX_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang++)
- set(CMAKE_C_COMPILER_TARGET aarch64-linux-gnu)
- set(CMAKE_CXX_COMPILER_TARGET aarch64-linux-gnu)
- set(CMAKE_C_FLAGS_INIT "-march=armv9-a -save-temps=obj")
- set(CMAKE_CXX_FLAGS_INIT "-march=armv9-a -save-temps=obj")
- set(CMAKE_SYSTEM_PROCESSOR arm64)
- set(CMAKE_LINKER_TYPE LLD)
- EOF
- cat << EOF > x86_64.cmake
- set(CMAKE_C_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang)
- set(CMAKE_CXX_COMPILER ${GITHUB_WORKSPACE}/build/bin/clang++)
- set(CMAKE_C_FLAGS_INIT "-save-temps=obj")
- set(CMAKE_CXX_FLAGS_INIT "-save-temps=obj")
- set(CMAKE_LINKER_TYPE LLD)
- EOF
- build_llvm_test_suite () {
- cmake -B build.$1 -C cmake/caches/O3.cmake --toolchain $2 -DTEST_SUITE_BENCHMARKING_ONLY=ON -DTEST_SUITE_RUN_BENCHMARKS=OFF -GNinja
- ninja -C build.$1
- $GITHUB_WORKSPACE/build/bin/llvm-lit build.$1 -o results.$1.json
- }
- build_llvm_test_suite rva23u64-O3-b rva23u64.cmake
- build_llvm_test_suite armv9-a-O3-b armv9-a.cmake
- build_llvm_test_suite x86_64-O3-b x86_64.cmake
+ build-and-configure.sh rva23u64-O3-b riscv64.cmake
+ build-and-configure.sh armv9-a-O3-b aarch64.cmake
+ build-and-configure.sh x86_64-O3-b x86_64.cmake
working-directory: llvm-test-suite
- name: Build test-suite @ base
run: git checkout $BASE_SHA && ninja -C build
- name: Configure and build test-suite @ base
run: |
- build_llvm_test_suite () {
- cmake -B build.$1 -C cmake/caches/O3.cmake --toolchain $2 -DTEST_SUITE_BENCHMARKING_ONLY=ON -DTEST_SUITE_RUN_BENCHMARKS=OFF -GNinja
- ninja -C build.$1
- $GITHUB_WORKSPACE/build/bin/llvm-lit build.$1 -o results.$1.json
- }
- build_llvm_test_suite rva23u64-O3-a rva23u64.cmake
- build_llvm_test_suite armv9-a-O3-a armv9-a.cmake
- build_llvm_test_suite x86_64-O3-a x86_64.cmake
+ build-and-configure.sh rva23u64-O3-a riscv64.cmake
+ build-and-configure.sh armv9-a-O3-a aarch64.cmake
+ build-and-configure.sh x86_64-O3-a x86_64.cmake
working-directory: llvm-test-suite
- run: |
mkdir diffs
diff --git a/.github/workflows/test-suite/aarch64.cmake b/.github/workflows/test-suite/aarch64.cmake
new file mode 100644
index 0000000000000..4215cf8e9d5eb
--- /dev/null
+++ b/.github/workflows/test-suite/aarch64.cmake
@@ -0,0 +1,9 @@
+set(CMAKE_SYSTEM_NAME Linux)
+set(CMAKE_C_COMPILER $ENV{GITHUB_WORKSPACE}/build/bin/clang)
+set(CMAKE_CXX_COMPILER $ENV{GITHUB_WORKSPACE}/build/bin/clang++)
+set(CMAKE_C_COMPILER_TARGET aarch64-linux-gnu)
+set(CMAKE_CXX_COMPILER_TARGET aarch64-linux-gnu)
+set(CMAKE_C_FLAGS_INIT "-march=armv9-a -save-temps=obj")
+set(CMAKE_CXX_FLAGS_INIT "-march=armv9-a -save-temps=obj")
+set(CMAKE_SYSTEM_PROCESSOR arm64)
+set(CMAKE_LINKER_TYPE LLD)
diff --git a/.github/workflows/test-suite/configure-and-build.sh b/.github/workflows/test-suite/configure-and-build.sh
new file mode 100755
index 0000000000000..248e408e1ca9b
--- /dev/null
+++ b/.github/workflows/test-suite/configure-and-build.sh
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+set -eux
+
+cmake -B build.$1 \
+ --toolchain $GITHUB_WORKSPACE/.github/workflows/test-suite/$2 \
+ -C cmake/caches/O3.cmake \
+ -GNinja \
+ -DTEST_SUITE_BENCHMARKING_ONLY=ON \
+ -DTEST_SUITE_RUN_BENCHMARKS=OFF
+ninja -C build.$1
+$GITHUB_WORKSPACE/build/bin/llvm-lit build.$1 -o results.$1.json
diff --git a/.github/workflows/test-suite/riscv64.cmake b/.github/workflows/test-suite/riscv64.cmake
new file mode 100644
index 0000000000000..19f924bcd6003
--- /dev/null
+++ b/.github/workflows/test-suite/riscv64.cmake
@@ -0,0 +1,9 @@
+set(CMAKE_SYSTEM_NAME Linux)
+set(CMAKE_C_COMPILER $ENV{GITHUB_WORKSPACE}/build/bin/clang)
+set(CMAKE_CXX_COMPILER $ENV{GITHUB_WORKSPACE}/build/bin/clang++)
+set(CMAKE_C_COMPILER_TARGET riscv64-linux-gnu)
+set(CMAKE_CXX_COMPILER_TARGET riscv64-linux-gnu)
+set(CMAKE_C_FLAGS_INIT "-march=rva23u64 -save-temps=obj")
+set(CMAKE_CXX_FLAGS_INIT "-march=rva23u64 -save-temps=obj")
+set(CMAKE_SYSTEM_PROCESSOR riscv64)
+set(CMAKE_LINKER_TYPE LLD)
diff --git a/.github/workflows/test-suite/x86_64.cmake b/.github/workflows/test-suite/x86_64.cmake
new file mode 100644
index 0000000000000..d1150ac1a42e7
--- /dev/null
+++ b/.github/workflows/test-suite/x86_64.cmake
@@ -0,0 +1,5 @@
+set(CMAKE_C_COMPILER $ENV{GITHUB_WORKSPACE}/build/bin/clang)
+set(CMAKE_CXX_COMPILER $ENV{GITHUB_WORKSPACE}/build/bin/clang++)
+set(CMAKE_C_FLAGS_INIT "-save-temps=obj")
+set(CMAKE_CXX_FLAGS_INIT "-save-temps=obj")
+set(CMAKE_LINKER_TYPE LLD)
>From 68707da10b18f55a403724133986c0911fe7f9b0 Mon Sep 17 00:00:00 2001
From: Luke Lau <luke at igalia.com>
Date: Thu, 2 Apr 2026 11:04:55 +0800
Subject: [PATCH 3/4] Fix permissions
---
.github/workflows/test-suite.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/test-suite.yml b/.github/workflows/test-suite.yml
index 3cd3946a80161..2a0725d43625c 100644
--- a/.github/workflows/test-suite.yml
+++ b/.github/workflows/test-suite.yml
@@ -15,7 +15,7 @@ jobs:
name: Build and diff
runs-on: ubuntu-24.04
permissions:
- issues: write
+ pull-requests: write
if: >-
!startswith(github.event.comment.body, '<!--IGNORE-->') &&
github.event.issue.pull_request && contains(github.event.comment.body, '/test-suite')
>From af377f061e65adf6b1777de5c4de5a1659feb4a4 Mon Sep 17 00:00:00 2001
From: Luke Lau <luke at igalia.com>
Date: Thu, 2 Apr 2026 11:30:14 +0800
Subject: [PATCH 4/4] Use scripts + cmake files from default branch of upstream
repo
Instead of using untrusted PR repo
---
.github/workflows/test-suite.yml | 58 +++++++++++++------
.../test-suite/configure-and-build.sh | 2 +-
2 files changed, 40 insertions(+), 20 deletions(-)
diff --git a/.github/workflows/test-suite.yml b/.github/workflows/test-suite.yml
index 2a0725d43625c..d6048bf05a93f 100644
--- a/.github/workflows/test-suite.yml
+++ b/.github/workflows/test-suite.yml
@@ -20,7 +20,8 @@ jobs:
!startswith(github.event.comment.body, '<!--IGNORE-->') &&
github.event.issue.pull_request && contains(github.event.comment.body, '/test-suite')
steps:
- - id: get-pr
+ - name: Get pull request
+ id: get-pr
uses: actions/github-script at ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
with:
script: |
@@ -36,9 +37,11 @@ jobs:
body: "Can't diff PR, PR isn't mergeable"
})
return pr
- - if: ${{ !fromJSON(steps.get-pr.outputs.result).mergeable }}
+ - name: Check pull request is mergeable
+ if: ${{ !fromJSON(steps.get-pr.outputs.result).mergeable }}
run: exit 1
- - uses: actions/github-script at ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
+ - name: Thumbs up comment
+ uses: actions/github-script at ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
with:
script: |
github.rest.reactions.createForIssueComment({
@@ -47,19 +50,32 @@ jobs:
comment_id: context.payload.comment.id,
content: '+1'
})
- - uses: actions/checkout at 8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
+ - name: Checkout pull request
+ uses: actions/checkout at 8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
with:
ref: ${{ fromJSON(steps.get-pr.outputs.result).merge_commit_sha }}
repository: ${{ fromJSON(steps.get-pr.outputs.result).head.repo.full_name }}
fetch-depth: 2
- - run: |
- echo "HEAD_SHA=$(git rev-parse HEAD)" >> $GITHUB_ENV
- echo "BASE_SHA=$(git rev-parse HEAD^)" >> $GITHUB_ENV
- echo "$GITHUB_WORKSPACE/.github/workflows/test-suite" >> $GITHUB_PATH
- - uses: actions/checkout at 8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
+ # Check out the test-suite scripts and CMake files from the default branch
+ # on the upstream repository. Use these instead of the scripts in the pull
+ # request's branch since it's untrusted.
+ - name: Checkout scripts and CMake files
+ uses: actions/checkout at 8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
+ with:
+ path: test-suite
+ sparse-checkout: .github/workflows/test-suite
+ - name: Checkout llvm/llvm-test-suite
+ uses: actions/checkout at 8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
with:
repository: llvm/llvm-test-suite
path: llvm-test-suite
+ - name: Setup environment variables
+ run: |
+ echo "HEAD_SHA=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ echo "BASE_SHA=$(git rev-parse HEAD^)" >> $GITHUB_ENV
+ SCRIPTS_DIR="$GITHUB_WORKSPACE/test-suite/.github/workflows/test-suite"
+ echo "SCRIPTS_DIR=$SCRIPTS_DIR" >> $GITHUB_ENV
+ echo "$SCRIPTS_DIR" >> $GITHUB_PATH
- name: Install system dependencies
run: |
sudo apt-get update
@@ -70,34 +86,38 @@ jobs:
run: ninja -C build
- name: Configure and build test-suite @ head
run: |
- build-and-configure.sh rva23u64-O3-b riscv64.cmake
- build-and-configure.sh armv9-a-O3-b aarch64.cmake
- build-and-configure.sh x86_64-O3-b x86_64.cmake
+ build-and-configure.sh rva23u64-O3-b $SCRIPTS_DIR/riscv64.cmake
+ build-and-configure.sh armv9-a-O3-b $SCRIPTS_DIR/aarch64.cmake
+ build-and-configure.sh x86_64-O3-b $SCRIPTS_DIR/x86_64.cmake
working-directory: llvm-test-suite
- name: Build test-suite @ base
run: git checkout $BASE_SHA && ninja -C build
- name: Configure and build test-suite @ base
run: |
- build-and-configure.sh rva23u64-O3-a riscv64.cmake
- build-and-configure.sh armv9-a-O3-a aarch64.cmake
- build-and-configure.sh x86_64-O3-a x86_64.cmake
+ build-and-configure.sh rva23u64-O3-a $SCRIPTS_DIR/riscv64.cmake
+ build-and-configure.sh armv9-a-O3-a $SCRIPTS_DIR/aarch64.cmake
+ build-and-configure.sh x86_64-O3-a $SCRIPTS_DIR/x86_64.cmake
working-directory: llvm-test-suite
- - run: |
+ - name: Compute diffs
+ run: |
mkdir diffs
./utils/tdiff.py -a build.rva23u64-O3-a -b build.rva23u64-O3-b -s all > diffs/rva23u64-O3.diff || true
./utils/tdiff.py -a build.armv9-a-O3-a -b build.armv9-a-O3-b -s all > diffs/armv9-a-O3.diff || true
./utils/tdiff.py -a build.x86_64-O3-a -b build.x86_64-O3-b -s all > diffs/x86_64-O3.diff || true
working-directory: llvm-test-suite
- - uses: actions/upload-artifact at bbbca2ddaa5d8feaa63e36b76fdaad77386f024f #v7.0.0
+ - name: Upload diffs
+ uses: actions/upload-artifact at bbbca2ddaa5d8feaa63e36b76fdaad77386f024f #v7.0.0
id: upload-diffs
with:
name: diffs
path: llvm-test-suite/diffs
- - uses: actions/upload-artifact at bbbca2ddaa5d8feaa63e36b76fdaad77386f024f #v7.0.0
+ - name: Upload results
+ uses: actions/upload-artifact at bbbca2ddaa5d8feaa63e36b76fdaad77386f024f #v7.0.0
with:
name: results
path: llvm-test-suite/results*.json
- - uses: actions/github-script at ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
+ - name: Create comment
+ uses: actions/github-script at ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
env:
DIFF_URL: ${{ steps.upload-diffs.outputs.artifact-url }}
with:
diff --git a/.github/workflows/test-suite/configure-and-build.sh b/.github/workflows/test-suite/configure-and-build.sh
index 248e408e1ca9b..14303a5015493 100755
--- a/.github/workflows/test-suite/configure-and-build.sh
+++ b/.github/workflows/test-suite/configure-and-build.sh
@@ -3,7 +3,7 @@
set -eux
cmake -B build.$1 \
- --toolchain $GITHUB_WORKSPACE/.github/workflows/test-suite/$2 \
+ --toolchain $2 \
-C cmake/caches/O3.cmake \
-GNinja \
-DTEST_SUITE_BENCHMARKING_ONLY=ON \
More information about the llvm-commits
mailing list