[libcxx] [llvm] [libc++] Add a job that runs the benchmarks as part of the CI (PR #157567)
Louis Dionne via llvm-commits
llvm-commits at lists.llvm.org
Thu Sep 11 09:31:52 PDT 2025
https://github.com/ldionne updated https://github.com/llvm/llvm-project/pull/157567
>From c7306f453076f0c8a962281ed406aad6ce6e21eb Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Mon, 8 Sep 2025 18:12:17 -0400
Subject: [PATCH 1/3] [libc++] Add a job that runs the benchmarks as part of
the CI
This will make it easier to see the performance impact of individual
patches. This is a first step towards better performance tracking, but
many things still need to be figured out:
- We should get a dedicated set of machines to run these tests on
- We could probably reuse regularly-generated baseline results published
to LNT instead of re-generating the baselines in this job every time.
---
.github/workflows/libcxx-build-and-test.yaml | 22 ++++++++++++++++++++
libcxx/utils/ci/run-buildbot | 18 ++++++++++++++++
2 files changed, 40 insertions(+)
diff --git a/.github/workflows/libcxx-build-and-test.yaml b/.github/workflows/libcxx-build-and-test.yaml
index 2e6ff7f91b6fc..80da80bdc59e7 100644
--- a/.github/workflows/libcxx-build-and-test.yaml
+++ b/.github/workflows/libcxx-build-and-test.yaml
@@ -284,3 +284,25 @@ jobs:
- name: Build and test
run: |
bash libcxx/utils/ci/run-buildbot ${{ matrix.config }}
+
+ benchmarks:
+ runs-on: llvm-premerge-libcxx-next-runners # TODO: This should run on a dedicated set of machines
+ # needs: [ stage3 ] # TODO: Run this at the end of the pipeline once we're done testing the job
+ steps:
+ - uses: actions/checkout at 08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ - name: benchmarks
+ run: libcxx/utils/ci/run-buildbot benchmark-comparison
+ env:
+ CC: clang-22
+ CXX: clang++-22
+ - uses: actions/upload-artifact at 26f96dfa697d77e81fd5907df203aa23a56210a8 # v4.3.0
+ if: always()
+ with:
+ name: benchmarks-results
+ path: |
+ **/test-results.xml
+ **/*.abilist
+ **/CMakeConfigureLog.yaml
+ **/CMakeError.log
+ **/CMakeOutput.log
+ **/crash_diagnostics/*
diff --git a/libcxx/utils/ci/run-buildbot b/libcxx/utils/ci/run-buildbot
index 57ecf1e49dbf2..6a776b6645723 100755
--- a/libcxx/utils/ci/run-buildbot
+++ b/libcxx/utils/ci/run-buildbot
@@ -542,6 +542,24 @@ generic-optimized-speed)
generate-cmake -C "${MONOREPO_ROOT}/libcxx/cmake/caches/Generic-optimized-speed.cmake"
check-runtimes
;;
+benchmark-comparison)
+ clean
+ LIT_ARGS="-sv -j1 --param optimization=speed libcxx/test/benchmarks/hash.bench.cpp libcxx/test/benchmarks/join_view.bench.cpp"
+
+ baseline_commit=$(git merge-base main HEAD)
+ step "Building and running the baseline at ${baseline_commit}"
+ rm -rf "${BUILD_DIR}-baseline"
+ ./libcxx/utils/test-at-commit --commit ${baseline_commit} -B "${BUILD_DIR}-baseline" -- ${LIT_ARGS}
+
+ # Build and run the candidate
+ step "Building and running the candidate"
+ generate-cmake -C "${MONOREPO_ROOT}/libcxx/cmake/caches/Generic-optimized-speed.cmake"
+ ./libcxx/utils/libcxx-lit "${BUILD_DIR}" ${LIT_ARGS}
+
+ step "Comparison of the baseline and the candidate"
+ ./libcxx/utils/compare-benchmarks <(./libcxx/utils/consolidate-benchmarks "${BUILD_DIR}-baseline") \
+ <(./libcxx/utils/consolidate-benchmarks "${BUILD_DIR}")
+;;
apple-configuration)
clean
>From 567efcabb14f4b695a9b267b505443d0d592c05b Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Thu, 11 Sep 2025 12:25:40 -0400
Subject: [PATCH 2/3] Try to fix baseline commit
---
libcxx/utils/ci/run-buildbot | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/libcxx/utils/ci/run-buildbot b/libcxx/utils/ci/run-buildbot
index 6a776b6645723..b888faed975da 100755
--- a/libcxx/utils/ci/run-buildbot
+++ b/libcxx/utils/ci/run-buildbot
@@ -546,10 +546,12 @@ benchmark-comparison)
clean
LIT_ARGS="-sv -j1 --param optimization=speed libcxx/test/benchmarks/hash.bench.cpp libcxx/test/benchmarks/join_view.bench.cpp"
- baseline_commit=$(git merge-base main HEAD)
- step "Building and running the baseline at ${baseline_commit}"
+ if [[ "${BASELINE_COMMIT}" == "" ]]; then
+ BASELINE_COMMIT=$(git merge-base main HEAD)
+ fi
+ step "Building and running the baseline at ${BASELINE_COMMIT}"
rm -rf "${BUILD_DIR}-baseline"
- ./libcxx/utils/test-at-commit --commit ${baseline_commit} -B "${BUILD_DIR}-baseline" -- ${LIT_ARGS}
+ ./libcxx/utils/test-at-commit --commit ${BASELINE_COMMIT} -B "${BUILD_DIR}-baseline" -- ${LIT_ARGS}
# Build and run the candidate
step "Building and running the candidate"
>From 9ac9dc92978a281f2e0d0e652aca74250be2858d Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Thu, 11 Sep 2025 12:31:40 -0400
Subject: [PATCH 3/3] Try again
---
libcxx/utils/ci/run-buildbot | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/libcxx/utils/ci/run-buildbot b/libcxx/utils/ci/run-buildbot
index b888faed975da..373e298183d86 100755
--- a/libcxx/utils/ci/run-buildbot
+++ b/libcxx/utils/ci/run-buildbot
@@ -546,12 +546,15 @@ benchmark-comparison)
clean
LIT_ARGS="-sv -j1 --param optimization=speed libcxx/test/benchmarks/hash.bench.cpp libcxx/test/benchmarks/join_view.bench.cpp"
- if [[ "${BASELINE_COMMIT}" == "" ]]; then
- BASELINE_COMMIT=$(git merge-base main HEAD)
+ if [[ "${GITHUB_BASE_REF}" != "" ]]; then # triggered from a PR
+ baseline_commit=$(git merge-base ${GITHUB_BASE_REF} ${GITHUB_SHA})
+ else
+ baseline_commit=$(git merge-base main HEAD)
fi
- step "Building and running the baseline at ${BASELINE_COMMIT}"
+
+ step "Building and running the baseline at ${baseline_commit}"
rm -rf "${BUILD_DIR}-baseline"
- ./libcxx/utils/test-at-commit --commit ${BASELINE_COMMIT} -B "${BUILD_DIR}-baseline" -- ${LIT_ARGS}
+ ./libcxx/utils/test-at-commit --commit ${baseline_commit} -B "${BUILD_DIR}-baseline" -- ${LIT_ARGS}
# Build and run the candidate
step "Building and running the candidate"
More information about the llvm-commits
mailing list