[libcxx-commits] [libcxx] [llvm] [libc++] Pin down the compiler in the various benchmark scripts and jobs (PR #211563)
Louis Dionne via libcxx-commits
libcxx-commits at lists.llvm.org
Thu Jul 23 07:03:42 PDT 2026
https://github.com/ldionne created https://github.com/llvm/llvm-project/pull/211563
We have various scripts that build and test the library at pinpointed commits: benchmark-historical, run-benchmarks, build-at-commit and test-at-commit. They were handling the compiler in different ways: some scripts would just run the libc++ build (or test suite configuration) without specifying the compiler, which means the $CXX environment variable was used if present. Other scripts (e.g. run-benchmarks) would accept the compiler as an argument, but would fail to pass it down when configuring the test suite, which led to issues.
This patch passes the compiler explicitly in all scripts: this removes any potential confusion around how the compiler should be specified (env var or argument). The only exception is build-at-commit, where the compiler is still specified by passing the appropriate CMake arguments. The reason for this exception is that passing arguments to CMake is actually the way we want to configure aspects of the build (and the test suite) in the long term, it's just that the test suite doesn't support this cleanly due to the CMake/Lit split at the moment.
In the longer term, `test-at-commit` should also lose its `--compiler` argument in favour of being able to pass CMake parameters to the test suite configuration, but we are not there yet.
>From ec040e5440e49ccbbd244f197352bf979d8118c1 Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Thu, 23 Jul 2026 09:48:34 -0400
Subject: [PATCH] [libc++] Pin down the compiler in the various benchmark
scripts and jobs
We have various scripts that build and test the library at pinpointed
commits: benchmark-historical, run-benchmarks, build-at-commit and
test-at-commit. They were handling the compiler in different ways:
some scripts would just run the libc++ build (or test suite configuration)
without specifying the compiler, which means the $CXX environment variable
was used if present. Other scripts (e.g. run-benchmarks) would accept
the compiler as an argument, but would fail to pass it down when
configuring the test suite, which led to issues.
This patch passes the compiler explicitly in all scripts: this removes
any potential confusion around how the compiler should be specified
(env var or argument). The only exception is build-at-commit, where
the compiler is still specified by passing the appropriate CMake
arguments. The reason for this exception is that passing arguments
to CMake is actually the way we want to configure aspects of the build
(and the test suite) in the long term, it's just that the test suite
doesn't support this cleanly due to the CMake/Lit split at the moment.
In the longer term, `test-at-commit` should also lose its `--compiler`
argument in favour of being able to pass CMake parameters to the test
suite configuration, but we are not there yet.
---
.github/workflows/libcxx-benchmark-commit.yml | 6 ++--
.github/workflows/libcxx-pr-benchmark.yml | 34 +++++++------------
libcxx/utils/benchmark-historical | 6 +++-
libcxx/utils/ci/lnt/run-benchmarks | 1 +
libcxx/utils/test-at-commit | 3 ++
5 files changed, 26 insertions(+), 24 deletions(-)
diff --git a/.github/workflows/libcxx-benchmark-commit.yml b/.github/workflows/libcxx-benchmark-commit.yml
index 4d72f1d126432..f2096cc968b26 100644
--- a/.github/workflows/libcxx-benchmark-commit.yml
+++ b/.github/workflows/libcxx-benchmark-commit.yml
@@ -52,6 +52,8 @@ jobs:
install-macos-dependencies: false
fail-fast: false
runs-on: ${{ matrix.runner }}
+ env:
+ COMPILER: ${{ matrix.cxx }}
steps:
- name: Checkout the LLVM monorepo
uses: actions/checkout at df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
@@ -76,7 +78,7 @@ jobs:
run: |
cmake --version
ninja --version
- ${{ matrix.cxx }} --version
+ "${COMPILER}" --version
- name: Run the benchmarks
env:
@@ -91,7 +93,7 @@ jobs:
libcxx/utils/ci/lnt/run-benchmarks \
--test-suite-commit "${BENCHMARK_SUITE_VERSION}" \
--machine ${{ matrix.lnt-machine }} \
- --compiler ${{ matrix.cxx }} \
+ --compiler "${COMPILER}" \
--benchmark-commit "${COMMIT}" \
"${filter_arg[@]}" \
--output "${COMMIT}.json"
diff --git a/.github/workflows/libcxx-pr-benchmark.yml b/.github/workflows/libcxx-pr-benchmark.yml
index 1603962b6883f..7e7cf12ec9921 100644
--- a/.github/workflows/libcxx-pr-benchmark.yml
+++ b/.github/workflows/libcxx-pr-benchmark.yml
@@ -91,13 +91,11 @@ jobs:
include:
- platform: macOS 26.5 arm64
runner: ["self-hosted", "macOS", "ARM64", "26", "26.5"]
- cc: clang
cxx: clang++
install-python: false
install-macos-dependencies: true
- platform: Linux x86_64
runner: llvm-premerge-libcxx-runners
- cc: clang-22
cxx: clang++-22
install-python: true
install-macos-dependencies: false
@@ -108,8 +106,11 @@ jobs:
needs:
- extract-info
env:
- CC: ${{ matrix.cc }}
- CXX: ${{ matrix.cxx }}
+ BENCHMARKS: ${{ needs.extract-info.outputs.benchmarks }}
+ COMPILER: ${{ matrix.cxx }}
+ PLATFORM: ${{ matrix.platform }}
+ PR_HEAD: ${{ needs.extract-info.outputs.pr_head }}
+ PR_BASE: ${{ needs.extract-info.outputs.pr_base }}
steps:
- uses: actions/checkout at df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
@@ -136,34 +137,29 @@ jobs:
python -m pip install -r libcxx/utils/requirements.txt
- name: Build the baseline and the candidate
- env:
- PR_HEAD: ${{ needs.extract-info.outputs.pr_head }}
- PR_BASE: ${{ needs.extract-info.outputs.pr_base }}
run: |
source .venv/bin/activate
baseline_commit=$(git merge-base $PR_BASE $PR_HEAD)
- ./libcxx/utils/build-at-commit --commit ${baseline_commit} --install-dir install/baseline -- -DCMAKE_BUILD_TYPE=RelWithDebInfo
- ./libcxx/utils/build-at-commit --commit $PR_HEAD --install-dir install/candidate -- -DCMAKE_BUILD_TYPE=RelWithDebInfo
+ ./libcxx/utils/build-at-commit --commit ${baseline_commit} --install-dir install/baseline -- -DCMAKE_CXX_COMPILER="${COMPILER}" -DCMAKE_BUILD_TYPE=RelWithDebInfo
+ ./libcxx/utils/build-at-commit --commit $PR_HEAD --install-dir install/candidate -- -DCMAKE_CXX_COMPILER="${COMPILER}" -DCMAKE_BUILD_TYPE=RelWithDebInfo
- name: Run baseline and candidate interleaved
- env:
- BENCHMARKS: ${{ needs.extract-info.outputs.benchmarks }}
run: |
source .venv/bin/activate
# Run 3 times so we can pick the median, and interleave to mitigate the impact of environmental noise
- ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline -- -sv -j1 --param optimization=speed "$BENCHMARKS"
+ ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee baseline.lnt
- ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate -- -sv -j1 --param optimization=speed "$BENCHMARKS"
+ ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee candidate.lnt
- ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline -- -sv -j1 --param optimization=speed "$BENCHMARKS"
+ ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee -a baseline.lnt
- ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate -- -sv -j1 --param optimization=speed "$BENCHMARKS"
+ ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee -a candidate.lnt
- ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline -- -sv -j1 --param optimization=speed "$BENCHMARKS"
+ ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee -a baseline.lnt
- ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate -- -sv -j1 --param optimization=speed "$BENCHMARKS"
+ ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee -a candidate.lnt
- name: Compare baseline and candidate runs
@@ -173,8 +169,6 @@ jobs:
- name: Update comment with results
uses: actions/github-script at 3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
- env:
- PLATFORM: ${{ matrix.platform }}
with:
script: |
const fs = require('fs');
@@ -206,8 +200,6 @@ jobs:
- name: Report failure in the comment
if: failure()
uses: actions/github-script at 3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
- env:
- PLATFORM: ${{ matrix.platform }}
with:
script: |
const { data: comment } = await github.rest.issues.getComment({
diff --git a/libcxx/utils/benchmark-historical b/libcxx/utils/benchmark-historical
index 8a315be522d5c..48946e4bee075 100755
--- a/libcxx/utils/benchmark-historical
+++ b/libcxx/utils/benchmark-historical
@@ -40,6 +40,8 @@ def main(argv):
're-run on a potentially-overlapping set of commits, such as after pulling new commits with Git.')
parser.add_argument('--output', '-o', type=pathlib.Path, required=True,
help='Path to the directory where the resulting .lnt files are stored.')
+ parser.add_argument('--compiler', type=str, required=True,
+ help='Path to the compiler to use to build libc++ and run the tests.')
parser.add_argument('--commit-list', type=argparse.FileType('r'), default=sys.stdin,
help='Path to a file containing a whitespace separated list of commits to test. '
'By default, this is read from standard input.')
@@ -83,10 +85,12 @@ def main(argv):
build_cmd = [PARENT_DIR / 'build-at-commit', '--git-repo', args.git_repo,
'--commit', commit,
'--install-dir', libcxx_install_dir,
- '--', '-DCMAKE_BUILD_TYPE=RelWithDebInfo']
+ '--', '-DCMAKE_BUILD_TYPE=RelWithDebInfo',
+ f'-DCMAKE_CXX_COMPILER={args.compiler}']
test_cmd = [PARENT_DIR / 'test-at-commit', '--git-repo', args.git_repo,
'--libcxx-installation', libcxx_install_dir,
+ '--compiler', args.compiler,
'--build-dir', build_dir]
test_cmd += ['--'] + lit_options
diff --git a/libcxx/utils/ci/lnt/run-benchmarks b/libcxx/utils/ci/lnt/run-benchmarks
index ce4705c0fafa1..2fd53008cae72 100755
--- a/libcxx/utils/ci/lnt/run-benchmarks
+++ b/libcxx/utils/ci/lnt/run-benchmarks
@@ -167,6 +167,7 @@ def main(argv):
'--build-dir', artifacts / 'benchmarks',
'--test-suite-commit', args.test_suite_commit,
'--libcxx-installation', artifacts / 'libcxx-install',
+ '--compiler', args.compiler,
'--',
'-j1', '--time-tests', '--test-output=failed',
'--param', f'compiler={args.compiler}',
diff --git a/libcxx/utils/test-at-commit b/libcxx/utils/test-at-commit
index 4249d9c1ce86e..b02c571ca9b1b 100755
--- a/libcxx/utils/test-at-commit
+++ b/libcxx/utils/test-at-commit
@@ -85,6 +85,8 @@ def main(argv):
'are located in that directory after the run.')
parser.add_argument('--libcxx-installation', type=pathlib.Path, required=True,
help='Path to the directory where a copy of libc++ to run tests on is installed.')
+ parser.add_argument('--compiler', type=pathlib.Path, required=True,
+ help='Path to the compiler to use to run the tests.')
parser.add_argument('--test-suite-commit', type=str, required=False,
help='Commit to use for the test suite. If left unspecified, the currently checked-out version of the '
'test suite is used. Otherwise, the requested version is checked out in a separate directory and '
@@ -125,6 +127,7 @@ def main(argv):
f.write(LIT_CONFIG_FILE.format(INSTALL_ROOT=args.libcxx_installation))
test_suite_cmd = ['cmake', '-B', args.build_dir, '-S', test_suite_sources / 'runtimes', '-G', 'Ninja']
+ test_suite_cmd += ['-D', f'CMAKE_CXX_COMPILER={args.compiler}']
test_suite_cmd += ['-D', 'LLVM_ENABLE_RUNTIMES=libcxx;libcxxabi']
test_suite_cmd += ['-D', 'LIBCXXABI_USE_LLVM_UNWINDER=OFF']
test_suite_cmd += ['-D', f'LIBCXX_TEST_CONFIG={lit_cfg}']
More information about the libcxx-commits
mailing list