[libcxx] [llvm] [libc++] Encode Lit configuration parameters in machines.json (PR #225601)
Louis Dionne via llvm-commits
llvm-commits at lists.llvm.org
Tue Sep 22 22:55:30 PDT 2026
https://github.com/ldionne created https://github.com/llvm/llvm-project/pull/225601
Instead of hardcoding a bunch of Lit parameters in run-benchmarks, encode them in machines.json and thread them through. This makes machines.json closer to being the authoritative place describing the full benchmark configuration.
>From 005498395c7ce8f4a800971ed92fbefdb4273d25 Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Fri, 18 Sep 2026 12:32:46 -0700
Subject: [PATCH] [libc++] Encode Lit configuration parameters in machines.json
Instead of hardcoding a bunch of Lit parameters in run-benchmarks,
encode them in machines.json and thread them through. This makes
machines.json closer to being the authoritative place describing
the full benchmark configuration.
---
.github/workflows/libcxx-benchmark-commit.yml | 10 ++++++-
.github/workflows/libcxx-pr-benchmark.yml | 12 ++++++--
libcxx/utils/ci/lnt/README.md | 11 +++----
libcxx/utils/ci/lnt/machines.json | 3 ++
libcxx/utils/ci/lnt/run-benchmarks | 30 ++++++++++++-------
libcxx/utils/ci/run-buildbot | 3 +-
6 files changed, 50 insertions(+), 19 deletions(-)
diff --git a/.github/workflows/libcxx-benchmark-commit.yml b/.github/workflows/libcxx-benchmark-commit.yml
index f1f8965760fdb..a73574ac474ae 100644
--- a/.github/workflows/libcxx-benchmark-commit.yml
+++ b/.github/workflows/libcxx-benchmark-commit.yml
@@ -167,6 +167,7 @@ jobs:
BENCHMARK_SUITE_VERSION: ${{ matrix.benchmark-suite-version }}
COMMIT: ${{ inputs.commit }}
FILTER: ${{ inputs.filter }}
+ LIT_PARAMS: ${{ join(matrix.lit-params, ' ') }}
LNT_MACHINE: ${{ matrix.lnt-machine }}
run: |
source .venv/bin/activate
@@ -175,6 +176,12 @@ jobs:
filter_arg=(--filter "${FILTER}")
fi
+ read -ra configured_params <<< "${LIT_PARAMS}"
+ lit_params=()
+ for param in "${configured_params[@]}"; do
+ lit_params+=(--param "${param}")
+ done
+
# If the library build failed, still create an install directory so we
# run the benchmarks. The benchmarks will fail, and we'll submit an empty
# LNT report.
@@ -187,7 +194,8 @@ jobs:
--benchmark-commit "${COMMIT}" \
--libcxx-installation "${INSTALL_DIR}" \
"${filter_arg[@]}" \
- --output "${COMMIT}.json"
+ --output "${COMMIT}.json" \
+ -- "${lit_params[@]}"
cat "${COMMIT}.json"
diff --git a/.github/workflows/libcxx-pr-benchmark.yml b/.github/workflows/libcxx-pr-benchmark.yml
index b9e540c986d17..ac9d1082ddd45 100644
--- a/.github/workflows/libcxx-pr-benchmark.yml
+++ b/.github/workflows/libcxx-pr-benchmark.yml
@@ -111,6 +111,7 @@ jobs:
env:
BENCHMARKS: ${{ needs.extract-info.outputs.benchmarks }}
COMPILER: ${{ matrix.cxx }}
+ LIT_PARAMS: ${{ join(matrix.lit-params, ' ') }}
LNT_MACHINE: ${{ matrix.lnt-machine }}
PR_HEAD: ${{ needs.extract-info.outputs.pr_head }}
PR_BASE: ${{ needs.extract-info.outputs.pr_base }}
@@ -175,12 +176,19 @@ jobs:
- name: Run baseline and candidate interleaved
run: |
source .venv/bin/activate
+
+ read -ra configured_params <<< "${LIT_PARAMS}"
+ lit_params=()
+ for param in "${configured_params[@]}"; do
+ lit_params+=(--param "${param}")
+ done
+
# Run 5 times so we can pick the median, and interleave baseline and candidate to mitigate the impact of
# environmental noise
for _ in $(seq 1 5); do
- ./libcxx/utils/test-at-commit --test-config "${PWD}/libcxx/test/configs/installed-libc++.cfg.in" -B benchmarks/baseline --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed --param libcxx_installation="${PWD}/install/baseline" "$BENCHMARKS"
+ ./libcxx/utils/test-at-commit --test-config "${PWD}/libcxx/test/configs/installed-libc++.cfg.in" -B benchmarks/baseline --compiler "${COMPILER}" -- -sv -j1 "${lit_params[@]}" --param libcxx_installation="${PWD}/install/baseline" "$BENCHMARKS"
./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee -a baseline.lnt
- ./libcxx/utils/test-at-commit --test-config "${PWD}/libcxx/test/configs/installed-libc++.cfg.in" -B benchmarks/candidate --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed --param libcxx_installation="${PWD}/install/candidate" "$BENCHMARKS"
+ ./libcxx/utils/test-at-commit --test-config "${PWD}/libcxx/test/configs/installed-libc++.cfg.in" -B benchmarks/candidate --compiler "${COMPILER}" -- -sv -j1 "${lit_params[@]}" --param libcxx_installation="${PWD}/install/candidate" "$BENCHMARKS"
./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee -a candidate.lnt
done
diff --git a/libcxx/utils/ci/lnt/README.md b/libcxx/utils/ci/lnt/README.md
index 1b40d562d9391..ed1ecae79857c 100644
--- a/libcxx/utils/ci/lnt/README.md
+++ b/libcxx/utils/ci/lnt/README.md
@@ -62,10 +62,10 @@ it for each machine defined in `machines.json` on a schedule.
## Configuring the benchmark machines
-`machines.json` describes the machines we benchmark on. It is the single source of truth
-for both the workflow that runs the benchmarks (`libcxx-benchmark-commit.yml`) and the cron
-that requests those runs (`libcxx-benchmark-cron.yml`). Each entry contains variables used
-by the various workflows and the LNT machine name that the results will be reported under.
+`machines.json` describes the machines we benchmark on and their configuration. It is the
+single source of truth for all workflows that run benchmarks (PR benchmarking, running
+historical benchmarks, etc). Each entry contains variables used by the various workflows
+and the LNT machine name that the results will be reported under.
## Running benchmarks locally
@@ -77,7 +77,8 @@ which can be used to benchmark locally:
run-benchmarks --test-suite-commit <SHA1> --machine <MACHINE> \
--compiler clang++ --benchmark-commit <SHA2> \
--libcxx-installation <PATH> \
- --output result.json
+ --output result.json \
+ -- --param std=c++26 --param optimization=speed
```
This will run the benchmarks (using the test suite at the specified `SHA1`) against the installation
diff --git a/libcxx/utils/ci/lnt/machines.json b/libcxx/utils/ci/lnt/machines.json
index ca55a7fc3c241..21814b13b4c91 100644
--- a/libcxx/utils/ci/lnt/machines.json
+++ b/libcxx/utils/ci/lnt/machines.json
@@ -6,6 +6,7 @@
"xcode-version": "26.5",
"benchmark-suite-version": "8bb5e216937e6b541f351aa1637c67e85a43ada0",
"cmake-cache": "libcxx/utils/ci/lnt/cmake/generic.cmake",
+ "lit-params": ["std=c++26", "optimization=speed"],
"coverage": {
"since": "2023-01-01",
"every": "week",
@@ -21,6 +22,7 @@
"xcode-version": "26.5",
"benchmark-suite-version": "8bb5e216937e6b541f351aa1637c67e85a43ada0",
"cmake-cache": "libcxx/utils/ci/lnt/cmake/hardened-fast.cmake",
+ "lit-params": ["std=c++26", "optimization=speed"],
"coverage": {
"since": "2023-12-01",
"every": "week",
@@ -35,6 +37,7 @@
"cxx": "clang++-22",
"benchmark-suite-version": "8bb5e216937e6b541f351aa1637c67e85a43ada0",
"cmake-cache": "libcxx/utils/ci/lnt/cmake/generic.cmake",
+ "lit-params": ["std=c++26", "optimization=speed"],
"coverage": {
"since": "2023-01-01",
"every": "week",
diff --git a/libcxx/utils/ci/lnt/run-benchmarks b/libcxx/utils/ci/lnt/run-benchmarks
index 8ecc653c8fb79..d610f83acb55b 100755
--- a/libcxx/utils/ci/lnt/run-benchmarks
+++ b/libcxx/utils/ci/lnt/run-benchmarks
@@ -47,7 +47,7 @@ def gather_machine_information(args):
return info
-def gather_lit_features(lit, lit_params, tests):
+def gather_lit_features(lit, lit_args, tests):
"""
Gather the Lit features in effect for the configuration we benchmarked, as a dictionary of
feature name to feature value.
@@ -64,7 +64,7 @@ def gather_lit_features(lit, lit_params, tests):
return {}
# Lit prints the features of the configuration it discovered and exits without running anything.
- command = [str(c) for c in [lit, '--show-suites', *lit_params, tests]]
+ command = [str(c) for c in [lit, '--show-suites', *lit_args, tests]]
logging.debug(f'$ {" ".join(command)}')
try:
output = subprocess.check_output(command, stderr=subprocess.PIPE).decode()
@@ -122,6 +122,9 @@ def main(argv):
parser.add_argument('--output', type=pathlib.Path, required=True,
help='Path where the resulting LNT JSON report is written. It is an error for the file to '
'already exist.')
+ parser.add_argument('lit_options', nargs=argparse.REMAINDER,
+ help='Optional arguments passed to Lit when running the benchmarks. Should be provided last and '
+ 'separated from other arguments with a `--`.')
parser.add_argument('--filter', type=str, required=False,
help="Optional test filter to pass to lit when running the benchmarks. This allows "
"running only a subset of the benchmarks.")
@@ -139,6 +142,13 @@ def main(argv):
help='Verbosity level: passing the option multiple times increases the level.')
args = parser.parse_args(argv)
+ # Gather Lit options
+ lit_options = []
+ if args.lit_options:
+ if args.lit_options[0] != '--':
+ sys.exit('error: for clarity, Lit options must be separated from other options by --')
+ lit_options = args.lit_options[1:]
+
if args.verbose == 0:
logging.basicConfig(level=logging.INFO)
elif args.verbose >= 1:
@@ -191,13 +201,13 @@ def main(argv):
logging.info(f'Storing artifacts in {artifacts}')
logging.info(f'Running benchmarks from {args.test_suite_commit} against libc++ at {args.libcxx_installation}')
- lit_params = ['--param', f'compiler={args.compiler}',
- '--param', f'libcxx_installation={args.libcxx_installation}',
- '--param', 'optimization=speed',
- '--param', 'std=c++26',
- '--param', 'enable_werror=False'] # older versions of the library trigger new warnings, don't fail
+ # Always add some lit parameters that are required or make sense when running benchmarks.
+ lit_args = ['--param', f'compiler={args.compiler}',
+ '--param', f'libcxx_installation={args.libcxx_installation}',
+ '--param', 'enable_werror=False'] # older versions of the library trigger new warnings, don't fail
if args.spec_dir is not None:
- lit_params += ['--param', f'spec_dir={args.spec_dir}']
+ lit_args += ['--param', f'spec_dir={args.spec_dir}']
+ lit_args += lit_options
cmd = [args.git_repo / 'libcxx/utils/test-at-commit',
'--git-repo', args.git_repo,
'--build-dir', artifacts / 'benchmarks-build',
@@ -207,7 +217,7 @@ def main(argv):
'--compiler', args.compiler,
'--',
'-j1', '--time-tests', '--test-output=failed',
- *lit_params,
+ *lit_args,
artifacts / 'benchmarks-build/libcxx/test/benchmarks']
if args.filter is not None:
cmd += ['--filter', args.filter]
@@ -226,7 +236,7 @@ def main(argv):
for arg in dict_to_params(gather_run_information(args)):
importreport += ['--run-info', arg]
machine_info = {**gather_machine_information(args),
- **gather_lit_features(artifacts / 'benchmarks-build/bin/llvm-lit', lit_params,
+ **gather_lit_features(artifacts / 'benchmarks-build/bin/llvm-lit', lit_args,
artifacts / 'benchmarks-build/libcxx/test/benchmarks')}
for arg in dict_to_params(machine_info):
importreport += ['--machine-info', arg]
diff --git a/libcxx/utils/ci/run-buildbot b/libcxx/utils/ci/run-buildbot
index 02f9a87d01a07..572f326c4fb0a 100755
--- a/libcxx/utils/ci/run-buildbot
+++ b/libcxx/utils/ci/run-buildbot
@@ -343,7 +343,8 @@ EOF
--machine test-tools \
--compiler "${CXX}" \
--filter hash.bench.cpp \
- --output "${BUILD_DIR}/report.json"
+ --output "${BUILD_DIR}/report.json" \
+ -- --param optimization=speed
step "Submit a LNT report (dry-run)"
"${MONOREPO_ROOT}/libcxx/utils/ci/lnt/submit-benchmarks" \
More information about the llvm-commits
mailing list