[libcxx] [llvm] [libc++] Use a self-contained job to run libstdc++ benchmarks (PR #226985)
Louis Dionne via llvm-commits
llvm-commits at lists.llvm.org
Mon Sep 28 06:49:32 PDT 2026
https://github.com/ldionne created https://github.com/llvm/llvm-project/pull/226985
After trying to retrofit libstdc++ benchmarking into our existing libc++ pipeline, I am giving up and setting up a self-contained job to do it instead. It's much simpler as we can just run exactly what we need instead of fighting to decouple libc++ specific logic from the libc++ benchmarking pipeline.
Various parts can still be extracted (e.g. gathering hardware information for LNT reports, reading machines.json, etc) and will be extracted in future iterations.
Also, add the corresponding libstdcxx LNT test suite schema so we can start submitting results to it.
>From 0c31418817b2a4bad6454968d0d9d98eb657d827 Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Fri, 25 Sep 2026 16:18:22 -0400
Subject: [PATCH] [libc++] Use a self-contained job to run libstdc++ benchmarks
After trying to retrofit libstdc++ benchmarking into our existing
libc++ pipeline, I am giving up and setting up a self-contained job
to do it instead. It's much simpler as we can just run exactly what
we need instead of fighting to decouple libc++ specific logic from
the libc++ benchmarking pipeline.
Various parts can still be extracted (e.g. gathering hardware information
for LNT reports, reading machines.json, etc) and will be extracted in
future iterations.
Also, add the corresponding libstdcxx LNT test suite schema so we can
start submitting results to it.
---
.github/workflows/libcxx-benchmark-commit.yml | 8 +-
.../workflows/libcxx-benchmark-libstdcxx.yml | 182 ++++++++++++++++++
libcxx/utils/ci/lnt/README.md | 2 +-
libcxx/utils/ci/lnt/machines.json | 6 +-
libcxx/utils/ci/lnt/run-libstdcxx-benchmarks | 115 +++++++++++
.../lnt/{schema.yaml => schemas/libcxx.yaml} | 0
libcxx/utils/ci/lnt/schemas/libstdcxx.yaml | 43 +++++
7 files changed, 345 insertions(+), 11 deletions(-)
create mode 100644 .github/workflows/libcxx-benchmark-libstdcxx.yml
create mode 100755 libcxx/utils/ci/lnt/run-libstdcxx-benchmarks
rename libcxx/utils/ci/lnt/{schema.yaml => schemas/libcxx.yaml} (100%)
create mode 100644 libcxx/utils/ci/lnt/schemas/libstdcxx.yaml
diff --git a/.github/workflows/libcxx-benchmark-commit.yml b/.github/workflows/libcxx-benchmark-commit.yml
index 4c62087f2eede..17b3f612e46ab 100644
--- a/.github/workflows/libcxx-benchmark-commit.yml
+++ b/.github/workflows/libcxx-benchmark-commit.yml
@@ -18,7 +18,7 @@ on:
required: true
type: string
lnt-machine:
- description: 'The LNT machine to run the benchmarks on'
+ description: 'The LNT machine configuration to use when benchmarking and to report results as'
required: true
type: string
benchmark-suite-override:
@@ -131,13 +131,9 @@ jobs:
- name: Install dependencies via Homebrew
if: runner.os == 'macOS'
- env:
- # Extra packages needed by the test configuration
- EXTRA_BREW_PACKAGES: ${{ join(matrix.brew-packages, ' ') }}
run: |
- read -ra extra_packages <<< "${EXTRA_BREW_PACKAGES}"
brew update
- brew install ninja cmake python at 3.14 "${extra_packages[@]}"
+ brew install ninja cmake python at 3.14
echo "$(brew --prefix python at 3.14)/bin" >> "$GITHUB_PATH"
- name: Diagnose tools in use
diff --git a/.github/workflows/libcxx-benchmark-libstdcxx.yml b/.github/workflows/libcxx-benchmark-libstdcxx.yml
new file mode 100644
index 0000000000000..c8e2735c50c02
--- /dev/null
+++ b/.github/workflows/libcxx-benchmark-libstdcxx.yml
@@ -0,0 +1,182 @@
+# This file defines a workflow that runs the libc++ benchmarks against the specified version
+# of libstdc++ and (optionally) submits the results to a LNT instance.
+
+name: "[libc++] Run benchmark suite against libstdc++"
+
+permissions:
+ contents: read
+
+on:
+ workflow_dispatch:
+ inputs:
+ libstdcxx-version:
+ description: 'The version of libstdc++ to benchmark. This is the version of GCC being installed to get libstdc++.'
+ required: true
+ type: string
+ lnt-machine:
+ description: 'The LNT machine configuration to use when benchmarking and to report results as'
+ required: true
+ type: string
+ benchmark-suite-override:
+ description: |
+ Override the version of the benchmark suite to use (a LLVM monorepo SHA). By default, the version pinned
+ for this machine in machines.json is used. This override can be used to dry-run benchmarks with arbitrary
+ commits of the benchmark suite, but only the version pinned in machines.json can be used when actually
+ submitting to LNT.
+ required: false
+ type: string
+ filter:
+ description: 'An optional filter to determine which benchmarks to run'
+ required: false
+ type: string
+ submit-lnt:
+ description: 'Whether to submit the results to LNT -- dry-run unless opted in'
+ required: false
+ type: boolean
+ default: false
+ lnt-url:
+ description: 'The URL of the LNT instance to submit to'
+ required: false
+ type: string
+ default: https://lnt.llvm.org
+
+jobs:
+ # Determine which configuration to run based on the `lnt-machine` input.
+ select-machine:
+ runs-on: ubuntu-26.04
+ outputs:
+ matrix: ${{ steps.select.outputs.matrix }}
+ steps:
+ - name: Checkout the machine definitions
+ uses: actions/checkout at df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+ with:
+ persist-credentials: false
+ # Disabling cone mode allows checking out exactly the single file we need.
+ sparse-checkout: libcxx/utils/ci/lnt/machines.json
+ sparse-checkout-cone-mode: false
+
+ - name: Select the configuration to benchmark on
+ id: select
+ uses: actions/github-script at 3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
+ env:
+ LNT_MACHINE: ${{ inputs.lnt-machine }}
+ BENCHMARK_SUITE_OVERRIDE: ${{ inputs.benchmark-suite-override }}
+ SUBMIT_LNT: ${{ inputs.submit-lnt }}
+ with:
+ script: |
+ const config = JSON.parse(require('fs').readFileSync('libcxx/utils/ci/lnt/machines.json', 'utf8'));
+
+ const requested = process.env.LNT_MACHINE;
+ const selected = config.filter(cfg => cfg['lnt-machine'] === requested);
+ if (selected.length === 0) {
+ const known = config.map(cfg => cfg['lnt-machine']).join(', ');
+ core.setFailed(`Unknown LNT machine '${requested}' (known machines: ${known})`);
+ return;
+ }
+
+ // Honor benchmark suite version override and make sure we don't submit if an incorrect
+ // override is provided.
+ const version_override = (process.env.BENCHMARK_SUITE_OVERRIDE || '').trim().toLowerCase();
+ for (const cfg of selected) {
+ const pinned = cfg['benchmark-suite-version'];
+ if (process.env.SUBMIT_LNT === 'true' && version_override && version_override !== pinned) {
+ core.setFailed(`Refusing to submit results for ${cfg['lnt-machine']}, since the benchmark suite was `
+ + `overridden to version ${version_override}, which is different from the version `
+ + `pinned in machines.json (${pinned}).`);
+ return;
+ }
+
+ if (version_override) {
+ cfg['benchmark-suite-version'] = version_override;
+ }
+ }
+
+ core.setOutput('matrix', JSON.stringify(selected));
+
+ run-benchmarks:
+ needs:
+ - select-machine
+ strategy:
+ matrix:
+ include: ${{ fromJSON(needs.select-machine.outputs.matrix) }}
+ fail-fast: false
+ runs-on: ${{ matrix.runner }}
+ env:
+ COMPILER: ${{ matrix.cxx }}
+ LIBSTDCXX_VERSION: ${{ inputs.libstdcxx-version }}
+ steps:
+ - name: Checkout the LLVM monorepo
+ uses: actions/checkout at df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+ with:
+ persist-credentials: false
+ # The benchmark suite is checked out at the version pinned in machines.json, which requires the
+ # full history to be available.
+ fetch-depth: 0
+ fetch-tags: true
+
+ - name: Install Python
+ if: runner.os == 'Linux' # installed via Homebrew on macOS
+ uses: actions/setup-python at ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+ with:
+ python-version: '3.14'
+
+ - name: Select Xcode
+ if: runner.os == 'macOS'
+ run: echo "DEVELOPER_DIR=/Applications/Xcode_${{ matrix.xcode-version }}.app/Contents/Developer" >> $GITHUB_ENV
+
+ - name: Install dependencies via Homebrew
+ if: runner.os == 'macOS'
+ run: |
+ brew update
+ brew install ninja cmake python at 3.14 gcc@${LIBSTDCXX_VERSION}
+ echo "$(brew --prefix python at 3.14)/bin" >> "$GITHUB_PATH"
+
+ - name: Diagnose tools in use
+ run: |
+ cmake --version
+ ninja --version
+ "${COMPILER}" --version
+ python3 --version
+
+ - name: Setup virtual environment
+ run: |
+ python3 -m venv .venv
+ source .venv/bin/activate
+ pip install -r libcxx/utils/ci/lnt/requirements.txt
+
+ - name: Run the benchmarks
+ env:
+ BENCHMARK_SUITE_VERSION: ${{ matrix.benchmark-suite-version }}
+ FILTER: ${{ inputs.filter }}
+ LIT_PARAMS: ${{ join(matrix.lit-params, ' ') }}
+ LNT_MACHINE: ${{ matrix.lnt-machine }}
+ run: |
+ read -ra configured_params <<< "${LIT_PARAMS}"
+ lit_args=()
+ for param in "${configured_params[@]}"; do
+ lit_args+=(--param "${param}")
+ done
+ if [ -n "${FILTER}" ]; then
+ lit_args+=(--filter "${FILTER}")
+ fi
+
+ libcxx/utils/ci/lnt/run-libstdcxx-benchmarks \
+ --gcc "g++-${LIBSTDCXX_VERSION}" \
+ --compiler "${COMPILER}" \
+ --test-suite-commit "${BENCHMARK_SUITE_VERSION}" \
+ --machine "${LNT_MACHINE}" \
+ --output result.json \
+ -- "${lit_args[@]}"
+
+ cat result.json
+
+ - name: Submit to LNT
+ if: ${{ inputs.submit-lnt }}
+ env:
+ LNT_URL: ${{ inputs.lnt-url }}
+ run: |
+ source .venv/bin/activate
+ libcxx/utils/ci/lnt/submit-benchmarks \
+ --lnt-url "${LNT_URL}" \
+ --test-suite libstdcxx \
+ result.json
diff --git a/libcxx/utils/ci/lnt/README.md b/libcxx/utils/ci/lnt/README.md
index ff5dcc0cce9a9..a60fef83f506d 100644
--- a/libcxx/utils/ci/lnt/README.md
+++ b/libcxx/utils/ci/lnt/README.md
@@ -115,7 +115,7 @@ lnt_url: "http://localhost:8000"
database: default
auth_token: example_token
EOF
-lnt admin --config lnt-admin-config.yaml --testsuite libcxx test-suite add libcxx/utils/ci/lnt/schema.yaml
+lnt admin --config lnt-admin-config.yaml --testsuite libcxx test-suite add libcxx/utils/ci/lnt/schemas/libcxx.yaml
# Then submit to the local instance
submit-benchmarks --lnt-url http://localhost:8000 --test-suite libcxx result.json
diff --git a/libcxx/utils/ci/lnt/machines.json b/libcxx/utils/ci/lnt/machines.json
index 187cfec16d755..32a5ca0802e98 100644
--- a/libcxx/utils/ci/lnt/machines.json
+++ b/libcxx/utils/ci/lnt/machines.json
@@ -56,13 +56,11 @@
}
},
{
- "lnt-machine": "macos-26.6.2-arm64-libstdcxx16-20260925",
+ "lnt-machine": "macos-26.6.2-arm64-libstdcxx-20260928",
"runner": ["self-hosted", "macOS", "26.6.2", "ARM64", "apple-runners"],
"cxx": "clang++",
"xcode-version": "26.6",
"benchmark-suite-version": "4ebb19efe952ded7f9d937731418baf3b28babd0",
- "brew-packages": ["gcc at 16"],
- "lit-params": ["std=c++26", "optimization=speed", "libstdcxx_compiler=g++-16"],
- "test-config": "stdlib-libstdc++.cfg.in"
+ "lit-params": ["std=c++26", "optimization=speed"]
}
]
diff --git a/libcxx/utils/ci/lnt/run-libstdcxx-benchmarks b/libcxx/utils/ci/lnt/run-libstdcxx-benchmarks
new file mode 100755
index 0000000000000..b9f230b516023
--- /dev/null
+++ b/libcxx/utils/ci/lnt/run-libstdcxx-benchmarks
@@ -0,0 +1,115 @@
+#!/usr/bin/env python3
+# ===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ===----------------------------------------------------------------------===##
+
+import argparse
+import datetime
+import json
+import pathlib
+import platform
+import subprocess
+import sys
+import tempfile
+
+
+MONOREPO_ROOT = pathlib.Path(__file__).resolve().parents[4]
+
+def gather_machine_information(args):
+ """
+ Gather the machine information to upload to LNT as part of the submission.
+ """
+ info = {'name': args.machine}
+ if platform.system() == 'Darwin':
+ profiler_info = json.loads(subprocess.check_output(['system_profiler', 'SPHardwareDataType', 'SPSoftwareDataType', '-json']).decode())
+ info['hardware'] = profiler_info['SPHardwareDataType'][0]['chip_type']
+ info['os'] = profiler_info['SPSoftwareDataType'][0]['os_version']
+ info['sdk'] = subprocess.check_output(['xcrun', '--show-sdk-version']).decode().strip()
+
+ info['compiler'] = subprocess.check_output([args.compiler, '--version']).decode().strip().splitlines()[0]
+ info['test_suite_commit'] = subprocess.check_output(['git', '-C', MONOREPO_ROOT, 'rev-parse', args.test_suite_commit]).decode().strip()
+ return info
+
+def parse_lnt_results(results):
+ """
+ Parse `<test>.<metric> <value>` lines (as produced by consolidate-benchmarks) into the list of tests
+ expected in a LNT JSON report.
+ """
+ tests = {}
+ for line in results.splitlines():
+ if not line.strip():
+ continue
+ (key, value) = line.split()
+ (name, metric) = key.rsplit('.', 1)
+ tests.setdefault(name, {}).setdefault(metric, []).append(float(value))
+ return [{'name': name, **{m: v if len(v) > 1 else v[0] for (m, v) in metrics.items()}}
+ for (name, metrics) in tests.items()]
+
+def main(argv):
+ parser = argparse.ArgumentParser(
+ prog='run-libstdcxx-benchmarks',
+ description='Run the libc++ benchmark suite against the libstdc++ of the given GCC and produce a LNT JSON '
+ 'report for the libstdcxx LNT test suite. The results are attributed to the version of that GCC.')
+ parser.add_argument('--gcc', type=str, required=True,
+ help='The GCC whose libstdc++ is benchmarked, e.g. g++-16. This is not the compiler used to build the benchmarks.')
+ parser.add_argument('--compiler', type=str, required=True,
+ help='The compiler used to build the benchmarks.')
+ parser.add_argument('--test-suite-commit', type=str, required=True,
+ help='The SHA representing the version of the test suite to use for benchmarking.')
+ parser.add_argument('--machine', type=str, required=True,
+ help='The name of the machine for reporting LNT results.')
+ parser.add_argument('--output', type=pathlib.Path, required=True,
+ help='Path where the resulting LNT JSON report is written.')
+ parser.add_argument('lit_options', nargs=argparse.REMAINDER,
+ help='Optional arguments passed to Lit when running the benchmarks (e.g. --filter). Should be provided '
+ 'last and separated from other arguments with a `--`.')
+ args = parser.parse_args(argv)
+
+ lit_options = []
+ if args.lit_options:
+ if args.lit_options[0] != '--':
+ sys.exit('error: for clarity, Lit options must be separated from other options by --')
+ lit_options = args.lit_options[1:]
+
+ with tempfile.TemporaryDirectory() as build_dir:
+ build_dir = pathlib.Path(build_dir)
+ start_time = datetime.datetime.now(datetime.timezone.utc)
+ # Some benchmarks may fail to build or run, and that's okay: we report the ones that succeeded.
+ subprocess.run([MONOREPO_ROOT / 'libcxx/utils/test-at-commit',
+ '--git-repo', MONOREPO_ROOT,
+ '--build-dir', build_dir,
+ '--test-suite-commit', args.test_suite_commit,
+ '--test-config', 'stdlib-libstdc++.cfg.in',
+ '--compiler', args.compiler,
+ '--',
+ '-j1', '--test-output=failed',
+ '--param', f'compiler={args.compiler}',
+ '--param', f'libstdcxx_compiler={args.gcc}',
+ *lit_options,
+ build_dir / 'libcxx/test/benchmarks'])
+ end_time = datetime.datetime.now(datetime.timezone.utc)
+
+ results = subprocess.check_output([MONOREPO_ROOT / 'libcxx/utils/consolidate-benchmarks', build_dir]).decode()
+ tests = parse_lnt_results(results)
+
+ report = {
+ 'format_version': '2',
+ 'machine': gather_machine_information(args),
+ 'run': {
+ # This is the version of libstdc++, but LNT hardcodes the name of the order field (see schemas/libstdcxx.yaml).
+ 'llvm_project_revision': subprocess.check_output([args.gcc, '-dumpfullversion']).decode().strip(),
+ 'start_time': start_time.strftime('%Y-%m-%d %H:%M:%S'),
+ 'end_time': end_time.strftime('%Y-%m-%d %H:%M:%S'),
+ },
+ 'tests': tests,
+ }
+ with open(args.output, 'w') as f:
+ json.dump(report, f, indent=4, sort_keys=True)
+
+
+if __name__ == '__main__':
+ main(sys.argv[1:])
diff --git a/libcxx/utils/ci/lnt/schema.yaml b/libcxx/utils/ci/lnt/schemas/libcxx.yaml
similarity index 100%
rename from libcxx/utils/ci/lnt/schema.yaml
rename to libcxx/utils/ci/lnt/schemas/libcxx.yaml
diff --git a/libcxx/utils/ci/lnt/schemas/libstdcxx.yaml b/libcxx/utils/ci/lnt/schemas/libstdcxx.yaml
new file mode 100644
index 0000000000000..16437377cb578
--- /dev/null
+++ b/libcxx/utils/ci/lnt/schemas/libstdcxx.yaml
@@ -0,0 +1,43 @@
+# Schema definition for the libc++ benchmark suite when running it against libstdc++ and
+# submitting to a LNT instance. The metrics in this schema correspond to the metrics
+# gathered by libc++'s benchmark suite.
+format_version: '2'
+name: libstdcxx
+metrics:
+- name: execution_time
+ type: Real
+ display_name: Execution Time
+ unit: seconds
+ unit_abbrev: s
+- name: instructions
+ type: Real
+ display_name: Instructions Retired
+ unit: instructions
+ unit_abbrev: instr
+- name: max_rss
+ type: Real
+ display_name: Maximum Resident Set Size
+ unit: megabytes
+ unit_abbrev: mb
+- name: cycles
+ type: Real
+ display_name: Cycles Elapsed
+ unit: cycles
+ unit_abbrev: cycles
+- name: peak_memory
+ type: Real
+ display_name: Peak Memory Footprint
+ unit: megabytes
+ unit_abbrev: mb
+run_fields:
+# This is the version of libstdc++ being benchmarked (e.g. 16.2.0), not a LLVM revision. However,
+# LNT currently hardcodes the name of the order field as `llvm_project_revision` in several places,
+# so we must use that name.
+- name: llvm_project_revision
+ order: true
+machine_fields:
+- name: hardware
+- name: os
+- name: test_suite_commit
+- name: compiler
+- name: sdk
More information about the llvm-commits
mailing list