[libcxx-commits] [libcxx] [llvm] [libc++][CI] Add a cron job to trigger benchmarking jobs (PR #212859)
via libcxx-commits
libcxx-commits at lists.llvm.org
Mon Aug 10 06:30:28 PDT 2026
llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-github-workflow
Author: Louis Dionne (ldionne)
<details>
<summary>Changes</summary>
This patch introduces a GitHub workflow that runs on a schedule and
determines which benchmarking jobs to trigger to fill the LNT instances
with performance data.
As a drive-by, it also consolidates the information describing libc++
LNT machines into a single JSON file.
The added cron workflow will run every hour, but since the benchmarks
typically take more than an hour to run, it is expected that some of
those triggers will not actually trigger new jobs.
---
Full diff: https://github.com/llvm/llvm-project/pull/212859.diff
5 Files Affected:
- (modified) .github/workflows/libcxx-benchmark-commit.yml (+13-23)
- (added) .github/workflows/libcxx-benchmark-cron.yml (+132)
- (modified) .github/workflows/libcxx-pr-benchmark.yml (+17-13)
- (modified) libcxx/utils/ci/lnt/README.md (+10)
- (added) libcxx/utils/ci/lnt/machines.json (+31)
``````````diff
diff --git a/.github/workflows/libcxx-benchmark-commit.yml b/.github/workflows/libcxx-benchmark-commit.yml
index 80614796d4269..f55c11b0d93c6 100644
--- a/.github/workflows/libcxx-benchmark-commit.yml
+++ b/.github/workflows/libcxx-benchmark-commit.yml
@@ -20,15 +20,11 @@ on:
lnt-machine:
description: 'The LNT machine to run the benchmarks on'
required: true
- type: choice
- options:
- - macos-26.5-arm64
- - linux-x86_64
+ type: string
benchmark-suite-version:
description: 'The version of the benchmark suite to use (a LLVM monorepo SHA)'
- required: false
+ required: true
type: string
- default: 0eefb2682bf8c04954c46e91916b5164d8424702
filter:
description: 'An optional filter to determine which benchmarks to run'
required: false
@@ -51,6 +47,14 @@ jobs:
outputs:
matrix: ${{ steps.select.outputs.matrix }}
steps:
+ - name: Checkout the machine definitions
+ uses: actions/checkout at df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+ with:
+ persist-credentials: false
+ # Disabling cone mode allows checking out exactly the single file we need.
+ sparse-checkout: libcxx/utils/ci/lnt/machines.json
+ sparse-checkout-cone-mode: false
+
- name: Select the configuration to benchmark on
id: select
uses: actions/github-script at 3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
@@ -58,26 +62,12 @@ jobs:
LNT_MACHINE: ${{ inputs.lnt-machine }}
with:
script: |
- const MACHINES = [
- {
- 'lnt-machine': 'macos-26.5-arm64',
- runner: ['self-hosted', 'macOS', 'ARM64', '26', '26.5'],
- cxx: 'clang++',
- 'running-on': 'macos',
- 'xcode-version': '26.5',
- },
- {
- 'lnt-machine': 'linux-x86_64',
- runner: 'llvm-premerge-libcxx-runners',
- cxx: 'clang++-22',
- 'running-on': 'linux',
- },
- ];
+ const config = JSON.parse(require('fs').readFileSync('libcxx/utils/ci/lnt/machines.json', 'utf8'));
const requested = process.env.LNT_MACHINE;
- const selected = MACHINES.filter(m => m['lnt-machine'] === requested);
+ const selected = config.filter(cfg => cfg['lnt-machine'] === requested);
if (selected.length === 0) {
- const known = MACHINES.map(m => m['lnt-machine']).join(', ');
+ const known = config.map(cfg => cfg['lnt-machine']).join(', ');
core.setFailed(`Unknown LNT machine '${requested}' (known machines: ${known})`);
return;
}
diff --git a/.github/workflows/libcxx-benchmark-cron.yml b/.github/workflows/libcxx-benchmark-cron.yml
new file mode 100644
index 0000000000000..c71ab8112d1d8
--- /dev/null
+++ b/.github/workflows/libcxx-benchmark-cron.yml
@@ -0,0 +1,132 @@
+# This file defines a workflow that periodically requests benchmark runs for the commits
+# that should have historical performance data but don't yet.
+#
+# The workflow keeps no state: on every invocation, the commits that should be benchmarked
+# are recomputed from Git and the ones that have been benchmarked are recomputed from LNT
+# and from the Github Actions API. It then dispatches libcxx-benchmark-commit.yml for the
+# difference, within a budget. This makes the system converge towards having data for all
+# the desired commits, without necessarily ever reaching that state (as new commits land).
+#
+# The actual configuration driving this job is in libcxx/utils/ci/lnt/machines.json.
+
+name: "[libc++] Request historical benchmark data"
+
+permissions:
+ contents: read
+ actions: write # to dispatch libcxx-benchmark-commit.yml
+
+on:
+ schedule:
+ # Trigger every hour, off the hour boundary to avoid popular times.
+ - cron: '37 * * * *'
+ workflow_dispatch:
+ inputs:
+ dry-run:
+ description: 'Report what would be requested, but request nothing'
+ required: false
+ type: boolean
+ default: false
+ allow-missing-machine:
+ description: 'Plan for machines that have no data in LNT yet. Needed to seed a new machine.'
+ required: false
+ type: boolean
+ default: false
+
+concurrency:
+ group: ${{ github.workflow }}
+ cancel-in-progress: false
+
+jobs:
+ # Turn the machine definitions stored in libcxx/utils/ci/lnt/machines.json into actual
+ # dispatch jobs.
+ select-machines:
+ if: github.repository_owner == 'llvm'
+ runs-on: ubuntu-24.04
+ outputs:
+ matrix: ${{ steps.select.outputs.matrix }}
+ steps:
+ - name: Checkout the machine definitions
+ uses: actions/checkout at df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+ with:
+ persist-credentials: false
+ # Disabling cone mode allows checking out exactly the single file we need.
+ sparse-checkout: libcxx/utils/ci/lnt/machines.json
+ sparse-checkout-cone-mode: false
+
+ - name: Select the machines to request runs for
+ id: select
+ uses: actions/github-script at 3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
+ with:
+ script: |
+ const config = JSON.parse(require('fs').readFileSync('libcxx/utils/ci/lnt/machines.json', 'utf8'));
+ core.setOutput('matrix', JSON.stringify(config.map(cfg => ({
+ machine: cfg['lnt-machine'],
+ ...cfg.coverage,
+ }))));
+
+ request-runs:
+ needs:
+ - select-machines
+ strategy:
+ matrix:
+ include: ${{ fromJSON(needs.select-machines.outputs.matrix) }}
+ fail-fast: false
+ name: ${{ matrix.machine }}
+ runs-on: ubuntu-24.04
+ timeout-minutes: 30
+ env:
+ MACHINE: ${{ matrix.machine }}
+ SINCE: ${{ matrix.since }}
+ EVERY: ${{ matrix.every }}
+ SAMPLES: ${{ matrix.samples }}
+ MAX_IN_FLIGHT: ${{ matrix.max-in-flight }}
+ BENCHMARK_SUITE_VERSION: ${{ matrix.benchmark-suite-version }}
+ LNT_URL: ${{ matrix.lnt-url }}
+
+ ALLOW_MISSING_MACHINE: ${{ inputs.allow-missing-machine || 'false' }}
+ DRY_RUN: ${{ inputs.dry-run || 'false' }}
+ steps:
+ - name: Checkout the LLVM monorepo
+ uses: actions/checkout at df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+ with:
+ persist-credentials: false
+ # Selecting anchor commits requires full Git history, but not the blob content.
+ fetch-depth: 0
+ filter: blob:none
+
+ - name: Install Python
+ uses: actions/setup-python at ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+ with:
+ python-version: '3.14'
+ cache: pip
+ cache-dependency-path: libcxx/utils/requirements.txt
+
+ - name: Install dependencies
+ run: pip install -r libcxx/utils/requirements.txt
+
+ - name: Determine which commits should have benchmark data
+ run: libcxx/utils/ci/lnt/select-anchor-commits --since "${SINCE}" --every "${EVERY}" --output anchors.txt
+
+ - name: Determine which of them are missing from LNT
+ run: |
+ allow_missing=()
+ if [ "${ALLOW_MISSING_MACHINE}" = "true" ]; then
+ allow_missing=(--allow-missing-machine)
+ fi
+
+ libcxx/utils/ci/lnt/plan-benchmarks --commit-list anchors.txt --lnt-url "${LNT_URL}" --test-suite libcxx \
+ --machine "${MACHINE}" --samples "${SAMPLES}" "${allow_missing[@]}" \
+ --output plan.jsonl
+
+ - name: Request the missing runs
+ env:
+ GH_TOKEN: ${{ github.token }}
+ run: |
+ dry_run=()
+ if [ "${DRY_RUN}" = "true" ]; then
+ dry_run=(--dry-run)
+ fi
+
+ libcxx/utils/ci/lnt/dispatch-benchmarks --work-items plan.jsonl --test-suite-commit "${BENCHMARK_SUITE_VERSION}" \
+ --max-in-flight "${MAX_IN_FLIGHT}" --lnt-url "${LNT_URL}" \
+ --output dispatched.jsonl "${dry_run[@]}"
diff --git a/.github/workflows/libcxx-pr-benchmark.yml b/.github/workflows/libcxx-pr-benchmark.yml
index 59e9a511ba5e0..9b5bdb5adb834 100644
--- a/.github/workflows/libcxx-pr-benchmark.yml
+++ b/.github/workflows/libcxx-pr-benchmark.yml
@@ -48,6 +48,14 @@ jobs:
permissions:
pull-requests: write
steps:
+ - name: Checkout the LNT configurations
+ uses: actions/checkout at df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+ with:
+ persist-credentials: false
+ # Disabling cone mode allows checking out exactly the single file we need.
+ sparse-checkout: libcxx/utils/ci/lnt/machines.json
+ sparse-checkout-cone-mode: false
+
- name: Extract information from the PR
id: vars
uses: actions/github-script at 3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
@@ -63,6 +71,10 @@ jobs:
const match = context.payload.comment.body.match(/\/libcxx-bot benchmark (.+)/);
core.setOutput('benchmarks', match ? match[1] : '');
+ // Benchmark on the same configurations that we track performance on.
+ const config = JSON.parse(require('fs').readFileSync('libcxx/utils/ci/lnt/machines.json', 'utf8'));
+ core.setOutput('matrix', JSON.stringify(config.map(({coverage, ...machine}) => machine)));
+
- name: Update comment with link to the run
uses: actions/github-script at 3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
@@ -84,20 +96,12 @@ jobs:
pr_base: ${{ steps.vars.outputs.pr_base }}
pr_head: ${{ steps.vars.outputs.pr_head }}
benchmarks: ${{ steps.vars.outputs.benchmarks }}
+ matrix: ${{ steps.vars.outputs.matrix }}
run-benchmarks:
strategy:
matrix:
- include:
- - platform: macOS 26.5 arm64
- runner: ["self-hosted", "macOS", "ARM64", "26", "26.5"]
- cxx: clang++
- running-on: macos
- xcode-version: '26.5'
- - platform: Linux x86_64
- runner: llvm-premerge-libcxx-runners
- cxx: clang++-22
- running-on: linux
+ include: ${{ fromJSON(needs.extract-info.outputs.matrix) }}
fail-fast: false
permissions:
pull-requests: write
@@ -107,7 +111,7 @@ jobs:
env:
BENCHMARKS: ${{ needs.extract-info.outputs.benchmarks }}
COMPILER: ${{ matrix.cxx }}
- PLATFORM: ${{ matrix.platform }}
+ LNT_MACHINE: ${{ matrix.lnt-machine }}
PR_HEAD: ${{ needs.extract-info.outputs.pr_head }}
PR_BASE: ${{ needs.extract-info.outputs.pr_base }}
steps:
@@ -186,7 +190,7 @@ jobs:
const details = [
'<details>',
'<summary>',
- `Benchmark results for ${process.env.PLATFORM}:`,
+ `Benchmark results for ${process.env.LNT_MACHINE}:`,
'</summary>',
'',
'```',
@@ -213,7 +217,7 @@ jobs:
comment_id: context.payload.comment.id,
});
const run_url = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
- const note = `> _:x: Benchmarks for ${process.env.PLATFORM} failed. See ${run_url} for details._`;
+ const note = `> _:x: Benchmarks for ${process.env.LNT_MACHINE} failed. See ${run_url} for details._`;
await github.rest.issues.updateComment({
owner: context.repo.owner,
repo: context.repo.repo,
diff --git a/libcxx/utils/ci/lnt/README.md b/libcxx/utils/ci/lnt/README.md
index d51715d435587..f3345d343d2d6 100644
--- a/libcxx/utils/ci/lnt/README.md
+++ b/libcxx/utils/ci/lnt/README.md
@@ -58,6 +58,16 @@ Note that since `dispatch-benchmarks` both reads and creates workflow runs, it r
GitHub token with write access to the repository. That token can either be passed as an
argument or picked up from the `GITHUB_TOKEN` environment variable.
+In production, this pipeline is run by the `libcxx-benchmark-cron.yml` workflow, which runs
+it for each machine defined in `machines.json` on a schedule.
+
+## Configuring the benchmark machines
+
+`machines.json` describes the machines we benchmark on. It is the single source of truth
+for both the workflow that runs the benchmarks (`libcxx-benchmark-commit.yml`) and the cron
+that requests those runs (`libcxx-benchmark-cron.yml`). Each entry contains variables used
+by the various workflows and the LNT machine name that the results will be reported under.
+
## Running benchmarks locally
On GitHub, the `libcxx-benchmark-commit.yml` workflow is used to run benchmarks and report
diff --git a/libcxx/utils/ci/lnt/machines.json b/libcxx/utils/ci/lnt/machines.json
new file mode 100644
index 0000000000000..4b9321a21330a
--- /dev/null
+++ b/libcxx/utils/ci/lnt/machines.json
@@ -0,0 +1,31 @@
+[
+ {
+ "lnt-machine": "macos-26.5-arm64",
+ "runner": ["self-hosted", "macOS", "ARM64", "26", "26.5"],
+ "cxx": "clang++",
+ "running-on": "macos",
+ "xcode-version": "26.5",
+ "coverage": {
+ "benchmark-suite-version": "810062e9771d8b47c25a89311139c621c13bbf9f",
+ "since": "2025-01-01",
+ "every": "week",
+ "samples": 3,
+ "max-in-flight": 4,
+ "lnt-url": "http://lnt.llvm.org"
+ }
+ },
+ {
+ "lnt-machine": "linux-x86_64",
+ "runner": "llvm-premerge-libcxx-runners",
+ "cxx": "clang++-22",
+ "running-on": "linux",
+ "coverage": {
+ "benchmark-suite-version": "810062e9771d8b47c25a89311139c621c13bbf9f",
+ "since": "2025-01-01",
+ "every": "week",
+ "samples": 3,
+ "max-in-flight": 8,
+ "lnt-url": "http://lnt.llvm.org"
+ }
+ }
+]
``````````
</details>
https://github.com/llvm/llvm-project/pull/212859
More information about the libcxx-commits
mailing list