[libcxx-commits] [libcxx] [llvm] [libc++] Extract running the benchmarks into reusable workflows (PR #232168)
Louis Dionne via libcxx-commits
libcxx-commits at lists.llvm.org
Wed Oct 14 20:10:52 PDT 2026
https://github.com/ldionne created https://github.com/llvm/llvm-project/pull/232168
This patch moves the jobs that actually run the benchmarks into their own workflows and defines the full "benchmarking pipeline" in a dedicated workflow as well. This allows seeing the full set of steps in the pipeline in a concise way, and removes most of the remaining duplication of logic.
>From 609c1cec73327623bb444d215347d27f6ef2f38e Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Wed, 14 Oct 2026 22:42:40 -0400
Subject: [PATCH] [libc++] Extract running the benchmarks into reusable
workflows
This patch moves the jobs that actually run the benchmarks into their
own workflows and defines the full "benchmarking pipeline" in a
dedicated workflow as well. This allows seeing the full set of steps
in the pipeline in a concise way, and removes most of the remaining
duplication of logic.
---
.github/workflows/libcxx-benchmark-cron.yml | 4 +-
.../libcxx-lnt-pipeline-libstdcxx.yml | 73 ++++++++++++++++
.github/workflows/libcxx-lnt-pipeline.yml | 75 ++++++++++++++++
...k-commit.yml => libcxx-run-benchmarks.yml} | 86 ++++++-------------
...ml => libcxx-run-libstdcxx-benchmarks.yml} | 66 ++++----------
libcxx/utils/ci/lnt/README.md | 2 +-
libcxx/utils/ci/lnt/dispatch-benchmarks | 6 +-
7 files changed, 200 insertions(+), 112 deletions(-)
create mode 100644 .github/workflows/libcxx-lnt-pipeline-libstdcxx.yml
create mode 100644 .github/workflows/libcxx-lnt-pipeline.yml
rename .github/workflows/{libcxx-benchmark-commit.yml => libcxx-run-benchmarks.yml} (60%)
rename .github/workflows/{libcxx-benchmark-libstdcxx.yml => libcxx-run-libstdcxx-benchmarks.yml} (59%)
diff --git a/.github/workflows/libcxx-benchmark-cron.yml b/.github/workflows/libcxx-benchmark-cron.yml
index 8d98d3f84b6248..386966604252ab 100644
--- a/.github/workflows/libcxx-benchmark-cron.yml
+++ b/.github/workflows/libcxx-benchmark-cron.yml
@@ -3,7 +3,7 @@
#
# The workflow keeps no state: on every invocation, the commits that should be benchmarked
# are recomputed from Git and the ones that have been benchmarked are recomputed from LNT
-# and from the Github Actions API. It then dispatches libcxx-benchmark-commit.yml for the
+# and from the Github Actions API. It then dispatches libcxx-lnt-pipeline.yml for the
# difference, within a budget. This makes the system converge towards having data for all
# the desired commits, without necessarily ever reaching that state (as new commits land).
#
@@ -64,7 +64,7 @@ jobs:
request-runs:
permissions:
contents: read
- actions: write # to dispatch libcxx-benchmark-commit.yml
+ actions: write # to dispatch libcxx-lnt-pipeline.yml
needs:
- select-machines
strategy:
diff --git a/.github/workflows/libcxx-lnt-pipeline-libstdcxx.yml b/.github/workflows/libcxx-lnt-pipeline-libstdcxx.yml
new file mode 100644
index 00000000000000..4fabb2b7d57b76
--- /dev/null
+++ b/.github/workflows/libcxx-lnt-pipeline-libstdcxx.yml
@@ -0,0 +1,73 @@
+# This file defines a workflow that runs the libc++ benchmarks against the specified version
+# of libstdc++ and (optionally) submits the results to a LNT instance.
+
+name: "[libc++] LNT pipeline (libstdc++)"
+
+run-name: "[libc++] LNT pipeline: libstdc++ ${{ inputs.libstdcxx-version }} on ${{ inputs.lnt-machine }}${{ !inputs.submit-lnt && ' (dry run)' || '' }}"
+
+permissions:
+ contents: read
+
+on:
+ workflow_dispatch:
+ inputs:
+ libstdcxx-version:
+ description: 'The version of libstdc++ to benchmark. This is the major version of GCC being installed to get libstdc++ (e.g. 16).'
+ required: true
+ type: string
+ lnt-machine:
+ description: 'The LNT machine configuration to use when benchmarking and to report results as'
+ required: true
+ type: string
+ benchmark-suite-override:
+ description: |
+ Override the version of the benchmark suite to use (a LLVM monorepo SHA). By default, the version pinned
+ for this machine in machines.json is used. This override can be used to dry-run benchmarks with arbitrary
+ commits of the benchmark suite, but only the version pinned in machines.json can be used when actually
+ submitting to LNT.
+ required: false
+ type: string
+ filter:
+ description: 'An optional filter to determine which benchmarks to run'
+ required: false
+ type: string
+ submit-lnt:
+ description: 'Whether to submit the results to LNT -- dry-run unless opted in'
+ required: false
+ type: boolean
+ default: false
+ lnt-url:
+ description: 'The URL of the LNT instance to submit to'
+ required: false
+ type: string
+ default: https://lnt.llvm.org
+
+jobs:
+ # Validate the inputs and select the configuration of the machine to benchmark on.
+ validate:
+ uses: $/.github/workflows/libcxx-lnt-validate.yml
+ with:
+ lnt-machine: ${{ inputs.lnt-machine }}
+ benchmark-suite-override: ${{ inputs.benchmark-suite-override }}
+ submit-lnt: ${{ inputs.submit-lnt }}
+
+ benchmark:
+ needs:
+ - validate
+ uses: $/.github/workflows/libcxx-run-libstdcxx-benchmarks.yml
+ with:
+ libstdcxx-version: ${{ inputs.libstdcxx-version }}
+ lnt-machine: ${{ inputs.lnt-machine }}
+ setup: ${{ needs.validate.outputs.setup }}
+ filter: ${{ inputs.filter }}
+ artifact-name: lnt-report
+
+ submit:
+ needs:
+ - benchmark
+ if: ${{ inputs.submit-lnt }}
+ uses: $/.github/workflows/libcxx-lnt-submit.yml
+ with:
+ artifact-name: lnt-report
+ test-suite: libstdcxx
+ lnt-url: ${{ inputs.lnt-url }}
diff --git a/.github/workflows/libcxx-lnt-pipeline.yml b/.github/workflows/libcxx-lnt-pipeline.yml
new file mode 100644
index 00000000000000..9b32a7fa491f7f
--- /dev/null
+++ b/.github/workflows/libcxx-lnt-pipeline.yml
@@ -0,0 +1,75 @@
+# This file defines a workflow that runs the libc++ benchmarks at the given commit.
+# This workflow is intended to be triggered manually or via the Github API. As such,
+# it requires several inputs that allow customizing its behavior.
+
+name: "[libc++] LNT pipeline"
+
+# Keep in sync with libcxx/utils/ci/lnt/dispatch-benchmarks
+run-name: "[libc++] LNT pipeline: ${{ inputs.commit }} on ${{ inputs.lnt-machine }}${{ !inputs.submit-lnt && ' (dry run)' || '' }}"
+
+permissions:
+ contents: read
+
+on:
+ workflow_dispatch:
+ inputs:
+ commit:
+ description: 'The libc++ commit to benchmark'
+ required: true
+ type: string
+ lnt-machine:
+ description: 'The LNT machine configuration to use when benchmarking and to report results as'
+ required: true
+ type: string
+ benchmark-suite-override:
+ description: |
+ Override the version of the benchmark suite to use (a LLVM monorepo SHA). By default, the version pinned
+ for this machine in machines.json is used. This override can be used to dry-run benchmarks with arbitrary
+ commits of the benchmark suite, but only the version pinned in machines.json can be used when actually
+ submitting to LNT.
+ required: false
+ type: string
+ filter:
+ description: 'An optional filter to determine which benchmarks to run'
+ required: false
+ type: string
+ submit-lnt:
+ description: 'Whether to submit the results to LNT -- dry-run unless opted in'
+ required: false
+ type: boolean
+ default: false
+ lnt-url:
+ description: 'The URL of the LNT instance to submit to'
+ required: false
+ type: string
+ default: https://lnt.llvm.org
+
+jobs:
+ # Validate the inputs and select the configuration of the machine to benchmark on.
+ validate:
+ uses: $/.github/workflows/libcxx-lnt-validate.yml
+ with:
+ lnt-machine: ${{ inputs.lnt-machine }}
+ benchmark-suite-override: ${{ inputs.benchmark-suite-override }}
+ submit-lnt: ${{ inputs.submit-lnt }}
+
+ benchmark:
+ needs:
+ - validate
+ uses: $/.github/workflows/libcxx-run-benchmarks.yml
+ with:
+ commit: ${{ inputs.commit }}
+ lnt-machine: ${{ inputs.lnt-machine }}
+ setup: ${{ needs.validate.outputs.setup }}
+ filter: ${{ inputs.filter }}
+ artifact-name: lnt-report
+
+ submit:
+ needs:
+ - benchmark
+ if: ${{ inputs.submit-lnt }}
+ uses: $/.github/workflows/libcxx-lnt-submit.yml
+ with:
+ artifact-name: lnt-report
+ test-suite: libcxx
+ lnt-url: ${{ inputs.lnt-url }}
diff --git a/.github/workflows/libcxx-benchmark-commit.yml b/.github/workflows/libcxx-run-benchmarks.yml
similarity index 60%
rename from .github/workflows/libcxx-benchmark-commit.yml
rename to .github/workflows/libcxx-run-benchmarks.yml
index ff9374e401e1bd..aa3d7f3a71d237 100644
--- a/.github/workflows/libcxx-benchmark-commit.yml
+++ b/.github/workflows/libcxx-run-benchmarks.yml
@@ -1,75 +1,51 @@
-# This file defines a workflow that runs the libc++ benchmarks at the given commit.
-# This workflow is intended to be triggered manually or via the Github API. As such,
-# it requires several inputs that allow customizing its behavior.
+# This file defines a reusable workflow that runs the libc++ benchmarks at the given commit, using the
+# given machine setup, and uploads the resulting LNT report as an artifact.
-name: "[libc++] Run benchmark suite against commit"
-
-# Keep in sync with libcxx/utils/ci/lnt/dispatch-benchmarks
-run-name: "[libc++] Run benchmark suite against ${{ inputs.commit }} on ${{ inputs.lnt-machine }}${{ !inputs.submit-lnt && ' (dry run)' || '' }}"
+name: "[libc++] Run benchmarks"
permissions:
contents: read
on:
- workflow_dispatch:
+ workflow_call:
inputs:
commit:
description: 'The libc++ commit to benchmark'
required: true
type: string
lnt-machine:
- description: 'The LNT machine configuration to use when benchmarking and to report results as'
+ description: 'The name of the LNT machine to report the results as'
required: true
type: string
- benchmark-suite-override:
- description: |
- Override the version of the benchmark suite to use (a LLVM monorepo SHA). By default, the version pinned
- for this machine in machines.json is used. This override can be used to dry-run benchmarks with arbitrary
- commits of the benchmark suite, but only the version pinned in machines.json can be used when actually
- submitting to LNT.
- required: false
+ setup:
+ description: 'The setup of the machine to benchmark on, as a JSON object (see libcxx/utils/ci/lnt/README.md)'
+ required: true
type: string
filter:
description: 'An optional filter to determine which benchmarks to run'
required: false
type: string
- submit-lnt:
- description: 'Whether to submit the results to LNT -- dry-run unless opted in'
- required: false
- type: boolean
- default: false
- lnt-url:
- description: 'The URL of the LNT instance to submit to'
- required: false
+ artifact-name:
+ description: 'The name of the artifact to upload the LNT report as'
+ required: true
type: string
- default: https://lnt.llvm.org
jobs:
- # Validate the inputs and select the configuration of the machine to benchmark on.
- validate:
- uses: $/.github/workflows/libcxx-lnt-validate.yml
- with:
- lnt-machine: ${{ inputs.lnt-machine }}
- benchmark-suite-override: ${{ inputs.benchmark-suite-override }}
- submit-lnt: ${{ inputs.submit-lnt }}
-
run-benchmarks:
name: Run benchmarks on ${{ inputs.lnt-machine }}
- needs:
- - validate
- runs-on: ${{ fromJSON(needs.validate.outputs.setup).runner }}
- container: ${{ fromJSON(needs.validate.outputs.setup).container || null }}
+ runs-on: ${{ fromJSON(inputs.setup).runner }}
+ container: ${{ fromJSON(inputs.setup).container || null }}
defaults:
run:
shell: bash
env:
- BENCHMARK_SUITE_VERSION: ${{ fromJSON(needs.validate.outputs.setup).benchmark-suite-version }}
- CMAKE_CACHE: ${{ fromJSON(needs.validate.outputs.setup).cmake-cache }}
- COMPILER: ${{ fromJSON(needs.validate.outputs.setup).cxx }}
- LIT_PARAMS: ${{ join(fromJSON(needs.validate.outputs.setup).lit-params, ' ') }}
+ BENCHMARK_SUITE_VERSION: ${{ fromJSON(inputs.setup).benchmark-suite-version }}
+ CMAKE_CACHE: ${{ fromJSON(inputs.setup).cmake-cache }}
+ COMPILER: ${{ fromJSON(inputs.setup).cxx }}
+ LIT_PARAMS: ${{ join(fromJSON(inputs.setup).lit-params, ' ') }}
LNT_MACHINE: ${{ inputs.lnt-machine }}
- TEST_CONFIG: ${{ fromJSON(needs.validate.outputs.setup).test-config }}
- XCODE_VERSION: ${{ fromJSON(needs.validate.outputs.setup).xcode-version }}
+ TEST_CONFIG: ${{ fromJSON(inputs.setup).test-config }}
+ XCODE_VERSION: ${{ fromJSON(inputs.setup).xcode-version }}
# Where we install the library. This lives inside the workspace so that actions/checkout
# cleans it up between runs on self-hosted runners.
INSTALL_DIR: ${{ github.workspace }}/install
@@ -114,8 +90,9 @@ jobs:
# Build the library.
#
# A build failure is tolerated on purpose, since the library doesn't build at every historical commit.
- # We still submit an empty run for such commits.
+ # We still produce an empty report for such commits.
- name: Build libc++ at ${{ inputs.commit }}
+ id: build
continue-on-error: true
env:
COMMIT: ${{ inputs.commit }}
@@ -135,6 +112,7 @@ jobs:
env:
COMMIT: ${{ inputs.commit }}
FILTER: ${{ inputs.filter }}
+ LIBRARY_BUILD: ${{ steps.build.outcome }}
run: |
source .venv/bin/activate
filter_arg=()
@@ -149,9 +127,11 @@ jobs:
done
# If the library build failed, still create an install directory so we
- # run the benchmarks. The benchmarks will fail, and we'll submit an empty
- # LNT report.
- mkdir -p "${INSTALL_DIR}"
+ # run the benchmarks. The benchmarks will fail, producing an empty report.
+ if [ "${LIBRARY_BUILD}" != "success" ]; then
+ echo "::warning::libc++ failed to build at ${COMMIT}, producing an empty report"
+ mkdir -p "${INSTALL_DIR}"
+ fi
libcxx/utils/ci/lnt/run-benchmarks \
--test-suite-commit "${BENCHMARK_SUITE_VERSION}" \
@@ -168,16 +148,6 @@ jobs:
- name: Upload the LNT report
uses: actions/upload-artifact at cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
- name: lnt-report
+ name: ${{ inputs.artifact-name }}
path: ${{ inputs.commit }}.json
if-no-files-found: error
-
- submit:
- needs:
- - run-benchmarks
- if: ${{ inputs.submit-lnt }}
- uses: $/.github/workflows/libcxx-lnt-submit.yml
- with:
- artifact-name: lnt-report
- test-suite: libcxx
- lnt-url: ${{ inputs.lnt-url }}
diff --git a/.github/workflows/libcxx-benchmark-libstdcxx.yml b/.github/workflows/libcxx-run-libstdcxx-benchmarks.yml
similarity index 59%
rename from .github/workflows/libcxx-benchmark-libstdcxx.yml
rename to .github/workflows/libcxx-run-libstdcxx-benchmarks.yml
index 08a851f53ea4c9..0ba4ee59420c31 100644
--- a/.github/workflows/libcxx-benchmark-libstdcxx.yml
+++ b/.github/workflows/libcxx-run-libstdcxx-benchmarks.yml
@@ -1,70 +1,50 @@
-# This file defines a workflow that runs the libc++ benchmarks against the specified version
-# of libstdc++ and (optionally) submits the results to a LNT instance.
+# This file defines a reusable workflow that runs the libc++ benchmarks against the given version of
+# libstdc++, using the given machine setup, and uploads the resulting LNT report as an artifact.
-name: "[libc++] Run benchmark suite against libstdc++"
+name: "[libc++] Run benchmarks against libstdc++"
permissions:
contents: read
on:
- workflow_dispatch:
+ workflow_call:
inputs:
libstdcxx-version:
description: 'The version of libstdc++ to benchmark. This is the major version of GCC being installed to get libstdc++ (e.g. 16).'
required: true
type: string
lnt-machine:
- description: 'The LNT machine configuration to use when benchmarking and to report results as'
+ description: 'The name of the LNT machine to report the results as'
required: true
type: string
- benchmark-suite-override:
- description: |
- Override the version of the benchmark suite to use (a LLVM monorepo SHA). By default, the version pinned
- for this machine in machines.json is used. This override can be used to dry-run benchmarks with arbitrary
- commits of the benchmark suite, but only the version pinned in machines.json can be used when actually
- submitting to LNT.
- required: false
+ setup:
+ description: 'The setup of the machine to benchmark on, as a JSON object (see libcxx/utils/ci/lnt/README.md)'
+ required: true
type: string
filter:
description: 'An optional filter to determine which benchmarks to run'
required: false
type: string
- submit-lnt:
- description: 'Whether to submit the results to LNT -- dry-run unless opted in'
- required: false
- type: boolean
- default: false
- lnt-url:
- description: 'The URL of the LNT instance to submit to'
- required: false
+ artifact-name:
+ description: 'The name of the artifact to upload the LNT report as'
+ required: true
type: string
- default: https://lnt.llvm.org
jobs:
- # Validate the inputs and select the configuration of the machine to benchmark on.
- validate:
- uses: $/.github/workflows/libcxx-lnt-validate.yml
- with:
- lnt-machine: ${{ inputs.lnt-machine }}
- benchmark-suite-override: ${{ inputs.benchmark-suite-override }}
- submit-lnt: ${{ inputs.submit-lnt }}
-
run-benchmarks:
name: Run benchmarks on ${{ inputs.lnt-machine }}
- needs:
- - validate
- runs-on: ${{ fromJSON(needs.validate.outputs.setup).runner }}
- container: ${{ fromJSON(needs.validate.outputs.setup).container || null }}
+ runs-on: ${{ fromJSON(inputs.setup).runner }}
+ container: ${{ fromJSON(inputs.setup).container || null }}
defaults:
run:
shell: bash
env:
- BENCHMARK_SUITE_VERSION: ${{ fromJSON(needs.validate.outputs.setup).benchmark-suite-version }}
- COMPILER: ${{ fromJSON(needs.validate.outputs.setup).cxx }}
+ BENCHMARK_SUITE_VERSION: ${{ fromJSON(inputs.setup).benchmark-suite-version }}
+ COMPILER: ${{ fromJSON(inputs.setup).cxx }}
LIBSTDCXX_VERSION: ${{ inputs.libstdcxx-version }}
- LIT_PARAMS: ${{ join(fromJSON(needs.validate.outputs.setup).lit-params, ' ') }}
+ LIT_PARAMS: ${{ join(fromJSON(inputs.setup).lit-params, ' ') }}
LNT_MACHINE: ${{ inputs.lnt-machine }}
- XCODE_VERSION: ${{ fromJSON(needs.validate.outputs.setup).xcode-version }}
+ XCODE_VERSION: ${{ fromJSON(inputs.setup).xcode-version }}
steps:
- name: Checkout the LLVM monorepo
uses: actions/checkout at 3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -130,16 +110,6 @@ jobs:
- name: Upload the LNT report
uses: actions/upload-artifact at cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
- name: lnt-report
+ name: ${{ inputs.artifact-name }}
path: result.json
if-no-files-found: error
-
- submit:
- needs:
- - run-benchmarks
- if: ${{ inputs.submit-lnt }}
- uses: $/.github/workflows/libcxx-lnt-submit.yml
- with:
- artifact-name: lnt-report
- test-suite: libstdcxx
- lnt-url: ${{ inputs.lnt-url }}
diff --git a/libcxx/utils/ci/lnt/README.md b/libcxx/utils/ci/lnt/README.md
index 6bf5a181db899a..f846631d79a549 100644
--- a/libcxx/utils/ci/lnt/README.md
+++ b/libcxx/utils/ci/lnt/README.md
@@ -91,7 +91,7 @@ with `/libcxx-bot benchmark`.
## Running benchmarks locally
-On GitHub, the `libcxx-benchmark-commit.yml` workflow is used to run benchmarks and report
+On GitHub, the `libcxx-lnt-pipeline.yml` workflow is used to run benchmarks and report
results to a LNT instance. This workflow wraps the `libcxx/utils/ci/lnt/run-benchmarks` script,
which can be used to benchmark locally:
diff --git a/libcxx/utils/ci/lnt/dispatch-benchmarks b/libcxx/utils/ci/lnt/dispatch-benchmarks
index 93f348fe7be0cd..c1eebfa3548efa 100755
--- a/libcxx/utils/ci/lnt/dispatch-benchmarks
+++ b/libcxx/utils/ci/lnt/dispatch-benchmarks
@@ -53,14 +53,14 @@ class Runs(NamedTuple):
# The repository and the workflow driven by this tool, and the ref to use for the
# definition of the workflow.
REPOSITORY = 'llvm/llvm-project'
-WORKFLOW = 'libcxx-benchmark-commit.yml'
+WORKFLOW = 'libcxx-lnt-pipeline.yml'
WORKFLOW_REF = 'main'
# The name the benchmark workflow gives its runs. What a run is benchmarking can
# only be recovered from its name (the inputs a workflow was dispatched with are
# not available from the API) so this format is a contract with the
-# libcxx-benchmark-commit.yml workflow.
-RUN_NAME_PREFIX = '[libc++] Run benchmark suite against '
+# libcxx-lnt-pipeline.yml workflow.
+RUN_NAME_PREFIX = '[libc++] LNT pipeline: '
RUN_NAME = re.compile(re.escape(RUN_NAME_PREFIX) + r'([0-9a-fA-F]{40}) on (\S+)( \(dry run\))?$')
More information about the libcxx-commits
mailing list