[libcxx-commits] [libcxx] [llvm] [libc++] Add tools for gathering historical benchmark data (PR #212775)
via libcxx-commits
libcxx-commits at lists.llvm.org
Wed Jul 29 06:51:45 PDT 2026
llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-github-workflow
Author: Louis Dionne (ldionne)
<details>
<summary>Changes</summary>
Benchmarking every commit of libc++ is prohibitively expensive: a single run of the benchmark suite takes hours, and the data has to be regenerated from scratch whenever the compiler, the OS or the benchmark machines change. These tools instead sample the history at a coarse granularity and drive libcxx-benchmark-commit.yml to fill in what is missing.
Three tools cooperate, meant to be run periodically:
select-anchor-commits picks one commit per calendar bucket from Git
plan-benchmarks diffs that against what LNT already holds
dispatch-benchmarks requests the corresponding workflow runs
They keep no state of their own. They recompute the current and target states from LNT and the GitHub Actions API, which allows running them in a CRON. The dispatching of workflows is done using a budget, to avoid launching tens of jobs and competing with other uses of the CI resources.
The first pass of these tools was assisted by Claude, but I reviewed and tweaked everything that needed it.
---
Patch is 48.97 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/212775.diff
5 Files Affected:
- (modified) .github/workflows/libcxx-benchmark-commit.yml (+2)
- (modified) libcxx/utils/ci/lnt/README.md (+60-15)
- (added) libcxx/utils/ci/lnt/dispatch-benchmarks (+483)
- (added) libcxx/utils/ci/lnt/plan-benchmarks (+228)
- (added) libcxx/utils/ci/lnt/select-anchor-commits (+211)
``````````diff
diff --git a/.github/workflows/libcxx-benchmark-commit.yml b/.github/workflows/libcxx-benchmark-commit.yml
index d2ad03a180cf5..37112bbc822a9 100644
--- a/.github/workflows/libcxx-benchmark-commit.yml
+++ b/.github/workflows/libcxx-benchmark-commit.yml
@@ -3,6 +3,8 @@
# it requires several inputs that allow customizing its behavior.
name: "[libc++] Run benchmark suite against commit"
+
+# Keep in sync with libcxx/utils/ci/lnt/dispatch-benchmarks
run-name: "[libc++] Run benchmark suite against ${{ inputs.commit }} on ${{ inputs.lnt-machine }}"
permissions:
diff --git a/libcxx/utils/ci/lnt/README.md b/libcxx/utils/ci/lnt/README.md
index ac5cbc11f4c40..d724877e4d82e 100644
--- a/libcxx/utils/ci/lnt/README.md
+++ b/libcxx/utils/ci/lnt/README.md
@@ -1,29 +1,74 @@
+# LNT tools for libc++ performance tracking
+
This directory contains utilities for continuous benchmarking of libc++ with LNT.
-This can be done locally using a local instance, or using a public instance like http://lnt.llvm.org.
-## Running a benchmark bot
+## Gathering historical performance data
+
+When generating historical performance data, benchmarking every commit of libc++
+is prohibitively expensive since a single run of the benchmark suite takes a few
+hours. Furthermore, generating this data from scratch is expected to be common,
+since it must happen whenever a fixed parameter like the compiler or the OS changes.
+
+Instead, the tools in this directory aim to make it possible to generate historical
+performance data quickly with coarse granularity, with the goal of then generating
+finer granularity performance data based on coarse granularity observations (e.g.
+finding regressions between two distant commits). At this time, triggering finer
+granularity data points is done manually.
+
+Coarse grained performance data is obtained by determining "anchor commits", which
+are libc++ commits that fall at specific intervals (e.g. the first commit of every
+week). These anchor commits remain stable through time: they don't change as new
+commits are introduced. This makes it possible to re-generate performance data for
+the same anchor commits with a different configuration, and to compare across
+configurations.
-The `run-benchbot` script is the main entry point for running benchmarks. That script
-is where libc++'s pre-defined LNT bot configurations are defined. To benchmark specific
-commits:
+For coarse grained performance data, the system is meant to be run periodically on
+a schedule. It keeps no state of its own: on every invocation, what should be measured
+is recomputed from Git and what has been measured is recomputed from LNT and from the
+Github Actions API. This makes the overall system converge towards a state where all
+the desired commits have been benchmarked, without necessarily ever reaching it (as
+new desired commits are added).
+
+To achieve this, three tools work together:
```
-libcxx/utils/ci/lnt/run-benchbot --llvm-root <monorepo> <builder> -- <commit1> <commit2> ...
+# What should have benchmark data: one commit per week since 2023.
+select-anchor-commits --since 2023-01-02 --every week > anchor-commits.txt
+
+# What is missing from LNT (we want at least 3 samples for each commit).
+plan-benchmarks --commit-list anchor-commits.txt \
+ --lnt-url http://lnt.llvm.org --test-suite libcxx \
+ --machine <machine> --samples 3 > plan.jsonl
+
+# Request the corresponding workflow runs, at most 4 at a time to be a good citizen.
+dispatch-benchmarks --work-items plan.jsonl --test-suite-commit <benchmark suite SHA> \
+ --max-in-flight 4 --dry-run
```
-Results are stored as LNT JSON files in `<llvm-root>/build/<builder>/` by default.
-Use `--results-dir <dir>` to override where these reports are written.
+In a nutshell, `select-anchor-commits` produces the list of anchor commits that we want
+data for. `plan-benchmarks` then looks at which commits we actually already have data for
+in the LNT instance and produces a plan of what runs we need to trigger in order to
+get data for the missing commits, taking into account the number of samples we want for
+each commit. Finally, `dispatch-benchmarks` interprets this plan and actually dispatches
+the Github workflows based on a budget, taking into account currently running workflows
+and previously failed runs, if any (to avoid requesting runs that fail indefinitely).
-By default, build artifacts are stored in a temporary directory and discarded after
-each run. Pass `--build-dir <dir>` to keep them; artifacts for each run are then stored
-under `<dir>/<builder>/<commit>`.
+## Running benchmarks locally
-To continuously poll for un-benchmarked commits and submit results to a LNT instance:
+On GitHub, the `libcxx-benchmark-commit.yml` workflow is used to run benchmarks and report
+results to a LNT instance. This workflow wraps the `libcxx/utils/ci/lnt/run-benchmarks` script,
+which can be used to benchmark locally:
```
-libcxx/utils/ci/lnt/run-benchbot --llvm-root <monorepo> --lnt-url http://lnt.llvm.org <builder>
+run-benchmarks --test-suite-commit <SHA1> --machine <MACHINE> \
+ --compiler clang++ --benchmark-commit <SHA2> \
+ --output result.json
```
+This will run the benchmarks (using the test suite at the specified `SHA1`) against libc++
+as-of the specified `SHA2`, and produce a LNT-ready JSON report. The results can then be
+submitted to a LNT instance if desired.
+
## Setting up a local LNT instance
```
@@ -40,6 +85,6 @@ auth_token: example_token
EOF
lnt admin --config lnt-admin-config.yaml --testsuite libcxx test-suite add libcxx/utils/ci/lnt/schema.yaml
-# Then run the benchbot against the local instance
-libcxx/utils/ci/lnt/run-benchbot --llvm-root <monorepo> --lnt-url http://localhost:8000 <builder>
+# Then submit to the local instance
+submit-benchmarks --lnt-url http://localhost:8000 --test-suite libcxx result.json
```
diff --git a/libcxx/utils/ci/lnt/dispatch-benchmarks b/libcxx/utils/ci/lnt/dispatch-benchmarks
new file mode 100755
index 0000000000000..b031959dfde88
--- /dev/null
+++ b/libcxx/utils/ci/lnt/dispatch-benchmarks
@@ -0,0 +1,483 @@
+#!/usr/bin/env python3
+# ===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ===----------------------------------------------------------------------===##
+
+from typing import Dict, List, NamedTuple, Optional, Sequence, TextIO
+import argparse
+import json
+import logging
+import pathlib
+import re
+import subprocess
+import sys
+import tabulate
+import urllib.parse
+
+
+class WorkItem(NamedTuple):
+ """
+ One line of the plan as produced by `plan-benchmarks`.
+ """
+ commit: str
+ machine: str
+ samples_have: int
+ samples_want: int
+ reason: str = ''
+
+ @staticmethod
+ def parse(line: str) -> 'WorkItem':
+ """
+ Read one line of a plan, validating every field.
+
+ A plan is normally produced by `plan-benchmarks`, but it can also be
+ hand-written, so we are careful about validating the data here.
+ """
+ item = json.loads(line)
+ if not isinstance(item, dict):
+ raise ValueError(f'expected a JSON object, got {type(item).__name__}')
+
+ def field(name: str, types, required: bool = True, default=None):
+ if name not in item:
+ if required:
+ raise ValueError(f'missing required field {name}')
+ return default
+ value = item[name]
+ if isinstance(value, bool) or not isinstance(value, types):
+ raise ValueError(f'{name} has the wrong type: {value!r}')
+ return value
+
+ commit = field('commit', str)
+ if not is_sha(commit):
+ raise ValueError(f'commit is not a full 40-character SHA: {commit!r}')
+ machine = field('machine', str)
+ if not machine:
+ raise ValueError('machine is empty')
+ samples_have = field('samples_have', int)
+ samples_want = field('samples_want', int)
+ if samples_have < 0:
+ raise ValueError(f'samples_have is negative: {samples_have}')
+ if samples_want <= samples_have:
+ raise ValueError(f'samples_want {samples_want} is not more than samples_have '
+ f'{samples_have}, so there is nothing to request')
+ return WorkItem(commit=commit.lower(),
+ machine=machine,
+ samples_have=samples_have,
+ samples_want=samples_want,
+ reason=field('reason', str, required=False, default=''))
+
+ @property
+ def target(self) -> 'Target':
+ """The unit of work this item is about."""
+ return Target(self.commit, self.machine)
+
+ @property
+ def runs_to_request(self) -> int:
+ """How many more runs this item is asking for."""
+ return self.samples_want - self.samples_have
+
+
+class Target(NamedTuple):
+ """
+ A unit of work: one commit benchmarked on one machine.
+
+ Plans, workflow runs and retry budgets are all keyed by this pair.
+ """
+ commit: str
+ machine: str
+
+
+class WorkflowRun(NamedTuple):
+ """A run of the benchmark workflow, as the Github API describes it."""
+ identifier: int
+ target: Target
+ finished: bool
+
+
+class Decision(NamedTuple):
+ """What to do with one work item, and why."""
+ item: WorkItem
+ action: str # 'dispatch', 'skip' or 'defer'
+ jobs: int # how many runs to request now
+ reason: str
+
+
+class Runs(NamedTuple):
+ """What the workflow is currently doing, counted per target."""
+ in_flight: Dict[Target, int]
+ completed: Dict[Target, int]
+ ids: Dict[Target, List[int]] # in-flight run ids, for the summary
+
+ def in_flight_on(self, machine: str) -> int:
+ """How many runs are in flight on a machine, whatever they are benchmarking."""
+ return sum(n for (target, n) in self.in_flight.items() if target.machine == machine)
+
+
+# The name the benchmark workflow gives its runs. What a run is benchmarking can
+# only be recovered from its name (the inputs a workflow was dispatched with are
+# not available from the API) so this format is a contract with the
+# libcxx-benchmark-commit.yml workflow.
+RUN_NAME_PREFIX = '[libc++] Run benchmark suite against '
+RUN_NAME = re.compile(re.escape(RUN_NAME_PREFIX) + r'([0-9a-fA-F]{40}) on (\S+)$')
+
+
+class HelpFormatter(argparse.ArgumentDefaultsHelpFormatter,
+ argparse.RawDescriptionHelpFormatter):
+ """Show defaults, but keep the paragraphs of the description intact."""
+
+def is_sha(string: str) -> bool:
+ return len(string) == 40 and all(c in '0123456789abcdef' for c in string.lower())
+
+def positive(string: str) -> int:
+ value = int(string)
+ if value < 1:
+ raise argparse.ArgumentTypeError(f'expected a positive integer, got {string}')
+ return value
+
+def non_negative(string: str) -> int:
+ value = int(string)
+ if value < 0:
+ raise argparse.ArgumentTypeError(f'expected a non-negative integer, got {string}')
+ return value
+
+class GithubError(Exception):
+ pass
+
+
+def parse_plan(stream: TextIO) -> List[WorkItem]:
+ """Parse a plan into a list of work items, in the order they were written."""
+ items: List[WorkItem] = []
+ for (number, line) in enumerate(stream, start=1):
+ line = line.strip()
+ if not line:
+ continue
+ try:
+ items.append(WorkItem.parse(line))
+ except ValueError as error:
+ raise ValueError(f'line {number} of the plan is not a valid work item: {error}')
+ return items
+
+
+def collapse(items: Sequence[WorkItem]) -> List[WorkItem]:
+ """
+ Keep one item per commit and machine, the one asking for the most runs.
+
+ A plan can be concatenated from multiple sources or handwritten, so we
+ de-duplicate commits against the same machine to avoid duplicate work.
+ """
+ best: Dict[Target, WorkItem] = {}
+ for item in items:
+ existing = best.get(item.target)
+ if existing is None or item.runs_to_request > existing.runs_to_request:
+ best[item.target] = item
+ # Preserve the order of the plan: it is the order the work should be done in.
+ seen = set()
+ ordered: List[WorkItem] = []
+ for item in items:
+ if item.target not in seen:
+ seen.add(item.target)
+ ordered.append(best[item.target])
+ return ordered
+
+
+def target_of_run(title: str) -> Optional[Target]:
+ """
+ Return what a workflow run is benchmarking, or None if it can't be told.
+
+ The name has to match the workflow's `run-name` exactly.
+ """
+ match = RUN_NAME.match(title.strip())
+ return Target(match.group(1).lower(), match.group(2)) if match else None
+
+
+def run_gh(args: Sequence[str], stdin: Optional[str] = None) -> str:
+ command = ['gh', *args]
+ logging.debug(f'$ {" ".join(command)}')
+ try:
+ result = subprocess.run(command, check=True, text=True, input=stdin,
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ except FileNotFoundError:
+ raise GithubError('gh was not found. This tool needs the Github CLI to be installed.')
+ except subprocess.CalledProcessError as error:
+ raise GithubError(f'{" ".join(command)} failed with status {error.returncode}: '
+ f'{error.stderr.strip()}')
+ return result.stdout
+
+
+def workflow_runs(repo: str, workflow: str) -> List[WorkflowRun]:
+ """
+ Return every run of a workflow whose name says what it benchmarks.
+
+ The whole history is walked, however Github only retains runs for 90 days.
+ Runs are extracted without filtering on status.
+
+ Cancelled runs or runs named in any other format are ignored: they either
+ predate this tooling or don't represent expended capacity towards benchmarking
+ a target.
+ """
+ query = urllib.parse.urlencode({'per_page': 100})
+ # The projection also flattens paginated output into one object per line;
+ # without one, `--paginate` emits a whole JSON document per page.
+ output = run_gh(['api', '--paginate', '-q',
+ '.workflow_runs[] | {id, display_title, status, conclusion}',
+ f'/repos/{repo}/actions/workflows/{workflow}/runs?{query}'])
+ runs: Dict[int, WorkflowRun] = {}
+ for line in output.splitlines():
+ line = line.strip()
+ if not line:
+ continue
+ try:
+ data = json.loads(line)
+ except json.JSONDecodeError as error:
+ raise GithubError(f'could not parse a workflow run from gh output: {error}: {line}')
+ if 'id' not in data:
+ raise GithubError(f'a workflow run came back without an id: {line}')
+ target = target_of_run(data.get('display_title') or '')
+ if target is None or data.get('conclusion') == 'cancelled':
+ continue
+ # Deduplicate by id: pages can overlap while runs are being created.
+ runs[data['id']] = WorkflowRun(identifier=data['id'], target=target,
+ finished=(data.get('status') or '') == 'completed')
+ return list(runs.values())
+
+
+def summarize(runs: Sequence[WorkflowRun]) -> Runs:
+ """Count what the workflow is doing, per target."""
+ in_flight: Dict[Target, int] = {}
+ completed: Dict[Target, int] = {}
+ ids: Dict[Target, List[int]] = {}
+ for run in runs:
+ if run.finished:
+ completed[run.target] = completed.get(run.target, 0) + 1
+ else:
+ in_flight[run.target] = in_flight.get(run.target, 0) + 1
+ ids.setdefault(run.target, []).append(run.identifier)
+ return Runs(in_flight=in_flight, completed=completed, ids=ids)
+
+
+def decide(items: Sequence[WorkItem],
+ runs: Runs,
+ max_in_flight: int,
+ extra_attempts: int) -> List[Decision]:
+ """
+ Return a list of Decisions describing what to do with each item.
+
+ Items are considered in the order the plan gave them, which is the order the
+ work should be done in.
+
+ Two budgets are enforced. Capacity: no more than `max_in_flight` runs may exist
+ at once on a machine, including queued runs since they occupy the queue just as
+ much as currently running jobs. Per machine rather than overall because the
+ machines are separate pools of runners.
+
+ Attempts: a target whose runs keep completing without the results reaching LNT
+ (e.g. cancelled, timed out, or failing before the submission step) would
+ otherwise risk being requested forever, since the planner only sees what reached
+ LNT. That budget is per machine too, and runs already in flight count against it.
+ """
+ remaining = {machine: max(0, max_in_flight - runs.in_flight_on(machine))
+ for machine in {item.machine for item in items}}
+ decisions: List[Decision] = []
+
+ def in_flight_detail(target: Target, running: int) -> str:
+ ids = ', '.join(str(i) for i in runs.ids.get(target, [])[:3])
+ return f'{running} already in flight' + (f' (run {ids})' if ids else '')
+
+ for item in items:
+ target = item.target
+ running = runs.in_flight.get(target, 0)
+ finished = runs.completed.get(target, 0)
+ budget = item.samples_want + extra_attempts
+ # Only a target whose budget is spent with nothing left running is really abandoned.
+ if finished + running >= budget and running == 0:
+ reason = f'giving up: {item.samples_have}/{item.samples_want} runs recorded in LNT after {finished} completed runs, budget is {budget}'
+ decisions.append(Decision(item, action='skip', jobs=0, reason=reason))
+ continue
+ pending = min(item.runs_to_request - running, budget - finished - running)
+ if pending <= 0:
+ decisions.append(Decision(item, action='skip', jobs=0,
+ reason=in_flight_detail(target, running)))
+ continue
+ if remaining[item.machine] == 0:
+ reason = 'no capacity'
+ if running:
+ reason = f'{reason}, {in_flight_detail(target, running)}'
+ decisions.append(Decision(item, action='defer', jobs=0, reason=reason))
+ continue
+ dispatching = min(pending, remaining[item.machine])
+ remaining[item.machine] -= dispatching
+ deferred = pending - dispatching
+ reason = item.reason
+ if deferred:
+ reason = f'{reason} ({deferred} more deferred, no capacity)'
+ decisions.append(Decision(item, action='dispatch', jobs=dispatching, reason=reason))
+ return decisions
+
+
+def workflow_inputs(args: argparse.Namespace, item: WorkItem) -> Dict[str, str]:
+ """Return the inputs to dispatch the workflow with for a work item."""
+ inputs = {'commit': item.commit,
+ 'lnt-machine': item.machine,
+ 'benchmark-suite-version': args.test_suite_commit,
+ # Submission is not optional: a run whose results are not recorded
+ # anywhere would be requested again on every future invocation.
+ # During a dry run nothing runs, so nothing is submitted either.
+ 'submit-lnt': 'false' if args.dry_run else 'true'}
+ if args.lnt_url:
+ inputs['lnt-url'] = args.lnt_url
+ return inputs
+
+
+def report(args: argparse.Namespace, decisions: Sequence[Decision], runs: Runs) -> None:
+ """Print a human-readable summary of what is about to be done."""
+ out = sys.stderr
+ print(f'Submitting to: {args.lnt_url or "whatever the workflow defaults to"}', file=out)
+ # One line per machine the plan is about, since capacity is per machine.
+ for machine in sorted({d.item.machine for d in decisions}):
+ in_flight = runs.in_flight_on(machine)
+ available = max(0, args.max_in_flight - in_flight)
+ print(f'Capacity on {machi...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/212775
More information about the libcxx-commits
mailing list