[libcxx-commits] [libcxx] [llvm] [libc++] Add tools for gathering historical benchmark data (PR #212775)
Louis Dionne via libcxx-commits
libcxx-commits at lists.llvm.org
Wed Jul 29 06:50:57 PDT 2026
https://github.com/ldionne created https://github.com/llvm/llvm-project/pull/212775
Benchmarking every commit of libc++ is prohibitively expensive: a single run of the benchmark suite takes hours, and the data has to be regenerated from scratch whenever the compiler, the OS or the benchmark machines change. These tools instead sample the history at a coarse granularity and drive libcxx-benchmark-commit.yml to fill in what is missing.
Three tools cooperate, meant to be run periodically:
select-anchor-commits picks one commit per calendar bucket from Git
plan-benchmarks diffs that against what LNT already holds
dispatch-benchmarks requests the corresponding workflow runs
They keep no state of their own. They recompute the current and target states from LNT and the GitHub Actions API, which allows running them in a CRON. The dispatching of workflows is done using a budget, to avoid launching tens of jobs and competing with other uses of the CI resources.
The first pass of these tools was assisted by Claude, but I reviewed and tweaked everything that needed it.
>From 72c6583248d2ee3312de35c87fef3fa2ac9ff468 Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Wed, 29 Jul 2026 08:31:17 -0400
Subject: [PATCH] [libc++] Add tools for gathering historical benchmark data
Benchmarking every commit of libc++ is prohibitively expensive: a single run
of the benchmark suite takes hours, and the data has to be regenerated from
scratch whenever the compiler, the OS or the benchmark machines change. These
tools instead sample the history at a coarse granularity and drive
libcxx-benchmark-commit.yml to fill in what is missing.
Three tools cooperate, meant to be run periodically:
select-anchor-commits picks one commit per calendar bucket from Git
plan-benchmarks diffs that against what LNT already holds
dispatch-benchmarks requests the corresponding workflow runs
They keep no state of their own. They recompute the current and target
states from LNT and the GitHub Actions API, which allows running them
in a CRON. The dispatching of workflows is done using a budget, to
avoid launching tens of jobs and competing with other uses of the CI
resources.
The first pass of these tools was assisted by Claude, but I reviewed
and tweaked everything that needed it.
---
.github/workflows/libcxx-benchmark-commit.yml | 2 +
libcxx/utils/ci/lnt/README.md | 75 ++-
libcxx/utils/ci/lnt/dispatch-benchmarks | 483 ++++++++++++++++++
libcxx/utils/ci/lnt/plan-benchmarks | 228 +++++++++
libcxx/utils/ci/lnt/select-anchor-commits | 211 ++++++++
5 files changed, 984 insertions(+), 15 deletions(-)
create mode 100755 libcxx/utils/ci/lnt/dispatch-benchmarks
create mode 100755 libcxx/utils/ci/lnt/plan-benchmarks
create mode 100755 libcxx/utils/ci/lnt/select-anchor-commits
diff --git a/.github/workflows/libcxx-benchmark-commit.yml b/.github/workflows/libcxx-benchmark-commit.yml
index d2ad03a180cf5..37112bbc822a9 100644
--- a/.github/workflows/libcxx-benchmark-commit.yml
+++ b/.github/workflows/libcxx-benchmark-commit.yml
@@ -3,6 +3,8 @@
# it requires several inputs that allow customizing its behavior.
name: "[libc++] Run benchmark suite against commit"
+
+# Keep in sync with libcxx/utils/ci/lnt/dispatch-benchmarks
run-name: "[libc++] Run benchmark suite against ${{ inputs.commit }} on ${{ inputs.lnt-machine }}"
permissions:
diff --git a/libcxx/utils/ci/lnt/README.md b/libcxx/utils/ci/lnt/README.md
index ac5cbc11f4c40..d724877e4d82e 100644
--- a/libcxx/utils/ci/lnt/README.md
+++ b/libcxx/utils/ci/lnt/README.md
@@ -1,29 +1,74 @@
+# LNT tools for libc++ performance tracking
+
This directory contains utilities for continuous benchmarking of libc++ with LNT.
-This can be done locally using a local instance, or using a public instance like http://lnt.llvm.org.
-## Running a benchmark bot
+## Gathering historical performance data
+
+When generating historical performance data, benchmarking every commit of libc++
+is prohibitively expensive since a single run of the benchmark suite takes a few
+hours. Furthermore, generating this data from scratch is expected to be common,
+since it must happen whenever a fixed parameter like the compiler or the OS changes.
+
+Instead, the tools in this directory aim to make it possible to generate historical
+performance data quickly with coarse granularity, with the goal of then generating
+finer granularity performance data based on coarse granularity observations (e.g.
+finding regressions between two distant commits). At this time, triggering finer
+granularity data points is done manually.
+
+Coarse grained performance data is obtained by determining "anchor commits", which
+are libc++ commits that fall at specific intervals (e.g. the first commit of every
+week). These anchor commits remain stable through time: they don't change as new
+commits are introduced. This makes it possible to re-generate performance data for
+the same anchor commits with a different configuration, and to compare across
+configurations.
-The `run-benchbot` script is the main entry point for running benchmarks. That script
-is where libc++'s pre-defined LNT bot configurations are defined. To benchmark specific
-commits:
+For coarse grained performance data, the system is meant to be run periodically on
+a schedule. It keeps no state of its own: on every invocation, what should be measured
+is recomputed from Git and what has been measured is recomputed from LNT and from the
+Github Actions API. This makes the overall system converge towards a state where all
+the desired commits have been benchmarked, without necessarily ever reaching it (as
+new desired commits are added).
+
+To achieve this, three tools work together:
```
-libcxx/utils/ci/lnt/run-benchbot --llvm-root <monorepo> <builder> -- <commit1> <commit2> ...
+# What should have benchmark data: one commit per week since 2023.
+select-anchor-commits --since 2023-01-02 --every week > anchor-commits.txt
+
+# What is missing from LNT (we want at least 3 samples for each commit).
+plan-benchmarks --commit-list anchor-commits.txt \
+ --lnt-url http://lnt.llvm.org --test-suite libcxx \
+ --machine <machine> --samples 3 > plan.jsonl
+
+# Request the corresponding workflow runs, at most 4 at a time to be a good citizen.
+dispatch-benchmarks --work-items plan.jsonl --test-suite-commit <benchmark suite SHA> \
+ --max-in-flight 4 --dry-run
```
-Results are stored as LNT JSON files in `<llvm-root>/build/<builder>/` by default.
-Use `--results-dir <dir>` to override where these reports are written.
+In a nutshell, `select-anchor-commits` produces the list of anchor commits that we want
+data for. `plan-benchmarks` then looks at which commits we actually already have data for
+in the LNT instance and produces a plan of what runs we need to trigger in order to
+get data for the missing commits, taking into account the number of samples we want for
+each commit. Finally, `dispatch-benchmarks` interprets this plan and actually dispatches
+the Github workflows based on a budget, taking into account currently running workflows
+and previously failed runs, if any (to avoid requesting runs that fail indefinitely).
-By default, build artifacts are stored in a temporary directory and discarded after
-each run. Pass `--build-dir <dir>` to keep them; artifacts for each run are then stored
-under `<dir>/<builder>/<commit>`.
+## Running benchmarks locally
-To continuously poll for un-benchmarked commits and submit results to a LNT instance:
+On GitHub, the `libcxx-benchmark-commit.yml` workflow is used to run benchmarks and report
+results to a LNT instance. This workflow wraps the `libcxx/utils/ci/lnt/run-benchmarks` script,
+which can be used to benchmark locally:
```
-libcxx/utils/ci/lnt/run-benchbot --llvm-root <monorepo> --lnt-url http://lnt.llvm.org <builder>
+run-benchmarks --test-suite-commit <SHA1> --machine <MACHINE> \
+ --compiler clang++ --benchmark-commit <SHA2> \
+ --output result.json
```
+This will run the benchmarks (using the test suite at the specified `SHA1`) against libc++
+as-of the specified `SHA2`, and produce a LNT-ready JSON report. The results can then be
+submitted to a LNT instance if desired.
+
## Setting up a local LNT instance
```
@@ -40,6 +85,6 @@ auth_token: example_token
EOF
lnt admin --config lnt-admin-config.yaml --testsuite libcxx test-suite add libcxx/utils/ci/lnt/schema.yaml
-# Then run the benchbot against the local instance
-libcxx/utils/ci/lnt/run-benchbot --llvm-root <monorepo> --lnt-url http://localhost:8000 <builder>
+# Then submit to the local instance
+submit-benchmarks --lnt-url http://localhost:8000 --test-suite libcxx result.json
```
diff --git a/libcxx/utils/ci/lnt/dispatch-benchmarks b/libcxx/utils/ci/lnt/dispatch-benchmarks
new file mode 100755
index 0000000000000..b031959dfde88
--- /dev/null
+++ b/libcxx/utils/ci/lnt/dispatch-benchmarks
@@ -0,0 +1,483 @@
+#!/usr/bin/env python3
+# ===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ===----------------------------------------------------------------------===##
+
+from typing import Dict, List, NamedTuple, Optional, Sequence, TextIO
+import argparse
+import json
+import logging
+import pathlib
+import re
+import subprocess
+import sys
+import tabulate
+import urllib.parse
+
+
+class WorkItem(NamedTuple):
+ """
+ One line of the plan as produced by `plan-benchmarks`.
+ """
+ commit: str
+ machine: str
+ samples_have: int
+ samples_want: int
+ reason: str = ''
+
+ @staticmethod
+ def parse(line: str) -> 'WorkItem':
+ """
+ Read one line of a plan, validating every field.
+
+ A plan is normally produced by `plan-benchmarks`, but it can also be
+ hand-written, so we are careful about validating the data here.
+ """
+ item = json.loads(line)
+ if not isinstance(item, dict):
+ raise ValueError(f'expected a JSON object, got {type(item).__name__}')
+
+ def field(name: str, types, required: bool = True, default=None):
+ if name not in item:
+ if required:
+ raise ValueError(f'missing required field {name}')
+ return default
+ value = item[name]
+ if isinstance(value, bool) or not isinstance(value, types):
+ raise ValueError(f'{name} has the wrong type: {value!r}')
+ return value
+
+ commit = field('commit', str)
+ if not is_sha(commit):
+ raise ValueError(f'commit is not a full 40-character SHA: {commit!r}')
+ machine = field('machine', str)
+ if not machine:
+ raise ValueError('machine is empty')
+ samples_have = field('samples_have', int)
+ samples_want = field('samples_want', int)
+ if samples_have < 0:
+ raise ValueError(f'samples_have is negative: {samples_have}')
+ if samples_want <= samples_have:
+ raise ValueError(f'samples_want {samples_want} is not more than samples_have '
+ f'{samples_have}, so there is nothing to request')
+ return WorkItem(commit=commit.lower(),
+ machine=machine,
+ samples_have=samples_have,
+ samples_want=samples_want,
+ reason=field('reason', str, required=False, default=''))
+
+ @property
+ def target(self) -> 'Target':
+ """The unit of work this item is about."""
+ return Target(self.commit, self.machine)
+
+ @property
+ def runs_to_request(self) -> int:
+ """How many more runs this item is asking for."""
+ return self.samples_want - self.samples_have
+
+
+class Target(NamedTuple):
+ """
+ A unit of work: one commit benchmarked on one machine.
+
+ Plans, workflow runs and retry budgets are all keyed by this pair.
+ """
+ commit: str
+ machine: str
+
+
+class WorkflowRun(NamedTuple):
+ """A run of the benchmark workflow, as the Github API describes it."""
+ identifier: int
+ target: Target
+ finished: bool
+
+
+class Decision(NamedTuple):
+ """What to do with one work item, and why."""
+ item: WorkItem
+ action: str # 'dispatch', 'skip' or 'defer'
+ jobs: int # how many runs to request now
+ reason: str
+
+
+class Runs(NamedTuple):
+ """What the workflow is currently doing, counted per target."""
+ in_flight: Dict[Target, int]
+ completed: Dict[Target, int]
+ ids: Dict[Target, List[int]] # in-flight run ids, for the summary
+
+ def in_flight_on(self, machine: str) -> int:
+ """How many runs are in flight on a machine, whatever they are benchmarking."""
+ return sum(n for (target, n) in self.in_flight.items() if target.machine == machine)
+
+
+# The name the benchmark workflow gives its runs. What a run is benchmarking can
+# only be recovered from its name (the inputs a workflow was dispatched with are
+# not available from the API) so this format is a contract with the
+# libcxx-benchmark-commit.yml workflow.
+RUN_NAME_PREFIX = '[libc++] Run benchmark suite against '
+RUN_NAME = re.compile(re.escape(RUN_NAME_PREFIX) + r'([0-9a-fA-F]{40}) on (\S+)$')
+
+
+class HelpFormatter(argparse.ArgumentDefaultsHelpFormatter,
+ argparse.RawDescriptionHelpFormatter):
+ """Show defaults, but keep the paragraphs of the description intact."""
+
+def is_sha(string: str) -> bool:
+ return len(string) == 40 and all(c in '0123456789abcdef' for c in string.lower())
+
+def positive(string: str) -> int:
+ value = int(string)
+ if value < 1:
+ raise argparse.ArgumentTypeError(f'expected a positive integer, got {string}')
+ return value
+
+def non_negative(string: str) -> int:
+ value = int(string)
+ if value < 0:
+ raise argparse.ArgumentTypeError(f'expected a non-negative integer, got {string}')
+ return value
+
+class GithubError(Exception):
+ pass
+
+
+def parse_plan(stream: TextIO) -> List[WorkItem]:
+ """Parse a plan into a list of work items, in the order they were written."""
+ items: List[WorkItem] = []
+ for (number, line) in enumerate(stream, start=1):
+ line = line.strip()
+ if not line:
+ continue
+ try:
+ items.append(WorkItem.parse(line))
+ except ValueError as error:
+ raise ValueError(f'line {number} of the plan is not a valid work item: {error}')
+ return items
+
+
+def collapse(items: Sequence[WorkItem]) -> List[WorkItem]:
+ """
+ Keep one item per commit and machine, the one asking for the most runs.
+
+ A plan can be concatenated from multiple sources or handwritten, so we
+ de-duplicate commits against the same machine to avoid duplicate work.
+ """
+ best: Dict[Target, WorkItem] = {}
+ for item in items:
+ existing = best.get(item.target)
+ if existing is None or item.runs_to_request > existing.runs_to_request:
+ best[item.target] = item
+ # Preserve the order of the plan: it is the order the work should be done in.
+ seen = set()
+ ordered: List[WorkItem] = []
+ for item in items:
+ if item.target not in seen:
+ seen.add(item.target)
+ ordered.append(best[item.target])
+ return ordered
+
+
+def target_of_run(title: str) -> Optional[Target]:
+ """
+ Return what a workflow run is benchmarking, or None if it can't be told.
+
+ The name has to match the workflow's `run-name` exactly.
+ """
+ match = RUN_NAME.match(title.strip())
+ return Target(match.group(1).lower(), match.group(2)) if match else None
+
+
+def run_gh(args: Sequence[str], stdin: Optional[str] = None) -> str:
+ command = ['gh', *args]
+ logging.debug(f'$ {" ".join(command)}')
+ try:
+ result = subprocess.run(command, check=True, text=True, input=stdin,
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ except FileNotFoundError:
+ raise GithubError('gh was not found. This tool needs the Github CLI to be installed.')
+ except subprocess.CalledProcessError as error:
+ raise GithubError(f'{" ".join(command)} failed with status {error.returncode}: '
+ f'{error.stderr.strip()}')
+ return result.stdout
+
+
+def workflow_runs(repo: str, workflow: str) -> List[WorkflowRun]:
+ """
+ Return every run of a workflow whose name says what it benchmarks.
+
+ The whole history is walked, however Github only retains runs for 90 days.
+ Runs are extracted without filtering on status.
+
+ Cancelled runs or runs named in any other format are ignored: they either
+ predate this tooling or don't represent expended capacity towards benchmarking
+ a target.
+ """
+ query = urllib.parse.urlencode({'per_page': 100})
+ # The projection also flattens paginated output into one object per line;
+ # without one, `--paginate` emits a whole JSON document per page.
+ output = run_gh(['api', '--paginate', '-q',
+ '.workflow_runs[] | {id, display_title, status, conclusion}',
+ f'/repos/{repo}/actions/workflows/{workflow}/runs?{query}'])
+ runs: Dict[int, WorkflowRun] = {}
+ for line in output.splitlines():
+ line = line.strip()
+ if not line:
+ continue
+ try:
+ data = json.loads(line)
+ except json.JSONDecodeError as error:
+ raise GithubError(f'could not parse a workflow run from gh output: {error}: {line}')
+ if 'id' not in data:
+ raise GithubError(f'a workflow run came back without an id: {line}')
+ target = target_of_run(data.get('display_title') or '')
+ if target is None or data.get('conclusion') == 'cancelled':
+ continue
+ # Deduplicate by id: pages can overlap while runs are being created.
+ runs[data['id']] = WorkflowRun(identifier=data['id'], target=target,
+ finished=(data.get('status') or '') == 'completed')
+ return list(runs.values())
+
+
+def summarize(runs: Sequence[WorkflowRun]) -> Runs:
+ """Count what the workflow is doing, per target."""
+ in_flight: Dict[Target, int] = {}
+ completed: Dict[Target, int] = {}
+ ids: Dict[Target, List[int]] = {}
+ for run in runs:
+ if run.finished:
+ completed[run.target] = completed.get(run.target, 0) + 1
+ else:
+ in_flight[run.target] = in_flight.get(run.target, 0) + 1
+ ids.setdefault(run.target, []).append(run.identifier)
+ return Runs(in_flight=in_flight, completed=completed, ids=ids)
+
+
+def decide(items: Sequence[WorkItem],
+ runs: Runs,
+ max_in_flight: int,
+ extra_attempts: int) -> List[Decision]:
+ """
+ Return a list of Decisions describing what to do with each item.
+
+ Items are considered in the order the plan gave them, which is the order the
+ work should be done in.
+
+ Two budgets are enforced. Capacity: no more than `max_in_flight` runs may exist
+ at once on a machine, including queued runs since they occupy the queue just as
+ much as currently running jobs. Per machine rather than overall because the
+ machines are separate pools of runners.
+
+ Attempts: a target whose runs keep completing without the results reaching LNT
+ (e.g. cancelled, timed out, or failing before the submission step) would
+ otherwise risk being requested forever, since the planner only sees what reached
+ LNT. That budget is per machine too, and runs already in flight count against it.
+ """
+ remaining = {machine: max(0, max_in_flight - runs.in_flight_on(machine))
+ for machine in {item.machine for item in items}}
+ decisions: List[Decision] = []
+
+ def in_flight_detail(target: Target, running: int) -> str:
+ ids = ', '.join(str(i) for i in runs.ids.get(target, [])[:3])
+ return f'{running} already in flight' + (f' (run {ids})' if ids else '')
+
+ for item in items:
+ target = item.target
+ running = runs.in_flight.get(target, 0)
+ finished = runs.completed.get(target, 0)
+ budget = item.samples_want + extra_attempts
+ # Only a target whose budget is spent with nothing left running is really abandoned.
+ if finished + running >= budget and running == 0:
+ reason = f'giving up: {item.samples_have}/{item.samples_want} runs recorded in LNT after {finished} completed runs, budget is {budget}'
+ decisions.append(Decision(item, action='skip', jobs=0, reason=reason))
+ continue
+ pending = min(item.runs_to_request - running, budget - finished - running)
+ if pending <= 0:
+ decisions.append(Decision(item, action='skip', jobs=0,
+ reason=in_flight_detail(target, running)))
+ continue
+ if remaining[item.machine] == 0:
+ reason = 'no capacity'
+ if running:
+ reason = f'{reason}, {in_flight_detail(target, running)}'
+ decisions.append(Decision(item, action='defer', jobs=0, reason=reason))
+ continue
+ dispatching = min(pending, remaining[item.machine])
+ remaining[item.machine] -= dispatching
+ deferred = pending - dispatching
+ reason = item.reason
+ if deferred:
+ reason = f'{reason} ({deferred} more deferred, no capacity)'
+ decisions.append(Decision(item, action='dispatch', jobs=dispatching, reason=reason))
+ return decisions
+
+
+def workflow_inputs(args: argparse.Namespace, item: WorkItem) -> Dict[str, str]:
+ """Return the inputs to dispatch the workflow with for a work item."""
+ inputs = {'commit': item.commit,
+ 'lnt-machine': item.machine,
+ 'benchmark-suite-version': args.test_suite_commit,
+ # Submission is not optional: a run whose results are not recorded
+ # anywhere would be requested again on every future invocation.
+ # During a dry run nothing runs, so nothing is submitted either.
+ 'submit-lnt': 'false' if args.dry_run else 'true'}
+ if args.lnt_url:
+ inputs['lnt-url'] = args.lnt_url
+ return inputs
+
+
+def report(args: argparse.Namespace, decisions: Sequence[Decision], runs: Runs) -> None:
+ """Print a human-readable summary of what is about to be done."""
+ out = sys.stderr
+ print(f'Submitting to: {args.lnt_url or "whatever the workflow defaults to"}', file=out)
+ # One line per machine the plan is about, since capacity is per machine.
+ for machine in sorted({d.item.machine for d in decisions}):
+ in_flight = runs.in_flight_on(machine)
+ available = max(0, args.max_in_flight - in_flight)
+ print(f'Capacity on {machine}: {args.max_in_flight} runs allowed at once, {in_flight} '
+ f'in flight -> {available} slots available', file=out)
+ print('', file=out)
+
+ symbols = {'dispatch': '+', 'skip': '~', 'defer': '.'}
+ rows = [(f'{symbols[d.action]} {d.action}', d.item.commit[:12], d.item.machine,
+ str(d.jobs) if d.action == 'dispatch' else '-', d.reason)
+ for d in decisions]
+ print(tabulate.tabulate(rows, headers=['ACTION', 'COMMIT', 'MACHINE', 'JOBS', 'REASON'],
+ tablefmt='plain', disable_numparse=True,
+ colalign=('left', 'left', 'left', 'right', 'left'))
+ if rows else ' nothing to do', file=out)
+ print('', file=out)
+
+ counts = {'dispatch': 0, 'skip': 0, 'defer': 0}
+ for decision in decisions:
+ counts[decision.action] += 1
+ jobs = sum(d.jobs for d in decisions if d.action == 'dispatch')
+ print(f'Summary: {len(decisions)} commit/machine pairs requested by the plan, '
+ f'{counts["skip"]} already in flight or given up on, '
+ f'{counts["dispatch"]} being requested now ({jobs} runs), '
+ f'{counts["defer"]} left for later.', file=out)
+ if args.dry_run:
+ print('DRY RUN: nothing was requested.', file=out)
+
+
+def main(argv: List[str]) -> int:
+ parser = argparse.ArgumentParser(
+ prog='dispatch-benchmarks',
+ description='Read a benchmark plan in a compatible format (see plan-benchmarks), and request the '
+ 'corresponding Github Actions workflow runs.\n'
+ '\n'
+ 'The plan is read as one unit of work per line. A unit of work is a commit to benchmark '
+ 'on a specific machine. It is represented as a JSON object containing these fields: '
+ '`commit`, the commit to benchmark; `machine`, the LNT machine to benchmark on; '
+ '`samples_have` and `samples_want`, how many runs it already has and how many it '
+ 'should end up with; and `reason`, a human-readable explanation.\n'
+ '\n'
+ 'Work that is already running is not requested again, and no more than a fixed '
+ 'number of runs are allowed to exist at once, so that the benchmark machines '
+ 'remain available for other jobs. Work that does not fit is left for a later '
+ 'invocation: since the plan is recomputed from scratch every time, nothing is '
+ 'lost by deferring it.\n'
+ '\n'
+ 'Results are submitted to LNT unless this is a dry run. That is not optional: '
+ 'results that are not recorded anywhere would be requested again on every '
+ 'invocation, forever.',
+ epilog='This script depends on the modules listed in `libcxx/utils/requirements.txt`.',
+ formatter_class=HelpFormatter)
+ parser.add_argument('--work-items', type=argparse.FileType('r'), default=sys.stdin,
+ help='A file containing the plan, one JSON object per line. By default, this is read from '
+ 'standard input.')
+ parser.add_argument('--test-suite-commit', type=str, required=True,
+ help='The version of the benchmark suite to use, as a monorepo SHA. Pinning this is what '
+ 'makes results comparable across commits.')
+ parser.add_argument('--max-in-flight', type=positive, default=4,
+ help='The largest number of workflow runs allowed to exist at once for a single LNT machine. '
+ 'This counts runs that are queued but not yet started.')
+ parser.add_argument('--extra-attempts', type=non_negative, default=2,
+ help='How many runs beyond the desired number of samples a commit may consume on a machine '
+ 'before it is given up on. This bounds the cost of a job that keeps dying before it '
+ 'can submit, since such a run leaves no trace in LNT and would otherwise be requested '
+ 'forever. It is counted per machine.')
+ parser.add_argument('--lnt-url', type=str, required=False,
+ help='The LNT instance the runs should submit to. By default the workflow submits to '
+ 'whichever instance it defaults to.')
+ parser.add_argument('--dry-run', action='store_true',
+ help='Do not request anything, just report what would be requested.')
+ parser.add_argument('--output', type=pathlib.Path, default=None,
+ help='Where to write the record of what was requested, one JSON object per line. Defaults '
+ 'to standard output.')
+ parser.add_argument('-q', '--quiet', action='store_true',
+ help='Do not print the human-readable summary to standard error.')
+ parser.add_argument('-v', '--verbose', action='count', default=0,
+ help='Verbosity level: passing the option multiple times increases the level.')
+ args = parser.parse_args(argv)
+
+ # --quiet raises the logging threshold rather than silencing everything, since we
+ # still want to see warnings.
+ logging.basicConfig(format='%(levelname)s: %(message)s',
+ level=logging.DEBUG if args.verbose else
+ logging.WARNING if args.quiet else logging.INFO)
+
+ items = collapse(parse_plan(args.work_items))
+
+ runs = summarize(workflow_runs('llvm/llvm-project', 'libcxx-benchmark-commit.yml'))
+
+ decisions = decide(items, runs, args.max_in_flight, args.extra_attempts)
+
+ # Warn then we give up for good on a commit/machine pair (because enough attempts have been done).
+ for decision in decisions:
+ if decision.action == 'skip' and decision.reason.startswith('giving up'):
+ logging.warning(f'{decision.item.commit[:12]} on {decision.item.machine}: {decision.reason}')
+
+ if not args.quiet:
+ report(args, decisions, runs)
+
+ # Requests are made one at a time and recorded as they succeed. A failure halfway through must still
+ # leave an accurate record, since the runs already requested actually exist in Github.
+ records: List[str] = []
+ dispatched = 0
+ try:
+ for decision in decisions:
+ if decision.action != 'dispatch':
+ continue
+ inputs = workflow_inputs(args, decision.item)
+ body = json.dumps({'ref': 'main', 'inputs': inputs}) # always use the workflow on `main`
+ for _ in range(decision.jobs):
+ if not args.dry_run:
+ run_gh(['api', '-X', 'POST', '--input', '-',
+ '/repos/llvm/llvm-project/actions/workflows/'
+ 'libcxx-benchmark-commit.yml/dispatches'],
+ stdin=body)
+ dispatched += 1
+ records.append(json.dumps({'commit': decision.item.commit,
+ 'machine': decision.item.machine,
+ 'reason': decision.item.reason,
+ 'dry_run': args.dry_run,
+ 'inputs': inputs}, sort_keys=True) + '\n')
+ except GithubError:
+ logging.error(f'requested {dispatched} runs before failing')
+ raise
+ finally:
+ # Write what we managed to do even when we fail: those runs exist and will
+ # consume machine time, so we want to report them.
+ contents = ''.join(records)
+ if args.output is None:
+ sys.stdout.write(contents)
+ else:
+ args.output.write_text(contents)
+
+ if not args.quiet and not args.dry_run:
+ print(f'Requested {dispatched} runs.', file=sys.stderr)
+ return 0
+
+
+if __name__ == '__main__':
+ try:
+ sys.exit(main(sys.argv[1:]))
+ except (ValueError, OSError, GithubError) as error:
+ sys.exit(f'error: {error}')
diff --git a/libcxx/utils/ci/lnt/plan-benchmarks b/libcxx/utils/ci/lnt/plan-benchmarks
new file mode 100755
index 0000000000000..5d959f1d39385
--- /dev/null
+++ b/libcxx/utils/ci/lnt/plan-benchmarks
@@ -0,0 +1,228 @@
+#!/usr/bin/env python3
+# ===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ===----------------------------------------------------------------------===##
+
+from typing import Dict, List, Optional, Sequence, TypedDict
+import argparse
+import json
+import logging
+import pathlib
+import sys
+import tabulate
+import urllib.error
+import urllib.parse
+import urllib.request
+
+
+# LNT instances can be slow: timeout after that many seconds.
+TIMEOUT_SECONDS = 120
+
+# One line of the produced plan.
+class WorkItem(TypedDict):
+ commit: str
+ machine: str
+ samples_have: int
+ samples_want: int
+ reason: str
+
+
+class HelpFormatter(argparse.ArgumentDefaultsHelpFormatter,
+ argparse.RawDescriptionHelpFormatter):
+ """Show defaults, but keep the paragraphs of the description intact."""
+
+def is_sha(string: str) -> bool:
+ return len(string) == 40 and all(c in '0123456789abcdef' for c in string.lower())
+
+def positive(string: str) -> int:
+ value = int(string)
+ if value < 1:
+ raise argparse.ArgumentTypeError(f'expected a positive integer, got {string}')
+ return value
+
+class LNTError(Exception):
+ pass
+
+
+def get(url: str) -> Optional[dict]:
+ """GET a JSON resource, returning None if the server answers 404."""
+ logging.debug(f'GET {url}')
+ try:
+ with urllib.request.urlopen(url, timeout=TIMEOUT_SECONDS) as response:
+ return json.loads(response.read().decode())
+ except urllib.error.HTTPError as error:
+ if error.code == 404:
+ return None
+ raise LNTError(f'GET {url} failed with HTTP {error.code}: {error.reason}')
+ except urllib.error.URLError as error:
+ raise LNTError(f'GET {url} failed: {error.reason}')
+ except (json.JSONDecodeError, UnicodeDecodeError) as error:
+ raise LNTError(f'GET {url} returned a malformed response: {error}')
+
+
+def benchmarked_commits(lnt_url: str, test_suite: str, machine: str) -> Optional[Dict[str, int]]:
+ """
+ Return {commit: number of runs} for a machine, or None if the instance has no
+ such machine.
+ """
+ base = f'{lnt_url.rstrip("/")}/api/db_default/v4/{test_suite}'
+ result = get(f'{base}/machines/{urllib.parse.quote(machine)}')
+ if result is None:
+ listing = get(f'{base}/machines')
+ if listing is None:
+ raise LNTError(f'{lnt_url} has no test suite named {test_suite}')
+ named = [m for m in listing.get('machines', []) if m.get('name') == machine]
+ if len(named) > 1:
+ ids = ', '.join(str(m.get('id')) for m in named)
+ raise LNTError(f'ambiguous name: {lnt_url} has {len(named)} machines named {machine} (ids {ids})')
+ return None
+
+ counts: Dict[str, int] = {}
+ for run in result.get('runs', []):
+ if 'git_sha' not in run:
+ raise LNTError(f'run {run.get("id")} has no git_sha field. Is {test_suite} the right test suite?')
+ commit = str(run['git_sha']).strip().lower() # normalize commit SHAs
+ counts[commit] = counts.get(commit, 0) + 1
+ return counts
+
+
+def plan(commits: Sequence[str],
+ have: Dict[str, int],
+ want: int,
+ machine: str) -> List[WorkItem]:
+ """
+ Return the work items needed to bring every commit up to `want` runs.
+
+ The order of the input is preserved. A commit repeated on the input is
+ planned once, at the position it first appeared.
+ """
+ items: List[WorkItem] = []
+ for commit in dict.fromkeys(commits):
+ existing = have.get(commit, 0)
+ if existing >= want:
+ continue
+ items.append({'commit': commit,
+ 'machine': machine,
+ 'samples_have': existing,
+ 'samples_want': want,
+ 'reason': f'{existing}/{want} runs'})
+ return items
+
+
+def main(argv: List[str]) -> int:
+ parser = argparse.ArgumentParser(
+ prog='plan-benchmarks',
+ description='Determine which commits need benchmark data based on the set of desired commits '
+ 'and the content of an existing LNT instance.\n'
+ '\n'
+ 'This tool effectively compares a desired state (represented by a set of desired '
+ 'commits and a number of samples for each) to a current state (obtained from a '
+ 'LNT instance containing data), and determines the actions to take in order to '
+ 'fill that gap. This plan is printed to standard output as one JSON object per line, '
+ 'in the order the commits were given: that order is the priority, since the '
+ 'dispatcher works through the plan in order and stops when it runs out of '
+ 'capacity.\n'
+ '\n'
+ 'This tool does not manage state of its own, and running it repeatedly as the LNT '
+ 'instance gains more data makes it converge towards the desired state.\n'
+ '\n'
+ 'Note that a run counts towards a commit whether or not it produced any results. '
+ 'Building the library or its benchmarks can fail at an arbitrary commit, so this '
+ 'tool considers a commit with enough runs in LNT to have been measured, even if '
+ 'those runs contain no (or few) valid benchmark results.\n'
+ '\n'
+ 'Each line of the produced plan carries the commit to benchmark, the machine the '
+ 'plan is about, how many runs the commit already has and how many it should end '
+ 'up with, and a human-readable reason.',
+ epilog='This script depends on the modules listed in `libcxx/utils/requirements.txt`.',
+ formatter_class=HelpFormatter)
+ parser.add_argument('--commit-list', type=argparse.FileType('r'), default=sys.stdin,
+ help='A file of whitespace separated commits for which benchmark data is desired. By default, '
+ 'this is read from standard input.')
+ parser.add_argument('--lnt-url', type=str, required=True,
+ help='The URL of the LNT instance holding the benchmark data.')
+ parser.add_argument('--test-suite', type=str, required=True,
+ help='The LNT test suite holding the benchmark data.')
+ parser.add_argument('--machine', type=str, required=True,
+ help='The LNT machine to plan work for.')
+ parser.add_argument('--samples', type=positive, default=3,
+ help='How many benchmark runs each commit should have.')
+ parser.add_argument('--allow-missing-machine', action='store_true',
+ help='Accept that the LNT instance has no such machine, and plan despite that. '
+ 'Needed when first populating a machine, and withheld by default because a '
+ 'mistyped or renamed machine is otherwise indistinguishable from one that '
+ 'has no data yet.')
+ parser.add_argument('--output', type=pathlib.Path, default=None,
+ help='Where to write the plan. Defaults to standard output.')
+ parser.add_argument('-q', '--quiet', action='store_true',
+ help='Do not print the human-readable summary to standard error.')
+ parser.add_argument('-v', '--verbose', action='count', default=0,
+ help='Verbosity level: passing the option multiple times increases the level.')
+ args = parser.parse_args(argv)
+
+ # --quiet raises the logging threshold rather than silencing everything, since we
+ # still want to see warnings.
+ logging.basicConfig(format='%(levelname)s: %(message)s',
+ level=logging.DEBUG if args.verbose else
+ logging.WARNING if args.quiet else logging.INFO)
+
+ requested = [c.lower() for line in args.commit_list for c in line.split()] # normalize commit SHAs
+ invalid = [c for c in requested if not is_sha(c)]
+ if invalid:
+ raise ValueError(f'expected full 40-character SHAs on the input, got {invalid[0]!r}')
+
+ have = benchmarked_commits(args.lnt_url, args.test_suite, args.machine)
+ if have is None:
+ if not args.allow_missing_machine:
+ raise ValueError(f'{args.lnt_url} has no machine named {args.machine} in test suite '
+ f'{args.test_suite}. Pass --allow-missing-machine if that is intended, '
+ f'otherwise check your usage of the tool to avoid generating a potentially '
+ f'large list of commits to benchmark.')
+ logging.warning(f'{args.lnt_url} has no machine named {args.machine} yet: planning as if it had no data')
+ have = {}
+
+ items = plan(requested, have, args.samples, args.machine)
+
+ if not args.quiet:
+ runs_to_request = lambda item: item['samples_want'] - item['samples_have']
+ unique = list(dict.fromkeys(requested))
+ complete = sum(1 for c in unique if have.get(c, 0) >= args.samples)
+ partial = sum(1 for c in unique if 0 < have.get(c, 0) < args.samples)
+ print(f'Machine: {args.machine} Suite: {args.test_suite} LNT: {args.lnt_url}', file=sys.stderr)
+ print(f'Requested commits: {len(unique)} '
+ f'({len(unique) - complete - partial} with no data, {partial} partial, {complete} complete)',
+ file=sys.stderr)
+ planned = {i['commit']: i for i in items}
+ rows = []
+ for commit in unique:
+ item = planned.get(commit)
+ if item is None and not args.verbose:
+ continue
+ action = f'request {runs_to_request(item)}' if item else 'ok'
+ rows.append((commit[:12], have.get(commit, 0), args.samples, action))
+ if rows:
+ print(tabulate.tabulate(rows, headers=['COMMIT', 'HAVE', 'WANT', 'ACTION'],
+ tablefmt='plain', disable_numparse=True,
+ colalign=('left', 'right', 'right', 'left')),
+ file=sys.stderr)
+ print(f'Plan: {sum(runs_to_request(i) for i in items)} runs across {len(items)} commits',
+ file=sys.stderr)
+
+ # Write the output all at once at the end, to avoid partial output in case of failure.
+ contents = ''.join(json.dumps(i, sort_keys=True) + '\n' for i in items)
+ if args.output is None:
+ sys.stdout.write(contents)
+ else:
+ args.output.write_text(contents)
+ return 0
+
+
+if __name__ == '__main__':
+ try:
+ sys.exit(main(sys.argv[1:]))
+ except (ValueError, OSError, LNTError) as error:
+ sys.exit(f'error: {error}')
diff --git a/libcxx/utils/ci/lnt/select-anchor-commits b/libcxx/utils/ci/lnt/select-anchor-commits
new file mode 100755
index 0000000000000..2a2d86faec97c
--- /dev/null
+++ b/libcxx/utils/ci/lnt/select-anchor-commits
@@ -0,0 +1,211 @@
+#!/usr/bin/env python3
+# ===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ===----------------------------------------------------------------------===##
+
+from typing import Dict, List, Optional, Sequence, Tuple
+import argparse
+import datetime
+import logging
+import os
+import pathlib
+import subprocess
+import sys
+import tabulate
+
+
+# Commits that cannot change the performance of the library are not worth
+# benchmarking. This is the set of paths we consider for libc++ changes.
+BENCHMARK_PATHS = ['libcxx/include', 'libcxx/src']
+
+GRANULARITIES = ('day', 'week', 'month')
+
+# The UTC timestamp at which a commit landed, in seconds since the epoch, and the
+# commit SHA itself. That order is deliberate: it sorts chronologically by default,
+# and comparing two of them falls back to the SHA when the timestamps are equal, which
+# makes this tool deterministic.
+Commit = Tuple[int, str]
+
+
+class HelpFormatter(argparse.ArgumentDefaultsHelpFormatter,
+ argparse.RawDescriptionHelpFormatter):
+ """Show defaults, but keep the paragraphs of the description intact."""
+
+def directory_path(string: str) -> pathlib.Path:
+ if not os.path.isdir(string):
+ raise argparse.ArgumentTypeError(f'{string} is not a directory')
+ return pathlib.Path(string)
+
+def date(string: str) -> datetime.date:
+ try:
+ return datetime.date.fromisoformat(string)
+ except ValueError:
+ raise argparse.ArgumentTypeError(f'{string} is not a valid ISO 8601 date (YYYY-MM-DD)')
+
+def git(git_repo: pathlib.Path, *args: str) -> str:
+ command = ['git', '-C', str(git_repo), *args]
+ logging.debug(f'$ {" ".join(command)}')
+ try:
+ return subprocess.run(command, check=True, text=True, stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE).stdout
+ except subprocess.CalledProcessError as error:
+ raise ValueError(f'{" ".join(command)} failed: {error.stderr.strip()}')
+
+
+def benchmarkable_commits(git_repo: pathlib.Path, rev: str,
+ paths: Sequence[str]) -> List[Commit]:
+ """
+ Return the commits that are candidates for benchmarking, oldest first.
+ """
+ output = git(git_repo, 'log', '--reverse', '--first-parent', '--no-show-signature',
+ '--format=%ct %H', rev, '--', *paths)
+ commits: List[Commit] = []
+ for line in output.splitlines():
+ fields = line.split()
+ if len(fields) != 2:
+ raise ValueError(f'unexpected output from git log: {line!r}')
+ commits.append((int(fields[0]), fields[1]))
+ return commits
+
+
+def clamp_date(day: datetime.date, granularity: str, *, direction: str) -> datetime.date:
+ """
+ Clamp a date to the nearest day/week/month, going in the specified direction.
+
+ For example, clamping a date that falls on a Wednesday using 'week' granularity
+ will produce the next date that is a Monday in forward direction, and the previous
+ date that was a Monday in backward direction.
+ """
+ if granularity == 'day':
+ at_boundary = lambda d: True
+ elif granularity == 'week':
+ at_boundary = lambda d: d.weekday() == 0 # Monday
+ elif granularity == 'month':
+ at_boundary = lambda d: d.day == 1 # first of the month
+ else:
+ raise ValueError(f'invalid granularity {granularity}, expected one of {GRANULARITIES}')
+
+ if direction == 'forward':
+ step = datetime.timedelta(days=1)
+ elif direction == 'backward':
+ step = datetime.timedelta(days=-1)
+ else:
+ raise ValueError(f'invalid direction {direction}, expected one of backward, forward')
+
+ while not at_boundary(day):
+ day += step
+ return day
+
+
+def select_anchor_commits(commits: Sequence[Commit],
+ granularity: str,
+ since: Optional[datetime.date] = None,
+ until: Optional[datetime.date] = None) -> List[Commit]:
+ """
+ Select one commit per calendar bucket: the oldest one in each, oldest first.
+
+ `since` is inclusive and `until` exclusive, both interpreted as UTC dates.
+ """
+ buckets: Dict[datetime.date, Commit] = {}
+ for (timestamp, sha) in commits:
+ day = datetime.datetime.fromtimestamp(timestamp, datetime.timezone.utc).date()
+ if since is not None and day < since:
+ continue
+ if until is not None and day >= until:
+ continue
+ bucket = clamp_date(day, granularity, direction='backward')
+ existing = buckets.get(bucket)
+ if existing is None or (timestamp, sha) < existing:
+ buckets[bucket] = (timestamp, sha)
+ return [anchor for (_, anchor) in sorted(buckets.items())]
+
+
+def main(argv: List[str]) -> int:
+ parser = argparse.ArgumentParser(
+ prog='select-anchor-commits',
+ description='Select the libc++ commits that should have benchmark data, by picking one '
+ 'commit in each calendar bucket (one per week by default). The commits are '
+ 'printed to standard output from oldest to newest, one per line.\n'
+ '\n'
+ 'The selection depends only on the calendar and on the history of the '
+ 'repository, so it is stable as new commits land: the same options always '
+ 'produce the same commits. This makes it possible to recompute the list in '
+ 'a reproducible fashion, which is useful when e.g. re-generating historical '
+ 'performance data after a configuration change.',
+ epilog='This script depends on the modules listed in `libcxx/utils/requirements.txt`.',
+ formatter_class=HelpFormatter)
+ parser.add_argument('--since', type=date, required=True,
+ help='Only select commits on or after this date (UTC). Selection starts at the first '
+ 'bucket beginning on or after it, so a date partway through a bucket skips that '
+ 'bucket rather than truncating it -- a truncated bucket would yield a different '
+ 'commit than a run starting earlier would.')
+ parser.add_argument('--until', type=date, required=False,
+ help='Only select commits strictly before this date (UTC). Unlike --since this needs no '
+ 'adjustment: a bucket cut short at the end still yields the same commit, since the '
+ 'oldest one in it is the one selected. By default, there is no limit.')
+ parser.add_argument('--every', type=str, choices=GRANULARITIES, default='week',
+ help='The calendar granularity at which to select commits.')
+ parser.add_argument('--paths', type=str, nargs='+', default=BENCHMARK_PATHS,
+ help='Only consider commits touching these paths.')
+ parser.add_argument('--rev', type=str, default='HEAD',
+ help='The revision whose history is walked.')
+ parser.add_argument('--git-repo', type=directory_path, default=pathlib.Path(os.getcwd()),
+ help='Path to the Git repository to use.')
+ parser.add_argument('--output', type=pathlib.Path, default=None,
+ help='Where to write the selected commits. Defaults to standard output.')
+ parser.add_argument('-q', '--quiet', action='store_true',
+ help='Do not print a human-readable summary to standard error.')
+ parser.add_argument('-v', '--verbose', action='count', default=0,
+ help='Verbosity level: passing the option multiple times increases the level.')
+ args = parser.parse_args(argv)
+
+ # --quiet raises the logging threshold rather than silencing everything, since we
+ # still want to see warnings.
+ logging.basicConfig(format='%(levelname)s: %(message)s',
+ level=logging.DEBUG if args.verbose else
+ logging.WARNING if args.quiet else logging.INFO)
+
+ since = clamp_date(args.since, args.every, direction='forward')
+ if args.until is not None and args.until <= since:
+ raise ValueError(f'--until {args.until} leaves no whole {args.every} between it and '
+ f'--since {args.since}: the first one starts on {since}.')
+
+ git(args.git_repo, 'rev-parse', '--verify', f'{args.rev}^{{commit}}')
+ commits = benchmarkable_commits(args.git_repo, args.rev, args.paths)
+ anchors = select_anchor_commits(commits, args.every, since=since,
+ until=args.until)
+
+ if not args.quiet:
+ print(f'Anchor commits from {" ".join(str(p) for p in args.paths)} in {args.git_repo} '
+ f'at {args.rev}', file=sys.stderr)
+ print(f'Range: {since} to {args.until if args.until else "now"}, '
+ f'one per {args.every}', file=sys.stderr)
+ print(f'Candidate commits: {len(commits)} Selected: {len(anchors)}', file=sys.stderr)
+ rows = []
+ for (timestamp, sha) in anchors:
+ when = datetime.datetime.fromtimestamp(timestamp, datetime.timezone.utc)
+ bucket = clamp_date(when.date(), args.every, direction='backward')
+ label = f'{bucket} ({bucket:%G-W%V})' if args.every == 'week' else str(bucket)
+ rows.append((label, sha, when.strftime('%Y-%m-%d %H:%M')))
+ if rows:
+ print(tabulate.tabulate(rows, headers=['BUCKET', 'COMMIT', 'DATE (UTC)'],
+ tablefmt='plain', disable_numparse=True), file=sys.stderr)
+
+ # Write the output all at once at the end, to avoid partial output in case of failure.
+ contents = ''.join(f'{sha}\n' for (_, sha) in anchors)
+ if args.output is None:
+ sys.stdout.write(contents)
+ else:
+ args.output.write_text(contents)
+ return 0
+
+
+if __name__ == '__main__':
+ try:
+ sys.exit(main(sys.argv[1:]))
+ except (ValueError, OSError) as error:
+ sys.exit(f'error: {error}')
More information about the libcxx-commits
mailing list