[libcxx-commits] [libcxx] [libc++] Add a tool to find when a benchmark changed (PR #233656)
via libcxx-commits
libcxx-commits at lists.llvm.org
Mon Oct 19 19:33:47 PDT 2026
llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-libcxx
Author: Louis Dionne (ldionne)
<details>
<summary>Changes</summary>
Add `detect-changepoints`, which reads a time series line-by-line and reports the changepoints in the time series.
The method is E-Divisive (Matteson & James, 2014), which has been used for similar applications (e.g. MongoDB performance tooling similar to what we're doing here, see [1]). Note that we're not using the `signal-processing-algorithms` package [2] since that repo seems barely maintained and does not support Python > 3.11.
[1]: https://dl.acm.org/doi/10.1145/3491204.3527488
[2]: https://github.com/mongodb/signal-processing-algorithms
Assisted by Claude
---
Patch is 22.98 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/233656.diff
3 Files Affected:
- (modified) libcxx/utils/ci/lnt/README.md (+30)
- (added) libcxx/utils/ci/lnt/detect-changepoints (+441)
- (modified) libcxx/utils/ci/lnt/requirements.txt (+1)
``````````diff
diff --git a/libcxx/utils/ci/lnt/README.md b/libcxx/utils/ci/lnt/README.md
index 8155a3ec94b31..e5c475d9fe4e7 100644
--- a/libcxx/utils/ci/lnt/README.md
+++ b/libcxx/utils/ci/lnt/README.md
@@ -113,6 +113,36 @@ Note that `run-benchmarks` does not build anything: the library being benchmarke
or installed beforehand. How to pick up that library is determined by `--test-config` and any Lit parameters
passed.
+## Finding when a benchmark changed
+
+`detect-changepoints` reports the points at which a single benchmark changed, which allows finding regressions
+and progressions in a benchmark. In many cases, pin-pointing the exact commit at which the change happened
+requires gathering finer-grained data (essentially benchmarking between anchor commits). `detect-changepoints`
+reads JSONL with one object per measurement:
+
+```
+# Print every measurement of a benchmark on a machine, one page of the query API at a time.
+fetch_series() {
+ local cursor=null page
+ while
+ page=$(jq -n --arg machine "$1" --arg test "$2" --argjson cursor "${cursor}" \
+ '{metric: "execution_time", machine: $machine, test: [$test],
+ sort: "commit", limit: 10000, cursor: $cursor}' \
+ | curl -sS --fail-with-body -X POST https://lnt.llvm.org/api/suites/libcxx/query \
+ -H 'Content-Type: application/json' -d @-)
+ jq -c '.items[]' <<< "${page}"
+ cursor=$(jq '.cursor.next' <<< "${page}")
+ [ "${cursor}" != null ]
+ do :; done
+}
+
+fetch_series linux-x86_64-20260812 'rng::find_if_not(vector<bool>)_(process_all)/1024' \
+ | detect-changepoints --aggregate median
+```
+
+Repeated measurements of a commit must be combined with `--aggregate`. See
+`detect-changepoints --help` for details and other options.
+
## Test suite schemas
`schemas/` holds the schemas of the `libcxx` and `libstdcxx` test suites as we submit them to LNT.
diff --git a/libcxx/utils/ci/lnt/detect-changepoints b/libcxx/utils/ci/lnt/detect-changepoints
new file mode 100755
index 0000000000000..6aead205b24bc
--- /dev/null
+++ b/libcxx/utils/ci/lnt/detect-changepoints
@@ -0,0 +1,441 @@
+#!/usr/bin/env python3
+# ===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ===----------------------------------------------------------------------===##
+
+from typing import Dict, List, NamedTuple, Optional, Sequence, Tuple
+import argparse
+import common
+import json
+import logging
+import math
+import numpy
+import pathlib
+import statistics
+import sys
+import tabulate
+
+
+# Ways to aggregate repeated measurements of one point into the single value taken into
+# account by the detection algorithm.
+AGGREGATORS = {
+ 'median': statistics.median,
+ 'mean': statistics.fmean,
+ 'min': min,
+}
+
+
+class Point(NamedTuple):
+ """
+ One position of the series: a label, its place in the order, and every sample
+ measured there.
+
+ A point is normally a commit, and its samples are the individual benchmark runs
+ performed at that commit. The ordinal exists only to put the points in order while
+ the series is being read, and is implicit from the point's position in the series
+ afterwards.
+ """
+ label: str
+ ordinal: Optional[int]
+ samples: List[float]
+
+
+class Split(NamedTuple):
+ """A candidate division of a segment: where it falls, and how good it is."""
+ point: int
+ statistic: float
+
+
+class Candidate(NamedTuple):
+ """A split that survived its permutation test."""
+ point: int
+ p_value: float
+
+
+class Changepoint(NamedTuple):
+ """A detected change, located between two consecutive points of the series."""
+ before_x: str # last point before the change
+ after_x: str # first point after the change
+ before_y: float # representative value of the segment before the change
+ after_y: float # representative value of the segment after the change
+ change: float # relative change, as a fraction of the value before
+ p_value: float
+
+
+class Settings(NamedTuple):
+ """The parameters the detection runs with, fixed for the whole series."""
+ min_points: int
+ p_value: float
+ permutations: int
+ aggregate: str
+
+
+class HelpFormatter(argparse.ArgumentDefaultsHelpFormatter,
+ argparse.RawDescriptionHelpFormatter):
+ """Show defaults, but keep the paragraphs of the description intact."""
+
+
+def read_series(stream, x: str, y: str, order_by: str) -> List[Point]:
+ """
+ Read a series from a JSONL stream, grouping the samples of each point together.
+
+ The `y` field holds the measurement, and the `x` field labels the points and gathers the
+ records that belong to the same one. Both are required. The optional `order_by` field
+ places the points in order; without it, the order of the records is the order of the
+ series.
+ """
+ points: Dict[str, Point] = {} # in order of first appearance
+ has_order: Optional[bool] = None
+ for (number, line) in enumerate(stream, start=1):
+ line = line.strip()
+ if not line:
+ continue
+ try:
+ record = json.loads(line)
+ except json.JSONDecodeError as error:
+ raise ValueError(f'line {number} is not valid JSON: {error}')
+ if not isinstance(record, dict):
+ raise ValueError(f'line {number} is not a JSON object, but a {type(record).__name__}')
+
+ if y not in record:
+ raise ValueError(f'line {number} has no `{y}` field')
+ value = record[y]
+ if not isinstance(value, (int, float)) or isinstance(value, bool): # bool is a subclass of int
+ raise ValueError(f'line {number}: `{y}` is not a number: {value!r}')
+ value = float(value)
+ if not math.isfinite(value):
+ raise ValueError(f'line {number}: `{y}` is not finite: {record[y]!r}')
+ if value <= 0:
+ raise ValueError(f'line {number}: `{y}` is not positive: {value!r}')
+
+ ordinal = record.get(order_by)
+ if has_order is None:
+ has_order = ordinal is not None
+ elif (ordinal is not None) != has_order:
+ raise ValueError(f'line {number}: `{order_by}` must be given for every record or for none of them.')
+ if ordinal is not None and (isinstance(ordinal, bool) or not isinstance(ordinal, int)):
+ raise ValueError(f'line {number}: `{order_by}` is not an integer: {ordinal!r}')
+
+ if record.get(x) is None:
+ raise ValueError(f'line {number} has no `{x}` field')
+ label = str(record[x])
+ if label in points:
+ existing = points[label]
+ if existing.ordinal != ordinal:
+ raise ValueError(f'line {number}: {label} appears with two different ordinals '
+ f'({existing.ordinal} and {ordinal})')
+ existing.samples.append(value)
+ else:
+ points[label] = Point(label=label, ordinal=ordinal, samples=[value])
+
+ series = list(points.values())
+ if has_order:
+ series.sort(key=lambda point: point.ordinal)
+ return series
+
+
+def cumulative_distances(values: numpy.ndarray) -> numpy.ndarray:
+ """
+ Return the prefix sums of the pairwise distance matrix of `values`.
+
+ Element [i, j] holds the sum of the distances between the first `i` values and the
+ first `j` values, which makes the sum over any rectangular block of points computable
+ with four lookups. The scan for a changepoint needs exactly such block sums for every
+ candidate location, and getting them in constant time keeps the permutation test affordable.
+ """
+ # The energy statistic is consistent for any exponent in (0, 2) applied to the
+ # distances. One is the usual choice and the one this tool is calibrated against.
+ distances = numpy.abs(values[:, None] - values[None, :])
+ prefix = numpy.zeros((len(values) + 1, len(values) + 1))
+ prefix[1:, 1:] = distances.cumsum(axis=0).cumsum(axis=1)
+ return prefix
+
+
+def block(prefix: numpy.ndarray, row_begin, row_end, column_begin, column_end):
+ """Sum the distance matrix over the given half-open ranges of points."""
+ return (prefix[row_end, column_end] - prefix[row_begin, column_end]
+ - prefix[row_end, column_begin] + prefix[row_begin, column_begin])
+
+
+def energy_statistic(prefix: numpy.ndarray, first: int, last: int,
+ splits: numpy.ndarray) -> numpy.ndarray:
+ """
+ Return the scaled energy statistic for dividing the points [first, last) at each of
+ `splits`.
+
+ This is the two-sample energy distance: how far apart the two candidate segments are
+ on average, minus how spread out each of them is internally.
+
+ The result is scaled by `n * m / (n + m)` so that the statistic of an unbalanced split
+ is comparable to that of a balanced one. Without it, splits near the middle of a
+ segment would win on size alone.
+ """
+ before = splits - first
+ after = last - splits
+ # The two within-segment blocks count every unordered pair twice, so pairing them with
+ # the doubled pair count `n * (n - 1)` averages over each pair exactly once.
+ across = block(prefix, first, splits, splits, last)
+ within_before = block(prefix, first, splits, first, splits)
+ within_after = block(prefix, splits, last, splits, last)
+ distance = (2.0 * across / (before * after)
+ - within_before / (before * (before - 1))
+ - within_after / (after * (after - 1)))
+ return distance * before * after / (before + after)
+
+
+def best_split(prefix: numpy.ndarray, first: int, last: int,
+ min_points: int) -> Optional[Split]:
+ """
+ Return the most likely changepoint in the points [first, last), or None when the
+ segment is too short to be divided.
+
+ Since `--min-points` is at least two, every candidate leaves at least two points on
+ each side. The statistic needs that in order to have a notion of within-segment dispersion.
+ """
+ candidates = numpy.arange(first + min_points, last - min_points + 1)
+ if candidates.size == 0:
+ return None
+ energies = energy_statistic(prefix, first, last, candidates)
+ best = int(numpy.argmax(energies))
+ return Split(point=int(candidates[best]), statistic=float(energies[best]))
+
+
+def significance(values: numpy.ndarray, segments: Sequence[Tuple[int, int]],
+ min_points: int, observed: float, permutations: int,
+ rng: numpy.random.Generator) -> float:
+ """
+ Return the permutation p-value of the best candidate changepoint across `segments`.
+
+ Under the hypothesis that nothing changes within any of the segments, the points of
+ each segment are exchangeable, so reordering them at random is as likely to produce
+ the data as the order we observed. Shuffling many times and counting how often chance
+ alone explains a split at least as good as the observed one is therefore a direct
+ estimate of how surprising that split is.
+
+ The points are exchangeable, but not the measurements behind them, so `analyze`
+ combines each point down to one value before any of this runs.
+
+ The count is offset by one on both sides, which keeps the estimate unbiased and
+ avoids reporting a p-value of exactly zero, which no finite number of permutations
+ can justify.
+ """
+ atypical = 0
+ shuffled = values.copy()
+ for _ in range(permutations):
+ for (first, last) in segments:
+ shuffled[first:last] = rng.permutation(values[first:last])
+ # Distances between points of different segments end up in the prefix sums, but
+ # are never looked at: every block a split needs lies within a single segment.
+ prefix = cumulative_distances(shuffled)
+ splits = [best_split(prefix, first, last, min_points) for (first, last) in segments]
+ if max(split.statistic for split in splits if split is not None) >= observed:
+ atypical += 1
+ return (atypical + 1) / (permutations + 1)
+
+
+def detect(values: numpy.ndarray, labels: Sequence[str],
+ settings: Settings, rng: numpy.random.Generator) -> List[Candidate]:
+ """
+ Return the points at which the series changes, oldest first.
+
+ Changepoints are found one at a time: the most likely one anywhere in the series is
+ tested, and if it survives, the segment containing it is divided in two and the search
+ continues inside both halves. This allows the method to find several changes without
+ knowing how many to look for in the first place.
+
+ The search stops as soon as the best remaining candidate fails its test. Because the
+ statistic is scaled to be comparable across segments, that candidate is the one most
+ likely to have survived, so the ones it was preferred over would not have survived either.
+ """
+ prefix = cumulative_distances(values)
+ segments = [(0, len(values))]
+ found: List[Candidate] = []
+ while True:
+ best: Optional[Tuple[Split, Tuple[int, int]]] = None
+ for (first, last) in segments:
+ split = best_split(prefix, first, last, settings.min_points)
+ if split is None:
+ continue
+ if best is None or split.statistic > best[0].statistic:
+ best = (split, (first, last))
+ if best is None:
+ logging.debug('no segment is long enough to be divided further')
+ break
+
+ (split, (first, last)) = best
+ p_value = significance(values, segments, settings.min_points,
+ split.statistic, settings.permutations, rng)
+ label = labels[split.point]
+ if p_value > settings.p_value:
+ logging.debug(f'stopping: the best remaining candidate ({label}, statistic '
+ f'{split.statistic:.4g}) has p-value {p_value:.4g}')
+ break
+ logging.debug(f'accepted a changepoint at {label} (statistic {split.statistic:.4g}, '
+ f'p-value {p_value:.4g}), dividing points [{first}, {last})')
+ found.append(Candidate(split.point, p_value))
+ segments.remove((first, last))
+ segments += [(first, split.point), (split.point, last)]
+
+ return sorted(found)
+
+
+def describe(series: Sequence[Point], values: numpy.ndarray,
+ found: Sequence[Candidate]) -> List[Changepoint]:
+ """
+ Turn the detected points into reportable changes.
+ """
+ boundaries = [0] + [candidate.point for candidate in found] + [len(series)]
+ changes: List[Changepoint] = []
+ for (start, candidate, stop) in zip(boundaries, found, boundaries[2:]):
+ before = float(numpy.median(values[start:candidate.point]))
+ after = float(numpy.median(values[candidate.point:stop]))
+ changes.append(Changepoint(before_x=series[candidate.point - 1].label,
+ after_x=series[candidate.point].label,
+ before_y=before,
+ after_y=after,
+ change=(after - before) / before,
+ p_value=candidate.p_value))
+ return changes
+
+
+def analyze(series: Sequence[Point], settings: Settings,
+ rng: numpy.random.Generator) -> List[Changepoint]:
+ """Find and describe the changes in a series."""
+ if len(series) < 2 * settings.min_points:
+ logging.warning(f'the series has {len(series)} points, which is fewer than the '
+ f'{2 * settings.min_points} needed to divide it into two segments of '
+ f'--min-points; not looking for changepoints')
+ return []
+
+ aggregator = AGGREGATORS[settings.aggregate]
+ values: numpy.ndarray = numpy.array([aggregator(point.samples) for point in series], dtype=float)
+
+ labels = [point.label for point in series]
+ return describe(series, values, detect(values, labels, settings, rng))
+
+
+def serialize(changes: Sequence[Changepoint]) -> str:
+ """Render the changes as one JSON object per line."""
+ return ''.join(json.dumps(change._asdict(), sort_keys=True) + '\n' for change in changes)
+
+
+def report(series: Sequence[Point], changes: Sequence[Changepoint],
+ settings: Settings) -> None:
+ """Print a human-readable summary of the series and of what was found in it."""
+ out = sys.stderr
+ samples = sum(len(point.samples) for point in series)
+ print(f'Series: {len(series)} points, {samples} samples '
+ f'({samples / len(series):.1f} per point)', file=out)
+ if samples > len(series):
+ print(f'Combining the measurements of each point with their '
+ f'{settings.aggregate}', file=out)
+ print(f'Detected {len(changes)} changepoint(s) at p <= {settings.p_value} using '
+ f'{settings.permutations} permutations, with at least {settings.min_points} points '
+ f'per segment', file=out)
+ if not changes:
+ return
+
+ print('', file=out)
+ rows = [(f'{index}', change.before_x[:12], change.after_x[:12],
+ f'{change.before_y:.4g}', f'{change.after_y:.4g}',
+ f'{change.change:+.2%}', f'{change.p_value:.4g}')
+ for (index, change) in enumerate(changes, start=1)]
+ print(tabulate.tabulate(rows, headers=['#', 'BETWEEN', 'AND', 'BEFORE', 'AFTER',
+ 'CHANGE', 'P-VALUE'],
+ tablefmt='plain', disable_numparse=True,
+ colalign=('right', 'left', 'left', 'right', 'right', 'right', 'right')),
+ file=out)
+
+
+def main(argv: List[str]) -> int:
+ parser = argparse.ArgumentParser(
+ prog='detect-changepoints',
+ description='Find the points at which a benchmark time series changed, using the '
+ 'E-Divisive method.\n'
+ '\n'
+ 'The changepoints are written to standard output as one JSON object per line, '
+ 'including the two points the change falls between, the relative change and its '
+ 'p-value.\n'
+ '\n'
+ 'The method used by this script is described in "A Nonparametric Approach for '
+ 'Multiple Change Point Analysis of Multivariate Data" (Matteson & James, 2014, '
+ 'https://arxiv.org/abs/1306.4933).',
+ epilog='This script depends on the modules listed in `libcxx/utils/ci/lnt/requirements.txt`.',
+ formatter_class=HelpFormatter)
+ parser.add_argument('--input', type=argparse.FileType('r'), default='-', metavar='FILE',
+ help='The JSONL file holding the series, as one JSON object per measurement. Only the fields '
+ 'denoted by `--x` and `--y` are required in the input objects. If the input contains '
+ 'multiple samples for a point, --aggregate must be provided. Defaults to standard input.')
+ parser.add_argument('--x', default='commit', metavar='FIELD',
+ help='The field that labels each point.')
+ parser.add_argument('--y', default='value', metavar='FIELD',
+ help='The field holding the measurement (this must be a number).')
+ parser.add_argument('--order-by', default='ordinal', metavar='FIELD',
+ help='A field that establishes a total order between the points. If not provided, '
+ 'the order of the points in the input is used.')
+ parser.add_argument('--aggregate', choices=sorted(AGGREGATORS), default=None,
+ help='How to combine repeated samples of the same point into the single value that '
+ 'E-Divisive is run over. This must be provided explicitly when the input contains '
+ 'multiple samples for at least one point.')
+ parser.add_argument('--p-value', type=float, default=0.05,
+ help='How unlikely a change must be, under the hypothesis that nothing changed, for it '
+ 'to be reported. Lowering this causes the script to report fe...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/233656
More information about the libcxx-commits
mailing list