[all-commits] [llvm/llvm-project] d9f6db: [libc++] Compute a confidence interval in compare-...

[Louis Dionne via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: d9f6dbca81714fd5ee1c5aaf656fed6b0b5a4c01
      https://github.com/llvm/llvm-project/commit/d9f6dbca81714fd5ee1c5aaf656fed6b0b5a4c01
  Author: Louis Dionne <ldionne.2 at gmail.com>
  Date:   2026-07-14 (Tue, 14 Jul 2026)

  Changed paths:
    M libcxx/utils/compare-benchmarks
    M libcxx/utils/requirements.txt

  Log Message:
  -----------
  [libc++] Compute a confidence interval in compare-benchmarks (#208090)

When comparing benchmark results with more than one sample per
benchmark, compute a confidence interval and flag rows that are
statistically significant. This should make the A/B comparison PR job
more robust to noise and easier to rely on. After this change, output
for a multi-sample run looks like:

```
Benchmark                         Baseline    Candidate    Difference    % Difference   Significant?   95% C.I. of %diff
------------------------------  ----------  -----------  ------------  --------------  --------------  -------------------
std::any_of(list<int>)/32            39.92        39.22         -0.70          -1.75%                  [-6.5%, +1.6%]
std::any_of(list<int>)/32768      54071.34     65395.51      11324.17          20.94%                  [-2.8%, +34.5%]
std::any_of(list<int>)/50            67.39        69.08          1.69           2.51%        x         [+0.2%, +4.7%]
std::any_of(list<int>)/8              7.21         6.46         -0.75         -10.40%        x         [-25.0%, -0.7%]
std::any_of(list<int>)/8192       21745.58     22299.32        553.74           2.55%                  [-20.3%, +38.4%]
std::any_of(vector<int>)/32          24.21        14.61         -9.60         -39.65%        x         [-41.2%, -38.4%]
std::any_of(vector<int>)/32768    24365.14     12498.21     -11866.93         -48.70%        x         [-48.9%, -48.6%]
Geomean                             324.15       247.83        -76.32         -23.54%
```

The confidence interval is computed based on the mean of the N samples,
but we still present the median of N in the results. Because of that,
the result is not centered within the confidence interval, and we must
report it as [low, high] instead of a single number. The general idea is
that if 0% falls within the confidence interval, we can't really be
certain that the result changed at all. Similarly, if significant
negative and positive % diffs are in the confidence interval, we can't
be confident that we are introducing an optimization, regression, or any
change at all.

I think this approach is reasonable given the small number of samples
(usually around 3), which severely limits the statistical methods
available to us.

Assisted by Claude



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications