[libcxx-commits] [libcxx] 0c27a66 - [libc++] Lower the default benchmark_min_time to 0.2s (#214499)
via libcxx-commits
libcxx-commits at lists.llvm.org
Fri Aug 7 05:47:44 PDT 2026
Author: Louis Dionne
Date: 2026-08-07T08:47:39-04:00
New Revision: 0c27a66627a5077c52e549dae787294c4570d4d0
URL: https://github.com/llvm/llvm-project/commit/0c27a66627a5077c52e549dae787294c4570d4d0
DIFF: https://github.com/llvm/llvm-project/commit/0c27a66627a5077c52e549dae787294c4570d4d0.diff
LOG: [libc++] Lower the default benchmark_min_time to 0.2s (#214499)
We were using GoogleBenchmark's default benchmark_min_time of 0.5s,
which makes the benchmark suite slow to run for little benefit. This
patch adds a benchmark_min_time Lit parameter and defaults it to 0.2s,
which runs the suite over 2x faster. It also bumps the PR benchmark job
from median-of-3 to median-of-5 to make up for the slightly noisier
samples.
To pick the value of 0.2s, every benchmark in the suite was run at 0.5s
and 0.2s back-to-back, 5 times. Over the whole benchmark suite:
```
median CV p90 p99
0.5s 0.40% 2.34% 16.93%
0.2s 0.53% 2.45% 17.03%
```
So while 0.2s is noisier, the noise increase sits mostly in the body of
the distribution instead of the tail. The tail of noisier benchmarks is
where the noise is already a problem for detecting regressions, but this
change doesn't have a large impact on that. I think that addressing
those noisy benchmarks directly (by e.g. rewriting or finding
alternative ways to benchmark the same thing) would be more effective.
Also, making it faster to run the benchmarks means that we can gather
more samples (i.e. do more "independent" runs on each commit), which
should reduce the noise more significantly than just running each
benchmark for longer.
Also note that going below 0.2s is not a clear win. Indeed, the speed
gain we obtain is not linear, since some benchmarks go past the
specified min time anyway, and the setup time still exists.
Assisted by Claude for the measurements and reasoning that led to
choosing 0.2s over other values.
Fixes #214055
Added:
Modified:
.github/workflows/libcxx-pr-benchmark.yml
libcxx/utils/libcxx/test/format.py
libcxx/utils/libcxx/test/params.py
Removed:
################################################################################
diff --git a/.github/workflows/libcxx-pr-benchmark.yml b/.github/workflows/libcxx-pr-benchmark.yml
index ef1bdc21d33a5..59e9a511ba5e0 100644
--- a/.github/workflows/libcxx-pr-benchmark.yml
+++ b/.github/workflows/libcxx-pr-benchmark.yml
@@ -158,21 +158,14 @@ jobs:
- name: Run baseline and candidate interleaved
run: |
source .venv/bin/activate
- # Run 3 times so we can pick the median, and interleave to mitigate the impact of environmental noise
- ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
- ./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee baseline.lnt
- ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
- ./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee candidate.lnt
-
- ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
- ./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee -a baseline.lnt
- ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
- ./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee -a candidate.lnt
-
- ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
- ./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee -a baseline.lnt
- ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
- ./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee -a candidate.lnt
+ # Run 5 times so we can pick the median, and interleave baseline and candidate to mitigate the impact of
+ # environmental noise
+ for _ in $(seq 1 5); do
+ ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
+ ./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee -a baseline.lnt
+ ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
+ ./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee -a candidate.lnt
+ done
- name: Compare baseline and candidate runs
run: |
diff --git a/libcxx/utils/libcxx/test/format.py b/libcxx/utils/libcxx/test/format.py
index e3e81fd1cf36f..2f449e234e6d5 100644
--- a/libcxx/utils/libcxx/test/format.py
+++ b/libcxx/utils/libcxx/test/format.py
@@ -363,7 +363,7 @@ def execute(self, test, litConfig):
"%dbg(COMPILED WITH) %{cxx} %s %{flags} %{compile_flags} %{benchmark_flags} %{link_flags} -o %t.exe",
]
if "enable-benchmarks=run" in test.config.available_features:
- steps += ["%dbg(EXECUTED AS) %{exec} %t.exe --benchmark_out=%{temp}/benchmark-result.json --benchmark_out_format=json"]
+ steps += ["%dbg(EXECUTED AS) %{exec} %t.exe --benchmark_min_time=%{benchmark_min_time} --benchmark_out=%{temp}/benchmark-result.json --benchmark_out_format=json"]
parse_results = os.path.join(LIBCXX_UTILS, 'parse-google-benchmark-results')
steps += [f"{parse_results} %{{temp}}/benchmark-result.json --output-format=lnt > %{{temp}}/results.lnt"]
return self._executeShTest(test, litConfig, steps)
diff --git a/libcxx/utils/libcxx/test/params.py b/libcxx/utils/libcxx/test/params.py
index 4a14ed0697b84..21c88e4ea667f 100644
--- a/libcxx/utils/libcxx/test/params.py
+++ b/libcxx/utils/libcxx/test/params.py
@@ -377,6 +377,17 @@ def getSuitableClangTidy(cfg):
help="Whether to run the benchmarks in the test suite, to only dry-run them or to disable them entirely.",
actions=lambda mode: [AddFeature(f"enable-benchmarks={mode}")],
),
+ Parameter(
+ name="benchmark_min_time",
+ type=str,
+ default="0.2s",
+ help="The minimum amount of time each benchmark is run for, passed to GoogleBenchmark's "
+ "--benchmark_min_time flag. By default, we use a lower value than GoogleBenchmark's "
+ "default of 0.5s because that speeds up the test suite without severely impacting "
+ "noise. This can be increased to get more precise results, but running the benchmark "
+ "suite several times may reduce noise more than increasing this threshold.",
+ actions=lambda min_time: [AddSubstitution("%{benchmark_min_time}", min_time)],
+ ),
Parameter(
name="spec_dir",
type=str,
More information about the libcxx-commits
mailing list