[libcxx-commits] [libcxx] [llvm] [libc++] Lower the default benchmark_min_time to 0.2s (PR #214499)

Thu Aug 6 07:12:54 PDT 2026

llvmorg-github-actions[bot] wrote:




@llvm/pr-subscribers-libcxx

Author: Louis Dionne (ldionne)

<details>
<summary>Changes</summary>

We were using GoogleBenchmark's default benchmark_min_time of 0.5s, which makes the benchmark suite slow to run for little benefit. This patch adds a benchmark_min_time Lit parameter and defaults it to 0.2s, which runs the suite over 2x faster. It also bumps the PR benchmark job from median-of-3 to median-of-5 to make up for the slightly noisier samples.

To pick the value of 0.2s, every benchmark in the suite was run at 0.5s and 0.2s back-to-back, 5 times. Over the whole benchmark suite:

```
              median CV    p90      p99
  0.5s          0.40%     2.34%   16.93%
  0.2s          0.53%     2.45%   17.03%
```

So while 0.2s is noisier, the noise increase sits mostly in the body of the distribution instead of the tail. The tail of noisier benchmarks is where the noise is already a problem for detecting regressions, but this change doesn't have a large impact on that. I think that addressing those noisy benchmarks directly (by e.g. rewriting or finding alternative ways to benchmark the same thing) would be more effective.

Also, making it faster to run the benchmarks means that we can gather more samples (i.e. do more "independent" runs on each commit), which should reduce the noise more significantly than just running each benchmark for longer.

Also note that going below 0.2s is not a clear win. Indeed, the speed gain we obtain is not linear, since some benchmarks go past the specified min time anyway, and the setup time still exists.

Assisted by Claude for the measurements and reasoning that led to choosing 0.2s over other values.

Fixes #214055

---
Full diff: https://github.com/llvm/llvm-project/pull/214499.diff


3 Files Affected:

- (modified) .github/workflows/libcxx-pr-benchmark.yml (+8-15) 
- (modified) libcxx/utils/libcxx/test/format.py (+1-1) 
- (modified) libcxx/utils/libcxx/test/params.py (+11) 


``````````diff

diff --git a/.github/workflows/libcxx-pr-benchmark.yml b/.github/workflows/libcxx-pr-benchmark.yml
index ef1bdc21d33a5..59e9a511ba5e0 100644
--- a/.github/workflows/libcxx-pr-benchmark.yml
+++ b/.github/workflows/libcxx-pr-benchmark.yml
@@ -158,21 +158,14 @@ jobs:
       - name: Run baseline and candidate interleaved
         run: |
           source .venv/bin/activate
-          # Run 3 times so we can pick the median, and interleave to mitigate the impact of environmental noise
-          ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
-          ./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee baseline.lnt
-          ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
-          ./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee candidate.lnt
-
-          ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
-          ./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee -a baseline.lnt
-          ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
-          ./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee -a candidate.lnt
-
-          ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
-          ./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee -a baseline.lnt
-          ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
-          ./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee -a candidate.lnt
+          # Run 5 times so we can pick the median, and interleave baseline and candidate to mitigate the impact of
+          # environmental noise
+          for _ in $(seq 1 5); do
+            ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
+            ./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee -a baseline.lnt
+            ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
+            ./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee -a candidate.lnt
+          done
 
       - name: Compare baseline and candidate runs
         run: |
diff --git a/libcxx/utils/libcxx/test/format.py b/libcxx/utils/libcxx/test/format.py
index e3e81fd1cf36f..2f449e234e6d5 100644
--- a/libcxx/utils/libcxx/test/format.py
+++ b/libcxx/utils/libcxx/test/format.py
@@ -363,7 +363,7 @@ def execute(self, test, litConfig):
                 "%dbg(COMPILED WITH) %{cxx} %s %{flags} %{compile_flags} %{benchmark_flags} %{link_flags} -o %t.exe",
             ]
             if "enable-benchmarks=run" in test.config.available_features:
-                steps += ["%dbg(EXECUTED AS) %{exec} %t.exe --benchmark_out=%{temp}/benchmark-result.json --benchmark_out_format=json"]
+                steps += ["%dbg(EXECUTED AS) %{exec} %t.exe --benchmark_min_time=%{benchmark_min_time} --benchmark_out=%{temp}/benchmark-result.json --benchmark_out_format=json"]
                 parse_results = os.path.join(LIBCXX_UTILS, 'parse-google-benchmark-results')
                 steps += [f"{parse_results} %{{temp}}/benchmark-result.json --output-format=lnt > %{{temp}}/results.lnt"]
             return self._executeShTest(test, litConfig, steps)
diff --git a/libcxx/utils/libcxx/test/params.py b/libcxx/utils/libcxx/test/params.py
index 909815694c2cc..6919f79deb679 100644
--- a/libcxx/utils/libcxx/test/params.py
+++ b/libcxx/utils/libcxx/test/params.py
@@ -378,6 +378,17 @@ def getSuitableClangTidy(cfg):
         help="Whether to run the benchmarks in the test suite, to only dry-run them or to disable them entirely.",
         actions=lambda mode: [AddFeature(f"enable-benchmarks={mode}")],
     ),
+    Parameter(
+        name="benchmark_min_time",
+        type=str,
+        default="0.2s",
+        help="The minimum amount of time each benchmark is run for, passed to GoogleBenchmark's "
+             "--benchmark_min_time flag. By default, we use a lower value than GoogleBenchmark's "
+             "default of 0.5s because that speeds up the test suite without severely impacting "
+             "noise. This can be increased to get more precise results, but running the benchmark "
+             "suite several times may reduce noise more than increasing this threshold.",
+        actions=lambda min_time: [AddSubstitution("%{benchmark_min_time}", min_time)],
+    ),
     Parameter(
         name="spec_dir",
         type=str,

``````````

</details>


https://github.com/llvm/llvm-project/pull/214499
