[libcxx] [llvm] [libc++] Lower the default benchmark_min_time to 0.2s (PR #214499)

[Louis Dionne via llvm-commits]

https://github.com/ldionne created https://github.com/llvm/llvm-project/pull/214499

We were using GoogleBenchmark's default benchmark_min_time of 0.5s, which makes the benchmark suite slow to run for little benefit. This patch adds a benchmark_min_time Lit parameter and defaults it to 0.2s, which runs the suite over 2x faster. It also bumps the PR benchmark job from median-of-3 to median-of-5 to make up for the slightly noisier samples.

To pick the value of 0.2s, every benchmark in the suite was run at 0.5s and 0.2s back-to-back, 5 times. Over the whole benchmark suite:

```
              median CV    p90      p99
  0.5s          0.40%     2.34%   16.93%
  0.2s          0.53%     2.45%   17.03%
```

So while 0.2s is noisier, the noise increase sits mostly in the body of the distribution instead of the tail. The tail of noisier benchmarks is where the noise is already a problem for detecting regressions, but this change doesn't have a large impact on that. I think that addressing those noisy benchmarks directly (by e.g. rewriting or finding alternative ways to benchmark the same thing) would be more effective.

Also, making it faster to run the benchmarks means that we can gather more samples (i.e. do more "independent" runs on each commit), which should reduce the noise more significantly than just running each benchmark for longer.

Also note that going below 0.2s is not a clear win. Indeed, the speed gain we obtain is not linear, since some benchmarks go past the specified min time anyway, and the setup time still exists.

Assisted by Claude for the measurements and reasoning that led to choosing 0.2s over other values.

Fixes #214055

>From 57ac08d51b8eac84fbf0d9ff0049ecdb05209b3e Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Tue, 4 Aug 2026 18:21:33 -0400
Subject: [PATCH] [libc++] Lower the default benchmark_min_time to 0.2s

We were using GoogleBenchmark's default benchmark_min_time of 0.5s, which
makes the benchmark suite slow to run for little benefit. This patch adds
a benchmark_min_time Lit parameter and defaults it to 0.2s, which runs the
suite over 2x faster. It also bumps the PR benchmark job from median-of-3
to median-of-5 to make up for the slightly noisier samples.

To pick the value of 0.2s, every benchmark in the suite was run at 0.5s
and 0.2s back-to-back, 5 times. Over the whole benchmark suite:

              median CV    p90      p99
  0.5s          0.40%     2.34%   16.93%
  0.2s          0.53%     2.45%   17.03%

So while 0.2s is noisier, the noise increase sits mostly in the body of
the distribution instead of the tail. The tail of noisier benchmarks is
where the noise is already a problem for detecting regressions, but this
change doesn't have a large impact on that. I think that addressing those
noisy benchmarks directly (by e.g. rewriting or finding alternative ways
to benchmark the same thing) would be more effective.

Also, making it faster to run the benchmarks means that we can gather
more samples (i.e. do more "independent" runs on each commit), which
should reduce the noise more significantly than just running each
benchmark for longer.

Also note that going below 0.2s is not a clear win. Indeed, the speed gain
we obtain is not linear, since some benchmarks go past the specified min
time anyway, and the setup time still exists.

Assisted by Claude for the measurements and reasoning that led to choosing
0.2s over other values.

Fixes #214055
---
 .github/workflows/libcxx-pr-benchmark.yml | 23 ++++++++---------------
 libcxx/utils/libcxx/test/format.py        |  2 +-
 libcxx/utils/libcxx/test/params.py        | 11 +++++++++++
 3 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/.github/workflows/libcxx-pr-benchmark.yml b/.github/workflows/libcxx-pr-benchmark.yml
index ef1bdc21d33a5..59e9a511ba5e0 100644
--- a/.github/workflows/libcxx-pr-benchmark.yml
+++ b/.github/workflows/libcxx-pr-benchmark.yml
@@ -158,21 +158,14 @@ jobs:
       - name: Run baseline and candidate interleaved
         run: |
           source .venv/bin/activate
-          # Run 3 times so we can pick the median, and interleave to mitigate the impact of environmental noise
-          ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
-          ./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee baseline.lnt
-          ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
-          ./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee candidate.lnt
-
-          ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
-          ./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee -a baseline.lnt
-          ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
-          ./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee -a candidate.lnt
-
-          ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
-          ./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee -a baseline.lnt
-          ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
-          ./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee -a candidate.lnt
+          # Run 5 times so we can pick the median, and interleave baseline and candidate to mitigate the impact of
+          # environmental noise
+          for _ in $(seq 1 5); do
+            ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
+            ./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee -a baseline.lnt
+            ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate --compiler "${COMPILER}" -- -sv -j1 --param optimization=speed "$BENCHMARKS"
+            ./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee -a candidate.lnt
+          done
 
       - name: Compare baseline and candidate runs
         run: |
diff --git a/libcxx/utils/libcxx/test/format.py b/libcxx/utils/libcxx/test/format.py
index e3e81fd1cf36f..2f449e234e6d5 100644
--- a/libcxx/utils/libcxx/test/format.py
+++ b/libcxx/utils/libcxx/test/format.py
@@ -363,7 +363,7 @@ def execute(self, test, litConfig):
                 "%dbg(COMPILED WITH) %{cxx} %s %{flags} %{compile_flags} %{benchmark_flags} %{link_flags} -o %t.exe",
             ]
             if "enable-benchmarks=run" in test.config.available_features:
-                steps += ["%dbg(EXECUTED AS) %{exec} %t.exe --benchmark_out=%{temp}/benchmark-result.json --benchmark_out_format=json"]
+                steps += ["%dbg(EXECUTED AS) %{exec} %t.exe --benchmark_min_time=%{benchmark_min_time} --benchmark_out=%{temp}/benchmark-result.json --benchmark_out_format=json"]
                 parse_results = os.path.join(LIBCXX_UTILS, 'parse-google-benchmark-results')
                 steps += [f"{parse_results} %{{temp}}/benchmark-result.json --output-format=lnt > %{{temp}}/results.lnt"]
             return self._executeShTest(test, litConfig, steps)
diff --git a/libcxx/utils/libcxx/test/params.py b/libcxx/utils/libcxx/test/params.py
index 909815694c2cc..6919f79deb679 100644
--- a/libcxx/utils/libcxx/test/params.py
+++ b/libcxx/utils/libcxx/test/params.py
@@ -378,6 +378,17 @@ def getSuitableClangTidy(cfg):
         help="Whether to run the benchmarks in the test suite, to only dry-run them or to disable them entirely.",
         actions=lambda mode: [AddFeature(f"enable-benchmarks={mode}")],
     ),
+    Parameter(
+        name="benchmark_min_time",
+        type=str,
+        default="0.2s",
+        help="The minimum amount of time each benchmark is run for, passed to GoogleBenchmark's "
+             "--benchmark_min_time flag. By default, we use a lower value than GoogleBenchmark's "
+             "default of 0.5s because that speeds up the test suite without severely impacting "
+             "noise. This can be increased to get more precise results, but running the benchmark "
+             "suite several times may reduce noise more than increasing this threshold.",
+        actions=lambda min_time: [AddSubstitution("%{benchmark_min_time}", min_time)],
+    ),
     Parameter(
         name="spec_dir",
         type=str,