[libcxx-commits] [libcxx] [libc++] Compute a confidence interval in compare-benchmarks (PR #208090)

[Louis Dionne via libcxx-commits]

ldionne wrote:

> I would do the inverse and only flag rows that have a high confidence/are statistically significant. I would assume most of the time there will be no significant difference.

Ah, I think we're coming at this from different angles! In my mind, we mostly use this job to measure changes that we know will actually be significant, so I would be tempted to say that your proposed approach would flag most rows (artificial example):

```
Benchmark                         Baseline    Candidate    Difference    % Difference  95% Confidence interval of %diff     High confidence?
------------------------------  ----------  -----------  ------------  --------------  ----------------------------------  -----------------
std::any_of(list<int>)/32            39.92        39.22         -0.70          -1.75%  [-6.5%, +1.6%]                              
std::any_of(list<int>)/32768      54071.34     65395.51      11324.17          20.94%  [-2.8%, +34.5%]                             
std::any_of(list<int>)/50            67.39        69.08          1.69           2.51%  [+0.2%, +4.7%]                              x
std::any_of(list<int>)/8              7.21         6.46         -0.75         -10.40%  [-25.0%, -0.7%]                             x
std::any_of(list<int>)/8192       21745.58     22299.32        553.74           2.55%  [-20.3%, +38.4%]                            
std::any_of(vector<int>)/32          24.21        14.61         -9.60         -39.65%  [-41.2%, -38.4%]                            x
std::any_of(vector<int>)/33         24.21        14.61         -9.60         -39.65%  [-41.2%, -38.4%]                             x
std::any_of(vector<int>)/34          24.21        14.61         -9.60         -39.65%  [-41.2%, -38.4%]                            x
std::any_of(vector<int>)/35          24.21        14.61         -9.60         -39.65%  [-41.2%, -38.4%]                            x
std::any_of(vector<int>)/36          24.21        14.61         -9.60         -39.65%  [-41.2%, -38.4%]                            x
std::any_of(vector<int>)/32768    24365.14     12498.21     -11866.93         -48.70%  [-48.9%, -48.6%]
Geomean                             324.15       247.83        -76.32         -23.54%
```

However, IIUC you're suggesting that the main use case here would be to validate that there's no regression, and in that case most of the rows would end up being neutral (or within the noise). Is that your thinking? I think it also makes sense, and indeed that would make it more useful to flag rows where there is a high-confidence change.

https://github.com/llvm/llvm-project/pull/208090