[libcxx-commits] [libcxx] [llvm] [libc++] Unify the benchmarks with the test suite (PR #101399)

[Louis Dionne via libcxx-commits]

ldionne wrote:

> After this patch were are the benchmark executables stored? When benchmarking I typically want to run the before and after version.

The benchmarks are stored in the same location as a normal test executable is stored. For example: `<build>/test/benchmarks/algorithms/Output/for_each.bench.cpp.dir/t.tmp.exe`.

However, the biggest benefit of this patch is that you don't need to care anymore. You can simply run a benchmark like you run a test, with `libcxx-lit <build> path/to/benchmark.bench.cpp`. So you don't need to know where exactly the executable is, just like when you run tests.

> 
> Are the benchmarks still executed in the benchmark CI configuration?

I removed that CI configuration. After this patch, the benchmarks are always dry-run as part of the test suite, but we don't have a dedicated CI job that runs the full benchmarks. We were not gaining anything from having that since we never looked at those results. Instead, as a follow-up I would like to work on setting up a dedicated machine that we can run the benchmarks on and ideally even have something to prevent regressions. That CI job could be triggered by a comment in the PR or something like that: we probably don't want to run it all the time since we'd just be wasting cycles.

https://github.com/llvm/llvm-project/pull/101399