[libcxx-commits] [libcxx] [llvm] [libc++] Unify the benchmarks with the test suite (PR #101399)

[Mark de Wever via libcxx-commits]

mordante wrote:

> > After this patch were are the benchmark executables stored? When benchmarking I typically want to run the before and after version.
> 
> The benchmarks are stored in the same location as a normal test executable is stored. For example: `<build>/test/benchmarks/algorithms/Output/for_each.bench.cpp.dir/t.tmp.exe`.
> 
> However, the biggest benefit of this patch is that you don't need to care anymore. You can simply run a benchmark like you run a test, with `libcxx-lit <build> path/to/benchmark.bench.cpp`. So you don't need to know where exactly the executable is, just like when you run tests.

I do care. I sometimes like to compare benchmarks from different iterations. Currently when I build a benchmark I get a message where the executable is stored. The new approach makes harder to find the location of the executable. I wonder whether we can show a message where the benchmark is stored, maybe always print the location in `format.py`?

In the case I only want to run it once the new approach is indeed nicer.

> > Are the benchmarks still executed in the benchmark CI configuration?
> 
> I removed that CI configuration. After this patch, the benchmarks are always dry-run as part of the test suite, but we don't have a dedicated CI job that runs the full benchmarks. We were not gaining anything from having that since we never looked at those results. Instead, as a follow-up I would like to work on setting up a dedicated machine that we can run the benchmarks on and ideally even have something to prevent regressions. That CI job could be triggered by a comment in the PR or something like that: we probably don't want to run it all the time since we'd just be wasting cycles.

I see.



https://github.com/llvm/llvm-project/pull/101399