[all-commits] [llvm/llvm-project] e236a5: [libc++] Unify the benchmarks with the test suite ...

[Louis Dionne via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: e236a52a88956968f318fb908c584e5cb80b5b03
      https://github.com/llvm/llvm-project/commit/e236a52a88956968f318fb908c584e5cb80b5b03
  Author: Louis Dionne <ldionne.2 at gmail.com>
  Date:   2024-11-07 (Thu, 07 Nov 2024)

  Changed paths:
    M .github/workflows/libcxx-build-and-test.yaml
    M libcxx/CMakeLists.txt
    M libcxx/docs/TestingLibcxx.rst
    M libcxx/docs/VendorDocumentation.rst
    M libcxx/test/CMakeLists.txt
    M libcxx/test/benchmarks/CMakeLists.txt
    M libcxx/test/benchmarks/algorithms/min.bench.cpp
    M libcxx/test/benchmarks/atomic_wait_vs_mutex_lock.bench.cpp
    M libcxx/test/benchmarks/formatter_int.bench.cpp
    R libcxx/test/benchmarks/lit.cfg.py.in
    R libcxx/test/benchmarks/lit.site.cfg.py.in
    M libcxx/test/benchmarks/util_smartptr.bench.cpp
    M libcxx/test/configs/cmake-bridge.cfg.in
    M libcxx/utils/ci/buildkite-pipeline.yml
    M libcxx/utils/ci/run-buildbot
    M libcxx/utils/libcxx/test/config.py
    M libcxx/utils/libcxx/test/format.py
    R libcxx/utils/libcxx/test/googlebenchmark.py
    M libcxx/utils/libcxx/test/params.py
    M libcxxabi/test/configs/cmake-bridge.cfg.in
    M libunwind/test/configs/cmake-bridge.cfg.in

  Log Message:
  -----------
  [libc++] Unify the benchmarks with the test suite (#101399)

Instead of building the benchmarks separately via CMake and running them
separately from the test suite, this patch merges the benchmarks into
the test suite and handles both uniformly.

As a result:
- It is now possible to run individual benchmarks like we run tests
  (e.g. using libcxx-lit), which is a huge quality-of-life improvement.

- The benchmarks will be run under exactly the same configuration as
  the rest of the tests, which is a nice simplification. This does
  mean that one has to be careful to enable the desired optimization
  flags when running benchmarks, but that is easy with e.g.
  `libcxx-lit <...> --param optimization=speed`.

- Benchmarks can use the same annotations as the rest of the test
  suite, such as `// UNSUPPORTED` & friends.

When running the tests via `check-cxx`, we only compile the benchmarks
because running them would be too time consuming. This introduces a bit
of complexity in the testing setup, and instead it would be better to
allow passing a --dry-run flag to GoogleBenchmark executables, which is
the topic of https://github.com/google/benchmark/issues/1827.

I am not really satisfied with the layering violation of adding the
%{benchmark_flags} substitution to cmake-bridge, however I believe
this can be improved in the future.



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications