[libcxx] [llvm] [libc++] Add a job that runs the benchmarks as part of the CI (PR #157567)

Louis Dionne via llvm-commits llvm-commits at lists.llvm.org
Thu Sep 11 10:56:05 PDT 2025


https://github.com/ldionne updated https://github.com/llvm/llvm-project/pull/157567

>From c7306f453076f0c8a962281ed406aad6ce6e21eb Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Mon, 8 Sep 2025 18:12:17 -0400
Subject: [PATCH 01/10] [libc++] Add a job that runs the benchmarks as part of
 the CI

This will make it easier to see the performance impact of individual
patches. This is a first step towards better performance tracking, but
many things still need to be figured out:
- We should get a dedicated set of machines to run these tests on
- We could probably reuse regularly-generated baseline results published
  to LNT instead of re-generating the baselines in this job every time.
---
 .github/workflows/libcxx-build-and-test.yaml | 22 ++++++++++++++++++++
 libcxx/utils/ci/run-buildbot                 | 18 ++++++++++++++++
 2 files changed, 40 insertions(+)

diff --git a/.github/workflows/libcxx-build-and-test.yaml b/.github/workflows/libcxx-build-and-test.yaml
index 2e6ff7f91b6fc..80da80bdc59e7 100644
--- a/.github/workflows/libcxx-build-and-test.yaml
+++ b/.github/workflows/libcxx-build-and-test.yaml
@@ -284,3 +284,25 @@ jobs:
       - name: Build and test
         run: |
           bash libcxx/utils/ci/run-buildbot ${{ matrix.config }}
+
+  benchmarks:
+    runs-on: llvm-premerge-libcxx-next-runners # TODO: This should run on a dedicated set of machines
+    # needs: [ stage3 ] # TODO: Run this at the end of the pipeline once we're done testing the job
+    steps:
+      - uses: actions/checkout at 08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+      - name: benchmarks
+        run: libcxx/utils/ci/run-buildbot benchmark-comparison
+        env:
+          CC: clang-22
+          CXX: clang++-22
+      - uses: actions/upload-artifact at 26f96dfa697d77e81fd5907df203aa23a56210a8 # v4.3.0
+        if: always()
+        with:
+          name: benchmarks-results
+          path: |
+            **/test-results.xml
+            **/*.abilist
+            **/CMakeConfigureLog.yaml
+            **/CMakeError.log
+            **/CMakeOutput.log
+            **/crash_diagnostics/*
diff --git a/libcxx/utils/ci/run-buildbot b/libcxx/utils/ci/run-buildbot
index 57ecf1e49dbf2..6a776b6645723 100755
--- a/libcxx/utils/ci/run-buildbot
+++ b/libcxx/utils/ci/run-buildbot
@@ -542,6 +542,24 @@ generic-optimized-speed)
     generate-cmake -C "${MONOREPO_ROOT}/libcxx/cmake/caches/Generic-optimized-speed.cmake"
     check-runtimes
 ;;
+benchmark-comparison)
+    clean
+    LIT_ARGS="-sv -j1 --param optimization=speed libcxx/test/benchmarks/hash.bench.cpp libcxx/test/benchmarks/join_view.bench.cpp"
+
+    baseline_commit=$(git merge-base main HEAD)
+    step "Building and running the baseline at ${baseline_commit}"
+    rm -rf "${BUILD_DIR}-baseline"
+    ./libcxx/utils/test-at-commit --commit ${baseline_commit} -B "${BUILD_DIR}-baseline" -- ${LIT_ARGS}
+
+    # Build and run the candidate
+    step "Building and running the candidate"
+    generate-cmake -C "${MONOREPO_ROOT}/libcxx/cmake/caches/Generic-optimized-speed.cmake"
+    ./libcxx/utils/libcxx-lit "${BUILD_DIR}" ${LIT_ARGS}
+
+    step "Comparison of the baseline and the candidate"
+    ./libcxx/utils/compare-benchmarks <(./libcxx/utils/consolidate-benchmarks "${BUILD_DIR}-baseline") \
+                                      <(./libcxx/utils/consolidate-benchmarks "${BUILD_DIR}")
+;;
 apple-configuration)
     clean
 

>From 567efcabb14f4b695a9b267b505443d0d592c05b Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Thu, 11 Sep 2025 12:25:40 -0400
Subject: [PATCH 02/10] Try to fix baseline commit

---
 libcxx/utils/ci/run-buildbot | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/libcxx/utils/ci/run-buildbot b/libcxx/utils/ci/run-buildbot
index 6a776b6645723..b888faed975da 100755
--- a/libcxx/utils/ci/run-buildbot
+++ b/libcxx/utils/ci/run-buildbot
@@ -546,10 +546,12 @@ benchmark-comparison)
     clean
     LIT_ARGS="-sv -j1 --param optimization=speed libcxx/test/benchmarks/hash.bench.cpp libcxx/test/benchmarks/join_view.bench.cpp"
 
-    baseline_commit=$(git merge-base main HEAD)
-    step "Building and running the baseline at ${baseline_commit}"
+    if [[ "${BASELINE_COMMIT}" == "" ]]; then
+        BASELINE_COMMIT=$(git merge-base main HEAD)
+    fi
+    step "Building and running the baseline at ${BASELINE_COMMIT}"
     rm -rf "${BUILD_DIR}-baseline"
-    ./libcxx/utils/test-at-commit --commit ${baseline_commit} -B "${BUILD_DIR}-baseline" -- ${LIT_ARGS}
+    ./libcxx/utils/test-at-commit --commit ${BASELINE_COMMIT} -B "${BUILD_DIR}-baseline" -- ${LIT_ARGS}
 
     # Build and run the candidate
     step "Building and running the candidate"

>From 9ac9dc92978a281f2e0d0e652aca74250be2858d Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Thu, 11 Sep 2025 12:31:40 -0400
Subject: [PATCH 03/10] Try again

---
 libcxx/utils/ci/run-buildbot | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/libcxx/utils/ci/run-buildbot b/libcxx/utils/ci/run-buildbot
index b888faed975da..373e298183d86 100755
--- a/libcxx/utils/ci/run-buildbot
+++ b/libcxx/utils/ci/run-buildbot
@@ -546,12 +546,15 @@ benchmark-comparison)
     clean
     LIT_ARGS="-sv -j1 --param optimization=speed libcxx/test/benchmarks/hash.bench.cpp libcxx/test/benchmarks/join_view.bench.cpp"
 
-    if [[ "${BASELINE_COMMIT}" == "" ]]; then
-        BASELINE_COMMIT=$(git merge-base main HEAD)
+    if [[ "${GITHUB_BASE_REF}" != "" ]]; then # triggered from a PR
+        baseline_commit=$(git merge-base ${GITHUB_BASE_REF} ${GITHUB_SHA})
+    else
+        baseline_commit=$(git merge-base main HEAD)
     fi
-    step "Building and running the baseline at ${BASELINE_COMMIT}"
+
+    step "Building and running the baseline at ${baseline_commit}"
     rm -rf "${BUILD_DIR}-baseline"
-    ./libcxx/utils/test-at-commit --commit ${BASELINE_COMMIT} -B "${BUILD_DIR}-baseline" -- ${LIT_ARGS}
+    ./libcxx/utils/test-at-commit --commit ${baseline_commit} -B "${BUILD_DIR}-baseline" -- ${LIT_ARGS}
 
     # Build and run the candidate
     step "Building and running the candidate"

>From f16b1a7988c3f0e90a75a3fce868f11d30672292 Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Thu, 11 Sep 2025 12:45:04 -0400
Subject: [PATCH 04/10] Try to fix the fetch

---
 .github/workflows/libcxx-build-and-test.yaml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.github/workflows/libcxx-build-and-test.yaml b/.github/workflows/libcxx-build-and-test.yaml
index 80da80bdc59e7..73e01611025ef 100644
--- a/.github/workflows/libcxx-build-and-test.yaml
+++ b/.github/workflows/libcxx-build-and-test.yaml
@@ -290,6 +290,8 @@ jobs:
     # needs: [ stage3 ] # TODO: Run this at the end of the pipeline once we're done testing the job
     steps:
       - uses: actions/checkout at 08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+        with:
+          fetch-depth: 0 # This job requires access to all the Git branches so it can diff against (usually) main
       - name: benchmarks
         run: libcxx/utils/ci/run-buildbot benchmark-comparison
         env:

>From 354e90385aaaa20031a10b47b1111012823c1284 Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Thu, 11 Sep 2025 13:05:07 -0400
Subject: [PATCH 05/10] Try fetching tags

---
 .github/workflows/libcxx-build-and-test.yaml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.github/workflows/libcxx-build-and-test.yaml b/.github/workflows/libcxx-build-and-test.yaml
index 73e01611025ef..cd125e7e1a52f 100644
--- a/.github/workflows/libcxx-build-and-test.yaml
+++ b/.github/workflows/libcxx-build-and-test.yaml
@@ -292,6 +292,7 @@ jobs:
       - uses: actions/checkout at 08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
         with:
           fetch-depth: 0 # This job requires access to all the Git branches so it can diff against (usually) main
+          fetch-tags: true
       - name: benchmarks
         run: libcxx/utils/ci/run-buildbot benchmark-comparison
         env:

>From 716aa962b46afcd9926de8f4818bf4c4d662addf Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Thu, 11 Sep 2025 13:15:59 -0400
Subject: [PATCH 06/10] Try to fix merge base

---
 libcxx/utils/ci/run-buildbot | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libcxx/utils/ci/run-buildbot b/libcxx/utils/ci/run-buildbot
index 373e298183d86..1b3d1a7861c48 100755
--- a/libcxx/utils/ci/run-buildbot
+++ b/libcxx/utils/ci/run-buildbot
@@ -547,7 +547,7 @@ benchmark-comparison)
     LIT_ARGS="-sv -j1 --param optimization=speed libcxx/test/benchmarks/hash.bench.cpp libcxx/test/benchmarks/join_view.bench.cpp"
 
     if [[ "${GITHUB_BASE_REF}" != "" ]]; then # triggered from a PR
-        baseline_commit=$(git merge-base ${GITHUB_BASE_REF} ${GITHUB_SHA})
+        baseline_commit=$(git merge-base refs/heads/${GITHUB_BASE_REF} ${GITHUB_SHA})
     else
         baseline_commit=$(git merge-base main HEAD)
     fi

>From a9b70dfc099e95cb838b78c05e83fc8720bb3e91 Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Thu, 11 Sep 2025 13:26:57 -0400
Subject: [PATCH 07/10] Try checkout main

---
 .github/workflows/libcxx-build-and-test.yaml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.github/workflows/libcxx-build-and-test.yaml b/.github/workflows/libcxx-build-and-test.yaml
index cd125e7e1a52f..6ede9a312e443 100644
--- a/.github/workflows/libcxx-build-and-test.yaml
+++ b/.github/workflows/libcxx-build-and-test.yaml
@@ -293,6 +293,7 @@ jobs:
         with:
           fetch-depth: 0 # This job requires access to all the Git branches so it can diff against (usually) main
           fetch-tags: true
+          ref: main
       - name: benchmarks
         run: libcxx/utils/ci/run-buildbot benchmark-comparison
         env:

>From dcc96726f024caa13f6334bcae0b2f63af8b146b Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Thu, 11 Sep 2025 13:39:00 -0400
Subject: [PATCH 08/10] Try to fix checkout

---
 .github/workflows/libcxx-build-and-test.yaml | 2 +-
 libcxx/utils/ci/run-buildbot                 | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/libcxx-build-and-test.yaml b/.github/workflows/libcxx-build-and-test.yaml
index 6ede9a312e443..b30e126181a83 100644
--- a/.github/workflows/libcxx-build-and-test.yaml
+++ b/.github/workflows/libcxx-build-and-test.yaml
@@ -293,7 +293,7 @@ jobs:
         with:
           fetch-depth: 0 # This job requires access to all the Git branches so it can diff against (usually) main
           fetch-tags: true
-          ref: main
+          # ref: main
       - name: benchmarks
         run: libcxx/utils/ci/run-buildbot benchmark-comparison
         env:
diff --git a/libcxx/utils/ci/run-buildbot b/libcxx/utils/ci/run-buildbot
index 1b3d1a7861c48..3daf524503710 100755
--- a/libcxx/utils/ci/run-buildbot
+++ b/libcxx/utils/ci/run-buildbot
@@ -547,7 +547,7 @@ benchmark-comparison)
     LIT_ARGS="-sv -j1 --param optimization=speed libcxx/test/benchmarks/hash.bench.cpp libcxx/test/benchmarks/join_view.bench.cpp"
 
     if [[ "${GITHUB_BASE_REF}" != "" ]]; then # triggered from a PR
-        baseline_commit=$(git merge-base refs/heads/${GITHUB_BASE_REF} ${GITHUB_SHA})
+        baseline_commit=$(git merge-base refs/remotes/origin/${GITHUB_BASE_REF} ${GITHUB_SHA})
     else
         baseline_commit=$(git merge-base main HEAD)
     fi

>From 900f6b520de8746db135db7eff4bfcb2b78edaae Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Thu, 11 Sep 2025 13:48:18 -0400
Subject: [PATCH 09/10] Install python requirements'

---
 .github/workflows/libcxx-build-and-test.yaml | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/libcxx-build-and-test.yaml b/.github/workflows/libcxx-build-and-test.yaml
index b30e126181a83..7cb5c0609d56d 100644
--- a/.github/workflows/libcxx-build-and-test.yaml
+++ b/.github/workflows/libcxx-build-and-test.yaml
@@ -293,9 +293,12 @@ jobs:
         with:
           fetch-depth: 0 # This job requires access to all the Git branches so it can diff against (usually) main
           fetch-tags: true
-          # ref: main
       - name: benchmarks
-        run: libcxx/utils/ci/run-buildbot benchmark-comparison
+        run: |
+          python3 -m venv .venv
+          source .venv/bin/activate
+          python -m pip install -r libcxx/utils/requirements.txt
+          libcxx/utils/ci/run-buildbot benchmark-comparison
         env:
           CC: clang-22
           CXX: clang++-22

>From 38eab49b1af376b7b76379a47f1fc87c07e97dc2 Mon Sep 17 00:00:00 2001
From: Louis Dionne <ldionne.2 at gmail.com>
Date: Thu, 11 Sep 2025 13:55:51 -0400
Subject: [PATCH 10/10] Try to fix issue with venv

---
 .github/workflows/libcxx-build-and-test.yaml | 2 ++
 libcxx/utils/ci/Dockerfile                   | 1 +
 2 files changed, 3 insertions(+)

diff --git a/.github/workflows/libcxx-build-and-test.yaml b/.github/workflows/libcxx-build-and-test.yaml
index 7cb5c0609d56d..1ea99abaa7c8e 100644
--- a/.github/workflows/libcxx-build-and-test.yaml
+++ b/.github/workflows/libcxx-build-and-test.yaml
@@ -294,7 +294,9 @@ jobs:
           fetch-depth: 0 # This job requires access to all the Git branches so it can diff against (usually) main
           fetch-tags: true
       - name: benchmarks
+        # TODO: python3-venv should be taken from the Docker image instead
         run: |
+          sudo apt-get install -y python3-venv
           python3 -m venv .venv
           source .venv/bin/activate
           python -m pip install -r libcxx/utils/requirements.txt
diff --git a/libcxx/utils/ci/Dockerfile b/libcxx/utils/ci/Dockerfile
index 79e11569c0d08..03c830499a0fd 100644
--- a/libcxx/utils/ci/Dockerfile
+++ b/libcxx/utils/ci/Dockerfile
@@ -100,6 +100,7 @@ RUN sudo apt-get update \
         python3-packaging \
         python3-setuptools \
         python3-psutil \
+        python3-venv \
         software-properties-common \
         swig \
         unzip \



More information about the llvm-commits mailing list