[libcxx-commits] [libcxx] [libc++] Improvements to the benchmark runners (PR #194659)

[via libcxx-commits]

llvmbot wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-libcxx

Author: Louis Dionne (ldionne)

<details>
<summary>Changes</summary>

- Run the ref workloads on SPEC
- Record the SPEC version in the machine info
- Allow filtering which benchmarks are run in run-benchbot

---
Full diff: https://github.com/llvm/llvm-project/pull/194659.diff


3 Files Affected:

- (modified) libcxx/test/benchmarks/spec.gen.py (+3-3) 
- (modified) libcxx/utils/ci/lnt/run-benchbot (+12) 
- (modified) libcxx/utils/ci/lnt/run-benchmarks (+4) 


``````````diff
diff --git a/libcxx/test/benchmarks/spec.gen.py b/libcxx/test/benchmarks/spec.gen.py
index ab546e53e0692..a76010b6e1460 100644
--- a/libcxx/test/benchmarks/spec.gen.py
+++ b/libcxx/test/benchmarks/spec.gen.py
@@ -76,7 +76,7 @@
     print(f'RUN: %{{spec_dir}}/bin/runcpu --config %{{temp}}/spec-config.cfg --action build --output_root %{{temp}} {benchmark}')
 
     # Run the benchmark
-    print(f'RUN: /usr/bin/time -l -o %{{temp}}/time.txt %{{spec_dir}}/bin/runcpu --config %{{temp}}/spec-config.cfg --action run --size train --output_root %{{temp}} {benchmark}')
+    print(f'RUN: /usr/bin/time -l -o %{{temp}}/time.txt %{{spec_dir}}/bin/runcpu --config %{{temp}}/spec-config.cfg --action run --size ref --output_root %{{temp}} {benchmark}')
 
     # Clean up, since there can be lots of content created
     print(f'RUN: rm -rf %{{temp}}/benchspec')
@@ -84,10 +84,10 @@
     # The `runcpu` command above doesn't fail even if the benchmark fails to run. To determine failure, parse the CSV
     # results and ensure there are no compilation errors or runtime errors in the status row. Also print the logs and
     # fail if there are no CSV files at all, which implies a SPEC error.
-    print(f'RUN: %{{libcxx-dir}}/utils/parse-spec-results --extract "Base Status" --keep-failed %{{temp}}/result/*.train.csv > %{{temp}}/status || ! cat %{{temp}}/result/*.log')
+    print(f'RUN: %{{libcxx-dir}}/utils/parse-spec-results --extract "Base Status" --keep-failed %{{temp}}/result/*.refrate.csv > %{{temp}}/status || ! cat %{{temp}}/result/*.log')
     print(f'RUN: ! grep -E "CE|RE" %{{temp}}/status || ! cat %{{temp}}/result/*.log')
 
     # If there were no errors, parse the SPEC results and the `time` output into LNT-compatible format and print them.
-    print(f'RUN: %{{libcxx-dir}}/utils/parse-spec-results %{{temp}}/result/*.train.csv --output-format=lnt > %{{temp}}/results.lnt')
+    print(f'RUN: %{{libcxx-dir}}/utils/parse-spec-results %{{temp}}/result/*.refrate.csv --output-format=lnt > %{{temp}}/results.lnt')
     print(f'RUN: %{{libcxx-dir}}/utils/parse-time-output %{{temp}}/time.txt --benchmark {benchmark.replace(".", "_")} --extract instructions max_rss cycles peak_memory >> %{{temp}}/results.lnt')
     print(f'RUN: cat %{{temp}}/results.lnt')
diff --git a/libcxx/utils/ci/lnt/run-benchbot b/libcxx/utils/ci/lnt/run-benchbot
index 266a064f4987f..52fb813b9321e 100755
--- a/libcxx/utils/ci/lnt/run-benchbot
+++ b/libcxx/utils/ci/lnt/run-benchbot
@@ -29,6 +29,8 @@ ${PROGNAME} [options] <BUILDER> [-- commit ...]
 [--lnt-url <URL>]   The optional URL of the LNT instance to submit results to. By
                     default, results are not submitted to any LNT instance.
 
+[--filter <FILTER>] Optional filter to pass to lit to run only a subset of benchmarks.
+
 Environment variables
 SPEC_DIR            Optional path to a SPEC installation. If set and the directory
                     exists, SPEC benchmarks are included in the run.
@@ -62,6 +64,10 @@ while [[ $# -gt 0 ]]; do
             LNT_URL="${2}"
             shift; shift
             ;;
+        --filter)
+            FILTER="${2}"
+            shift; shift
+            ;;
         --)
             shift
             COMMITS=("$@")
@@ -124,6 +130,11 @@ run_benchmarks() {
         spec_arg="--spec-dir ${SPEC_DIR}"
     fi
 
+    local filter_arg=""
+    if [ -n "${FILTER}" ]; then
+        filter_arg="--filter ${FILTER}"
+    fi
+
     local output=$(next_available_path "${BUILD_DIR}/${1}.json")
     ${MONOREPO_ROOT}/libcxx/utils/ci/lnt/run-benchmarks                                                         \
         --test-suite-commit ${BENCHMARK_SUITE_VERSION}                                                          \
@@ -132,6 +143,7 @@ run_benchmarks() {
         --compiler ${COMPILER}                                                                                  \
         --benchmark-commit ${1}                                                                                 \
         ${spec_arg}                                                                                             \
+        ${filter_arg}                                                                                           \
         --output ${output}
 
     if [ -n "${LNT_URL}" ]; then
diff --git a/libcxx/utils/ci/lnt/run-benchmarks b/libcxx/utils/ci/lnt/run-benchmarks
index e0d2c153f4f27..611bd9c4d45b9 100755
--- a/libcxx/utils/ci/lnt/run-benchmarks
+++ b/libcxx/utils/ci/lnt/run-benchmarks
@@ -38,6 +38,10 @@ def gather_machine_information(args):
     info['compiler'] = subprocess.check_output([args.compiler, '--version']).decode().strip().splitlines()[0]
     info['test_suite_commit'] = subprocess.check_output(['git', '-C', args.git_repo, 'rev-parse', args.test_suite_commit]).decode().strip()
 
+    if args.spec_dir is not None:
+        with open(args.spec_dir / 'version.txt', 'r') as f:
+            info['spec'] = f.read().strip()
+
     return info
 
 def gather_run_information(args):

``````````

</details>


https://github.com/llvm/llvm-project/pull/194659