[lld] Add lld benchmarking script. (PR #138367)

[Reid Kleckner via llvm-commits]

================
@@ -0,0 +1,131 @@
+#!/usr/bin/env python3
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ==------------------------------------------------------------------------==#
+
+import argparse
+import os
+import shutil
+import subprocess
+import tempfile
+
+# The purpose of this script is to measure the performance effect
+# of an lld change in a statistically sound way, automating all the
+# tedious parts of doing so. It copies the test case into /tmp as well as
+# running the test binaries from /tmp to reduce the influence on the test
+# machine's storage medium on the results. It accounts for measurement
+# bias caused by binary layout (using the --randomize-section-padding
+# flag to link the test binaries) and by environment variable size
+# (implemented by hyperfine [1]). Runs of the base and test case are
+# interleaved to account for environmental factors which may influence
+# the result due to the passage of time. The results of running hyperfine
+# are collected into a results.csv file in the output directory and may
+# be analyzed by the user with a tool such as ministat.
+#
+# Requirements: Linux host, hyperfine [2] in $PATH, run from a build directory
+# configured to use ninja and a recent version of lld that supports
+# --randomize-section-padding, /tmp is tmpfs.
+#
+# [1] https://github.com/sharkdp/hyperfine/blob/3cedcc38d0c430cbf38b4364b441c43a938d2bf3/src/util/randomized_environment_offset.rs#L1
+# [2] https://github.com/sharkdp/hyperfine
+#
+# Example invocation for comparing the performance of the current commit
+# against the previous commit which is treated as the baseline, without
+# linking debug info:
+#
+# lld/utils/run_benchmark.py \
+#   --base-commit HEAD^ \
+#   --test-commit HEAD \
+#   --test-case lld/utils/speed-test-reproducers/result/firefox-x64/response.txt \
+#   --num-iterations 512 \
+#   --num-binary-variants 16 \
+#   --output-dir outdir \
+#   --ldflags=-S
+#
+# Then this bash command will compare the real time of the base and test cases.
+#
+# ministat -A \
+#   <(grep lld-base outdir/results.csv | cut -d, -f2) \
+#   <(grep lld-test outdir/results.csv | cut -d, -f2)
+
+# We don't want to copy stat() information when we copy the reproducer
+# to the temporary directory. Files in the Nix store are read-only so this will
+# cause trouble when the linker writes the output file and when we want to clean
+# up the temporary directory. Python doesn't provide a way to disable copying
+# stat() information in shutil.copytree so we just monkeypatch shutil.copystat
+# to do nothing.
+shutil.copystat = lambda *args, **kwargs: 0
----------------
rnk wrote:

11/10

https://github.com/llvm/llvm-project/pull/138367