[llvm-dev] Questions About LLVM Test Suite: Time Units, Re-running benchmarks

[Stefanos Baziotis via llvm-dev]

Hi,

I'm not very familiar with the LLVM test suite and I'd like to ask some
questions.
I wanted to get a feeling of the impact to runtime performance of some
changes inside LLVM,
so I thought of running llvm test-suite benchmarks.

Build options: O3.cmake + DTEST_SUITE_BENCHMARKING_ONLY=True
Run:
llvm-lit -v -j 1 -o out.json .

What I think I should be looking for is the "metrics" -> "exec_time" in the
JSON file.

Now, to the questions. First, there doesn't seem to be a common time unit
for
"exec_time" among the different tests. For instance, SingleSource/ seem to
use
seconds while MicroBenchmarks seem to use μs. So, we can't reliably judge
changes. Although I get the fact that micro-benchmarks are different in
nature
than Single/MultiSource benchmarks, so maybe one should focus only on
the one or the other depending on what they're interested in.

In any case, it would at least be great if the JSON data contained the time
unit per test,
but that is not happening either.

Do you think that the lack of time unit info is a problem ? If yes, do you
like the
solution of adding the time unit in the JSON or do you want to propose an
alternative?

The second question has to do with re-running the benchmarks: I do
cmake + make + llvm-lit -v -j 1 -o out.json .
but if I try to do the latter another time, it just does/shows nothing. Is
there any reason
that the benchmarks can't be run a second time? Could I somehow run it a
second time ?

Lastly, slightly off-topic but while we're on the subject of benchmarking,
do you think it's reliable to run with -j <number of cores> ? I'm a little
bit afraid of
the shared caches (because misses should be counted in the CPU time, which
is what is measured in "exec_time" AFAIU)
and any potential multi-threading that the tests may use.

Best,
Stefanos
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210718/44eb79fe/attachment.html>