[llvm-dev] Noisy benchmark results?
Mikael Holmén via llvm-dev
llvm-dev at lists.llvm.org
Mon Feb 27 00:46:54 PST 2017
Hi,
I'm trying to run the benchmark suite:
http://llvm.org/docs/TestingGuide.html#test-suite-quickstart
I'm doing it the lnt way, as described at:
http://llvm.org/docs/lnt/quickstart.html
I don't know what to expect but the results seems to be quite noisy and
unstable. E.g I've done two runs on two different commits that only
differ by a space in CODE_OWNERS.txt on my 12 core ubuntu 14.04 machine
with:
lnt runtest nt --sandbox SANDBOX --cc <path-to-my-clang> --test-suite
/data/repo/test-suite -j 8
And then I get the following top execution time regressions:
http://i.imgur.com/sv1xzlK.png
The numbers bounce around a lot if I do more runs.
Given the amount of noise I see here I don't know to sort out
significant regressions if I actually do a real change in the compiler.
Are the above results expected?
How to use this?
As a bonus question, if I instead run the benchmarks with an added -m32:
lnt runtest nt --sandbox SANDBOX --cflag=-m32 --cc <path-to-my-clang>
--test-suite /data/repo/test-suite -j 8
I get three failures:
--- Tested: 2465 tests --
FAIL: MultiSource/Applications/ClamAV/clamscan.compile_time (1 of 2465)
FAIL: MultiSource/Applications/ClamAV/clamscan.execution_time (494 of 2465)
FAIL:
MultiSource/Benchmarks/DOE-ProxyApps-C/XSBench/XSBench.execution_time
(495 of 2465)
Is this known/expected or do I do something stupid?
Thanks,
Mikael
More information about the llvm-dev
mailing list