[llvm-dev] Noisy benchmark results?

[Kristof Beyls via llvm-dev]

On 1 Mar 2017, at 08:36, Mikael Holmén via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:

Hi,

Thank you to everyone that responded.

Good hints!

Now, I'm sure that I haven't read every piece of documentation about the test suite, but don't you think the tips and tricks you've responded with here should make into the quick start web page to help the next test-suite newbie that wants to run this and get stable results?

It definitely should.
It's sometimes hard for the non-newbies to figure out what documentation is missing the most, so thank you very much for pointing this out!

I've added some documentation in the patch under review at https://reviews.llvm.org/D30488.
Please have a look and leave your comments. I'll leave the patch in review until the end of the week before committing it.

Thanks,

Kristof


E.g. at
http://llvm.org/docs/lnt/quickstart.html
for the lnt way?

There could be a bullet 3 under "Running Tests" or just some extra proposed flags under "2" to describe a few things one could do if the results bounce around a lot.

Thanks again,
Mikael

On 02/27/2017 09:46 AM, Mikael Holmén via llvm-dev wrote:
Hi,

I'm trying to run the benchmark suite:
http://llvm.org/docs/TestingGuide.html#test-suite-quickstart

I'm doing it the lnt way, as described at:
http://llvm.org/docs/lnt/quickstart.html

I don't know what to expect but the results seems to be quite noisy and
unstable. E.g I've done two runs on two different commits that only
differ by a space in CODE_OWNERS.txt on my 12 core ubuntu 14.04 machine
with:

lnt runtest nt --sandbox SANDBOX --cc <path-to-my-clang> --test-suite
/data/repo/test-suite -j 8

And then I get the following top execution time regressions:
http://i.imgur.com/sv1xzlK.png

The numbers bounce around a lot if I do more runs.

Given the amount of noise I see here I don't know to sort out
significant regressions if I actually do a real change in the compiler.

Are the above results expected?

How to use this?


As a bonus question, if I instead run the benchmarks with an added -m32:
lnt runtest nt --sandbox SANDBOX --cflag=-m32 --cc <path-to-my-clang>
--test-suite /data/repo/test-suite -j 8

I get three failures:

--- Tested: 2465 tests --
FAIL: MultiSource/Applications/ClamAV/clamscan.compile_time (1 of 2465)
FAIL: MultiSource/Applications/ClamAV/clamscan.execution_time (494 of 2465)
FAIL:
MultiSource/Benchmarks/DOE-ProxyApps-C/XSBench/XSBench.execution_time
(495 of 2465)

Is this known/expected or do I do something stupid?

Thanks,
Mikael
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170301/6fa60777/attachment.html>