<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 1 Mar 2017, at 08:36, Mikael Holmén via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">Hi,<br class="">
<br class="">
Thank you to everyone that responded.<br class="">
<br class="">
Good hints!<br class="">
<br class="">
Now, I'm sure that I haven't read every piece of documentation about the test suite, but don't you think the tips and tricks you've responded with here should make into the quick start web page to help the next test-suite newbie that wants to run this and get
stable results?<br class="">
</div>
</blockquote>
<div><br class="">
</div>
<div>It definitely should.</div>
<div>It's sometimes hard for the non-newbies to figure out what documentation is missing the most, so thank you very much for pointing this out!</div>
<div><br class="">
</div>
<div>I've added some documentation in the patch under review at <a href="https://reviews.llvm.org/D30488" class="">https://reviews.llvm.org/D30488</a>.</div>
<div>Please have a look and leave your comments. I'll leave the patch in review until the end of the week before committing it.</div>
<div><br class="">
</div>
<div>Thanks,</div>
<div><br class="">
</div>
<div>Kristof</div>
<br class="">
<blockquote type="cite" class="">
<div class=""><br class="">
E.g. at<br class="">
<a href="http://llvm.org/docs/lnt/quickstart.html" class="">http://llvm.org/docs/lnt/quickstart.html</a><br class="">
for the lnt way?<br class="">
<br class="">
There could be a bullet 3 under "Running Tests" or just some extra proposed flags under "2" to describe a few things one could do if the results bounce around a lot.<br class="">
<br class="">
Thanks again,<br class="">
Mikael<br class="">
<br class="">
On 02/27/2017 09:46 AM, Mikael Holmén via llvm-dev wrote:<br class="">
<blockquote type="cite" class="">Hi,<br class="">
<br class="">
I'm trying to run the benchmark suite:<br class="">
<a href="http://llvm.org/docs/TestingGuide.html#test-suite-quickstart" class="">http://llvm.org/docs/TestingGuide.html#test-suite-quickstart</a><br class="">
<br class="">
I'm doing it the lnt way, as described at:<br class="">
<a href="http://llvm.org/docs/lnt/quickstart.html" class="">http://llvm.org/docs/lnt/quickstart.html</a><br class="">
<br class="">
I don't know what to expect but the results seems to be quite noisy and<br class="">
unstable. E.g I've done two runs on two different commits that only<br class="">
differ by a space in CODE_OWNERS.txt on my 12 core ubuntu 14.04 machine<br class="">
with:<br class="">
<br class="">
lnt runtest nt --sandbox SANDBOX --cc <path-to-my-clang> --test-suite<br class="">
/data/repo/test-suite -j 8<br class="">
<br class="">
And then I get the following top execution time regressions:<br class="">
<a href="http://i.imgur.com/sv1xzlK.png" class="">http://i.imgur.com/sv1xzlK.png</a><br class="">
<br class="">
The numbers bounce around a lot if I do more runs.<br class="">
<br class="">
Given the amount of noise I see here I don't know to sort out<br class="">
significant regressions if I actually do a real change in the compiler.<br class="">
<br class="">
Are the above results expected?<br class="">
<br class="">
How to use this?<br class="">
<br class="">
<br class="">
As a bonus question, if I instead run the benchmarks with an added -m32:<br class="">
lnt runtest nt --sandbox SANDBOX --cflag=-m32 --cc <path-to-my-clang><br class="">
--test-suite /data/repo/test-suite -j 8<br class="">
<br class="">
I get three failures:<br class="">
<br class="">
--- Tested: 2465 tests --<br class="">
FAIL: MultiSource/Applications/ClamAV/clamscan.compile_time (1 of 2465)<br class="">
FAIL: MultiSource/Applications/ClamAV/clamscan.execution_time (494 of 2465)<br class="">
FAIL:<br class="">
MultiSource/Benchmarks/DOE-ProxyApps-C/XSBench/XSBench.execution_time<br class="">
(495 of 2465)<br class="">
<br class="">
Is this known/expected or do I do something stupid?<br class="">
<br class="">
Thanks,<br class="">
Mikael<br class="">
_______________________________________________<br class="">
LLVM Developers mailing list<br class="">
<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a><br class="">
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<br class="">
</blockquote>
<br class="">
_______________________________________________<br class="">
LLVM Developers mailing list<br class="">
<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a><br class="">
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<br class="">
</div>
</blockquote>
</div>
<br class="">
</body>
</html>