[llvm-dev] Noisy benchmark results?

[Mikael Holmén via llvm-dev]

Hi,

Thank you to everyone that responded.

Good hints!

Now, I'm sure that I haven't read every piece of documentation about the 
test suite, but don't you think the tips and tricks you've responded 
with here should make into the quick start web page to help the next 
test-suite newbie that wants to run this and get stable results?

E.g. at
  http://llvm.org/docs/lnt/quickstart.html
for the lnt way?

There could be a bullet 3 under "Running Tests" or just some extra 
proposed flags under "2" to describe a few things one could do if the 
results bounce around a lot.

Thanks again,
Mikael

On 02/27/2017 09:46 AM, Mikael Holmén via llvm-dev wrote:
> Hi,
>
> I'm trying to run the benchmark suite:
>  http://llvm.org/docs/TestingGuide.html#test-suite-quickstart
>
> I'm doing it the lnt way, as described at:
>  http://llvm.org/docs/lnt/quickstart.html
>
> I don't know what to expect but the results seems to be quite noisy and
> unstable. E.g I've done two runs on two different commits that only
> differ by a space in CODE_OWNERS.txt on my 12 core ubuntu 14.04 machine
> with:
>
>  lnt runtest nt --sandbox SANDBOX --cc <path-to-my-clang> --test-suite
> /data/repo/test-suite -j 8
>
> And then I get the following top execution time regressions:
>  http://i.imgur.com/sv1xzlK.png
>
> The numbers bounce around a lot if I do more runs.
>
> Given the amount of noise I see here I don't know to sort out
> significant regressions if I actually do a real change in the compiler.
>
> Are the above results expected?
>
> How to use this?
>
>
> As a bonus question, if I instead run the benchmarks with an added -m32:
>  lnt runtest nt --sandbox SANDBOX --cflag=-m32 --cc <path-to-my-clang>
> --test-suite /data/repo/test-suite -j 8
>
> I get three failures:
>
> --- Tested: 2465 tests --
> FAIL: MultiSource/Applications/ClamAV/clamscan.compile_time (1 of 2465)
> FAIL: MultiSource/Applications/ClamAV/clamscan.execution_time (494 of 2465)
> FAIL:
> MultiSource/Benchmarks/DOE-ProxyApps-C/XSBench/XSBench.execution_time
> (495 of 2465)
>
> Is this known/expected or do I do something stupid?
>
> Thanks,
> Mikael
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev