Some tips for benchmarking

[Matthias Braun via llvm-commits]

> On Oct 26, 2016, at 11:07 AM, Michael Kruse via llvm-commits <llvm-commits at lists.llvm.org> wrote:
> 
> 2016-10-20 4:39 GMT+02:00 Rafael Espíndola via llvm-commits
> <llvm-commits at lists.llvm.org <mailto:llvm-commits at lists.llvm.org>>:
>> After much pain I decided that instead of just running tests many
>> times I should put some effort in figuring out why perf numbers were
>> so hard to reproduce on my machine.
>> 
>> I was already disabling address space randomization and setting the
>> scaling_governor to performance. Other things I found
>> 
>> * Use https://github.com/lpechacek/cpuset to reserve a cpu for just
>> the program you are benchmarking.
>> * Disable the SMP pair (echo 0 > /sys/devices/system/cpu/cpuX/online)
>> of the cpu you will use for the benchmark.
>> * disable turbo mode (echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo).
Thanks for sharing! Getting a system stable for benchmarking is always a bit of a black art, I applaud the effort to document several of those non-obvious tricks (even though those tricks don't help me on my mac :-( )

>> 
>> The results are pretty awesome:
>> 
>> 3.012355647 seconds time elapsed         ( +-  0.01% )
> 
> Cool. Would it be possible to teach lnt runtests to do that by itself?

In my experience applying those automagically is dangerous. Among other things you need root access and get your machine into a really weird state.

I see the value in codifying the things in a script but I would NOT integrate this into LNT. Among other things it will probably be really hard to maintain and do the right for everyone. So yeah put the stuff in a script, but keep it separate from LNT and let users run it manually if they need it.

- Matthias
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20161026/652f1ca9/attachment.html>