[LLVMdev] Use perf tool for more accurate time measuring on Linux

[Yi Kong]

Also some screenshots. Both running on a quiet x86 machine.

Timeit: 10 samples per run(aggregated by minimum, 0.05 MWU significance level)
Perf stat: 8 samples per run(aggregated by minimum, 0.05 MWU significance level)

On 16 May 2014 17:37, Yi Kong <kongy.dev at gmail.com> wrote:
> Hi all,
>
> The LLVM benchmarking system produces very noisy results even on quiet
> machines. One of the sources of inaccuracy is the timing tool we are
> using. Because it is a user-space tool, the OS can context switch it
> and we will get an outlier result. Perf stat uses SW_TASK_CLOCK
> counter in kernel to measure time, therefore more accurate. It also
> does not get context switched.
>
> I've implemented a wrapper script over perf stat which mimics the
> behaviour of timeit tool in test suite, so that nothing else needs to
> be modified. The script is not yet feature complete as timeit, but
> enough to run nightly tests.
>
> I carried out experiments on several machines and saw different level
> of improvements. I am no longer seeing outlier results, and MAD is
> considerably lower. The run-by-run changes results over the same
> revision shrank from around 10 to only 2-3. The MAD reduced from
> around 0.01 to 0.003 on a quiet machine.
>
> I've attached the patch and please experiment with it.
>
> Cheers,
> Yi Kong
-------------- next part --------------
A non-text attachment was scrubbed...
Name: perfit.png
Type: image/png
Size: 39723 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140516/6f1e46e5/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: timeit.png
Type: image/png
Size: 70089 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140516/6f1e46e5/attachment-0001.png>