[LLVMdev] Use perf tool for more accurate time measuring on Linux

[Hal Finkel]

----- Original Message -----
> From: "Tobias Grosser" <tobias at grosser.es>
> To: "Yi Kong" <kongy.dev at gmail.com>, "Hal Finkel" <hfinkel at anl.gov>, "Renato Golin" <renato.golin at linaro.org>
> Cc: "LLVM Dev" <llvmdev at cs.uiuc.edu>
> Sent: Friday, May 16, 2014 1:13:43 PM
> Subject: Re: Use perf tool for more accurate time measuring on Linux
> 
> On 16/05/2014 18:37, Yi Kong wrote:
> > Hi all,
> >
> > The LLVM benchmarking system produces very noisy results even on
> > quiet
> > machines. One of the sources of inaccuracy is the timing tool we
> > are
> > using. Because it is a user-space tool, the OS can context switch
> > it
> > and we will get an outlier result. Perf stat uses SW_TASK_CLOCK
> > counter in kernel to measure time, therefore more accurate. It also
> > does not get context switched.
> >
> > I've implemented a wrapper script over perf stat which mimics the
> > behaviour of timeit tool in test suite, so that nothing else needs
> > to
> > be modified. The script is not yet feature complete as timeit, but
> > enough to run nightly tests.
> >
> > I carried out experiments on several machines and saw different
> > level
> > of improvements. I am no longer seeing outlier results, and MAD is
> > considerably lower. The run-by-run changes results over the same
> > revision shrank from around 10 to only 2-3. The MAD reduced from
> > around 0.01 to 0.003 on a quiet machine.
> >
> > I've attached the patch and please experiment with it.
> 
> I think this is a great idea. I installed perf on all my performance
> trackers. As you already experimented with it on x86, I would be fine
> with just committing the patch. We can adjust it if the results are
> not as expected.

This is a general adjustment for anything running the test suite on a Linux target, I'd wait until we have some testing on non-x86 platforms before we enable it by default for all architectures; and we have other x86 buildbots besides yours, right?

 -Hal

> 
> Tobias
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory