[LLVMdev] Use perf tool for more accurate time measuring on Linux

[Hal Finkel]

----- Original Message -----
> From: "Yi Kong" <kongy.dev at gmail.com>
> To: "Hal Finkel" <hfinkel at anl.gov>, "Renato Golin" <renato.golin at linaro.org>, "Tobias Grosser" <tobias at grosser.es>
> Cc: "LLVM Dev" <llvmdev at cs.uiuc.edu>
> Sent: Friday, May 16, 2014 11:37:28 AM
> Subject: Use perf tool for more accurate time measuring on Linux
> 
> Hi all,
> 
> The LLVM benchmarking system produces very noisy results even on
> quiet
> machines. One of the sources of inaccuracy is the timing tool we are
> using. Because it is a user-space tool, the OS can context switch it
> and we will get an outlier result. Perf stat uses SW_TASK_CLOCK
> counter in kernel to measure time, therefore more accurate. It also
> does not get context switched.
> 
> I've implemented a wrapper script over perf stat which mimics the
> behaviour of timeit tool in test suite, so that nothing else needs to
> be modified. The script is not yet feature complete as timeit, but
> enough to run nightly tests.
> 
> I carried out experiments on several machines and saw different level
> of improvements. I am no longer seeing outlier results, and MAD is
> considerably lower. The run-by-run changes results over the same
> revision shrank from around 10 to only 2-3. The MAD reduced from
> around 0.01 to 0.003 on a quiet machine.

That sounds great, thanks for working on this!

First, we'd definitely need more documentation on what perf is and how to get it. The testing guide (and lnt dependencies), at least, need to be updated. FWIW, I don't have any machines on which this is already installed (and so it certainly is not installed by default). On Ubuntu, it is claimed that both linux-base and linux-tools-common provide a 'pref' utility, and on rpm systems it looks like there is a perf package.

+# FIXME: How to measure sys time?
+echo sys 0.0000 >> $REPORT

Is this just the difference between the real time and the task time (or would that be a reasonable approximation)?

Thanks again,
Hal

> 
> I've attached the patch and please experiment with it.
> 
> Cheers,
> Yi Kong
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory