[LLVMdev] Use perf tool for more accurate time measuring on Linux

[Yi Kong]

On 16 May 2014 18:08, Hal Finkel <hfinkel at anl.gov> wrote:
> ----- Original Message -----
>> From: "Yi Kong" <kongy.dev at gmail.com>
>> To: "Hal Finkel" <hfinkel at anl.gov>, "Renato Golin" <renato.golin at linaro.org>, "Tobias Grosser" <tobias at grosser.es>
>> Cc: "LLVM Dev" <llvmdev at cs.uiuc.edu>
>> Sent: Friday, May 16, 2014 11:37:28 AM
>> Subject: Use perf tool for more accurate time measuring on Linux
>>
>> Hi all,
>>
>> The LLVM benchmarking system produces very noisy results even on
>> quiet
>> machines. One of the sources of inaccuracy is the timing tool we are
>> using. Because it is a user-space tool, the OS can context switch it
>> and we will get an outlier result. Perf stat uses SW_TASK_CLOCK
>> counter in kernel to measure time, therefore more accurate. It also
>> does not get context switched.
>>
>> I've implemented a wrapper script over perf stat which mimics the
>> behaviour of timeit tool in test suite, so that nothing else needs to
>> be modified. The script is not yet feature complete as timeit, but
>> enough to run nightly tests.
>>
>> I carried out experiments on several machines and saw different level
>> of improvements. I am no longer seeing outlier results, and MAD is
>> considerably lower. The run-by-run changes results over the same
>> revision shrank from around 10 to only 2-3. The MAD reduced from
>> around 0.01 to 0.003 on a quiet machine.
>
> That sounds great, thanks for working on this!
>
> First, we'd definitely need more documentation on what perf is and how to get it. The testing guide (and lnt dependencies), at least, need to be updated. FWIW, I don't have any machines on which this is already installed (and so it certainly is not installed by default). On Ubuntu, it is claimed that both linux-base and linux-tools-common provide a 'pref' utility, and on rpm systems it looks like there is a perf package.
>
> +# FIXME: How to measure sys time?
> +echo sys 0.0000 >> $REPORT
>
> Is this just the difference between the real time and the task time (or would that be a reasonable approximation)?

Not on some occasions. But for most programs, that should a reasonable
approximation. Since test suite does not care about the sys time, we
can leave it for now.
$ time -p sleep 5
real 5.00
user 0.00
sys 0.00

>
> Thanks again,
> Hal
>
>>
>> I've attached the patch and please experiment with it.
>>
>> Cheers,
>> Yi Kong
>>
>
> --
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory