[LLVMdev] Use perf tool for more accurate time measuring on Linux

[Chandler Carruth]

Why not use the cycle count which perf exposes from hardware? That would
seem even better to me, but data would be better. =]


On Fri, May 16, 2014 at 11:17 AM, Yi Kong <kongy.dev at gmail.com> wrote:

> On 16 May 2014 18:08, Hal Finkel <hfinkel at anl.gov> wrote:
> > ----- Original Message -----
> >> From: "Yi Kong" <kongy.dev at gmail.com>
> >> To: "Hal Finkel" <hfinkel at anl.gov>, "Renato Golin" <
> renato.golin at linaro.org>, "Tobias Grosser" <tobias at grosser.es>
> >> Cc: "LLVM Dev" <llvmdev at cs.uiuc.edu>
> >> Sent: Friday, May 16, 2014 11:37:28 AM
> >> Subject: Use perf tool for more accurate time measuring on Linux
> >>
> >> Hi all,
> >>
> >> The LLVM benchmarking system produces very noisy results even on
> >> quiet
> >> machines. One of the sources of inaccuracy is the timing tool we are
> >> using. Because it is a user-space tool, the OS can context switch it
> >> and we will get an outlier result. Perf stat uses SW_TASK_CLOCK
> >> counter in kernel to measure time, therefore more accurate. It also
> >> does not get context switched.
> >>
> >> I've implemented a wrapper script over perf stat which mimics the
> >> behaviour of timeit tool in test suite, so that nothing else needs to
> >> be modified. The script is not yet feature complete as timeit, but
> >> enough to run nightly tests.
> >>
> >> I carried out experiments on several machines and saw different level
> >> of improvements. I am no longer seeing outlier results, and MAD is
> >> considerably lower. The run-by-run changes results over the same
> >> revision shrank from around 10 to only 2-3. The MAD reduced from
> >> around 0.01 to 0.003 on a quiet machine.
> >
> > That sounds great, thanks for working on this!
> >
> > First, we'd definitely need more documentation on what perf is and how
> to get it. The testing guide (and lnt dependencies), at least, need to be
> updated. FWIW, I don't have any machines on which this is already installed
> (and so it certainly is not installed by default). On Ubuntu, it is claimed
> that both linux-base and linux-tools-common provide a 'pref' utility, and
> on rpm systems it looks like there is a perf package.
> >
> > +# FIXME: How to measure sys time?
> > +echo sys 0.0000 >> $REPORT
> >
> > Is this just the difference between the real time and the task time (or
> would that be a reasonable approximation)?
>
> Not on some occasions. But for most programs, that should a reasonable
> approximation. Since test suite does not care about the sys time, we
> can leave it for now.
> $ time -p sleep 5
> real 5.00
> user 0.00
> sys 0.00
>
> >
> > Thanks again,
> > Hal
> >
> >>
> >> I've attached the patch and please experiment with it.
> >>
> >> Cheers,
> >> Yi Kong
> >>
> >
> > --
> > Hal Finkel
> > Assistant Computational Scientist
> > Leadership Computing Facility
> > Argonne National Laboratory
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140516/7484aed8/attachment.html>