[llvm-dev] Using Google Benchmark Library

Wed May 30 12:21:26 PDT 2018

On Wed, May 30, 2018 at 4:07 AM, mbraun via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> Not going into all the detail, but from my side the big question is
> whether the benchmarks inner loop is small/fine grained enough that
> stabilization with google benchmark doesn't lead to dozens of seconds
> benchmark runtimes. Given that you typically see thousandsd or millions of
> invocations for small functions...
>

Google benchmarks executes the kernel at max 1e9 times or until CPU time is
greater than the minimum time or the wallclock time is 5x minimum time, by
default min time is 0.5s but we can change it using "MinTime(X)" or
 "--benchmark_min_time=X".
So the stabilization of small/fine grained kernel with google benchmark
should not be any problem.

>
>
> > On May 29, 2018, at 2:06 PM, Michael Kruse via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >
> > Thanks for your remarks.
> >
> > 2018-05-27 5:19 GMT-05:00 Dean Michael Berris <dean.berris at gmail.com>:
> >> I think you might run into artificial overhead here if you’re not
> careful. In particular you might run into:
> >>
> >> - Missed in-lining opportunity in the benchmark. If you expect the
> kernels to be potentially inlined, this might be a problem.
> >
> > For the kind of benchmarks we have in mind, one function call overhead
> > is not significant, nor would we expect compilers to inline them in
> > the applications that use them.
> > Inlining may even be counterproductive: After inlining might see from
> > the array initialization code that elements are initialized to 0 (or
> > some other deterministic value) and use that knowledge while
> > optimizing the kernel.
> > We might prevent such things by annotating the kernel with
> > __attribute__((noinline)).
> >
> >
> >> - The link order might cause interference depending on the linker being
> used.
> >>
> >> - If you’re doing LTO then that would add an additional wrinkle.
> >>
> >> They’re not show-stoppers, but these are some of the things to look out
> for and consider.
> >
> > I'd consider the benchmark to be specific to a compiler+linker
> > combination. As long as we can measure the kernel in isolation (and
> > consider cold/warm caches), it should be fine.
> > I'd switch off LTO here since any code that could be inlined into the
> > kernel should already be in its translation unit.
> >
> >
> >>
> >>> - Instruct the driver to run the kernel with a small problem size and
> >>> check the correctness.
> >>
> >> In practice, what I’ve seen is mixing unit tests which perform
> correctness checks (using Google Test/Mock) and then co-locating the
> benchmarks in the same file. This way you can choose to run just the tests
> or the benchmarks in the same compilation mode. I’m not sure whether
> there’s already a copy of the Google Test/Mock libraries in the test-suite,
> but I’d think those shouldn’t be too hard (nor controversial) to add.
> >
> > Google Test is already part of LLVM. Since the test-suite already has
> > a dependency on LLVM (e.g. for llvm-lit, itself already supporting
> > Google Test), we could just use that one.
> > I don't know yet one would combine them to run the same code.
> >
> >
> >>> - Instructs Google Benchmark to run the kernel to get a reliable
> >>> average execution time of the kernel (without the input data
> >>> initialization)
> >>
> >> There’s ways to write the benchmarks so that you only measure a small
> part of the actual benchmark. The manuals will be really helpful in
> pointing out how to do that.
> >>
> >> https://github.com/google/benchmark#passing-arguments
> >>
> >> In particular, you can pause the timing when you’re doing the data
> initialisation and then resume just before you run the kernel.
> >
> > Sounds great.
> >
> >
> >>> - LNT's --exec-multisample does not need to run the benchmarks
> >>> multiple times, as Google Benchmark already did so.
> >>
> >> I thought recent patches already does some of this. Hal would know.
> >
> > I haven't found any special handling for --exec-multisample in
> MicroBenchmarks.
>
> This LNT option means run the whole benchmarking multiple times. It's more
> or less a loop the whole of the benchmark run: This even means we compile
> the source code multiple times.
>
> FWIW I'd really like to see an alternative option to compile once and then
> run each benchmark executable multiple times before moving to the next
> executable; but given that we don't have that today I'm just not using the
> multisample option personally as it's not better than submitting multiple
> benchmarking jobs anyway...
>

I think its better to use "benchmark_repetitions=n" option of google
benchmark than "--exec-multisample=n" as in multisample data initialization
will also be executed again along with the main kernel but
benchmark_repetitions option runs only the main kernel n times. The
repeated execution of data inititalization will only add unwanted execution
time.

Also, google benchmark library gives the result for each run along with the
mean, stddev and median of all runs. This can be usefull if there is some
tool which can parse the stdout and write some summary in a sheet.

One thing that I would like to have is some tool/API that can automatically
verifiy the output like lit does using reference output.

Regards,
Pankaj Kukreja
Computer Science Department
IIT Hyderabad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180531/1f69c1aa/attachment.html>