[llvm-dev] Using Google Benchmark Library

[Michael Kruse via llvm-dev]

Thanks for your remarks.

2018-05-27 5:19 GMT-05:00 Dean Michael Berris <dean.berris at gmail.com>:
> I think you might run into artificial overhead here if you’re not careful. In particular you might run into:
>
> - Missed in-lining opportunity in the benchmark. If you expect the kernels to be potentially inlined, this might be a problem.

For the kind of benchmarks we have in mind, one function call overhead
is not significant, nor would we expect compilers to inline them in
the applications that use them.
Inlining may even be counterproductive: After inlining might see from
the array initialization code that elements are initialized to 0 (or
some other deterministic value) and use that knowledge while
optimizing the kernel.
We might prevent such things by annotating the kernel with
__attribute__((noinline)).


> - The link order might cause interference depending on the linker being used.
>
> - If you’re doing LTO then that would add an additional wrinkle.
>
> They’re not show-stoppers, but these are some of the things to look out for and consider.

I'd consider the benchmark to be specific to a compiler+linker
combination. As long as we can measure the kernel in isolation (and
consider cold/warm caches), it should be fine.
I'd switch off LTO here since any code that could be inlined into the
kernel should already be in its translation unit.


>
>> - Instruct the driver to run the kernel with a small problem size and
>> check the correctness.
>
> In practice, what I’ve seen is mixing unit tests which perform correctness checks (using Google Test/Mock) and then co-locating the benchmarks in the same file. This way you can choose to run just the tests or the benchmarks in the same compilation mode. I’m not sure whether there’s already a copy of the Google Test/Mock libraries in the test-suite, but I’d think those shouldn’t be too hard (nor controversial) to add.

Google Test is already part of LLVM. Since the test-suite already has
a dependency on LLVM (e.g. for llvm-lit, itself already supporting
Google Test), we could just use that one.
I don't know yet one would combine them to run the same code.


>> - Instructs Google Benchmark to run the kernel to get a reliable
>> average execution time of the kernel (without the input data
>> initialization)
>
> There’s ways to write the benchmarks so that you only measure a small part of the actual benchmark. The manuals will be really helpful in pointing out how to do that.
>
> https://github.com/google/benchmark#passing-arguments
>
> In particular, you can pause the timing when you’re doing the data initialisation and then resume just before you run the kernel.

Sounds great.


>> - LNT's --exec-multisample does not need to run the benchmarks
>> multiple times, as Google Benchmark already did so.
>
> I thought recent patches already does some of this. Hal would know.

I haven't found any special handling for --exec-multisample in MicroBenchmarks.

>
> PS. I’d be happy to do some reviews of uses of the Google Benchmark library, if you need additional reviewers.

Thank you. We will notify you when we have something.

Michael