[llvm-dev] Using Google Benchmark Library

[Michael Kruse via llvm-dev]

2018-05-25 13:49 GMT-05:00 Pankaj Kukreja via llvm-dev
<llvm-dev at lists.llvm.org>:
> Hi,
> I am adding some benchmarks to the test-suite as a part of my GSoC project.
> I am planning to use the google benchmark library on some benchmarks. I
> would like to know your opinion/suggestion on how I should proceed with this
> library and how the design should be(like limiting the number of times a
> kernel is executed so that overall runtime of test-suite can be controlled,
> multiple inputs sizes,  adding it to small kernels etc.).
> I would love to hear any other suggestions that you may have.

I would like to add some details of the intent.

We would like to benchmark common algorithms that appear in many
sources (Think of linear algebra, image processing, etc.). Usually,
they only consist of a single function. Only the kernel itself is of
interest, but not e.g. the data initialization. Depending on the
optimization (e.g. by Polly, parallelization, offloading), the
execution time can vary widely.

Pankaj already added a review at
https://reviews.llvm.org/D46735
where the idea to use Google Benchmark came up. However, as such it
does not fulfill all the requirements, in particular, it does not
check for correctness.

Here is a list of things I would like to see:
- Check correct output
- Execution time
- Compilation time
- LLVM pass statistics
- Code size
- Hardware performance counters
- Multiple problem sizes
- Measuring the above for the kernel only.
- Optional very large problem sizes (that test every cache level),
disabled by default
- Repeated execution to average-out noise
- Taking cold/warm cache into account

Here is an idea on how this could be implemented:
Every benchmark consists of two files: The source for the kernel and a
driver that initializes the input data, knows how to call the kernel
in the other file and can check the correct output.
The framework recognizes all benchmarks and their drivers and does the
following:
- Compile the driver
- Time the compilation of the kernel using -save-stats=obj
- Get the kernel's code size using llvm-size
- Link driver, kernel and Google Benchmark together.
- Instruct the driver to run the kernel with a small problem size and
check the correctness.
- Instructs Google Benchmark to run the kernel to get a reliable
average execution time of the kernel (without the input data
initialization)
- LNT's --exec-multisample does not need to run the benchmarks
multiple times, as Google Benchmark already did so.

Michael