[llvm-dev] [RFC] Implementing the BHive methodology in llvm-exegesis

[Ondrej Sykora via llvm-dev]

Hi all,

In a recent IISWC paper
<http://groups.csail.mit.edu/commit/papers/19/ithemal-measurement.pdf>,
we've proposed BHive - a new methodology for benchmarking arbitrary basic
blocks that has several advantages over the one currently used in
llvm-exegesis. In particular, the new methodology:
- automatically handles memory accesses in the basic block, without the
need to manually annotate live-ins,
- maps all memory addresses accessed by the basic block to the same page,
significantly reducing the probability of cache misses during benchmarking,
- the benchmarked code runs in a separate process, reducing risks of
compromising the monitor process memory,
- computes the throughput in a way that subtracts away the effects of the
scaffolding code.

A possible challenge is increased complexity of the code: BHive uses a
separate process to run the benchmarked basic block and changes memory
mapping of the process to ensure that all memory accesses lead to the same
page. Most operating systems have the necessary APIs, but these may differ
significantly. In particular, the Windows API for memory mapping and
process creation/control is very different from the Unix world. Initially,
we might be able to support the new methodology only on Linux and Unix-like
systems.

Before we start the implementation, we would like to collect feedback on
the proposed design:
- We're planning to implement the methodology as a new implementation of
BenchmarkRunner::FunctionExecutor that will exist alongside the current
runner. The existing functionality will be preserved, and the user will be
able to select the benchmark runner using a command-line flag.
- We're considering using the LLDB API to control the execution of the
benchmarking process in a platform-independent way.

You can find a more detailed proposal here
<https://docs.google.com/document/d/1Z6sYes0jBRwHUkUZmI2JieDZZBm-HHhAzHlKJOFLLtU/edit#>.
A stand-alone Linux implementation of the methodology used in the paper is
available on GitHub <https://github.com/ithemal/timing-harness>.

Comments and suggestions are most welcome!

Ondrej and Tom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200116/32c003ab/attachment.html>