<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi all,
<div class=""><br class="">
</div>
<div class="">This is an exciting development. </div>
<div class=""><br class="">
</div>
<div class="">BHive code base now supports both x86-64 and ARM timing. It would be great if these developments are integrated into LLVM-exegesis in a more portable manner. </div>
<div class=""><br class="">
</div>
<div class="">From the perspective of the compiler research community, this would enable us to use precise timing data inside LLVM IR passes if we want more precision than what TTI supports. We will not have to worry about segmentation faults when timing memory
heavy basic blocks if BHive’s timing methodology is adopted. Further, this will also pave way to interesting new directions in developing new performance models that are tightly coupled with the LLVM infrastructure. </div>
<div class=""><br class="">
</div>
<div class="">Therefore, I welcome and support this contribution by Ondrej and his colleagues in getting the BHive timing infrastructure embedded within the LLVM eco-system.</div>
<div class=""><br class="">
</div>
<div class="">Best,</div>
<div class="">Charith.</div>
<div class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Apr 21, 2021, at 9:32 AM, Ondrej Sykora <<a href="mailto:ondrasej@google.com" class="">ondrasej@google.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class=""><br class="">
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">---------- Forwarded message ---------<br class="">
From: <strong class="gmail_sendername" dir="auto">Ondrej Sykora</strong> <span dir="auto" class="">
<<a href="mailto:ondrasej@google.com" class="">ondrasej@google.com</a>></span><br class="">
Date: Fri, Mar 19, 2021 at 10:16 AM<br class="">
Subject: Re: [llvm-dev] [RFC] Implementing the BHive methodology in llvm-exegesis<br class="">
To: Clement Courbet <<a href="mailto:courbet@google.com" class="">courbet@google.com</a>><br class="">
Cc: llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>>, Tom Chen <<a href="mailto:cyt046@gmail.com" class="">cyt046@gmail.com</a>><br class="">
</div>
<br class="">
<br class="">
<div dir="ltr" class="">Hi all,
<div class=""><br class="">
</div>
<div class="">I'm sorry to revive such an old thread. Due to lack of time, we did not make progress as fast as we planned. We started building a prototype based on the <a href="https://urldefense.com/v3/__https://docs.google.com/document/d/1Z6sYes0jBRwHUkUZmI2JieDZZBm-HHhAzHlKJOFLLtU/edit*__;Iw!!DZ3fjg!tKYTdxu_GJ9pdWz4Qm6bknG4ija8etDoHkiUtzu1cFhAElkj1SoJb30bHUiE_8J5VK0$" target="_blank" class="">original
proposal</a>, but we found a couple of blockers:</div>
<div class="">- we found it very difficult to implement the interaction between the llvm-exegesis process and the child process running the benchmarked code using the LLDB API.</div>
<div class="">- in the meantime, the MIT team continued development of the BHive algorithm, and replaced most of the assembly with C. The new code is simpler and easier to port to other architectures.</div>
<div class=""><br class="">
</div>
<div class="">Based on our experience, we're considering a simpler approach compared to the <a href="https://urldefense.com/v3/__https://docs.google.com/document/d/1Z6sYes0jBRwHUkUZmI2JieDZZBm-HHhAzHlKJOFLLtU/edit*__;Iw!!DZ3fjg!tKYTdxu_GJ9pdWz4Qm6bknG4ija8etDoHkiUtzu1cFhAElkj1SoJb30bHUiE_8J5VK0$" target="_blank" class="">original
proposal</a>:</div>
<div class="">- if possible, we will use the same C-oriented design as the latest version of the tool developed at MIT.</div>
<div class="">- we will focus on a Linux and x86-64 implementation first, with porting to other architectures (but not operating systems) in mind. This would allow us to depend on the stable and well-defined Linux syscall interface. To our best knowledge, llvm-exegesis
is already limited to Linux because of its dependence on the Linux perf subsystem, so this does not create any new portability restrictions.</div>
<div class="">- we would use <a href="https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Ptrace__;!!DZ3fjg!tKYTdxu_GJ9pdWz4Qm6bknG4ija8etDoHkiUtzu1cFhAElkj1SoJb30bHUiEJ32eXN8$" target="_blank" class="">ptrace</a> as a simpler and more powerful alternative
to LLDB. It is a syscall, so it does not introduce any new external library dependencies.</div>
<div class="">- in the same spirit, we will depend on the mmap and munmap syscalls rather than on their abstractions.</div>
<div class=""><br class="">
</div>
<div class="">Let us know what you think!</div>
<div class=""><br class="">
</div>
<div class="">Best regards</div>
<div class=""><br class="">
</div>
<div class="">Ondrej</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Jan 27, 2020 at 1:21 PM Ondrej Sykora <<a href="mailto:ondrasej@google.com" target="_blank" class="">ondrasej@google.com</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">
<div dir="ltr" class="">Hi Clement,
<div class=""><br class="">
</div>
<div class="">thanks for the feedback!</div>
<div class=""><br class="">
</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Jan 17, 2020 at 11:47 AM Clement Courbet <<a href="mailto:courbet@google.com" target="_blank" class="">courbet@google.com</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Jan 16, 2020 at 6:32 PM Ondrej Sykora via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">
<div class="">In a <a href="https://urldefense.com/v3/__http://groups.csail.mit.edu/commit/papers/19/ithemal-measurement.pdf__;!!DZ3fjg!tKYTdxu_GJ9pdWz4Qm6bknG4ija8etDoHkiUtzu1cFhAElkj1SoJb30bHUiEKQ0dQVk$" target="_blank" class="">recent IISWC paper</a>, we've
proposed BHive - a new methodology for benchmarking arbitrary basic blocks that has several advantages over the one currently used in llvm-exegesis. In particular, the new methodology:<br class="">
</div>
<div class="">- automatically handles memory accesses in the basic block, without the need to manually annotate live-ins,<br class="">
- maps all memory addresses accessed by the basic block to the same page, significantly reducing the probability of cache misses during benchmarking,<br class="">
- the benchmarked code runs in a separate process, reducing risks of compromising the monitor process memory,<br class="">
- computes the throughput in a way that subtracts away the effects of the scaffolding code.<br class="">
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">I've never actually seen a case where the scaffolding code had much influence on the results (at least on X86), especially in loop mode. However, I can see some value in snippet mode (not generated code mode): this allows the snippet code to exhaust
all available registers and still be measurable.</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Yes, our main goal is benchmarking arbitrary basic blocks, where we do not control the register allocation.</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">
<div class="">A possible challenge is increased complexity of the code: BHive uses a separate process to run the benchmarked basic block and changes memory mapping of the process to ensure that all memory accesses lead to the same page. Most operating systems
have the necessary APIs, but these may differ significantly. In particular, the Windows API for memory mapping and process creation/control is very different from the Unix world. Initially, we might be able to support the new methodology only on Linux and
Unix-like systems.</div>
</div>
</blockquote>
<div class=""> </div>
<div class="">Though I think it's fine to have linux only as an initial implementation, I think there should be a clear plan to support windows: there are people in the LLVM community who are using llvm-exegesis on windows (e.g. folks at Sony). Note that you
might be able to reuse some code in LLVM: compiler-rt already has an abstraction layer in "WindowsMMap.c" on top of MapViewOfFile.</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Thanks for the pointers! That said, replacing mmap is relatively straightforward. The difficult part is replacing munmap, which does not have a direct equivalent on Windows and you need to query the system for all mapped blocks, and then unmap
them one by one. This is a very specific functionality, and I'd be surprised if someone implemented that.</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">
<div class="gmail_quote">
<div class=""></div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">
<div class="">Before we start the implementation, we would like to collect feedback on the proposed design:<br class="">
- We're planning to implement the methodology as a new implementation of BenchmarkRunner::FunctionExecutor that will exist alongside the current runner. The existing functionality will be preserved, and the user will be able to select the benchmark runner using
a command-line flag.<br class="">
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">LGTM.</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">
<div class="">- We're considering using the LLDB API to control the execution of the benchmarking process in a platform-independent way.<br class="">
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class=""><i class=""> </i>I think it's a great idea to avoid introducing any other external dependencies.</div>
</div>
</div>
</blockquote>
</div>
<div class=""><br class="">
</div>
-- <br class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div class="">Ondrej</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all" class="">
<div class=""><br class="">
</div>
-- <br class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div class="">Ondrej</div>
</div>
</div>
</div>
<br clear="all" class="">
<div class=""><br class="">
</div>
-- <br class="">
<div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr" class="">
<div class="">Ondrej</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>