<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:monospace;font-size:small;color:#000000"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 16, 2021 at 5:09 PM Ellis Hoag via Phabricator <<a href="mailto:reviews@reviews.llvm.org">reviews@reviews.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">ellis added a comment.<br>
<br>
Section Layout<br>
--------------<br>
<br>
MIP's major feature is the ability to extract all metadata from the instrumented binary to reduce its size. This is possible by using two sections, `__llvm_mipraw` to store profile data and `__llvm_mipmap` to store function metadata. The map section is extracted from the binary before runtime, and the raw section is dumped at runtime. Both sections are then combined to a final .mip profile. Both sections have a 24 byte header and a box of data for each instrumented function.<br>
<br>
`__llvm_mipraw`<br>
---------------<br>
<br>
The contents of this section depends on the type of instrumentation used, but the data is always "unreadable" without the map section. For function coverage, we simply allocate a single byte for each instrumented function and initialize it to all ones.<br>
<br>
_Z3foov$RAW:<br>
.byte 0xff<br>
<br>
<br>
<br>
`__llvm_mipmap`<br>
---------------<br>
<br>
The map section allows us to map instrumented functions to their profile data in the raw section. The two most important values are the function name and the offset to the profile data in the raw section. With these values we can read a function's profile data using the offset.<br>
<br>
_Z3foov$MAP:<br>
.Lref:<br>
.long __start___llvm_mipraw-.Lref # Raw Section Start PC Offset<br>
.long _Z3foov$RAW-.Lref # Raw Profile Symbol PC Offset<br>
.long _Z3foov-.Lref # Function PC Offset<br>
.long [[FOO_END]]-_Z3foov # Function Size<br>
.long 0x70c9fa27 # CFG Signature<br>
.long 2 # Non-entry Block Count<br>
.long [[FOO_BLOCK0]]-_Z3foov # Block 0 Offset<br>
.long [[FOO_BLOCK1]]-_Z3foov # Block 1 Offset<br>
.long 7 # Function Name Length<br>
.ascii "_Z3foov"<br>
<br>
The main challenge is storing the offset to the profile data without using dynamic relocations. This is complicated by the fact that we use comdat sections within the `__llvm_mipraw` section and that ELF does not seem to directly support section relative addresses. The solution is to use PC relative relocations. `__start___llvm_mipraw-.Lref` gives us the PC relative offset to the start of the raw section and `_Z3foov$RAW-.Lref` gives us the offset to the profile data for this function relative to the same PC. After we extract the map section, we can subtract these to get the value we want, the section relative raw profile data offset.<br>
<br>
(_Z3foov$RAW-.Lref) - (__start___llvm_mipraw-.Lref) = _Z3foov$RAW - __start___llvm_mipraw<br>
<br>
We can use the same trick to encode the function address, we just need to also add the address of the raw section which can be looked up in the binary. This is useful to lookup debug info and join it with our final profile data.<br>
<br>
The other values are relatively straightforward. The function size and block offsets allow us to map block coverage data to debug info. The control flow graph signature is used to verify that the function has not changed since the profile was collected. And last we have the mangled function name to help identify the function.<br>
<br>
Integration with Existing PGI?<br>
------------------------------<br>
<br>
There seems to be a question about why we chose to implement MIP from scratch instead of extending an existing framework. If we were to extend `-fprofile-instr-generate` we would need to have extractable metadata, which may be too invasive to implement.<br></blockquote><div><br></div><div><div class="gmail_default" style="font-family:monospace;font-size:small;color:rgb(0,0,0)">I believe you mean -fprofile-generate or -fcs-profile-generate. -fprofile-instr-generate is based on front end AST and eventually will be hidden under -fcoverage-test for source coveraging purpose only.</div><div class="gmail_default" style="font-family:monospace;font-size:small;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:monospace;font-size:small;color:rgb(0,0,0)">As you can see we are currently making an effort to unify/simplify the PGO implementation. Having yet another instrumentation mechanism is doing the opposite.</div></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
- As shown above, a lot of work was done to make sure the metadata can be extracted correctly.<br>
- Existing pgi has structured raw data that would need to be moved to the extractable metadata section.<br>
- Our MIP tool has a `llvm-mipdata create` command which converts the map section to an “empty” profile that can later be merged with raw profiles. Existing pgi tools do not have this extra step.<br>
<br></blockquote><div><br></div><div><div class="gmail_default" style="font-family:monospace;font-size:small;color:rgb(0,0,0)">As I said, size improvement efforts (under options) are welcome for the existing IRPGO. Another benefit is that we can have consolidated effort on improving related toolings.</div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
MIP Edge Profile<br>
----------------<br>
<br>
We omitted the code that adds call edge profiles to reduce the complexity of the current review, but we do plan to upload it. For each instrumented function, we sample return address values so we can build a dynamic call graph from the profile. The format is largely the same, but we added a size-configurable static buffer to hold the return address values. This approach correctly profiles dynamic dispatch calls that are found in Objective-C and Swift as well as any normal function calls. We can, for example, identify candidate for `objc_direct` using this call edge data.<br>
<br></blockquote><div><br></div><div><div class="gmail_default" style="font-family:monospace;font-size:small;color:rgb(0,0,0)">Adding this duplicate functionality (edge profiling) makes it even less compelling to do it MIR level.</div><div class="gmail_default" style="font-family:monospace;font-size:small;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:monospace;font-size:small;color:rgb(0,0,0)">For the dynamic dispatching profiling, does it handle any number of indirect targets or only supports topN?</div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Optimization<br>
------------<br>
<br>
We also omitted the profile consumption and optimization code from the current review. Our focus for optimization is on outlining to reduce size and function ordering to reduce page faults. When we consume profile data with function coverage, block coverage, or function call counts we can make smarter inlining and outlining decisions based on hotness. When we have profile data with timestamps, call counts, or the dynamic call graph then we can generate an optimal function order that reduces the number of page faults.<br>
<br></blockquote><div><span style="color:rgb(0,0,0);font-family:monospace"><span class="gmail_default" style="font-family:monospace;font-size:small;color:rgb(0,0,0)"></span></span></div><div><span style="color:rgb(0,0,0);font-family:monospace"><span class="gmail_default" style="font-family:monospace;font-size:small;color:rgb(0,0,0)">See above, adding any missing features in the existing framework is a more prefered approach. I am yet to see convincing arguments that spinning off a new instrumentation framework is the way to go.</span></span></div><div><span style="color:rgb(0,0,0);font-family:monospace"><span class="gmail_default" style="font-family:monospace;font-size:small;color:rgb(0,0,0)"><br></span></span></div><div><span style="color:rgb(0,0,0);font-family:monospace"><span class="gmail_default" style="font-family:monospace;font-size:small;color:rgb(0,0,0)">David </span></span><br></div><div><br></div><div class="gmail_default" style="font-family:monospace;font-size:small;color:rgb(0,0,0)"></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Repository:<br>
rG LLVM Github Monorepo<br>
<br>
CHANGES SINCE LAST ACTION<br>
<a href="https://reviews.llvm.org/D104060/new/" rel="noreferrer" target="_blank">https://reviews.llvm.org/D104060/new/</a><br>
<br>
<a href="https://reviews.llvm.org/D104060" rel="noreferrer" target="_blank">https://reviews.llvm.org/D104060</a><br>
<br>
</blockquote></div></div>