<div dir="ltr">Hi Rafael,<div><br></div><div>Thanks a lot for the detailed description. I think this would help me a lot.</div><div><br></div><div>Regards,</div><div>Venu.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 10, 2020 at 12:35 AM Rafael Auler <<a href="mailto:rafaelauler@fb.com">rafaelauler@fb.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_8230196964602917893WordSection1">
<p class="MsoNormal">Hi Venugopal,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Running a project with source code under a debugger is the fastest way I know to create a mental model of the most important classes and how a project is organized. I definitely recommend doing that. BOLT’s documentation would be a high-level
view described in the 2019 CGO paper and some slides, but the technical details are only available in source code.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">However, I can give a quick overview to get you better prepared. Here are some key points:<br>
<br>
<u></u><u></u></p>
<p class="MsoNormal">1. Binary format is mostly abstracted away by LLVM’s libObject (see Binary.h and ObjectFile.h). We are currently working in a new bolt-only BinaryFormat abstraction to better encapsulate the gory details of manipulating object files.<br>
2. llvm-bolt.cpp is the main tool entry point. All class hierarchy where real work happens is designed as a library, LLVM-style, and llvm-bolt.cpp is the main user of BOLT as a library.<br>
3. The main control class in BOLT would be RewriteInstance. This represents the concept of a single binary rewrite that was requested by the user.<br>
4. RewriteInstance will coordinate the entire rewrite process by first reading the binary, building BOLT’s IR to represent its contents, perform a pipeline of modifications (BinaryPasses) and then rewriting it in a separate output file.<br>
<br>
If your interest in BOLT is to write a pass, I would suggest setting breakpoints and paying special attention to BinaryPassManager. You can find the simplest passes and easier to understand under Passes/BinaryPasses.cpp. Just copy one of them and register your
copy in BinaryPassManager. Your pass will be exposed to BOLT’s view of the world and you can write code to dump a snapshot of this view for a quick analysis (look for dump() methods in the objects exposed to your pass).<br>
<br>
BOLT’s IR top-level class is BinaryFunction. A BinaryFunction holds BinaryBasicBlock instances, which holds MCInst instances. Since BOLT operates with an augmented MCInst in comparison with the regular MCInst from LLVM, we have a special class to deal with
operations on instructions and abstract the target machine. You can play around by using MCPlusBuilder and its subclass X86MCPlusBuilder to do all sort of work, such as creating/checking calls, branches, etc. If you want to instrument your binary, for example,
take a look at Passes/Instrumentation.cpp to see how it accomplishes branch instrumentation.<br>
<br>
I hope this helps,<u></u><u></u></p>
<p class="MsoNormal">Rafael<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From: </span></b><span style="font-size:12pt;color:black">Venugopal Raghavan <<a href="mailto:venur2005@gmail.com" target="_blank">venur2005@gmail.com</a>><br>
<b>Date: </b>Thursday, April 9, 2020 at 6:10 AM<br>
<b>To: </b>Rafael Auler <<a href="mailto:rafaelauler@fb.com" target="_blank">rafaelauler@fb.com</a>><br>
<b>Subject: </b>Re: [llvm-dev] Error with perf2bolt in LLVM BOLT<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Hi Rafael, <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks for your reply. I think I understand the issue now. I had run BOLT a week ago successfully and I must have attempted perf2bolt specifying the binary that was optimized earlier.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I think I will do a fresh run.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I would like to understand the code in BOLT, the general flow and the data structures it uses. Is there any documentation on the code structure? What would suggest the fastest way to get some understanding of the code. My idea was to run
llvm-bolt under gdb and step through the code, but I am not sure that is the best way. I am not familiar with binary formats and so on, so that is another obstacle I need to face.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Regards,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Venu.<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, Apr 8, 2020 at 11:16 PM Rafael Auler <<a href="mailto:rafaelauler@fb.com" target="_blank">rafaelauler@fb.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">Hi Venugopal,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">perf2bolt has strict demands on its inputs when generating the profile data file. The input binary to perf2bolt must be the same that was running when you launched perf record,
and it will try to verify that by checking the build id, if the binary has one. It also assumes this will be the binary you will later optimize. Ordinarily, BOLT doesn’t optimize a binary that was already optimized by BOLT itself. That’s the message you are
getting. You should try collecting data on a binary that you did not already optimize with BOLT.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">That said, if you really want it, it is possible to collect data in binaries that were already bolted, but you need some non-standard flags for that. You need to use -enable-bat
when generating that binary. This flag will embed a translation table in your binary that perf2bolt uses to build the profile data suitable to be consumed in the original binary. This is non-standard because it is only really necessary in some large scale
deployments where collecting the data in a special “no bolt” configuration can be inconvenient.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Best,<u></u><u></u></p>
<p class="MsoNormal" style="margin-bottom:12pt">Rafael<br>
<br>
<u></u><u></u></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">llvm-dev <<a href="mailto:llvm-dev-bounces@lists.llvm.org" target="_blank">llvm-dev-bounces@lists.llvm.org</a>> on behalf of Venugopal Raghavan via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br>
<b>Reply-To: </b>Venugopal Raghavan <<a href="mailto:venur2005@gmail.com" target="_blank">venur2005@gmail.com</a>><br>
<b>Date: </b>Tuesday, April 7, 2020 at 10:53 PM<br>
<b>To: </b>"<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>" <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br>
<b>Subject: </b>[llvm-dev] Error with perf2bolt in LLVM BOLT</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Hi,
<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I was interested in trying out LLVM BOLT and generated profile data using Linux perf using the following:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">perf record -e cycles:u -o perf.data <command><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">This is without the use of LBR so I understand the performance improvements may not be much but this was more for becoming familiar with BOLT's commands.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I then run:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">perf2bolt -nl -p perf.data -o perf.fdata <binary><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">and I get the following:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">PERF2BOLT: Starting data aggregation job for perf.data<br>
PERF2BOLT: spawning perf job to read events without LBR<br>
PERF2BOLT: spawning perf job to read mem events<br>
PERF2BOLT: spawning perf job to read process events<br>
PERF2BOLT: spawning perf job to read task events<br>
BOLT-INFO: Target architecture: x86_64<br>
<b>BOLT-ERROR: input file was processed by BOLT. Cannot re-optimize.</b><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Not sure why I get the above error. Can someone who has used BOLT help me?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Regards,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Venugopal Raghavan.<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote></div>