[llvm-dev] Error with perf2bolt in LLVM BOLT

[Rafael Auler via llvm-dev]

Hi Venugopal,

Running a project with source code under a debugger is the fastest way I know to create a mental model of the most important classes and how a project is organized. I definitely recommend doing that. BOLT’s documentation would be a high-level view described in the 2019 CGO paper and some slides, but the technical details are only available in source code.

However, I can give a quick overview to get you better prepared. Here are some key points:

1. Binary format is mostly abstracted away by LLVM’s libObject (see Binary.h and ObjectFile.h). We are currently working in a new bolt-only BinaryFormat abstraction to better encapsulate the gory details of manipulating object files.
2. llvm-bolt.cpp is the main tool entry point. All class hierarchy where real work happens is designed as a library, LLVM-style, and llvm-bolt.cpp is the main user of BOLT as a library.
3. The main control class in BOLT would be RewriteInstance. This represents the concept of a single binary rewrite that was requested by the user.
4. RewriteInstance will coordinate the entire rewrite process by first reading the binary, building BOLT’s IR to represent its contents, perform a pipeline of modifications (BinaryPasses) and then rewriting it in a separate output file.

If your interest in BOLT is to write a pass, I would suggest setting breakpoints and paying special attention to BinaryPassManager. You can find the simplest passes and easier to understand under Passes/BinaryPasses.cpp. Just copy one of them and register your copy in BinaryPassManager.  Your pass will be exposed to BOLT’s view of the world and you can write code to dump a snapshot of this view for a quick analysis (look for dump() methods in the objects exposed to your pass).

BOLT’s IR top-level class is BinaryFunction. A BinaryFunction holds BinaryBasicBlock instances, which holds MCInst instances. Since BOLT operates with an augmented MCInst in comparison with the regular MCInst from LLVM, we have a special class to deal with operations on instructions and abstract the target machine. You can play around by using MCPlusBuilder and its subclass X86MCPlusBuilder to do all sort of work, such as creating/checking calls, branches, etc. If you want to instrument your binary, for example, take a look at Passes/Instrumentation.cpp to see how it accomplishes branch instrumentation.

I hope this helps,
Rafael

From: Venugopal Raghavan <venur2005 at gmail.com>
Date: Thursday, April 9, 2020 at 6:10 AM
To: Rafael Auler <rafaelauler at fb.com>
Subject: Re: [llvm-dev] Error with perf2bolt in LLVM BOLT

Hi Rafael,

Thanks for your reply. I think I understand the issue now. I had run BOLT a week ago successfully and I must have attempted perf2bolt specifying the binary that was optimized earlier.

I think I will do a fresh run.

I would like to understand the code in BOLT, the general flow and the data structures it uses. Is there any documentation on the code structure? What would suggest the fastest way to get some understanding of the code. My idea was to run llvm-bolt under gdb and step through the code, but I am not sure that is the best way. I am not familiar with binary formats and so on, so that is another obstacle I need to face.

Thanks.

Regards,
Venu.

On Wed, Apr 8, 2020 at 11:16 PM Rafael Auler <rafaelauler at fb.com<mailto:rafaelauler at fb.com>> wrote:
Hi Venugopal,

perf2bolt has strict demands on its inputs when generating the profile data file. The input binary to perf2bolt must be the same that was running when you launched perf record, and it will try to verify that by checking the build id, if the binary has one. It also assumes this will be the binary you will later optimize. Ordinarily, BOLT doesn’t optimize a binary that was already optimized by BOLT itself. That’s the message you are getting. You should try collecting data on a binary that you did not already optimize with BOLT.

That said, if you really want it, it is possible to collect data in binaries that were already bolted, but you need some non-standard flags for that. You need to use -enable-bat when generating that binary. This flag will embed a translation table in your binary that perf2bolt uses to build the profile data suitable to be consumed in the original binary. This is non-standard because it is only really necessary in some large scale deployments where collecting the data in a special “no bolt” configuration can be inconvenient.

Best,
Rafael

From: llvm-dev <llvm-dev-bounces at lists.llvm.org<mailto:llvm-dev-bounces at lists.llvm.org>> on behalf of Venugopal Raghavan via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>>
Reply-To: Venugopal Raghavan <venur2005 at gmail.com<mailto:venur2005 at gmail.com>>
Date: Tuesday, April 7, 2020 at 10:53 PM
To: "llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>" <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>>
Subject: [llvm-dev] Error with perf2bolt in LLVM BOLT

Hi,

I was interested in trying out LLVM BOLT and generated profile data using Linux perf using the following:

perf record -e cycles:u -o perf.data <command>

This is without the use of LBR so I understand the performance improvements may not be much but this was more for becoming familiar with BOLT's commands.

I then run:

perf2bolt -nl -p perf.data -o perf.fdata <binary>

and I get the following:

PERF2BOLT: Starting data aggregation job for perf.data
PERF2BOLT: spawning perf job to read events without LBR
PERF2BOLT: spawning perf job to read mem events
PERF2BOLT: spawning perf job to read process events
PERF2BOLT: spawning perf job to read task events
BOLT-INFO: Target architecture: x86_64
BOLT-ERROR: input file was processed by BOLT. Cannot re-optimize.

Not sure why I get the above error. Can someone who has used BOLT help me?

Thanks.

Regards,
Venugopal Raghavan.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200409/d86df3d3/attachment.html>