[lldb-dev] [RFC] lldb integration with (user mode) qemu

[Greg Clayton via lldb-dev]

> On Oct 28, 2021, at 6:33 AM, Pavel Labath via lldb-dev <lldb-dev at lists.llvm.org> wrote:
> 
> Hello everyone,
> 
> I'd like to propose a new plugin for better lldb+qemu integration.
> 
> As you're probably aware qemu has an integrated gdb stub. Lldb is able
> to communicate with it, but currently this process is somewhat tedious.
> One has to manually start qemu, giving it a port number, and then
> separately start lldb, and have it connect to that port.
> 
> The chief purpose of this feature would be to automate this behavior,
> ideally to the point where one can just point lldb to an executable,
> type "run", and everything would just work. It would take the form of a
> platform plugin (PlatformQemuUser, perhaps). This would be a non-host,
> always-connected plugin, and it's heart would be the DebugProcess
> method, which would ensure the emulator gets started when the user wants
> to start debugging. It would operate the same way as our host platforms
> do, except that it would start qemu instead of debug/lldb-server. Most
> of the other methods would be implemented by delegating to the host
> platform (as the process will be running on the host), possibly with
> some minor adjustments like prepending sysroot to the paths, etc. (My
> initial proof-of-concept implementation was 200 LOC.)

> The plugin would be configured via multiple settings, which would let
> the user specify, the path to the emulator, the kind of cpu it should
> emulate and the path to the system libraries, and any other arguments
> that the user wishes to pass to the emulator. The user could then
> configure it in their lldbinit file to match their system setup.

Yeah, I would create a "PlatformQemuEmulator" and allow multiple instances of this to be created. The setup for the architecture would then happen during the "platform connect" command. The "platform connect" command has different options for each platform, so you can customize the platform connect options to make sense for QEMU. Something like:

(lldb) platform select qemu-emulator
(lldb) platform connect --arch arm64 --sysroot /path/to/arm64/qemu/sysroot --emulator-path /path/to/arm64/emulator ...

> 
> 
> The needs of this plugin should match the existing Platform abstraction
> fairly well, so I don't anticipate (*) the need to add new entry points
> or modify existing ones.

Totally fine to add new virtual functions as needed if necessary.

> There is one tricky aspect which I see, and it
> relates to platform selection. Our current platform selection code gives
> each platform instance (while preferring the current platform) a chance
> to "claim" an executable, and aborts if the choice is ambiguous. The
> introduction of a qemu platform would introduce such an ambiguity, since
> (when running on a linux host) a linux executable would be claimed by
> both the qemu plugin and the existing remote-linux platform. This would
> prevent "target create arm-linux.exe" from working out-of-the-box.
> 
> To resolve this, I'd like to create some kind of a mechanism to give
> preference to some plugin. This could either be something internal,
> where a plugin indicates "strong" preference for an executable (the qemu
> platform could e.g. do this when the user sets the emulator path, the
> remote platform when it is connected), or some external mechanism like a
> global setting giving the preferred platform order. I'd very much like
> hear your thoughts on this.

Seems like selecting the platform first and then connecting to it, and specifying the architecture in the "platform connect --arch <triple" would allow the current QEMU platform to either accept the next:

(lldb) file arm-linux.exe

if the arch matches the currently selected platform for QEMU, or rejecting it of the architecture is wrong.
> 
> I'm also not sure how to handle the case of multiple emulated
> architectures. Qemu can emulate any processor architecture (of those
> that lldb supports, anyway), but the path to the emulator, sysroot, and
> probably other settings as well are going to be different. I see two
> possible ways to go about this:
> 
> a) have just a single set of settings, effectively limiting the user to
> emulating just a single architecture per session. While it would most
> likely be enough for most use cases, this kind of limitation seems
> artificial. It would also likely require the introduction of another
> setting, which would specify which architecture the plugin should
> actually emulate (and return from GetSupportedArchitectureAtIndex,
> etc.). On the flip side, this would be consistent with the how our
> remote-plugins work, although there it is given by the need to connect
> to something, and the supported architecture is then determined by the
> remote machine.
> 
> b) have multiple platform instances for each architecture. This solution
> be a more general solution, but it would mean that our "platform list"
> output would double, and half of it would consist of qemu platforms

I would vote for one platform plug-in that can have multiple instances created where each instance is configured by a call to "platform connect". Then you can do:

(lldb) platform select qemu-emulator
(lldb) platform connect --arch arm64 --sysroot /path/to/arm64/qemu/sysroot --emulator-path /path/to/arm64/emulator ...
(lldb) platform select qemu-emulator
(lldb) platform connect --arch x86_64 --sysroot /path/to/x86_64/qemu/sysroot --emulator-path /path/to/x86_64/emulator ...
(lldb) file arm-linux-arm64.exe
(lldb) file arm-linux-x86_64.exe

And each one would find the platforms that were created with some sort of precedence being needed to be added to LLDB. So maybe we keep a stack of most recently created selected platform instances and always check them in order when a new target is created. In the above case after the two platform select calls, we would have a stack like:

[0] qemu-emulator (x86_64)
[1] qemu-emulator (arm64)
[2] host 

And each new target would run through the platforms in order of most recently selected first. And we would need to be able to check all instances of a given platform as each one may or may not be compatible with each new target that is created.

I would also vote for such binaries to be marked in a way that would allow LLDB to auto select the right platform. So if the arm-linux.exe binary had some ELF notes inside of it that could specify the details of the QEMU needed, then we can just do:

$ lldb
(lldb) file arm-linux.exe

The details in the ELF file would help us to know that we wanted to use the QEMU platform, by making sure that the triple that we extract from the ObjectFileELF has QEMU specified in the environment. So the triple could be something like:

arm64-gnu-linux-qemu

The ELF file knows how to dig through ELF files and look at the OSABI in the ELF header + ELF notes to help refine the triple. Darwin binaries do this really well for macOS, iOS, watchOS and tvOS. Each mach-o file has enough info to create a unique triple that allows us to just create a target and we will select the right platform.

> 
> As far as testing is concerned I'm planning to reuse parts of our
> gdb-client test suite for this. Namely, I want to write a small python
> script which would act as a fake emulator. It would be sending out
> pre-programmed gdb-remote responses, much like our client test suite
> does. Since the main purpose of this is to validate that the emulator
> was started with the correct arguments, I don't expect the need for
> emulating any complex behavior -- the existing client classes should
> completely suffice.
> 
> If you got all the way here, I want to thank you for taking your time to
> read this, and urge you to let me know what you think.
> 
> regards,
> pavel
> 
> (*) There is one refactor of the Platform class implementations that I'd like to do first, but this (a) is not strictly necessary for this; and (b) is valueable independently of this RFC; so I am leaving that for a separate discussion.
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev