[lldb-dev] Adding support for FreeBSD kernel coredumps (and live memory lookup)

[Pavel Labath via lldb-dev]

On 30/11/2021 14:49, Michał Górny via lldb-dev wrote:
> Hi,
> 
> I'm working on a FreeBSD-sponsored project aiming at improving LLDB's
> support for debugging FreeBSD kernel to achieve feature parity with
> KGDB.  As a part of that, I'd like to improve LLDB's ability of working
> with kernel coredumps ("vmcores"), plus add the ability to read kernel
> memory via special character device /dev/mem.
> 
> 
> The FreeBSD kernel supports two coredump formats that are of interest to
> us:
> 
> 1. The (older) "full memory" coredumps that use an ELF container.
> 
> 2. The (newer) minidumps that dump only the active memory and use
> a custom format.
> 
> At this point, LLDB recognizes the ELF files but doesn't handle them
> correctly, and outright rejects the FreeBSD minidump format.  In both
> cases some additional logic is required.  This is because kernel
> coredumps contain physical contents of memory, and for user convenience
> the debugger needs to be able to read memory maps from the physical
> memory and use them to translate virtual addresses to physical
> addresses.
> 
> Unless I'm mistaken, the rationale for using this format is that
> coredumps are -- after all -- usually created when something goes wrong
> with the kernel.  In that case, we want the process for dumping core to
> be as simple as possible, and coredumps need to be small enough to fit
> in swap space (that's where they're being usually written).
> The complexity of memory translation should then naturally fall into
> userspace processes used to debug them.
> 
> FreeBSD (following Solaris and other BSDs) provides a helper libkvm
> library that can be used by userspace programs to access both coredumps
> and running kernel memory.  Additionally, we have split the routines
> related to coredumps and made them portable to other operating systems
> via libfbsdvmcore [1].  We have also included a program that can convert
> minidump into a debugger-compatible ELF core file.
> 
> 
> We'd like to discuss the possible approaches to integrating this
> additional functionality to LLDB.  At this point, our goal is to make it
> possible for LLDB to correctly read memory from coredumps and live
> system.
> 
> 
> Plan A: new FreeBSDKernel plugin
> ================================
> I think the preferable approach is to write a new plugin that would
> enable out-of-the-box support for the new functions in LLDB.  The plugin
> would be based on using both libraries.  When available, libfbsdvmcore
> will be used as the primary provider for vmcore support on all operating
> systems.  Additionally, libkvm will be usable on FreeBSD as a fallback
> provider for coredump support, and as the provider of live memory
> support.
> 
> support using system-installed libfbsdvmcore to read coredumps and
> libkvm to read coredumps (as a fallback) and to read live memory.
> 
> The two main challenges with this approach are:
> 
> 1) "Full memory" vmcores are currently recognized by LLDB's elf-core
> plugin.  I haven't investigated LLDB's plugin architecture in detail yet
> but I think the cleanest solution here would be to teach elf-core to
> distinguish and reject FreeBSD vmcores, in order to have the new plugin
> handle them.
> 
> 2) How to integrate "live kernel" support into the current user
> interface?  I don't think we should make major UI modifications to
> support this specific case but I'd also like to avoid gross hacks.
> My initial thought is to allow specifying "/dev/mem" as core path, that
> would match how libkvm handles it.
> 
> Nevertheless, I think this is the cleanest approach and I think we
> should go with it if possible.
> 
> 
> Plan B: GDB Remote Protocol-based wrapper
> =========================================
> If we cannot integrate FreeBSD vmcore support into LLDB directly,
> I think the next best approach is to create a minimal GDB Remote
> Protocol server for it.  The rough idea is that the server implements
> the minimal subset of the protocol necessary for LLDB to connect,
> and implements memory read operations via the aforementioned libraries.
> 
> The advantage of this solution is that it is still relatively clean
> and can be implemented outside LLDB.  It still provides quite good
> performance but probably requires more work than the alternatives
> and does not provide out-of-box support in LLDB.
> 
> 
> Plan C: converting vmcores
> ==========================
> Our final option, one that's practically implemented already is to
> require the user to explicitly convert vmcore into an ELF core
> understood by LLDB.  This is the simplest solution but it has a few
> drawbacks:
> 
> 1. it is limited to minidumps right now
> 
> 2. it requires storing a converted coredump which means that at least
> temporarily it doubles the disk space use
> 
> 3. there is possibility of cleanly supporting live kernel memory
> operations and therefore reaching KGDB feature parity
> 
> We could create a wrapper to avoid having users convert coredumps
> explicitly but well, we think other options are better.
> 
> 
> WDYT?
> 
> 
> [1] https://github.com/Moritz-Systems/libfbsdvmcore
> 

Having a new plugin for opening these kinds of core files seems 
reasonable to me. The extra dependency is unfortunate, but I guess 
that's what we have to work with.

Since this is still an elf file, you might still be able to use yaml2obj 
to create realistic-looking core files (it's support for program headers 
is pretty good these days).


The live kernel debugging sounds... scary. Can you explain how would 
this actually work? Like, what would be the supported operations? I 
presume you won't be able to actually "stop" the kernel, but what will 
you actually be able to do?

Without more details its hard for me to say whether this should be a 
separate plugin (or even a server, since that's what we tend to use for 
live debugging).

pl