[lldb-dev] Adding support for FreeBSD kernel coredumps (and live memory lookup)

[Greg Clayton via lldb-dev]

I am fine with a new plug-in to handle this, but I want to verify a few things first:

Can this core dump file format basically allow debugging of multiple targets? For example could you for example want to examine the kernel itself as is, but also provide a view into any of the user space processes that exist? Mach-o kernel dumps can currently do this, but I am not sure how much of this code is public. The idea was you connect to the kernel dump, but you can create new targets that represent each user space process as it's own target within LLDB. The Apple tool would vend a new GDB remote protocol for each user space process and all memory reads that are asked of this GDB remote protocol that is created for each process can be asked for memory and each instance would translate the address correctly using the TLB entries in the kernel and give the user a user space view of this process. 

So the idea is connect to the kernel core file and display only the things that belong to the kernel, including all data structures and kernel threads in the target that represents the kernel. Have a way to list all of the user space processes that can have targets created so that each user space process can be debugged by a separate target in LLDB.

The natural area to do this would with a new lldb_private::Platform, or extending the existing PlatformFreeBSD. If you did a "platform select remote-freebsd", followed by a "platform connect --kernel-core-file /path/to/kernel/core.file", then the platform can be asked to list all available processes, one of which will be the kernel itself, and one process for each user space process that can have a target created for it. Then you can "process attach --pid <pid>" to attach to the kernel (we would need to make up a process ID for the kernel, and use the native process ID for all user space processes). The the new core file plug-in can be used to create a ProcessFreeBSDKernelCore instance that can be created and knows how to correctly answer all of the process questions for the targeted process.



> On Nov 30, 2021, at 5:49 AM, Michał Górny via lldb-dev <lldb-dev at lists.llvm.org> wrote:
> 
> Hi,
> 
> I'm working on a FreeBSD-sponsored project aiming at improving LLDB's
> support for debugging FreeBSD kernel to achieve feature parity with
> KGDB.  As a part of that, I'd like to improve LLDB's ability of working
> with kernel coredumps ("vmcores"), plus add the ability to read kernel
> memory via special character device /dev/mem.
> 
> 
> The FreeBSD kernel supports two coredump formats that are of interest to
> us:
> 
> 1. The (older) "full memory" coredumps that use an ELF container.
> 
> 2. The (newer) minidumps that dump only the active memory and use
> a custom format.
> 
> At this point, LLDB recognizes the ELF files but doesn't handle them
> correctly, and outright rejects the FreeBSD minidump format.  In both
> cases some additional logic is required.  This is because kernel
> coredumps contain physical contents of memory, and for user convenience
> the debugger needs to be able to read memory maps from the physical
> memory and use them to translate virtual addresses to physical
> addresses.
> 
> Unless I'm mistaken, the rationale for using this format is that
> coredumps are -- after all -- usually created when something goes wrong
> with the kernel.  In that case, we want the process for dumping core to
> be as simple as possible, and coredumps need to be small enough to fit
> in swap space (that's where they're being usually written).
> The complexity of memory translation should then naturally fall into
> userspace processes used to debug them.
> 
> FreeBSD (following Solaris and other BSDs) provides a helper libkvm
> library that can be used by userspace programs to access both coredumps
> and running kernel memory.  Additionally, we have split the routines
> related to coredumps and made them portable to other operating systems
> via libfbsdvmcore [1].  We have also included a program that can convert
> minidump into a debugger-compatible ELF core file.
> 
> 
> We'd like to discuss the possible approaches to integrating this
> additional functionality to LLDB.  At this point, our goal is to make it
> possible for LLDB to correctly read memory from coredumps and live
> system.
> 
> 
> Plan A: new FreeBSDKernel plugin
> ================================
> I think the preferable approach is to write a new plugin that would
> enable out-of-the-box support for the new functions in LLDB.  The plugin
> would be based on using both libraries.  When available, libfbsdvmcore
> will be used as the primary provider for vmcore support on all operating
> systems.  Additionally, libkvm will be usable on FreeBSD as a fallback
> provider for coredump support, and as the provider of live memory
> support.
> 
> support using system-installed libfbsdvmcore to read coredumps and
> libkvm to read coredumps (as a fallback) and to read live memory.
> 
> The two main challenges with this approach are:
> 
> 1) "Full memory" vmcores are currently recognized by LLDB's elf-core
> plugin.  I haven't investigated LLDB's plugin architecture in detail yet
> but I think the cleanest solution here would be to teach elf-core to
> distinguish and reject FreeBSD vmcores, in order to have the new plugin
> handle them.
> 
> 2) How to integrate "live kernel" support into the current user
> interface?  I don't think we should make major UI modifications to
> support this specific case but I'd also like to avoid gross hacks.
> My initial thought is to allow specifying "/dev/mem" as core path, that
> would match how libkvm handles it.
> 
> Nevertheless, I think this is the cleanest approach and I think we
> should go with it if possible.
> 
> 
> Plan B: GDB Remote Protocol-based wrapper
> =========================================
> If we cannot integrate FreeBSD vmcore support into LLDB directly,
> I think the next best approach is to create a minimal GDB Remote
> Protocol server for it.  The rough idea is that the server implements
> the minimal subset of the protocol necessary for LLDB to connect,
> and implements memory read operations via the aforementioned libraries.
> 
> The advantage of this solution is that it is still relatively clean
> and can be implemented outside LLDB.  It still provides quite good
> performance but probably requires more work than the alternatives
> and does not provide out-of-box support in LLDB.
> 
> 
> Plan C: converting vmcores
> ==========================
> Our final option, one that's practically implemented already is to
> require the user to explicitly convert vmcore into an ELF core
> understood by LLDB.  This is the simplest solution but it has a few
> drawbacks:
> 
> 1. it is limited to minidumps right now
> 
> 2. it requires storing a converted coredump which means that at least
> temporarily it doubles the disk space use
> 
> 3. there is possibility of cleanly supporting live kernel memory
> operations and therefore reaching KGDB feature parity
> 
> We could create a wrapper to avoid having users convert coredumps
> explicitly but well, we think other options are better.
> 
> 
> WDYT?
> 
> 
> [1] https://github.com/Moritz-Systems/libfbsdvmcore
> 
> -- 
> Best regards,
> Michał Górny
> 
> 
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev