[lldb-dev] Adding support for FreeBSD kernel coredumps (and live memory lookup)

[David Spickett via lldb-dev]

> 1. The (older) "full memory" coredumps that use an ELF container.
>
> 2. The (newer) minidumps that dump only the active memory and use
a custom format.

Maybe a silly question, is the "minidumps" here the same sort of
minidump as lldb already supports
(https://chromium.googlesource.com/breakpad/breakpad/+/master/docs/getting_started_with_breakpad.md#the-minidump-file-format)?
Or mini meaning small and/or sparse relative to the ELF container core
files.

I see that the minidump tests use yaml2obj to make their files, but if
you end up only needing 1 file and it would need changes to yaml2obj
probably not worth pursuing.

On Thu, 2 Dec 2021 at 13:38, Michał Górny <mgorny at moritz.systems> wrote:
>
> On Thu, 2021-12-02 at 11:50 +0000, David Spickett wrote:
> > > Right now, the idea is that when the kernel crashes, the developer can
> > > take the vmcore file use LLDB to look the kernel state up.
> >
> > Thanks for the explanation. (FWIW your first email is clear now that I
> > read it properly but this still helped me :))
> >
> > > 2) How to integrate "live kernel" support into the current user
> > > interface?  I don't think we should make major UI modifications to
> > > support this specific case but I'd also like to avoid gross hacks.
> >
> > Do you think it will always be one or the other, corefile or live
> > memory? I assume you wouldn't want to fall back to live memory because
> > that memory might not have been in use at the time of the core dump.
>
> Yes, it's always one or the other.  When you're debugging crashed
> kernel, you want to see the state of the crashed kernel and not
> the kernel that's running right now.
>
> Reading the memory of running kernel seems less useful but I've been
> told that it sometimes helps debugging non-crash kernel bugs.
>
> > But I'm thinking about debuggers where they use the ELF file as a
> > quicker way to read memory. Not sure if lldb does this already but you
> > could steal some ideas from there if so.
> >
> > Using /dev/mem as the path seems fine unless you do need some
> > combination of that and a corefile. Is /dev/mem format identical to
> > the corefile format? (probably not an issue anyway because the plugin
> > is what will decide how to use it)
>
> No, the formats are distinct (well, /dev/mem doesn't really have
> a container format, to be precise) but libkvm distinguishes this case
> and handles it specially.
>
> > Your plans B and C seem like they are enablement of the initial use
> > case but have limited scope for improvements. The gdb-remote wrapper
> > for example would work fine but would you hit issues where the current
> > FreeBSD plugin is making userspace assumptions? For example the
> > AArch64 Linux plugin assumes that addresses will be in certain ranges,
> > so if you connected it to an in kernel stub you'd probably get some
> > surprises.
> >
> > So I agree a new plugin would make the most sense. Only reason I'd be
> > against it is if it added significant maintenance or build issues but
> > I'm not aware of any. (beyond checking for some libraries and plenty
> > of bits of llvm do that) And it'll be able to give the best
> > experience.
>
> Well, my initial attempt turned out quite trivial, primarily because
> the external library does most of the work:
>
> https://reviews.llvm.org/D114911
>
> Right now it just supports reading memory and printing variables.
> I still need to extend it to recognize kernel threads through the memory
> dump, and then add support for grabbing registers out of that to get
> backtraces.
>
> > Do you have a plan to test this if it is an in tree plugin? Will the
> > corefiles take up a lot of space or would you be able to craft minimal
> > files just for testing?
>
> I have some ideas but I don't have small core files right now.  I need
> to write more code to determine what exactly is necessary, and then
> decide to pursue either:
>
> a. trying to build a minimal FreeBSD kernel and run it in a VM with
> minimal amount of RAM to get a small minicore
>
> b. trying to strip unnecessary data from real minicore
>
> c. trying to construct a minicore file directly
>
> But as I said, I don't have enough data to decide which route would
> involve the least amount of work.
>
> --
> Best regards,
> Michał Górny
>