[lldb-dev] Adding support for FreeBSD kernel coredumps (and live memory lookup)

[David Spickett via lldb-dev]

> Right now, the idea is that when the kernel crashes, the developer can
> take the vmcore file use LLDB to look the kernel state up.

Thanks for the explanation. (FWIW your first email is clear now that I
read it properly but this still helped me :))

> 2) How to integrate "live kernel" support into the current user
> interface?  I don't think we should make major UI modifications to
> support this specific case but I'd also like to avoid gross hacks.

Do you think it will always be one or the other, corefile or live
memory? I assume you wouldn't want to fall back to live memory because
that memory might not have been in use at the time of the core dump.
But I'm thinking about debuggers where they use the ELF file as a
quicker way to read memory. Not sure if lldb does this already but you
could steal some ideas from there if so.

Using /dev/mem as the path seems fine unless you do need some
combination of that and a corefile. Is /dev/mem format identical to
the corefile format? (probably not an issue anyway because the plugin
is what will decide how to use it)

Your plans B and C seem like they are enablement of the initial use
case but have limited scope for improvements. The gdb-remote wrapper
for example would work fine but would you hit issues where the current
FreeBSD plugin is making userspace assumptions? For example the
AArch64 Linux plugin assumes that addresses will be in certain ranges,
so if you connected it to an in kernel stub you'd probably get some
surprises.

So I agree a new plugin would make the most sense. Only reason I'd be
against it is if it added significant maintenance or build issues but
I'm not aware of any. (beyond checking for some libraries and plenty
of bits of llvm do that) And it'll be able to give the best
experience.

Do you have a plan to test this if it is an in tree plugin? Will the
corefiles take up a lot of space or would you be able to craft minimal
files just for testing?

On Thu, 2 Dec 2021 at 10:03, Michał Górny <mgorny at moritz.systems> wrote:
>
> On Thu, 2021-12-02 at 09:40 +0000, David Spickett wrote:
> > Can you give an example workflow of how these core files are used by a
> > developer? For some background.
>
> Right now, the idea is that when the kernel crashes, the developer can
> take the vmcore file use LLDB to look the kernel state up.  Initially,
> this means reading the "raw" memory, i.e. looking up basic symbol values
> but eventually (like kGDB) we'd like to add basic support for looking up
> kernel thread states.
>
> > Most of my experience is in userspace, the corefile is "offline" debug
> > and then you have "live" debug of the running process. Is that the
> > same here or do we have a mix since you can access some of the live
> > memory after the core has been dumped?
>
> It's roughly the same, i.e. you either use a crash dump (i.e. saved
> kernel state) or you use /dev/mem to read memory from the running
> kernel.
>
> > I'm wondering if a FreeBSD Kernel plugin would support these corefiles
> > and/or live debug, or if they are just two halves of the same
> > solution. Basically, would you end up with a FreeBSDKernelCoreDump and
> > a FreeBSDKernelLive plugin?
>
> I think one plugin is the correct approach here.  Firstly, because
> the interface for reading memory is abstracted out to a single library
> and the API is the same for both cases.  Secondly, because the actual
> interpreting logic would also be shared.
>
> --
> Best regards,
> Michał Górny
>