[lldb-dev] Supporting FreeBSD kernel debugging project

[Ed Maste]

Thanks for the insight Greg - a few comments:

On 22 May 2014 12:39, Greg Clayton <gclayton at apple.com> wrote:
> Core file kernel debugging:
> - You shouldn't have to do too much to the current ProcessELFCore except help it to locate hint address for the list of shared libraries and the main kernel file in memory. This is done in:

Note that kernel core files on FreeBSD aren't (currently) ELF files,
and may be a different format on different architectures, so there's
some extra work here.  Dumps are written in a contiguous blob
(typically to swap) at the time of a crash, and then copied to the
filesystem after the next reboot.

The dump function for amd64/x86_64 is minidumpsys in
minidump_machdep.c[1], and consists of:

1. Dump header (stripped during copy to filesystem)
2. Minidump header
3. Copy of kernel message buffer
4. Bitmap of pages present in the dump
5. Page directory pages
6. Pages themselves
7. Trailer

All of this is abstracted by the libkvm library, which provide
kvm_read and kvm_write functions which a VA address parameter.  I
think this is the right way to introduce support for FreeBSD kernel
cores at first, as I'll explain below.

> Live kernel debugging:
> - You will make a new process subclass like you were saying unless the GDB remote protocol is used to talk to the kernel?

We currently have separate ways of interacting with the kernel for
remote and local debugging.  For remote debugging the kernel has a
built-in GDB protocol stub which is available over a serial or
firewire interface, and this should mostly work with the existing LLDB
support.  We do local debugging (examining threads, global objects,
etc.) via /dev/mem, abstracted by the same kvm_read and kvm_write
functions in libkvm.  Since local FreeBSD debugging will be done on a
FreeBSD host (by definition), using libkvm makes sense.  And since
we'll have that, using it also for cores makes sense to get started.

Of course we'll want to allow debugging a FreeBSD kernel core on OS X
or Linux too.  I think the way forward for us there is to create a
utility in the FreeBSD base system that converts the minidump format
to ELF, rather than teaching LLDB about our core formats in a
cross-platform way.  This utility could be built-in to savecore(8),
the tool which reads the dump out of a raw partition and writes to the
filesystem.  There is also ongoing discussion about bringing a KDP
target to the kernel for live debugging.

> Extra credit:
> - Get threads that are context switched in memory to show up. In the MacOSX kernel debugger we have the notion of 1 core we can talk to through the KDP interface which is the code that runs the debugger.

We have code[2] for this already in kgdb, our GDB 6.1 port with kernel
support, and we should be able to re-use most of it as is.  The kernel
threads just appear as regular threads in kgdb:

(kgdb) info threads
  827 Thread 102126 (PID=28738: kgdb)  sched_switch
(td=0xfffffe021882b000, newtd=0xfffffe0008392920, flags=<value
optimized out>)
    at /tank/emaste/src/git-stable-9/sys/kern/sched_ule.c:1904
  826 Thread 102334 (PID=28737: sudo)  sched_switch
(td=0xfffffe02d2669920, newtd=0xfffffe0008392920, flags=<value
optimized out>)
    at /tank/emaste/src/git-stable-9/sys/kern/sched_ule.c:1904
  825 Thread 104699 (PID=28715: more)  sched_switch
(td=0xfffffe049cec7920, newtd=0xfffffe0008390000, flags=<value
optimized out>)
    at /tank/emaste/src/git-stable-9/sys/kern/sched_ule.c:1904
...

-Ed

[1] http://svnweb.freebsd.org/base/head/sys/amd64/amd64/minidump_machdep.c?annotate=257216#l219
[2] http://svnweb.freebsd.org/base/head/gnu/usr.bin/gdb/kgdb/kthr.c?annotate=246893