[lldb-dev] post-mortem debugging with lldb

[Jason Molenda]

> On Nov 29, 2014, at 1:35 PM, René J.V. Bertin <rjvbertin at gmail.com> wrote:
> 
> 
> I currently have the following for using lldb:
> 
> lldb -s %tempfile -p %pid < /dev/null
> %tempfile:
> set set term-width 200
> thread info
> bt all
> quit
> 
> The redirection from /dev/null is necessary because otherwise lldb will not respect the quit command (when read from a batch file; that must be a bug?)

It's worth noting that Jim added a real batch mode to lldb a month ago:

r219654 | jingham | 2014-10-13 18:20:07 -0700 (Mon, 13 Oct 2014) | 12 lines

This adds a "batch mode" to lldb kinda like the gdb batch mode.  It will quit the debugger
after all the commands have been executed except if one of the commands was an execution control
command that stopped because of a signal or exception.


this is in the svn repository but hasn't made it in to an Apple release of lldb yet.  There have been other fixes made related to batch mode as well since then -- I would not be surprised if your code works correctly with the current TOT without redirecting from /dev/null.



> In itself this works, except that I do not always get useful information. A full discussion can be found at https://git.reviewboard.kde.org/r/121286/ , and it shows an example where one can only "see back to when" the crash handler was invoked.

The short backtrace in that discussion is a tricky one -- _sigtramp followed by objc_msgSend.  Both of these can be difficult for the unwinder to backtrace out of (_sigtramp because the register context is saved out-of-band by the kernel and we rely on accurate eh_frame instructions to find it) and objc_msgSend because it is hand-written assembly with some hand-written eh_frame instructions that are accurate at many -- but not all -- points in the function.

I have seen some edge cases on Mavericks (Mac OS X 10.10) where _sigtramp unwinding is not completely accurate.  It's on my todo list to figure out what's going on there.

If your process was at a location in objc_msgSend that did not have accurate eh_frame unwind descriptions, that would also account for this.

I think it will be difficult to hit this backtrace again, it is likely to be rare.  Your process of attaching and collecting information looks reasonable to me.

I gather you're scraping the output of lldb for information about the crash.  This can be a problem as the debugger output changes over time ... if I were writing a tool like this, I would probably write it in Python using the SB API that lldb supports.  lldb is actually a debugger *library* and the lldb command line program is one client of that library (Xcode is another).  You can write a Python script (or C++ program) that uses the library to attach to the process, iterate over the threads, print the backtrace information you want, etc.

It's probably more work than you want to do right now but for long-term maintainability, it would be the way to go.  

J