[lldb-dev] Thread created / exited notifications

[Greg Clayton]

> On Jul 2, 2015, at 1:26 AM, Bruce Mitchener <bruce.mitchener at gmail.com> wrote:
> 
> I was hoping that I could get events for threads being created and exiting.
> 
> Right now, it looks like this is sitting on top of a mountain of various issues. So, I'm hoping that I can go through some of the steps involved to see if I'm on track.
> 
> First up, is this something that would be useful to have?
> 
> It looks like that you basically have to get the current thread list when the process stops, and see how it differs from the last time you saw the thread list.  This means that you miss out on any threads that were created and destroyed in between process stops.
> 
> One issue is platform availability for this information:
> 
> 	• Windows: information about thread creation and destruction is provided directly by the Windows debug events. Right now, it is queued up and not processed until process stops though to maintain the current thread list.
> 	• Linux: it looks like you have to use thread_db, if it is available. (For example, if someone's using musl libc, there is no equivalent functionality.) Thread lists are gotten via enumerating /proc/%d/task.
> 	• Mac OS X and iOS: it looks like this information is not available. Thread lists are gotten by calling Mach's task_threads.
> 	• FreeBSD: there's a thread_db, but this isn't currently used. Right now, thread lists are gotten via PT_GETLWPLIST via ptrace.
> 
> LLDB/MI provides this information using async records:
> 
> 	• =thread-created,id="id",group-id="gid"
> 	• =thread-exited,id="id",group-id="gid"A thread either was created, or has exited. The id field contains the gdb identifier of the thread. The gid field identifies the thread group this thread belongs to. 
> 
> So, looking at what we might need to do in LLDB:
> 
> 	• We'd have a ThreadEventData.
> 	• We'd have an SBThread::GetThreadFromEvent().
> 	• For Targets where the stub supports these events, they would issue them as an asynchronous notification.
> 		• This asynchronous notification would get broadcast as an event.
> 	• For targets where the stub doesn't support these, then perhaps some (new?) code in Process::UpdateThreadListIfNeeded() could broadcast created / exited events?
> 	• LLDB/MI could be updated to use the new events.
> 	• The GDB RSP support needs to be updated to handle the new event. It isn't clear whether or not this is supported by GDB or if this would be an LLDB extension. (The GDB docs don't mention this.)
> 	• Perhaps methods on SBProcess to allow OperatingSystem plugins to issue created / exited events.
> 
> Some questions:
> 
> 	• First up, again, should we support thread creation / exit events?

I can be implemented as optional support, but we shouldn't make _anything_ that relies on it. We should probably have a call on SBProcess that tries to enable this feature. Something like:

SBError
SBProcess::EnableSynchronousThreadNotifications();

Macs would return an error, and any target that doesn't support it would return an error. 

> 	• When the target doesn't provide this information, is it okay for them to be batched up as described above by diffing the thread list when updating the thread list?

Yes this could easily be done.

> 	• If non-stop mode starts to work, how would the current model work where we just update a thread list at specific points in time? Wouldn't thread creation / exit events be more appropriate at that point?

There is a whole bunch of work needed to get non-stop mode working. You would need your changes to detect new threads and report them. You would to start listening to SBThread events instead of listening to the process state changed events. Much much more. Each command that does anything to the process (continue, halt, kill) need to be updated to support this new mode. All thread based commands (stepping in/out/over/return, etc) need to be updated to try and deal with other threads that can be running. All ThreadPlans need to be completely reworked to deal with non-stop. Hitting breakpoints is a real pain in non-stop mode because when you hit a breakpoint you need to disable the BP, single step the thread, re-enable the BP and continue, but of course you will need to halt ALL threads when doing this (more thread plan modifications) so you don't miss the BP on other threads. The only correct way to enable non-stop mode in my opinion is to improve the EmulateInstruction classes for the current architecture to be able to emulate ALL opcodes for the current architecture so when you hit a breakpoint, you can emulate the original opcode using the read/write memory/read callbacks. In this case you wouldn't have to disable the BP, single step, and re-enable the BP, you would just emulate it and do the actual read/write reg/memory. This is the way dtrace does things. I know some people think non-stop mode is going to be enabled soon, but in reality it is a long way off and there is a ton of work to make it work correctly.


> 	• Any thoughts on how this might work or integrate with the Operating System plugin interface? (Right now, this is all about UpdateThreadList and doesn't seem to provide a means for doing asynchronous notification like calling a method on SBProcess to create / exit threads...)

We would send out thread events. lldb_private::Thread would make a new eBroadcastBitStateChanged bit just like lldb_private::Process does, and send out events for itself. Clients could listen for thread events and be able to catch them. The question is: do OS threads get treated just like actual threads? Do they send out events?

> 	• If extensions are made, is there anything for communicating this clearly to other people implementing things, like Facebook's ds2?
> 
> It took about 10 hours to track through everything and read the relevant docs and look at other implementations (like Facebook's ds2) to get a better idea for what's going on in various areas here, but I may well have gotten something wrong ... so any corrections are welcome. :)

So overall the thread notifications might be nice to add using the correct format where you can listen to for thread created/exit notifications, maybe using a eBroadcastBitStateChanged bit just like lldb_private::Process and send a eStateLaunching state for threads being created and eStateExited for the thread exiting. Doing this only when the process stops seems like a good default (send out the new thread created/exited notifications). Doing anything synchronously seems like it will slow down debugging for no real gain as non-stop mode is really not close to being a reality...

Greg