[lldb-dev] Race condition crashes during launching LLDB
Greg Clayton via lldb-dev
lldb-dev at lists.llvm.org
Fri Feb 12 10:04:58 PST 2016
The main problem you are running into is in async mode when you say "launch" or "continue", those calls will return immediately and you must consume the events to make sure it is safe to do things. If your process is stopped, then until your actually resume your process with process.Continue() or any thread.StepXXX() calls, your process will stay stopped. But when you ask to make it run, you must consume events to know what the process is actually doing...
> On Feb 12, 2016, at 10:02 AM, Greg Clayton via lldb-dev <lldb-dev at lists.llvm.org> wrote:
>
> If you are going to set async to true, then you must consume the events by waiting for events. See the following example:
>
> svn cat http://llvm.org/svn/llvm-project/lldb/trunk/examples/python/process_events.py
>
> So you can rewrite your wait_for_process_stop to use the debugger to fetch the events and do things correctly:
>
> def wait_for_process_stop(debugger):
> event_timeout_in_seconds = 10
> listener = debugger.GetListener()
> event = lldb.SBEvent()
> stopped = False
> while not stopped:
> if listener.WaitForEvent (event_timeout_in_seconds, event):
> if lldb.SBProcess.EventIsProcessEvent(event):
> state = lldb.SBProcess.GetStateFromEvent (event)
> if state == lldb.eStateStopped:
> // Watch for something that stopped and restarted automatically
> if SBProcess::GetRestartedFromEvent(event) == False:
> stopped = True
>
>
> The other option is to do it to set Async to False and then your target.Launch won't return until the process is stopped. Also if you do "process.Continue()" it won't return until the process is stopped. But you lose your ability to stop the process if things go wrong...
>
> Greg
>
>> On Feb 4, 2016, at 9:09 PM, Jeffrey Tan via lldb-dev <lldb-dev at lists.llvm.org> wrote:
>>
>> After adding some logging I figured out that the race condition is caused by process.Continue() did not guarantee process has been really resumed yet in async mode, so the second wait_for_process_stop() is skipped immediately to kill listener thread and destroying debugger. I know I have a race condition bug here because of polling for process state, but why is lldb crashing when listener thread has exited and SBDebugger.Destroy() is called? What is the situation that SBDebugger.Destroy() can be called safely?
>>
>> ==================do_test==================
>> Launch result: success
>> <Listener> Listening Thread ID: 4660334592
>> WaitForEvent...
>> Target event: ModulesLoaded
>> WaitForEvent...
>> Process event: StateChanged, Stopped
>> Stop reason: 5
>> WaitForEvent...
>> Breakpoint event: [Added] SBBreakpoint: id = 1, name = 'main', locations = 1
>> WaitForEvent...
>> [main] killing listener thread
>> Process event: StateChanged, Running
>> Stop reason: 0
>> <Listener> Exiting listener thread
>> [main] destroy debugger
>> Segmentation fault: 11
>>
>> On Thu, Feb 4, 2016 at 8:22 PM, Jeffrey Tan <jeffrey.fudan at gmail.com> wrote:
>> Sorry, I have actually tried to exit the event listening thread before debugger destroy, but still go the crash randomly(1 out of 5 runs). Here is the code:
>>
>>
>> ==========Main Thread==========
>> def wait_for_process_stop(process):
>> while not process.is_stopped:
>> time.sleep(0.1)
>>
>> def launch_debugging(debugger, stop_at_entry):
>> error = lldb.SBError()
>> listener = lldb.SBListener('Chrome Dev Tools Listener')
>> target = debugger.GetSelectedTarget()
>> process = target.Launch (listener,
>> None, # argv
>> None, # envp
>> None, # stdin_path
>> None, # stdout_path
>> None, # stderr_path
>> None, # working directory
>> 0, # launch flags
>> stop_at_entry, # Stop at entry
>> error) # error
>> print 'Launch result: %s' % str(error)
>> listener_thread = LLDBListenerThread(debugger)
>> listener_thread.start()
>> return listener_thread
>>
>> def do_test():
>> debugger = lldb.SBDebugger.Create()
>> debugger.SetAsync(True)
>> executable_path = '~/Personal/compiler/CompilerConstruction/code/compiler'
>> target = debugger.CreateTargetWithFileAndArch(executable_path, lldb.LLDB_ARCH_DEFAULT)
>>
>> listener_thread = launch_debugging(debugger, stop_at_entry=True)
>> process = debugger.GetSelectedTarget().process
>>
>> wait_for_process_stop(process) # wait for entry breakpoint.
>> target.BreakpointCreateByName('main')
>> process.Continue()
>> wait_for_process_stop(process) # wait for main breakpoint.
>>
>> listener_thread.should_quit = True
>> listener_thread.join()
>>
>> lldb.SBDebugger.Destroy(debugger)
>>
>> def main():
>> do_test()
>> do_test()
>>
>> ==========Listening Thread==========
>> class LLDBListenerThread(Thread):
>> should_quit = False
>>
>> def __init__(self, debugger):
>> Thread.__init__(self)
>> process = debugger.GetSelectedTarget().process
>> self.listener = debugger.GetListener()
>> self._add_listener_to_process(process)
>> self._add_listener_to_target(process.target)
>>
>> def _add_listener_to_target(self, target):
>> # Listen for breakpoint/watchpoint events (Added/Removed/Disabled/etc).
>> broadcaster = target.GetBroadcaster()
>> mask = lldb.SBTarget.eBroadcastBitBreakpointChanged | lldb.SBTarget.eBroadcastBitWatchpointChanged | lldb.SBTarget.eBroadcastBitModulesLoaded
>> broadcaster.AddListener(self.listener, mask)
>>
>> def _add_listener_to_process(self, process):
>> # Listen for process events (Start/Stop/Interrupt/etc).
>> broadcaster = process.GetBroadcaster()
>> mask = lldb.SBProcess.eBroadcastBitStateChanged | lldb.SBProcess.eBroadcastBitSTDOUT | lldb.SBProcess.eBroadcastBitSTDERR | lldb.SBProcess.eBroadcastBitInterrupt
>> broadcaster.AddListener(self.listener, mask)
>>
>> def run(self):
>> while not self.should_quit:
>> event = lldb.SBEvent()
>> print 'WaitForEvent...'
>> if self.listener.WaitForEvent(1, event):
>> if lldb.SBTarget.EventIsTargetEvent(event):
>> self._handle_target_event(event)
>> elif lldb.SBProcess.EventIsProcessEvent(event):
>> self._handle_process_event(event)
>> elif lldb.SBBreakpoint.EventIsBreakpointEvent(event):
>> self._handle_breakpoint_event(event)
>> elif lldb.SBThread.EventIsThreadEvent(event):
>> self._handle_thread_event(event)
>> else:
>> self._handle_unknown_event(event)
>>
>> _hand_XXX_event() methods just dump some logging information of the debug events.
>>
>>
>> Crashed Thread: 0 Dispatch queue: com.apple.main-thread
>>
>> Exception Type: EXC_BAD_ACCESS (SIGSEGV)
>> Exception Codes: EXC_I386_GPFLT
>>
>> Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
>> 0 _lldb.so 0x0000000111528179 EventMatcher::operator()(std::__1::shared_ptr<lldb_private::Event> const&) const + 21
>> 1 _lldb.so 0x00000001115275d2 lldb_private::Listener::FindNextEventInternal(lldb_private::Broadcaster*, lldb_private::ConstString const*, unsigned int, unsigned int, std::__1::shared_ptr<lldb_private::Event>&, bool) + 176
>> 2 _lldb.so 0x0000000111527952 lldb_private::Listener::WaitForEventsInternal(lldb_private::TimeValue const*, lldb_private::Broadcaster*, lldb_private::ConstString const*, unsigned int, unsigned int, std::__1::shared_ptr<lldb_private::Event>&) + 134
>> 3 _lldb.so 0x0000000111527ae9 lldb_private::Listener::WaitForEventForBroadcasterWithType(lldb_private::TimeValue const*, lldb_private::Broadcaster*, unsigned int, std::__1::shared_ptr<lldb_private::Event>&) + 27
>> 4 _lldb.so 0x000000011171de6c lldb_private::Process::WaitForStateChangedEvents(lldb_private::TimeValue const*, std::__1::shared_ptr<lldb_private::Event>&, lldb_private::Listener*) + 112
>> 5 _lldb.so 0x000000011171dc95 lldb_private::Process::WaitForProcessToStop(lldb_private::TimeValue const*, std::__1::shared_ptr<lldb_private::Event>*, bool, lldb_private::Listener*, lldb_private::Stream*) + 377
>> 6 _lldb.so 0x000000011172616a lldb_private::Process::HaltForDestroyOrDetach(std::__1::shared_ptr<lldb_private::Event>&) + 216
>> 7 _lldb.so 0x000000011171d8b0 lldb_private::Process::Destroy(bool) + 146
>> 8 _lldb.so 0x000000011171d56d lldb_private::Process::Finalize() + 91
>> 9 _lldb.so 0x00000001115173c4 lldb_private::Debugger::Clear() + 148
>> 10 _lldb.so 0x00000001115171fd lldb_private::Debugger::Destroy(std::__1::shared_ptr<lldb_private::Debugger>&) + 37
>> 11 _lldb.so 0x000000010f83c144 lldb::SBDebugger::Destroy(lldb::SBDebugger&) + 116
>> 12 _lldb.so 0x000000010f884daf _wrap_SBDebugger_Destroy(_object*, _object*) + 120
>> 13 org.python.python 0x000000010e53e75f PyEval_EvalFrameEx + 12761
>>
>>
>> On Thu, Feb 4, 2016 at 5:37 PM, Jim Ingham <jingham at apple.com> wrote:
>> I don't know what:
>>
>> event_thread = LLDBListenerThread(debugger)
>>
>> does, but from your little sketch it looks like you are starting up a thread listening on this debugger, and so far as I can see you destroy the debugger out from under it without ever closing down that thread. That doesn't seem like a good idea.
>>
>> Jim
>>
>>
>>
>>
>>> On Feb 4, 2016, at 5:27 PM, Jeffrey Tan via lldb-dev <lldb-dev at lists.llvm.org> wrote:
>>>
>>> Hi,
>>>
>>> I am revising our lldb automation tests into async mode. However, I found it randomly crashes depends on timing. And the crash happens mostly while launching lldb twice in a row. I have narrowed down the code into a simple repro below. Any assumption I made wrong with the LLDB API here?
>>>
>>> The crash stack seems to be not consistently. In the small repro below, the crash stack is:
>>> Crashed Thread: 0 Dispatch queue: com.apple.main-thread
>>>
>>> Exception Type: EXC_BAD_ACCESS (SIGSEGV)
>>> Exception Codes: EXC_I386_GPFLT
>>>
>>> Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
>>> 0 _lldb.so 0x00000001088c7179 EventMatcher::operator()(std::__1::shared_ptr<lldb_private::Event> const&) const + 21
>>> 1 _lldb.so 0x00000001088c65d2 lldb_private::Listener::FindNextEventInternal(lldb_private::Broadcaster*, lldb_private::ConstString const*, unsigned int, unsigned int, std::__1::shared_ptr<lldb_private::Event>&, bool) + 176
>>> 2 _lldb.so 0x00000001088c6952 lldb_private::Listener::WaitForEventsInternal(lldb_private::TimeValue const*, lldb_private::Broadcaster*, lldb_private::ConstString const*, unsigned int, unsigned int, std::__1::shared_ptr<lldb_private::Event>&) + 134
>>> 3 _lldb.so 0x00000001088c6ae9 lldb_private::Listener::WaitForEventForBroadcasterWithType(lldb_private::TimeValue const*, lldb_private::Broadcaster*, unsigned int, std::__1::shared_ptr<lldb_private::Event>&) + 27
>>> 4 _lldb.so 0x0000000108abce6c lldb_private::Process::WaitForStateChangedEvents(lldb_private::TimeValue const*, std::__1::shared_ptr<lldb_private::Event>&, lldb_private::Listener*) + 112
>>> 5 _lldb.so 0x0000000108abcc95 lldb_private::Process::WaitForProcessToStop(lldb_private::TimeValue const*, std::__1::shared_ptr<lldb_private::Event>*, bool, lldb_private::Listener*, lldb_private::Stream*) + 377
>>> 6 _lldb.so 0x0000000108ac516a lldb_private::Process::HaltForDestroyOrDetach(std::__1::shared_ptr<lldb_private::Event>&) + 216
>>> 7 _lldb.so 0x0000000108abc8b0 lldb_private::Process::Destroy(bool) + 146
>>> 8 _lldb.so 0x0000000108abc56d lldb_private::Process::Finalize() + 91
>>> 9 _lldb.so 0x00000001088b63c4 lldb_private::Debugger::Clear() + 148
>>> 10 _lldb.so 0x00000001088b61fd lldb_private::Debugger::Destroy(std::__1::shared_ptr<lldb_private::Debugger>&) + 37
>>> 11 _lldb.so 0x0000000106bdb144 lldb::SBDebugger::Destroy(lldb::SBDebugger&) + 116
>>> 12 _lldb.so 0x0000000106c23daf _wrap_SBDebugger_Destroy(_object*, _object*) + 120
>>> 13 org.python.python 0x00000001058dd75f PyEval_EvalFrameEx + 12761
>>>
>>> while in the real unit test it is crashing at:
>>> Thread 12 Crashed:
>>> 0 libsystem_kernel.dylib 0x00007fff8635a286 __pthread_kill + 10
>>> 1 libsystem_c.dylib 0x00007fff919409b3 abort + 129
>>> 2 libc++abi.dylib 0x00007fff8a94ea21 abort_message + 257
>>> 3 libc++abi.dylib 0x00007fff8a9769d1 default_terminate_handler() + 267
>>> 4 libobjc.A.dylib 0x00007fff935e77eb _objc_terminate() + 124
>>> 5 libc++abi.dylib 0x00007fff8a9740a1 std::__terminate(void (*)()) + 8
>>> 6 libc++abi.dylib 0x00007fff8a973b30 __cxa_throw + 121
>>> 7 com.apple.LLDB.framework 0x000000010b994c6b std::__1::shared_ptr<lldb_private::Process>::shared_ptr<lldb_private::Process>(std::__1::weak_ptr<lldb_private::Process> const&, std::__1::enable_if<is_convertible<lldb_private::Process*, lldb_private::Process*>::value, std::__1::shared_ptr<lldb_private::Process>::__nat>::type) + 99
>>> 8 com.apple.LLDB.framework 0x000000010b8ac762 lldb_private::Process::AppendSTDOUT(char const*, unsigned long) + 86
>>> 9 com.apple.LLDB.framework 0x000000010b6951d7 lldb_private::Communication::ReadThread(void*) + 287
>>> 10 libsystem_pthread.dylib 0x00007fff8d92c05a _pthread_body + 131
>>> 11 libsystem_pthread.dylib 0x00007fff8d92bfd7 _pthread_start + 176
>>>
>>>
>>> ================Repro Code====================
>>>
>>> def wait_for_process_stop(process):
>>> while not process.is_stopped:
>>> time.sleep(0.1)
>>>
>>> def launch_debugging(debugger, stop_at_entry):
>>> error = lldb.SBError()
>>> listener = lldb.SBListener('Chrome Dev Tools Listener')
>>> target = debugger.GetSelectedTarget()
>>> process = target.Launch (listener,
>>> None, # argv
>>> None, # envp
>>> None, # stdin_path
>>> None, # stdout_path
>>> None, # stderr_path
>>> None, # working directory
>>> 0, # launch flags
>>> stop_at_entry, # Stop at entry
>>> error) # error
>>> print 'Launch result: %s' % str(error)
>>> event_thread = LLDBListenerThread(debugger)
>>> event_thread.start()
>>> return process
>>>
>>> def do_test():
>>> debugger = lldb.SBDebugger.Create()
>>> debugger.SetAsync(True)
>>> target = debugger.CreateTargetWithFileAndArch(executable_path, lldb.LLDB_ARCH_DEFAULT)
>>>
>>> process = launch_debugging(debugger, stop_at_entry=True)
>>>
>>> wait_for_process_stop(process) # wait for entry breakpoint.
>>> target.BreakpointCreateByName('main')
>>> process.Continue()
>>> wait_for_process_stop(process) # wait for main breakpoint.
>>> lldb.SBDebugger.Destroy(debugger)
>>>
>>> def main():
>>> do_test()
>>> do_test()
>>> _______________________________________________
>>> lldb-dev mailing list
>>> lldb-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>>
>>
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
More information about the lldb-dev
mailing list