[lldb-dev] ThreadSanitizer reports lock-order-inversion running lldb on Ubuntu 14.04

Mon Mar 25 11:18:38 PDT 2019

Some more info from my team:
Debugger:Clear() is calling StopEventHandlerThread followed by a m_listener_sp->Clear() and from what I can tell the EventHandlerThread isn't fully stopped prior to the Clear being called.

From: lldb-dev <lldb-dev-bounces at lists.llvm.org> On Behalf Of Ted Woodward via lldb-dev
Sent: Friday, March 22, 2019 3:28 PM
To: LLDB <lldb-dev at lists.llvm.org>
Subject: [EXT] [lldb-dev] ThreadSanitizer reports lock-order-inversion running lldb on Ubuntu 14.04

We've been seeing intermittent test failures in some lit tests with this error message:

terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument

We've only seen it on a buildbot running on SLES 11 - we can't reproduce it on our Ubuntu machines.

We build lldb with tsan turned on, and see a lock-order-inversion error with just a run/quit. Could this be the cause of our crash?

Build was done with ToT on Ubuntu 14.04, using clang 7.0.1 with this cmake line:
CC=clang CXX=clang++ cmake -G Ninja -DLLVM_USE_SANITIZER=Thread -DLLVM_ENABLE_LIBCXX=On ../llvm/

Output from the run:

> TSAN_OPTIONS=second_deadlock_stack=1 bin/lldb
(lldb) quit
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=11107)
  Cycle in lock order graph: M1739 (0x7b1c000000b0) => M1745 (0x7b3c00000040) => M1739

  Mutex M1745 acquired here while holding mutex M1739 in thread T1:
    #0 pthread_mutex_lock /local/mnt/workspace/bcain_clang_bcain-ubuntu_14427/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:4071:3 (lldb+0x4441ae)
    #1 std::__1::recursive_mutex::lock() <null> (libc++.so.1+0x52ea5)
    #2 lldb_private::Debugger::DefaultEventHandler() /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:1537:16 (liblldb.so.9svn+0xd5f783)
    #3 lldb_private::Debugger::EventHandlerThread(void*) /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:1609:22 (liblldb.so.9svn+0xd5fba9)
    #4 lldb_private::HostNativeThreadBase::ThreadCreateTrampoline(void*) /local/mnt/ted/tip/llvm/tools/lldb/source/Host/common/HostNativeThreadBase.cpp:69:10 (liblldb.so.9svn+0xe438f2)

  Mutex M1739 previously acquired by the same thread here:
    #0 pthread_mutex_lock /local/mnt/workspace/bcain_clang_bcain-ubuntu_14427/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:4071:3 (lldb+0x4441ae)
    #1 std::__1::recursive_mutex::lock() <null> (libc++.so.1+0x52ea5)
    #2 lldb_private::Debugger::DefaultEventHandler() /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:1537:16 (liblldb.so.9svn+0xd5f783)
    #3 lldb_private::Debugger::EventHandlerThread(void*) /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:1609:22 (liblldb.so.9svn+0xd5fba9)
    #4 lldb_private::HostNativeThreadBase::ThreadCreateTrampoline(void*) /local/mnt/ted/tip/llvm/tools/lldb/source/Host/common/HostNativeThreadBase.cpp:69:10 (liblldb.so.9svn+0xe438f2)

  Mutex M1739 acquired here while holding mutex M1745 in main thread:
    #0 pthread_mutex_lock /local/mnt/workspace/bcain_clang_bcain-ubuntu_14427/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:4071:3 (lldb+0x4441ae)
    #1 std::__1::recursive_mutex::lock() <null> (libc++.so.1+0x52ea5)
    #2 lldb_private::Listener::Clear() /local/mnt/ted/tip/llvm/tools/lldb/source/Utility/Listener.cpp:78:19 (liblldb.so.9svn+0x1005e58)
    #3 lldb_private::Debugger::Clear()::$_0::operator()() const /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:841:20 (liblldb.so.9svn+0xd60237)
    #4 decltype(std::__1::forward<lldb_private::Debugger::Clear()::$_0>(fp)()) std::__1::__invoke<lldb_private::Debugger::Clear()::$_0>(lldb_private::Debugger::Clear()::$_0&&) /opt/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-14.04/bin/../include/c++/v1/type_traits:4345:1 (liblldb.so.9svn+0xd601cc)
    #5 void std::__1::__call_once_param<std::__1::tuple<lldb_private::Debugger::Clear()::$_0&&> >::__execute<>(std::__1::__tuple_indices<>) /opt/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-14.04/bin/../include/c++/v1/mutex:621 (liblldb.so.9svn+0xd601cc)
    #6 std::__1::__call_once_param<std::__1::tuple<lldb_private::Debugger::Clear()::$_0&&> >::operator()() /opt/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-14.04/bin/../include/c++/v1/mutex:613 (liblldb.so.9svn+0xd601cc)
    #7 void std::__1::__call_once_proxy<std::__1::tuple<lldb_private::Debugger::Clear()::$_0&&> >(void*) /opt/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-14.04/bin/../include/c++/v1/mutex:649 (liblldb.so.9svn+0xd601cc)
    #8 std::__1::__call_once(unsigned long volatile&, void*, void (*)(void*)) <null> (libc++.so.1+0x53317)
    #9 lldb_private::Debugger::Clear() /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:837:3 (liblldb.so.9svn+0xd5b358)
    #10 lldb_private::Debugger::Destroy(std::__1::shared_ptr<lldb_private::Debugger>&) /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:699:16 (liblldb.so.9svn+0xd5b9e6)
    #11 lldb::SBDebugger::Destroy(lldb::SBDebugger&) /local/mnt/ted/tip/llvm/tools/lldb/source/API/SBDebugger.cpp:276:3 (liblldb.so.9svn+0x9ddef5)
    #12 Driver::MainLoop() /local/mnt/ted/tip/llvm/tools/lldb/tools/driver/Driver.cpp:764:3 (lldb+0x4b518f)
    #13 main /local/mnt/ted/tip/llvm/tools/lldb/tools/driver/Driver.cpp:957:26 (lldb+0x4b5aa6)

  Mutex M1745 previously acquired by the same thread here:
    #0 pthread_mutex_lock /local/mnt/workspace/bcain_clang_bcain-ubuntu_14427/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:4071:3 (lldb+0x4441ae)
    #1 std::__1::recursive_mutex::lock() <null> (libc++.so.1+0x52ea5)
    #2 lldb_private::Debugger::Clear()::$_0::operator()() const /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:841:20 (liblldb.so.9svn+0xd60237)
    #3 decltype(std::__1::forward<lldb_private::Debugger::Clear()::$_0>(fp)()) std::__1::__invoke<lldb_private::Debugger::Clear()::$_0>(lldb_private::Debugger::Clear()::$_0&&) /opt/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-14.04/bin/../include/c++/v1/type_traits:4345:1 (liblldb.so.9svn+0xd601cc)
    #4 void std::__1::__call_once_param<std::__1::tuple<lldb_private::Debugger::Clear()::$_0&&> >::__execute<>(std::__1::__tuple_indices<>) /opt/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-14.04/bin/../include/c++/v1/mutex:621 (liblldb.so.9svn+0xd601cc)
    #5 std::__1::__call_once_param<std::__1::tuple<lldb_private::Debugger::Clear()::$_0&&> >::operator()() /opt/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-14.04/bin/../include/c++/v1/mutex:613 (liblldb.so.9svn+0xd601cc)
    #6 void std::__1::__call_once_proxy<std::__1::tuple<lldb_private::Debugger::Clear()::$_0&&> >(void*) /opt/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-14.04/bin/../include/c++/v1/mutex:649 (liblldb.so.9svn+0xd601cc)
    #7 std::__1::__call_once(unsigned long volatile&, void*, void (*)(void*)) <null> (libc++.so.1+0x53317)
    #8 lldb_private::Debugger::Clear() /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:837:3 (liblldb.so.9svn+0xd5b358)
    #9 lldb_private::Debugger::Destroy(std::__1::shared_ptr<lldb_private::Debugger>&) /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:699:16 (liblldb.so.9svn+0xd5b9e6)
    #10 lldb::SBDebugger::Destroy(lldb::SBDebugger&) /local/mnt/ted/tip/llvm/tools/lldb/source/API/SBDebugger.cpp:276:3 (liblldb.so.9svn+0x9ddef5)
    #11 Driver::MainLoop() /local/mnt/ted/tip/llvm/tools/lldb/tools/driver/Driver.cpp:764:3 (lldb+0x4b518f)
   #12 main /local/mnt/ted/tip/llvm/tools/lldb/tools/driver/Driver.cpp:957:26 (lldb+0x4b5aa6)

  Thread T1 'dbg.evt-handler' (tid=11111, finished) created by main thread at:
    #0 pthread_create /local/mnt/workspace/bcain_clang_bcain-ubuntu_14427/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:965:3 (lldb+0x4286a5)
    #1 lldb_private::ThreadLauncher::LaunchThread(llvm::StringRef, void* (*)(void*), void*, lldb_private::Status*, unsigned long) /local/mnt/ted/tip/llvm/tools/lldb/source/Host/common/ThreadLauncher.cpp:70:7 (liblldb.so.9svn+0xe2da36)
    #2 lldb_private::Debugger::StartEventHandlerThread() /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:1629:30 (liblldb.so.9svn+0xd5fc91)
    #3 lldb_private::CommandInterpreter::RunCommandInterpreter(bool, bool, lldb_private::CommandInterpreterRunOptions&) /local/mnt/ted/tip/llvm/tools/lldb/source/Interpreter/CommandInterpreter.cpp:3002:16 (liblldb.so.9svn+0xe5f0bf)
    #4 lldb::SBDebugger::RunCommandInterpreter(bool, bool) /local/mnt/ted/tip/llvm/tools/lldb/source/API/SBDebugger.cpp:1097:42 (liblldb.so.9svn+0x9e4729)
    #5 Driver::MainLoop() /local/mnt/ted/tip/llvm/tools/lldb/tools/driver/Driver.cpp:756:18 (lldb+0x4b5088)
    #6 main /local/mnt/ted/tip/llvm/tools/lldb/tools/driver/Driver.cpp:957:26 (lldb+0x4b5aa6)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) (/local/mnt/ted/tip/tsan/bin/../lib/libc++.so.1+0x52ea5) in std::__1::recursive_mutex::lock()
==================
ThreadSanitizer: reported 1 warnings

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20190325/986520be/attachment.html>