<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Some more info from my team:<o:p></o:p></p>
<p class="MsoNormal">Debugger:Clear() is calling StopEventHandlerThread followed by a m_listener_sp->Clear() and from what I can tell the EventHandlerThread isn't fully stopped prior to the Clear being called.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> lldb-dev <lldb-dev-bounces@lists.llvm.org> <b>On Behalf Of
</b>Ted Woodward via lldb-dev<br>
<b>Sent:</b> Friday, March 22, 2019 3:28 PM<br>
<b>To:</b> LLDB <lldb-dev@lists.llvm.org><br>
<b>Subject:</b> [EXT] [lldb-dev] ThreadSanitizer reports lock-order-inversion running lldb on Ubuntu 14.04<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We’ve been seeing intermittent test failures in some lit tests with this error message:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We’ve only seen it on a buildbot running on SLES 11 – we can’t reproduce it on our Ubuntu machines.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We build lldb with tsan turned on, and see a lock-order-inversion error with just a run/quit. Could this be the cause of our crash?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Build was done with ToT on Ubuntu 14.04, using clang 7.0.1 with this cmake line:<o:p></o:p></p>
<p class="MsoNormal">CC=clang CXX=clang++ cmake -G Ninja -DLLVM_USE_SANITIZER=Thread -DLLVM_ENABLE_LIBCXX=On ../llvm/<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Output from the run:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">> TSAN_OPTIONS=second_deadlock_stack=1 bin/lldb<o:p></o:p></p>
<p class="MsoNormal">(lldb) quit<o:p></o:p></p>
<p class="MsoNormal">==================<o:p></o:p></p>
<p class="MsoNormal">WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=11107)<o:p></o:p></p>
<p class="MsoNormal"> Cycle in lock order graph: M1739 (0x7b1c000000b0) => M1745 (0x7b3c00000040) => M1739<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Mutex M1745 acquired here while holding mutex M1739 in thread T1:<o:p></o:p></p>
<p class="MsoNormal"> #0 pthread_mutex_lock /local/mnt/workspace/bcain_clang_bcain-ubuntu_14427/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:4071:3 (lldb+0x4441ae)<o:p></o:p></p>
<p class="MsoNormal"> #1 std::__1::recursive_mutex::lock() <null> (libc++.so.1+0x52ea5)<o:p></o:p></p>
<p class="MsoNormal"> #2 lldb_private::Debugger::DefaultEventHandler() /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:1537:16 (liblldb.so.9svn+0xd5f783)<o:p></o:p></p>
<p class="MsoNormal"> #3 lldb_private::Debugger::EventHandlerThread(void*) /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:1609:22 (liblldb.so.9svn+0xd5fba9)<o:p></o:p></p>
<p class="MsoNormal"> #4 lldb_private::HostNativeThreadBase::ThreadCreateTrampoline(void*) /local/mnt/ted/tip/llvm/tools/lldb/source/Host/common/HostNativeThreadBase.cpp:69:10 (liblldb.so.9svn+0xe438f2)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Mutex M1739 previously acquired by the same thread here:<o:p></o:p></p>
<p class="MsoNormal"> #0 pthread_mutex_lock /local/mnt/workspace/bcain_clang_bcain-ubuntu_14427/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:4071:3 (lldb+0x4441ae)<o:p></o:p></p>
<p class="MsoNormal"> #1 std::__1::recursive_mutex::lock() <null> (libc++.so.1+0x52ea5)<o:p></o:p></p>
<p class="MsoNormal"> #2 lldb_private::Debugger::DefaultEventHandler() /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:1537:16 (liblldb.so.9svn+0xd5f783)<o:p></o:p></p>
<p class="MsoNormal"> #3 lldb_private::Debugger::EventHandlerThread(void*) /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:1609:22 (liblldb.so.9svn+0xd5fba9)<o:p></o:p></p>
<p class="MsoNormal"> #4 lldb_private::HostNativeThreadBase::ThreadCreateTrampoline(void*) /local/mnt/ted/tip/llvm/tools/lldb/source/Host/common/HostNativeThreadBase.cpp:69:10 (liblldb.so.9svn+0xe438f2)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Mutex M1739 acquired here while holding mutex M1745 in main thread:<o:p></o:p></p>
<p class="MsoNormal"> #0 pthread_mutex_lock /local/mnt/workspace/bcain_clang_bcain-ubuntu_14427/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:4071:3 (lldb+0x4441ae)<o:p></o:p></p>
<p class="MsoNormal"> #1 std::__1::recursive_mutex::lock() <null> (libc++.so.1+0x52ea5)<o:p></o:p></p>
<p class="MsoNormal"> #2 lldb_private::Listener::Clear() /local/mnt/ted/tip/llvm/tools/lldb/source/Utility/Listener.cpp:78:19 (liblldb.so.9svn+0x1005e58)<o:p></o:p></p>
<p class="MsoNormal"> #3 lldb_private::Debugger::Clear()::$_0::operator()() const /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:841:20 (liblldb.so.9svn+0xd60237)<o:p></o:p></p>
<p class="MsoNormal"> #4 decltype(std::__1::forward<lldb_private::Debugger::Clear()::$_0>(fp)()) std::__1::__invoke<lldb_private::Debugger::Clear()::$_0>(lldb_private::Debugger::Clear()::$_0&&) /opt/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-14.04/bin/../include/c++/v1/type_traits:4345:1
(liblldb.so.9svn+0xd601cc)<o:p></o:p></p>
<p class="MsoNormal"> #5 void std::__1::__call_once_param<std::__1::tuple<lldb_private::Debugger::Clear()::$_0&&> >::__execute<>(std::__1::__tuple_indices<>) /opt/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-14.04/bin/../include/c++/v1/mutex:621 (liblldb.so.9svn+0xd601cc)<o:p></o:p></p>
<p class="MsoNormal"> #6 std::__1::__call_once_param<std::__1::tuple<lldb_private::Debugger::Clear()::$_0&&> >::operator()() /opt/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-14.04/bin/../include/c++/v1/mutex:613 (liblldb.so.9svn+0xd601cc)<o:p></o:p></p>
<p class="MsoNormal"> #7 void std::__1::__call_once_proxy<std::__1::tuple<lldb_private::Debugger::Clear()::$_0&&> >(void*) /opt/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-14.04/bin/../include/c++/v1/mutex:649 (liblldb.so.9svn+0xd601cc)<o:p></o:p></p>
<p class="MsoNormal"> #8 std::__1::__call_once(unsigned long volatile&, void*, void (*)(void*)) <null> (libc++.so.1+0x53317)<o:p></o:p></p>
<p class="MsoNormal"> #9 lldb_private::Debugger::Clear() /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:837:3 (liblldb.so.9svn+0xd5b358)<o:p></o:p></p>
<p class="MsoNormal"> #10 lldb_private::Debugger::Destroy(std::__1::shared_ptr<lldb_private::Debugger>&) /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:699:16 (liblldb.so.9svn+0xd5b9e6)<o:p></o:p></p>
<p class="MsoNormal"> #11 lldb::SBDebugger::Destroy(lldb::SBDebugger&) /local/mnt/ted/tip/llvm/tools/lldb/source/API/SBDebugger.cpp:276:3 (liblldb.so.9svn+0x9ddef5)<o:p></o:p></p>
<p class="MsoNormal"> #12 Driver::MainLoop() /local/mnt/ted/tip/llvm/tools/lldb/tools/driver/Driver.cpp:764:3 (lldb+0x4b518f)<o:p></o:p></p>
<p class="MsoNormal"> #13 main /local/mnt/ted/tip/llvm/tools/lldb/tools/driver/Driver.cpp:957:26 (lldb+0x4b5aa6)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Mutex M1745 previously acquired by the same thread here:<o:p></o:p></p>
<p class="MsoNormal"> #0 pthread_mutex_lock /local/mnt/workspace/bcain_clang_bcain-ubuntu_14427/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:4071:3 (lldb+0x4441ae)<o:p></o:p></p>
<p class="MsoNormal"> #1 std::__1::recursive_mutex::lock() <null> (libc++.so.1+0x52ea5)<o:p></o:p></p>
<p class="MsoNormal"> #2 lldb_private::Debugger::Clear()::$_0::operator()() const /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:841:20 (liblldb.so.9svn+0xd60237)<o:p></o:p></p>
<p class="MsoNormal"> #3 decltype(std::__1::forward<lldb_private::Debugger::Clear()::$_0>(fp)()) std::__1::__invoke<lldb_private::Debugger::Clear()::$_0>(lldb_private::Debugger::Clear()::$_0&&) /opt/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-14.04/bin/../include/c++/v1/type_traits:4345:1
(liblldb.so.9svn+0xd601cc)<o:p></o:p></p>
<p class="MsoNormal"> #4 void std::__1::__call_once_param<std::__1::tuple<lldb_private::Debugger::Clear()::$_0&&> >::__execute<>(std::__1::__tuple_indices<>) /opt/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-14.04/bin/../include/c++/v1/mutex:621 (liblldb.so.9svn+0xd601cc)<o:p></o:p></p>
<p class="MsoNormal"> #5 std::__1::__call_once_param<std::__1::tuple<lldb_private::Debugger::Clear()::$_0&&> >::operator()() /opt/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-14.04/bin/../include/c++/v1/mutex:613 (liblldb.so.9svn+0xd601cc)<o:p></o:p></p>
<p class="MsoNormal"> #6 void std::__1::__call_once_proxy<std::__1::tuple<lldb_private::Debugger::Clear()::$_0&&> >(void*) /opt/clang+llvm-7.0.1-x86_64-linux-gnu-ubuntu-14.04/bin/../include/c++/v1/mutex:649 (liblldb.so.9svn+0xd601cc)<o:p></o:p></p>
<p class="MsoNormal"> #7 std::__1::__call_once(unsigned long volatile&, void*, void (*)(void*)) <null> (libc++.so.1+0x53317)<o:p></o:p></p>
<p class="MsoNormal"> #8 lldb_private::Debugger::Clear() /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:837:3 (liblldb.so.9svn+0xd5b358)<o:p></o:p></p>
<p class="MsoNormal"> #9 lldb_private::Debugger::Destroy(std::__1::shared_ptr<lldb_private::Debugger>&) /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:699:16 (liblldb.so.9svn+0xd5b9e6)<o:p></o:p></p>
<p class="MsoNormal"> #10 lldb::SBDebugger::Destroy(lldb::SBDebugger&) /local/mnt/ted/tip/llvm/tools/lldb/source/API/SBDebugger.cpp:276:3 (liblldb.so.9svn+0x9ddef5)<o:p></o:p></p>
<p class="MsoNormal"> #11 Driver::MainLoop() /local/mnt/ted/tip/llvm/tools/lldb/tools/driver/Driver.cpp:764:3 (lldb+0x4b518f)<o:p></o:p></p>
<p class="MsoNormal"> #12 main /local/mnt/ted/tip/llvm/tools/lldb/tools/driver/Driver.cpp:957:26 (lldb+0x4b5aa6)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Thread T1 'dbg.evt-handler' (tid=11111, finished) created by main thread at:<o:p></o:p></p>
<p class="MsoNormal"> #0 pthread_create /local/mnt/workspace/bcain_clang_bcain-ubuntu_14427/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:965:3 (lldb+0x4286a5)<o:p></o:p></p>
<p class="MsoNormal"> #1 lldb_private::ThreadLauncher::LaunchThread(llvm::StringRef, void* (*)(void*), void*, lldb_private::Status*, unsigned long) /local/mnt/ted/tip/llvm/tools/lldb/source/Host/common/ThreadLauncher.cpp:70:7 (liblldb.so.9svn+0xe2da36)<o:p></o:p></p>
<p class="MsoNormal"> #2 lldb_private::Debugger::StartEventHandlerThread() /local/mnt/ted/tip/llvm/tools/lldb/source/Core/Debugger.cpp:1629:30 (liblldb.so.9svn+0xd5fc91)<o:p></o:p></p>
<p class="MsoNormal"> #3 lldb_private::CommandInterpreter::RunCommandInterpreter(bool, bool, lldb_private::CommandInterpreterRunOptions&) /local/mnt/ted/tip/llvm/tools/lldb/source/Interpreter/CommandInterpreter.cpp:3002:16 (liblldb.so.9svn+0xe5f0bf)<o:p></o:p></p>
<p class="MsoNormal"> #4 lldb::SBDebugger::RunCommandInterpreter(bool, bool) /local/mnt/ted/tip/llvm/tools/lldb/source/API/SBDebugger.cpp:1097:42 (liblldb.so.9svn+0x9e4729)<o:p></o:p></p>
<p class="MsoNormal"> #5 Driver::MainLoop() /local/mnt/ted/tip/llvm/tools/lldb/tools/driver/Driver.cpp:756:18 (lldb+0x4b5088)<o:p></o:p></p>
<p class="MsoNormal"> #6 main /local/mnt/ted/tip/llvm/tools/lldb/tools/driver/Driver.cpp:957:26 (lldb+0x4b5aa6)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) (/local/mnt/ted/tip/tsan/bin/../lib/libc++.so.1+0x52ea5) in std::__1::recursive_mutex::lock()<o:p></o:p></p>
<p class="MsoNormal">==================<o:p></o:p></p>
<p class="MsoNormal">ThreadSanitizer: reported 1 warnings<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>