<div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Adam,</div><div dir="ltr"><br></div><div dir="ltr">Calls to the listeners should be protected by the RTDyldLayerMutex. Could you apply the attached patch and share the debugging output from one of the failing runs?</div><div dir="ltr"><br></div><div>Regards,</div><div>Lang.</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 20, 2020 at 8:00 PM David Blaikie <<a href="mailto:dblaikie@gmail.com">dblaikie@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">+Lang<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 20, 2020 at 4:44 PM Straw, Adam D via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div>
<p class="MsoNormal">Hi all,<u></u><u></u></p>
<p class="MsoNormal">Attention: Lang Hames<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I am developing the <a href="https://github.com/NervanaSystems/ngraph/tree/master/src/contrib/mlir" target="_blank">
nGraph MLIR</a> implementation and hitting the following assert while running nGraph unit tests:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> assert(ObjectBufferMap.find(K) == ObjectBufferMap.end() &&<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> "Second attempt to perform debug registration.");<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Here is a <a href="https://github.com/llvm/llvm-project/blob/3d5360a4398bfa6878f94ca9ac55bc568692c765/llvm/lib/ExecutionEngine/GDBRegistrationListener.cpp#L163" target="_blank">
permalink</a> for that line of code on GitHub. The failure only occurs when running multiple unit tests back-to-back. Inevitably, if I rerun the failing unit test, it passes. The failure also tends to move around with a different unit tests failing on successive
runs.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I am able to hit the failure in GDB. Here is a partial backtrace:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New"">#3 0x00007fffec4cf412 in __GI___assert_fail (assertion=assertion@entry=0x7ffff1d145e0 "ObjectBufferMap.find(K) == ObjectBufferMap.end() && \"Second attempt to perform debug
registration.\"", <u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New""> file=file@entry=0x7ffff1d144d8 "/localdisk/adstraw/ngraph/build/mlir_project/llvm-project/llvm/lib/ExecutionEngine/GDBRegistrationListener.cpp", line=line@entry=164,
<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New""> function=function@entry=0x7ffff1d15520 <(anonymous namespace)::GDBJITRegistrationListener::notifyObjectLoaded(unsigned long, llvm::object::ObjectFile const&, llvm::RuntimeDyld::LoadedObjectInfo
const&)::__PRETTY_FUNCTION__> "virtual void {anonymous}::GDBJITRegistrationListener::notifyObjectLoaded(llvm::JITEventListener::ObjectKey, const llvm::object::ObjectFile&, const llvm::RuntimeDyld::LoadedObjectInfo&)")<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New""> at assert.c:101<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New"">#4 0x00007ffff01ed4d3 in (anonymous namespace)::GDBJITRegistrationListener::notifyObjectLoaded (this=0x5555591a9270,
<span style="background-color:yellow;background-position:initial initial;background-repeat:initial initial">K=93825060893584</span>, Obj=..., L=...)<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New""> at /localdisk/adstraw/ngraph/build/mlir_project/llvm-project/llvm/lib/ExecutionEngine/GDBRegistrationListener.cpp:163<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New"">#5 0x00007ffff01b0eca in std::function<void (unsigned long, llvm::object::ObjectFile const&, llvm::RuntimeDyld::LoadedObjectInfo const&)>::operator()(unsigned long, llvm::object::ObjectFile
const&, llvm::RuntimeDyld::LoadedObjectInfo const&) const (__args#2=..., __args#1=..., __args#0=<optimized out>, this=0x5555594e09b8) at /usr/include/c++/7/bits/std_function.h:706<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New"">#6 llvm::orc::RTDyldObjectLinkingLayer::onObjLoad (this=<optimized out>,
<span style="background-color:yellow;background-position:initial initial;background-repeat:initial initial">K=3</span>, R=..., Obj=..., MemMgr=0x5555591c4160, LoadedObjInfo=std::unique_ptr<llvm::RuntimeDyld::LoadedObjectInfo> = {...},
<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New""> Resolved=std::map with 8 elements = {...}, InternalSymbols=std::set with 0 elements)<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New""> at /localdisk/adstraw/ngraph/build/mlir_project/llvm-project/llvm/lib/ExecutionEngine/Orc/RTDyldObjectLinkingLayer.cpp:267<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">A little debug context:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<ul style="margin-top:0in" type="disc">
<li style="margin-left:0in"><b>Frame #6</b> with K=3 we are about to execute the `NotifyLoaded` callback<u></u><u></u></li><li style="margin-left:0in"><b>Frame #5</b> we execute the `NotifyLoaded` callback which apparently transforms K from 3 to 93825060893584 (0x5555596CF390) which looks like a pointer. I don’t know the code well
enough to grok what’s going on here. I’m not even sure where this callback resides, but it could be
<a href="https://github.com/llvm/llvm-project/blob/c66f89005f6d23b6885d8f93f33ff27dc60ce7dd/llvm/lib/ExecutionEngine/Orc/RTDyldObjectLinkingLayer.cpp#L312" target="_blank">
here</a> in RTDyldObjectLinkingLater.cpp where we are using the address of the `MemMgr` as K.<u></u><u></u></li><li style="margin-left:0in"><b>Frame #4
</b>now in `notifyObjectLoaded` with K=93825060893584 which will go on to assert in
<b>frame #3</b> due to the fact that that key (K) already exists in the map<u></u><u></u></li></ul>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">All of this leads me to believe there is some sort of race. Theory: with two unit tests A and B running back to back…<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<ol style="margin-top:0in" start="1" type="1">
<li style="margin-left:0in">Unit test A allocates and deallocates a `MemMgr` (or whatever other object we are using for a key) at address X<u></u><u></u></li><li style="margin-left:0in">Unit test A calls ` notifyFreeingObject` to free the object at K=X but gets stuck (does not lock) the ` JITDebugLock` mutex
<a href="https://github.com/llvm/llvm-project/blob/3d5360a4398bfa6878f94ca9ac55bc568692c765/llvm/lib/ExecutionEngine/GDBRegistrationListener.cpp#L181" target="_blank">
here</a> for whatever reason<u></u><u></u></li><li style="margin-left:0in">Meanwhile unit test B allocates a `MemMgr` (or whatever other object we are using for a key) at newly freed address X<u></u><u></u></li><li style="margin-left:0in">Unit test B calls `notifyObjectLoaded` to register the object at K=X and locks the ` JITDebugLock`
<a href="https://github.com/llvm/llvm-project/blob/3d5360a4398bfa6878f94ca9ac55bc568692c765/llvm/lib/ExecutionEngine/GDBRegistrationListener.cpp#L162" target="_blank">
here</a> with unit test A still waiting in step #2 above<u></u><u></u></li><li style="margin-left:0in">This goes on to assert as the key was not erased (step #2) before it was added (step #4)<u></u><u></u></li></ol>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I could use a little help to debug the error further. Curiously, we do hit this issue in continuous integration (CI) --- only when running on development systems “by hand”. I am investigating the differences (perhaps single vs. multi
threaded?) as to why this might be the case.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Note that I am using a slightly old version of MLIR master, I’m at this commit:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New"">commit 2f8b4545f4960778e37114c024073d208751ca89<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New"">Author: Adam Straw <<a href="mailto:adam.d.straw@intel.com" target="_blank">adam.d.straw@intel.com</a>><u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New"">Date: Tue Apr 14 22:49:18 2020 +0300<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New""> [mlir] Fix assert on signed integer type in EDSC<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New"">
<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New""> Integer type in Std dialect is signless so we should be checking<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New""> for signless integer type instead of signed integer type in EDSC.<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New"">
<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Courier New""> Differential Revision:
<a href="https://reviews.llvm.org/D78144" target="_blank">https://reviews.llvm.org/D78144</a><u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks in advance for the help.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Adam<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Note: I am out of office this Fri and the following Mon but otherwise should be prompt with email replies.<u></u><u></u></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div>
</blockquote></div>