<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">While using a sampling-based profiler (HPCToolkit) to measure the performance of an application using a dynamically-linked version of the LLVM OpenMP runtime, I encountered a deadlock on x86_64. Although I haven’t considered other architectures in detail, I believe that they may be similarly affected.<div class=""><br class=""></div><div class="">Here’s what I believe I have observed: there is a subtle race condition between TLS setup for an OpenMP runtime and and a profiler that inspects it through the OMPT interface.</div><div class=""><br class=""></div><div class="">A thread executing code in <span style="caret-color: rgb(4, 51, 255);" class="">__kmp_launch_worker in</span> the context of the LLVM OpenMP runtime library acquired the lock controlling access to TLS state (</div><div class=""><span style="caret-color: rgb(4, 51, 255); color: rgb(4, 51, 255); font-family: Courier;" class="">__tls_get_addr </span><span style="caret-color: rgb(4, 51, 255);" class="">calls</span><span style="caret-color: rgb(4, 51, 255); color: rgb(4, 51, 255); font-family: Courier;" class=""> </span><span style="caret-color: rgb(4, 51, 255); color: rgb(4, 51, 255); font-family: Courier;" class="">tls_get_addr_tail </span><span style="caret-color: rgb(4, 51, 255);" class="">calls</span><span style="caret-color: rgb(4, 51, 255); color: rgb(4, 51, 255); font-family: Courier;" class=""> </span><span style="caret-color: rgb(4, 51, 255); color: rgb(4, 51, 255); font-family: Courier;" class="">pthread_mutex_lock)</span>to set up TLS needed for its access to its thread local variable <span style="caret-color: rgb(255, 38, 0);" class="">__kmp_gtid in frame 24 of the callstack shown below. Immediately after acquiring the TLS lock by setting its __lock field with a CMPXCHG but before recording the lock owner or finishing TLS setup, the thread was interrupted by our profiler. As a normal part of its operation to record a sample, our profiler uses the OMPT tools API to check if the thread is an OpenMP thread by inspecting the thread id being maintained by the OpenMP runtime. A call to a runtime entry point through the OMPT API led to an access to </span><span style="caret-color: rgb(255, 38, 0);" class="">__kmp_gtid in frame 5 of the call stack. However, TLS has still not been set up for the OpenMP runtime shared library for this thread and causing the access to </span><span style="caret-color: rgb(255, 38, 0);" class="">__kmp_gtid</span> to <span style="caret-color: rgb(255, 38, 0);" class="">go through the same protocol as before </span>(<span style="caret-color: rgb(4, 51, 255); color: rgb(4, 51, 255); font-family: Courier;" class="">__tls_get_addr </span><span style="caret-color: rgb(4, 51, 255);" class="">calls</span><span style="caret-color: rgb(4, 51, 255); color: rgb(4, 51, 255); font-family: Courier;" class=""> </span><span style="caret-color: rgb(4, 51, 255); color: rgb(4, 51, 255); font-family: Courier;" class="">tls_get_addr_tail </span><span style="caret-color: rgb(4, 51, 255);" class="">calls</span><span style="caret-color: rgb(4, 51, 255); color: rgb(4, 51, 255); font-family: Courier;" class=""> </span><span style="caret-color: rgb(4, 51, 255); color: rgb(4, 51, 255); font-family: Courier;" class="">pthread_mutex_lock).</span><span style="caret-color: rgb(255, 38, 0);" class=""> However, the lock has already been acquired in frame 21 so it is unavailable for acquistion in frames 0-2, causing deadlock. The TLS lock is implemented as a recursive lock, but the profiler interrupted the lock acquisition in libpthread before the owner field of the recursive lock was set, so the inner call to </span><font color="#0433ff" face="Courier" class="">pthread_mutex_lock </font>can't succeed.</div><div class=""><br class=""></div><div class=""><b class="">This is a serious problem if a profiler using the OMPT interface can cause a deadlock. </b></div><div class=""><span style="caret-color: rgb(34, 34, 34); color: rgb(34, 34, 34); font-family: Arial, Helvetica, Luxi-sans, "Nimbus Sans L", sans-serif; font-size: 12px;" class=""><br class=""></span></div><div class=""><font color="#222222" class="">We need a design of the OMPT interface and OpenMP runtime implementations that make this impossible.</font></div><div class=""><font color="#222222" class=""><br class=""></font></div><div class=""><font color="#222222" class="">After thinking about this for a while, I think that a profiler can arrange to receive the </font>ompt_callback_thread_begin and the profiler then set a thread local flag in its own TLS variables to note that a thread is an OpenMP thread. A profiler must not invoke any ompt runtime entry point on a thread that has not announced itself as an OpenMP thread by previously calling ompt_callback_thread_begin. An OpenMP runtime should ensure that its TLS is allocated before invoking the callback ompt_callback_thread_begin. Similarly, a profiler shouldn’t invoke an OMPT callback on a thread after receiving ompt_callback_thread_end.</div><div class=""><br class=""></div><div class="">If a profiler thread doesn’t use the OMPT interface to inspect a thread that hasn’t announced itself as an OpenMP thread, it won’t access any TLS state that the OpenMP library may maintain.</div><div class=""><br class=""></div><div class=""><span style="caret-color: rgb(4, 51, 255);" class="">Does anyone care to comment or offer a vision of a different solution?</span></div><div class=""><br class=""></div><div class=""><span style="caret-color: rgb(4, 51, 255);" class="">Below my signature block are some details of the thread state that I observed, in case you want to validate my assessment of the situation.</span></div><div class=""><br class=""></div><div class="">--<br class="">John Mellor-Crummey<span class="Apple-tab-span" style="white-space: pre;"> </span>Professor<br class="">Dept of Computer Science<span class="Apple-tab-span" style="white-space: pre;"> </span>Rice University<br class=""><a href="mailto:johnmc@rice.edu" class="">email: johnmc@rice.edu</a><span class="Apple-tab-span" style="white-space: pre;"> </span>phone: 713-348-5179</div><div class=""><span style="caret-color: rgb(34, 34, 34); color: rgb(34, 34, 34); font-family: Arial, Helvetica, Luxi-sans, "Nimbus Sans L", sans-serif; font-size: 12px;" class=""><br class=""></span></div><div class=""><span style="caret-color: rgb(34, 34, 34); color: rgb(34, 34, 34); font-family: Arial, Helvetica, Luxi-sans, "Nimbus Sans L", sans-serif; font-size: 12px;" class=""><br class=""></span></div><div class=""><br class=""></div><div class=""><div class=""><br class=""></div><div class="">In the 10 frames at the top of the stack, HPCToolkit is querying the OMPT interface to determine the thread identity. I’ve highlighted the top 5 frames in red. In frame 5, our profiler asks for the thread’s id by calling __kmp_get_global_thread_id. Below the callstack is a small snippet from the debugger that shows what is going on in frame 5 - on line 132, the runtime is trying to return the value of __kmp_gtid - a thread local variable. To resolve this variable reference, the runtime invokes __tls_get_addr which ends up trying to acquire the <span style="font-family: Monaco; font-size: 10px; background-color: rgb(255, 255, 255);" class="">dl_load_lock </span>as shown in frame 0.</div><div class=""><br class=""></div><div class="">The problem here is that the thread seems to have acquired the lock in frame 21 before getting interrupted by a sample. In frame 21, the thread just completed a cmpxchg to acquire the lock. I believe that this succeeded since RIP is now at JNE instruction (address <span style="caret-color: rgb(255, 147, 0); color: rgb(255, 147, 0);" class="">0x7f515ad25cd9) </span>immediately following the CMPXCHG at address 0x7f515ad25cd4 and the condition code ZF is set. As I read the machine state, the CAS succeeded in swapping in a 1 for a 0 to acquire the lock. </div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><font color="#ff2600" class="">The __kmp_gtid_mode was observed in the runtime to be 3 and __kmp_gtid is declared with modifier __thread.</font></div><div class=""><br class=""></div><div class=""><div class="">#if KMP_OS_LINUX && defined(KMP_TDATA_GTID)</div><div class="">int __kmp_gtid_mode = 3; /* use __declspec(thread) TLS to store gtid */</div><div class="">int __kmp_adjust_gtid_mode = FALSE;</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class="">#ifdef KMP_TDATA_GTID</div><div class="">#if KMP_OS_WINDOWS</div><div class="">__declspec(thread) int __kmp_gtid = KMP_GTID_DNE;</div><div class="">#else</div><div class="">__thread int __kmp_gtid = KMP_GTID_DNE;</div><div class="">#endif /* KMP_OS_WINDOWS - workaround because Intel(R) Many Integrated Core \</div><div class=""> compiler 20110316 doesn't accept __declspec */</div></div><div class=""><br class=""></div><div class="">The call stack <br class=""><div class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;"><br class=""></div>
</div>
<br class=""></div><div class=""><div class=""><font face="Courier" class="">#0 0x00007f515ad2a4cd in __lll_lock_wait () from /lib64/libpthread.so.0</font></div><div class=""><font face="Courier" class="">(gdb) where</font></div><div class=""><font color="#ff2600" face="Courier" class="">#0 0x00007f515ad2a4cd in __lll_lock_wait () from /lib64/libpthread.so.0</font></div><div class=""><font color="#ff2600" face="Courier" class="">#1 0x00007f515ad25de6 in _L_lock_870 () from /lib64/libpthread.so.0</font></div><div class=""><font color="#ff2600" face="Courier" class="">#2 0x00007f515ad25cdf in pthread_mutex_lock () from /lib64/libpthread.so.0</font></div><div class=""><font color="#ff2600" face="Courier" class="">#3 0x00007f515b7041cb in tls_get_addr_tail () from /lib64/ld-linux-x86-64.so.2</font></div><div class=""><font color="#ff2600" face="Courier" class="">#4 0x00007f515b709af8 in __tls_get_addr () from /lib64/ld-linux-x86-64.so.2</font></div><div class=""><font color="#ff2600" face="Courier" class="">#5 0x00007f515af7fd90 in __kmp_get_global_thread_id ()</font></div><div class=""><font face="Courier" class=""> at /home/johnmc/pkgs-src/openmp/runtime/src/kmp_runtime.cpp:132</font></div><div class=""><font face="Courier" class="">#6 0x00007f515b0068e4 in ompt_get_thread ()</font></div><div class=""><font face="Courier" class=""> at /home/johnmc/pkgs-src/openmp/runtime/src/ompt-specific.h:63</font></div><div class=""><font face="Courier" class="">#7 0x00007f515b003fd7 in __ompt_get_teaminfo (depth=0, size=0x0)</font></div><div class=""><font face="Courier" class=""> at /home/johnmc/pkgs-src/openmp/runtime/src/ompt-specific.cpp:47</font></div><div class=""><font face="Courier" class="">#8 0x00007f515b004381 in __ompt_get_parallel_id_internal (depth=0)</font></div><div class=""><font face="Courier" class=""> at /home/johnmc/pkgs-src/openmp/runtime/src/ompt-specific.cpp:216</font></div><div class=""><font face="Courier" class="">#9 0x00007f515b0062bc in ompt_get_parallel_id (ancestor_level=0)</font></div><div class=""><font face="Courier" class=""> at /home/johnmc/pkgs-src/openmp/runtime/src/ompt-general.cpp:348</font></div><div class=""><font face="Courier" class="">#10 0x00007f515b4a5e6c in hpcrun_ompt_get_parallel_id (level=0)</font></div><div class=""><font face="Courier" class=""> at ../../../../src/tool/hpcrun/ompt/ompt-interface.c:618</font></div><div class=""><font face="Courier" class="">#11 0x00007f515b4a4ef2 in need_defer_cntxt () at ../../../../src/tool/hpcrun/ompt/ompt-defer.c:148</font></div><div class=""><font face="Courier" class="">#12 0x00007f515b4a492a in ompt_backtrace_finalize (bt=0x7f513dbf5af0, isSync=0)</font></div><div class=""><font face="Courier" class=""> at ../../../../src/tool/hpcrun/ompt/ompt-callstack.c:551</font></div><div class=""><font face="Courier" class="">#13 0x00007f515b490dd0 in cct_backtrace_finalize (bt=0x7f513dbf5af0, isSync=0)</font></div><div class=""><font face="Courier" class=""> at ../../../../src/tool/hpcrun/cct_backtrace_finalize.c:51</font></div><div class=""><font face="Courier" class="">#14 0x00007f515b490bd9 in help_hpcrun_backtrace2cct (bundle=0x7f513d2f5fa8, context=0x7f513dbf6040, </font></div><div class=""><font face="Courier" class=""> metricId=0, metricIncr=..., skipInner=0, isSync=0, data=0x7f513dbf5d40)</font></div><div class=""><font face="Courier" class=""> at ../../../../src/tool/hpcrun/cct_insert_backtrace.c:401</font></div><div class=""><font face="Courier" class="">#15 0x00007f515b490651 in hpcrun_backtrace2cct (cct=0x7f513d2f5fa8, context=0x7f513dbf6040, metricId=0, </font></div><div class=""><font face="Courier" class=""> metricIncr=..., skipInner=0, isSync=0, data=0x7f513dbf5d40)</font></div><div class=""><font face="Courier" class=""> at ../../../../src/tool/hpcrun/cct_insert_backtrace.c:246</font></div><div class=""><font face="Courier" class="">#16 0x00007f515b493f5f in hpcrun_sample_callpath (context=0x7f513dbf6040, metricId=0, metricIncr=..., </font></div><div class=""><font face="Courier" class=""> skipInner=0, isSync=0, data=0x7f513dbf5cc0) at ../../../../src/tool/hpcrun/sample_event.c:239</font></div><div class=""><font face="Courier" class="">#17 0x00007f515b4a9020 in record_sample (current=0x7f513d2d9d68, mmap_data=0x7f513dbf5d40, </font></div><div class=""><font face="Courier" class=""> context=0x7f513dbf6040, sv=0x7f513dbf5f10)</font></div><div class=""><font face="Courier" class=""> at ../../../../src/tool/hpcrun/sample-sources/perf/linux_perf.c:567</font></div><div class=""><font face="Courier" class="">#18 0x00007f515b4aa388 in perf_event_handler (sig=38, siginfo=0x7f513dbf6170, context=0x7f513dbf6040)</font></div><div class=""><font face="Courier" class=""> at ../../../../src/tool/hpcrun/sample-sources/perf/linux_perf.c:1130</font></div><div class=""><font face="Courier" class="">#19 0x00007f515b25846d in monitor_signal_handler (sig=38, info=0x7f513dbf6170, context=0x7f513dbf6040)</font></div><div class=""><font face="Courier" class=""> at ../../libmonitor/src/signal.c:217</font></div><div class=""><font face="Courier" class="">#20 <signal handler called></font></div><div class=""><font color="#0433ff" face="Courier" class="">#21 0x00007f515ad25cd9 in pthread_mutex_lock () from /lib64/libpthread.so.0</font></div><div class=""><font color="#0433ff" face="Courier" class="">#22 0x00007f515b7041cb in tls_get_addr_tail () from /lib64/ld-linux-x86-64.so.2</font></div><div class=""><font color="#0433ff" face="Courier" class="">#23 0x00007f515b709af8 in __tls_get_addr () from /lib64/ld-linux-x86-64.so.2</font></div><div class=""><font color="#0433ff" face="Courier" class="">#24 0x00007f515aff3478 in __kmp_launch_worker (thr=0x15e1600)</font></div><div class=""><font color="#0433ff" face="Courier" class=""> at /home/johnmc/pkgs-src/openmp/runtime/src/z_Linux_util.cpp:510</font></div><div class=""><font face="Courier" class="">#25 0x00007f515b26112b in monitor_thread_fence2 () at ../../libmonitor/src/pthread.c:975</font></div><div class=""><font face="Courier" class="">#26 0x00007f515ad23dd5 in start_thread () from /lib64/libpthread.so.0</font></div><div class=""><font face="Courier" class="">#27 0x00007f515aa4db3d in clone () from /lib64/libc.so.6</font></div><div class=""><font face="Courier" class=""><br class=""></font></div><div class=""><font face="Courier" class=""><br class=""></font></div><div class=""><font color="#0433ff" face="Courier" class="">(gdb) frame 24</font></div><div class=""><font color="#0433ff" face="Courier" class="">#24 0x00007f515aff3478 in __kmp_launch_worker (thr=0x15e1600)</font></div><div class=""><font color="#0433ff" face="Courier" class=""> at /home/johnmc/pkgs-src/openmp/runtime/src/z_Linux_util.cpp:510</font></div><div class=""><font color="#0433ff" face="Courier" class="">510 __kmp_gtid = gtid;</font></div><div class=""><font face="Courier" class=""><br class=""></font></div><div class=""><font face="Courier" class=""><br class=""></font></div><div class=""><font color="#ff2600" face="Courier" class="">(gdb) frame 5</font></div><div class=""><font color="#ff2600" face="Courier" class="">#5 0x00007f515af7fd90 in __kmp_get_global_thread_id ()</font></div><div class=""><font color="#ff2600" face="Courier" class=""> at /home/johnmc/pkgs-src/openmp/runtime/src/kmp_runtime.cpp:132</font></div><div class=""><font color="#ff2600" face="Courier" class="">132 return __kmp_gtid;</font></div></div><div class=""><font face="Courier" class=""><br class=""></font></div></div><div class=""><font face="Courier" class=""><br class=""></font></div><div class=""><font face="Courier" class=""><br class=""></font></div><div class=""><font face="Courier" class="">Below is the state for frame 21, which shows that the PC is at the instruction following the cmpxchg and that the ZF bit is set, indicating the </font><span style="font-family: Courier;" class="">cmpxchg instruction succeeded, according to the instruction semantics below:</span></div><div class=""><font face="Courier" class=""><br class=""></font></div><div class=""><div class="">Semantics of the <span style="font-family: Courier;" class="">cmpxchg</span> instruction</div><div class=""><div class="page" title="Page 570"><div class="layoutArea"><div class="column"><p class=""><span style="font-size: 9pt; font-family: NeoSansIntel;" class="">(* Accumulator </span><span style="font-size: 11pt; font-family: Symbol;" class="">= </span><span style="font-size: 9pt; font-family: NeoSansIntel;" class="">AL, AX, EAX, or RAX depending on whether a byte, word, doubleword, or quadword comparison is being performed *)</span></p><p class=""><span style="font-size: 9pt; font-family: NeoSansIntel;" class="">IF EAX </span><span style="font-size: 11pt; font-family: Symbol;" class="">= </span><span style="font-size: 9pt; font-family: NeoSansIntel;" class="">DEST THEN</span></p><p class=""><span style="font-size: 9pt; font-family: NeoSansIntel;" class=""> ZF </span><span style="font-size: 9pt; font-family: Symbol;" class="">← </span><span style="font-size: 9pt; font-family: NeoSansIntel;" class="">1;</span></p><p class=""><span style="font-size: 9pt; font-family: NeoSansIntel;" class=""> DEST </span><span style="font-size: 9pt; font-family: Symbol;" class="">← </span><span style="font-size: 9pt; font-family: NeoSansIntel;" class="">SRC; </span></p><p class=""><span style="font-size: 9pt; font-family: NeoSansIntel;" class="">ELSE</span> </p></div></div><div class="layoutArea"><div class="column"></div></div></div><div class="page" title="Page 571"><div class="layoutArea"><div class="column"><p class=""><span style="font-size: 9pt; font-family: NeoSansIntel;" class=""> ZF </span><span style="font-size: 9pt; font-family: Symbol;" class="">← </span><span style="font-size: 9pt; font-family: NeoSansIntel;" class="">0;</span></p><p class=""><span style="font-size: 9pt; font-family: NeoSansIntel;" class=""> EAX </span><span style="font-size: 9pt; font-family: Symbol;" class="">← </span><span style="font-size: 9pt; font-family: NeoSansIntel;" class="">DEST; </span></p><p class=""><span style="font-size: 9pt; font-family: NeoSansIntel;" class="">FI;</span></p></div></div></div></div></div><div class=""><font face="Courier" class=""><br class=""></font></div><div class=""><div class=""><font face="Courier" class="">(gdb) frame 21</font></div><div class=""><font face="Courier" class="">#21 0x00007f515ad25cd9 in pthread_mutex_lock () from /lib64/libpthread.so.0</font></div><div class=""><font face="Courier" class="">(gdb) x/i 0x00007f515ad25cd9-5</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25cd4 <pthread_mutex_lock+164>: <font color="#ff40ff" class=""> lock cmpxchg %edi,(%r8)</font></font></div><div class=""><font face="Courier" class="">(gdb) </font></div><div class=""><font face="Courier" class="">=> 0x7f515ad25cd9 <pthread_mutex_lock+169>: jne 0x7f515ad25dd7 <_L_lock_870></font></div><div class=""><font face="Courier" class="">(gdb) info reg</font></div><div class=""><font face="Courier" class="">rax 0x0 0</font></div><div class=""><font face="Courier" class="">rbx 0x7f515b911b18 139987405314840</font></div><div class=""><font face="Courier" class="">rcx 0x43 67</font></div><div class=""><font face="Courier" class="">rdx 0x0 0</font></div><div class=""><font face="Courier" class="">rsi 0x0 0</font></div><div class=""><font face="Courier" class="">rdi 0x1 1</font></div><div class=""><font face="Courier" class="">rbp 0x7f513dbf6620 0x7f513dbf6620</font></div><div class=""><font face="Courier" class="">rsp 0x7f513dbf65d8 0x7f513dbf65d8</font></div><div class=""><font face="Courier" class="">r8 0x7f515b914908 139987405326600</font></div><div class=""><font face="Courier" class="">r9 0x0 0</font></div><div class=""><font face="Courier" class="">r10 0x22 34</font></div><div class=""><font face="Courier" class="">r11 0x7f515aadc440 139987390415936</font></div><div class=""><font face="Courier" class="">r12 0x7f515b241f28 139987398172456</font></div><div class=""><font face="Courier" class="">r13 0x501980 5249408</font></div><div class=""><font face="Courier" class="">r14 0x4 4</font></div><div class=""><font face="Courier" class="">r15 0x7f515b46f840 139987400456256</font></div><div class=""><font face="Courier" class="">rip 0x7f515ad25cd9 0x7f515ad25cd9 <pthread_mutex_lock+169></font></div><div class=""><font face="Courier" class="">eflags 0x246 [ PF <font color="#ff40ff" class="">ZF</font> IF ]</font></div><div class=""><font face="Courier" class="">cs 0x33 51</font></div><div class=""><font face="Courier" class="">ss 0x2b 43</font></div><div class=""><font face="Courier" class="">ds 0x0 0</font></div><div class=""><font face="Courier" class="">es 0x0 0</font></div><div class=""><font face="Courier" class="">fs 0x0 0</font></div><div class=""><font face="Courier" class="">gs 0x0 0</font></div></div><div class=""><font face="Courier" class=""><br class=""></font></div><div class=""><font face="Courier" class=""><br class=""></font></div><div class=""><font face="Courier" class=""><br class=""></font></div><div class=""><font face="Courier" class=""><br class=""></font></div><div class=""><div class="page" title="Page 571"><div class="layoutArea"><div class="column"><div class=""><font face="Courier" class="">Below is the CMPXCHG instruction in the context of the code for pthread_mutex_lock. The magenta instruction shows the compare and exchange that succeeded. The orange instructions show the path to return following a successful acquisition. The blue comments indicate the instructions completing the acquisition of the recursive lock by setting the count, owner, and nusers</font></div><div class=""><font face="Courier" class=""><br class=""></font></div><div class=""><div class=""><font face="Courier" class=""> 0x7f515ad25c30 <pthread_mutex_lock>: mov 0x10(%rdi),%esi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c33 <pthread_mutex_lock+3>: mov %rdi,%r8</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c36 <pthread_mutex_lock+6>: mov %esi,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c38 <pthread_mutex_lock+8>: and $0x17f,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c3d <pthread_mutex_lock+13>: nop</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c3e <pthread_mutex_lock+14>: mov %esi,%edx</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c40 <pthread_mutex_lock+16>: and $0x7c,%edx</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c43 <pthread_mutex_lock+19>: jne 0x7f515ad25ce9 <pthread_mutex_lock+185></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c49 <pthread_mutex_lock+25>: test %eax,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c4b <pthread_mutex_lock+27>: jne 0x7f515ad25cad <pthread_mutex_lock+125></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c4d <pthread_mutex_lock+29>: mov 0x21182d(%rip),%eax # 0x7f515af37480 <__pthread_force_elision></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c53 <pthread_mutex_lock+35>: test %eax,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c55 <pthread_mutex_lock+37>: je 0x7f515ad25c80 <pthread_mutex_lock+80></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c57 <pthread_mutex_lock+39>: test $0x300,%esi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c5d <pthread_mutex_lock+45>: jne 0x7f515ad25c80 <pthread_mutex_lock+80></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c5f <pthread_mutex_lock+47>: or $0x100,%esi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c65 <pthread_mutex_lock+53>: mov %esi,0x10(%rdi)</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c68 <pthread_mutex_lock+56>: mov %esi,%edx</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c6a <pthread_mutex_lock+58>: lea 0x16(%r8),%rsi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c6e <pthread_mutex_lock+62>: mov %r8,%rdi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c71 <pthread_mutex_lock+65>: and $0x80,%edx</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c77 <pthread_mutex_lock+71>: jmpq 0x7f515ad2c9a0 <__lll_lock_elision></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c7c <pthread_mutex_lock+76>: nopl 0x0(%rax)</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c80 <pthread_mutex_lock+80>: and $0x80,%esi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c86 <pthread_mutex_lock+86>: mov $0x1,%edi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c8b <pthread_mutex_lock+91>: xor %eax,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c8d <pthread_mutex_lock+93>: lock cmpxchg %edi,(%r8)</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25c92 <pthread_mutex_lock+98>: jne 0x7f515ad25dbc <_L_lock_812></font></div><div class=""><font color="#ff9300" face="Courier" class=""> 0x7f515ad25c98 <pthread_mutex_lock+104>: mov %fs:0x2d0,%eax </font><font color="#0433ff" class="">// get owner</font></div><div class=""><font color="#ff9300" style="font-family: Courier;" class=""> 0x7f515ad25ca0 <pthread_mutex_lock+112>: mov %eax,0x8(%r8) </font><font color="#0433ff" class=""><span style="caret-color: rgb(4, 51, 255);" class="">// set </span><span style="background-color: rgb(255, 255, 255);" class="">mutex->__data.</span><span style="caret-color: rgb(4, 51, 255);" class="">__owner</span></font></div><div class=""><font color="#ff9300" style="font-family: Courier;" class=""> 0x7f515ad25ca4 <pthread_mutex_lock+116>: addl $0x1,0xc(%r8) </font><font color="#0433ff" class="">// increment <span style="background-color: rgb(255, 255, 255);" class="">mutex->__data.</span><span style="background-color: rgb(255, 255, 255);" class="">__nusers</span></font></div><div class=""><font color="#ff9300" face="Courier" class=""> 0x7f515ad25ca9 <pthread_mutex_lock+121>: nop</font></div><div class=""><font color="#ff9300" face="Courier" class=""> 0x7f515ad25caa <pthread_mutex_lock+122>: xor %eax,%eax</font></div><div class=""><font face="Courier" class=""><font color="#ff9300" class=""> 0x7f515ad25cac <pthread_mutex_lock+124>: retq </font> </font></div><div class=""><font face="Courier" class=""> 0x7f515ad25cad <pthread_mutex_lock+125>: cmp $0x100,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25cb2 <pthread_mutex_lock+130>: je 0x7f515ad25c68 <pthread_mutex_lock+56></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25cb4 <pthread_mutex_lock+132>: mov %esi,%edi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25cb6 <pthread_mutex_lock+134>: and $0x7f,%edi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25cb9 <pthread_mutex_lock+137>: cmp $0x1,%edi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25cbc <pthread_mutex_lock+140>: jne 0x7f515ad25d05 <pthread_mutex_lock+213></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25cbe <pthread_mutex_lock+142>: mov %fs:0x2d0,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25cc6 <pthread_mutex_lock+150>: cmp 0x8(%r8),%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25cca <pthread_mutex_lock+154>: je 0x7f515ad25cee <pthread_mutex_lock+190></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25ccc <pthread_mutex_lock+156>: and $0x80,%esi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25cd2 <pthread_mutex_lock+162>: mov %edx,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25cd4 <pthread_mutex_lock+164>: </font><font color="#ff40ff" style="font-family: Courier;" class=""> lock cmpxchg %edi,(%r8)</font><font color="#0433ff" class=""><font face="Courier" class=""> </font>// successful CAS of <span style="background-color: rgb(255, 255, 255);" class="">mutex->__data.</span>__lock</font></div><div class=""><font face="Courier" class=""><font color="#ff9300" class="">=> 0x7f515ad25cd9 <pthread_mutex_lock+169>: jne 0x7f515ad25dd7 <_L_lock_870> </font><font color="#ff2600" class=""><b class=""> <--profiler interrupts here</b></font></font></div><div class=""><font color="#ff9300" style="font-family: Courier;" class=""> 0x7f515ad25cdf <pthread_mutex_lock+175>: movl $0x1,0x4(%r8) </font><font color="#0433ff" class="">// set <span style="background-color: rgb(255, 255, 255);" class="">mutex->__data.__count</span> to 1</font></div><div class=""><font color="#ff9300" face="Courier" class=""> 0x7f515ad25ce7 <pthread_mutex_lock+183>: jmp 0x7f515ad25c98 <pthread_mutex_lock+104></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25ce9 <pthread_mutex_lock+185>: jmpq 0x7f515ad218b0 <__pthread_mutex_lock_full></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25cee <pthread_mutex_lock+190>: mov 0x4(%r8),%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25cf2 <pthread_mutex_lock+194>: cmp $0xffffffff,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25cf5 <pthread_mutex_lock+197>: je 0x7f515ad25d84 <pthread_mutex_lock+340></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25cfb <pthread_mutex_lock+203>: add $0x1,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25cfe <pthread_mutex_lock+206>: mov %eax,0x4(%r8)</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d02 <pthread_mutex_lock+210>: xor %eax,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d04 <pthread_mutex_lock+212>: retq </font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d05 <pthread_mutex_lock+213>: cmp $0x3,%edi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d08 <pthread_mutex_lock+216>: jne 0x7f515ad25da4 <pthread_mutex_lock+372></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d0e <pthread_mutex_lock+222>: cmpl $0x0,0x2116cf(%rip) # 0x7f515af373e4 <__is_smp></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d15 <pthread_mutex_lock+229>: je 0x7f515ad25c80 <pthread_mutex_lock+80></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d1b <pthread_mutex_lock+235>: mov %edx,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d1d <pthread_mutex_lock+237>: mov $0x1,%edx</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d22 <pthread_mutex_lock+242>: lock cmpxchg %edx,(%r8)</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d27 <pthread_mutex_lock+247>: test %eax,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d29 <pthread_mutex_lock+249>: je 0x7f515ad25c98 <pthread_mutex_lock+104></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d2f <pthread_mutex_lock+255>: movswl 0x14(%r8),%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d34 <pthread_mutex_lock+260>: mov $0x1,%esi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d39 <pthread_mutex_lock+265>: lea 0xa(%rax,%rax,1),%ecx</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d3d <pthread_mutex_lock+269>: mov $0x64,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d42 <pthread_mutex_lock+274>: cmp $0x64,%ecx</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d45 <pthread_mutex_lock+277>: cmovg %eax,%ecx</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d48 <pthread_mutex_lock+280>: xor %dl,%dl</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d4a <pthread_mutex_lock+282>: xor %dil,%dil</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d4d <pthread_mutex_lock+285>: add $0x1,%edx</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d50 <pthread_mutex_lock+288>: lea -0x1(%rdx),%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d53 <pthread_mutex_lock+291>: cmp %eax,%ecx</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d55 <pthread_mutex_lock+293>: jle 0x7f515ad25d8a <pthread_mutex_lock+346></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d57 <pthread_mutex_lock+295>: pause </font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d59 <pthread_mutex_lock+297>: mov %edi,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d5b <pthread_mutex_lock+299>: lock cmpxchg %esi,(%r8)</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d60 <pthread_mutex_lock+304>: test %eax,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d62 <pthread_mutex_lock+306>: jne 0x7f515ad25d4d <pthread_mutex_lock+285></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d64 <pthread_mutex_lock+308>: movzwl 0x14(%r8),%ecx</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d69 <pthread_mutex_lock+313>: mov $0x8,%esi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d6e <pthread_mutex_lock+318>: movswl %cx,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d71 <pthread_mutex_lock+321>: sub %eax,%edx</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d73 <pthread_mutex_lock+323>: mov %edx,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d75 <pthread_mutex_lock+325>: cltd </font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d76 <pthread_mutex_lock+326>: idiv %esi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d78 <pthread_mutex_lock+328>: add %ecx,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d7a <pthread_mutex_lock+330>: mov %ax,0x14(%r8)</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d7f <pthread_mutex_lock+335>: jmpq 0x7f515ad25c98 <pthread_mutex_lock+104></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d84 <pthread_mutex_lock+340>: mov $0xb,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d89 <pthread_mutex_lock+345>: retq </font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d8a <pthread_mutex_lock+346>: mov 0x10(%r8),%esi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d8e <pthread_mutex_lock+350>: mov $0x1,%edi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d93 <pthread_mutex_lock+355>: xor %eax,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d95 <pthread_mutex_lock+357>: and $0x80,%esi</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25d9b <pthread_mutex_lock+363>: lock cmpxchg %edi,(%r8)</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25da0 <pthread_mutex_lock+368>: jne 0x7f515ad25df2 <_L_lock_1022></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25da2 <pthread_mutex_lock+370>: jmp 0x7f515ad25d64 <pthread_mutex_lock+308></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25da4 <pthread_mutex_lock+372>: mov %fs:0x2d0,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25dac <pthread_mutex_lock+380>: cmp 0x8(%r8),%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25db0 <pthread_mutex_lock+384>: jne 0x7f515ad25c80 <pthread_mutex_lock+80></font></div><div class=""><font face="Courier" class=""> 0x7f515ad25db6 <pthread_mutex_lock+390>: mov $0x23,%eax</font></div><div class=""><font face="Courier" class=""> 0x7f515ad25dbb <pthread_mutex_lock+395>: retq </font></div></div>
</div>
</div>
</div></div></div></body></html>