<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr">
<p>Yes, we have contacted RHEL people. It repros there too. </p>
<p><br>
</p>
<div id="Signature">
<p>Sent from <a href="http://aka.ms/weboutlook" id="LPNoLP">Outlook</a><br>
</p>
</div>
<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Pavel Labath <labath@google.com><br>
<b>Sent:</b> Monday, January 9, 2017 2:53 AM<br>
<b>To:</b> Eugene Birukov<br>
<b>Cc:</b> LLDB<br>
<b>Subject:</b> Re: [lldb-dev] Lldb-server spins forever in ptrace with 100% CPU on Linux Ubuntu 16.04</font>
<div> </div>
</div>
<div>
<div dir="ltr">Nice catch. I think this backtrace should be enough to bring the problem to the attention of the kernel developers. Have you tried contacting them already? I can help you with that if you want.</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 6 January 2017 at 23:34, Eugene Birukov <span dir="ltr">
<<a href="mailto:eugenebi@hotmail.com" target="_blank">eugenebi@hotmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div id="m_-8127642431370355060divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Hi,</p>
<p><br>
</p>
<p>It seems the problem is related to transparent huge pages. Disabling them helps.</p>
<p><br>
</p>
<p>The LLDB spins in this code:</p>
<p><br>
</p>
<p></p>
<div>[11428.634820] Call Trace:</div>
<div>[11428.634821] [<ffffffff811ec7de>] do_huge_pmd_wp_page+0x62e/<wbr>0xb80</div>
<div>[11428.634822] [<ffffffff811b0f75>] handle_mm_fault+0x705/0xfe0</div>
<div>[11428.634823] [<ffffffff811aa34d>] ? follow_page_mask+0x37d/0x7a0</div>
<div>[11428.634824] [<ffffffff811aa90b>] __get_user_pages+0x19b/0x660</div>
<div>[11428.634825] [<ffffffff811ab222>] get_user_pages+0x52/0x60</div>
<div>[11428.634825] [<ffffffff811ac00b>] __access_remote_vm+0xcb/0x1f0</div>
<div>[11428.634826] [<ffffffff811b2360>] access_process_vm+0x50/0x70</div>
<div>[11428.634827] [<ffffffff8109500a>] ptrace_request+0x2da/0x620</div>
<div>[11428.634828] [<ffffffff810c24dc>] ? wait_task_inactive+0x16c/0x1f0</div>
<div>[11428.634829] [<ffffffff81038cab>] arch_ptrace+0x29b/0x300</div>
<div>[11428.634830] [<ffffffff81094c6e>] SyS_ptrace+0xbe/0x110</div>
<div>[11428.634831] [<ffffffff816967c9>] system_call_fastpath+0x16/0x1b</div>
<div><br>
</div>
Eugene
<p></p>
<p><br>
</p>
<div id="m_-8127642431370355060Signature">
<p>Sent from <a href="http://aka.ms/weboutlook" id="m_-8127642431370355060LPNoLP" target="_blank">
Outlook</a><br>
</p>
</div>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div id="m_-8127642431370355060divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Pavel Labath <<a href="mailto:labath@google.com" target="_blank">labath@google.com</a>><br>
<b>Sent:</b> Tuesday, December 13, 2016 1:39 AM
<div>
<div class="h5"><br>
<b>To:</b> Eugene Birukov<br>
<b>Subject:</b> Re: [lldb-dev] Lldb-server spins forever in ptrace with 100% CPU on Linux Ubuntu 16.04</div>
</div>
</font>
<div> </div>
</div>
<div>
<div class="h5">
<div>
<div dir="ltr">Yeah, we support a couple of communication transports. For example, you can switch to using unix sockets instead. It's easiest if you just connect the server and client manually (for automatically starting up the server you would have to edit
the source, I think.)
<div><br>
</div>
<div>$ bin/lldb-server gdbserver unix-abstract:///tmp/X -- /bin/ls</div>
<div><br>
</div>
<div>
<div>$ bin/lldb</div>
<div>(lldb) process connect unix-abstract-connect:///tmp/X</div>
<div>Process 21731 stopped</div>
<div>* thread #1, name = 'ls', stop reason = signal SIGSTOP</div>
<div> frame #0: 0x00007f4b341bb2d0</div>
<div>-> 0x7f4b341bb2d0: movq %rsp, %rdi</div>
<div> 0x7f4b341bb2d3: callq 0x7f4b341bea40</div>
<div> 0x7f4b341bb2d8: movq %rax, %r12</div>
<div> 0x7f4b341bb2db: movl 0x221b17(%rip), %eax</div>
<div><br>
</div>
<div><br>
</div>
<div>Good luck. Let me know what you find out.</div>
<div>pl</div>
<div><br>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 12 December 2016 at 18:11, Eugene Birukov <span dir="ltr">
<<a href="mailto:eugenebi@hotmail.com" target="_blank">eugenebi@hotmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div id="m_-8127642431370355060m_-5865990061185604691divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>OK, thanks!</p>
<p>I'll work on getting reliable repro on something I can attach kgdb to.</p>
<p><br>
</p>
<p>Meanwhile - is there any options in lldb-server communication protocol I could play with? Say, using pipe instead of tcp/ip or vice versa? </p>
<p><br>
</p>
<div id="m_-8127642431370355060m_-5865990061185604691Signature">
<p>Sent from <a href="http://aka.ms/weboutlook" id="m_-8127642431370355060m_-5865990061185604691LPNoLP" target="_blank">
Outlook</a><br>
</p>
</div>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div id="m_-8127642431370355060m_-5865990061185604691divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Pavel Labath <<a href="mailto:labath@google.com" target="_blank">labath@google.com</a>><br>
<b>Sent:</b> Friday, December 9, 2016 2:56 AM
<div>
<div class="m_-8127642431370355060h5"><br>
<b>To:</b> Eugene Birukov<br>
<b>Subject:</b> Re: [lldb-dev] Lldb-server spins forever in ptrace with 100% CPU on Linux Ubuntu 16.04</div>
</div>
</font>
<div> </div>
</div>
<div>
<div class="m_-8127642431370355060h5">
<div>
<div dir="ltr">It looks like debugging this remotely will not lead anywhere. I don't think I have enough knowledge about the kernel to help you with that. If I were able to reproduce it myself, maybe I'd get somewhere by random experiments, but doing it over
the email is too slow. :)
<div><br>
<div>I suggest you ask some of the kernel developers about it. They should be able to direct you where to look further, and may be aware of kernel changes between the two versions that would affect this.</div>
</div>
<div><br>
</div>
<div>cheers,</div>
<div>pl</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 8 December 2016 at 17:12, Eugene Birukov <span dir="ltr">
<<a href="mailto:eugenebi@hotmail.com" target="_blank">eugenebi@hotmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div id="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<span>
<p><i>> </i><span style="font-family:Calibri,Arial,Helvetica,sans-serif,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:12.8px"><i>From your description is sounds like un-predicability comes from
the tracee side</i></span></p>
<p><br>
</p>
</span>
<p>Not exactly. If problem is there, it repro's 100%. Things that affect repro - make it go away or shift to different address:</p>
<p><br>
</p>
<p></p>
<ol>
<li>Delays in the client. If I put ::sleep(0) right after sending write memory request (not sure - are we writing in pipe or in socket?) - it might kill repro. <br>
</li><li>Changes in VM. Copying all bits into would be identical VM hurts repro.</li><li>Changes in the target application. I assume this is mainly layout of different VMA that shift after significant code change because target is fully stopped and it is hard to believe that the contents of the memory matters... Just in case - I tried to write
different values and change write size to 8 (not to trigger the recursion in llldb-server).</li></ol>
<p></p>
<p>Things that do not matter:</p>
<p></p>
<ul>
<li>Delays in the server. As soon as we are about to call ptrace, no amount of delays caused by stepping, etc. matters, the damage is already there.</li></ul>
<p></p>
<p>So, it seems that the problem is really not in ptrace but in the communication channel between the client and the server. Ptrace looks like just an innocent bystander caught in the gang crossfire on the street.</p>
<p><br>
</p>
<p>BTW, I am on short vacation, will be back on Monday 12/12.</p>
<p><br>
</p>
<div id="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226Signature">
<p>Sent from <a href="http://aka.ms/weboutlook" id="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226LPNoLP" target="_blank">
Outlook</a><br>
</p>
</div>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div id="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Pavel Labath <<a href="mailto:labath@google.com" target="_blank">labath@google.com</a>><br>
<b>Sent:</b> Thursday, December 8, 2016 3:19 AM<br>
<b>To:</b> Eugene Birukov
<div>
<div class="m_-8127642431370355060m_-5865990061185604691h5"><br>
<b>Subject:</b> Re: [lldb-dev] Lldb-server spins forever in ptrace with 100% CPU on Linux Ubuntu 16.04</div>
</div>
</font>
<div> </div>
</div>
<div>
<div class="m_-8127642431370355060m_-5865990061185604691h5">
<div>
<div dir="ltr">Thanks for the info. I agree with your assessment, it definitely looks like a kernel bug.
<div><br>
</div>
<div>I don't know how to trigger a kernel core dump. I have done a bit of kernel debugging/development, but it mainly consisted of adding a bunch of printf's and watching the dmesg output. I think we should contact the linux kernel mailing list (LKML.org) with
this problem. I've had pretty good success with this in the past. The one contact I have there is <span style="font-size:12.8px">Oleg Nesterov <<a href="mailto:oleg@redhat.com" target="_blank">oleg@redhat.com</a>>. He's quite a ptrace expert, and was able
to fix my bug in a day when I contacted him with a not-too-unsimilar problem to yours (although this does sound more complicated, I suspect some interaction between ptrace and virtual memory manager).</span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px">Before we do that, we may want to reduce the scope of the problem a bit. From your description is sounds like un-predicability comes from the tracee side, and once it gets to a bad state, all tracer writes to the affected
area will hang. In that case, it should be possible to take lldb out of the equation, and instead write a trivial app, which takes a pid and an address, and does a PTRACE_ATTACH + a bunch of PTRACE_POKEDATA to the given address. I think this would be enough
to get kernel folks interested, as it clearly demonstrates it's not lldb being funny, and they may direct us how to debug this further. I have adapted one of my test apps to do this (attached). Could you try this out on your system?</span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px">pl</span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 8 December 2016 at 00:57, Eugene Birukov <span dir="ltr">
<<a href="mailto:eugenebi@hotmail.com" target="_blank">eugenebi@hotmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div id="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226m_-2065094377441467094divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Some additional info.</p>
<p><br>
</p>
<p></p>
<p class="MsoNormal">I stepped through individual instructions… It hangs in syscall<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">37 in ../sysdeps/unix/sysv/linux/ptr<wbr>ace.c<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">=> 0x00007fcac11ee21b <ptrace+43>: 48 63 70 08 movslq 0x8(%rax),%rsi<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">(gdb)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">38 in ../sysdeps/unix/sysv/linux/ptr<wbr>ace.c<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">=> 0x00007fcac11ee21f <ptrace+47>: 48 8b 50 10 mov 0x10(%rax),%rdx<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">(gdb)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">45 in ../sysdeps/unix/sysv/linux/ptr<wbr>ace.c<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">=> 0x00007fcac11ee223 <ptrace+51>: 89 ff mov %edi,%edi<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">(gdb)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">36 in ../sysdeps/unix/sysv/linux/ptr<wbr>ace.c<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">=> 0x00007fcac11ee225 <ptrace+53>: 48 89 44 24 c8 mov %rax,-0x38(%rsp)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">(gdb)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">45 in ../sysdeps/unix/sysv/linux/ptr<wbr>ace.c<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">=> 0x00007fcac11ee22a <ptrace+58>: 4c 0f 47 50 18 cmova 0x18(%rax),%r10<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New""> 0x00007fcac11ee22f <ptrace+63>: b8 65 00 00 00 mov $0x65,%eax<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">(gdb)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">0x00007fcac11ee22f 45 in ../sysdeps/unix/sysv/linux/ptr<wbr>ace.c<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New""> 0x00007fcac11ee22a <ptrace+58>: 4c 0f 47 50 18 cmova 0x18(%rax),%r10<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">=> 0x00007fcac11ee22f <ptrace+63>: b8 65 00 00 00 mov $0x65,%eax<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">(gdb)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">38 in ../sysdeps/unix/sysv/linux/ptr<wbr>ace.c<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">=> 0x00007fcac11ee234 <ptrace+68>: c7 44 24 b8 18 00 00 00 movl $0x18,-0x48(%rsp)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">(gdb)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">45 in ../sysdeps/unix/sysv/linux/ptr<wbr>ace.c<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8pt; font-family:"Courier New"; background-color:rgb(255,255,0)">=> 0x00007fcac11ee23c <ptrace+76>: 0f 05 syscall<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New""> 0x00007fcac11ee23e <ptrace+78>: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New""> 0x00007fcac11ee244 <ptrace+84>: 77 2a ja 0x7fcac11ee270 <ptrace+128><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt; font-family:"Courier New"">(gdb)<u></u><u></u></span></p>
<br>
<p></p>
<p>Also, user vs. kernel times are about 1:1000 ratio according to proc/pid/stat:</p>
<p><br>
</p>
<p></p>
<p style="margin:0in; font-family:"Courier New"; font-size:9.0pt"># cat /proc/21809/stat</p>
<p style="margin:0in; font-family:"Courier New"; font-size:9.0pt">21809 (lldb-server) R 21780 21809 21809 0 -1 4194304 1175 1709 0 0
<span style="background:yellow">18 18398</span> 4 0 20 0 1 0 1602761 48611328 2102 18446744073709551615 4194304 11143636 140733729492144 140733729487496 140187105837630 262144 65537 4096 65537 0 0 0 17 1 0 0 0 0 0 13244688 13945048 29155328 140733729499314
140733729499429 140733729499429 140733729501128 0</p>
<br>
<p></p>
<p>Eugene</p>
<p><br>
</p>
<div id="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226m_-2065094377441467094Signature">
<p>Sent from <a href="http://aka.ms/weboutlook" id="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226m_-2065094377441467094LPNoLP" target="_blank">
Outlook</a><br>
</p>
<div class="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226hm m_-8127642431370355060m_-5865990061185604691m_1629372973991788226HOEnZb">
</div>
<p></p>
<div class="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226hm m_-8127642431370355060m_-5865990061185604691m_1629372973991788226HOEnZb">
</div>
</div>
<div class="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226hm m_-8127642431370355060m_-5865990061185604691m_1629372973991788226HOEnZb">
<br>
<br>
</div>
<div style="color:rgb(0,0,0)">
<div class="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226hm m_-8127642431370355060m_-5865990061185604691m_1629372973991788226HOEnZb">
<hr style="display:inline-block; width:98%">
</div>
<div id="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226m_-2065094377441467094divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt">
<div class="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226hm m_-8127642431370355060m_-5865990061185604691m_1629372973991788226HOEnZb">
<b>From:</b> Eugene Birukov <<a href="mailto:eugenebi@hotmail.com" target="_blank">eugenebi@hotmail.com</a>><br>
<b>Sent:</b> Wednesday, December 7, 2016 1:15 PM<br>
<b>To:</b> Pavel Labath; Eugene Birukov</div>
<div>
<div class="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226h5">
<br>
<b>Subject:</b> Re: [lldb-dev] Lldb-server spins forever in ptrace with 100% CPU on Linux Ubuntu 16.04</div>
</div>
</font>
<div> </div>
</div>
<div>
<div class="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226h5">
<div>
<div id="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226m_-2065094377441467094divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>And yes - I suspect kernel bug because we did not see it on 15.10 but on 16.04 I have several people reporting it. In one case it would repro on 3 out of 4 kind of "identical" machines.</p>
<p><br>
</p>
<p>I do not have simple repro, but I can find victim VM that we do not care much about. Is there a way to cause Linux kernel core dump there?</p>
<p><br>
</p>
<div id="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226m_-2065094377441467094Signature">
<p>Sent from <a href="http://aka.ms/weboutlook" id="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226m_-2065094377441467094LPNoLP" target="_blank">
Outlook</a><br>
</p>
</div>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div id="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226m_-2065094377441467094divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> lldb-dev <<a href="mailto:lldb-dev-bounces@lists.llvm.org" target="_blank">lldb-dev-bounces@lists.llvm.o<wbr>rg</a>> on behalf of Eugene Birukov via lldb-dev <<a href="mailto:lldb-dev@lists.llvm.org" target="_blank">lldb-dev@lists.llvm.org</a>><br>
<b>Sent:</b> Wednesday, December 7, 2016 11:36 AM<br>
<b>To:</b> Pavel Labath<br>
<b>Cc:</b> LLDB<br>
<b>Subject:</b> Re: [lldb-dev] Lldb-server spins forever in ptrace with 100% CPU on Linux Ubuntu 16.04</font>
<div> </div>
</div>
<div>
<div id="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226m_-2065094377441467094divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<blockquote style="margin:0 0 0 40px; border:none; padding:0px">
<p><i>> <span style="font-family:Calibri,Arial,Helvetica,sans-serif,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:13.3333px">could you get a backtrace of lldb-server when it is in the "stuck"</span></i></p>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:13.3333px"><i>state (just attach with lldb/gdb after it hangs and do "bt")?</i></span></p>
</blockquote>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:13.3333px"><br>
</span></p>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:13.3333px"><span style="font-size:12pt">You wish
</span><img naturalheight="19" naturalwidth="19" class="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226m_-2065094377441467094EmojiInsert" id="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226m_-2065094377441467094OWAEmoji35851" alt="☹" style="vertical-align: bottom; user-select: none;" aria-expanded="false" tabindex="0" src="cid:eb2d8a09-d75b-49ee-b1d4-c804a990d92a"><span style="font-size:12pt"> The </span><span style="font-size:12pt">lldb-server
does not react to any signals including SIGSTOP, so gdb just hangs forever.</span></span></p>
<p><br>
</p>
<blockquote style="margin:0 0 0 40px; border:none; padding:0px">
<p><i>> <span style="font-family:Calibri,Arial,Helvetica,sans-serif,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:13.3333px">If you can get me reasonably detailed repro steps, I can try to </span></i><span style="font-family:Calibri,Arial,Helvetica,sans-serif,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:13.3333px"><i>investigate</i></span></p>
</blockquote>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:13.3333px"><i></i></span></p>
<p><br>
</p>
<p>Unfortunately I do not have repro myself. It happens semi-randomly on some machines and I need to borrow the machine with the problem. Here are some details from my records:</p>
<p></p>
<ul>
<li>It is pretty big piece of RX memory, /proc/<pid>/maps shows this: <br>
409701000-40b49c000 r-xp 00000000 00:00 0</li><li>Writing into some locations within that VMA works</li><li>When it repros, it is pretty consistent, but changing in the target may shift it - i.e. make no repro or fail at different address.</li></ul>
<p></p>
<blockquote style="margin:0 0 0 40px; border:none; padding:0px">
<p><i>> </i><span style="font-family:Calibri,Arial,Helvetica,sans-serif,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:13.3333px"><i>are you able to still reproduce the bug with logging enabled?</i></span></p>
</blockquote>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:13.3333px"><i></i></span></p>
<p><br>
</p>
<p></p>
<div>Yes. Here are a few last lines from the log:</div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:8pt">1481139040.768961000 0x7fff253c9780 Communication::Write (src = 0x7fff253c8f48, src_len = 7) connection = 0x24a6bd0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:8pt">1481139040.768963000 0x24a6bd0 ConnectionFileDescriptor::Writ<wbr>e (src = 0x7fff253c8f48, src_len = 7)</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:8pt">1481139040.768973000 0x24a6cc0 Socket::Write() (socket = 6, src = 0x7fff253c8f48, src_len = 7, flags = 0) => 7 (error = (null))</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:8pt">1481139040.768976000 0x24a6bd0 ConnectionFileDescriptor::Writ<wbr>e(fd = 6, src = 0x7fff253c8f48, src_len = 7) => 7 (error = (null))</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:8pt">1481139040.768979000 0x7fff253c9780 Communication::Read (dst = 0x7fff253c7140, dst_len = 8192, timeout = 0 usec) connection = 0x24a6bd0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:8pt">1481139040.768982000 0x24a6bd0 ConnectionFileDescriptor::Byte<wbr>sAvailable (timeout_usec = 0)</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:8pt">1481139040.768984000 0x24a6bd0 ConnectionFileDescriptor::Byte<wbr>sAvailable() ::select (nfds=7, fds={6, 4}, NULL, NULL, timeout=0x7fff253c6d80)...</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:8pt">1481139040.768986000 0x24a6bd0 ConnectionFileDescriptor::Byte<wbr>sAvailable() ::select (nfds=7, fds={6, 4}, NULL, NULL, timeout=0x7fff253c6d80) => 0, error = (null)</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:8pt">1481139090.788317000 0x7fff253c9780 Communication::Read (dst = 0x7fff253c7140, dst_len = 8192, timeout = 0 usec) connection = 0x24a6bd0</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:8pt">1481139090.788356000 0x24a6bd0 ConnectionFileDescriptor::Byte<wbr>sAvailable (timeout_usec = 0)</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:8pt">1481139090.788364000 0x24a6bd0 ConnectionFileDescriptor::Byte<wbr>sAvailable() ::select (nfds=7, fds={6, 4}, NULL, NULL, timeout=0x7fff253c6d80)...</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:8pt">1481139090.788368000 0x24a6bd0 ConnectionFileDescriptor::Byte<wbr>sAvailable() ::select (nfds=7, fds={6, 4}, NULL, NULL, timeout=0x7fff253c6d80) => 1, error = (null)</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:8pt">1481139090.788378000 0x24a6cc0 Socket::Read() (socket = 6, src = 0x7fff253c7140, src_len = 25, flags = 0) => 25 (error = (null))</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:8pt">1481139090.788382000 0x24a6bd0 ConnectionFileDescriptor::Read<wbr>() fd = 6, dst = 0x7fff253c7140, dst_len = 8192) => 25, error = (null)</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:8pt">1481139090.788395000 NativeProcessLinux::WriteMemor<wbr>y(0x409d5a7d0, 0x25271d0, 4)</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:8pt">1481139090.788409000 NativeProcessLinux::ReadMemory using process_vm_readv to read 8 bytes from inferior address 0x409d5a7d0: Success</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:8pt">1481139090.788414000 PTRACE_POKEDATA [1][0][0][0][57][41][54][41]</span></div>
<div><br>
</div>
<p></p>
<p>Thanks,</p>
<p>Eugene</p>
<p><br>
</p>
<div id="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226m_-2065094377441467094Signature">
<p>Sent from <a href="http://aka.ms/weboutlook" id="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226m_-2065094377441467094LPNoLP" target="_blank">
Outlook</a><br>
</p>
</div>
<br>
<br>
<div style="color:rgb(0,0,0)">
<div>
<hr style="display:inline-block; width:98%">
<div id="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226m_-2065094377441467094x_divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Pavel Labath <<a href="mailto:labath@google.com" target="_blank">labath@google.com</a>><br>
<b>Sent:</b> Wednesday, December 7, 2016 2:34 AM<br>
<b>To:</b> Eugene Birukov<br>
<b>Cc:</b> LLDB<br>
<b>Subject:</b> Re: [lldb-dev] Lldb-server spins forever in ptrace with 100% CPU on Linux Ubuntu 16.04</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt">
<div class="m_-8127642431370355060m_-5865990061185604691m_1629372973991788226m_-2065094377441467094PlainText">
Hello Eugene,<br>
<br>
this sounds troubling, and I'd like to get to the bottom of this. If<br>
you can get me a bit more information about this, I believe we can<br>
figure it out:<br>
<br>
- could you get a backtrace of lldb-server when it is in the "stuck"<br>
state (just attach with lldb/gdb after it hangs and do "bt")? I want<br>
to see the where is it spinning, as I don't see any obvious infinite<br>
loop there.<br>
<br>
- are you able to still reproduce the bug with logging enabled? If so,<br>
I'd like to see the log file to understand this better. (You can<br>
enable logging by starting lldb-server with: --log-file XXX.log<br>
--log-channels "lldb all:linux all". If you're starting it via lldb<br>
client you can set the LLDB_DEBUGSERVER_LOG_FILE and<br>
LLDB_SERVER_LOG_CHANNELS environment vars to achieve this)<br>
<br>
- If you can get me reasonably detailed repro steps, I can try to<br>
investigate (I am fine with the first step being "install ubuntu 16.04<br>
in virtualbox")<br>
<br>
On 6 December 2016 at 23:41, Eugene Birukov via lldb-dev<br>
<<a href="mailto:lldb-dev@lists.llvm.org" target="_blank">lldb-dev@lists.llvm.org</a>> wrote:<br>
> Hi,<br>
> 1. I believe that lldb-server spins inside ptrace. I put breakpoint on the<br>
> highlighted line, and it does not hit. If I put breakpoint on line before,<br>
> it hits but lldb-server hangs.<br>
<br>
Do you mean actually inside the ptrace(2) syscall? Your description<br>
would certainly fit that, but that sounds scary, as it would mean a<br>
kernel bug. If that's the case, then we have to start looking in the<br>
kernel. I have some experience with that, but If we can boil this down<br>
to a simple use case, we can also ask the kernel ptrace folks for<br>
help.<br>
<br>
<br>
> 2. It seems that hang is caused by the client trying to read response too<br>
> fast. I mean, if I step through the client code it works - i.e. there is<br>
> significant delay between client writing into pipe and issuing ::select to<br>
> wait for response.<br>
<br>
I am not sure how this fits in with the item above. I find it hard to<br>
believe that the presence of select(2) in one process would affect the<br>
outcome of ptrace() in another. Unless we are actually encountering a<br>
kernel scheduler bug, which I find unlikely. Hopefully we can get more<br>
insight here with additional logging information.<br>
<br>
<br>
> Any advice how to deal with the situation except putting random sleeps in<br>
> random places?<br>
Inserting sleeps in various places is a valid (albeit very slow)<br>
strategy for debugging races. If you can push the sleep down, you will<br>
eventually reach the place where it will be obvious what is racing<br>
(or, at least, which component is to blame). Hopefully we can do<br>
something smarter though.<br>
<br>
If you are suspecting a kernel bug, I'd recommend recreating it in a<br>
simple standalone application (fork, trace the child, write its<br>
memory), as then it is easy to ask for help<br>
<br>
pl<br>
</div>
</span></font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</body>
</html>