[lldb-dev] Unreliable process attach on Linux

[Pavel Labath via lldb-dev]

On 07/01/2019 09:29, Florian Weimer wrote:
> * Pavel Labath:
> 
>> On 04/01/2019 17:38, Florian Weimer via lldb-dev wrote:
>>> Consider this example program:
>>>
>>> #include <err.h>
>>> #include <sys/wait.h>
>>> #include <unistd.h>
>>>
>>> #include <lldb/API/SBDebugger.h>
>>> #include <lldb/API/SBProcess.h>
>>> #include <lldb/API/SBTarget.h>
>>>
>>> int
>>> main(void)
>>> {
>>>     // Target process for the debugger.
>>>     pid_t pid = fork();
>>>     if (pid < 0)
>>>       err(1, "fork");
>>>     if (pid == 0)
>>>       while (true)
>>>         pause();
>>>
>>>     lldb::SBDebugger::Initialize();
>>>     {
>>>       auto debugger(lldb::SBDebugger::Create());
>>>       if (!debugger.IsValid())
>>>         errx(1, "SBDebugger::Create failed");
>>>
>>>       auto target(debugger.CreateTarget(nullptr));
>>>       if (!target.IsValid())
>>>         errx(1, "SBDebugger::CreateTarget failed");
>>>
>>>       lldb::SBAttachInfo attachinfo(pid);
>>>       lldb::SBError error;
>>>       auto process(target.Attach(attachinfo, error));
>>>       if (!process.IsValid())
>>>         errx(1, "SBTarget::Attach failed: %s", error.GetCString());
>>>       error = process.Detach();
>>>       if (error.Fail())
>>>         errx(1, "SBProcess::Detach failed: %s", error.GetCString());
>>>     }
>>>     lldb::SBDebugger::Terminate();
>>>
>>>     if (kill(pid, SIGKILL) != 0)
>>>       err(1, "kill");
>>>     if (waitpid(pid, NULL, 0) < 0)
>>>       err(1, "waitpid");
>>>
>>>     return 0;
>>> }
>>>
>>> Run it in a loop like this:
>>>
>>> $ while ./test-attach ; do date; done
>>>
>>> On Linux x86-64 (Fedora 29), with LLDB 7 (lldb-7.0.0-1.fc29.x86_64) and
>>> kernel 4.19.12 (kernel-4.19.12-301.fc29.x86_64), after 100 iterations or
>>> so, attaching to the newly created process fails:
>>>
>>> test-attach: SBTarget::Attach failed: lost connection
>>>
>>> This also reproduces occasionally with LLDB itself (with “lldb -p PID”).
>>>
>>> Any suggestions how to get more information about the cause of this
>>> error?
>>>
>>
>> I would recommend enabling gdb-remote logging (so something like:
>> debugger.HandleCommand("log enable gdb-remote packets")) to see at
>> which stage do we actually lose the gdb-server connection.
> 
> Thanks.  I enabled logging like this:
> 
>      auto debugger(lldb::SBDebugger::Create());
>      if (!debugger.IsValid())
>        errx(1, "SBDebugger::Create failed");
> 
>      debugger.HandleCommand("log enable gdb-remote packets");
> 
>      auto target(debugger.CreateTarget(nullptr));
>      if (!target.IsValid())
>        errx(1, "SBDebugger::CreateTarget failed");
> 
> And here's the output I get:
> 
> test-attach      <   1> send packet: +
> test-attach      history[1] tid=0x1cab <   1> send packet: +
> test-attach      <  19> send packet: $QStartNoAckMode#b0
> test-attach      <   1> read packet: +
> test-attach      <   6> read packet: $OK#9a
> test-attach      <   1> send packet: +
> test-attach      <  41> send packet: $qSupported:xmlRegisters=i386,arm,mips#12
> test-attach      < 124> read packet: $PacketSize=20000;QStartNoAckMode+;QThreadSuffixSupported+;QListThreadsInStopReply+;qEcho+;QPassSignals+;qXfer:auxv:read+#be
> test-attach      <  26> send packet: $QThreadSuffixSupported#e4
> test-attach      <   6> read packet: $OK#9a
> test-attach      <  27> send packet: $QListThreadsInStopReply#21
> test-attach      <   6> read packet: $OK#9a
> test-attach      <  13> send packet: $qHostInfo#9b
> test-attach      <  11> send packet: $qEcho:1#5b
> test-attach: SBTarget::Attach failed: lost connection
> 
> Florian
> 

Thanks. I think this is what I suspected. The server is extremely slow 
in responding to the qHostInfo packet. This timeout for this was 
recently increased to 10 seconds, but it looks like 7.0 still has the 
default (1 second) timeout.

If you don't want to recompile or update, you should be able to work 
around this by increasing the default timeout with the following command 
"settings set plugin.process.gdb-remote.packet-timeout 10".

regards,
pavel