[lldb-dev] Remote debugging ARM target from x86 host

Thu Aug 24 09:40:58 PDT 2017

> On Aug 24, 2017, at 8:33 AM, Ted Woodward <ted.woodward at codeaurora.org> wrote:
> 
> The lldb-server launch line looks ok; it should take the port 0 arg and get a port from the OS. 
> lldb is getting the port back from lldb-server in 4.0.1, as seen here:
> 
>> GDBRemoteCommunication::StartDebugserverProcess() debugserver listens 56543 port
> 
> But for 5.0.0 we see it fail the debugserver launch, and get:
> 
>> GDBRemoteCommunication::StartDebugserverProcess() debugserver listens 0 port
> 
> That log message comes right after the pipe read, which succeeds because error.Success() is true:
> 
>        // Read port from pipe with 10 second timeout.
>        error = socket_pipe.ReadWithTimeout(
>            port_cstr, num_bytes, std::chrono::seconds{10}, num_bytes);
>        if (error.Success() && (port != nullptr)) {
>          assert(num_bytes > 0 && port_cstr[num_bytes - 1] == '\0');
>          uint16_t child_port = StringConvert::ToUInt32(port_cstr, 0);
>          if (*port == 0 || *port == child_port) {
>            *port = child_port;
>            if (log)
>              log->Printf("GDBRemoteCommunication::%s() "
>                          "debugserver listens %u port",
>                          __FUNCTION__, *port);
> 
> As an aside, I don't like that assert - if we get garbage back from the pipe, we should error out, not take down lldb.

The assert should be removed and the code should work correctly without the assert.

> Another aside, the definition of port_cstr is odd:
>        char port_cstr[PATH_MAX] = {0};
>        port_cstr[0] = '\0';
> The size is way to big - max port number is 65535, so we don't need PATH_MAX bytes. And 2 assignments to make it a null string?
> 
> 
> Ramana, can you stick in a log message to print port_cstr? I suspect it's actually getting 0 back from lldb-server, which would tell us the error is in the server code, not the client code.
> 

With the following args:

argv[0]="/mnt/var/binaries/arm_release/bin/lldb-server"
argv[1]="gdbserver"
argv[2]="tcp://10.10.12.3:0"
argv[3]="--native-regs"
argv[4]="--pipe"
argv[5]="7"
argv[6]=NULL

lldb-server should bind to port 0, figure out the port, and write the port number into the file descriptor 7 ("--pipe 7" argument) to let the other side know what port to report back to the remote LLDB. The reply to the qLaunchGDBServer packet should then contain this valid port number. 

Ted's comments are correct and I am guessing we will find the "lldb-server gdb-server" is not doing the right thing and it isn't returning the correctly bound port. 

When we are doing remote stuff we must use TCP so there should be lldb-server should be opening a TCP socket, binding, listening and accepting a connection from the remote LLDB. 

Greg

> Ted
> 
> --
> Qualcomm Innovation Center, Inc.
> The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
> 
>> -----Original Message-----
>> From: Ramana [mailto:ramana.venkat83 at gmail.com]
>> Sent: Thursday, August 24, 2017 4:00 AM
>> To: Ted Woodward <ted.woodward at codeaurora.org>
>> Cc: Greg Clayton <clayborg at gmail.com>; Hans Wennborg
>> <hans at chromium.org>; lldb-dev at lists.llvm.org
>> Subject: Re: [lldb-dev] Remote debugging ARM target from x86 host
>> 
>> Greg, Ted
>> 
>> Thanks for your comments.
>> 
>> On Thu, Aug 24, 2017 at 12:01 AM, Ted Woodward
>> <ted.woodward at codeaurora.org> wrote:
>>> 
>>> What should be happening is Handle_qLaunchGDBServer calls
>> LaunchGDBServer with a port of UINT16_MAX, which tells it to use a port in its
>> port list. It shouldn't have a port list, so should return 0. LaunchGDBServer calls
>> StartDebugServerProcess with a port of 0 in the url. StartDebugServerProcess
>> launches the gdbserver with a named pipe, and reads the actual port from the
>> pipe.
>> 
>> Yes, the port list is empty unless --(min/max)-gdbserver-port is specified and
>> the gdbserver reads port number from the named pipe which in the failing case
>> is zero.
>> 
>>> I suggest turning on more logging - specifically, "gdb-remote process". That's
>> the channel used in StartDebugServerProcess.
>>> 
>> 
>> In fact I have given the relevant log line of "gdb-remote process" in the bug
>> details reporting port zero. Anyhow, the full log of "gdb-remote process" for
>> both lldb v4.0.1 and v5.0.0 is given at the bottom of my reply.
>> 
>> I thought https://reviews.llvm.org/rL295947 (Ensure lldb-server waits for child
>> debug servers to start up when passing them) got something to do with this bug
>> but both reversing
>> 
>>    if (pass_comm_fd == -1) {   at
>> source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp:1086
>> 
>> to original
>> 
>>   if (pass_comm_fd == -1 && ((port != nullptr && *port == 0) || port ==
>> nullptr)) {
>> 
>> and
>> 
>>    increasing the time out to 30 sec from 10 sec in
>> socket_pipe.ReadWithTimeout()    at
>> source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp:1255
>> 
>> did not solve the issue.
>> 
>> By the way, the remote debugging of x86 (linux) from x86 (linux) with lldb
>> v5.0.0 (but works with v4.0.1) also fails with assertion
>> 
>>    assert(num_bytes > 0 && port_cstr[num_bytes - 1] == '\0');  at
>> source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp:1258
>> 
>> due to the reason that socket_pipe.ReadWithTimeout() could not successfully
>> read the port number from the named pipe.
>> 
>> Based on the above, though I am not sure, the other patch I could think of
>> having an effect on this bug is
>> https://reviews.llvm.org/rL300579 (Update LLDB Host to support IPv6 over TCP)
>> which changed the socket implementation.
>> 
>> 
>> lldb-server log for "gdb-remote process" with lldb v4.0.1 (passing case)
>> 
>> GDBRemoteCommunication::StartDebugserverProcess(url=tcp://10.10.12.3:0,
>> port=0)
>> GDBRemoteCommunication::StartDebugserverProcess() found gdb-remote stub
>> exe '/mnt/var/binaries/arm_release/bin/lldb-server'
>> launch info for gdb-remote stub:
>> Executable: lldb-server
>> Triple: *-*-*
>> Arguments:
>> argv[0]="/mnt/var/binaries/arm_release/bin/lldb-server"
>> argv[1]="gdbserver"
>> argv[2]="tcp://10.10.12.3:0"
>> argv[3]="--native-regs"
>> argv[4]="--pipe"
>> argv[5]="7"
>> argv[6]=NULL
>> 
>> Environment:
>> env[0]="XDG_SESSION_ID=c3"
>> env[1]="TERM=xterm-256color"
>> env[2]="SHELL=/bin/sh"
>> env[3]="SSH_CLIENT=10.10.33.99 53542 22"
>> env[4]="SSH_TTY=/dev/pts/0"
>> env[5]="USER=root"
>> env[6]="MAIL=/var/mail/root"
>> env[7]="PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin"
>> env[8]="PWD=/home/root"
>> env[9]="EDITOR=vi"
>> env[10]="PS1=\u@\h:\w\$ "
>> env[11]="SHLVL=1"
>> env[12]="HOME=/home/root"
>> env[13]="LOGNAME=root"
>> env[14]="SSH_CONNECTION=10.10.33.99 53542 10.10.2.4 22"
>> env[15]="XDG_RUNTIME_DIR=/run/user/0"
>> env[16]="_=/mnt/var/binaries/arm_release/bin/lldb-server"
>> env[17]=NULL
>> 
>> GDBRemoteCommunication::StartDebugserverProcess() debugserver listens
>> 56543 port
>> 
>> 
>> lldb-server log for "gdb-remote process" with lldb v5.0.0 (failing case)
>> 
>> 
>> GDBRemoteCommunication::StartDebugserverProcess(url=tcp://10.10.12.3:0,
>> port=0)
>> GDBRemoteCommunication::StartDebugserverProcess() found gdb-remote stub
>> exe '/mnt/var/binaries/arm_v5.0_orig/bin/lldb-server'
>> launch info for gdb-remote stub:
>> Executable: lldb-server
>> Triple: *-*-*
>> Arguments:
>> argv[0]="/mnt/var/binaries/arm_v5.0_orig/bin/lldb-server"
>> argv[1]="gdbserver"
>> argv[2]="tcp://10.10.12.3:0"
>> argv[3]="--native-regs"
>> argv[4]="--pipe"
>> argv[5]="7"
>> argv[6]=NULL
>> 
>> Environment:
>> env[0]="XDG_SESSION_ID=c3"
>> env[1]="TERM=xterm-256color"
>> env[2]="SHELL=/bin/sh"
>> env[3]="SSH_CLIENT=10.10.33.99 53542 22"
>> env[4]="SSH_TTY=/dev/pts/0"
>> env[5]="USER=root"
>> env[6]="MAIL=/var/mail/root"
>> env[7]="PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin"
>> env[8]="PWD=/home/root"
>> env[9]="EDITOR=vi"
>> env[10]="PS1=\u@\h:\w\$ "
>> env[11]="SHLVL=1"
>> env[12]="HOME=/home/root"
>> env[13]="LOGNAME=root"
>> env[14]="SSH_CONNECTION=10.10.33.99 53542 10.10.2.4 22"
>> env[15]="XDG_RUNTIME_DIR=/run/user/0"
>> env[16]="_=/mnt/var/binaries/arm_v5.0_orig/bin/lldb-server"
>> env[17]=NULL
>> 
>> GDBRemoteCommunication::StartDebugserverProcess() debugserver listens 0
>> port
>> 
>> 
>> Thanks,
>> Ramana
>> 
>>> --
>>> Qualcomm Innovation Center, Inc.
>>> The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
>>> a Linux Foundation Collaborative Project
>>> 
>>>> -----Original Message-----
>>>> From: lldb-dev [mailto:lldb-dev-bounces at lists.llvm.org] On Behalf Of
>>>> Greg Clayton via lldb-dev
>>>> Sent: Wednesday, August 23, 2017 12:45 PM
>>>> To: Hans Wennborg <hans at chromium.org>
>>>> Cc: Ramana <ramana.venkat83 at gmail.com>; LLDB Dev <lldb-
>>>> dev at lists.llvm.org>
>>>> Subject: Re: [lldb-dev] Remote debugging ARM target from x86 host
>>>> 
>>>> Port zero should never be returned as a valid port. We do bind to
>>>> port zero just so we don't try and pick a port at random just to find
>>>> it is being used. When we bind to port 9, we must find the actual
>>>> port we bound to and return that. Seems something has gone wrong with
>>>> the code that discovers the port that was actually bound and is not reporting
>> that back correctly.
>>>> 
>>>> 
>>>> Should be straight forward to do by debugging the function
>>>> GDBRemoteCommunicationServerPlatform::Handle_qLaunchGDBServer(...)
>> in
>>>> GDBRemoteCommunicationServerPlatform.cpp and see what is going on
>> and
>>>> why it is returning 0 as the port.
>>>> 
>>>> Greg
>>>> 
>>>>> On Aug 23, 2017, at 9:44 AM, Hans Wennborg via lldb-dev <lldb-
>>>> dev at lists.llvm.org> wrote:
>>>>> 
>>>>> This was marked as an lldb 5.0.0 release blocker since it's a
>>>>> regression from 4.0.1: https://bugs.llvm.org/show_bug.cgi?id=34183
>>>>> 
>>>>> lldb-dev: Is there any interest in fixing this bug?
>>>>> 
>>>>> On Fri, Aug 4, 2017 at 10:13 PM, Ramana via lldb-dev
>>>>> <lldb-dev at lists.llvm.org> wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> I am trying to remote debug ARM (linux) target from x86 (linux)
>>>>>> host and I am getting the following error while trying to launch a process.
>>>>>> The local debugging on ARM works.
>>>>>> 
>>>>>> error: connect remote failed (invalid host:port specification:
>>>>>> '10.10.2.3')
>>>>>> error: process launch failed: invalid host:port specification: '10.10.2.3'
>>>>>> 
>>>>>> It appears the above error is because the gdb-remote is returning
>>>>>> the communication port as zero.
>>>>>> 
>>>>>> <  36> send packet: $qLaunchGDBServer;host:svrlin249;#bb
>>>>>> <  19> read packet: $pid:298;port:0;#bf
>>>>>> 
>>>>>> What are the possible reasons for the above behavior from
>>>>>> gdb-remote and how I could resolve this?
>>>>>> 
>>>>>> If it helps, below is the full log.
>>>>>> 
>>>>>> (lldb) log enable lldb comm
>>>>>> (lldb) log enable gdb-remote packets
>>>>>> (lldb) platform select remote-linux
>>>>>> Platform: remote-linux
>>>>>> Connected: no
>>>>>> (lldb) platform connect connect://10.10.2.3:500
>>>>>> 0x915bd78 Communication::Communication (name = gdb-remote.client)
>>>>>> 0x915bd78 Communication::Disconnect ()
>>>>>> 0x915bd78 Communication::Disconnect ()
>>>>>> 0x915bd78 Communication::Connect (url = connect://10.10.2.3:500)
>>>>>> Socket::TcpConnect (host/port = 10.10.2.3:500) TCPSocket::Connect
>>>>>> (host/port = 10.10.2.3:500)
>>>>>> 0x915bd78 Communication::Write (src = 0xbfcb7433, src_len = 1)
>>>>>> connection = 0x915f578
>>>>>> 0x915f608 Socket::Write() (socket = 7, src = 0xbfcb7433, src_len =
>>>>>> 1, flags = 0) => 1 (error = (null))
>>>>>> <   1> send packet: +
>>>>>> this = 0x0915BD78, dst = 0xBFCB53EC, dst_len = 8192, timeout =
>>>>>> 10000 us, connection = 0x0915F578
>>>>>> 0x915bd78 Communication::Write (src = 0x916022c, src_len = 19)
>>>>>> connection = 0x915f578
>>>>>> 0x915f608 Socket::Write() (socket = 7, src = 0x916022c, src_len =
>>>>>> 19, flags = 0) => 19 (error = (null))
>>>>>> history[1] tid=0x7cbf <   1> send packet: +
>>>>>> <  19> send packet: $QStartNoAckMode#b0 this = 0x0915BD78, dst =
>>>>>> 0xBFCB51AC, dst_len = 8192, timeout = 6000000 us, connection =
>>>>>> 0x0915F578
>>>>>> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb51ac, src_len =
>>>>>> 7, flags = 0) => 7 (error = (null))
>>>>>> <   1> read packet: +
>>>>>> <   6> read packet: $OK#9a
>>>>>> 0x915bd78 Communication::Write (src = 0xbfcb50f3, src_len = 1)
>>>>>> connection = 0x915f578
>>>>>> 0x915f608 Socket::Write() (socket = 7, src = 0xbfcb50f3, src_len =
>>>>>> 1, flags = 0) => 1 (error = (null))
>>>>>> <   1> send packet: +
>>>>>> 0x915bd78 Communication::Write (src = 0x9161ff4, src_len = 13)
>>>>>> connection = 0x915f578
>>>>>> 0x915f608 Socket::Write() (socket = 7, src = 0x9161ff4, src_len =
>>>>>> 13, flags = 0) => 13 (error = (null)) <  13> send packet:
>>>>>> $qHostInfo#9b this = 0x0915BD78, dst = 0xBFCB510C, dst_len = 8192,
>>>>>> timeout =
>>>>>> 1000000 us, connection = 0x0915F578
>>>>>> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb510c, src_len =
>>>>>> 316, flags = 0) => 316 (error = (null)) < 316> read packet:
>>>>>> 
>>>> 
>> $triple:61726d2d2d6c696e75782d676e75656162696866;ptrsize:4;watchpoint
>>>>>> _exceptions_received:before;endian:little;os_version:3.10.31;os_bu
>>>>>> ild
>>>>>> 
>>>> 
>> :332e31302e33312d6c7473692d30323836312d6738303161343066;os_kernel:2
>>>> 33
>>>>>> 
>>>> 
>> 520534d5020467269204d61792031332031353a35383a3232204953542032303
>>>> 136;h
>>>>>> ostname:736f
>>>>>> 63667067615f617272696135;#0a
>>>>>> 0x915bd78 Communication::Write (src = 0x915fe9c, src_len = 18)
>>>>>> connection = 0x915f578
>>>>>> 0x915f608 Socket::Write() (socket = 7, src = 0x915fe9c, src_len =
>>>>>> 18, flags = 0) => 18 (error = (null)) <  18> send packet:
>>>>>> $qGetWorkingDir#91 this = 0x0915BD78, dst = 0xBFCB50FC, dst_len =
>>>>>> 8192, timeout = 1000000 us, connection = 0x0915F578
>>>>>> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb50fc, src_len =
>>>>>> 24, flags = 0) => 24 (error = (null)) <  24> read packet:
>>>>>> $2f686f6d652f726f6f74#4b
>>>>>> 0x915bd78 Communication::Write (src = 0x915fe9c, src_len = 19)
>>>>>> connection = 0x915f578
>>>>>> 0x915f608 Socket::Write() (socket = 7, src = 0x915fe9c, src_len =
>>>>>> 19, flags = 0) => 19 (error = (null)) <  19> send packet:
>>>>>> $qQueryGDBServer#cb this = 0x0915BD78, dst = 0xBFCB531C, dst_len =
>>>>>> 8192, timeout = 1000000 us, connection = 0x0915F578
>>>>>> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb531c, src_len =
>>>>>> 7, flags = 0) => 7 (error = (null))
>>>>>> <   7> read packet: $E04#a9
>>>>>> Platform: remote-linux
>>>>>>   Triple: arm-*-linux-gnueabihf
>>>>>> OS Version: 3.10.31 (3.10.31-ltsi-02861-g801a40f)
>>>>>>   Kernel: #5 SMP Fri May 13 15:58:22 IST 2016
>>>>>> Hostname: socfpga_arria5
>>>>>> Connected: yes
>>>>>> WorkingDir: /home/root
>>>>>> (lldb) file main
>>>>>> 0x915bd78 Communication::Write (src = 0x91638fc, src_len = 137)
>>>>>> connection = 0x915f578
>>>>>> 0x915f608 Socket::Write() (socket = 7, src = 0x91638fc, src_len =
>>>>>> 137, flags = 0) => 137 (error = (null)) < 137> send packet:
>>>>>> 
>>>> 
>> $qModuleInfo:2f686f6d652f72616d616e616e2f776f726b5f726f6f742f546f545f
>>>>>> 
>>>> 
>> 6c6c64622f74657374732f6d61696e;61726d2d2d6c696e75782d656162696866#
>>>> f1
>>>>>> this = 0x0915BD78, dst = 0xBFCB172C, dst_len = 8192, timeout =
>>>>>> 1000000 us, connection = 0x0915F578
>>>>>> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb172c, src_len =
>>>>>> 7, flags = 0) => 7 (error = (null))
>>>>>> <   7> read packet: $E03#a8
>>>>>> Current executable set to 'main' (arm).
>>>>>> (lldb) b main
>>>>>> Breakpoint 1: where = main`main + 4 at main.c:4, address =
>>>>>> 0x000104a0
>>>>>> (lldb) run
>>>>>> 0x915bd78 Communication::Write (src = 0x917bae4, src_len = 36)
>>>>>> connection = 0x915f578
>>>>>> 0x915f608 Socket::Write() (socket = 7, src = 0x917bae4, src_len =
>>>>>> 36, flags = 0) => 36 (error = (null)) <  36> send packet:
>>>>>> $qLaunchGDBServer;host:svrlin249;#bb
>>>>>> this = 0x0915BD78, dst = 0xBFCB4FDC, dst_len = 8192, timeout =
>>>>>> 10000000 us, connection = 0x0915F578
>>>>>> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb4fdc, src_len =
>>>>>> 19, flags = 0) => 19 (error = (null)) <  19> read packet:
>>>>>> $pid:298;port:0;#bf
>>>>>> 0x92b0a84 Communication::Communication (name = process.stdio)
>>>>>> 0x92b0d78 Communication::Communication (name = gdb-remote.client)
>>>>>> 0x92b0a84 Communication::Disconnect () Socket::TcpConnect
>>>>>> (host/port = 10.10.2.3) TCPSocket::Connect (host/port = 10.10.2.3)
>>>>>> Socket::TcpConnect (host/port = 10.10.2.3) TCPSocket::Connect
>>>>>> (host/port = 10.10.2.3) Socket::TcpConnect (host/port = 10.10.2.3)
>>>>>> TCPSocket::Connect (host/port = 10.10.2.3) Socket::TcpConnect
>>>>>> (host/port = 10.10.2.3) TCPSocket::Connect (host/port = 10.10.2.3)
>>>>>> Socket::TcpConnect (host/port = 10.10.2.3) TCPSocket::Connect
>>>>>> (host/port = 10.10.2.3) Socket::TcpConnect (host/port = 10.10.2.3)
>>>>>> TCPSocket::Connect (host/port = 10.10.2.3) Socket::TcpConnect
>>>>>> (host/port = 10.10.2.3) TCPSocket::Connect (host/port = 10.10.2.3)
>>>>>> Socket::TcpConnect (host/port = 10.10.2.3) ..................
>>>>>> ..................
>>>>>> ..................
>>>>>> TCPSocket::Connect (host/port = 10.10.2.3) Socket::TcpConnect
>>>>>> (host/port = 10.10.2.3) TCPSocket::Connect (host/port = 10.10.2.3)
>>>>>> Socket::TcpConnect (host/port = 10.10.2.3) TCPSocket::Connect
>>>>>> (host/port = 10.10.2.3) Socket::TcpConnect (host/port = 10.10.2.3)
>>>>>> TCPSocket::Connect (host/port = 10.10.2.3) Socket::TcpConnect
>>>>>> (host/port = 10.10.2.3) TCPSocket::Connect (host/port = 10.10.2.3)
>>>>>> Socket::TcpConnect (host/port = 10.10.2.3) TCPSocket::Connect
>>>>>> (host/port = 10.10.2.3)
>>>>>> error: connect remote failed (invalid host:port specification:
>>>>>> '10.10.2.3')
>>>>>> 0x915bd78 Communication::Write (src = 0x92b38c4, src_len = 27)
>>>>>> connection = 0x915f578
>>>>>> 0x915f608 Socket::Write() (socket = 7, src = 0x92b38c4, src_len =
>>>>>> 27, flags = 0) => 27 (error = (null)) <  27> send packet:
>>>>>> $qKillSpawnedProcess:298#8b this = 0x0915BD78, dst = 0xBFCB509C,
>>>>>> dst_len = 8192, timeout = 1000000 us, connection = 0x0915F578
>>>>>> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb509c, src_len =
>>>>>> 7, flags = 0) => 7 (error = (null))
>>>>>> <   7> read packet: $E0a#d6
>>>>>> error: process launch failed: invalid host:port specification: '10.10.2.3'
>>>>>> (lldb)
>>>>>> 
>>>>>> 
>>>>>> Regards,
>>>>>> Ramana
>>>>>> _______________________________________________
>>>>>> lldb-dev mailing list
>>>>>> lldb-dev at lists.llvm.org
>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>> _______________________________________________
>>>>> lldb-dev mailing list
>>>>> lldb-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>> 
>>>> _______________________________________________
>>>> lldb-dev mailing list
>>>> lldb-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>> 
> 