[lldb-dev] Remote debugging ARM target from x86 host

Thu Aug 24 02:00:13 PDT 2017

Greg, Ted

Thanks for your comments.

On Thu, Aug 24, 2017 at 12:01 AM, Ted Woodward
<ted.woodward at codeaurora.org> wrote:
>
> What should be happening is Handle_qLaunchGDBServer calls LaunchGDBServer with a port of UINT16_MAX, which tells it to use a port in its port list. It shouldn't have a port list, so should return 0. LaunchGDBServer calls StartDebugServerProcess with a port of 0 in the url. StartDebugServerProcess launches the gdbserver with a named pipe, and reads the actual port from the pipe.

Yes, the port list is empty unless --(min/max)-gdbserver-port is
specified and the gdbserver reads port number from the named pipe
which in the failing case is zero.

> I suggest turning on more logging - specifically, "gdb-remote process". That's the channel used in StartDebugServerProcess.
>

In fact I have given the relevant log line of "gdb-remote process" in
the bug details reporting port zero. Anyhow, the full log of
"gdb-remote process" for both lldb v4.0.1 and v5.0.0 is given at the
bottom of my reply.

I thought https://reviews.llvm.org/rL295947 (Ensure lldb-server waits
for child debug servers to start up when passing them) got something
to do with this bug but both reversing

    if (pass_comm_fd == -1) {   at
source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp:1086

to original

   if (pass_comm_fd == -1 && ((port != nullptr && *port == 0) || port
== nullptr)) {

and

    increasing the time out to 30 sec from 10 sec in
socket_pipe.ReadWithTimeout()    at
source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp:1255

did not solve the issue.

By the way, the remote debugging of x86 (linux) from x86 (linux) with
lldb v5.0.0 (but works with v4.0.1) also fails with assertion

    assert(num_bytes > 0 && port_cstr[num_bytes - 1] == '\0');  at
source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp:1258

due to the reason that socket_pipe.ReadWithTimeout() could not
successfully read the port number from the named pipe.

Based on the above, though I am not sure, the other patch I could
think of having an effect on this bug is
https://reviews.llvm.org/rL300579 (Update LLDB Host to support IPv6
over TCP) which changed the socket implementation.

lldb-server log for "gdb-remote process" with lldb v4.0.1 (passing case)

GDBRemoteCommunication::StartDebugserverProcess(url=tcp://10.10.12.3:0, port=0)
GDBRemoteCommunication::StartDebugserverProcess() found gdb-remote
stub exe '/mnt/var/binaries/arm_release/bin/lldb-server'
launch info for gdb-remote stub:
Executable: lldb-server
Triple: *-*-*
Arguments:
argv[0]="/mnt/var/binaries/arm_release/bin/lldb-server"
argv[1]="gdbserver"
argv[2]="tcp://10.10.12.3:0"
argv[3]="--native-regs"
argv[4]="--pipe"
argv[5]="7"
argv[6]=NULL

Environment:
env[0]="XDG_SESSION_ID=c3"
env[1]="TERM=xterm-256color"
env[2]="SHELL=/bin/sh"
env[3]="SSH_CLIENT=10.10.33.99 53542 22"
env[4]="SSH_TTY=/dev/pts/0"
env[5]="USER=root"
env[6]="MAIL=/var/mail/root"
env[7]="PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin"
env[8]="PWD=/home/root"
env[9]="EDITOR=vi"
env[10]="PS1=\u@\h:\w\$ "
env[11]="SHLVL=1"
env[12]="HOME=/home/root"
env[13]="LOGNAME=root"
env[14]="SSH_CONNECTION=10.10.33.99 53542 10.10.2.4 22"
env[15]="XDG_RUNTIME_DIR=/run/user/0"
env[16]="_=/mnt/var/binaries/arm_release/bin/lldb-server"
env[17]=NULL

GDBRemoteCommunication::StartDebugserverProcess() debugserver listens 56543 port

lldb-server log for "gdb-remote process" with lldb v5.0.0 (failing case)

GDBRemoteCommunication::StartDebugserverProcess(url=tcp://10.10.12.3:0, port=0)
GDBRemoteCommunication::StartDebugserverProcess() found gdb-remote
stub exe '/mnt/var/binaries/arm_v5.0_orig/bin/lldb-server'
launch info for gdb-remote stub:
Executable: lldb-server
Triple: *-*-*
Arguments:
argv[0]="/mnt/var/binaries/arm_v5.0_orig/bin/lldb-server"
argv[1]="gdbserver"
argv[2]="tcp://10.10.12.3:0"
argv[3]="--native-regs"
argv[4]="--pipe"
argv[5]="7"
argv[6]=NULL

Environment:
env[0]="XDG_SESSION_ID=c3"
env[1]="TERM=xterm-256color"
env[2]="SHELL=/bin/sh"
env[3]="SSH_CLIENT=10.10.33.99 53542 22"
env[4]="SSH_TTY=/dev/pts/0"
env[5]="USER=root"
env[6]="MAIL=/var/mail/root"
env[7]="PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin"
env[8]="PWD=/home/root"
env[9]="EDITOR=vi"
env[10]="PS1=\u@\h:\w\$ "
env[11]="SHLVL=1"
env[12]="HOME=/home/root"
env[13]="LOGNAME=root"
env[14]="SSH_CONNECTION=10.10.33.99 53542 10.10.2.4 22"
env[15]="XDG_RUNTIME_DIR=/run/user/0"
env[16]="_=/mnt/var/binaries/arm_v5.0_orig/bin/lldb-server"
env[17]=NULL

GDBRemoteCommunication::StartDebugserverProcess() debugserver listens 0 port

Thanks,
Ramana

> --
> Qualcomm Innovation Center, Inc.
> The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
>
>> -----Original Message-----
>> From: lldb-dev [mailto:lldb-dev-bounces at lists.llvm.org] On Behalf Of Greg
>> Clayton via lldb-dev
>> Sent: Wednesday, August 23, 2017 12:45 PM
>> To: Hans Wennborg <hans at chromium.org>
>> Cc: Ramana <ramana.venkat83 at gmail.com>; LLDB Dev <lldb-
>> dev at lists.llvm.org>
>> Subject: Re: [lldb-dev] Remote debugging ARM target from x86 host
>>
>> Port zero should never be returned as a valid port. We do bind to port zero just
>> so we don't try and pick a port at random just to find it is being used. When we
>> bind to port 9, we must find the actual port we bound to and return that. Seems
>> something has gone wrong with the code that discovers the port that was
>> actually bound and is not reporting that back correctly.
>>
>>
>> Should be straight forward to do by debugging the function
>> GDBRemoteCommunicationServerPlatform::Handle_qLaunchGDBServer(...) in
>> GDBRemoteCommunicationServerPlatform.cpp and see what is going on and
>> why it is returning 0 as the port.
>>
>> Greg
>>
>> > On Aug 23, 2017, at 9:44 AM, Hans Wennborg via lldb-dev <lldb-
>> dev at lists.llvm.org> wrote:
>> >
>> > This was marked as an lldb 5.0.0 release blocker since it's a
>> > regression from 4.0.1: https://bugs.llvm.org/show_bug.cgi?id=34183
>> >
>> > lldb-dev: Is there any interest in fixing this bug?
>> >
>> > On Fri, Aug 4, 2017 at 10:13 PM, Ramana via lldb-dev
>> > <lldb-dev at lists.llvm.org> wrote:
>> >> Hi,
>> >>
>> >> I am trying to remote debug ARM (linux) target from x86 (linux) host
>> >> and I am getting the following error while trying to launch a process.
>> >> The local debugging on ARM works.
>> >>
>> >> error: connect remote failed (invalid host:port specification:
>> >> '10.10.2.3')
>> >> error: process launch failed: invalid host:port specification: '10.10.2.3'
>> >>
>> >> It appears the above error is because the gdb-remote is returning the
>> >> communication port as zero.
>> >>
>> >> <  36> send packet: $qLaunchGDBServer;host:svrlin249;#bb
>> >> <  19> read packet: $pid:298;port:0;#bf
>> >>
>> >> What are the possible reasons for the above behavior from gdb-remote
>> >> and how I could resolve this?
>> >>
>> >> If it helps, below is the full log.
>> >>
>> >> (lldb) log enable lldb comm
>> >> (lldb) log enable gdb-remote packets
>> >> (lldb) platform select remote-linux
>> >>  Platform: remote-linux
>> >> Connected: no
>> >> (lldb) platform connect connect://10.10.2.3:500
>> >> 0x915bd78 Communication::Communication (name = gdb-remote.client)
>> >> 0x915bd78 Communication::Disconnect ()
>> >> 0x915bd78 Communication::Disconnect ()
>> >> 0x915bd78 Communication::Connect (url = connect://10.10.2.3:500)
>> >> Socket::TcpConnect (host/port = 10.10.2.3:500) TCPSocket::Connect
>> >> (host/port = 10.10.2.3:500)
>> >> 0x915bd78 Communication::Write (src = 0xbfcb7433, src_len = 1)
>> >> connection = 0x915f578
>> >> 0x915f608 Socket::Write() (socket = 7, src = 0xbfcb7433, src_len = 1,
>> >> flags = 0) => 1 (error = (null))
>> >> <   1> send packet: +
>> >> this = 0x0915BD78, dst = 0xBFCB53EC, dst_len = 8192, timeout = 10000
>> >> us, connection = 0x0915F578
>> >> 0x915bd78 Communication::Write (src = 0x916022c, src_len = 19)
>> >> connection = 0x915f578
>> >> 0x915f608 Socket::Write() (socket = 7, src = 0x916022c, src_len = 19,
>> >> flags = 0) => 19 (error = (null))
>> >> history[1] tid=0x7cbf <   1> send packet: +
>> >> <  19> send packet: $QStartNoAckMode#b0 this = 0x0915BD78, dst =
>> >> 0xBFCB51AC, dst_len = 8192, timeout = 6000000 us, connection =
>> >> 0x0915F578
>> >> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb51ac, src_len = 7,
>> >> flags = 0) => 7 (error = (null))
>> >> <   1> read packet: +
>> >> <   6> read packet: $OK#9a
>> >> 0x915bd78 Communication::Write (src = 0xbfcb50f3, src_len = 1)
>> >> connection = 0x915f578
>> >> 0x915f608 Socket::Write() (socket = 7, src = 0xbfcb50f3, src_len = 1,
>> >> flags = 0) => 1 (error = (null))
>> >> <   1> send packet: +
>> >> 0x915bd78 Communication::Write (src = 0x9161ff4, src_len = 13)
>> >> connection = 0x915f578
>> >> 0x915f608 Socket::Write() (socket = 7, src = 0x9161ff4, src_len = 13,
>> >> flags = 0) => 13 (error = (null)) <  13> send packet: $qHostInfo#9b
>> >> this = 0x0915BD78, dst = 0xBFCB510C, dst_len = 8192, timeout =
>> >> 1000000 us, connection = 0x0915F578
>> >> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb510c, src_len =
>> >> 316, flags = 0) => 316 (error = (null)) < 316> read packet:
>> >>
>> $triple:61726d2d2d6c696e75782d676e75656162696866;ptrsize:4;watchpoint
>> >> _exceptions_received:before;endian:little;os_version:3.10.31;os_build
>> >>
>> :332e31302e33312d6c7473692d30323836312d6738303161343066;os_kernel:2
>> 33
>> >>
>> 520534d5020467269204d61792031332031353a35383a3232204953542032303
>> 136;h
>> >> ostname:736f
>> >> 63667067615f617272696135;#0a
>> >> 0x915bd78 Communication::Write (src = 0x915fe9c, src_len = 18)
>> >> connection = 0x915f578
>> >> 0x915f608 Socket::Write() (socket = 7, src = 0x915fe9c, src_len = 18,
>> >> flags = 0) => 18 (error = (null)) <  18> send packet:
>> >> $qGetWorkingDir#91 this = 0x0915BD78, dst = 0xBFCB50FC, dst_len =
>> >> 8192, timeout = 1000000 us, connection = 0x0915F578
>> >> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb50fc, src_len = 24,
>> >> flags = 0) => 24 (error = (null)) <  24> read packet:
>> >> $2f686f6d652f726f6f74#4b
>> >> 0x915bd78 Communication::Write (src = 0x915fe9c, src_len = 19)
>> >> connection = 0x915f578
>> >> 0x915f608 Socket::Write() (socket = 7, src = 0x915fe9c, src_len = 19,
>> >> flags = 0) => 19 (error = (null)) <  19> send packet:
>> >> $qQueryGDBServer#cb this = 0x0915BD78, dst = 0xBFCB531C, dst_len =
>> >> 8192, timeout = 1000000 us, connection = 0x0915F578
>> >> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb531c, src_len = 7,
>> >> flags = 0) => 7 (error = (null))
>> >> <   7> read packet: $E04#a9
>> >>  Platform: remote-linux
>> >>    Triple: arm-*-linux-gnueabihf
>> >> OS Version: 3.10.31 (3.10.31-ltsi-02861-g801a40f)
>> >>    Kernel: #5 SMP Fri May 13 15:58:22 IST 2016
>> >>  Hostname: socfpga_arria5
>> >> Connected: yes
>> >> WorkingDir: /home/root
>> >> (lldb) file main
>> >> 0x915bd78 Communication::Write (src = 0x91638fc, src_len = 137)
>> >> connection = 0x915f578
>> >> 0x915f608 Socket::Write() (socket = 7, src = 0x91638fc, src_len =
>> >> 137, flags = 0) => 137 (error = (null)) < 137> send packet:
>> >>
>> $qModuleInfo:2f686f6d652f72616d616e616e2f776f726b5f726f6f742f546f545f
>> >>
>> 6c6c64622f74657374732f6d61696e;61726d2d2d6c696e75782d656162696866#
>> f1
>> >> this = 0x0915BD78, dst = 0xBFCB172C, dst_len = 8192, timeout =
>> >> 1000000 us, connection = 0x0915F578
>> >> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb172c, src_len = 7,
>> >> flags = 0) => 7 (error = (null))
>> >> <   7> read packet: $E03#a8
>> >> Current executable set to 'main' (arm).
>> >> (lldb) b main
>> >> Breakpoint 1: where = main`main + 4 at main.c:4, address = 0x000104a0
>> >> (lldb) run
>> >> 0x915bd78 Communication::Write (src = 0x917bae4, src_len = 36)
>> >> connection = 0x915f578
>> >> 0x915f608 Socket::Write() (socket = 7, src = 0x917bae4, src_len = 36,
>> >> flags = 0) => 36 (error = (null)) <  36> send packet:
>> >> $qLaunchGDBServer;host:svrlin249;#bb
>> >> this = 0x0915BD78, dst = 0xBFCB4FDC, dst_len = 8192, timeout =
>> >> 10000000 us, connection = 0x0915F578
>> >> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb4fdc, src_len = 19,
>> >> flags = 0) => 19 (error = (null)) <  19> read packet:
>> >> $pid:298;port:0;#bf
>> >> 0x92b0a84 Communication::Communication (name = process.stdio)
>> >> 0x92b0d78 Communication::Communication (name = gdb-remote.client)
>> >> 0x92b0a84 Communication::Disconnect () Socket::TcpConnect (host/port
>> >> = 10.10.2.3) TCPSocket::Connect (host/port = 10.10.2.3)
>> >> Socket::TcpConnect (host/port = 10.10.2.3) TCPSocket::Connect
>> >> (host/port = 10.10.2.3) Socket::TcpConnect (host/port = 10.10.2.3)
>> >> TCPSocket::Connect (host/port = 10.10.2.3) Socket::TcpConnect
>> >> (host/port = 10.10.2.3) TCPSocket::Connect (host/port = 10.10.2.3)
>> >> Socket::TcpConnect (host/port = 10.10.2.3) TCPSocket::Connect
>> >> (host/port = 10.10.2.3) Socket::TcpConnect (host/port = 10.10.2.3)
>> >> TCPSocket::Connect (host/port = 10.10.2.3) Socket::TcpConnect
>> >> (host/port = 10.10.2.3) TCPSocket::Connect (host/port = 10.10.2.3)
>> >> Socket::TcpConnect (host/port = 10.10.2.3) ..................
>> >> ..................
>> >> ..................
>> >> TCPSocket::Connect (host/port = 10.10.2.3) Socket::TcpConnect
>> >> (host/port = 10.10.2.3) TCPSocket::Connect (host/port = 10.10.2.3)
>> >> Socket::TcpConnect (host/port = 10.10.2.3) TCPSocket::Connect
>> >> (host/port = 10.10.2.3) Socket::TcpConnect (host/port = 10.10.2.3)
>> >> TCPSocket::Connect (host/port = 10.10.2.3) Socket::TcpConnect
>> >> (host/port = 10.10.2.3) TCPSocket::Connect (host/port = 10.10.2.3)
>> >> Socket::TcpConnect (host/port = 10.10.2.3) TCPSocket::Connect
>> >> (host/port = 10.10.2.3)
>> >> error: connect remote failed (invalid host:port specification:
>> >> '10.10.2.3')
>> >> 0x915bd78 Communication::Write (src = 0x92b38c4, src_len = 27)
>> >> connection = 0x915f578
>> >> 0x915f608 Socket::Write() (socket = 7, src = 0x92b38c4, src_len = 27,
>> >> flags = 0) => 27 (error = (null)) <  27> send packet:
>> >> $qKillSpawnedProcess:298#8b this = 0x0915BD78, dst = 0xBFCB509C,
>> >> dst_len = 8192, timeout = 1000000 us, connection = 0x0915F578
>> >> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb509c, src_len = 7,
>> >> flags = 0) => 7 (error = (null))
>> >> <   7> read packet: $E0a#d6
>> >> error: process launch failed: invalid host:port specification: '10.10.2.3'
>> >> (lldb)
>> >>
>> >>
>> >> Regards,
>> >> Ramana
>> >> _______________________________________________
>> >> lldb-dev mailing list
>> >> lldb-dev at lists.llvm.org
>> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>> > _______________________________________________
>> > lldb-dev mailing list
>> > lldb-dev at lists.llvm.org
>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>