[lldb-dev] Remote debugging ARM target from x86 host
Ted Woodward via lldb-dev
lldb-dev at lists.llvm.org
Thu Aug 24 08:33:11 PDT 2017
The lldb-server launch line looks ok; it should take the port 0 arg and get a port from the OS.
lldb is getting the port back from lldb-server in 4.0.1, as seen here:
> GDBRemoteCommunication::StartDebugserverProcess() debugserver listens 56543 port
But for 5.0.0 we see it fail the debugserver launch, and get:
> GDBRemoteCommunication::StartDebugserverProcess() debugserver listens 0 port
That log message comes right after the pipe read, which succeeds because error.Success() is true:
// Read port from pipe with 10 second timeout.
error = socket_pipe.ReadWithTimeout(
port_cstr, num_bytes, std::chrono::seconds{10}, num_bytes);
if (error.Success() && (port != nullptr)) {
assert(num_bytes > 0 && port_cstr[num_bytes - 1] == '\0');
uint16_t child_port = StringConvert::ToUInt32(port_cstr, 0);
if (*port == 0 || *port == child_port) {
*port = child_port;
if (log)
log->Printf("GDBRemoteCommunication::%s() "
"debugserver listens %u port",
__FUNCTION__, *port);
As an aside, I don't like that assert - if we get garbage back from the pipe, we should error out, not take down lldb.
Another aside, the definition of port_cstr is odd:
char port_cstr[PATH_MAX] = {0};
port_cstr[0] = '\0';
The size is way to big - max port number is 65535, so we don't need PATH_MAX bytes. And 2 assignments to make it a null string?
Ramana, can you stick in a log message to print port_cstr? I suspect it's actually getting 0 back from lldb-server, which would tell us the error is in the server code, not the client code.
Ted
--
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
> -----Original Message-----
> From: Ramana [mailto:ramana.venkat83 at gmail.com]
> Sent: Thursday, August 24, 2017 4:00 AM
> To: Ted Woodward <ted.woodward at codeaurora.org>
> Cc: Greg Clayton <clayborg at gmail.com>; Hans Wennborg
> <hans at chromium.org>; lldb-dev at lists.llvm.org
> Subject: Re: [lldb-dev] Remote debugging ARM target from x86 host
>
> Greg, Ted
>
> Thanks for your comments.
>
> On Thu, Aug 24, 2017 at 12:01 AM, Ted Woodward
> <ted.woodward at codeaurora.org> wrote:
> >
> > What should be happening is Handle_qLaunchGDBServer calls
> LaunchGDBServer with a port of UINT16_MAX, which tells it to use a port in its
> port list. It shouldn't have a port list, so should return 0. LaunchGDBServer calls
> StartDebugServerProcess with a port of 0 in the url. StartDebugServerProcess
> launches the gdbserver with a named pipe, and reads the actual port from the
> pipe.
>
> Yes, the port list is empty unless --(min/max)-gdbserver-port is specified and
> the gdbserver reads port number from the named pipe which in the failing case
> is zero.
>
> > I suggest turning on more logging - specifically, "gdb-remote process". That's
> the channel used in StartDebugServerProcess.
> >
>
> In fact I have given the relevant log line of "gdb-remote process" in the bug
> details reporting port zero. Anyhow, the full log of "gdb-remote process" for
> both lldb v4.0.1 and v5.0.0 is given at the bottom of my reply.
>
> I thought https://reviews.llvm.org/rL295947 (Ensure lldb-server waits for child
> debug servers to start up when passing them) got something to do with this bug
> but both reversing
>
> if (pass_comm_fd == -1) { at
> source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp:1086
>
> to original
>
> if (pass_comm_fd == -1 && ((port != nullptr && *port == 0) || port ==
> nullptr)) {
>
> and
>
> increasing the time out to 30 sec from 10 sec in
> socket_pipe.ReadWithTimeout() at
> source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp:1255
>
> did not solve the issue.
>
> By the way, the remote debugging of x86 (linux) from x86 (linux) with lldb
> v5.0.0 (but works with v4.0.1) also fails with assertion
>
> assert(num_bytes > 0 && port_cstr[num_bytes - 1] == '\0'); at
> source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp:1258
>
> due to the reason that socket_pipe.ReadWithTimeout() could not successfully
> read the port number from the named pipe.
>
> Based on the above, though I am not sure, the other patch I could think of
> having an effect on this bug is
> https://reviews.llvm.org/rL300579 (Update LLDB Host to support IPv6 over TCP)
> which changed the socket implementation.
>
>
> lldb-server log for "gdb-remote process" with lldb v4.0.1 (passing case)
>
> GDBRemoteCommunication::StartDebugserverProcess(url=tcp://10.10.12.3:0,
> port=0)
> GDBRemoteCommunication::StartDebugserverProcess() found gdb-remote stub
> exe '/mnt/var/binaries/arm_release/bin/lldb-server'
> launch info for gdb-remote stub:
> Executable: lldb-server
> Triple: *-*-*
> Arguments:
> argv[0]="/mnt/var/binaries/arm_release/bin/lldb-server"
> argv[1]="gdbserver"
> argv[2]="tcp://10.10.12.3:0"
> argv[3]="--native-regs"
> argv[4]="--pipe"
> argv[5]="7"
> argv[6]=NULL
>
> Environment:
> env[0]="XDG_SESSION_ID=c3"
> env[1]="TERM=xterm-256color"
> env[2]="SHELL=/bin/sh"
> env[3]="SSH_CLIENT=10.10.33.99 53542 22"
> env[4]="SSH_TTY=/dev/pts/0"
> env[5]="USER=root"
> env[6]="MAIL=/var/mail/root"
> env[7]="PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin"
> env[8]="PWD=/home/root"
> env[9]="EDITOR=vi"
> env[10]="PS1=\u@\h:\w\$ "
> env[11]="SHLVL=1"
> env[12]="HOME=/home/root"
> env[13]="LOGNAME=root"
> env[14]="SSH_CONNECTION=10.10.33.99 53542 10.10.2.4 22"
> env[15]="XDG_RUNTIME_DIR=/run/user/0"
> env[16]="_=/mnt/var/binaries/arm_release/bin/lldb-server"
> env[17]=NULL
>
> GDBRemoteCommunication::StartDebugserverProcess() debugserver listens
> 56543 port
>
>
> lldb-server log for "gdb-remote process" with lldb v5.0.0 (failing case)
>
>
> GDBRemoteCommunication::StartDebugserverProcess(url=tcp://10.10.12.3:0,
> port=0)
> GDBRemoteCommunication::StartDebugserverProcess() found gdb-remote stub
> exe '/mnt/var/binaries/arm_v5.0_orig/bin/lldb-server'
> launch info for gdb-remote stub:
> Executable: lldb-server
> Triple: *-*-*
> Arguments:
> argv[0]="/mnt/var/binaries/arm_v5.0_orig/bin/lldb-server"
> argv[1]="gdbserver"
> argv[2]="tcp://10.10.12.3:0"
> argv[3]="--native-regs"
> argv[4]="--pipe"
> argv[5]="7"
> argv[6]=NULL
>
> Environment:
> env[0]="XDG_SESSION_ID=c3"
> env[1]="TERM=xterm-256color"
> env[2]="SHELL=/bin/sh"
> env[3]="SSH_CLIENT=10.10.33.99 53542 22"
> env[4]="SSH_TTY=/dev/pts/0"
> env[5]="USER=root"
> env[6]="MAIL=/var/mail/root"
> env[7]="PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin"
> env[8]="PWD=/home/root"
> env[9]="EDITOR=vi"
> env[10]="PS1=\u@\h:\w\$ "
> env[11]="SHLVL=1"
> env[12]="HOME=/home/root"
> env[13]="LOGNAME=root"
> env[14]="SSH_CONNECTION=10.10.33.99 53542 10.10.2.4 22"
> env[15]="XDG_RUNTIME_DIR=/run/user/0"
> env[16]="_=/mnt/var/binaries/arm_v5.0_orig/bin/lldb-server"
> env[17]=NULL
>
> GDBRemoteCommunication::StartDebugserverProcess() debugserver listens 0
> port
>
>
> Thanks,
> Ramana
>
> > --
> > Qualcomm Innovation Center, Inc.
> > The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
> > a Linux Foundation Collaborative Project
> >
> >> -----Original Message-----
> >> From: lldb-dev [mailto:lldb-dev-bounces at lists.llvm.org] On Behalf Of
> >> Greg Clayton via lldb-dev
> >> Sent: Wednesday, August 23, 2017 12:45 PM
> >> To: Hans Wennborg <hans at chromium.org>
> >> Cc: Ramana <ramana.venkat83 at gmail.com>; LLDB Dev <lldb-
> >> dev at lists.llvm.org>
> >> Subject: Re: [lldb-dev] Remote debugging ARM target from x86 host
> >>
> >> Port zero should never be returned as a valid port. We do bind to
> >> port zero just so we don't try and pick a port at random just to find
> >> it is being used. When we bind to port 9, we must find the actual
> >> port we bound to and return that. Seems something has gone wrong with
> >> the code that discovers the port that was actually bound and is not reporting
> that back correctly.
> >>
> >>
> >> Should be straight forward to do by debugging the function
> >> GDBRemoteCommunicationServerPlatform::Handle_qLaunchGDBServer(...)
> in
> >> GDBRemoteCommunicationServerPlatform.cpp and see what is going on
> and
> >> why it is returning 0 as the port.
> >>
> >> Greg
> >>
> >> > On Aug 23, 2017, at 9:44 AM, Hans Wennborg via lldb-dev <lldb-
> >> dev at lists.llvm.org> wrote:
> >> >
> >> > This was marked as an lldb 5.0.0 release blocker since it's a
> >> > regression from 4.0.1: https://bugs.llvm.org/show_bug.cgi?id=34183
> >> >
> >> > lldb-dev: Is there any interest in fixing this bug?
> >> >
> >> > On Fri, Aug 4, 2017 at 10:13 PM, Ramana via lldb-dev
> >> > <lldb-dev at lists.llvm.org> wrote:
> >> >> Hi,
> >> >>
> >> >> I am trying to remote debug ARM (linux) target from x86 (linux)
> >> >> host and I am getting the following error while trying to launch a process.
> >> >> The local debugging on ARM works.
> >> >>
> >> >> error: connect remote failed (invalid host:port specification:
> >> >> '10.10.2.3')
> >> >> error: process launch failed: invalid host:port specification: '10.10.2.3'
> >> >>
> >> >> It appears the above error is because the gdb-remote is returning
> >> >> the communication port as zero.
> >> >>
> >> >> < 36> send packet: $qLaunchGDBServer;host:svrlin249;#bb
> >> >> < 19> read packet: $pid:298;port:0;#bf
> >> >>
> >> >> What are the possible reasons for the above behavior from
> >> >> gdb-remote and how I could resolve this?
> >> >>
> >> >> If it helps, below is the full log.
> >> >>
> >> >> (lldb) log enable lldb comm
> >> >> (lldb) log enable gdb-remote packets
> >> >> (lldb) platform select remote-linux
> >> >> Platform: remote-linux
> >> >> Connected: no
> >> >> (lldb) platform connect connect://10.10.2.3:500
> >> >> 0x915bd78 Communication::Communication (name = gdb-remote.client)
> >> >> 0x915bd78 Communication::Disconnect ()
> >> >> 0x915bd78 Communication::Disconnect ()
> >> >> 0x915bd78 Communication::Connect (url = connect://10.10.2.3:500)
> >> >> Socket::TcpConnect (host/port = 10.10.2.3:500) TCPSocket::Connect
> >> >> (host/port = 10.10.2.3:500)
> >> >> 0x915bd78 Communication::Write (src = 0xbfcb7433, src_len = 1)
> >> >> connection = 0x915f578
> >> >> 0x915f608 Socket::Write() (socket = 7, src = 0xbfcb7433, src_len =
> >> >> 1, flags = 0) => 1 (error = (null))
> >> >> < 1> send packet: +
> >> >> this = 0x0915BD78, dst = 0xBFCB53EC, dst_len = 8192, timeout =
> >> >> 10000 us, connection = 0x0915F578
> >> >> 0x915bd78 Communication::Write (src = 0x916022c, src_len = 19)
> >> >> connection = 0x915f578
> >> >> 0x915f608 Socket::Write() (socket = 7, src = 0x916022c, src_len =
> >> >> 19, flags = 0) => 19 (error = (null))
> >> >> history[1] tid=0x7cbf < 1> send packet: +
> >> >> < 19> send packet: $QStartNoAckMode#b0 this = 0x0915BD78, dst =
> >> >> 0xBFCB51AC, dst_len = 8192, timeout = 6000000 us, connection =
> >> >> 0x0915F578
> >> >> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb51ac, src_len =
> >> >> 7, flags = 0) => 7 (error = (null))
> >> >> < 1> read packet: +
> >> >> < 6> read packet: $OK#9a
> >> >> 0x915bd78 Communication::Write (src = 0xbfcb50f3, src_len = 1)
> >> >> connection = 0x915f578
> >> >> 0x915f608 Socket::Write() (socket = 7, src = 0xbfcb50f3, src_len =
> >> >> 1, flags = 0) => 1 (error = (null))
> >> >> < 1> send packet: +
> >> >> 0x915bd78 Communication::Write (src = 0x9161ff4, src_len = 13)
> >> >> connection = 0x915f578
> >> >> 0x915f608 Socket::Write() (socket = 7, src = 0x9161ff4, src_len =
> >> >> 13, flags = 0) => 13 (error = (null)) < 13> send packet:
> >> >> $qHostInfo#9b this = 0x0915BD78, dst = 0xBFCB510C, dst_len = 8192,
> >> >> timeout =
> >> >> 1000000 us, connection = 0x0915F578
> >> >> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb510c, src_len =
> >> >> 316, flags = 0) => 316 (error = (null)) < 316> read packet:
> >> >>
> >>
> $triple:61726d2d2d6c696e75782d676e75656162696866;ptrsize:4;watchpoint
> >> >> _exceptions_received:before;endian:little;os_version:3.10.31;os_bu
> >> >> ild
> >> >>
> >>
> :332e31302e33312d6c7473692d30323836312d6738303161343066;os_kernel:2
> >> 33
> >> >>
> >>
> 520534d5020467269204d61792031332031353a35383a3232204953542032303
> >> 136;h
> >> >> ostname:736f
> >> >> 63667067615f617272696135;#0a
> >> >> 0x915bd78 Communication::Write (src = 0x915fe9c, src_len = 18)
> >> >> connection = 0x915f578
> >> >> 0x915f608 Socket::Write() (socket = 7, src = 0x915fe9c, src_len =
> >> >> 18, flags = 0) => 18 (error = (null)) < 18> send packet:
> >> >> $qGetWorkingDir#91 this = 0x0915BD78, dst = 0xBFCB50FC, dst_len =
> >> >> 8192, timeout = 1000000 us, connection = 0x0915F578
> >> >> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb50fc, src_len =
> >> >> 24, flags = 0) => 24 (error = (null)) < 24> read packet:
> >> >> $2f686f6d652f726f6f74#4b
> >> >> 0x915bd78 Communication::Write (src = 0x915fe9c, src_len = 19)
> >> >> connection = 0x915f578
> >> >> 0x915f608 Socket::Write() (socket = 7, src = 0x915fe9c, src_len =
> >> >> 19, flags = 0) => 19 (error = (null)) < 19> send packet:
> >> >> $qQueryGDBServer#cb this = 0x0915BD78, dst = 0xBFCB531C, dst_len =
> >> >> 8192, timeout = 1000000 us, connection = 0x0915F578
> >> >> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb531c, src_len =
> >> >> 7, flags = 0) => 7 (error = (null))
> >> >> < 7> read packet: $E04#a9
> >> >> Platform: remote-linux
> >> >> Triple: arm-*-linux-gnueabihf
> >> >> OS Version: 3.10.31 (3.10.31-ltsi-02861-g801a40f)
> >> >> Kernel: #5 SMP Fri May 13 15:58:22 IST 2016
> >> >> Hostname: socfpga_arria5
> >> >> Connected: yes
> >> >> WorkingDir: /home/root
> >> >> (lldb) file main
> >> >> 0x915bd78 Communication::Write (src = 0x91638fc, src_len = 137)
> >> >> connection = 0x915f578
> >> >> 0x915f608 Socket::Write() (socket = 7, src = 0x91638fc, src_len =
> >> >> 137, flags = 0) => 137 (error = (null)) < 137> send packet:
> >> >>
> >>
> $qModuleInfo:2f686f6d652f72616d616e616e2f776f726b5f726f6f742f546f545f
> >> >>
> >>
> 6c6c64622f74657374732f6d61696e;61726d2d2d6c696e75782d656162696866#
> >> f1
> >> >> this = 0x0915BD78, dst = 0xBFCB172C, dst_len = 8192, timeout =
> >> >> 1000000 us, connection = 0x0915F578
> >> >> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb172c, src_len =
> >> >> 7, flags = 0) => 7 (error = (null))
> >> >> < 7> read packet: $E03#a8
> >> >> Current executable set to 'main' (arm).
> >> >> (lldb) b main
> >> >> Breakpoint 1: where = main`main + 4 at main.c:4, address =
> >> >> 0x000104a0
> >> >> (lldb) run
> >> >> 0x915bd78 Communication::Write (src = 0x917bae4, src_len = 36)
> >> >> connection = 0x915f578
> >> >> 0x915f608 Socket::Write() (socket = 7, src = 0x917bae4, src_len =
> >> >> 36, flags = 0) => 36 (error = (null)) < 36> send packet:
> >> >> $qLaunchGDBServer;host:svrlin249;#bb
> >> >> this = 0x0915BD78, dst = 0xBFCB4FDC, dst_len = 8192, timeout =
> >> >> 10000000 us, connection = 0x0915F578
> >> >> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb4fdc, src_len =
> >> >> 19, flags = 0) => 19 (error = (null)) < 19> read packet:
> >> >> $pid:298;port:0;#bf
> >> >> 0x92b0a84 Communication::Communication (name = process.stdio)
> >> >> 0x92b0d78 Communication::Communication (name = gdb-remote.client)
> >> >> 0x92b0a84 Communication::Disconnect () Socket::TcpConnect
> >> >> (host/port = 10.10.2.3) TCPSocket::Connect (host/port = 10.10.2.3)
> >> >> Socket::TcpConnect (host/port = 10.10.2.3) TCPSocket::Connect
> >> >> (host/port = 10.10.2.3) Socket::TcpConnect (host/port = 10.10.2.3)
> >> >> TCPSocket::Connect (host/port = 10.10.2.3) Socket::TcpConnect
> >> >> (host/port = 10.10.2.3) TCPSocket::Connect (host/port = 10.10.2.3)
> >> >> Socket::TcpConnect (host/port = 10.10.2.3) TCPSocket::Connect
> >> >> (host/port = 10.10.2.3) Socket::TcpConnect (host/port = 10.10.2.3)
> >> >> TCPSocket::Connect (host/port = 10.10.2.3) Socket::TcpConnect
> >> >> (host/port = 10.10.2.3) TCPSocket::Connect (host/port = 10.10.2.3)
> >> >> Socket::TcpConnect (host/port = 10.10.2.3) ..................
> >> >> ..................
> >> >> ..................
> >> >> TCPSocket::Connect (host/port = 10.10.2.3) Socket::TcpConnect
> >> >> (host/port = 10.10.2.3) TCPSocket::Connect (host/port = 10.10.2.3)
> >> >> Socket::TcpConnect (host/port = 10.10.2.3) TCPSocket::Connect
> >> >> (host/port = 10.10.2.3) Socket::TcpConnect (host/port = 10.10.2.3)
> >> >> TCPSocket::Connect (host/port = 10.10.2.3) Socket::TcpConnect
> >> >> (host/port = 10.10.2.3) TCPSocket::Connect (host/port = 10.10.2.3)
> >> >> Socket::TcpConnect (host/port = 10.10.2.3) TCPSocket::Connect
> >> >> (host/port = 10.10.2.3)
> >> >> error: connect remote failed (invalid host:port specification:
> >> >> '10.10.2.3')
> >> >> 0x915bd78 Communication::Write (src = 0x92b38c4, src_len = 27)
> >> >> connection = 0x915f578
> >> >> 0x915f608 Socket::Write() (socket = 7, src = 0x92b38c4, src_len =
> >> >> 27, flags = 0) => 27 (error = (null)) < 27> send packet:
> >> >> $qKillSpawnedProcess:298#8b this = 0x0915BD78, dst = 0xBFCB509C,
> >> >> dst_len = 8192, timeout = 1000000 us, connection = 0x0915F578
> >> >> 0x915f608 Socket::Read() (socket = 7, src = 0xbfcb509c, src_len =
> >> >> 7, flags = 0) => 7 (error = (null))
> >> >> < 7> read packet: $E0a#d6
> >> >> error: process launch failed: invalid host:port specification: '10.10.2.3'
> >> >> (lldb)
> >> >>
> >> >>
> >> >> Regards,
> >> >> Ramana
> >> >> _______________________________________________
> >> >> lldb-dev mailing list
> >> >> lldb-dev at lists.llvm.org
> >> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> >> > _______________________________________________
> >> > lldb-dev mailing list
> >> > lldb-dev at lists.llvm.org
> >> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> >>
> >> _______________________________________________
> >> lldb-dev mailing list
> >> lldb-dev at lists.llvm.org
> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> >
More information about the lldb-dev
mailing list