[lldb-dev] race condition using gdb-remote over ssh port forwarding

[Christopher Book via lldb-dev]

I have been using both a custom port-forwarding program as well as
OpenSSH.  The custom solution is a bit slower because it goes through a
proxy, so the problem doesn't manifest.  I've been trying to switch to
OpenSSH which is fast enough to consistent hit the race condition.

I would expect other solutions to be similar to OpenSSH because I don't
think there's a way to have the client connection fail based on the result
of a remote connection.  Our own forwarding solution was built using this
method because of this.

I agree that fixing this on the server seems like a better solution, since
the client only gets a port when it has one to connect to.  But I don't
know the best way to make this happen from lldb-server after launching the
gdb instance.

On Mon, Nov 27, 2017 at 4:48 PM Greg Clayton <clayborg at gmail.com> wrote:

> When I wrote the code I was assuming that when the platform lldb-server
> responds with the port that the gdb-remote would be ready to receive
> packets right away, so I can see how and why this is happening. Seems like
> we have the retry stuff under control when we don't get a connection right
> away, but we should fix this such that when we hand the port back to LLDB
> from platform lldb-server, it should be listening and ready to accept a
> connection right away with no delays needed, though this can wait until
> later since it currently works.
>
> What kind of port forwarding are you using? The main issue is I would
> assume that when someone tries to connect to a port on the port forwarder
> that it would fail to connect if it isn't able to connect on the other
> side. So this really comes down to a question of what a standard port
> forwarder's contract should really be.
>
> If anyone has extensive experience in port forward tech, please chime in.
>
> Short answer: not sure what the right solution is as it depends on what
> proper port forwarding etiquette is.
>
> Greg
>
>
>
>
>
> > On Nov 27, 2017, at 12:33 PM, Christopher Book via lldb-dev <
> lldb-dev at lists.llvm.org> wrote:
> >
> > Greetings, I've been using liblldb to remotely debug to a linux server
> with port forwarding.  To do this, I start lldb-server to with --listen
> specifying a localhost port, as well as with ----min-gdbserver-port and
> --max-gdbserver-port to specify a specific port for use by 'gdb remote'.
> Both ports are forwarded to the remote PC, where liblldb connects to
> localhost.
> >
> > This generally works fine, but there is a race condition.  When the
> client tells lldb-server to start gdb-remote, the port is returned to the
> client which may try to connect before the gdb-remote process is actually
> listening.  Without port-forwarding, this is okay because the client has
> retry logic:
> >
> > ProcessGDBRemote::ConnectToDebugserver
> > ...
> >        retry_count++;
> >         if (retry_count >= max_retry_count)
> >           break;
> >         usleep(100000);
> >
> > But with port-forwarding, the initial connection is always accepted by
> the port-forwarder, and only then does it try to establish a connection to
> the remote port.  It has no way to not accept the incoming local connection
> until it tries the remote end.
> >
> > lldb has some logic to detect this further in the function, by using a
> handshake to ensure the connection is actually made:
> >
> >   // We always seem to be able to open a connection to a local port
> >   // so we need to make sure we can then send data to it. If we can't
> >   // then we aren't actually connected to anything, so try and do the
> >   // handshake with the remote GDB server and make sure that goes
> >   // alright.
> >   if (!m_gdb_comm.HandshakeWithServer(&error)) {
> >     m_gdb_comm.Disconnect();
> >     if (error.Success())
> >       error.SetErrorString("not connected to remote gdb server");
> >     return error;
> >   }
> >
> > But the problem here is that no retry is performed on failure.  The
> caller to the 'attach' API also can't retry because the gdb server is
> terminated on the error.
> >
> > I would like to submit a patch, but first check to see if this solution
> would be acceptable:
> > - Include the handshake within the connection retry loop.
> > - This means fully disconnecting the re-establishing the connection in
> the loop if the handshake fails.
> > - Changing the timeout check to be based on a total absolute time
> instead of 50 iterations with a 100ms sleep.
> >
> > Thoughts?
> >
> > Alternatives could be:
> > - Have lldb-server delay responding to the 'start gdb server' request
> until it could tell (somehow) that the process is listening.
> > - A sleep of some kind on the client side after starting the server but
> before trying to connect.
> >
> > Thanks,
> > Chris
> >
> >
> > _______________________________________________
> > lldb-dev mailing list
> > lldb-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20171127/86050c50/attachment-0001.html>