[lldb-dev] race condition using gdb-remote over ssh port forwarding

[Greg Clayton via lldb-dev]

> On Nov 27, 2017, at 3:50 PM, Christopher Book <cbook at google.com> wrote:
> 
> I have been using both a custom port-forwarding program as well as OpenSSH.  The custom solution is a bit slower because it goes through a proxy, so the problem doesn't manifest.  I've been trying to switch to OpenSSH which is fast enough to consistent hit the race condition.
> 
> I would expect other solutions to be similar to OpenSSH because I don't think there's a way to have the client connection fail based on the result of a remote connection.  Our own forwarding solution was built using this method because of this.
> 
> I agree that fixing this on the server seems like a better solution, since the client only gets a port when it has one to connect to.  But I don't know the best way to make this happen from lldb-server after launching the gdb instance.
> 

You might need to connect to it from the platform lldb-server and then disconnect and have the gdb-server listen for another connection. Maybe if you pass an extra argument for the double connect?


> On Mon, Nov 27, 2017 at 4:48 PM Greg Clayton <clayborg at gmail.com <mailto:clayborg at gmail.com>> wrote:
> When I wrote the code I was assuming that when the platform lldb-server responds with the port that the gdb-remote would be ready to receive packets right away, so I can see how and why this is happening. Seems like we have the retry stuff under control when we don't get a connection right away, but we should fix this such that when we hand the port back to LLDB from platform lldb-server, it should be listening and ready to accept a connection right away with no delays needed, though this can wait until later since it currently works.
> 
> What kind of port forwarding are you using? The main issue is I would assume that when someone tries to connect to a port on the port forwarder that it would fail to connect if it isn't able to connect on the other side. So this really comes down to a question of what a standard port forwarder's contract should really be.
> 
> If anyone has extensive experience in port forward tech, please chime in.
> 
> Short answer: not sure what the right solution is as it depends on what proper port forwarding etiquette is.
> 
> Greg
> 
> 
> 
> 
> 
> > On Nov 27, 2017, at 12:33 PM, Christopher Book via lldb-dev <lldb-dev at lists.llvm.org <mailto:lldb-dev at lists.llvm.org>> wrote:
> >
> > Greetings, I've been using liblldb to remotely debug to a linux server with port forwarding.  To do this, I start lldb-server to with --listen specifying a localhost port, as well as with ----min-gdbserver-port and --max-gdbserver-port to specify a specific port for use by 'gdb remote'.  Both ports are forwarded to the remote PC, where liblldb connects to localhost.
> >
> > This generally works fine, but there is a race condition.  When the client tells lldb-server to start gdb-remote, the port is returned to the client which may try to connect before the gdb-remote process is actually listening.  Without port-forwarding, this is okay because the client has retry logic:
> >
> > ProcessGDBRemote::ConnectToDebugserver
> > ...
> >        retry_count++;
> >         if (retry_count >= max_retry_count)
> >           break;
> >         usleep(100000);
> >
> > But with port-forwarding, the initial connection is always accepted by the port-forwarder, and only then does it try to establish a connection to the remote port.  It has no way to not accept the incoming local connection until it tries the remote end.
> >
> > lldb has some logic to detect this further in the function, by using a handshake to ensure the connection is actually made:
> >
> >   // We always seem to be able to open a connection to a local port
> >   // so we need to make sure we can then send data to it. If we can't
> >   // then we aren't actually connected to anything, so try and do the
> >   // handshake with the remote GDB server and make sure that goes
> >   // alright.
> >   if (!m_gdb_comm.HandshakeWithServer(&error)) {
> >     m_gdb_comm.Disconnect();
> >     if (error.Success())
> >       error.SetErrorString("not connected to remote gdb server");
> >     return error;
> >   }
> >
> > But the problem here is that no retry is performed on failure.  The caller to the 'attach' API also can't retry because the gdb server is terminated on the error.
> >
> > I would like to submit a patch, but first check to see if this solution would be acceptable:
> > - Include the handshake within the connection retry loop.
> > - This means fully disconnecting the re-establishing the connection in the loop if the handshake fails.
> > - Changing the timeout check to be based on a total absolute time instead of 50 iterations with a 100ms sleep.
> >
> > Thoughts?
> >
> > Alternatives could be:
> > - Have lldb-server delay responding to the 'start gdb server' request until it could tell (somehow) that the process is listening.
> > - A sleep of some kind on the client side after starting the server but before trying to connect.
> >
> > Thanks,
> > Chris
> >
> >
> > _______________________________________________
> > lldb-dev mailing list
> > lldb-dev at lists.llvm.org <mailto:lldb-dev at lists.llvm.org>
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev>
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20171127/255b9442/attachment.html>