[lldb-dev] race condition using gdb-remote over ssh port forwarding

[Greg Clayton via lldb-dev]

When I wrote the code I was assuming that when the platform lldb-server responds with the port that the gdb-remote would be ready to receive packets right away, so I can see how and why this is happening. Seems like we have the retry stuff under control when we don't get a connection right away, but we should fix this such that when we hand the port back to LLDB from platform lldb-server, it should be listening and ready to accept a connection right away with no delays needed, though this can wait until later since it currently works.

What kind of port forwarding are you using? The main issue is I would assume that when someone tries to connect to a port on the port forwarder that it would fail to connect if it isn't able to connect on the other side. So this really comes down to a question of what a standard port forwarder's contract should really be. 

If anyone has extensive experience in port forward tech, please chime in.

Short answer: not sure what the right solution is as it depends on what proper port forwarding etiquette is.

Greg





> On Nov 27, 2017, at 12:33 PM, Christopher Book via lldb-dev <lldb-dev at lists.llvm.org> wrote:
> 
> Greetings, I've been using liblldb to remotely debug to a linux server with port forwarding.  To do this, I start lldb-server to with --listen specifying a localhost port, as well as with ----min-gdbserver-port and --max-gdbserver-port to specify a specific port for use by 'gdb remote'.  Both ports are forwarded to the remote PC, where liblldb connects to localhost.
> 
> This generally works fine, but there is a race condition.  When the client tells lldb-server to start gdb-remote, the port is returned to the client which may try to connect before the gdb-remote process is actually listening.  Without port-forwarding, this is okay because the client has retry logic:
> 
> ProcessGDBRemote::ConnectToDebugserver
> ...
>        retry_count++;
>         if (retry_count >= max_retry_count)
>           break;
>         usleep(100000);
> 
> But with port-forwarding, the initial connection is always accepted by the port-forwarder, and only then does it try to establish a connection to the remote port.  It has no way to not accept the incoming local connection until it tries the remote end.
> 
> lldb has some logic to detect this further in the function, by using a handshake to ensure the connection is actually made:
> 
>   // We always seem to be able to open a connection to a local port
>   // so we need to make sure we can then send data to it. If we can't
>   // then we aren't actually connected to anything, so try and do the
>   // handshake with the remote GDB server and make sure that goes
>   // alright.
>   if (!m_gdb_comm.HandshakeWithServer(&error)) {
>     m_gdb_comm.Disconnect();
>     if (error.Success())
>       error.SetErrorString("not connected to remote gdb server");
>     return error;
>   }
> 
> But the problem here is that no retry is performed on failure.  The caller to the 'attach' API also can't retry because the gdb server is terminated on the error.
> 
> I would like to submit a patch, but first check to see if this solution would be acceptable:
> - Include the handshake within the connection retry loop.
> - This means fully disconnecting the re-establishing the connection in the loop if the handshake fails.
> - Changing the timeout check to be based on a total absolute time instead of 50 iterations with a 100ms sleep.
> 
> Thoughts?
> 
> Alternatives could be:
> - Have lldb-server delay responding to the 'start gdb server' request until it could tell (somehow) that the process is listening.
> - A sleep of some kind on the client side after starting the server but before trying to connect.
> 
> Thanks,
> Chris
> 
> 
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev