[Lldb-commits] [lldb] [lldb][test] Support remote run of Shell tests (PR #95986)

[Vladislav Dzhidzhoev via lldb-commits]

dzhidzhoev wrote:

> That bug shows we're not able to create a test for a very simple thing (an unaligned stack pointer) that works reliably for everyone.

Honestly, based on https://github.com/llvm/llvm-project/issues/101710#issuecomment-2291180977, I'd say that the mentioned test has a problem of too many redundant dependencies, rather than that it shows the backside of tests universality.

> In the end, that also reduces test coverage, because people will start to (even proactively) add REQUIRES: my-system to their tests to avoid breaking other people's CI.

LLDB tests will still somehow depend on at least three targets in the foreseeable future (Linux/Windows/Darwin). If a test fails or is not supported on Linux target, it's unlikely that it will work on Linux target with remote debugging. So I don't see a perspective of having test coverage reduced because of `UNSUPPORTED: remote-` annotations if they appear.

> This is important as it slows down the velocity of all developers (old and new).

I see a situation of a development velocity slowdown if a test the developer wrote is not supposed to work for remote debugging, but is missing a proper annotation for that. For other cases, remote debugging support is claimed as an LLDB feature. If tests are written/executed using the LLDB command line, remote execution shouldn't hinder their execution.

BTW, most of the cases where tests should've been disabled for remote debugging were the cases where platform mismatch happened. Tests were already disabled for "system-windows", but they were attempted to be run against Linux target. Probably, "system-X" semantics should be redefined to indicate a target platform instead of a host platform, or a feature like "target-system-xxx" should be introduced to show that a test should be run on a Windows/Darwin target regardless of the host. I think this will eliminate the need for `UNSUPPORTED: remote-` annotations for most of the cases. 

Considering the complexity of setting up a remote debugging environment, public buildbots with it will be available https://discourse.llvm.org/t/rfc-lldb-support-remote-run-of-shell-tests/80072/4, https://lab.llvm.org/staging/#/builders/195.

Also, the case when no new features are added, but existing ones are changed, shouldn't be neglected. In such cases, it is hard to justify disabling tests without finding out the reason.

> For me, this just sends the wrong message/sets the wrong incentive.

>From my point of view, the current incentive given is that remote debugging compatibility may be ignored when new commits are made since it's something rare enough/hard to test properly (because testing with it requires writing an API test). 
The ability to run Shell tests remotely (and regular testing) may reduce the barrier for those who take remote debugging into account in their patches. 

https://github.com/llvm/llvm-project/pull/95986