[Lldb-commits] [lldb] [lldb][test] Support remote run of Shell tests (PR #95986)

[Pavel Labath via lldb-commits]

labath wrote:

> > Can't say I'm excited to see this feature (although I admit it is looking less daunting than I originally feared). Can you explain why you need this feature? Is there some particular aspect of lldb's remote operation that you think isn't well covered by the existing tests? Is there a particular category of tests that you'd like to have more than others? How many of the existing tests would exercise the remote platform capability in a meaningful way (a lot of these tests don't even build runnable binaries)?
> 
> I'm sorry, it took a bit longer time to answer than I expected, my apologies.
> 
> In general, our goal is to maximize the functionality of the testing toolkit in case of cross-compilation+remote launch setup.
> 
> I agree that not every Shell test launches binaries. According to my grep-based statistics, it's roughly one-fifth of the tests currently (117/533 test files that contain "r", "run", or "process launch" commands). Though it's still a number.

Thank you for finding that number.

> 
> Also, launching the "platform select" and "platform connect ..." commands before executing the rest of a Shell test script may be useful. Here is an example case of a potentially unexpected behavior #94165. So it's not always necessary to have a Shell test build and run a binary to reveal some nuances.

That's true, but this still seems like a pretty big hammer for that.


> And it seems to me, that it won't cause excessive maintenance overhead since we essentially add just two extra commands to %lldb invocation.

Maybe. Like I said, it's cleaner than I though, but I'm still somewhat worried about where this will lead. If you can support running these tests on a different machine, then I think the next question becomes "why not run them with a different compiler as well". And then you need to XFAIL the tests for a particular compiler, so you end up replicating all of the `@skipIfClang71ExceptOnTuesdays` logic.

I don't know if this is a slippery slope fallacy or an actual slippery slope, but this extra bit of test coverage does not seem worth it to me, and I sort of liked the separation how API tests can run in a multitude of configurations while a Shell test runs in one. I know that isn't really a good separation since "the way I write the tests" is in principle independent of "where I want to run it", but since the two tests use completely separate infrastructures, I think it would make sense to keep this functionality in just one of them.

That said I don't want to be only person blocking this, so I'd like to hear what others think as well. @JDevlieghere ?


> I haven't opened an RFC since I thought it would be clearer to discuss that idea with the showcase of implementation. I'm open to moving a discussion there if it's considered as a better approach in this case (please let me know😅).

Maybe the best would be an RFC with a link to a POC, but now that we're here, I think we can stick with it and see how it goes.

https://github.com/llvm/llvm-project/pull/95986