[lldb-dev] lldb-server tests

[Pavel Labath via lldb-dev]

Hello all,

In case you haven't noticed it, I'd like to draw your attention to
D32930, where we're proposing a new test framework for lldb-server
tests. The discussion has so far been about low-level implementation
details, so you don't have to read through it if you don't feel like
to (but I do encourage it)

However, I'd like to explain some of the high-level motivations which
led to our proposed design and open a discussion on them here. I'll do
this in an FAQ form: :)

- why new framework?:

Lldb-server tests were never really suited for the dotest.py model
anyway (for example they end up creating an SBDebugger, only to be
completely ignoring it and opening a socket connection to lldb-server
directly). Perhaps for this reason, they are also harder to write than
usual lldb tests, and a lot more messy internally (e.g., after Chris's
ipv6 patch, which caused all lldb-server connections in the test to
fail, I've learned that the test harness will attempt the lldb-server
connection 400(!!!) times before conceding defeat). The test suite
operation is also very illogical when it comes to doing remote tests:
to test lldb-server on a remote target, you first have to build
lldb-server for the target, THEN you have to build lldb for the HOST,
and THEN you run dotest.py in the HOST build folder while passing
funny dotest.py arguments.


- why lldb-server ?:
We'd like this to be a first step in porting the existing test off of
the dotest.py test runner. Unlike the full test suite, the number of
lldb-server tests is not that big, so porting them is an task
achievable in a not too long timeframe, and it can serve as a proof of
concept when considering further steps. Also, lldb-server already
performs a relatively well-defined and simple task, which means it
fits the llvm model of testing isolated components of functionality
without the need for a massive refactor.

- why c++ (aka, if the existing test suite is broken, why not just fix it) ?:
There are two fundamental issues with the current test suite which
cannot be easily "fixed". The first one is the remote execution (which
is where a large part of the test harness complexity comes from). By
writing the test in c++ we can run the test *and* lldb-server remotely
(***), avoiding the network communication and flakyness that comes
with it. The other issue is the fact that it needs to have a
completely independent reimplementation of the gdb-remote protocol.
Sure, some duplication is expected from tests, but it does not have to
be that big. If we write the test in c++ we can reuse parts of the
gdb-remote client code (thereby increasing test coverage of that as
well), and only resort to manual packet parsing when we really want to
(e.g., when testing the server response to a malformed packet or
similar).

- ok, but why not have the test described by a text file, and have a
c++ FileCheck-like utility which interprets it?:
Due to the unpredictable (e.g. we cannot control the addresses of
objects in the inferior), and interactive nature of the tests, we
believe they would be easier to write imperatively, instead of a more
declarative FileCheck style. E.g. for one test you need to send
qRegisterInfoN for N=1,... until you find the pc register then pluck a
field with that number from a stop-reply packet, and compare that the
result of another packet, while possible reversing endianness. To
describe something like this in a text file, you will either need
primitives to describe loops, conditionals, etc (which will then tend
towards implementing a full scripting language), or have a very
high-level primitive operation which does exactly this, which will
tend towards having many specialized primitive operations.

regards,
pavel

(***) To achieve this, we want to propose adding the ability to
execute tests remotely to llvm-lit, which we hope will be useful to
more people. I'll write more about this in a separate email with
llvm-dev included.