[lldb-dev] lldb-server tests

[Kamil Rytarowski via lldb-dev]

Thank you for this work.

I will add a usual requirement - please make sure that "check-lldb-unit"
works in standalone builds.

On 15.05.2017 16:33, Pavel Labath wrote:
> Hello all,
> 
> In case you haven't noticed it, I'd like to draw your attention to
> D32930, where we're proposing a new test framework for lldb-server
> tests. The discussion has so far been about low-level implementation
> details, so you don't have to read through it if you don't feel like
> to (but I do encourage it)
> 
> However, I'd like to explain some of the high-level motivations which
> led to our proposed design and open a discussion on them here. I'll do
> this in an FAQ form: :)
> 
> - why new framework?:
> 
> Lldb-server tests were never really suited for the dotest.py model
> anyway (for example they end up creating an SBDebugger, only to be
> completely ignoring it and opening a socket connection to lldb-server
> directly). Perhaps for this reason, they are also harder to write than
> usual lldb tests, and a lot more messy internally (e.g., after Chris's
> ipv6 patch, which caused all lldb-server connections in the test to
> fail, I've learned that the test harness will attempt the lldb-server
> connection 400(!!!) times before conceding defeat). The test suite
> operation is also very illogical when it comes to doing remote tests:
> to test lldb-server on a remote target, you first have to build
> lldb-server for the target, THEN you have to build lldb for the HOST,
> and THEN you run dotest.py in the HOST build folder while passing
> funny dotest.py arguments.
> 
> 
> - why lldb-server ?:
> We'd like this to be a first step in porting the existing test off of
> the dotest.py test runner. Unlike the full test suite, the number of
> lldb-server tests is not that big, so porting them is an task
> achievable in a not too long timeframe, and it can serve as a proof of
> concept when considering further steps. Also, lldb-server already
> performs a relatively well-defined and simple task, which means it
> fits the llvm model of testing isolated components of functionality
> without the need for a massive refactor.
> 
> - why c++ (aka, if the existing test suite is broken, why not just fix it) ?:
> There are two fundamental issues with the current test suite which
> cannot be easily "fixed". The first one is the remote execution (which
> is where a large part of the test harness complexity comes from). By
> writing the test in c++ we can run the test *and* lldb-server remotely
> (***), avoiding the network communication and flakyness that comes
> with it. The other issue is the fact that it needs to have a
> completely independent reimplementation of the gdb-remote protocol.
> Sure, some duplication is expected from tests, but it does not have to
> be that big. If we write the test in c++ we can reuse parts of the
> gdb-remote client code (thereby increasing test coverage of that as
> well), and only resort to manual packet parsing when we really want to
> (e.g., when testing the server response to a malformed packet or
> similar).
> 
> - ok, but why not have the test described by a text file, and have a
> c++ FileCheck-like utility which interprets it?:
> Due to the unpredictable (e.g. we cannot control the addresses of
> objects in the inferior), and interactive nature of the tests, we
> believe they would be easier to write imperatively, instead of a more
> declarative FileCheck style. E.g. for one test you need to send
> qRegisterInfoN for N=1,... until you find the pc register then pluck a
> field with that number from a stop-reply packet, and compare that the
> result of another packet, while possible reversing endianness. To
> describe something like this in a text file, you will either need
> primitives to describe loops, conditionals, etc (which will then tend
> towards implementing a full scripting language), or have a very
> high-level primitive operation which does exactly this, which will
> tend towards having many specialized primitive operations.
> 
> regards,
> pavel
> 
> (***) To achieve this, we want to propose adding the ability to
> execute tests remotely to llvm-lit, which we hope will be useful to
> more people. I'll write more about this in a separate email with
> llvm-dev included.
> 



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20170515/59d13cd5/attachment.sig>