[lldb-dev] lldb-server tests

[Zachary Turner via lldb-dev]

One thing about lit that most people either don't understand or forget
about is that FileCheck has nothing to do with lit. You can have lit tests
without FileCheck. It's more work because you would have to define an
LLDBServerTestFormat and invent some DSL that isn't just a bunch of run
lines and check statements. You could then write a c++ program like
lldb-server-test, which gives you all the benefits of code reuse and packet
parsing that you're talking about, and have your test consist of something
like:

check-lldb-server --prefix=TEST_ERROR_RESPONSE < %s
check-lldb-server --prefix=TEST_SUCCESS_RESPONSE < %s

TEST_ERROR_RESPONSE: SEND: <some hex packet string>
TEST_ERROR_RESPONSE: RECV-ERROR: 62

TEST_SUCCESS_RESPONSE: SEND: <some hex packet string>
TEST_SUCCESS_RESPONSE: RECV-SUCCESS: eax = 7

I think this would make tests both easier to write and easier to understand
than what is being proposed here.

That said, what is being proposed here can't exactly be called a lateral
move, because I do agree it's better.  So because of that, I'm willing to
let it go in.  But I'm 100% confident that a better solution can be devised
in lit with some thought.  Unfortunately, given that I don't work on
lldb-server, all I can really do is offer some high level ideas, and it
will be up to you guys to figure out the details.


On Mon, May 15, 2017 at 7:34 AM Pavel Labath <labath at google.com> wrote:

> Hello all,
>
> In case you haven't noticed it, I'd like to draw your attention to
> D32930, where we're proposing a new test framework for lldb-server
> tests. The discussion has so far been about low-level implementation
> details, so you don't have to read through it if you don't feel like
> to (but I do encourage it)
>
> However, I'd like to explain some of the high-level motivations which
> led to our proposed design and open a discussion on them here. I'll do
> this in an FAQ form: :)
>
> - why new framework?:
>
> Lldb-server tests were never really suited for the dotest.py model
> anyway (for example they end up creating an SBDebugger, only to be
> completely ignoring it and opening a socket connection to lldb-server
> directly). Perhaps for this reason, they are also harder to write than
> usual lldb tests, and a lot more messy internally (e.g., after Chris's
> ipv6 patch, which caused all lldb-server connections in the test to
> fail, I've learned that the test harness will attempt the lldb-server
> connection 400(!!!) times before conceding defeat). The test suite
> operation is also very illogical when it comes to doing remote tests:
> to test lldb-server on a remote target, you first have to build
> lldb-server for the target, THEN you have to build lldb for the HOST,
> and THEN you run dotest.py in the HOST build folder while passing
> funny dotest.py arguments.
>
>
> - why lldb-server ?:
> We'd like this to be a first step in porting the existing test off of
> the dotest.py test runner. Unlike the full test suite, the number of
> lldb-server tests is not that big, so porting them is an task
> achievable in a not too long timeframe, and it can serve as a proof of
> concept when considering further steps. Also, lldb-server already
> performs a relatively well-defined and simple task, which means it
> fits the llvm model of testing isolated components of functionality
> without the need for a massive refactor.
>
> - why c++ (aka, if the existing test suite is broken, why not just fix it)
> ?:
> There are two fundamental issues with the current test suite which
> cannot be easily "fixed". The first one is the remote execution (which
> is where a large part of the test harness complexity comes from). By
> writing the test in c++ we can run the test *and* lldb-server remotely
> (***), avoiding the network communication and flakyness that comes
> with it. The other issue is the fact that it needs to have a
> completely independent reimplementation of the gdb-remote protocol.
> Sure, some duplication is expected from tests, but it does not have to
> be that big. If we write the test in c++ we can reuse parts of the
> gdb-remote client code (thereby increasing test coverage of that as
> well), and only resort to manual packet parsing when we really want to
> (e.g., when testing the server response to a malformed packet or
> similar).
>
> - ok, but why not have the test described by a text file, and have a
> c++ FileCheck-like utility which interprets it?:
> Due to the unpredictable (e.g. we cannot control the addresses of
> objects in the inferior), and interactive nature of the tests, we
> believe they would be easier to write imperatively, instead of a more
> declarative FileCheck style. E.g. for one test you need to send
> qRegisterInfoN for N=1,... until you find the pc register then pluck a
> field with that number from a stop-reply packet, and compare that the
> result of another packet, while possible reversing endianness. To
> describe something like this in a text file, you will either need
> primitives to describe loops, conditionals, etc (which will then tend
> towards implementing a full scripting language), or have a very
> high-level primitive operation which does exactly this, which will
> tend towards having many specialized primitive operations.
>
> regards,
> pavel
>
> (***) To achieve this, we want to propose adding the ability to
> execute tests remotely to llvm-lit, which we hope will be useful to
> more people. I'll write more about this in a separate email with
> llvm-dev included.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20170515/39275a76/attachment-0001.html>