[Lldb-commits] [PATCH] D32930: New framework for lldb client-server communication tests.

[Pavel Labath via lldb-commits]

I think the discussion has veered of from the "lldb-server testing"
topic into the more murky "general testing" waters. I'll put my
thoughts on those first, and then get back to lldb-server test plans.

On 31 May 2017 at 18:37, Zachary Turner via Phabricator
<reviews at reviews.llvm.org> wrote:
> What *would* help would be to ignore how difficult it may or may not be, and just take a couple of tests and re-write them in some DSL that you invent specifically for this purpose that is as concise as possible yet as expressive as you need, and we go from there.  I did this with a couple of fairly hairy tests a few months ago and it didn't seem that bad to me.

All DSL examples I've seen from you were toy-ish in one aspect or
another and I could not see a straightforward (or even a complicated)
way to extend them to handle some of the hairy test scenarios. If you
want to see a more complicated example look at
<https://reviews.llvm.org/D33426>, which has been crawling for a
couple of weeks, large part of it due to the complexity of coming up
with a reliable test vector. I don't see a way to create a reasonable
DSL for this test -- you'd either have to add loops and conditions and
very fine-grained primitives for accessing SBAPI (at which point you
might as well use SBAPI itself), or you need to introduce a very high
level primitive operation, which would drive the threads to a desired
state (and end up with a very large number of primitives most of which
are used only in a single test).


What I think that makes writing a lot of the tests difficult is the
fact that all of the tests are responsible for building their own
executables and driving them to the state where you want to perform
the actual test. This makes the test very undeterministic and
environment-dependent.
- to run a libstdc++ data formatter test, you need to have a system
capable of running libstdc++ and have the library installed. To write
a test that verifies that the data formatter handles corrupt STL data
structures reasonably, you need to write some very
non-standard-compliant code to corrupt them yourself. And if the STL
structure layout changes, and you want your data formatter to handle
both layouts, you cannot write tests that will verify that both of
them still work.

- to write a test for the handling of some DWARF corner case, you need
to figure out how to coax the compiler into producing the dwarf you
want. And even then, you have no guarantee that when you update the
compiler, it will still produce the same dwarf that you were
interested in testing.

- etc.


Now, I think you actually agree with most of this. What differs (I
believe) is the conclusion we draw from it. My conclusion is that
there is no point in trying to invent a DSL yet. A lot of other things
need to happen first. For example, my ideal testing strategy for data
formatters would not involve running a process *at all*. All the data
formatters (at least the sane ones which do not evaluate expressions)
need is something that provides contents of some memory and a bit of
debug info. This is pretty much what a core file does. However, we're
not ready to start using core files everywhere yet, because we don't
have a reasonable way to generate small cores (they can be small, if
you try enough, but it's not easy), and checking in binaries is gross
(so we need to have a textual representation of them and tool for
converting to/from it).

At that point it would be easy to write a DSL for data formatter
tests. For example, you could have a yaml description of the state of
the stl container and a tool which ingests that and spits out a "fake"
core file. Then you load that core file in lldb, run a bunch of "frame
variable" commands and FileCheck their output. No run commands
necessary, no makefiles, and the test will work on every platform.

However, this requires a lot of prep work, and I don't see anyone
committing to doing that right now.


As for lldb-server tests, I think the case for a lldb-server DSL is
much weaker than that for many other things that lldb does. That DSL
would need to handle all of the 100 or so packets that the program
supports, and need to be able to construct them using quite
complicated expressions. It's possible, but I don't believe it's worth
the effort right now. You're welcome to try, I can give you the
requirements (first one: it needs to be able to run remotely), but
we're not going to attempt it right now. We hope to address a couple
of other problematic issues with the new test suite (complicated test
framework, Makefiles, code duplication in the tests, paralelization
problems) and I believe any future DSL for these tests can build on
this by providing adding a front-end to the test client we are adding
here.