[lldb-dev] [RFC] Testsuite in lldb & possible future directions

[Pavel Labath via lldb-dev]

On 6 February 2018 at 15:41, Davide Italiano <dccitaliano at gmail.com> wrote:
> On Tue, Feb 6, 2018 at 7:09 AM, Pavel Labath <labath at google.com> wrote:
>> On 6 February 2018 at 04:11, Davide Italiano via lldb-dev
>>
>> So, I guess my question is: are you guys looking into making sure that
>> others are also able to reproduce the 0-fail+0-xpass state? I would
>> love to run the mac test suite locally, as I tend to touch a lot of
>> stuff that impacts all targets, but as it stands now, I have very
>> little confidence that the test I am running reflect in any way the
>> results you will get when you run the test on your end.
>>
>> I am ready to supply any test logs or information you need if you want
>> to try to tackle this.
>>
>
> Yes, I'm definitely interested in making the testusuite
> working/reliable on any configuration.
> I was afraid there were a lot of latent issues, that's why I sent this
> mail in the first place.
> It's also the reason why I started thinking about `lldb-test` as a
> driver for testing, because I found out the testsuite being a little
> inconsistent/brittle depending on the environment it's run on (which,
> FWIW, doesn't happen when you run lit/FileCheck or even the unit tests
> in lldb). I'm not currently claiming switching to a different method
> would improve the situation, but it's worth a shot.
>

Despite Zachary's claims, I do not believe this is caused by the test
driver (dotest). It's definitely not beautiful, but I haven't seen an
issue that would be caused by this in a long time. The issue is that
the tests are doing too much -- even the simplest involves compiling a
fully working executable, which pulls in a lot of stuff from the
environment (runtime libraries, dynamic linker, ...) that we have no
control of. And of course it makes it impossible to test the debugging
functionality of any other platform than what you currently have in
front of you.

In this sense, the current setup makes an excellent integration test
suite -- if you run the tests and they pass, you can be fairly
confident that the debugging on your system is setup correctly.
However, it makes a very bad regression test suite, as the tests will
be checking something different on each machine.

So I believe we need more lightweight tests, and lldb-test can provide
us with that. The main question for me (and that's something I don't
really have an answer to) is how to make writing tests like that easy.
E.g. for these "foreign" language plugins, the only way to make a
self-contained regression test would be to check-in some dwarf which
mimics what the compiler in question would produce. But doing that is
extremely tedious as we don't have any tooling for that. Since debug
info is very central to what we do, having something like that would
go a long way towards improving the testing situation, and it would be
useful for C/C++ as well, as we generally need to make sure that we
work with a wide range of compiler versions, not just accept what ToT
clang happens to produce.


PS: I saw your second email as well. I'm going to try out what you
propose, most likely tomorrow.