[lldb-dev] [RFC] Testsuite in lldb & possible future directions

[Davide Italiano via lldb-dev]

On Tue, Feb 6, 2018 at 8:18 AM, Pavel Labath <labath at google.com> wrote:
> On 6 February 2018 at 15:41, Davide Italiano <dccitaliano at gmail.com> wrote:
>> On Tue, Feb 6, 2018 at 7:09 AM, Pavel Labath <labath at google.com> wrote:
>>> On 6 February 2018 at 04:11, Davide Italiano via lldb-dev
>>>
>>> So, I guess my question is: are you guys looking into making sure that
>>> others are also able to reproduce the 0-fail+0-xpass state? I would
>>> love to run the mac test suite locally, as I tend to touch a lot of
>>> stuff that impacts all targets, but as it stands now, I have very
>>> little confidence that the test I am running reflect in any way the
>>> results you will get when you run the test on your end.
>>>
>>> I am ready to supply any test logs or information you need if you want
>>> to try to tackle this.
>>>
>>
>> Yes, I'm definitely interested in making the testusuite
>> working/reliable on any configuration.
>> I was afraid there were a lot of latent issues, that's why I sent this
>> mail in the first place.
>> It's also the reason why I started thinking about `lldb-test` as a
>> driver for testing, because I found out the testsuite being a little
>> inconsistent/brittle depending on the environment it's run on (which,
>> FWIW, doesn't happen when you run lit/FileCheck or even the unit tests
>> in lldb). I'm not currently claiming switching to a different method
>> would improve the situation, but it's worth a shot.
>>
>
> Despite Zachary's claims, I do not believe this is caused by the test
> driver (dotest). It's definitely not beautiful, but I haven't seen an
> issue that would be caused by this in a long time. The issue is that
> the tests are doing too much -- even the simplest involves compiling a
> fully working executable, which pulls in a lot of stuff from the
> environment (runtime libraries, dynamic linker, ...) that we have no
> control of. And of course it makes it impossible to test the debugging
> functionality of any other platform than what you currently have in
> front of you.
>
> In this sense, the current setup makes an excellent integration test
> suite -- if you run the tests and they pass, you can be fairly
> confident that the debugging on your system is setup correctly.
> However, it makes a very bad regression test suite, as the tests will
> be checking something different on each machine.
>

Yes, I didn't complain about "dotest" in general, but, as you say, the
fact that it pull in lots of stuffs we don't really have control on.
Also, most of the times I actually found out we've been sloppy watching
bots for a while, or XFAILING tests instead of fixing them and that resulted in
issues piling up). This is a more general problem not necessarily tied to
`dotest` as a driver.

> So I believe we need more lightweight tests, and lldb-test can provide
> us with that. The main question for me (and that's something I don't

+1.

> really have an answer to) is how to make writing tests like that easy.
> E.g. for these "foreign" language plugins, the only way to make a
> self-contained regression test would be to check-in some dwarf which
> mimics what the compiler in question would produce. But doing that is
> extremely tedious as we don't have any tooling for that. Since debug
> info is very central to what we do, having something like that would
> go a long way towards improving the testing situation, and it would be
> useful for C/C++ as well, as we generally need to make sure that we
> work with a wide range of compiler versions, not just accept what ToT
> clang happens to produce.
>

I think the plan here (and I'd love to spend some time on this once we
have stability, which seems we're slowly getting) is that of enhancing
`yaml2*` to do the work for us.
I do agree is a major undertaken but even spending a month on it will
go a long way IMHO. I will try to come up with a plan after discussing
with folks in my team (I'd really love to get also inputs from DWARF
people in llvm, e.g. Eric or David Blake).

>
> PS: I saw your second email as well. I'm going to try out what you
> propose, most likely tomorrow.

Thanks!

--
Davide