[lldb-dev] [RFC] Testsuite in lldb & possible future directions

[Pavel Labath via lldb-dev]

On 6 February 2018 at 18:53, Zachary Turner <zturner at google.com> wrote:
> I'm not claiming that it's definitely caused by dotest and that moving away
> from dotest is going to fix all the problems.  Rather, I'm claiming that
> dotest has an unknown amount of flakiness (which may be 0, but may be
> large), and the alternative has a known amount of flakiness (which is very

Well, it may be unknown to you, but as someone who has managed a bot
running tests for a long time, I can tell you that the it's pretty
close to 0. Some test still fail sometimes, but the failure rate is
approximately at the same level as failures caused by the bot not
being able to reach the svn server to fetch the sources.

That said, I'm still in favor of replacing the test runner with lit. I
just think it needs to be done with a steady hand.


>> So I believe we need more lightweight tests, and lldb-test can provide
>> us with that. The main question for me (and that's something I don't
>> really have an answer to) is how to make writing tests like that easy.
>> E.g. for these "foreign" language plugins, the only way to make a
>> self-contained regression test would be to check-in some dwarf which
>> mimics what the compiler in question would produce. But doing that is
>> extremely tedious as we don't have any tooling for that.
>
>
>  Most of these other language plugins are being removed anyway.  Which
> language plugins are going to still remain that aren't some flavor of c/c++?

Well, right now we have another thread proposing the addition of a
Rust plugin, and we will want to resurrect Java support sooner or
later. Go/Ocaml folks may want to do the same, if doing that will not
involve them inventing a whole test framework.

So, I'm not sure where you were heading with that question..

On 6 February 2018 at 18:53, Zachary Turner <zturner at google.com> wrote:
>
>
> On Tue, Feb 6, 2018 at 8:19 AM Pavel Labath via lldb-dev
> <lldb-dev at lists.llvm.org> wrote:
>>
>> On 6 February 2018 at 15:41, Davide Italiano <dccitaliano at gmail.com>
>> wrote:
>> > On Tue, Feb 6, 2018 at 7:09 AM, Pavel Labath <labath at google.com> wrote:
>> >> On 6 February 2018 at 04:11, Davide Italiano via lldb-dev
>> >>
>> >> So, I guess my question is: are you guys looking into making sure that
>> >> others are also able to reproduce the 0-fail+0-xpass state? I would
>> >> love to run the mac test suite locally, as I tend to touch a lot of
>> >> stuff that impacts all targets, but as it stands now, I have very
>> >> little confidence that the test I am running reflect in any way the
>> >> results you will get when you run the test on your end.
>> >>
>> >> I am ready to supply any test logs or information you need if you want
>> >> to try to tackle this.
>> >>
>> >
>> > Yes, I'm definitely interested in making the testusuite
>> > working/reliable on any configuration.
>> > I was afraid there were a lot of latent issues, that's why I sent this
>> > mail in the first place.
>> > It's also the reason why I started thinking about `lldb-test` as a
>> > driver for testing, because I found out the testsuite being a little
>> > inconsistent/brittle depending on the environment it's run on (which,
>> > FWIW, doesn't happen when you run lit/FileCheck or even the unit tests
>> > in lldb). I'm not currently claiming switching to a different method
>> > would improve the situation, but it's worth a shot.
>> >
>>
>> Despite Zachary's claims, I do not believe this is caused by the test
>> driver (dotest). It's definitely not beautiful, but I haven't seen an
>> issue that would be caused by this in a long time. The issue is that
>> the tests are doing too much -- even the simplest involves compiling a
>> fully working executable, which pulls in a lot of stuff from the
>> environment (runtime libraries, dynamic linker, ...) that we have no
>> control of. And of course it makes it impossible to test the debugging
>> functionality of any other platform than what you currently have in
>> front of you.
>
> I'm not claiming that it's definitely caused by dotest and that moving away
> from dotest is going to fix all the problems.  Rather, I'm claiming that
> dotest has an unknown amount of flakiness (which may be 0, but may be
> large), and the alternative has a known amount of flakiness (which is very
> close to, if not equal to 0).  So we should do it because, among other
> benefits, it replaces an unknown with a known that is at least as good, if
> not better.
>
>
>>
>>
>> In this sense, the current setup makes an excellent integration test
>> suite -- if you run the tests and they pass, you can be fairly
>> confident that the debugging on your system is setup correctly.
>> However, it makes a very bad regression test suite, as the tests will
>> be checking something different on each machine.
>>
>> So I believe we need more lightweight tests, and lldb-test can provide
>> us with that. The main question for me (and that's something I don't
>> really have an answer to) is how to make writing tests like that easy.
>> E.g. for these "foreign" language plugins, the only way to make a
>> self-contained regression test would be to check-in some dwarf which
>> mimics what the compiler in question would produce. But doing that is
>> extremely tedious as we don't have any tooling for that.
>
>
>  Most of these other language plugins are being removed anyway.  Which
> language plugins are going to still remain that aren't some flavor of c/c++?