[Lldb-commits] [PATCH] D46005: [test] Add a utility for dumping all tests in the dotest suite

[Zachary Turner via lldb-commits]

But is there a reason why that is more valuable with LLDB than it is with
LLVM?  In LLVM when a test fails it stops and doesn't run subsequent run
lines.  So you have the same issue there.  The way this is handled in LLVM
is that if you think tests are sufficiently different that they could be
broken by different types of changes, you split them up into different
files.

Do you feel LLDB is fundamentally different in this regard?

On Wed, Apr 25, 2018 at 8:27 AM Adrian Prantl <aprantl at apple.com> wrote:

> Disregarding the added complexity I believe that being able to communicate
> the number of tests filed/passed inside a single file would be very
> valuable. For example when one test gets broken by upstream clang changes
> and it takes a few days to fix it properly, we don't want to loose signal
> on the other tests in the same file during that period. I think it's worth
> it.
>
> -- adrian
>
>
> On Apr 25, 2018, at 8:13 AM, Zachary Turner <zturner at google.com> wrote:
>
> Well let’s see what Davide and Adrian think. I’m more of an outsider these
> days so consider my perspective an llvm-centric one, which would sometimes
> (but not always) be the best for lldb
> On Wed, Apr 25, 2018 at 12:31 AM Pavel Labath via Phabricator <
> reviews at reviews.llvm.org> wrote:
>
>> labath added a comment.
>>
>> In https://reviews.llvm.org/D46005#1077016, @zturner wrote:
>>
>> > Note that there's currently no precedent (that i'm aware of anwyay) in
>> LLVM or any of its subprojects for splitting the running of a single file
>> into multiple parallel jobs.  All of LLVM's other projects parallelize at
>> file granularity, and so it's up to to the person writing tests to ensure
>> that the file granularity matches that with which they want tests to be
>> parallelized at.
>>
>>
>> There's always gtest. Regardless of whether you consider a .cpp file or
>> the built exectable to be a "test file", it will generally contain a number
>> of tests. And arguably, our test suite is more similar to the gtest suite
>> than the traditional lit (ShTest) suites (for one, it hijacks a third party
>> unit testing library, and then tries to make it run under lit). And we run
>> all gtest tests individually (I've often wondered what kind of performance
>> impact that has, but they seem to be running fast enough anyway...).
>>
>> For what it's worth, paralelization is not my motivation here. There some
>> tests which run disproportionately long, and this will speed them up, but
>> that may be offset by the fact we need to start more work this way (if
>> anyone is interested I can try to do some measurements). My main reason for
>> doing this is so that we can have better mapping of test result states. As
>> it stands now, we only have two possible results for a test: passed or
>> failed. Lit has more results than that, and they roughly match the existing
>> dotest states (the different treatment of XFAILs is the biggest
>> difference), so it should be possible to represent them with more fidelity.
>> However, right now it's not possible to translate them reasonably, as a
>> "single test" can result in any number of fails/xfails/skips/... (or no
>> results at all).
>>
>> > That doesn't mean it's not possible (as you've shown here), but it adds
>> some additional complexity and I'm not sure it's worth it.
>>
>> It adds complexity, but not much imho. I was actually pleasantly
>> surprised at how easy it was to pull this off. The entire implementation is
>> 78 LOC right now. The changes to the test format will probably push it over
>> a 100, but not by much.
>>
>> That said, I am not interested in forcing this onto everyone. I did it
>> because it seemed like a nice thing to have and I was curious if it is
>> doable (easily). However, if there's no interest for it, then I can just
>> abandon it..
>>
>>
>> https://reviews.llvm.org/D46005
>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-commits/attachments/20180425/6fb8440c/attachment-0001.html>