[Lldb-commits] [PATCH] D46005: [test] Add a utility for dumping all tests in the dotest suite

[Zachary Turner via lldb-commits]

Sure multiple run lines can be commands being executed that depend on each
other, but it's very common for a test to have multiple check labels as well

On Wed, Apr 25, 2018 at 9:19 AM Adrian Prantl <aprantl at apple.com> wrote:

>
> On Apr 25, 2018, at 9:08 AM, Zachary Turner <zturner at google.com> wrote:
>
> But is there a reason why that is more valuable with LLDB than it is with
> LLVM?  In LLVM when a test fails it stops and doesn't run subsequent run
> lines.  So you have the same issue there.
>
>
> That's not a good analogy. Multiple RUN lines don't necessarily mean
> separate tests, they mean separate commands being executed that may depend
> on each other. As Pavel pointed out before, this would be closer to how
> gtests behave.
>
> -- adrian
>
> The way this is handled in LLVM is that if you think tests are
> sufficiently different that they could be broken by different types of
> changes, you split them up into different files.
>
> Do you feel LLDB is fundamentally different in this regard?
>
> On Wed, Apr 25, 2018 at 8:27 AM Adrian Prantl <aprantl at apple.com> wrote:
>
>> Disregarding the added complexity I believe that being able to
>> communicate the number of tests filed/passed inside a single file would be
>> very valuable. For example when one test gets broken by upstream clang
>> changes and it takes a few days to fix it properly, we don't want to loose
>> signal on the other tests in the same file during that period. I think it's
>> worth it.
>>
>> -- adrian
>>
>>
>> On Apr 25, 2018, at 8:13 AM, Zachary Turner <zturner at google.com> wrote:
>>
>> Well let’s see what Davide and Adrian think. I’m more of an outsider
>> these days so consider my perspective an llvm-centric one, which would
>> sometimes (but not always) be the best for lldb
>> On Wed, Apr 25, 2018 at 12:31 AM Pavel Labath via Phabricator <
>> reviews at reviews.llvm.org> wrote:
>>
>>> labath added a comment.
>>>
>>> In https://reviews.llvm.org/D46005#1077016, @zturner wrote:
>>>
>>> > Note that there's currently no precedent (that i'm aware of anwyay) in
>>> LLVM or any of its subprojects for splitting the running of a single file
>>> into multiple parallel jobs.  All of LLVM's other projects parallelize at
>>> file granularity, and so it's up to to the person writing tests to ensure
>>> that the file granularity matches that with which they want tests to be
>>> parallelized at.
>>>
>>>
>>> There's always gtest. Regardless of whether you consider a .cpp file or
>>> the built exectable to be a "test file", it will generally contain a number
>>> of tests. And arguably, our test suite is more similar to the gtest suite
>>> than the traditional lit (ShTest) suites (for one, it hijacks a third party
>>> unit testing library, and then tries to make it run under lit). And we run
>>> all gtest tests individually (I've often wondered what kind of performance
>>> impact that has, but they seem to be running fast enough anyway...).
>>>
>>> For what it's worth, paralelization is not my motivation here. There
>>> some tests which run disproportionately long, and this will speed them up,
>>> but that may be offset by the fact we need to start more work this way (if
>>> anyone is interested I can try to do some measurements). My main reason for
>>> doing this is so that we can have better mapping of test result states. As
>>> it stands now, we only have two possible results for a test: passed or
>>> failed. Lit has more results than that, and they roughly match the existing
>>> dotest states (the different treatment of XFAILs is the biggest
>>> difference), so it should be possible to represent them with more fidelity.
>>> However, right now it's not possible to translate them reasonably, as a
>>> "single test" can result in any number of fails/xfails/skips/... (or no
>>> results at all).
>>>
>>> > That doesn't mean it's not possible (as you've shown here), but it
>>> adds some additional complexity and I'm not sure it's worth it.
>>>
>>> It adds complexity, but not much imho. I was actually pleasantly
>>> surprised at how easy it was to pull this off. The entire implementation is
>>> 78 LOC right now. The changes to the test format will probably push it over
>>> a 100, but not by much.
>>>
>>> That said, I am not interested in forcing this onto everyone. I did it
>>> because it seemed like a nice thing to have and I was curious if it is
>>> doable (easily). However, if there's no interest for it, then I can just
>>> abandon it..
>>>
>>>
>>> https://reviews.llvm.org/D46005
>>>
>>>
>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-commits/attachments/20180425/2fce504e/attachment-0001.html>