[llvm-dev] [cfe-dev] RFC: End-to-end testing

Tue Oct 8 19:14:54 PDT 2019

> I have a bit of concern about this sort of thing - worrying it'll lead to
> people being less cautious about writing the more isolated tests.
>

I have the same concern. I really believe we need to be careful about
testing at the right granularity to keep things both modular and the
testing maintainable (for instance checking vectorized ASM from a C++
source through clang has always been considered a bad FileCheck practice).
(Not saying that there is no space for better integration testing in some
areas).

> That said, clearly there's value in end-to-end testing for all the reasons
> you've mentioned (& we do see these problems in practice - recently DWARF
> indexing broke when support for more nuanced language codes were added to
> Clang).
>
> Dunno if they need a new place or should just be more stuff in test-suite,
> though.
>
> On Tue, Oct 8, 2019 at 9:50 AM David Greene via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
>> [ I am initially copying only a few lists since they seem like
>>   the most impacted projects and I didn't want to spam all the mailing
>>   lists.  Please let me know if other lists should be included. ]
>>
>> I submitted D68230 for review but this is not about that patch per se.
>> The patch allows update_cc_test_checks.py to process tests that should
>> check target asm rather than LLVM IR.  We use this facility downstream
>> for our end-to-end tests.  It strikes me that it might be useful for
>> upstream to do similar end-to-end testing.
>>
>> Now that the monorepo is about to become the canonical source of truth,
>> we have an opportunity for convenient end-to-end testing that we didn't
>> easily have before with svn (yes, it could be done but in an ugly way).
>> AFAIK the only upstream end-to-end testing we have is in test-suite and
>> many of those codes are very large and/or unfocused tests.
>>
>> With the monorepo we have a place to put lit-style tests that exercise
>> multiple subprojects, for example tests that ensure the entire clang
>> compilation pipeline executes correctly.
>
>
I don't think I agree with the relationship to the monorepo: there was
nothing that prevented tests inside the clang project to exercise the full
pipeline already. I don't believe that the SVN repo structure was really a
factor in the way the testing was setup, but instead it was a deliberate
choice in the way we structure our testing.
For instance I remember asking about implementing test based on checking if
some loops written in C source file were properly vectorized by the -O2 /
-O3 pipeline and it was deemed like the kind of test that we don't want to
maintain: instead I was pointed at the test-suite to add better benchmarks
there for the end-to-end story. What is interesting is that the test-suite
is not gonna be part of the monorepo!

To be clear: I'm not saying here we can't change our way of testing, I just
don't think the monorepo has anything to do with it and that it should
carefully motivated and scoped into what belongs/doesn't belong to
integration tests.

> We could, for example, create
>> a top-level "test" directory and put end-to-end tests there.  Some of
>> the things that could be tested include:
>>
>> - Pipeline execution (debug-pass=Executions)
>>
> - Optimization warnings/messages
>> - Specific asm code sequences out of clang (e.g. ensure certain loops
>>   are vectorized)
>> - Pragma effects (e.g. ensure loop optimizations are honored)
>> - Complete end-to-end PGO (generate a profile and re-compile)
>> - GPU/accelerator offloading
>> - Debuggability of clang-generated code
>>
>> Each of these things is tested to some degree within their own
>> subprojects, but AFAIK there are currently no dedicated tests ensuring
>> such things work through the entire clang pipeline flow and with other
>> tools that make use of the results (debuggers, etc.).  It is relatively
>> easy to break the pipeline while the individual subproject tests
>> continue to pass.
>>
>

I'm not sure I really see much in your list that isn't purely about testing
clang itself here?
Actually the first one seems more of a pure LLVM test.

> I realize that some folks prefer to work on only a portion of the
>> monorepo (for example, they just hack on LLVM).  I am not sure how to
>> address those developers WRT end-to-end testing.  On the one hand,
>> requiring them to run end-to-end testing means they will have to at
>> least check out and build the monorepo.  On the other hand, it seems
>> less than ideal to have people developing core infrastructure and not
>> running tests.
>>
>
I think we already expect LLVM developers to update clang APIs? And we
revert LLVM patches when clang testing is broken. So I believe the
acknowledgment to maintain the other in-tree projects isn't really new, it
is true that the monorepo will make this easy for everyone to reproduce
locally most failure, and find all the use of an API across projects (which
was provided as a motivation to move to a monorepo model:
https://llvm.org/docs/Proposals/GitHubMove.html#monorepo ).

-- 
Mehdi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191008/55b4c71f/attachment.html>