[llvm-dev] [cfe-dev] RFC: End-to-end testing
David Blaikie via llvm-dev
llvm-dev at lists.llvm.org
Tue Oct 8 12:22:39 PDT 2019
I have a bit of concern about this sort of thing - worrying it'll lead to
people being less cautious about writing the more isolated tests. That
said, clearly there's value in end-to-end testing for all the reasons
you've mentioned (& we do see these problems in practice - recently DWARF
indexing broke when support for more nuanced language codes were added to
Dunno if they need a new place or should just be more stuff in test-suite,
On Tue, Oct 8, 2019 at 9:50 AM David Greene via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
> [ I am initially copying only a few lists since they seem like
> the most impacted projects and I didn't want to spam all the mailing
> lists. Please let me know if other lists should be included. ]
> I submitted D68230 for review but this is not about that patch per se.
> The patch allows update_cc_test_checks.py to process tests that should
> check target asm rather than LLVM IR. We use this facility downstream
> for our end-to-end tests. It strikes me that it might be useful for
> upstream to do similar end-to-end testing.
> Now that the monorepo is about to become the canonical source of truth,
> we have an opportunity for convenient end-to-end testing that we didn't
> easily have before with svn (yes, it could be done but in an ugly way).
> AFAIK the only upstream end-to-end testing we have is in test-suite and
> many of those codes are very large and/or unfocused tests.
> With the monorepo we have a place to put lit-style tests that exercise
> multiple subprojects, for example tests that ensure the entire clang
> compilation pipeline executes correctly. We could, for example, create
> a top-level "test" directory and put end-to-end tests there. Some of
> the things that could be tested include:
> - Pipeline execution (debug-pass=Executions)
> - Optimization warnings/messages
> - Specific asm code sequences out of clang (e.g. ensure certain loops
> are vectorized)
> - Pragma effects (e.g. ensure loop optimizations are honored)
> - Complete end-to-end PGO (generate a profile and re-compile)
> - GPU/accelerator offloading
> - Debuggability of clang-generated code
> Each of these things is tested to some degree within their own
> subprojects, but AFAIK there are currently no dedicated tests ensuring
> such things work through the entire clang pipeline flow and with other
> tools that make use of the results (debuggers, etc.). It is relatively
> easy to break the pipeline while the individual subproject tests
> continue to pass.
> I realize that some folks prefer to work on only a portion of the
> monorepo (for example, they just hack on LLVM). I am not sure how to
> address those developers WRT end-to-end testing. On the one hand,
> requiring them to run end-to-end testing means they will have to at
> least check out and build the monorepo. On the other hand, it seems
> less than ideal to have people developing core infrastructure and not
> running tests.
> I don't yet have a formal proposal but wanted to put this out to spur
> discussion and gather feedback and ideas. Thank you for your interest
> and participation!
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the llvm-dev