<div dir="ltr">I have a bit of concern about this sort of thing - worrying it'll lead to people being less cautious about writing the more isolated tests. That said, clearly there's value in end-to-end testing for all the reasons you've mentioned (& we do see these problems in practice - recently DWARF indexing broke when support for more nuanced language codes were added to Clang).<br><br>Dunno if they need a new place or should just be more stuff in test-suite, though.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Oct 8, 2019 at 9:50 AM David Greene via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">[ I am initially copying only a few lists since they seem like<br>
the most impacted projects and I didn't want to spam all the mailing<br>
lists. Please let me know if other lists should be included. ]<br>
<br>
I submitted D68230 for review but this is not about that patch per se.<br>
The patch allows update_cc_test_checks.py to process tests that should<br>
check target asm rather than LLVM IR. We use this facility downstream<br>
for our end-to-end tests. It strikes me that it might be useful for<br>
upstream to do similar end-to-end testing.<br>
<br>
Now that the monorepo is about to become the canonical source of truth,<br>
we have an opportunity for convenient end-to-end testing that we didn't<br>
easily have before with svn (yes, it could be done but in an ugly way).<br>
AFAIK the only upstream end-to-end testing we have is in test-suite and<br>
many of those codes are very large and/or unfocused tests.<br>
<br>
With the monorepo we have a place to put lit-style tests that exercise<br>
multiple subprojects, for example tests that ensure the entire clang<br>
compilation pipeline executes correctly. We could, for example, create<br>
a top-level "test" directory and put end-to-end tests there. Some of<br>
the things that could be tested include:<br>
<br>
- Pipeline execution (debug-pass=Executions)<br>
- Optimization warnings/messages<br>
- Specific asm code sequences out of clang (e.g. ensure certain loops<br>
are vectorized)<br>
- Pragma effects (e.g. ensure loop optimizations are honored)<br>
- Complete end-to-end PGO (generate a profile and re-compile)<br>
- GPU/accelerator offloading<br>
- Debuggability of clang-generated code<br>
<br>
Each of these things is tested to some degree within their own<br>
subprojects, but AFAIK there are currently no dedicated tests ensuring<br>
such things work through the entire clang pipeline flow and with other<br>
tools that make use of the results (debuggers, etc.). It is relatively<br>
easy to break the pipeline while the individual subproject tests<br>
continue to pass.<br>
<br>
I realize that some folks prefer to work on only a portion of the<br>
monorepo (for example, they just hack on LLVM). I am not sure how to<br>
address those developers WRT end-to-end testing. On the one hand,<br>
requiring them to run end-to-end testing means they will have to at<br>
least check out and build the monorepo. On the other hand, it seems<br>
less than ideal to have people developing core infrastructure and not<br>
running tests.<br>
<br>
I don't yet have a formal proposal but wanted to put this out to spur<br>
discussion and gather feedback and ideas. Thank you for your interest<br>
and participation!<br>
<br>
-David<br>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
</blockquote></div>