[llvm-dev] RFC: End-to-end testing
llvm-dev at lists.llvm.org
Tue Oct 8 11:51:22 PDT 2019
> -----Original Message-----
> From: cfe-dev <cfe-dev-bounces at lists.llvm.org> On Behalf Of David Greene
> via cfe-dev
> Sent: Tuesday, October 08, 2019 12:50 PM
> To: llvm-dev at lists.llvm.org; cfe-dev at lists.llvm.org; openmp-
> dev at lists.llvm.org; lldb-dev at lists.llvm.org
> Subject: [cfe-dev] RFC: End-to-end testing
> [ I am initially copying only a few lists since they seem like
> the most impacted projects and I didn't want to spam all the mailing
> lists. Please let me know if other lists should be included. ]
> I submitted D68230 for review but this is not about that patch per se.
> The patch allows update_cc_test_checks.py to process tests that should
> check target asm rather than LLVM IR. We use this facility downstream
> for our end-to-end tests. It strikes me that it might be useful for
> upstream to do similar end-to-end testing.
> Now that the monorepo is about to become the canonical source of truth,
> we have an opportunity for convenient end-to-end testing that we didn't
> easily have before with svn (yes, it could be done but in an ugly way).
> AFAIK the only upstream end-to-end testing we have is in test-suite and
> many of those codes are very large and/or unfocused tests.
> With the monorepo we have a place to put lit-style tests that exercise
> multiple subprojects, for example tests that ensure the entire clang
> compilation pipeline executes correctly. We could, for example, create
> a top-level "test" directory and put end-to-end tests there. Some of
> the things that could be tested include:
> - Pipeline execution (debug-pass=Executions)
> - Optimization warnings/messages
> - Specific asm code sequences out of clang (e.g. ensure certain loops
> are vectorized)
> - Pragma effects (e.g. ensure loop optimizations are honored)
> - Complete end-to-end PGO (generate a profile and re-compile)
> - GPU/accelerator offloading
> - Debuggability of clang-generated code
> Each of these things is tested to some degree within their own
> subprojects, but AFAIK there are currently no dedicated tests ensuring
> such things work through the entire clang pipeline flow and with other
> tools that make use of the results (debuggers, etc.). It is relatively
> easy to break the pipeline while the individual subproject tests
> continue to pass.
I agree with all your points. End-to-end testing is a major hole in the
project infrastructure; it has been largely left up to the individual
vendors/packagers/distributors. The Clang tests verify that Clang will
produce some sort of not-unreasonable IR for given situations; the LLVM
tests verify that some (other) set of input IR will produce something
that looks not-unreasonable on the target side. Very little connects
There is more than nothing:
- test-suite has some quantity of code that is compiled end-to-end for
- lldb can be set up to use the just-built Clang to compile its tests,
but those are focused on debug info and are nothing like comprehensive.
- libcxx likely also can use the just-built Clang, although I've never
tried it so I don't know for sure. It obviously exercises just the
runtime side of things.
- compiler-rt likewise. The sanitizer tests in particular I'd expect to
be using the just-built Clang.
- debuginfo-tests also uses the just-built Clang but is a pathetically
small set, and again focused on debug info.
I'm not saying the LLVM Project should invest in a commercial suite
(although I'd expect vendors to do so; Sony does). But a place to *put*
end-to-end tests seems entirely reasonable and useful. Although I would
resist calling it simply "tests" (we have too many directories with that
> I realize that some folks prefer to work on only a portion of the
> monorepo (for example, they just hack on LLVM). I am not sure how to
> address those developers WRT end-to-end testing. On the one hand,
> requiring them to run end-to-end testing means they will have to at
> least check out and build the monorepo. On the other hand, it seems
> less than ideal to have people developing core infrastructure and not
> running tests.
People should obviously be running the tests for the project(s) they're
modifying. People aren't expected to run everything. That's why...
Bots. "Don't argue with the bots." I don't checkout and build and test
everything, and I've broken LLDB, compiler-rt, and probably others from
time to time. Probably everybody has broken other projects unexpectedly.
That's what bots are for, to run the combinations and the projects that
I don't have the infrastructure or resources to do myself. It's not up
to me to run everything possible before committing; it IS up to me to
respond promptly to bot failures for my changes. I don't see a new
end-to-end test project being any different in that respect.
> I don't yet have a formal proposal but wanted to put this out to spur
> discussion and gather feedback and ideas. Thank you for your interest
> and participation!
Thanks for bringing it up! It's been a pebble in my shoe for a long time.
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
More information about the llvm-dev