[llvm-dev] [Openmp-dev] [cfe-dev] RFC: End-to-end testing
David Greene via llvm-dev
llvm-dev at lists.llvm.org
Tue Oct 8 12:46:18 PDT 2019
David Blaikie via Openmp-dev <openmp-dev at lists.llvm.org> writes:
> I have a bit of concern about this sort of thing - worrying it'll lead to
> people being less cautious about writing the more isolated tests.
That's a fair concern. Reviewers will still need to insist on small
component-level tests to go along with patches. We don't have to
sacrifice one to get the other.
> Dunno if they need a new place or should just be more stuff in test-suite,
There are at least two problems I see with using test-suite for this:
- It is a separate repository and thus is not as convenient as tests
that live with the code. One cannot commit an end-to-end test
atomically with the change meant to be tested.
- It is full of large codes which is not the kind of testing I'm talking
Let me describe how I recently added some testing in our downstream
- I implemented a new feature along with a C source test.
- I used clang to generate asm from that test and captured the small
piece of it I wanted to check in an end-to-end test.
- I used clang to generate IR just before the feature kicked in and
created an opt-style test for it. Generating this IR is not always
straightfoward and it would be great to have better tools to do this,
but that's another discussion.
- I took the IR out of opt (after running my feature) and created an
llc-style test out of it to check the generated asm. The checks are
the same as in the original C end-to-end test.
So the tests are checking at each stage that the expected input is
generating the expected output and the end-to-end test checks that we go
from source to asm correctly.
These are all really small tests, easily runnable as part of the normal
"make check" process.
More information about the llvm-dev