[llvm-dev] [Openmp-dev] [cfe-dev] RFC: End-to-end testing
David Blaikie via llvm-dev
llvm-dev at lists.llvm.org
Tue Oct 8 13:05:05 PDT 2019
On Tue, Oct 8, 2019 at 12:46 PM David Greene <dag at cray.com> wrote:
> David Blaikie via Openmp-dev <openmp-dev at lists.llvm.org> writes:
> > I have a bit of concern about this sort of thing - worrying it'll lead to
> > people being less cautious about writing the more isolated tests.
> That's a fair concern. Reviewers will still need to insist on small
> component-level tests to go along with patches. We don't have to
> sacrifice one to get the other.
> > Dunno if they need a new place or should just be more stuff in
> > though.
> There are at least two problems I see with using test-suite for this:
> - It is a separate repository and thus is not as convenient as tests
> that live with the code. One cannot commit an end-to-end test
> atomically with the change meant to be tested.
> - It is full of large codes which is not the kind of testing I'm talking
Oh, right - I'd forgotten that the test-suite wasn't part of the monorepo
(due to size, I can understand why) - fair enough. Makes sense to me to
have lit-style lightweight, targeted, but intentionally end-to-end tests.
> Let me describe how I recently added some testing in our downstream
> - I implemented a new feature along with a C source test.
> - I used clang to generate asm from that test and captured the small
> piece of it I wanted to check in an end-to-end test.
> - I used clang to generate IR just before the feature kicked in and
> created an opt-style test for it. Generating this IR is not always
> straightfoward and it would be great to have better tools to do this,
> but that's another discussion.
> - I took the IR out of opt (after running my feature) and created an
> llc-style test out of it to check the generated asm. The checks are
> the same as in the original C end-to-end test.
> So the tests are checking at each stage that the expected input is
> generating the expected output and the end-to-end test checks that we go
> from source to asm correctly.
> These are all really small tests, easily runnable as part of the normal
> "make check" process.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the llvm-dev