[llvm-dev] [Openmp-dev] [cfe-dev] RFC: End-to-end testing

[David Blaikie via llvm-dev]

On Tue, Oct 8, 2019 at 12:46 PM David Greene <dag at cray.com> wrote:

> David Blaikie via Openmp-dev <openmp-dev at lists.llvm.org> writes:
>
> > I have a bit of concern about this sort of thing - worrying it'll lead to
> > people being less cautious about writing the more isolated tests.
>
> That's a fair concern.  Reviewers will still need to insist on small
> component-level tests to go along with patches.  We don't have to
> sacrifice one to get the other.
>
> > Dunno if they need a new place or should just be more stuff in
> test-suite,
> > though.
>
> There are at least two problems I see with using test-suite for this:
>
> - It is a separate repository and thus is not as convenient as tests
>   that live with the code.  One cannot commit an end-to-end test
>   atomically with the change meant to be tested.
>
> - It is full of large codes which is not the kind of testing I'm talking
>   about.
>

Oh, right - I'd forgotten that the test-suite wasn't part of the monorepo
(due to size, I can understand why) - fair enough. Makes sense to me to
have lit-style lightweight, targeted, but intentionally end-to-end tests.


>
> Let me describe how I recently added some testing in our downstream
> fork.
>
> - I implemented a new feature along with a C source test.
>
> - I used clang to generate asm from that test and captured the small
>   piece of it I wanted to check in an end-to-end test.
>
> - I used clang to generate IR just before the feature kicked in and
>   created an opt-style test for it.  Generating this IR is not always
>   straightfoward and it would be great to have better tools to do this,
>   but that's another discussion.
>
> - I took the IR out of opt (after running my feature) and created an
>   llc-style test out of it to check the generated asm.  The checks are
>   the same as in the original C end-to-end test.
>
> So the tests are checking at each stage that the expected input is
> generating the expected output and the end-to-end test checks that we go
> from source to asm correctly.
>
> These are all really small tests, easily runnable as part of the normal
> "make check" process.
>
>                      -David
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191008/96ec0836/attachment.html>