[llvm-dev] [Openmp-dev] [cfe-dev] RFC: End-to-end testing

[David Greene via llvm-dev]

David Blaikie via llvm-dev <llvm-dev at lists.llvm.org> writes:

> & I generally agree that end-to-end testing should be very limited - but
> there are already some end-to-end-ish tests in clang and I don't think
> they're entirely wrong there. I don't know much about the vectorization
> tests - but any test that requires a tool to maintain/generate makes me a
> bit skeptical and doubly-so if we were testing all of those end-to-end too.
> (I'd expect maybe one or two sample/example end-to-end tests, to test
> certain integration points, but exhaustive testing would usually be left to
> narrower tests (so if you have one subsystem with three codepaths {1, 2, 3}
> and another subsystem with 3 codepaths {A, B, C}, you don't test the full
> combination of {1, 2, 3} X {A, B, C} (9 tests), you test each set
> separately, and maybe one representative sample end-to-end (so you end up
> with maybe 7-8 tests))

That sounds reasonable.  End-to-end tests are probably going to be very
much a case-by-case thing.  I imagine we'd start with the component
tests as is done today and then if we see some failure in end-to-end
operation that isn't covered by the existing component tests we'd add an
end-to-end test.  Or maybe we create some new component tests to cover
it.

> Possible I know so little about the vectorization issues in particular that
> my thoughts on testing don't line up with the realities of that particular
> domain.

Vectorization is one only small part of what I imagine we'd want to test
in an end-to-end fashion.  There are lots of examples of "we want this
code generated" beyond vectorization.

                           -David