[cfe-dev] [Openmp-dev] [llvm-dev] RFC: End-to-end testing

[David Blaikie via cfe-dev]

On Wed, Oct 16, 2019 at 6:05 PM David Greene <dag at cray.com> wrote:

> > I'm inclined to the direction suggested by others that the monorepo is
> > orthogonal to this issue and top level tests might not be the right
> thing.
> >
> > lldb already does end-to-end testing in its tests, for instance.
> >
> > Clang does in some tests (the place I always hit is anything that's
> > configured API-wise on the MCContext - there's no way to test that
> > configuration on the clang boundary, so the only test that we can write
> is
> > one that tests the effect of that API/programmatic configuration done by
> > clang to the MCContext (function sections, for instance) - in some cases
> > I've just skipped the testing, in others I've written the end-to-end test
> > in clang (& an LLVM test for the functionality that uses llvm-mc or
> > similar)).
>
> I'd be totally happy putting such tests under clang.  This whole
> discussion was spurred by D68230 where some noted that previous
> discussion had determined we didn't want source-to-asm tests in clang
> and the test update script explicitly forbade it.
>
> If we're saying we want to reverse that decision, I'm very glad!
>

Unfortunately LLVM's community is by no means a monolith, so my opinion
here doesn't mean whoever expressed their opinion there has changed their
mind.

& I generally agree that end-to-end testing should be very limited - but
there are already some end-to-end-ish tests in clang and I don't think
they're entirely wrong there. I don't know much about the vectorization
tests - but any test that requires a tool to maintain/generate makes me a
bit skeptical and doubly-so if we were testing all of those end-to-end too.
(I'd expect maybe one or two sample/example end-to-end tests, to test
certain integration points, but exhaustive testing would usually be left to
narrower tests (so if you have one subsystem with three codepaths {1, 2, 3}
and another subsystem with 3 codepaths {A, B, C}, you don't test the full
combination of {1, 2, 3} X {A, B, C} (9 tests), you test each set
separately, and maybe one representative sample end-to-end (so you end up
with maybe 7-8 tests))

Possible I know so little about the vectorization issues in particular that
my thoughts on testing don't line up with the realities of that particular
domain.

- Dave


>
>                     -David
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20191016/5fe6e142/attachment.html>