[llvm-dev] [cfe-dev] [Openmp-dev] RFC: End-to-end testing

[David Blaikie via llvm-dev]

On Wed, Oct 16, 2019 at 12:54 PM David Greene via cfe-dev <
cfe-dev at lists.llvm.org> wrote:

> Renato Golin via Openmp-dev <openmp-dev at lists.llvm.org> writes:
>
> > But if we have some consensus on doing a clean job, then I would
> > actually like to have that kind of intermediary check (diagnostics,
> > warnings, etc) on most test-suite tests, which would cover at least
> > the main vectorisation issues. Later, we could add more analysis
> > tools, if we want.
>
> I think this makes a lot of sense.
>
> > It would be as simple as adding CHECK lines on the execution of the
> > compilation process (in CMake? Make? wrapper?) and keep the check
> > files with the tests / per file.
>
> Yep.
>
> > I think we're on the same page regarding almost everything, but
> > perhaps I haven't been clear enough on the main point, which I think
> > it's pretty simple. :)
>
> Personally, I still find source-to-asm tests to be highly valuable and I
> don't think we need test-suite for that.  Such tests don't (usually)
> depend on system libraries (headers may occasionally be an issue but I
> would argue that the test is too fragile in that case).
>
> So maybe we separate concerns.  Use test-suite to do the kind of
> system-level testing you've discussed but still allow some tests in a
> monorepo top-level directory that test across components but don't
> depend on system configurations.
>

I'm inclined to the direction suggested by others that the monorepo is
orthogonal to this issue and top level tests might not be the right thing.

lldb already does end-to-end testing in its tests, for instance.

Clang does in some tests (the place I always hit is anything that's
configured API-wise on the MCContext - there's no way to test that
configuration on the clang boundary, so the only test that we can write is
one that tests the effect of that API/programmatic configuration done by
clang to the MCContext (function sections, for instance) - in some cases
I've just skipped the testing, in others I've written the end-to-end test
in clang (& an LLVM test for the functionality that uses llvm-mc or
similar)).


> If people really object to a top-level monorepo test directory I guess
> they could go into test-suite but that makes it much more cumbersome to
> run what really should be very simple tests.
>
>                    -David
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191016/bf3bded8/attachment.html>