[llvm-dev] [Openmp-dev] [cfe-dev] RFC: End-to-end testing

[David Greene via llvm-dev]

Renato Golin via Openmp-dev <openmp-dev at lists.llvm.org> writes:

> But if we have some consensus on doing a clean job, then I would
> actually like to have that kind of intermediary check (diagnostics,
> warnings, etc) on most test-suite tests, which would cover at least
> the main vectorisation issues. Later, we could add more analysis
> tools, if we want.

I think this makes a lot of sense.

> It would be as simple as adding CHECK lines on the execution of the
> compilation process (in CMake? Make? wrapper?) and keep the check
> files with the tests / per file.

Yep.

> I think we're on the same page regarding almost everything, but
> perhaps I haven't been clear enough on the main point, which I think
> it's pretty simple. :)

Personally, I still find source-to-asm tests to be highly valuable and I
don't think we need test-suite for that.  Such tests don't (usually)
depend on system libraries (headers may occasionally be an issue but I
would argue that the test is too fragile in that case).

So maybe we separate concerns.  Use test-suite to do the kind of
system-level testing you've discussed but still allow some tests in a
monorepo top-level directory that test across components but don't
depend on system configurations.

If people really object to a top-level monorepo test directory I guess
they could go into test-suite but that makes it much more cumbersome to
run what really should be very simple tests.

                   -David