[llvm-dev] [cfe-dev] [Openmp-dev] RFC: End-to-end testing

[Mehdi AMINI via llvm-dev]

On Wed, Oct 16, 2019 at 12:54 PM David Greene via cfe-dev <
cfe-dev at lists.llvm.org> wrote:

> Renato Golin via Openmp-dev <openmp-dev at lists.llvm.org> writes:
>
> > But if we have some consensus on doing a clean job, then I would
> > actually like to have that kind of intermediary check (diagnostics,
> > warnings, etc) on most test-suite tests, which would cover at least
> > the main vectorisation issues. Later, we could add more analysis
> > tools, if we want.
>
> I think this makes a lot of sense.
>
> > It would be as simple as adding CHECK lines on the execution of the
> > compilation process (in CMake? Make? wrapper?) and keep the check
> > files with the tests / per file.
>
> Yep.
>
> > I think we're on the same page regarding almost everything, but
> > perhaps I haven't been clear enough on the main point, which I think
> > it's pretty simple. :)
>
> Personally, I still find source-to-asm tests to be highly valuable and I
> don't think we need test-suite for that.  Such tests don't (usually)
> depend on system libraries (headers may occasionally be an issue but I
> would argue that the test is too fragile in that case).
>
> So maybe we separate concerns.  Use test-suite to do the kind of
> system-level testing you've discussed but still allow some tests in a
> monorepo top-level directory that test across components but don't
> depend on system configurations.
>
> If people really object to a top-level monorepo test directory I guess
> they could go into test-suite but that makes it much more cumbersome to
> run what really should be very simple tests.


The main thing I see that will justify push-back on such test is the
maintenance: you need to convince everyone that every component in LLVM
must also maintain (update, fix, etc.) the tests that are in other
components (clang, flang, other future subproject, etc.). Changing the
vectorizer in the middle-end may require now to understand the kind of
update a test written in Fortran (or Haskell?) is checking with some
Hexagon assembly. This is a non-trivial burden when you compute the full
matrix of possible frontend and backends.
Even if you write very small tests for checking vectorization, what is
next? What about unrolling, inlining, loop-fusion, etc. ? Why would we stop
the end-to-end FileCheck testing to vectorization?

So the monorepo vs the test-suite seems like a false dichotomy: if such
tests don't make it in the monorepo it will be (I believe) because folks
won't want to maintain them. Putting them "elsewhere" is fine but it does
not solve the question of the maintenance of the tests.

-- 
Mehdi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191016/a5331739/attachment.html>