[llvm-dev] [cfe-dev] [Openmp-dev] RFC: End-to-end testing

[David Greene via llvm-dev]

Mehdi AMINI via llvm-dev <llvm-dev at lists.llvm.org> writes:

> The main thing I see that will justify push-back on such test is the
> maintenance: you need to convince everyone that every component in LLVM
> must also maintain (update, fix, etc.) the tests that are in other
> components (clang, flang, other future subproject, etc.). Changing the
> vectorizer in the middle-end may require now to understand the kind of
> update a test written in Fortran (or Haskell?) is checking with some
> Hexagon assembly. This is a non-trivial burden when you compute the full
> matrix of possible frontend and backends.

That's true.  But don't we want to make sure the complete compiler works
as expected?  And don't we want to be alerted as soon as possible if
something breaks?  To my knowledge we have very few end-to-end tests of
the type I've been thinking about.  That worries me.

> Even if you write very small tests for checking vectorization, what is
> next? What about unrolling, inlining, loop-fusion, etc. ? Why would we stop
> the end-to-end FileCheck testing to vectorization?

I actually think vectorization is probably lower on the concern list for
end-to-end testing than more focused things like FMA generation,
prefetching and so on.  This is because there isn't a lot after the
vectorization pass that can be mess up vectorization.  Once something is
vectorized, it is likely to stay vectorized.  On the other hand, I have
for example frequently seen prefetches dropped or poorly scheduled by
code long after the prefetch got inserted into the IR.

> So the monorepo vs the test-suite seems like a false dichotomy: if such
> tests don't make it in the monorepo it will be (I believe) because folks
> won't want to maintain them. Putting them "elsewhere" is fine but it does
> not solve the question of the maintenance of the tests.

Agree 100%.

                      -David