[llvm-dev] [cfe-dev] RFC: End-to-end testing

[Renato Golin via llvm-dev]

On Thu, 10 Oct 2019 at 10:56, Florian Hahn <florian_hahn at apple.com> wrote:
> Have you considered alternatives to checking the assembly for ensuring vectorization or other transformations? For example, instead of checking the assembly, we could check LLVM’s statistics or optimization remarks. If you want to ensure a loop got vectorized, you could check the loop-vectorize remarks, which should give you the position of the loop in the source and vectorization/interleave factor used. There are few other things that could go wrong later on that would prevent vector instruction selection, but I think it should be sufficient to guard against most cases where we loose vectorization and should be much more robust to unrelated changes. If there are additional properties you want to ensure, they potentially could be added to the remark as well.

We used to have lots of them, at least in the initial implementation
of the loop vectoriser (I know, many years ago).

The thread has enough points not to repeat here, but I guess the main
point is to make sure we don't duplicate tests, increasing CI cost
(especially on slower hardware).

I'd recommend trying to move any e2e tests into the test-suite and
make it easier to run, and leave specific tests only in the repo (to
guarantee independence of components).

The last thing we want is to create direct paths from front-ends to
back-ends and make LLVM IR transformation less flexible.

cheers,
--renato