[llvm-dev] [cfe-dev] RFC: End-to-end testing

[David Greene via llvm-dev]

Florian Hahn via cfe-dev <cfe-dev at lists.llvm.org> writes:

> Have you considered alternatives to checking the assembly for ensuring
> vectorization or other transformations? For example, instead of
> checking the assembly, we could check LLVM’s statistics or
> optimization remarks.

Yes, absolutely.  We have tests that do things like that.  I don't want
to focus on the asm bit, that's just one type of test.  The larger issue
is end-to-end tests that ensure the compiler and other tools are working
correctly, be it from checking messages, statistics, asm or something
else.

> This idea of leveraging statistics and optimization remarks to track
> the impact of changes on overall optimization results is nothing new
> and I think several people already discussed it in various forms. For
> regular benchmark runs, in addition to tracking the existing
> benchmarks, we could also track selected optimization remarks
> (e.g. loop-vectorize, but not necessarily noisy ones like gvn) and
> statistics. Comparing those run-to-run could potentially highlight new
> end-to-end issues on a much larger scale, across all existing
> benchmarks integrated in test-suite. We might be able to detect loss
> in vectorization pro-actively, instead of requiring someone to file a
> bug report and then we add an isolated test after the fact.

That's an interesting idea!  I would love to get more use out of
test-suite.

                       -David