[llvm-dev] [cfe-dev] [Openmp-dev] RFC: End-to-end testing

[David Greene via llvm-dev]

David Blaikie <dblaikie at gmail.com> writes:

> On Wed, Oct 16, 2019 at 1:09 PM Roman Lebedev via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
>> FWIW I'm personally cautiously non-optimistic about this,
>> but maybe i'm just not seeing the whole picture of the proposal.
>>
>> Both checking final asm, and checking more than one layer of abstraction
>> feels overreaching and very prone to breakage/too restrictful.
>> Even minimal changes to the scheduling for particular CPU can cause many
>> instructions to reorder.
>> I'm not sure what effect that will have on middle-end pass development,
>> too.
>>
>> A change affects these end-to-end tests, what then?
>> Just blindly regenerate every affected test?
>> This will be further complicated once clang isn't the only upstream
>> front-end.
>>
>
> Agreed that the broader a test is, the more careful one has to be about
> making it reliable in spite of other changes - sometimes that's really
> difficult (if you're trying to get a particular instruction selection or
> register allocation) but in others it can be fairly reliable if done
> carefully to sufficiently restrict optimizations, etc. (having function
> calls to external functions to act as sinks/sources for values, etc, for
> instance - picking places where the output is already "optimal" and
> trivially/obviously so (for whatever set of constraints you've provided -
> not heroic optimizations, etc) to ensure that it's fairly stable)

Right.  That is our experience.  The kinds of asm tests we have test
things that are quite stable.  For example, did we generate a prefetch
in this loop?  For more detailed things a component-level test is likely
more appropriate.

                      -David