[llvm-dev] [cfe-dev] RFC: End-to-end testing

[David Greene via llvm-dev]

Sean Silva via cfe-dev <cfe-dev at lists.llvm.org> writes:

>> We have to support many different systems and those systems are always
>> changing (new processors, new BIOS, new OS, etc.).  Performance can vary
>> widely day to day from factors completely outside the compiler's
>> control.  As the performance changes you have to keep updating the tests
>> to expect the new performance numbers.  Relying on performance
>> measurements to ensure something like vectorization is happening just
>> isn't reliable in our experience.
>
> Could you compare performance with vectorization turned on and off?

That might catch more things but now you're running tests twice and it
still won't catch some cases.

                        -David