[LLVMdev] Test Suite - Livermore Loops

[Renato Golin]

On 7 January 2013 15:32, Michael Gottesman <mgottesman at apple.com> wrote:

> We are testing LTO internally and have not run into this issue IIRC. But
> on the other hand, we are doing a straight compilation (I.e. not doing it
> in parts as you said you were). Even so, you are right, we should have
> public lnt LTO testers.
>


My opinion is that tests should be intentional. If you spot a difference
between two calls, you either start testing both explicitly or ignore one
of them. Relying on side-effects for testing, in the majority of cases,
increase the perceived importance of small matters and takes away time to
fix real bugs.

If LTO is important (I think it is), then we should have explicit LTO
tests. If testing the order of passes is important, we should consistently
test it on all important configurations we have (ex. using bugpoint).
Keeping an old testing style *just* to have the side-effect of testing LTO
leads to confusion and noise which is worse in the end.

Random tests are one way of achieving a huge vector space in a fair way.
The hard bits is to know what to ignore (ie. it'll never happen in real
world) against the real bugs, that need fixing, or the real bugs that have
very little importance, etc. But all that should only be pursued when all
the other proper tests are set up and giving meaningful results.

cheers,
--renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130107/8778aba1/attachment.html>