[LLVMdev] opt and bugs

[Török Edwin]

On 2009-11-19 22:16, John Regehr wrote:
> A longstanding project on the LLVM "open projects" page is to run random 
> C code through random LLVM passes.
>
> We have found many LLVM bugs by doing the first of these, but not the 
> second: we test only -O[0123s].
>
> Slowly but surely, LLVM is evolving resistance to our random tester.  Of 
> course this is good, but it means that we need to keep expanding our 
> efforts if we want to continue to get solid return on investment for our 
> bugfinding hours.
>
> My first question is: When we start running random combinations of 
> passes using opt, which ones should we leave out, because you folks 
> don't want to hear that they're wrong?  For example, the other day I 
> learned that ABCD is not yet ready for this sort of testing.  A while 
> ago someone on the list said something similar about predsimplify 
> (though it looks like that pass has disappeared from the tree?).
>   

I think that if a pass is not part of -O[0123s] or -std-compile-opts, or
-std-link-opts
then it is not ready for random testing.

> Also I'd appreciate some advice about probabilities for each pass.  For 
> example, it seems like a lot of passes will have little to do if Mem2Reg 
> doesn't run, so perhaps that pass should get >0.5 probability of 
> running.  Does that make sense and do other examples come to mind?
>   

You should have loops and nested loops (in some tests), when you want to
test loop optimizers.
You should have some tests with EH, some without.

Also I think you should also try to test LLVM IR features that are less
often used:
byval, weak linkage, ssp, structs with unusual layout/alignment, vector
operations,
multiple return values, indirectbr, atomic ops, lifetime/invariant
annotations, etc.

Best regards,
--Edwin