[cfe-dev] [analyzer][RFC] Test all checkers on all tests

[Ilya Palachev via cfe-dev]

Hi,

There is an idea for CSA testing. Currently each checker has it own test 
file(s) on which it is tested. Each checker has its own development 
history, and many of them were tested on code bases highly different 
from each other. That's why one checker may crash (i.e. produce 
assertion fail or UNREACHABLE) for code base on which it was never tested.

We tried to launch all checkers on all tests, by replacing `-verify' 
flags with `-analyzer-checker=<list of all packages>' through the 
lit.local.cfg (using config.substitutions.append method). A lot of 
crashes are obtained on our version of CSA (which is somehow different 
from upstream, however).

Moreover, we even tried Sema/SemaCXX tests and obtained several more 
crashes. Test files usually describe some corner cases, so this method 
will allow to test the analyzer on them.

I believe that such testing mode can be useful to improve the analyzer 
stability. Does anybody have any ideas, would it be appropriate? Should 
we add this mode to the default make check, or create a separate target 
for it?

--
Best regards,
Ilya Palachev