[cfe-dev] [analyzer][RFC] Test all checkers on all tests

[Devin Coughlin via cfe-dev]

I think this is a great idea! I do think we’ll need to make sure not to run alpha checkers here since those often have assertion failures on purpose for some code constructs since they are work in progress.

I agree with Gabor that we should probably have this be a separate target — but we should have bots run it.

Devin

> On Jan 18, 2018, at 7:55 AM, Gábor Horváth <xazax.hun at gmail.com> wrote:
> 
> Hi!
> 
> I do like the idea of having a target for this but I would not make it part of the make check target. I think it would be a measurable overhead for the runtime of the tests and I am also not sure that we can/want to make that target clean. Maybe in some cases, the return on investment would be too low and we simply have other priorities or the side effect. 
> 
> Regards,
> Gábor
> 
> On 18 January 2018 at 13:59, Ilya Palachev <i.palachev at samsung.com <mailto:i.palachev at samsung.com>> wrote:
> Hi,
> 
> There is an idea for CSA testing. Currently each checker has it own test file(s) on which it is tested. Each checker has its own development history, and many of them were tested on code bases highly different from each other. That's why one checker may crash (i.e. produce assertion fail or UNREACHABLE) for code base on which it was never tested.
> 
> We tried to launch all checkers on all tests, by replacing `-verify' flags with `-analyzer-checker=<list of all packages>' through the lit.local.cfg (using config.substitutions.append method). A lot of crashes are obtained on our version of CSA (which is somehow different from upstream, however).
> 
> Moreover, we even tried Sema/SemaCXX tests and obtained several more crashes. Test files usually describe some corner cases, so this method will allow to test the analyzer on them.
> 
> I believe that such testing mode can be useful to improve the analyzer stability. Does anybody have any ideas, would it be appropriate? Should we add this mode to the default make check, or create a separate target for it?
> 
> --
> Best regards,
> Ilya Palachev
> 
> 
> 
> 
> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20180118/b32f99b3/attachment.html>