[cfe-dev] [analyzer][RFC] Test all checkers on all tests

[Artem Dergachev via cfe-dev]

Tried this. Found a couple of "Check name is not set properly." 
assertion failures! Namely:


   $ bin/clang -cc1 -internal-isystem lib/clang/7.0.0/include 
-nostdsysteminc -analyze 
-analyzer-checker=core,unix,osx,cplusplus,optin,nullability,valist,llvm 
-analyzer-checker=unix.cstring.BadSizeArg -analyzer-store=region 
-Wno-strncat-size -Wno-strlcpy-strlcat-size -Wno-sizeof-array-argument 
-Wno-sizeof-pointer-memaccess -verify 
../llvm/tools/clang/test/Analysis/cstring-syntax.c

   $ bin/clang -cc1 -internal-isystem lib/clang/7.0.0/include 
-nostdsysteminc -analyze 
-analyzer-checker=core,unix,osx,cplusplus,optin,nullability,valist,llvm 
-triple i386-apple-darwin10 
-analyzer-checker=security.insecureAPI,security.FloatLoopCounter 
../llvm/tools/clang/test/Analysis/security-syntax-checks.m -verify


That was it though, not much crashes on the whole Clang's test suite. I 
guess those guys who send us crashes from the GCC test suite did a 
really good job^^ (eg. https://reviews.llvm.org/D39800). Also i didn't 
enable alpha checkers.

I'd love to have this sort of stress test when i'm working with the AST 
that i don't fully understand. I guess it's a good idea for a buildbot 
(eg. add to 
http://green.lab.llvm.org/green/job/StaticAnalyzerBenchmarks/ or make a 
new one?).


On 18/01/2018 7:55 AM, Gábor Horváth wrote:
> Hi!
>
> I do like the idea of having a target for this but I would not make it 
> part of the make check target. I think it would be a measurable 
> overhead for the runtime of the tests and I am also not sure that we 
> can/want to make that target clean. Maybe in some cases, the return on 
> investment would be too low and we simply have other priorities or the 
> side effect.
>
> Regards,
> Gábor
>
> On 18 January 2018 at 13:59, Ilya Palachev <i.palachev at samsung.com 
> <mailto:i.palachev at samsung.com>> wrote:
>
>     Hi,
>
>     There is an idea for CSA testing. Currently each checker has it
>     own test file(s) on which it is tested. Each checker has its own
>     development history, and many of them were tested on code bases
>     highly different from each other. That's why one checker may crash
>     (i.e. produce assertion fail or UNREACHABLE) for code base on
>     which it was never tested.
>
>     We tried to launch all checkers on all tests, by replacing
>     `-verify' flags with `-analyzer-checker=<list of all packages>'
>     through the lit.local.cfg (using config.substitutions.append
>     method). A lot of crashes are obtained on our version of CSA
>     (which is somehow different from upstream, however).
>
>     Moreover, we even tried Sema/SemaCXX tests and obtained several
>     more crashes. Test files usually describe some corner cases, so
>     this method will allow to test the analyzer on them.
>
>     I believe that such testing mode can be useful to improve the
>     analyzer stability. Does anybody have any ideas, would it be
>     appropriate? Should we add this mode to the default make check, or
>     create a separate target for it?
>
>     --
>     Best regards,
>     Ilya Palachev
>
>
>
>
>
>
>