[cfe-dev] [analyzer][RFC] Test all checkers on all tests

[Gábor Horváth via cfe-dev]

Hi!

I do like the idea of having a target for this but I would not make it part
of the make check target. I think it would be a measurable overhead for the
runtime of the tests and I am also not sure that we can/want to make that
target clean. Maybe in some cases, the return on investment would be too
low and we simply have other priorities or the side effect.

Regards,
Gábor

On 18 January 2018 at 13:59, Ilya Palachev <i.palachev at samsung.com> wrote:

> Hi,
>
> There is an idea for CSA testing. Currently each checker has it own test
> file(s) on which it is tested. Each checker has its own development
> history, and many of them were tested on code bases highly different from
> each other. That's why one checker may crash (i.e. produce assertion fail
> or UNREACHABLE) for code base on which it was never tested.
>
> We tried to launch all checkers on all tests, by replacing `-verify' flags
> with `-analyzer-checker=<list of all packages>' through the lit.local.cfg
> (using config.substitutions.append method). A lot of crashes are obtained
> on our version of CSA (which is somehow different from upstream, however).
>
> Moreover, we even tried Sema/SemaCXX tests and obtained several more
> crashes. Test files usually describe some corner cases, so this method will
> allow to test the analyzer on them.
>
> I believe that such testing mode can be useful to improve the analyzer
> stability. Does anybody have any ideas, would it be appropriate? Should we
> add this mode to the default make check, or create a separate target for it?
>
> --
> Best regards,
> Ilya Palachev
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20180118/f8e573fe/attachment.html>