[cfe-dev] [analyzer] Clang Static Analyzer and Clang Tidy regression tests

[Jonas Toth via cfe-dev]

Hey Alexander,

I did work towards a clang-tidy buildbot some time ago but unfortunatly
had to stop because of time constraints.

My setup was(still is) along these lines:

1. build latest clang-tidy as a docker image
2. make a project-image, compile the project as it wants and create the
"compile_commands.json"
3. run each check-category over the whole project
    - deduplicating diagnostics, as the output is massive for some
project/check combinations (~GBs for one run)
    - applying fixes
    - checking if the project still compiles after fixing
    - displaying `git diff` to be able to see what has been fixed and to
find potential breaking changes
    - optionally silencing some checks, like style-related stuff that is
just too noisy.

I did run this over ~10 Projects (blender, opencv, llvm, curl, ...)
privatly while developing. In principle this buildbot works, but I would
refine some parts,
especially moving towards the mono-repo. Improving how to access the
diagnostics and the diff would help as well, I guess. Its kinda
prototypish still.

If there is more interest I can publish my current work to github or so
and we can setup a buildbot. I would definitly contribute with a worker
machine,
and the docker-based approach should help setting up more workers for
more projects.
The only obstacle I see personally, is the massive output some projects
generate and how to make something useful out of it.

IMHO it should not run on the main buildbot we have, as for example
"does still compile" is rarely the case when fixing is activated
(unfortunatly), so the
easy ways to measure success are not feasable to use.

Best Regards, Jonas

Am 26.04.19 um 11:06 schrieb Alexander Zaitsev via cfe-dev:
>
> Hi!
>
> For Clang Static Analyzer (CSA) and for Clang Tidy there are a lot of
> unit-tests. That's great, but from my point of view running CSA and
> clang tidy *regularly *on some set of real life projects and comparing
> the results between runs is also important to check any regressions.
>
> Now I know that for CSA there are set of useful scripts in
> 'clang/utils/analyzer' but didn't find anything similar for clang-tidy.
>
> My questions are:
>
>  1. Are there any buildbots which test CSA and/or Clang Tidy on some
>     set of projects regularly? If there are not any buildbots - is it
>     possible to setup it?
>  2. I didn't find anything about regression testing in official
>     documentation. I think a note about regression testing has to be
>     added to the documentation: how it works now, how to it for your
>     code, etc. Now we have just small note about running unit-tests -
>     https://clang-analyzer.llvm.org/checker_dev_manual.html#testing
>
> Thank you.
>
> -- 
> Best regards,
> Alexander Zaitsev
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20190428/4b02c683/attachment.html>