[cfe-dev] Proposal: Integrate static analysis test suites

[Anna Zaks via cfe-dev]

> On Jan 29, 2016, at 9:12 PM, Sean Silva <chisophugis at gmail.com> wrote:
> 
> 
> 
> On Fri, Jan 29, 2016 at 8:32 PM, Anna Zaks via cfe-dev <cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>> wrote:
> 
>> On Jan 29, 2016, at 4:33 PM, Philip Reames <listmail at philipreames.com <mailto:listmail at philipreames.com>> wrote:
>> 
>> 
>> 
>> On 01/28/2016 05:53 AM, Aaron Ballman via cfe-dev wrote:
>>> On Thu, Jan 28, 2016 at 2:31 AM, Anna Zaks <ganna at apple.com <mailto:ganna at apple.com>> wrote:
>>> <snip..>
>>>> This is by design. Many more people have compiler as part of their daily
>>>> flow so it’s best to have such errors being reported by the compiler.
>>>> Having the analyzer produce all of the compiler warnings is likely to be too
>>>> nosy for the users.
>>> Personally, I find that design to lead to a confusing user experience.
>>> When I run the analyzer, my mental model is that I am running the
>>> compiler plus some additional analyses. When I don't get compiler
>>> warnings that I would otherwise get, it feels like I (as the user)
>>> have configured things improperly and done something wrong. Put
>>> another way: the point to running a static analyzer is to find out
>>> what's wrong with some code, so it's surprising that we would disable
>>> some of those notices of what's wrong that would otherwise be enabled
>>> by default.
>>> 
>>> Perhaps my mental model is in the minority, but it's another anecdote
>>> to remember if this design is ever reconsidered again.
>> I'd also find the current design slightly confusing.  I generally don't expect to see *fewer* warnings when I tell the compiler to work harder unless the original warning really was a false positive.
> 
> By calling "$clang —analyze” you are not calling the compiler and asking it to work harder. You are calling another tool that is not going to compile for you but rather provide deep static code analysis. Calling "clang —analyze" could call the compiler behind the scenes and report the compiler warnings in addition to the static analyzer issues. However, when warnings from both tools are merged in a straightforward way on command line, the user experience could be confusing. For example, both tools report some issues such as warning on code like this:
>   int j = 5/0; // warning: Division by zero
>                    // warning: division by zero is undefined [-Wdivision-by-zero]
> 
> Most importantly, end users should never invoke the analyzer by calling “clang —analyze” since “clang —analyze” is an implementation detail of the static analyzer. The only documented user facing clang static analysis tool is scan-build (see http://clang-analyzer.llvm.org <http://clang-analyzer.llvm.org/>).
> 
> --analyze is in `clang -help`.

I guess, I should have said “advertised and recommended”, not “documented”. A lot of low-level options are documented in 'clang -help'.

> Also, clang-check advertises a `-analyze` option which was clearly intentionally added.

Is it advertised in places other than help? Again, I do not think we can provide a good user experience for consuming the analyzer results without showing the paths, which 'clang-check' does not do.

> So it seems spurious to say that the only user-facing way to invoke the analyzer is scan-build.
> 
> In fact, anecdotally I seem to remember --analyze as being considered a user-facing option. I'm pretty sure that if I go digging back through old devmtg slides or whatnot I'll find a presentation recommending its use.

Our team has always been careful about not recommending end users to run 'clang —analyze' directly, mainly, because their workflow would stop working if the analyzer ever adds whole program analysis. (Of cause LLVM community knows about the clang option but they are not average end users.)

> 
> -- Sean Silva
>  
> Here are some reasons for that. For one, it is almost impossible to understand why the static analyzer warns without examining the error paths. Second, the analyzer could be extended to perform whole project analysis in the future and "clang —analyze" works with a single TU at a time.
> 
> I agree that the best user experience is to report all warnings in one place, while still differentiating which warning was reported by which tool. It would be awesome if the results from all bug finding tools such as the clang static analyzer, the compiler, and clang-tidy would be reported through the same interface.
> 
> The CodeChecker team is working on a solution for that and I hope we can incorporate their technology in LLVM/clang.
> 
>> 
>> Philip
> 
> 
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20160130/3c6b1a2c/attachment.html>