[cfe-dev] Proposal: Integrate static analysis test suites

Sun Jan 31 22:08:31 PST 2016

>
> Whole-program analysis *is* a reasonably compelling reason for a
> separate tool, however I don't think it should drive the design for
> the user interface. For instance, Visual Studio does not require
> execution of a separate tool to enable their static analysis (which I
> believe does whole-program analysis).
>

I believe that VS /analyze does do some whole program analysis, but I can't
actually find anywhere that Microsoft says so.

I think it's largely dependent on annotations, and slide 23 of this
presentation
<https://github.com/ariccio/SALExamples/raw/master/5_StaticAnalysisPREfast.pdf>
seems to agree with that.

As a sidenote, I'm guessing that whole program analysis will need to distil
information about individual functions into something conceptually similar
to SAL (albeit machine, not human readable) so that the analyzer only needs
to move "abstract behavior models" (or whatever) across translation units,
instead of entire functions.

So, for instance, how do we intend for clang-cl to support the /analyze
> option?
>

I feel like I raised this issue somewhere else, but it seems like clang-cl
just isn't drop-in compatible with /analyze.

When it comes to diagnostics, the easier we can make it on the user to
> enable the functionality, the greater the chance of users actually using
> it.

This. It's like security: if it's not on by default, very few people will
even try to turn it on. If it's hard to turn on, nobody will.

The CodeChecker team is working on a solution for that and I hope we
can incorporate
> their technology in LLVM/clang.

That *does* sound fantastic! I should look into it.

Sincerely,
Alexander Riccio
--
"Change the world or go home."
about.me/ariccio

<http://about.me/ariccio>
If left to my own devices, I will build more.
⁂

On Sat, Jan 30, 2016 at 9:55 AM, Aaron Ballman <aaron at aaronballman.com>
wrote:

> On Fri, Jan 29, 2016 at 11:32 PM, Anna Zaks <ganna at apple.com> wrote:
> >
> > By calling "$clang —analyze” you are not calling the compiler and asking
> it
> > to work harder. You are calling another tool that is not going to compile
> > for you but rather provide deep static code analysis.
>
> This is not unlike the way clang-tidy works (which also runs the
> analyzer!), but clang-tidy still shows compiler diagnostics.
>
> > Calling "clang
> > —analyze" could call the compiler behind the scenes and report the
> compiler
> > warnings in addition to the static analyzer issues. However, when
> warnings
> > from both tools are merged in a straightforward way on command line, the
> > user experience could be confusing. For example, both tools report some
> > issues such as warning on code like this:
> >   int j = 5/0; // warning: Division by zero
> >                    // warning: division by zero is undefined
> > [-Wdivision-by-zero]
>
> This is unfortunate, but to me it shows that we have duplication of
> efforts in our tools. We run into the same general issue with
> clang-tidy checks and the compiler, but the goal is to find one home
> for that diagnostic functionality and only enable it there. If we have
> diagnostics that live in both the compiler and the analyzer, we're
> duplicating effort and we should strive to rectify that where
> possible. There's likely to be cases where this is harder (such as
> division by zero) because you want the diagnostic enabled by default
> without requiring the overhead of running path-sensitive checks, but I
> think there are ways we can manage that.
>
> > Most importantly, end users should never invoke the analyzer by calling
> > “clang —analyze” since “clang —analyze” is an implementation detail of
> the
> > static analyzer. The only documented user facing clang static analysis
> tool
> > is scan-build (see http://clang-analyzer.llvm.org). Here are some
> reasons
> > for that. For one, it is almost impossible to understand why the static
> > analyzer warns without examining the error paths. Second, the analyzer
> could
> > be extended to perform whole project analysis in the future and "clang
> > —analyze" works with a single TU at a time.
>
> As a counter-example to requiring examining the code paths, the
> compiler has thread role analysis diagnostics (among others) that are
> also flow-sensitive and it's never been an issue that users must
> examine the error paths, so I'm not certain that's a particularly
> compelling reason to require a separate tool. Even templates and
> macros require a lot of "path" archaeology, and we've found some
> excellent ways to surface that from the compiler.
>
> Whole-program analysis *is* a reasonably compelling reason for a
> separate tool, however I don't think it should drive the design for
> the user interface. For instance, Visual Studio does not require
> execution of a separate tool to enable their static analysis (which I
> believe does whole-program analysis). So, for instance, how do we
> intend for clang-cl to support the /analyze option? Since we don't
> have whole-program analysis currently, it seems like such a feature
> could be designed to operate from a compiler flag (for instance, in
> conjunction with compilation databases) that is responsible for
> spawning off that secondary tool when required. (Note, I think a
> similar approach could be used to support running clang-tidy from the
> compiler via a command line flag.)
>
> > I agree that the best user experience is to report all warnings in one
> > place, while still differentiating which warning was reported by which
> tool.
> > It would be awesome if the results from all bug finding tools such as the
> > clang static analyzer, the compiler, and clang-tidy would be reported
> > through the same interface.
>
> I think we are in agreement, but to verify what I think we're agreeing
> on: users don't particularly care about the *tool* used nearly so much
> as they care about getting the diagnostics themselves. (For instance,
> users don't care if it's a parser error, a semantic error, a
> path-sensitive error, etc.) When it comes to diagnostics, the easier
> we can make it on the user to enable the functionality, the greater
> the chance of users actually using it. Based on that, having a single
> mechanism the user can invoke to give them diagnostics (such as the
> clang driver itself) is something we should strive towards, even if
> that means executing different libraries or executables under the hood
> (like we do with cc1). Obviously, *reporting* all the diagnostics in a
> single place falls naturally out of invocation of a single tool. Does
> that agree with what you were saying, or am I misinterpreting?
>
> > The CodeChecker team is working on a solution for that and I hope we can
> > incorporate their technology in LLVM/clang.
>
> That's fantastic! Thank you for the explanations, as well as all the
> hard work on this tool.
>
> ~Aaron
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20160201/887c4858/attachment.html>