[cfe-dev] [analyzer] Summary IPA thoughts

[Anna Zaks via cfe-dev]

> On Oct 21, 2015, at 10:02 AM, Artem Dergachev via cfe-dev <cfe-dev at lists.llvm.org> wrote:
> 
>> So this implementation doesn't give the performance gain
>> you were expecting, but what kind of performance gain
>> does it give?
> 
> It varies, some things better, some things worse, not exactly a dramatic improvement (yet, hopefully), especially considering the complexity trade-off. We hope it gets better while we keep debugging and/or porting it further.
> 
> While good metrics are actually hard to find, the following has been observed in our test runs:
> 
>  -1-  We can obtain ~15% more bug reports with similar analyzer options (eg. -analyzer-config max-nodes), but it takes ~twice more time and ~five times more memory.
>  -2-  We can find a similar number of bugs 2-3 times faster, with similar memory limitations, by reducing the max-nodes option; in this case, ~15% of the bugs do not overlap.
>  -3-  The true positive rate remains similar to inlining, newly obtained reports aren't like all incorrect or something like that.
> 
> In fact, we should be consuming more resources with similar max-nodes in our approach, because summary nodes are heavier and harder to obtain, and yeah, storing exploded graphs doesn't help either, so it's reasonable to reduce this parameter by default. So, eg., "-2-" was obtained when running summary with max-nodes=16000 and inlining with max-nodes=128000 (the default is 150000). However, by reducing max-nodes, we also reduce our possibilities to analyse functions with few IPA calls; and when we try to avoid this, we inevitably fail on memory improvement. So it's hard to obtain truly accurate comparison.
> 
> Our typical test run assumes analysis of the whole Android open-source project repo, and gathers around ~2000 warnings of two proof-of-concept checkers (these checkers usually don't find critical bugs, but they're noisy enough for statistical speculations); we only had time to evaluate a hundred of them by hand for each checker, more-or-less uniformly.

You can also measure the number of basic blocks visited if you want to compare coverage. Take a look at debug.Stats checker and -analyzer-stats option. These might give you a better understanding of what causes the slowdown.

We use various heuristics for balancing precision and performance for inlining. We do not actually inline all functions. This is why you see more reports with your summaries. Theoretically, inlining should always be more precise than any summary. There are several parameters controlling this apart from max-nodes:
 - We do not inline functions that are too large -analyzer-config max-inclinable-size (50 basic blocks).
 - We do not inline large functions(min-cfg-size-treat-functions-as-large) too many times (max-times-inline-large).

Anna.

> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev