[cfe-dev] [analyzer] Summary IPA thoughts

[Artem Dergachev via cfe-dev]

> So this implementation doesn't give the performance gain
> you were expecting, but what kind of performance gain
> does it give?

It varies, some things better, some things worse, not exactly a dramatic 
improvement (yet, hopefully), especially considering the complexity 
trade-off. We hope it gets better while we keep debugging and/or porting 
it further.

While good metrics are actually hard to find, the following has been 
observed in our test runs:

   -1-  We can obtain ~15% more bug reports with similar analyzer 
options (eg. -analyzer-config max-nodes), but it takes ~twice more time 
and ~five times more memory.
   -2-  We can find a similar number of bugs 2-3 times faster, with 
similar memory limitations, by reducing the max-nodes option; in this 
case, ~15% of the bugs do not overlap.
   -3-  The true positive rate remains similar to inlining, newly 
obtained reports aren't like all incorrect or something like that.

In fact, we should be consuming more resources with similar max-nodes in 
our approach, because summary nodes are heavier and harder to obtain, 
and yeah, storing exploded graphs doesn't help either, so it's 
reasonable to reduce this parameter by default. So, eg., "-2-" was 
obtained when running summary with max-nodes=16000 and inlining with 
max-nodes=128000 (the default is 150000). However, by reducing 
max-nodes, we also reduce our possibilities to analyse functions with 
few IPA calls; and when we try to avoid this, we inevitably fail on 
memory improvement. So it's hard to obtain truly accurate comparison.

Our typical test run assumes analysis of the whole Android open-source 
project repo, and gathers around ~2000 warnings of two proof-of-concept 
checkers (these checkers usually don't find critical bugs, but they're 
noisy enough for statistical speculations); we only had time to evaluate 
a hundred of them by hand for each checker, more-or-less uniformly.