[cfe-dev] ccc-analyzer progression/regression

[Anna Zaks]

On Sep 8, 2011, at 6:11 AM, John Smith wrote:

> Hi Anna,
> 
> 
> Thank you for your kind reply.
> 
> 
> 1.)
> Doing this automatically using a build-bot would of course be way more
> convenient and effective than doing things manually. I was unaware
> that you are currently setting this up.

We just started working on this a week ago and it has not been publicly discussed. (Perfect timing for your email:))

> I could of course still do
> something similar manually for projects that arent covered by the
> build-bot, but maybe it would be a better idea to just add those to
> the automatic process ?

The immediate goal of the bot is to determine regressions we introduce during development. It will be run daily, so adding too many projects to it might not be possible. In addition to this, it would be great if we could run the analyzer on a larger number of projects each month or so to catch the issues the bot did not catch. (You could still use the same scripts as our buildbot to automate the process.) It would also be useful to have someone file the bugs reporting the failures.

> ;)
> I ran the analyzer on a few projects a way back, but as a non-dev I
> didnt immediately know to turn that data into something useful (until
> I came up with the idea to compare results from different versions of
> the analyzer). The open source projects I tried were: bind, dhcp, gcc,
> gdb, glib, ntp, openldap, openssl, postfix. Some of those might be
> good candidates for the build-bot ?
> 
Definitely.
> 
> 2.)
> It's not that im not interested in determining which results are
> indeed bugs, which are false positives, and maybe even were the false
> negatives are... It's just that I dont have the required skills to do
> so. Which in turn made me wonder in what ways I could help out as a
> non-programmer, and resulted in the email below.

I understand. Thanks for the interest!

> Doing this
> automatically would of course be preferable.
Automating this is difficult (impossible in some cases). Think about it this way: the analyzer is an automated process which cannot determine if these are real bugs or not. If we knew how to automate this, we would add the logic to the analyzer.
> 
> 3.)
> Doing something like Coverity's open source scan results would indeed
> be the holy-grail. :)
> But as a start, it might be a lot easier, but still helpful, if just
> the reports would be published on some publicly accessible web server
> ?
What would these reports look like? I think it would be valuable to list which projects have been successfully analyzed and how many bugs have been found in each one. Going back to the HTML reports themselves, we could list them, but I am not 100% sure on how they'll be used by others. One scenario is that a project developer would try to investigate a reported bug and either fix it or mark it as a false positive.. (Again, would be great to provide the interface for feedback.)

Developing an interface to display analysis results would be useful even for providing feedback about the projects we analyze on our buildbot.

> There would be no fancy stuff like people
> categorizing/reporting/collaborating on stuff, but interested people
> would be able to easily compare results from different versions
> manually for a particular project. Which might be a good start ?
> 
> 
> 
> Anyway, thank you again for your time and reply,
> 
> 
> Regards,
> 
> 
> John Smith.
> 
> 
> On Thu, Sep 8, 2011 at 7:43 AM, Anna Zaks <ganna at apple.com> wrote:
>> Hi John,
>> I think that tracking the progress of the analyzer is very important.
>> Actually, we are in the process of setting up an internal buildbot which
>> would regularly analyze several open source projects(Python,
>> openssl, postresql, Adium,..). The goal is to be able to spot regressions as
>> quickly as possible as well as measure the progress. There are several
>> progress metrics which one could use, for example:
>> - reduction of analyzer failures/crashes
>> - number of new bugs reported / number of false positives removed
>> - increase in speed
>> - memory consumption reduction
>> Going back to your proposal, I think it would be helpful to:
>> 1) Run the analyzer on a bunch of projects and report failures if any.
>> (Ideally, these would be the projects which are not being tested in our
>> buildbot.)
>> 2) In regards to the analyzer reports, it would be VERY helpful to
>> categorize the (subset of) bugs into real bugs and false positives. An
>> increase in the number of bugs reported might mean both improvement and
>> regression in the analyzer. (Regression might be due to the analyzer
>> reporting more false positives.) Since you are not a programmer, I am not
>> sure if you are interested in doing this.
>> 3) A much more ambitious project would be setting up an external buildbot,
>> which would test the open source projects and, possibly, allow others to
>> categorize the reports. It could be something similar to Coverity's open
>> source scan results (http://scan.coverity.com/rung2.html).
>> Cheers,
>> Anna.
>> On Sep 6, 2011, at 12:35 PM, John Smith wrote:
>> 
>> Hi.
>> 
>> 
>> I really love the concept of static analysis, and would like to
>> contribute somehow. However, I am not a programmer myself, so i was
>> wondering what i could do to contribute to the ccc-analyzer ?
>> 
>> I have the following idea:
>> 
>> 1.) Let clang/ccc-analyzer devs pick a hand full reasonably well known
>> / used open source projects
>>    preferably code bases that the ccc-analyzer devs are at least
>> somewhat familiar with
>> 2.) Let me run the analyzer on these code bases with the latest trunk
>> clang/ccc-analyzer
>> 3.) Let me post the results on a website
>> 4.) a while later (some months ?) I could run the latest rev of
>> clang/ccc-analyzer on the same versions of the chosen codebases again
>> 
>> Then perhaps the differences in the results of running analyzer rev x
>> on codebase y versus running analyzer rev x+1 on the same code base y
>> could provide some insight into how well the analyzer is progressing
>> on real world code bases ?
>> 
>> The core idea here is not to find issues in the latest version of any
>> particular codebase, but rather to discover progression/regression of
>> the static analyzer. Of course, this would probably work best if the
>> code being analyzed are both 'reasonably well/widely used' real word
>> code bases, and the clang/ccc-analyzer devs are fairly familiar with
>> these code bases so that they can relatively easily interpret the
>> (differences in) results.
>> 
>> 
>> Just my 2$... Could this be useful ?
>> 
>> 
>> Let me know what you think ?
>> 
>> 
>> 
>> Regards,
>> 
>> 
>> John Smith.
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> 
>>