[cfe-dev] [analyzer] Regression testing for the static analyzer

Mon Jun 15 06:05:22 PDT 2020

First of all, thank you a lot for engaging in this conversation, sharing your ideas, and, of course, for your kind words :-)

>> Huge +1. Actually, I'd be even glad to see more extremes like running the analyzer multiple times making sure that the number of exploded graphs and other statistics are stable to avoid introducing non-deterministic behavior. 
> This one's not just about nondeterminism, it's also about reproducibility across machines with different systems and system headers. Like you'll be able to say "hey we broke something in our docker tests, take a look" and you'll no longer need to extract and send to me a preprocessed file. That's a lot if we try to collectively keep an eye on the effects of our changes on a single benchmark (or even if you have your own benchmark it's easy to share the project config because it's basically just a link to the project on github).
> 
Yes, and yes!  I thought about running the analyzer multiple times on one project for benchmarking.  In that mode we can also check for variations in stats between seemingly identical runs.
Sharing is probably the best part of this approach.

>> Dependencies are the bane of C++ at the moment. I'd love to see some other solutions for this problem. Some of them coming to my mind:
>> * Piggy backing on the source repositories of linux distributions. We could easily install all the build dependencies using the package manager automatically. The user would only need to specify the name of the source package, the rest could be automated without having to manually search for the names of the dependent packages.
>> * Supporting C++ package managers. There is Conan, vcpkg and some CMake based. We could use a base docker image that already has these installed.

Unfortunately, package managers could not be a solution across different platforms if we need reproducible results, one of the Conan’s main features is downloading packages appropriate for the current user setup pretty seemingly and consistently across different setups.  However it means that they are still different.

>> We could use a base docker image that already has these installed.

This is probably a good solution, but I didn’t have problem to build natively in docker so far.

>> While I think this is a great idea we also should not forget that the tested projects should exercise the right parts of the analyzer. For instance, a patch adding exception support should be tested on projects that are using exceptions extensively. Having a static set of projects will not solve this problem. Nevertheless, this is something that is far less important to solve. First, we need something that is very close to what you proposed.

I do agree, it should be definitely on a roadmap.  I also planned to have a tag system for projects, like “tiny”, “math”, “C++14”, “web”, and so on.  It is not exactly what you have in mind, but I think it could be a good first step in that direction.

>> How long do we want to retain Python 2 compatibility? I'm all in favor of not supporting it for long (or at all).
> As far as i understand we've still not "officially" transitioned to python3 in llvm. I don't think it actually matters for these scripts; it's not like they're run every day on an ancient buildbot that still doesn't have python3 (in fact as of now i don't think anybody uses them at all except us) but it sounds like in any case the only script that really needs to be python2 up to all possible formal requirements is `SATest.py` itself which is a trivial wrapper that parses some arguments and forwards them into docker; for everything else there's docker and you don't care what's within it.
> 

So, yes, as Artem said, SATest.py is the only script that is compatible with both versions.  All other parts of the system had been migrated to Python 3.

>> I think one very important feature is to collect/compare not only the analysis results but more fine-grained information like the statistics emitted by the analyzer (number of refuted reports in case of refutation, number of exploded nodes, and so on). 
>> It would be nice to be able to retrieve anything crash-related like call stacks and have an easy way to ssh into the docker image to debug the crash within the image. 
>> Also, the csa-testbench has a feature to define regular expressions and collect the matching lines of the analyzer output. This can be useful to count/collect log messages.

The usability of this system is probably the main point for me, so the ease of debugging should be the priority.  I already introduced a `—shell` option that provides an easy way to ssh into a docker without thinking too much about docker nature of things (if container is not running and how to clean it up afterwards).  And I believe pattern-matching certain erroneous situations and making a short summary out of those can be a really good feature.

>> While I see the value of having a minimal interface I wonder if it will be a bit limiting to the power users in the end (see extracting statistics and logs based on regexp)., 
> I think it's totally worth it to have both. When a newcomer tries to test their first checker there's nothing better than a simple one-liner that we can tell them to run. But on the other hand having fine-grained commands for controlling every step of the process is absolutely empowering and not going anywhere.
> 
It’s exactly right.  I believe that with a set of very reasonable defaults it can be achieved.  All additional tweaking should be probably categorized and put into README.

>> While I do my best to contribute non-breaking code, the tedious process of jumping on the company VPN, finding the appropriate server that isn't under heavy load to run an analysis that is thorough enough sometimes leaves me to commit seemingly miscellaneous patches after only running check-clang-analysis, which on occasions comes to bite back.

Yeah, it’s hard to predict sometimes what can be broken on a real code.  And if it is pretty scary to make changes for experienced users, what can be said about newcomers.

>> side effects on other platforms

It is pretty hard to avoid those, I guess we’ll have to keep our separate CI setups for that, but at least we’ll be able to get rid of the majority of issues common across platforms.

> On 11 Jun 2020, at 20:50, Artem Dergachev via cfe-dev <cfe-dev at lists.llvm.org> wrote:
> 
> 
> 11.06.2020 8:13 PM, Kristóf Umann via cfe-dev пишет:
>> +Ericssson gang
>> 
>> Endre and Gábor Márton in particular worked a lot of builtbots (CTU related ones in particular), so I wouldn't risk summarizing our current stance/progress on this issue.
>> 
>> What I will say however from my perspective is that I find committing stressful for all the reasons you mentioned. While I do my best to contribute non-breaking code, the tedious process of jumping on the company VPN, finding the appropriate server that isn't under heavy load to run an analysis that is thorough enough sometimes leaves me to commit seemingly miscellaneous patches after only running check-clang-analysis, which on occasions comes to bite back. Things like changes in the report count (in drastic cases changes in the bug reports themselves, such as new notes), side effects on other platforms, etc. makes this process really error prone as well, not to mention that its at the point where I'm just itching to commit and move on. While the responsibility of the committed or soon-to-be-commited code still falls on the contributor, the lack of builbots on a variety of platforms still makes this process very inconvenient and downright hostile to non-regulars. Not to mention the case where I fill the role of the reviewer.
>> 
>> All in all, I really appreciate this project and agree strongly with your goals!
>> 
>> On Thu, 11 Jun 2020 at 17:51, Gábor Horváth via cfe-dev <cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>> wrote:
>> Hi!
>> 
>> I'm glad that someone picked this up. Making it easier to test the analyzer on real-world topics is an important task that can ultimately make it much easier to contribute to the analyzer.
>> See some of my comments inline.
>> 
>> On Thu, 11 Jun 2020 at 16:23, Valeriy Savchenko via cfe-dev <cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>> wrote:
>> 
>> Person has to find at least a couple of projects, build them natively, and check
>> with the analyzer. ... It should be dead simple, maybe as simple as running
>> `lit` tests.
>> 
>> While I think this is a great idea we also should not forget that the tested projects should exercise the right parts of the analyzer. For instance, a patch adding exception support should be tested on projects that are using exceptions extensively. Having a static set of projects will not solve this problem. Nevertheless, this is something that is far less important to solve. First, we need something that is very close to what you proposed.
>>  
>> 
>> Another point that of interest, is reproducibility. 
>> 
>> Huge +1. Actually, I'd be even glad to see more extremes like running the analyzer multiple times making sure that the number of exploded graphs and other statistics are stable to avoid introducing non-deterministic behavior. 
> 
> This one's not just about nondeterminism, it's also about reproducibility across machines with different systems and system headers. Like you'll be able to say "hey we broke something in our docker tests, take a look" and you'll no longer need to extract and send to me a preprocessed file. That's a lot if we try to collectively keep an eye on the effects of our changes on a single benchmark (or even if you have your own benchmark it's easy to share the project config because it's basically just a link to the project on github).
> 
> 
> 
>> Short summary of what is there:
>>   * Info on 15 open-source projects to analyze, most of which are pretty small
>>   * Dockerfile with fixed versions of dependencies for these projects 
>> 
>> Dependencies are the bane of C++ at the moment. I'd love to see some other solutions for this problem. Some of them coming to my mind:
>> * Piggy backing on the source repositories of linux distributions. We could easily install all the build dependencies using the package manager automatically. The user would only need to specify the name of the source package, the rest could be automated without having to manually search for the names of the dependent packages.
>> * Supporting C++ package managers. There is Conan, vcpkg and some CMake based. We could use a base docker image that already has these installed.
> 
> Just curious, given that it's debian under the hood, can we replace our make scripts with "scan-build apt-build" or something like that?
> 
> 
> 
>> The system has two dependencies: python (2 or 3) and docker.  
>> 
>> How long do we want to retain Python 2 compatibility? I'm all in favor of not supporting it for long (or at all).
>>  
> 
> As far as i understand we've still not "officially" transitioned to python3 in llvm. I don't think it actually matters for these scripts; it's not like they're run every day on an ancient buildbot that still doesn't have python3 (in fact as of now i don't think anybody uses them at all except us) but it sounds like in any case the only script that really needs to be python2 up to all possible formal requirements is `SATest.py` itself which is a trivial wrapper that parses some arguments and forwards them into docker; for everything else there's docker and you don't care what's within it.
> 
> 
> 
>> 
>> (I am not a `csa-testbench` user, so please correct me if I'm wrong here)
>> 
>> Your assessment is 100% correct here. We always wanted to add docker support and support for rebuilding source deb packages to solve most of the issues you mentioned. 
>>  
>> 
>>   * I want it to cover all basic needs of the developer:
>>       - analyze a bunch of projects and show results
>>       - compare two given revisions
>>       - benchmark and compare performance
>> 
>> I think one very important feature is to collect/compare not only the analysis results but more fine-grained information like the statistics emitted by the analyzer (number of refuted reports in case of refutation, number of exploded nodes, and so on). 
>> It would be nice to be able to retrieve anything crash-related like call stacks and have an easy way to ssh into the docker image to debug the crash within the image. 
>> Also, the csa-testbench has a feature to define regular expressions and collect the matching lines of the analyzer output. This can be useful to count/collect log messages.
>>  
>> 
>>   * I want all commands to be as simple as possible, e.g.:
>> 
>> While I see the value of having a minimal interface I wonder if it will be a bit limiting to the power users in the end (see extracting statistics and logs based on regexp)., 
> 
> I think it's totally worth it to have both. When a newcomer tries to test their first checker there's nothing better than a simple one-liner that we can tell them to run. But on the other hand having fine-grained commands for controlling every step of the process is absolutely empowering and not going anywhere.
> 
> 
> 
>>   * Would you use a system like this?
>> 
>> In the case, it supports my needs, definitely. As you mentioned, there are multiple contenders here: csa-testbench and SATest. I do see why the testbench is not desirable (mainly because of the dependencies), but I wonder if it would make sense to have compatible configurations. I.e. one could copy and paste a project from one to the other have it working without any additional efforts. 
>>  
>> 
>>   * Does the proposed solution seem reasonable in this situation?
>> 
>> Looks good to me. 
>>  
>> 
>>   * What do you think about the directions?
>> 
>> +1
>>  
>> 
>>   * What other features do you want to see in the system?
>> 
>> See my other inlines above.
>>  
>> 
>>   * What are the priorities for the project and what is the minimal feature
>>     scope to start using it?
>> 
>> If we can run it reliably on big projects I'd say have a built bot as soon as possible (that only triggers when crashes are introduced). I think it could have prevented many errors. 
>>  
>> 
>> Thank you for taking your time and reading through this!
>> 
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev>
>> 
>> 
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20200615/0b5026d4/attachment-0001.html>