<div dir="ltr"><div dir="ltr"><div>Hi!<br><br></div><div>I'm glad that someone picked this up. Making it easier to test the analyzer on real-world topics is an important task that can ultimately make it much easier to contribute to the analyzer.</div><div>See some of my comments inline.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 11 Jun 2020 at 16:23, Valeriy Savchenko via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Person has to find at least a couple of projects, build them natively, and check<br>
with the analyzer. ... It should be dead simple, maybe as simple as running<br>
`lit` tests.<br></blockquote><div><br></div><div>While I think this is a great idea we also should not forget that the tested projects should exercise the right parts of the analyzer. For instance, a patch adding exception support should be tested on projects that are using exceptions extensively. Having a static set of projects will not solve this problem. Nevertheless, this is something that is far less important to solve. First, we need something that is very close to what you proposed.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Another point that of interest, is reproducibility. <br></blockquote><div><br></div><div>Huge +1. Actually, I'd be even glad to see more extremes like running the analyzer multiple times making sure that the number of exploded graphs and other statistics are stable to avoid introducing non-deterministic behavior. <br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Short summary of what is there:<br>
* Info on 15 open-source projects to analyze, most of which are pretty small<br>
* Dockerfile with fixed versions of dependencies for these projects </blockquote><div><br></div><div>Dependencies are the bane of C++ at the moment. I'd love to see some other solutions for this problem. Some of them coming to my mind:</div><div>* Piggy backing on the source repositories of linux distributions. We could easily install all the build dependencies using the package manager automatically. The user would only need to specify the name of the source package, the rest could be automated without having to manually search for the names of the dependent packages.</div><div>* Supporting C++ package managers. There is Conan, vcpkg and some CMake based. We could use a base docker image that already has these installed.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
The system has two dependencies: python (2 or 3) and docker. <br></blockquote><div><br></div><div>How long do we want to retain Python 2 compatibility? I'm all in favor of not supporting it for long (or at all).<br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
(I am not a `csa-testbench` user, so please correct me if I'm wrong here)<br></blockquote><div><br></div><div>Your assessment is 100% correct here. We always wanted to add docker support and support for rebuilding source deb packages to solve most of the issues you mentioned. <br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
* I want it to cover all basic needs of the developer:<br>
- analyze a bunch of projects and show results<br>
- compare two given revisions<br>
- benchmark and compare performance<br></blockquote><div><br></div><div><div>I think one very important feature is to collect/compare not only the analysis results but more fine-grained information like the statistics emitted by the analyzer (number of refuted reports in case of refutation, number of exploded nodes, and so on). <br></div><div>It would be nice to be able to retrieve anything crash-related like call stacks and have an easy way to ssh into the docker image to debug the crash within the image. <br></div><div>Also, the csa-testbench has a feature to define regular expressions and collect the matching lines of the analyzer output. This can be useful to count/collect log messages.<br></div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
* I want all commands to be as simple as possible, e.g.:<br></blockquote><div><br></div><div>While I see the value of having a minimal interface I wonder if it will be a bit limiting to the power users in the end (see extracting statistics and logs based on regexp)., <br></div><div> </div><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
* Would you use a system like this?<br></blockquote><div><br></div><div>In the case, it supports my needs, definitely. As you mentioned, there are multiple contenders here: csa-testbench and SATest. I do see why the testbench is not desirable (mainly because of the dependencies), but I wonder if it would make sense to have compatible configurations. I.e. one could copy and paste a project from one to the other have it working without any additional efforts. <br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
* Does the proposed solution seem reasonable in this situation?<br></blockquote><div><br></div><div>Looks good to me. <br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
* What do you think about the directions?<br></blockquote><div><br></div><div>+1<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
* What other features do you want to see in the system?<br></blockquote><div><br></div><div>See my other inlines above.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
* What are the priorities for the project and what is the minimal feature<br>
scope to start using it?<br></blockquote><div><br></div><div>If we can run it reliably on big projects I'd say have a built bot as soon as possible (that only triggers when crashes are introduced). I think it could have prevented many errors. <br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Thank you for taking your time and reading through this!<br>
<br>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
</blockquote></div></div>