<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">11.06.2020 8:13 PM, Kristóf Umann via
cfe-dev пишет:<br>
</div>
<blockquote type="cite"
cite="mid:CAGcXOD5PmQr+eu2LMF2Da=arZD10+ua2QBgK-DvKbAvmTiDJmw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">+Ericssson gang<br>
<div><br>
</div>
<div>Endre and Gábor Márton in particular worked a lot of
builtbots (CTU related ones in particular), so I wouldn't risk
summarizing our current stance/progress on this issue.</div>
<div><br>
</div>
<div>What I will say however from my perspective is that I find
committing stressful for all the reasons you mentioned. While
I do my best to contribute non-breaking code, the tedious
process of jumping on the company VPN, finding the appropriate
server that isn't under heavy load to run an analysis that is
thorough enough sometimes leaves me to commit seemingly
miscellaneous patches after only running check-clang-analysis,
which on occasions comes to bite back. Things like changes in
the report count (in drastic cases changes in the bug reports
themselves, such as new notes), side effects on other
platforms, etc. makes this process really error prone as well,
not to mention that its at the point where I'm just itching to
commit and move on. While the responsibility of the committed
or soon-to-be-commited code still falls on the contributor,
the lack of builbots on a variety of platforms still makes
this process very inconvenient and downright hostile to
non-regulars. Not to mention the case where I fill the role of
the reviewer.</div>
<div><br>
</div>
<div>All in all, I really appreciate this project and agree
strongly with your goals!</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, 11 Jun 2020 at 17:51,
Gábor Horváth via cfe-dev <<a
href="mailto:cfe-dev@lists.llvm.org" moz-do-not-send="true">cfe-dev@lists.llvm.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div>Hi!<br>
<br>
</div>
<div>I'm glad that someone picked this up. Making it
easier to test the analyzer on real-world topics is an
important task that can ultimately make it much easier
to contribute to the analyzer.</div>
<div>See some of my comments inline.<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, 11 Jun 2020 at
16:23, Valeriy Savchenko via cfe-dev <<a
href="mailto:cfe-dev@lists.llvm.org" target="_blank"
moz-do-not-send="true">cfe-dev@lists.llvm.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex"><br>
Person has to find at least a couple of projects, build
them natively, and check<br>
with the analyzer. ... It should be dead simple, maybe
as simple as running<br>
`lit` tests.<br>
</blockquote>
<div><br>
</div>
<div>While I think this is a great idea we also should not
forget that the tested projects should exercise the
right parts of the analyzer. For instance, a patch
adding exception support should be tested on projects
that are using exceptions extensively. Having a static
set of projects will not solve this problem.
Nevertheless, this is something that is far less
important to solve. First, we need something that is
very close to what you proposed.<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<br>
Another point that of interest, is reproducibility. <br>
</blockquote>
<div><br>
</div>
<div>Huge +1. Actually, I'd be even glad to see more
extremes like running the analyzer multiple times making
sure that the number of exploded graphs and other
statistics are stable to avoid introducing
non-deterministic behavior. <br>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
<p><br>
</p>
<p>This one's not just about nondeterminism, it's also about
reproducibility across machines with different systems and system
headers. Like you'll be able to say "hey we broke something in our
docker tests, take a look" and you'll no longer need to extract
and send to me a preprocessed file. That's a lot if we try to
collectively keep an eye on the effects of our changes on a single
benchmark (or even if you have your own benchmark it's easy to
share the project config because it's basically just a link to the
project on github).<br>
</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:CAGcXOD5PmQr+eu2LMF2Da=arZD10+ua2QBgK-DvKbAvmTiDJmw@mail.gmail.com">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
Short summary of what is there:<br>
* Info on 15 open-source projects to analyze, most of
which are pretty small<br>
* Dockerfile with fixed versions of dependencies for
these projects </blockquote>
<div><br>
</div>
<div>Dependencies are the bane of C++ at the moment. I'd
love to see some other solutions for this problem. Some
of them coming to my mind:</div>
<div>* Piggy backing on the source repositories of linux
distributions. We could easily install all the build
dependencies using the package manager automatically.
The user would only need to specify the name of the
source package, the rest could be automated without
having to manually search for the names of the dependent
packages.</div>
<div>* Supporting C++ package managers. There is Conan,
vcpkg and some CMake based. We could use a base docker
image that already has these installed.</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
<p><br>
</p>
<p>Just curious, given that it's debian under the hood, can we
replace our make scripts with "scan-build apt-build" or something
like that?</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:CAGcXOD5PmQr+eu2LMF2Da=arZD10+ua2QBgK-DvKbAvmTiDJmw@mail.gmail.com">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
The system has two dependencies: python (2 or 3) and
docker. <br>
</blockquote>
<div><br>
</div>
<div>How long do we want to retain Python 2 compatibility?
I'm all in favor of not supporting it for long (or at
all).<br>
</div>
<div> <br>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
<p><br>
</p>
<p>As far as i understand we've still not "officially" transitioned
to python3 in llvm. I don't think it actually matters for these
scripts; it's not like they're run every day on an ancient
buildbot that still doesn't have python3 (in fact as of now i
don't think anybody uses them at all except us) but it sounds like
in any case the only script that really needs to be python2 up to
all possible formal requirements is `SATest.py` itself which is a
trivial wrapper that parses some arguments and forwards them into
docker; for everything else there's docker and you don't care
what's within it.</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:CAGcXOD5PmQr+eu2LMF2Da=arZD10+ua2QBgK-DvKbAvmTiDJmw@mail.gmail.com">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<br>
(I am not a `csa-testbench` user, so please correct me
if I'm wrong here)<br>
</blockquote>
<div><br>
</div>
<div>Your assessment is 100% correct here. We always
wanted to add docker support and support for rebuilding
source deb packages to solve most of the issues you
mentioned. <br>
</div>
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<br>
* I want it to cover all basic needs of the developer:<br>
- analyze a bunch of projects and show results<br>
- compare two given revisions<br>
- benchmark and compare performance<br>
</blockquote>
<div><br>
</div>
<div>
<div>I think one very important feature is to
collect/compare not only the analysis results but more
fine-grained information like the statistics emitted
by the analyzer (number of refuted reports in case of
refutation, number of exploded nodes, and so on). <br>
</div>
<div>It would be nice to be able to retrieve anything
crash-related like call stacks and have an easy way to
ssh into the docker image to debug the crash within
the image. <br>
</div>
<div>Also, the csa-testbench has a feature to define
regular expressions and collect the matching lines of
the analyzer output. This can be useful to
count/collect log messages.<br>
</div>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<br>
* I want all commands to be as simple as possible,
e.g.:<br>
</blockquote>
<div><br>
</div>
<div>While I see the value of having a minimal interface I
wonder if it will be a bit limiting to the power users
in the end (see extracting statistics and logs based on
regexp)., <br>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
<p><br>
</p>
<p>I think it's totally worth it to have both. When a newcomer tries
to test their first checker there's nothing better than a simple
one-liner that we can tell them to run. But on the other hand
having fine-grained commands for controlling every step of the
process is absolutely empowering and not going anywhere.</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:CAGcXOD5PmQr+eu2LMF2Da=arZD10+ua2QBgK-DvKbAvmTiDJmw@mail.gmail.com">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
* Would you use a system like this?<br>
</blockquote>
<div><br>
</div>
<div>In the case, it supports my needs, definitely. As you
mentioned, there are multiple contenders here:
csa-testbench and SATest. I do see why the testbench is
not desirable (mainly because of the dependencies), but
I wonder if it would make sense to have compatible
configurations. I.e. one could copy and paste a project
from one to the other have it working without any
additional efforts. <br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<br>
* Does the proposed solution seem reasonable in this
situation?<br>
</blockquote>
<div><br>
</div>
<div>Looks good to me. <br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<br>
* What do you think about the directions?<br>
</blockquote>
<div><br>
</div>
<div>+1<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<br>
* What other features do you want to see in the
system?<br>
</blockquote>
<div><br>
</div>
<div>See my other inlines above.<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<br>
* What are the priorities for the project and what is
the minimal feature<br>
scope to start using it?<br>
</blockquote>
<div><br>
</div>
<div>If we can run it reliably on big projects I'd say
have a built bot as soon as possible (that only triggers
when crashes are introduced). I think it could have
prevented many errors. <br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<br>
Thank you for taking your time and reading through this!<br>
<br>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank"
moz-do-not-send="true">cfe-dev@lists.llvm.org</a><br>
<a
href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
</blockquote>
</div>
</div>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank"
moz-do-not-send="true">cfe-dev@lists.llvm.org</a><br>
<a
href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
cfe-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>
<a class="moz-txt-link-freetext" href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a>
</pre>
</blockquote>
</body>
</html>