<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p><br>
    </p>
    <div class="moz-cite-prefix">11.06.2020 8:13 PM, Kristóf Umann via
      cfe-dev пишет:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAGcXOD5PmQr+eu2LMF2Da=arZD10+ua2QBgK-DvKbAvmTiDJmw@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">+Ericssson gang<br>
        <div><br>
        </div>
        <div>Endre and Gábor Márton in particular worked a lot of
          builtbots (CTU related ones in particular), so I wouldn't risk
          summarizing our current stance/progress on this issue.</div>
        <div><br>
        </div>
        <div>What I will say however from my perspective is that I find
          committing stressful for all the reasons you mentioned. While
          I do my best to contribute non-breaking code, the tedious
          process of jumping on the company VPN, finding the appropriate
          server that isn't under heavy load to run an analysis that is
          thorough enough sometimes leaves me to commit seemingly
          miscellaneous patches after only running check-clang-analysis,
          which on occasions comes to bite back. Things like changes in
          the report count (in drastic cases changes in the bug reports
          themselves, such as new notes), side effects on other
          platforms, etc. makes this process really error prone as well,
          not to mention that its at the point where I'm just itching to
          commit and move on. While the responsibility of the committed
          or soon-to-be-commited code still falls on the contributor,
          the lack of builbots on a variety of platforms still makes
          this process very inconvenient and downright hostile to
          non-regulars. Not to mention the case where I fill the role of
          the reviewer.</div>
        <div><br>
        </div>
        <div>All in all, I really appreciate this project and agree
          strongly with your goals!</div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">On Thu, 11 Jun 2020 at 17:51,
          Gábor Horváth via cfe-dev <<a
            href="mailto:cfe-dev@lists.llvm.org" moz-do-not-send="true">cfe-dev@lists.llvm.org</a>>
          wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div dir="ltr">
            <div dir="ltr">
              <div>Hi!<br>
                <br>
              </div>
              <div>I'm glad that someone picked this up. Making it
                easier to test the analyzer on real-world topics is an
                important task that can ultimately make it much easier
                to contribute to the analyzer.</div>
              <div>See some of my comments inline.<br>
              </div>
            </div>
            <br>
            <div class="gmail_quote">
              <div dir="ltr" class="gmail_attr">On Thu, 11 Jun 2020 at
                16:23, Valeriy Savchenko via cfe-dev <<a
                  href="mailto:cfe-dev@lists.llvm.org" target="_blank"
                  moz-do-not-send="true">cfe-dev@lists.llvm.org</a>>
                wrote:<br>
              </div>
              <blockquote class="gmail_quote" style="margin:0px 0px 0px
                0.8ex;border-left:1px solid
                rgb(204,204,204);padding-left:1ex"><br>
                Person has to find at least a couple of projects, build
                them natively, and check<br>
                with the analyzer. ... It should be dead simple, maybe
                as simple as running<br>
                `lit` tests.<br>
              </blockquote>
              <div><br>
              </div>
              <div>While I think this is a great idea we also should not
                forget that the tested projects should exercise the
                right parts of the analyzer. For instance, a patch
                adding exception support should be tested on projects
                that are using exceptions extensively. Having a static
                set of projects will not solve this problem.
                Nevertheless, this is something that is far less
                important to solve. First, we need something that is
                very close to what you proposed.<br>
              </div>
              <div> </div>
              <blockquote class="gmail_quote" style="margin:0px 0px 0px
                0.8ex;border-left:1px solid
                rgb(204,204,204);padding-left:1ex">
                <br>
                Another point that of interest, is reproducibility. <br>
              </blockquote>
              <div><br>
              </div>
              <div>Huge +1. Actually, I'd be even glad to see more
                extremes like running the analyzer multiple times making
                sure that the number of exploded graphs and other
                statistics are stable to avoid introducing
                non-deterministic behavior. <br>
              </div>
            </div>
          </div>
        </blockquote>
      </div>
    </blockquote>
    <p><br>
    </p>
    <p>This one's not just about nondeterminism, it's also about
      reproducibility across machines with different systems and system
      headers. Like you'll be able to say "hey we broke something in our
      docker tests, take a look" and you'll no longer need to extract
      and send to me a preprocessed file. That's a lot if we try to
      collectively keep an eye on the effects of our changes on a single
      benchmark (or even if you have your own benchmark it's easy to
      share the project config because it's basically just a link to the
      project on github).<br>
    </p>
    <p><br>
    </p>
    <blockquote type="cite"
cite="mid:CAGcXOD5PmQr+eu2LMF2Da=arZD10+ua2QBgK-DvKbAvmTiDJmw@mail.gmail.com">
      <div class="gmail_quote">
        <blockquote class="gmail_quote" style="margin:0px 0px 0px
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div dir="ltr">
            <div class="gmail_quote">
              <blockquote class="gmail_quote" style="margin:0px 0px 0px
                0.8ex;border-left:1px solid
                rgb(204,204,204);padding-left:1ex">
                Short summary of what is there:<br>
                  * Info on 15 open-source projects to analyze, most of
                which are pretty small<br>
                  * Dockerfile with fixed versions of dependencies for
                these projects </blockquote>
              <div><br>
              </div>
              <div>Dependencies are the bane of C++ at the moment. I'd
                love to see some other solutions for this problem. Some
                of them coming to my mind:</div>
              <div>* Piggy backing on the source repositories of linux
                distributions. We could easily install all the build
                dependencies using the package manager automatically.
                The user would only need to specify the name of the
                source package, the rest could be automated without
                having to manually search for the names of the dependent
                packages.</div>
              <div>* Supporting C++ package managers. There is Conan,
                vcpkg and some CMake based. We could use a base docker
                image that already has these installed.</div>
            </div>
          </div>
        </blockquote>
      </div>
    </blockquote>
    <p><br>
    </p>
    <p>Just curious, given that it's debian under the hood, can we
      replace our make scripts with "scan-build apt-build" or something
      like that?</p>
    <p><br>
    </p>
    <blockquote type="cite"
cite="mid:CAGcXOD5PmQr+eu2LMF2Da=arZD10+ua2QBgK-DvKbAvmTiDJmw@mail.gmail.com">
      <div class="gmail_quote">
        <blockquote class="gmail_quote" style="margin:0px 0px 0px
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div dir="ltr">
            <div class="gmail_quote">
              <blockquote class="gmail_quote" style="margin:0px 0px 0px
                0.8ex;border-left:1px solid
                rgb(204,204,204);padding-left:1ex">
                The system has two dependencies: python (2 or 3) and
                docker.  <br>
              </blockquote>
              <div><br>
              </div>
              <div>How long do we want to retain Python 2 compatibility?
                I'm all in favor of not supporting it for long (or at
                all).<br>
              </div>
              <div> <br>
              </div>
            </div>
          </div>
        </blockquote>
      </div>
    </blockquote>
    <p><br>
    </p>
    <p>As far as i understand we've still not "officially" transitioned
      to python3 in llvm. I don't think it actually matters for these
      scripts; it's not like they're run every day on an ancient
      buildbot that still doesn't have python3 (in fact as of now i
      don't think anybody uses them at all except us) but it sounds like
      in any case the only script that really needs to be python2 up to
      all possible formal requirements is `SATest.py` itself which is a
      trivial wrapper that parses some arguments and forwards them into
      docker; for everything else there's docker and you don't care
      what's within it.</p>
    <p><br>
    </p>
    <blockquote type="cite"
cite="mid:CAGcXOD5PmQr+eu2LMF2Da=arZD10+ua2QBgK-DvKbAvmTiDJmw@mail.gmail.com">
      <div class="gmail_quote">
        <blockquote class="gmail_quote" style="margin:0px 0px 0px
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div dir="ltr">
            <div class="gmail_quote">
              <blockquote class="gmail_quote" style="margin:0px 0px 0px
                0.8ex;border-left:1px solid
                rgb(204,204,204);padding-left:1ex">
                <br>
                (I am not a `csa-testbench` user, so please correct me
                if I'm wrong here)<br>
              </blockquote>
              <div><br>
              </div>
              <div>Your assessment is 100% correct here. We always
                wanted to add docker support and support for rebuilding
                source deb packages to solve most of the issues you
                mentioned. <br>
              </div>
              <div> <br>
              </div>
              <blockquote class="gmail_quote" style="margin:0px 0px 0px
                0.8ex;border-left:1px solid
                rgb(204,204,204);padding-left:1ex">
                <br>
                  * I want it to cover all basic needs of the developer:<br>
                      - analyze a bunch of projects and show results<br>
                      - compare two given revisions<br>
                      - benchmark and compare performance<br>
              </blockquote>
              <div><br>
              </div>
              <div>
                <div>I think one very important feature is to
                  collect/compare not only the analysis results but more
                  fine-grained information like the statistics emitted
                  by the analyzer (number of refuted reports in case of
                  refutation, number of exploded nodes, and so on). <br>
                </div>
                <div>It would be nice to be able to retrieve anything
                  crash-related like call stacks and have an easy way to
                  ssh into the docker image to debug the crash within
                  the image. <br>
                </div>
                <div>Also, the csa-testbench has a feature to define
                  regular expressions and collect the matching lines of
                  the analyzer output. This can be useful to
                  count/collect log messages.<br>
                </div>
              </div>
              <div> </div>
              <blockquote class="gmail_quote" style="margin:0px 0px 0px
                0.8ex;border-left:1px solid
                rgb(204,204,204);padding-left:1ex">
                <br>
                  * I want all commands to be as simple as possible,
                e.g.:<br>
              </blockquote>
              <div><br>
              </div>
              <div>While I see the value of having a minimal interface I
                wonder if it will be a bit limiting to the power users
                in the end (see extracting statistics and logs based on
                regexp)., <br>
              </div>
            </div>
          </div>
        </blockquote>
      </div>
    </blockquote>
    <p><br>
    </p>
    <p>I think it's totally worth it to have both. When a newcomer tries
      to test their first checker there's nothing better than a simple
      one-liner that we can tell them to run. But on the other hand
      having fine-grained commands for controlling every step of the
      process is absolutely empowering and not going anywhere.</p>
    <p><br>
    </p>
    <blockquote type="cite"
cite="mid:CAGcXOD5PmQr+eu2LMF2Da=arZD10+ua2QBgK-DvKbAvmTiDJmw@mail.gmail.com">
      <div class="gmail_quote">
        <blockquote class="gmail_quote" style="margin:0px 0px 0px
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div dir="ltr">
            <div class="gmail_quote">
              <blockquote class="gmail_quote" style="margin:0px 0px 0px
                0.8ex;border-left:1px solid
                rgb(204,204,204);padding-left:1ex">
                  * Would you use a system like this?<br>
              </blockquote>
              <div><br>
              </div>
              <div>In the case, it supports my needs, definitely. As you
                mentioned, there are multiple contenders here:
                csa-testbench and SATest. I do see why the testbench is
                not desirable (mainly because of the dependencies), but
                I wonder if it would make sense to have compatible
                configurations. I.e. one could copy and paste a project
                from one to the other have it working without any
                additional efforts. <br>
              </div>
              <div> </div>
              <blockquote class="gmail_quote" style="margin:0px 0px 0px
                0.8ex;border-left:1px solid
                rgb(204,204,204);padding-left:1ex">
                <br>
                  * Does the proposed solution seem reasonable in this
                situation?<br>
              </blockquote>
              <div><br>
              </div>
              <div>Looks good to me. <br>
              </div>
              <div> </div>
              <blockquote class="gmail_quote" style="margin:0px 0px 0px
                0.8ex;border-left:1px solid
                rgb(204,204,204);padding-left:1ex">
                <br>
                  * What do you think about the directions?<br>
              </blockquote>
              <div><br>
              </div>
              <div>+1<br>
              </div>
              <div> </div>
              <blockquote class="gmail_quote" style="margin:0px 0px 0px
                0.8ex;border-left:1px solid
                rgb(204,204,204);padding-left:1ex">
                <br>
                  * What other features do you want to see in the
                system?<br>
              </blockquote>
              <div><br>
              </div>
              <div>See my other inlines above.<br>
              </div>
              <div> </div>
              <blockquote class="gmail_quote" style="margin:0px 0px 0px
                0.8ex;border-left:1px solid
                rgb(204,204,204);padding-left:1ex">
                <br>
                  * What are the priorities for the project and what is
                the minimal feature<br>
                    scope to start using it?<br>
              </blockquote>
              <div><br>
              </div>
              <div>If we can run it reliably on big projects I'd say
                have a built bot as soon as possible (that only triggers
                when crashes are introduced). I think it could have
                prevented many errors. <br>
              </div>
              <div> </div>
              <blockquote class="gmail_quote" style="margin:0px 0px 0px
                0.8ex;border-left:1px solid
                rgb(204,204,204);padding-left:1ex">
                <br>
                Thank you for taking your time and reading through this!<br>
                <br>
                _______________________________________________<br>
                cfe-dev mailing list<br>
                <a href="mailto:cfe-dev@lists.llvm.org" target="_blank"
                  moz-do-not-send="true">cfe-dev@lists.llvm.org</a><br>
                <a
                  href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev"
                  rel="noreferrer" target="_blank"
                  moz-do-not-send="true">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
              </blockquote>
            </div>
          </div>
          _______________________________________________<br>
          cfe-dev mailing list<br>
          <a href="mailto:cfe-dev@lists.llvm.org" target="_blank"
            moz-do-not-send="true">cfe-dev@lists.llvm.org</a><br>
          <a
            href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev"
            rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
        </blockquote>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <pre class="moz-quote-pre" wrap="">_______________________________________________
cfe-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>
<a class="moz-txt-link-freetext" href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a>
</pre>
    </blockquote>
  </body>
</html>