<div dir="ltr">Should I abandon 

<a href="https://reviews.llvm.org/D62092">https://reviews.llvm.org/D62092</a> and remove the beta category in 

<a href="https://reviews.llvm.org/D62094">https://reviews.llvm.org/D62094</a>, would be fine with me closing this project for good?<div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 23 May 2019 at 00:24, Kristóf Umann <<a href="mailto:dkszelethus@gmail.com">dkszelethus@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 22 May 2019 at 23:04, Artem Dergachev <<a href="mailto:noqnoqneo@gmail.com" target="_blank">noqnoqneo@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF">
    Mmm, i still don't understand what's the *purpose* of the beta
    package.<br>
    <br>
    I can understand the idea of "here's a new checker, we addressed all
    issues we know about but there may be other issues that we might
    have missed, so please try it out if you're into this sort of
    stuff". For now such checkers are usually outright turned on by
    default.<br>
    <br>
    But you're proposing to put checkers into beta when they have known
    classes of false positives that are relatively easy to fix. So it's
    not for gathering feedback (we already have enough) and there's no
    plan on addressing feedback rapidly either (because if you have the
    capability to address feedback rapidly, there's no reason to start
    with feedback not addressed). It sounds as if we'll still bounce
    back all the bug reports against beta checkers with "hey, this is
    beta, we know it's broken, right?". If that's the case, then they're
    not really different from alpha checkers and i wouldn't be
    comfortable providing them to the users.<br></div></blockquote><div><br></div><div>Hmm. I guess you do have a valid point here. My initial thoughts were to move alpha checkers we are actually using and found useful (which in itself imo is a good reason) to beta in order to display them under -analyzer-checker-help, and hide the ugliest stuff deep in the depths of -analyzer-checker-help-alpha. As you said very nicely, "let clang be the single source of truth".</div><div><br></div><div>At the end of the day, we could just make CodeChecker gather the list of checkers by using both -analyzer-checker-help and -analyzer-checker-help-alpha, which would include the really nasty stuff as well, but we've managed it so far.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
    <div class="gmail-m_5883736503230981326gmail-m_3051307291698322375moz-cite-prefix">On 5/22/19 10:57 AM, Kristóf Umann
      wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">
        <div dir="ltr">I implemented what we discussed here in the
          following manner: Each checker or checker option must either
          be in:<br>
          <br>
          * Alpha: Unfinished, users are strongly discouraged from using
          it.
          <div>* Beta: Stable but might produce more false positives and
            the emitted reports might lack proper explanation<br>
            * Released: Production ready, very few false positives with
            easy to understand bug reports.<br>
            <br>
            Any checker or checker option in addition can be marked as
            hidden, meaning that it's a developer feature.</div>
        </div>
        <br>
        <div class="gmail_quote">
          <div dir="ltr" class="gmail_attr">On Thu, 16 May 2019 at
            01:24, Artem Dergachev <<a href="mailto:noqnoqneo@gmail.com" target="_blank">noqnoqneo@gmail.com</a>>
            wrote:<br>
          </div>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
            <div bgcolor="#FFFFFF"> Just wanted to also point out that
              help is very welcome with implementing the missing
              functionality in the checkers that we've been discussing
              here.<br>
              <br>
              (more replies inline)<br>
              <br>
              <div class="gmail-m_5883736503230981326gmail-m_3051307291698322375gmail-m_-1071533723269754699moz-cite-prefix">On
                5/15/19 5:26 AM, Kristóf Umann wrote:<br>
              </div>
              <blockquote type="cite">
                <div dir="ltr">
                  <div>How about this:<br>
                    <br>
                    <font face="courier new, monospace">-analyzer-checker-help</font>:
                    Displays production ready (non-modeling) checkers,
                    and beta checkers with a disclaimer for each
                    description. Don't forget to patch clang-tidy!</div>
                  <div><font face="courier new, monospace">-analyzer-checker-help-developer</font>:
                    Displays only developer (modeling, debug) checkers
                    (renamed from *-hidden).</div>
                  <font face="courier new, monospace">-analyzer-checker-help-alpha</font>:
                  Displays only alpha (non-modeling) checkers with very
                  scary disclaimers both around the list and for each
                  checker description.
                  <div><br>
                  </div>
                  <div>Note how each of these is mutually exclusive.
                    I think we shouldn't make a flag that displays all
                    checkers, but should allow the following invocation
                    to do so:
                    <div>
                      <div><br>
                        <font face="courier new, monospace">clang -cc1
                          -analyzer-checker-help
                          -analyzer-checker-help-alpha
                          -analyzer-checker-help-hidden</font></div>
                    </div>
                  </div>
                </div>
              </blockquote>
              <br>
              This makes sense to me and i think this is how it should
              have always been done.<br>
              <br>
            </div>
          </blockquote>
          <div><br>
          </div>
          <div>Note that hidden checkers may only show up in
            -analyzer-checker-help-developer. Options for implicit
            checkers are NOT implicitly hidden, but options for alpha
            checkers are implicitly marked alpha. </div>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
            <div bgcolor="#FFFFFF">
              <blockquote type="cite">
                <div dir="ltr">
                  <div>
                    <div>
                      <div class="gmail_quote">
                        <div dir="ltr" class="gmail_attr">On Wed, 15 May
                          2019 at 12:39, Kristóf Umann <<a href="mailto:dkszelethus@gmail.com" target="_blank">dkszelethus@gmail.com</a>>
                          wrote:<br>
                        </div>
                        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                          <div dir="ltr">
                            <div dir="ltr">+ György Orbán</div>
                            <br>
                            <div class="gmail_quote">
                              <div dir="ltr" class="gmail_attr">On Wed,
                                15 May 2019 at 00:41, Artem Dergachev
                                <<a href="mailto:noqnoqneo@gmail.com" target="_blank">noqnoqneo@gmail.com</a>>
                                wrote:<br>
                              </div>
                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I
                                wanted to give more visibility to the
                                discussion on the status of <br>
                                different Static Analyzer "alpha"
                                (unfinished) checkers that's going on <br>
                                on Phabricator (<a href="https://reviews.llvm.org/D57858" rel="noreferrer" target="_blank">https://reviews.llvm.org/D57858</a>).
                                Story so far: We're <br>
                                trying to figure out how many of them
                                can be finished with relatively <br>
                                little effort (and therefore we should
                                probably ask around to gather <br>
                                feedback) and how many are clearly not
                                ready for any sort of use and <br>
                                should be hidden until the most glaring
                                bugs are fixed. For now we <br>
                                officially treat all alpha checkers as
                                the latter - so don't use them!<br>
                                <br>
                              </blockquote>
                              <div>Just to add to the summary: despite
                                our "disapproval" of using alpha
                                checkers, we always made them very
                                visible, since if you wanted to list the
                                available checkers, you inevitable came
                                across them (and they are also
                                lexicographically greater than the rest
                                of them).</div>
                              <div><br>
                              </div>
                              <div>Due to the lack of branches in our
                                SVN repository, we used the alpha
                                package to make development incremental,
                                which inevitable resulted in some
                                checkers in there being unfinished and
                                unstable, while others merely need some
                                finishing touches, and could be used
                                despite being rough around the patches.</div>
                            </div>
                          </div>
                        </blockquote>
                        <div>rough around the edges* </div>
                        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                          <div dir="ltr">
                            <div class="gmail_quote">
                              <div> </div>
                              <div>The discussion came up in a patch
                                that plans to expose checker options,
                                that always existed but were never
                                listable, unless you read the source
                                code. However, just like alpha checkers,
                                many of these also hide features under
                                development, while other would genuinely
                                be useful to fine tune the analyzer for
                                a specific project.</div>
                              <div><br>
                              </div>
                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> This
                                discussion is important because
                                different people's codebases are <br>
                                ridiculously different, so it's almost
                                impossible to estimate the <br>
                                quality and usefulness of static
                                analysis unless as many varied <br>
                                codebases as possible are involved.<br>
                                <br>
                                 >>! In D57858#1500668, @Szelethus
                                wrote:<br>
                                 > `IteratorChecker` is a prime
                                example that still suffers from not <br>
                                ideal FP/TP ratio, but users at Ericsson
                                value it plenty enough not to <br>
                                be bothered by it. Many of the changes
                                here and in private directly <br>
                                address user bug reports (That's just
                                one, but I do remember having <br>
                                others around too!).<br>
                                <br>
                                Once it has visitor path notes about
                                where did it get its iterators <br>
                                from, some of the iterator checks should
                                definitely be considered for <br>
                                being turned on by default. Especially
                                the mismatched iterator check <br>
                                that doesn't rely on hardcore constraint
                                solving. The current upstream <br>
                                version is not in good shape though; i
                                just tried it on LLVM and it <br>
                                currently crashes all over the place
                                with "Symbol comparison must be a <br>
                                `SymIntExpr`" assertion (pls ask me for
                                repros if they aren't obvious). <br>
                                Also it has false mismatched iterator
                                positives on `A.insert(B.begin(), <br>
                                B.end())`.<br>
                                <br>
                                <br>
                                 >> In <a href="https://reviews.llvm.org/D57858#1501065" rel="noreferrer" target="_blank">https://reviews.llvm.org/D57858#1501065</a>,
                                @dkrupp wrote:<br>
                                 > These are the alpha checkers that
                                we are testing in Ericsson:<br>
                                <br>
                                Let me undig my last year's attempt to
                                take a look at alpha checkers. <br>
                                The most common "limb" to "miss" in the
                                alpha checkers is the "bug <br>
                                visitor" functionality that'd add
                                checker-specific path notes to the <br>
                                report, which is almost inevitably
                                necessary for any path-sensitive <br>
                                checker. Bug reports without path notes
                                are hard to understand, but <br>
                                that's one thing that your users won't
                                tell you: they often just don't <br>
                                have their good taste to realize that
                                bug reports shouldn't be so hard <br>
                                to understand. The users often take it
                                for granted that they have to <br>
                                figure out themselves where do these
                                values come from, but it's still <br>
                                our job to not force them to.<br>
                                <br>
                              </blockquote>
                              <div><br>
                              </div>
                              <div>The fundamental problem here is, in
                                my opinion, that "alpha" doesn't
                                describe many of these checkers too
                                well. I think once a checker is stable
                                and has a "reasonable" true
                                positive/false positive ratio, we should
                                move them out of alpha status: not only
                                would we be able to gather invaluable
                                feedback for these, but users might
                                appreciate the feature even with a
                                couple shortcomings. However, just
                                because they are not falling apart on
                                their own, these aren't always
                                production ready -- how about we
                                introduce a "beta" package?</div>
                              <div><br>
                              </div>
                              <div>Alpha checkers would be incomplete,
                                incorrect and unstable by definition,
                                and would be hidden from non-developers.
                                Beta checkers would receive a disclaimer
                                that they might emit too much false
                                positives to be production ready and
                                don't emit ideal bug reports in terms of
                                readability, but are considered stable
                                and usable.</div>
                              <div> <br>
                              </div>
                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br>
                                 >  alpha.core.BoolAssignment<br>
                                <br>
                                Yes, i agree that this one's pretty
                                useful. It's currently missing a <br>
                                visitor that explains why does the
                                analyzer think that the value is <br>
                                non-true/false, which is often necessary
                                to understand the warning.<br>
                                <br>
                              </blockquote>
                              <div>This would be an ideal candidate to
                                be moved to a beta package.</div>
                            </div>
                          </div>
                        </blockquote>
                      </div>
                    </div>
                  </div>
                </div>
              </blockquote>
              <br>
              I'll be comfy to move it straight to core as long as
              someone stuffs a `bugreporter::trackExpressionValue()`
              into it and we both test it and see no obvious FP patterns
              specific to this checker. Unless we do this, i probably
              won't be comfy putting it into beta :)<br>
              <br>
              <blockquote type="cite">
                <div dir="ltr">
                  <div>
                    <div>
                      <div class="gmail_quote">
                        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                          <div dir="ltr">
                            <div class="gmail_quote">
                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br>
                                 >  alpha.core.CastSize<br>
                                <br>
                                This one's indeed relatively quiet, but
                                i'm seeing ~50 false positives <br>
                                caused by stuffing metadata at the
                                beginning of a dynamically allocated <br>
                                buffer. I.e., allocate a buffer that's 4
                                bytes larger than necessary, <br>
                                use these 4 bytes for our own
                                bookkeeping and provide a pointer to the
                                <br>
                                rest of the buffer to be used for the
                                actual value. I don't see an easy <br>
                                way to fix these false positives, so i
                                don't see how to move this out of <br>
                                alpha.<br>
                                <br>
                                <br>
                                 >  alpha.core.Conversion<br>
                                <br>
                                Interestingly, i haven't seen this one
                                trigger on our codebases. So i <br>
                                don't have an opinion here. There's a
                                chance it might be a good opt-in <br>
                                check. Do you have an open-source
                                project in mind on which this check is <br>
                                actually useful?<br>
                                <br>
                                <br>
                                 >  alpha.core.DynamicTypeChecker<br>
                                <br>
                                I really root for enabling this checker
                                (and generally improving our <br>
                                dynamic type-checking), but for now i'm
                                seeing ~600 false positives in <br>
                                projects that use custom RTTI (something
                                like `dyn_cast`) and ~300 more <br>
                                Objective-C-specific false positives. I
                                can take a look and try to <br>
                                reduce some of the custom RTTI ones if
                                you're interested in figuring out <br>
                                how to fix them; i don't remember if
                                they are easy to fix.<br>
                                <br>
                                <br>
                                 >  alpha.core.SizeofPtr<br>
                                <br>
                                This checker does indeed find
                                interesting bugs sometimes, but i'm <br>
                                overwhelmed by ~300 false positives in
                                which the sizeof of a pointer is <br>
                                taken intentionally. This is especially
                                annoying when the pointer is <br>
                                hidden behind a typedef and the user
                                doesn't need to know whether it's a <br>
                                pointer or not.<br>
                                <br>
                                <br>
                                 >  alpha.core.TestAfterDivZero<br>
                                <br>
                                I don't see any positives of this
                                checker, but this checker is crazy and <br>
                                shouldn't have been done this way. It's
                                a "must-problem" check and we <br>
                                don't have any sort of infrastructure to
                                even display this kind of bug <br>
                                report properly after we find it, let
                                alone to properly find it. We need <br>
                                a more-or-less full-featured data flow
                                analysis engine before we make an <br>
                                attempt on such checker.<br>
                                <br>
                                <br>
                                 > 
                                alpha.cplusplus.DeleteWithNonVirtualDtor<br>
                                <br>
                                I don't see any positives of this
                                checker. Is it any better than the <br>
                                compiler warning that we have for all
                                polymorphic classes that have no <br>
                                virtual destructors?<br>
                                <br>
                              </blockquote>
                              <div>Maybe this one too. <br>
                              </div>
                            </div>
                          </div>
                        </blockquote>
                      </div>
                    </div>
                  </div>
                </div>
              </blockquote>
              <br>
              Dunno, why duplicate a compiler warning that's already on
              by default and more aggressive than the checker?<br>
              <br>
              <blockquote type="cite">
                <div dir="ltr">
                  <div>
                    <div>
                      <div class="gmail_quote">
                        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                          <div dir="ltr">
                            <div class="gmail_quote">
                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br>
                                 >  alpha.security.MallocOverflow<br>
                                <br>
                                This one's extremely loud for me (~1500
                                false positives). It looks as if <br>
                                it warns on every `malloc(x *
                                sizeof(int))` (due to potential integer
                                <br>
                                overflow during multiplication) so i
                                just don't see it working as an <br>
                                AST-based check. We should probably
                                rewrite it on top of taint analysis <br>
                                and then it'll need a constraint solver
                                that knows how to multiply things.<br>
                                <br>
                                Like, this is the point where i'd like
                                to ask how does this happen that <br>
                                you don't see these false positives. Is
                                this checker actually quiet for <br>
                                you? Or are your users addressing these
                                warnings somehow?<br>
                                <br>
                                <br>
                                 >  alpha.security.MmapWriteExec<br>
                                <br>
                                I don't see any positives of this
                                checker. It probably needs a visitor <br>
                                (which is trivial) and it definitely
                                needs a solution for different <br>
                                values of macros on different platforms
                                that'd be better than just <br>
                                setting them as a config flag.<br>
                                <br>
                                <br>
                                 >  alpha.security.ReturnPtrRange<br>
                                <br>
                                I don't see any positives of this
                                checker. It definitely needs a visitor <br>
                                and we might as well join it with the
                                generic array bound checker. Also <br>
                                need to figure out how to deal with,
                                say, vector::end() which is <br>
                                supposed to return an out-of-bound
                                pointer.<br>
                                <br>
                                <br>
                                 > 
                                alpha.security.taint.TaintPropagation<br>
                                <br>
                                I believe that the generic taint engine
                                is currently very solid, but <br>
                                specific checks that we have on top of
                                it are very much unfinished: they <br>
                                simply don't react on any sort of
                                validation that can be used to remove <br>
                                the taint. Normally that'll involve
                                consulting the constraint manager <br>
                                (in case of tainted integers) or
                                modeling validation routines (in case <br>
                                of more complicated objects).<br>
                                <br>
                                <br>
                                 >  alpha.unix.BlockInCriticalSection<br>
                                <br>
                                I have ~10 positives and i didn't have a
                                look at them back then; might <br>
                                be good. This checker needs a visitor to
                                explain why do we think we're <br>
                                in a critical section.<br>
                                <br>
                                <br>
                                 >  alpha.unix.Chroot<br>
                                <br>
                                I don't see any positives of this
                                checker. This checker needs a visitor <br>
                                to highlight chroot.<br>
                                <br>
                                <br>
                                 >  alpha.unix.PthreadLock<br>
                                <br>
                                Uhm, i have a few patches on this
                                checker that i never committed:<br>
                                - <a href="https://reviews.llvm.org/D37806" rel="noreferrer" target="_blank">https://reviews.llvm.org/D37806</a><br>
                                - <a href="https://reviews.llvm.org/D37807" rel="noreferrer" target="_blank">https://reviews.llvm.org/D37807</a><br>
                                - <a href="https://reviews.llvm.org/D37809" rel="noreferrer" target="_blank">https://reviews.llvm.org/D37809</a><br>
                                - <a href="https://reviews.llvm.org/D37812" rel="noreferrer" target="_blank">https://reviews.llvm.org/D37812</a><br>
                                - <a href="https://reviews.llvm.org/D37963" rel="noreferrer" target="_blank">https://reviews.llvm.org/D37963</a><br>
                                And it still needs a visitor. And
                                support for more APIs, 'cause there <br>
                                are still false positives caused by
                                unobvious POSIX APIs that release <br>
                                the mutex (sometimes conditionally). And
                                once it's done, i'll be seeing <br>
                                no positives of this checker; it sounds
                                like a good checker to have, but <br>
                                it doesn't seem to find mistakes that
                                are *easy to make*.<br>
                                <br>
                                <br>
                                 >  alpha.unix.SimpleStream<br>
                                 >  alpha.unix.Stream<br>
                                <br>
                                Those need, at least:<br>
                                - A bug visitor.<br>
                                - A suppress-on-sink behavior. Otherwise
                                they warn on every assert <br>
                                between open and close (~200 false
                                positives for me).<br>
                                - Pointer escape support.<br>
                                Also i vaguely remember that the
                                non-simple checker is known to cause <br>
                                more state splits than necessary.<br>
                                <br>
                                <br>
                                 > 
                                alpha.unix.cstring.NotNullTerminated<br>
                                <br>
                                Hmm, this check looks like a walking
                                .addTransition() bug (unintended <br>
                                state split) when invoked from
                                getCStringLength(). Also it doesn't seem
                                <br>
                                to be disabled when the checker is
                                disabled, so i guess we're kinda <br>
                                "using" it too. But it's also too quiet
                                to matter, as it pretty much <br>
                                only warns when you're trying to compute
                                a strlen() of a function pointer.<br>
                                <br>
                                <br>
                                 >  alpha.unix.cstring.OutOfBounds<br>
                                <br>
                                <a href="https://bugs.llvm.org/show_bug.cgi?id=41729" rel="noreferrer" target="_blank">https://bugs.llvm.org/show_bug.cgi?id=41729</a>
                                and it also needs a visitor <br>
                                for the index. <br>
                              </blockquote>
                            </div>
                          </div>
                        </blockquote>
                      </div>
                    </div>
                  </div>
                </div>
              </blockquote>
              <br>
            </div>
          </blockquote>
        </div>
      </div>
    </blockquote>
    <br>
  </div>

</blockquote></div></div>
</blockquote></div>