[cfe-dev] [RFC] LLVM bug lifecycle BoF - triaging

Sat Oct 6 14:50:24 PDT 2018

Hello,

I am not a frequent poster on the LLVM mailing lists, but I happened to
notice this thread and thought I'd weigh in.

Over 2 years ago, I reported this bug:
https://bugs.llvm.org/show_bug.cgi?id=29102

We had to add a pretty ugly workaround in Mono to deal with that, and the
workaround is still to this day written to apply to *all* Clang versions on
ARM64 because we've gotten no response to the bug. This is what we're doing
currently: https://github.com/mono/mono/blob/master/mono/utils/atomic.h#L209

I think this looks to be a pretty serious bug that shouldn't have gone
unacknowledged for so long. If there had been any kind of response to the
bug, I would've even been happy to cook up a patch. But, frankly, without
any confirmation that a bug is valid, very few potential contributors are
going to put in the time and effort to write and submit a patch and risk
that it gets rejected because the issue it's trying to address isn't even
considered a bug by the project maintainers.

Don't get me wrong, though - I understand from experience that "triage all
the bugs" is much easier said than done, especially in an open source
project. I just wanted to back up Kristof's feeling that the project is
losing potential contributions with a concrete example of such, for what
it's worth.

Regards,
Alex

On Thu, Oct 4, 2018 at 11:55 AM Kristof Beyls via cfe-dev <
cfe-dev at lists.llvm.org> wrote:

> Hi all,
>
> I’d like to share a few thoughts and analysis results on the LLVM bug life
> cycle, especially the reporting/triaging part.
>
> As one of the few people creating llvm bugzilla accounts when people
> request an account, I started to have a feel that many reported bugs,
> especially by first-time reporters, never get any reply or feedback, let
> alone be acted on.
> If people go through the effort of requesting an account, and then
> reporting the bug, they show motivation to contribute to the project.
> However, if then they see zero return on their effort spent, even if it’s
> just a confirmation of the bug indeed being real or an explanation of what
> they thought to be a bug isn’t actually a bug, I fear as a community we
> disincentify a large number of potential long-term contributors.
>
> The above was all based on gut feel, so I tried to gather a bit more data
> to see if my feel was correct or not.
> I scraped the bugs in bugzilla and post-processed them a bit. Below is a
> chart showing, year by year, how long it takes for a reported bug to get
> any comment from anyone besides to original reporter. If the bug is still
> open and didn’t have any reaction after half a year the chart categorizes
> is as an “infinite” response time.
>
>
> It shows that in recent years the chance of never getting a response to a
> bug report has been increasing.
> For some bugs - e.g. an experienced LLVM developer records a
> not-that-important bug in bugzilla - that may be just fine.
> However, I assume that for people reporting a bug for the first time, the
> majority may look at least for confirmation that what they reported is
> actually a bug.
> The chart shows (blue bars) that about 50% of first-time bug reporters
> never get any reply.
>
> I also plotted which components get the most reported bugs that don’t get
> any reaction and remain open:
>
> The percentage at the top of the bars is the percentage of bugs against
> that component that never get any reaction. The bar height shows the
> absolute numbers.
>
>
> I hope that at the “Lifecycle of LLVM bug reports” BoF at the upcoming dev
> meeting in San Jose (https://llvmdev18.sched.com/event/H2T3, 17th of
> October, 10.30am), we can discuss what could be done to improve the
> experience for first-time reporters and also to reduce the number of bug
> reports that seemingly get ignored completely.
> By sending this email, I hope to trigger discussion before the BoF, both
> by attendees and non-attendees, so that we have a more fruitful outcome.
>
> At first sight, to me, it seems that the following actions would help:
>
>    - Let’s introduce some form of “triaged” state in bugzilla, to
>    represent that a bug report has been accepted as describing a real problem;
>    able to be acted on (e.g. has a suitable reproducer); and not being a
>    duplicate of another bug report. Looking at
>    https://bugzilla.readthedocs.io/en/5.0/using/editing.html#life-cycle-of-a-bug,
>    maybe the best way to achieve this would be for newly raised bugs to by
>    default go to an “UNCONFIRMED” state instead of “NEW”? Moving the status to
>    “NEW” or “CONFIRMED” would indicate the bug has been triaged.
>    - Would it help to have one or multiple people per component that
>    volunteer to triage new bugs?
>    - With the majority of developers being part of a team working on a
>    product based on LLVM, I would assume that it is in the interest of most
>    that reported bugs at least get evaluated/triaged? What is stopping those
>    developers to find the time to do some triaging? Would a better
>    notification mechanism be useful to notify when new bugs on a specific
>    component come in that you could triage? Maybe per component try to have a
>    few people on the “default CC list”, which seems easy to set up as a
>    bugzilla administrator.
>    - Should we get rid of the "new-bugs/new bugs” component if we won’t
>    have people triaging them?
>    - Should we have some description of what a reasonable triage of a bug
>    looks like? If we write such a page, we could also use that page to
>    describe what we think should get recorded when closing bugs.
>
>
> Thanks,
>
> Kristof
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20181006/023b241a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: llvmbugs_triage_response_time.png
Type: image/png
Size: 30609 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20181006/023b241a/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: llvmbugs_component_response_rate.png
Type: image/png
Size: 46056 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20181006/023b241a/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: llvmbugs_component_response_rate.png
Type: image/png
Size: 46056 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20181006/023b241a/attachment-0002.png>