[llvm-dev] [cfe-dev] [RFC] LLVM bug lifecycle BoF - triaging

Thu Oct 4 22:04:50 PDT 2018

> On 4 Oct 2018, at 19:55, Kristof Beyls via cfe-dev <cfe-dev at lists.llvm.org> wrote:
> 
> Hi all,
> 
> I’d like to share a few thoughts and analysis results on the LLVM bug life cycle, especially the reporting/triaging part.
> 
> As one of the few people creating llvm bugzilla accounts when people request an account, I started to have a feel that many reported bugs, especially by first-time reporters, never get any reply or feedback, let alone be acted on.
> If people go through the effort of requesting an account, and then reporting the bug, they show motivation to contribute to the project. However, if then they see zero return on their effort spent, even if it’s just a confirmation of the bug indeed being real or an explanation of what they thought to be a bug isn’t actually a bug, I fear as a community we disincentify a large number of potential long-term contributors.
> 
> The above was all based on gut feel, so I tried to gather a bit more data to see if my feel was correct or not.
> I scraped the bugs in bugzilla and post-processed them a bit. Below is a chart showing, year by year, how long it takes for a reported bug to get any comment from anyone besides to original reporter. If the bug is still open and didn’t have any reaction after half a year the chart categorizes is as an “infinite” response time.
> 
>  <llvmbugs_triage_response_time.png>
> It shows that in recent years the chance of never getting a response to a bug report has been increasing.
> For some bugs - e.g. an experienced LLVM developer records a not-that-important bug in bugzilla - that may be just fine.
> However, I assume that for people reporting a bug for the first time, the majority may look at least for confirmation that what they reported is actually a bug.
> The chart shows (blue bars) that about 50% of first-time bug reporters never get any reply.
> 
> I also plotted which components get the most reported bugs that don’t get any reaction and remain open:
> <llvmbugs_component_response_rate.png>
> The percentage at the top of the bars is the percentage of bugs against that component that never get any reaction. The bar height shows the absolute numbers.
> 
> 
> I hope that at the “Lifecycle of LLVM bug reports” BoF at the upcoming dev meeting in San Jose (https://llvmdev18.sched.com/event/H2T3, 17th of October, 10.30am), we can discuss what could be done to improve the experience for first-time reporters and also to reduce the number of bug reports that seemingly get ignored completely.
> By sending this email, I hope to trigger discussion before the BoF, both by attendees and non-attendees, so that we have a more fruitful outcome.
> 
> At first sight, to me, it seems that the following actions would help:
> 	• Let’s introduce some form of “triaged” state in bugzilla, to represent that a bug report has been accepted as describing a real problem; able to be acted on (e.g. has a suitable reproducer); and not being a duplicate of another bug report. Looking at https://bugzilla.readthedocs.io/en/5.0/using/editing.html#life-cycle-of-a-bug, maybe the best way to achieve this would be for newly raised bugs to by default go to an “UNCONFIRMED” state instead of “NEW”? Moving the status to “NEW” or “CONFIRMED” would indicate the bug has been triaged.
> 	• Would it help to have one or multiple people per component that volunteer to triage new bugs?
> 	• With the majority of developers being part of a team working on a product based on LLVM, I would assume that it is in the interest of most that reported bugs at least get evaluated/triaged? What is stopping those developers to find the time to do some triaging? Would a better notification mechanism be useful to notify when new bugs on a specific component come in that you could triage? Maybe per component try to have a few people on the “default CC list”, which seems easy to set up as a bugzilla administrator.

I set something up this way for XRay, and I try to address those as much as I can. However, sometimes reports go to a different component, and I have to look at the bugs which might be mentioning the components I work on.

This would be great if there were willing volunteers to triage all the bugs that come in. I would do this but unfortunately there’s only so much time in a day/week to make progress on other things as well. If people do step up to do this kind of work, you will have my eternal thanks and will do my part to help out by being as responsive as possible.

> 	• Should we get rid of the "new-bugs/new bugs” component if we won’t have people triaging them?
> 	• Should we have some description of what a reasonable triage of a bug looks like? If we write such a page, we could also use that page to describe what we think should get recorded when closing bugs.

Codifying the standards for what a good bug report should look like will definitely help. In my experience especially with open source projects, I’ve found that not everyone — even the experienced contributors — know how much detail is helpful in conveying the problem (this taking more time back-and-forth on getting more details). The default for bug reports seem to be “provide the minimum information as possible” with the hopes that these get a reaction.

If we somehow are able to give an expectation of responsiveness to bug reporters, then maybe we’ll get higher quality bug reports too. Instead of just saying “I tried X and it broke, fix it” maybe we’ll get more reports that say “I did steps 1, 2, 3 on my machine configured as X, Y, Z and got these results A, B, C instead of my expectations which are …”.

Bug report templates, I’ve found, have also been really helpful in this regard. If we don’t have those yet, maybe having a good starting template instead of a blank box to write a bug report in might prompt the reporter better. That also makes triage at least more enjoyable because it leaves very little to wonder when you do get a bug report.

> 
> Thanks,
> 

Thank you for starting this conversation! I look forward to the results of the BoF discussion summarised as well.

Cheers

-- Dean