<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:507599555;
mso-list-template-ids:1227809956;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">A couple of additional data points to consider…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Approximately 30% of all reported bugs are still open.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The number of open bugs grows by roughly 28 per week. This has been consistent for the past 6 years, when I started tracking it. I've reported this to the
mailing list a few times as an FYI.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">So, we do okay—70% of bugs get closed even though we have no defined process—but clearly we can do better.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Personally I think anything that raises bug-awareness in the community can only help. All of the ideas so far have sounded great. Replacing the "new bugs"
category with UNCONFIRMED or something like that should be good; making sure that everything at least gets looked at is important. Looking forward to the BoF.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">--paulr<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></a></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Alex Rønne Petersen [mailto:alex@alexrp.com]
<br>
<b>Sent:</b> Saturday, October 06, 2018 5:50 PM<br>
<b>To:</b> Kristof.Beyls@arm.com<br>
<b>Cc:</b> llvm-dev@lists.llvm.org; cfe-dev@lists.llvm.org; lldb-dev@lists.llvm.org; tanyalattner@llvm.org; nd@arm.com; Robinson, Paul<br>
<b>Subject:</b> Re: [cfe-dev] [RFC] LLVM bug lifecycle BoF - triaging<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">Hello,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I am not a frequent poster on the LLVM mailing lists, but I happened to notice this thread and thought I'd weigh in.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Over 2 years ago, I reported this bug: <a href="https://bugs.llvm.org/show_bug.cgi?id=29102">
https://bugs.llvm.org/show_bug.cgi?id=29102</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We had to add a pretty ugly workaround in Mono to deal with that, and the workaround is still to this day written to apply to *all* Clang versions on ARM64 because we've gotten no response to the bug. This is what we're doing currently:
<a href="https://github.com/mono/mono/blob/master/mono/utils/atomic.h#L209">https://github.com/mono/mono/blob/master/mono/utils/atomic.h#L209</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I think this looks to be a pretty serious bug that shouldn't have gone unacknowledged for so long. If there had been any kind of response to the bug, I would've even been happy to cook up a patch. But, frankly, without any confirmation
that a bug is valid, very few potential contributors are going to put in the time and effort to write and submit a patch and risk that it gets rejected because the issue it's trying to address isn't even considered a bug by the project maintainers.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Don't get me wrong, though - I understand from experience that "triage all the bugs" is much easier said than done, especially in an open source project. I just wanted to back up Kristof's feeling that the project is losing potential contributions
with a concrete example of such, for what it's worth.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Alex<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Oct 4, 2018 at 11:55 AM Kristof Beyls via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="color:#5856D6"><br>
</span>I’d like to share a few thoughts and analysis results on the LLVM bug life cycle, especially the reporting/triaging part.<br>
<span style="color:#5856D6"><br>
</span>As one of the few people creating llvm bugzilla accounts when people request an account, I started to have a feel that many reported bugs, especially by first-time reporters, never get any reply or feedback, let alone be acted on.<br>
If people go through the effort of requesting an account, and then reporting the bug, they show motivation to contribute to the project. However, if then they see zero return on their effort spent, even if it’s just a confirmation of the bug indeed being real
or an explanation of what they thought to be a bug isn’t actually a bug, I fear as a community we disincentify a large number of potential long-term contributors.<br>
<span style="color:#5856D6"><br>
</span>The above was all based on gut feel, so I tried to gather a bit more data to see if my feel was correct or not.<br>
I scraped the bugs in bugzilla and post-processed them a bit. Below is a chart showing, year by year, how long it takes for a reported bug to get any comment from anyone besides to original reporter. If the bug is still open and didn’t have any reaction after
half a year the chart categorizes is as an “infinite” response time.<br>
<span style="color:#5856D6"><br>
</span> <img border="0" width="40" height="40" id="m_-45147946744328517441E93BF73-F898-4729-A715-C867EEC4BD29" src="cid:DC7C978D-FC04-470F-BAAE-CC5C623999F0"><br>
It shows that in recent years the chance of never getting a response to a bug report has been increasing.<br>
For some bugs - e.g. an experienced LLVM developer records a not-that-important bug in bugzilla - that may be just fine.<br>
However, I assume that for people reporting a bug for the first time, the majority may look at least for confirmation that what they reported is actually a bug.<br>
The chart shows (blue bars) that about 50% of first-time bug reporters never get any reply.<br>
<span style="color:#5856D6"><br>
</span>I also plotted which components get the most reported bugs that don’t get any reaction and remain open:<br>
<img border="0" width="40" height="40" id="m_-4514794674432851744497C62EB-AA6C-41CE-9C59-E3738B9451FD" src="cid:130482D2-6DEF-4796-84EC-2968F16B635C"><br>
The percentage at the top of the bars is the percentage of bugs against that component that never get any reaction. The bar height shows the absolute numbers.<br>
<span style="color:#5856D6"><br>
<br>
</span>I hope that at the “Lifecycle of LLVM bug reports” BoF at the upcoming dev meeting in San Jose (<a href="https://llvmdev18.sched.com/event/H2T3" target="_blank">https://llvmdev18.sched.com/event/H2T3</a>, 17th of October, 10.30am), we can discuss what
could be done to improve the experience for first-time reporters and also to reduce the number of bug reports that seemingly get ignored completely.<br>
By sending this email, I hope to trigger discussion before the BoF, both by attendees and non-attendees, so that we have a more fruitful outcome.<br>
<span style="color:#5856D6"><br>
</span>At first sight, to me, it seems that the following actions would help:<o:p></o:p></p>
<ul type="disc">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Let’s introduce some form of “triaged” state in bugzilla, to represent that a bug report has been accepted as describing a real problem; able to be acted on (e.g. has a suitable reproducer); and not being a duplicate of another bug report. Looking at <a href="https://bugzilla.readthedocs.io/en/5.0/using/editing.html#life-cycle-of-a-bug" target="_blank">https://bugzilla.readthedocs.io/en/5.0/using/editing.html#life-cycle-of-a-bug</a>,
maybe the best way to achieve this would be for newly raised bugs to by default go to an “UNCONFIRMED” state instead of “NEW”? Moving the status to “NEW” or “CONFIRMED” would indicate the bug has been triaged.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Would it help to have one or multiple people per component that volunteer to triage new bugs?<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
With the majority of developers being part of a team working on a product based on LLVM, I would assume that it is in the interest of most that reported bugs at least get evaluated/triaged? What is stopping those developers to find the time to do some triaging?
Would a better notification mechanism be useful to notify when new bugs on a specific component come in that you could triage? Maybe per component try to have a few people on the “default CC list”, which seems easy to set up as a bugzilla administrator.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Should we get rid of the "new-bugs/new bugs” component if we won’t have people triaging them?<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Should we have some description of what a reasonable triage of a bug looks like? If we write such a page, we could also use that page to describe what we think should get recorded when closing bugs.<o:p></o:p></li></ul>
<p class="MsoNormal"><span style="color:#5856D6"><br>
</span>Thanks,<br>
<span style="color:#5856D6"><br>
</span>Kristof<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
</body>
</html>