<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Kristof + Paul, thank you for organising this BoF!<br>
</p>
A cleanup of the components would be welcome, we have multiple
components that get used as dumping grounds (new bugs, clang/new
bugs, clang/llvm codegen for instance), just merging those into a
single "new bugs"/"unconfirmed" component would at least mean bugs
are in the same place.<br>
<br>
We have components for things that don't exist any more (old
backends ....), and we are missing components for some major parts
of the codebase (various tools, should vectorizers be put in a
single component? etc.) - it's all scope for bugs getting filed,
lost and forgotten.<br>
<br>
Embedding more information in the bugzilla main page as well - for
instance we barely use keywords (including the beginner tag which
still holds promise), adding keyword links (or just the
<a class="moz-txt-link-freetext" href="https://bugs.llvm.org/describekeywords.cgi">https://bugs.llvm.org/describekeywords.cgi</a> table) to the bottom of
the main page could be trivial.<br>
<br>
We're also including links to godbolt (sorry, Compiler Explorer)
pages a lot more in bugs, there are probably things we can do to
make that more integral to the reporting/triage process - I keep
wondering if there'd be ways to make that part of web-based
reduction and bisection processes.<br>
<br>
Finally, finding ways to properly triage the oss-fuzz reports that
get sent to [llvm-bugs] - their signal:noise is poor (fuzz
tests....) and I have the feeling that most of those go completely
unchecked and many 'fixes' are by pure chance.<br>
<br>
Simon.<br>
<br>
<div class="moz-cite-prefix">On 10/10/2018 19:17, via llvm-dev
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:E3B07FDB86BFF041819DC057DEED8FEA0141AAFC31@USCULXMSG13.am.sony.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:507599555;
mso-list-template-ids:1227809956;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">A
couple of additional data points to consider…<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Approximately
30% of all reported bugs are still open.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The
number of open bugs grows by roughly 28 per week. This has
been consistent for the past 6 years, when I started
tracking it. I've reported this to the mailing list a few
times as an FYI.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">So,
we do okay—70% of bugs get closed even though we have no
defined process—but clearly we can do better.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Personally
I think anything that raises bug-awareness in the community
can only help. All of the ideas so far have sounded great.
Replacing the "new bugs" category with UNCONFIRMED or
something like that should be good; making sure that
everything at least gets looked at is important. Looking
forward to the BoF.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">--paulr<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"
moz-do-not-send="true"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></a></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in
0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
Alex Rønne Petersen [<a class="moz-txt-link-freetext" href="mailto:alex@alexrp.com">mailto:alex@alexrp.com</a>]
<br>
<b>Sent:</b> Saturday, October 06, 2018 5:50 PM<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:Kristof.Beyls@arm.com">Kristof.Beyls@arm.com</a><br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated" href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>;
<a class="moz-txt-link-abbreviated" href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>; <a class="moz-txt-link-abbreviated" href="mailto:lldb-dev@lists.llvm.org">lldb-dev@lists.llvm.org</a>;
<a class="moz-txt-link-abbreviated" href="mailto:tanyalattner@llvm.org">tanyalattner@llvm.org</a>; <a class="moz-txt-link-abbreviated" href="mailto:nd@arm.com">nd@arm.com</a>; Robinson, Paul<br>
<b>Subject:</b> Re: [cfe-dev] [RFC] LLVM bug lifecycle
BoF - triaging<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">Hello,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I am not a frequent poster on the
LLVM mailing lists, but I happened to notice this
thread and thought I'd weigh in.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Over 2 years ago, I reported this
bug: <a
href="https://bugs.llvm.org/show_bug.cgi?id=29102"
moz-do-not-send="true">
https://bugs.llvm.org/show_bug.cgi?id=29102</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We had to add a pretty ugly
workaround in Mono to deal with that, and the
workaround is still to this day written to apply to
*all* Clang versions on ARM64 because we've gotten no
response to the bug. This is what we're doing
currently:
<a
href="https://github.com/mono/mono/blob/master/mono/utils/atomic.h#L209"
moz-do-not-send="true">https://github.com/mono/mono/blob/master/mono/utils/atomic.h#L209</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I think this looks to be a pretty
serious bug that shouldn't have gone unacknowledged
for so long. If there had been any kind of response to
the bug, I would've even been happy to cook up a
patch. But, frankly, without any confirmation that a
bug is valid, very few potential contributors are
going to put in the time and effort to write and
submit a patch and risk that it gets rejected because
the issue it's trying to address isn't even considered
a bug by the project maintainers.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Don't get me wrong, though - I
understand from experience that "triage all the bugs"
is much easier said than done, especially in an open
source project. I just wanted to back up Kristof's
feeling that the project is losing potential
contributions with a concrete example of such, for
what it's worth.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Alex<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Oct 4, 2018 at 11:55 AM
Kristof Beyls via cfe-dev <<a
href="mailto:cfe-dev@lists.llvm.org"
moz-do-not-send="true">cfe-dev@lists.llvm.org</a>>
wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC
1.0pt;padding:0in 0in 0in
6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="color:#5856D6"><br>
</span>I’d like to share a few thoughts and analysis
results on the LLVM bug life cycle, especially the
reporting/triaging part.<br>
<span style="color:#5856D6"><br>
</span>As one of the few people creating llvm
bugzilla accounts when people request an account, I
started to have a feel that many reported bugs,
especially by first-time reporters, never get any
reply or feedback, let alone be acted on.<br>
If people go through the effort of requesting an
account, and then reporting the bug, they show
motivation to contribute to the project. However, if
then they see zero return on their effort spent,
even if it’s just a confirmation of the bug indeed
being real or an explanation of what they thought to
be a bug isn’t actually a bug, I fear as a community
we disincentify a large number of potential
long-term contributors.<br>
<span style="color:#5856D6"><br>
</span>The above was all based on gut feel, so I
tried to gather a bit more data to see if my feel
was correct or not.<br>
I scraped the bugs in bugzilla and post-processed
them a bit. Below is a chart showing, year by year,
how long it takes for a reported bug to get any
comment from anyone besides to original reporter. If
the bug is still open and didn’t have any reaction
after half a year the chart categorizes is as an
“infinite” response time.<br>
<span style="color:#5856D6"><br>
</span> <img
id="m_-45147946744328517441E93BF73-F898-4729-A715-C867EEC4BD29"
src="cid:DC7C978D-FC04-470F-BAAE-CC5C623999F0"
moz-do-not-send="true" height="40" width="40"
border="0"><br>
It shows that in recent years the chance of never
getting a response to a bug report has been
increasing.<br>
For some bugs - e.g. an experienced LLVM developer
records a not-that-important bug in bugzilla - that
may be just fine.<br>
However, I assume that for people reporting a bug
for the first time, the majority may look at least
for confirmation that what they reported is actually
a bug.<br>
The chart shows (blue bars) that about 50% of
first-time bug reporters never get any reply.<br>
<span style="color:#5856D6"><br>
</span>I also plotted which components get the most
reported bugs that don’t get any reaction and remain
open:<br>
<img
id="m_-4514794674432851744497C62EB-AA6C-41CE-9C59-E3738B9451FD"
src="cid:130482D2-6DEF-4796-84EC-2968F16B635C"
moz-do-not-send="true" height="40" width="40"
border="0"><br>
The percentage at the top of the bars is the
percentage of bugs against that component that never
get any reaction. The bar height shows the absolute
numbers.<br>
<span style="color:#5856D6"><br>
<br>
</span>I hope that at the “Lifecycle of LLVM bug
reports” BoF at the upcoming dev meeting in San Jose
(<a href="https://llvmdev18.sched.com/event/H2T3"
target="_blank" moz-do-not-send="true">https://llvmdev18.sched.com/event/H2T3</a>,
17th of October, 10.30am), we can discuss what could
be done to improve the experience for first-time
reporters and also to reduce the number of bug
reports that seemingly get ignored completely.<br>
By sending this email, I hope to trigger discussion
before the BoF, both by attendees and non-attendees,
so that we have a more fruitful outcome.<br>
<span style="color:#5856D6"><br>
</span>At first sight, to me, it seems that the
following actions would help:<o:p></o:p></p>
<ul type="disc">
<li class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0
level1 lfo1">
Let’s introduce some form of “triaged” state in
bugzilla, to represent that a bug report has been
accepted as describing a real problem; able to be
acted on (e.g. has a suitable reproducer); and not
being a duplicate of another bug report. Looking
at <a
href="https://bugzilla.readthedocs.io/en/5.0/using/editing.html#life-cycle-of-a-bug"
target="_blank" moz-do-not-send="true">https://bugzilla.readthedocs.io/en/5.0/using/editing.html#life-cycle-of-a-bug</a>,
maybe the best way to achieve this would be for
newly raised bugs to by default go to an
“UNCONFIRMED” state instead of “NEW”? Moving the
status to “NEW” or “CONFIRMED” would indicate the
bug has been triaged.<o:p></o:p></li>
<li class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0
level1 lfo1">
Would it help to have one or multiple people per
component that volunteer to triage new bugs?<o:p></o:p></li>
<li class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0
level1 lfo1">
With the majority of developers being part of a
team working on a product based on LLVM, I would
assume that it is in the interest of most that
reported bugs at least get evaluated/triaged? What
is stopping those developers to find the time to
do some triaging? Would a better notification
mechanism be useful to notify when new bugs on a
specific component come in that you could triage?
Maybe per component try to have a few people on
the “default CC list”, which seems easy to set up
as a bugzilla administrator.<o:p></o:p></li>
<li class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0
level1 lfo1">
Should we get rid of the "new-bugs/new bugs”
component if we won’t have people triaging them?<o:p></o:p></li>
<li class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0
level1 lfo1">
Should we have some description of what a
reasonable triage of a bug looks like? If we write
such a page, we could also use that page to
describe what we think should get recorded when
closing bugs.<o:p></o:p></li>
</ul>
<p class="MsoNormal"><span style="color:#5856D6"><br>
</span>Thanks,<br>
<span style="color:#5856D6"><br>
</span>Kristof<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank"
moz-do-not-send="true">cfe-dev@lists.llvm.org</a><br>
<a
href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev"
target="_blank" moz-do-not-send="true">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
cfe-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>
<a class="moz-txt-link-freetext" href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>