<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Sep 29, 2016, at 7:52 AM, David Blaikie <<a href="mailto:dblaikie@gmail.com" class="">dblaikie@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""><br class=""><div class="gmail_quote"><div dir="ltr" class="">On Wed, Sep 28, 2016 at 11:58 AM Robinson, Paul via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">On 28 September 2016 at 10:08, Chris Bieneman via llvm-dev<br class="gmail_msg"><<a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br class="gmail_msg">> I cannot think of any situation where a universally failing test<br class="gmail_msg">> should be in-tree unless it is a bug that someone is expecting to fix.<br class="gmail_msg"><br class="gmail_msg">It seems moderately common to mark something XFAIL temporarily to get<br class="gmail_msg">the bots green while then going ahead to fix the issue. Your proposal<br class="gmail_msg">would add extra overhead to that flow by requiring a PR as well. This<br class="gmail_msg">has value when it turns out that fix can't happen in the short term for<br class="gmail_msg">any reason. I don't have a feel for how common that is, although I'm<br class="gmail_msg">sure it does happen.<br class="gmail_msg">I think the overhead is worth the added value, but then I'm a process<br class="gmail_msg">kind of guy.<br class="gmail_msg"><br class="gmail_msg"><br class="gmail_msg">On 28 September 2016 at 10:28, Renato Golin via llvm-dev<br class="gmail_msg"><<a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br class="gmail_msg">> We already have an unwritten rule to create PRs for XFAILs, and we<br class="gmail_msg">> normally don't XFAIL lightly (I don't, at least). But creating one PR<br class="gmail_msg">> for every existing XFAIL may end up as a long list of never looked<br class="gmail_msg">> PRs. :)<br class="gmail_msg"><br class="gmail_msg">As opposed to the other ~9000 open PRs? At least they would be tracked.<br class="gmail_msg"></blockquote><div class=""><br class=""></div><div class="">I'd be inclined to agree (or at least voice the same concern) as Renato here - as a project we don't really have very good bug hygiene, so adding more bug filing doesn't seem to me like it'd give us much value.<br class=""></div></div></div></div></blockquote><div><br class=""></div><div>I haven’t done a full audit, but we have 257 XFAILs in LLVM.</div><div><br class=""></div><div>* 44 of those are vg_leak failures on TableGen tests which should be UNSUPPORTED because we allow TableGen to leak for performance the same way we allow clang to leak.</div><div>* 15 of them are vg_leak in the OCaml bindings. Someone familiar with OCaml should chime in on it, but I suspect those too should be UNSUPPORTED</div><div>* 125 of them are universal failure (XFAIL: *). Many of these have been marked this way for years. I suspect that if we take the time to go through these we will likely find that many of these test cases either should be tracked by bugs, or should be removed from the tree</div><div><br class=""></div><div>From there, many of the test cases are XFAIL on features where they really should be UNSUPPORTED. I suspect that if we do a full audit of the XFAILs we would likely find <100 which should actually be XFAIL, and tracking those seems valuable to me.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote"><div class=""><br class="">Auditing existing XFAILs can be done today without filing bugs for them.<br class=""></div></div></div></div></blockquote><div><br class=""></div><div>Yes, we can audit them today. Making bugs required for them makes it easier to audit them because there will (in theory) be a bug describing the justification for the XFAIL and what the underlying issue is. Digging up the reason why an XFAIL was added in 2009 is a little bit challenging today if there isn’t a PR associated with it (or a really good comment or commit message).</div><div><br class=""></div><div>-Chris</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote"><div class=""><br class="">And we still always have the option to (& in many cases do) file bugs for XFAILs to discuss them, etc.</div><div class=""><br class=""></div><div class="">But I don't feel strongly about it either way, so happy to leave the folks who do to make the call/do the work.<br class=""><br class="">- Dave</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">--paulr<br class="gmail_msg"><br class="gmail_msg">_______________________________________________<br class="gmail_msg">LLVM Developers mailing list<br class="gmail_msg"><a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" target="_blank">llvm-dev@lists.llvm.org</a><br class="gmail_msg"><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a></blockquote></div></div></div></blockquote></div><br class=""></body></html>