[llvm-dev] [RFC] Require PRs for XFAILing tests

[Robinson, Paul via llvm-dev]

> -----Original Message-----
> From: Alex Bradbury [mailto:asb at asbradbury.org]
> Sent: Saturday, October 01, 2016 1:06 PM
> To: Robinson, Paul
> Cc: Renato Golin; Chris Bieneman; llvm-dev at lists.llvm.org
> Subject: Re: [llvm-dev] [RFC] Require PRs for XFAILing tests
> 
> On 28 September 2016 at 19:58, Robinson, Paul via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> > On 28 September 2016 at 10:08, Chris Bieneman via llvm-dev
> > <llvm-dev at lists.llvm.org> wrote:
> >> I cannot think of any situation where a universally failing test
> >> should be in-tree unless it is a bug that someone is expecting to fix.
> >
> > It seems moderately common to mark something XFAIL temporarily to get
> > the bots green while then going ahead to fix the issue.  Your proposal
> > would add extra overhead to that flow by requiring a PR as well.  This
> > has value when it turns out that fix can't happen in the short term for
> > any reason.  I don't have a feel for how common that is, although I'm
> > sure it does happen.
> > I think the overhead is worth the added value, but then I'm a process
> > kind of guy.
> 
> I'm not saying I _like_ this solution, but if that were an issue we
> could always have an open issue e.g. "PRNNNN: Some tests are marked
> XFAIL but only have this generic PR listed as the reason", for use in
> these "quick fix" cases. It would also be easy to track if these
> "quick fixes" didn't happen shortly.

As David Blaikie mentioned, our bug hygiene is not really that good.
It would be easy to find the set of tests citing the generic PR, but
somebody would have to take it upon themselves to go looking for them.
By the time that happened, the kinds of details we'd want to see in a
bug would be just as missing as if we had no XFAIL-to-PR link at all.

Conversely, requiring short-term XFAILs to have their own PR means
that if somebody fixed the test and forgot to close the PR, that
dangling PR would be easy to recognize as something that could be
summarily closed if anybody decided to go look at all the XFAIL-linked
PRs.  This scenario leaves an open PR kicking around, O the horror,
but we have not lost any useful information.

Now, I think it would be a great and useful thing for somebody to take
on the role of PR Czar, to do that kind of sanitization of the bugs,
but I don't see it happening as an ongoing role.  Therefore I prefer
a process that is a bit more tedious but doesn't lose information
over a simpler but lossy process.
--paulr

> 
> Alex