[llvm-dev] [RFC] Require PRs for XFAILing tests

[Matthias Braun via llvm-dev]

> On Sep 28, 2016, at 1:01 PM, Chris Bieneman via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> I think there are a few interesting things that could follow from solidifying a policy requiring PRs for XFAILs.
> 
> First and foremost bugs can have way more context than than you would often find in a test case comment. That would make it a lot easier to audit XFAILs in the future and help keep the number of XFAILs to a minimum. I think this is important because many of our XFAILs are really old, and I’m not convinced that we shouldn’t just be deleting some of these tests.
> 
> For example, 2008-12-14-StrideAndSigned.ll and 7 other tests were marked with “XFAIL: *” in 2009, and the commit message doesn’t really explain what was going on:
> 
>> commit 789558db70d9513a017c11c5be30945839fdff1c
>> Author: Nick Lewycky <nicholas at mxc.ca>
>> Date:   Tue Jan 13 09:18:58 2009 +0000
>> 
>>    Wind SCEV back in time, to Nov 18th. This 'fixes' PR3275, PR3294, PR3295,
>>    PR3296 and PR3302.
>> 
>> 
>>    git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@62160 91177308-0d34-0410-b5e6-96231b3b80d8
> 
> 
> Requiring a PR doesn’t necessarily fix this problem because you could list the wrong PR, or even just a generic stub PR that didn’t add meaningful value, but I think our community is pretty good at keeping people honest via code reviews.

Other important aspects:
- We can immediately start a specific and meaningful conversation about the problem
- Progress/blockers in solving the problem can be documented if necessary
- It is actually easy to find the corresponding discussion to a problem (the alternative of looking up the commit adding the XFAIL and then searching the commits mailing list for threads on that commit is more trouble)

- Matthias