[llvm-dev] [RFC] Require PRs for XFAILing tests

[Mehdi Amini via llvm-dev]

> On Sep 28, 2016, at 9:39 AM, Chris Bieneman via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> Hello LLVM-Dev,
> 
> The other day as I was digging through lldb’s test suite I noticed they support something kinda neat. In their python test harness, the attribute they use to denote expected failures supports a parameter for specifying the bug number. This got me thinking.
> 
> I believe that any test that is marked XFAIL is a bug, and we can use LIT to enforce that. So I wrote a patch (https://reviews.llvm.org/D25035 <https://reviews.llvm.org/D25035>) to add a feature to LIT which would support mapping XFAILs to PRs, and a flag to turn XFAILS without PRs into failures.
> 
> My proposal is to add this feature to LIT (after proper code review, and I still need to write tests for it), and then to annotate all our XFAILS with PRs. Once all the PRs are annotated I think we should enable this behavior by default and require PRs tracking all XFAILs.
> 
> Thoughts?

I think that’s a great idea!

I wonder if there won’t be annoying cases that won’t fit well though, and a fake PR may be inserted just to please the system.
Have you survey the existing XFAIL test to see if it would be the case? (I can’t imagine why but…)

Thanks,

— 
Mehdi

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160928/14464b5e/attachment.html>