[llvm-dev] [RFC] Require PRs for XFAILing tests

[Chris Bieneman via llvm-dev]

Hello LLVM-Dev,

The other day as I was digging through lldb’s test suite I noticed they support something kinda neat. In their python test harness, the attribute they use to denote expected failures supports a parameter for specifying the bug number. This got me thinking.

I believe that any test that is marked XFAIL is a bug, and we can use LIT to enforce that. So I wrote a patch (https://reviews.llvm.org/D25035 <https://reviews.llvm.org/D25035>) to add a feature to LIT which would support mapping XFAILs to PRs, and a flag to turn XFAILS without PRs into failures.

My proposal is to add this feature to LIT (after proper code review, and I still need to write tests for it), and then to annotate all our XFAILS with PRs. Once all the PRs are annotated I think we should enable this behavior by default and require PRs tracking all XFAILs.

Thoughts?
-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160928/f8848b9c/attachment.html>