[llvm-dev] [RFC] Require PRs for XFAILing tests

[Chris Bieneman via llvm-dev]

> On Sep 29, 2016, at 2:52 AM, David Chisnall <David.Chisnall at cl.cam.ac.uk> wrote:
> 
> On 28 Sep 2016, at 17:39, Chris Bieneman via llvm-dev <llvm-dev at lists.llvm.org> wrote:
>> 
>> My proposal is to add this feature to LIT (after proper code review, and I still need to write tests for it), and then to annotate all our XFAILS with PRs. Once all the PRs are annotated I think we should enable this behavior by default and require PRs tracking all XFAILs.
> 
> This looks very useful for LLVM itself, but please consider downstream consumers with this feature.  We have a few XFAIL’d tests in our downstream fork and run llvm-lit in our CI setup.  We obviously can’t open an llvm.org bug for each one, because they don’t relate to bugs in upstream LLVM.  Please ensure that we have a mechanism for disabling this (or a different spelling of XFAIL for local XFAILs).

Absolutely. I expect to have two mechanisms for downstream users.

(1) The ability to specify your own bug tracker prefixes so that you can mark out-of-tree XFAILs with your own bugs if you’d like
(2) The ability to disable forcing failure. Additionally Matthias Braun suggested that I should add the ability to disable or add bug tracker URLs per-test suite, which I’ll be doing as well.

-Chris

> 
> David
> 
>