[llvm-dev] [RFC] Require PRs for XFAILing tests
Chris Bieneman via llvm-dev
llvm-dev at lists.llvm.org
Thu Sep 29 10:28:16 PDT 2016
> On Sep 29, 2016, at 2:52 AM, David Chisnall <David.Chisnall at cl.cam.ac.uk> wrote:
> On 28 Sep 2016, at 17:39, Chris Bieneman via llvm-dev <llvm-dev at lists.llvm.org> wrote:
>> My proposal is to add this feature to LIT (after proper code review, and I still need to write tests for it), and then to annotate all our XFAILS with PRs. Once all the PRs are annotated I think we should enable this behavior by default and require PRs tracking all XFAILs.
> This looks very useful for LLVM itself, but please consider downstream consumers with this feature. We have a few XFAIL’d tests in our downstream fork and run llvm-lit in our CI setup. We obviously can’t open an llvm.org bug for each one, because they don’t relate to bugs in upstream LLVM. Please ensure that we have a mechanism for disabling this (or a different spelling of XFAIL for local XFAILs).
Absolutely. I expect to have two mechanisms for downstream users.
(1) The ability to specify your own bug tracker prefixes so that you can mark out-of-tree XFAILs with your own bugs if you’d like
(2) The ability to disable forcing failure. Additionally Matthias Braun suggested that I should add the ability to disable or add bug tracker URLs per-test suite, which I’ll be doing as well.
More information about the llvm-dev