[llvm-dev] [lit] XFAIL individual RUN lines?

[James Henderson via llvm-dev]

It's not possible to XFAIL individual RUN lines any more than it is
possible to mark individual lines with the UNSUPPORTED keyword (though in
the latter case you could always just comment it out somehow, e.g. using
the COM: directive). To do that with the current test infrastructure, you'd
need to split the test into multiple files, I believe.

You could use the `not` command to invert the expected result, but that
would cause the test to PASS rather than XFAIL, which as I understand it,
isn't what you want.

One option to extend lit might be to add some kind of `--keep-going`
command which causes a test to fully execute even if a line in it failed.
That would allow you to use the global XFAIL, but still get the testing of
the subtargets (although you'd still have to display and trawl the failing
test output to see what exactly failed).


On Thu, 23 Sept 2021 at 19:22, Stephen Neuendorffer via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> I've wanted something similar, but for REQUIRES: lines, so that each RUN
> line could have a different set of required features in order to execute.
>
> Steve
>
> On Thu, Sep 23, 2021 at 8:42 AM David Greene via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> We have a number of tests where we want to run the same code through
>> different subtargets:
>>
>> # RUN: llc -mcpu=<subtarget1> ...
>> # RUN: llc -mcpu=<subtarget2> ...
>>
>> Not all subtargets are expected to pass, but some will pass.  We'd like
>> to know which tests are passing for which subtargets.  We don't want to
>> XFAIL the whole test (because it would stop testing of working
>> subtargets) but we do what to know which subtargets need work.
>>
>> Is it possible to somehow XFAIL an individual RUN line?  The
>> documentation seems to imply it's all-or-nothing.
>>
>> Alternatively, is it possible to get a test summary that shows PASS/FAIL
>> for individual RUN lines rather than for the test as a whole?
>>
>> If neither of these is possible, would patches to implement them be
>> welcome?
>>
>> Thanks for your help!
>>
>>                  -David
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210924/3ac1ea99/attachment.html>