[PATCH] D96662: [lit] Add --skip (inverse of --filter) and `--xfail`

Wed Feb 17 03:12:13 PST 2021

davezarzycki added a comment.

In D96662#2567744 <https://reviews.llvm.org/D96662#2567744>, @jhenderson wrote:

> In D96662#2566903 <https://reviews.llvm.org/D96662#2566903>, @yln wrote:
>
>> I can get behind `--skip` (patch LGTM), but would a little help to understand when `--xfail=path/to/test1;path/to/test2` would be useful (which workflow?).  Thanks!
>
> One example where it would be useful - in our downstream merge process, we may identify a test that has started failing due to an upstream bug. We could modify the test explicitly by adding XFAIL to the start of it, or we could add it to the build script that is used to build and test the changes before they are merged in. The --xfail option allows the latter to happen in such a way that when the issue is fixed by a later upstream change, we'll notice (due to the test starting to pass) and remove the option from the script, whereas with --skip, we won't notice (and thus lose the testing downstream the test gave us). I could imagine other cases along these lines too, where --xfail is superior to --skip.

Indeed, this is the problem I'm trying to solve.

> In D96662#2565257 <https://reviews.llvm.org/D96662#2565257>, @davezarzycki wrote:
>
>> If I might critique myself, I'm not sure that regular expression semantics are best for `--xfail`. One really wants and should use a precise list of some kind. Any thoughts on what that might look like? I'd lean toward a semicolon separated list.
>
> A precise list would make sense. I'd use commas rather than semicolons, because semicolons are more likely to require shell escaping, I think (commas are less likely to).

In my experience, quotes solve most "escaping" problems, so what is used as a separator mostly doesn't matter as long as the character isn't used for variable expansion. Using the shell's statement separator is a good choice because shell users/programmers naturally avoid files with that character embedded in the name.

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D96662/new/

https://reviews.llvm.org/D96662