[PATCH] D96662: [lit] Add --skip (inverse of --filter) and `--xfail`
Julian Lettner via Phabricator via llvm-commits
llvm-commits at lists.llvm.org
Thu Feb 18 11:30:17 PST 2021
yln added a comment.
`--skip`: is it okay if we rename this to `--filter-not`? It's an uglier name but
- more "self explanatory", i..e, "opposite of `--filter`". Anyone who knows `--filter` will know what it does just by looking the arg name.
- Discoverable: in the usage and help output, it will appear next to `--filter`.
- In the summary, we have a category of `Skipped` tests: tests that were not run because the user hit `[Ctrl+C]` or because of overall timeout. `--skip`ped tests would appear in the `Excluded` category (e.g., `--filter` and sharding feature). Not using the word skip may help prevent some confusion.
Can we also add the summary counts in the test, so that the test shows that `--filter-not` excluded tests are counted in the `Excluded` category? Thanks!
Otherwise, LGTM, will approve today if you split this part out.
`--xfail`: Thanks everyone for explaining. As someone who also has to deal with downstream test failures, you convinced me that this is useful to have!
Summarizing my understanding: this is about the desire to (temporarily) mark tests as `XFAIL` in a centralized location (instead of editing individual test files), especially useful for a downstream integration workflows.
@davezarzycki @jhenderson @thopre Is this accurate?
How about `--xfail-file` which names a file for which each line means one XFAILed test. I imagine that:
- Working with a file seems better (`echo "test/path" >> xfail.txt`, `sed '/test/d' ./file`), than fiddling with the lit invocation itself. Count how many XFAILed tests we have in upstream without even opening the file, etc.
- The file would "make sense" in version control: line-wise addition/removals. "Didn't this test fail before? What was the issue previously? `git log -- xfail.txt` and see commit for fix and line removal.
What do you think?
Repository:
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
https://reviews.llvm.org/D96662/new/
https://reviews.llvm.org/D96662
More information about the llvm-commits
mailing list