[PATCH] D96371: [lit] Add --ignore-fail

[Joel E. Denny via Phabricator via llvm-commits]

jdenny added a comment.

In D96371#2564848 <https://reviews.llvm.org/D96371#2564848>, @jhenderson wrote:

> I'm coming around to the benefits of this, but first have you considered whether the D96662 <https://reviews.llvm.org/D96662> would be sufficient for your needs? It seems that you could use that to ignore/xfail the bad tests and continue,

`--xfail` could prove frustrating for my current use case.  The problem is that openmp test failures are often racy.  I'd have to run check-all potentially many times trying to make my `--xfail` match the failing tests.

`--skip` avoids that problem because it doesn't care about the specified tests' results.  But I want to see those results, so I'd need to run check-all twice, once with the problematic tests, and once skipping them.  Seems ok, but...

> without using --ignore-fail and risk accidentally missing failures to do with your work (possibly in more obscure places).

This point almost convinced me in the case of `--skip`.  However, one advantage of `--ignore-fail` is that it makes it possible to just simply run everything.  With `--ignore-fail`, I don't have to figure out the right pattern that skips just tests in the first lit invocation without accidentally skipping other tests I didn't realize matched my pattern.  For example, some openmp-related patterns would match in both lit invocations because clang's test suite runs in the second lit invocation.  With `--ignore-fail`, I don't have to run check-all (almost 2 hours on my laptop) many times trying to get the `--skip` pattern just right.  I won't get through the second lit invocation and think I'm done without realizing a third lit invocation has been introduced (no, I'm not aware of a third right now).  `--ignore-fail` just runs everything.

The caveat is that I might miss test failures in the scroll, but anyone using `--ignore-fail` should know they're ignoring failures, so surely they're on the lookout.

To be clear, I think `--xfail` and `--skip` will prove useful for other purposes, especially when you're investigating a smaller set of tests or trying to write careful scripts.  In contrast, check-all can be huge.  I don't want to tinker trying to make it run everything.  After waiting two hours, I just want to see the test results... especially if I previously ran it once, came back two hours later, and discovered it skipped entire test suites.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D96371/new/

https://reviews.llvm.org/D96371