[PATCH] D76288: [lit] Add builtin support for flaky tests in lit

[Louis Dionne via Phabricator via llvm-commits]

ldionne added a comment.

Note that an alternative design would be to drop the integer from `ALLOW_FAILURES:` and make it a tag keyword only. Then, `config.test_retry_attempts` could be applied only on the tests that are marked with that keyword. I actually prefer this approach, however it is a behavior change for existing users of `config.test_retry_attempts`.

Also, I'm open to changing `ALLOW_FAILURES` to something else, like `ALLOW_RETRIES`. Suggestions welcome.

Finally, note that this doesn't handle `XFAIL` tests that would succeed repeatedly. This is actually an existing condition of `config.test_retry_attempts`, and the root cause is that we translate test results higher up in the call chain, near `_execute` in `worker.py`. Unfortunately, handling `ALLOW_FAILURES` at that point involves parsing the test several times, which messes up things like `test.xfails`. Since this is a pre-existing condition, I did not try to fix it cause it would have required a probably-breaking refactoring.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D76288/new/

https://reviews.llvm.org/D76288