[PATCH] D76288: [lit] Add builtin support for flaky tests in lit

[Louis Dionne via Phabricator via llvm-commits]

ldionne created this revision.
ldionne added reviewers: delcypher, rnk.
Herald added subscribers: llvm-commits, dexonsmith, jkorous.
Herald added a project: LLVM.
ldionne added a comment.
ldionne added a subscriber: EricWF.

Note that an alternative design would be to drop the integer from `ALLOW_FAILURES:` and make it a tag keyword only. Then, `config.test_retry_attempts` could be applied only on the tests that are marked with that keyword. I actually prefer this approach, however it is a behavior change for existing users of `config.test_retry_attempts`.

Also, I'm open to changing `ALLOW_FAILURES` to something else, like `ALLOW_RETRIES`. Suggestions welcome.

Finally, note that this doesn't handle `XFAIL` tests that would succeed repeatedly. This is actually an existing condition of `config.test_retry_attempts`, and the root cause is that we translate test results higher up in the call chain, near `_execute` in `worker.py`. Unfortunately, handling `ALLOW_FAILURES` at that point involves parsing the test several times, which messes up things like `test.xfails`. Since this is a pre-existing condition, I did not try to fix it cause it would have required a probably-breaking refactoring.


This commit adds a new keyword in lit called ALLOW_FAILURES. This keyword
takes a single integer as an argument, and it allows the test to fail that
number of times before it first succeeds.

This work attempts to make the existing test_retry_attempts more flexible
by allowing by-test customization, as well as eliminate libc++'s FLAKY_TEST
custom logic.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D76288

Files:
  llvm/utils/lit/lit/Test.py
  llvm/utils/lit/lit/TestRunner.py
  llvm/utils/lit/tests/Inputs/allowed-failures/does-not-succeed-within-limit.py
  llvm/utils/lit/tests/Inputs/allowed-failures/lit.cfg
  llvm/utils/lit/tests/Inputs/allowed-failures/more-than-one-allow-failures-lines.py
  llvm/utils/lit/tests/Inputs/allowed-failures/not-a-valid-integer.py
  llvm/utils/lit/tests/Inputs/allowed-failures/succeeds-within-limit.py
  llvm/utils/lit/tests/Inputs/testrunner-custom-parsers/test.txt
  llvm/utils/lit/tests/allowed-failures.py
  llvm/utils/lit/tests/unit/TestRunner.py

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D76288.250784.patch
Type: text/x-patch
Size: 11033 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20200317/8b18686b/attachment.bin>