[all-commits] [llvm/llvm-project] f951b0: [lit] Add builtin support for flaky tests in lit

[Louis Dionne via All-commits]

  Branch: refs/heads/master
  Home:   https://github.com/llvm/llvm-project
  Commit: f951b0f82dff97f91d9b89c67886740acf3373cd
      https://github.com/llvm/llvm-project/commit/f951b0f82dff97f91d9b89c67886740acf3373cd
  Author: Louis Dionne <ldionne at apple.com>
  Date:   2020-03-18 (Wed, 18 Mar 2020)

  Changed paths:
    M llvm/utils/lit/lit/Test.py
    M llvm/utils/lit/lit/TestRunner.py
    A llvm/utils/lit/tests/Inputs/allow-retries/does-not-succeed-within-limit.py
    A llvm/utils/lit/tests/Inputs/allow-retries/lit.cfg
    A llvm/utils/lit/tests/Inputs/allow-retries/more-than-one-allow-retries-lines.py
    A llvm/utils/lit/tests/Inputs/allow-retries/not-a-valid-integer.py
    A llvm/utils/lit/tests/Inputs/allow-retries/succeeds-within-limit.py
    A llvm/utils/lit/tests/Inputs/test_retry_attempts/lit.cfg
    A llvm/utils/lit/tests/Inputs/test_retry_attempts/test.py
    M llvm/utils/lit/tests/Inputs/testrunner-custom-parsers/test.txt
    A llvm/utils/lit/tests/allow-retries.py
    M llvm/utils/lit/tests/unit/TestRunner.py
    M llvm/utils/vim/syntax/llvm.vim
    M llvm/utils/vscode/llvm/syntaxes/ll.tmLanguage.yaml

  Log Message:
  -----------
  [lit] Add builtin support for flaky tests in lit

This commit adds a new keyword in lit called ALLOW_RETRIES. This keyword
takes a single integer as an argument, and it allows the test to fail that
number of times before it first succeeds.

This work attempts to make the existing test_retry_attempts more flexible
by allowing by-test customization, as well as eliminate libc++'s FLAKY_TEST
custom logic.

Differential Revision: https://reviews.llvm.org/D76288