[all-commits] [llvm/llvm-project] 84fd90: [lit] add --max-retries-per-test execution option ...

[Konrad Kleine via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 84fd907aa7de2c35721ab6d1fd0d64d7c9345747
      https://github.com/llvm/llvm-project/commit/84fd907aa7de2c35721ab6d1fd0d64d7c9345747
  Author: Konrad Kleine <kkleine at redhat.com>
  Date:   2025-05-31 (Sat, 31 May 2025)

  Changed paths:
    M llvm/docs/CommandGuide/lit.rst
    M llvm/utils/lit/lit/LitConfig.py
    M llvm/utils/lit/lit/TestingConfig.py
    M llvm/utils/lit/lit/cl_arguments.py
    M llvm/utils/lit/lit/main.py
    A llvm/utils/lit/tests/Inputs/max-retries-per-test/allow-retries-no-test_retry_attempts/lit.cfg
    A llvm/utils/lit/tests/Inputs/max-retries-per-test/allow-retries-no-test_retry_attempts/test.py
    A llvm/utils/lit/tests/Inputs/max-retries-per-test/allow-retries-test_retry_attempts/lit.cfg
    A llvm/utils/lit/tests/Inputs/max-retries-per-test/allow-retries-test_retry_attempts/test.py
    A llvm/utils/lit/tests/Inputs/max-retries-per-test/no-allow-retries-no-test_retry_attempts/lit.cfg
    A llvm/utils/lit/tests/Inputs/max-retries-per-test/no-allow-retries-no-test_retry_attempts/test.py
    A llvm/utils/lit/tests/Inputs/max-retries-per-test/no-allow-retries-test_retry_attempts/lit.cfg
    A llvm/utils/lit/tests/Inputs/max-retries-per-test/no-allow-retries-test_retry_attempts/test.py
    M llvm/utils/lit/tests/allow-retries.py

  Log Message:
  -----------
  [lit] add --max-retries-per-test execution option (#141851)

When packaging LLVM we've seen arbitrary tests fail.
It happened sporadically and most of the times the test
do work if they are run a second time on the next day.

The tests themselves were always different and we didn't
know ahead of time which ones we wanted to re-run.
That's we filter-out a lot of `libomp` and `libarcher` tests [1].

This change allows us to set
`LIT_OPTS="--max-retries-per-test=12"`
when running any "check-XXX" build target. Then any lit test
will at most be re-run 12 times, unless there's an `ALLOW_RETRIES:`
in one of the test scripts that's specifying a different value
than `12`. `12` is just an example here, any positive integer
will work.

Please note, that this only adds the possibility to re-run
lit tests. It does not actually do it until the caller specifies
`--max-retries-per-test=<POSITIVE_INT>` either on a call to `lit` or
in `LIT_OPTS`.

Also note, that one can still use `ALLOW_RETRIES:` in test scripts
and it will always rule over `--max-retries-per-test`. When
`--max-retries-per-test` is set too low, but the
`config.test_retry_attempts`
is high enough, it works as well.

Any option in the list below overrules its predecessor:

* `--max-retries-per-test`
* `config.test_retry_attempts`
* `ALLOW_RETRIES` keyword

>From the above options to re-run tests, `--max-retries-per-test` is the
only one that doesn't require a change in the test scripts or the test
config.

[1]:
https://src.fedoraproject.org/rpms/llvm/blob/rawhide/f/llvm.spec#_2326

Downstream PR to make use of the `--max-retries-per-test` option:
https://src.fedoraproject.org/rpms/llvm/pull-request/434
Downstream ticket: https://issues.redhat.com/browse/LLVM-145



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications