[llvm] [lit] Add --rerun-failed-serially <REGEX> mechanism (PR #223504)

[Adrian Prantl via llvm-commits]

================
@@ -237,6 +237,25 @@ require a change in the test scripts or the test config:
 Any option in the list above overrules its predecessor.
 :::
 
+:::{option} --rerun-failed-serially
+Once all tests have run, run the tests that failed a second time, with a
+single worker regardless of {option}`-j`. Tests that pass the second time
+are reported as `FLAKYPASS`, so they do not count as failures and do not
+affect the exit status.
+
+{option}`--max-retries-per-test` retries a failed test immediately, in the
+same worker and alongside the tests that are still running; this option
+waits until nothing else is running. Both can be used at once, in which
+case each attempt of the rerun is retried as well.
+:::
+
+:::{option} --rerun-failed-matching REGEX
+Only rerun those failures whose output matches `REGEX`, to allow
+working around very specific known issues without allowing other
+tests to become unreliable due to the retry mechanism. Implies
+{option}`--rerun-failed-serially`.
+:::
----------------
adrian-prantl wrote:

@JDevlieghere I merged the two options, please take another look.

As for reusing the `--max-retries-per-test` as a serial retry count isn't a great fit. My understanding is that the flag currently enables an unconditional retry for all tests (which tests also can manually opt into with ALLOW_RETRIES). So either we'd change the semantics of the option in the presence of the other one, or setting the flags would enable parallel retries in addition to enabling serial retries, which is just confusing behavior. I would like to punt on that decision unless it turns out it's really needed.

https://github.com/llvm/llvm-project/pull/223504