[Openmp-commits] [openmp] [openmp] Allow to re-run tests (PR #141851)

[Konrad Kleine via Openmp-commits]

https://github.com/kwk created https://github.com/llvm/llvm-project/pull/141851

When packaging LLVM we've seen arbitrary tests failing. It happened sporadically and most of the times the test do work if they are run a second time on the next day.

The tests themselves were always different and we didn't know ahead of time which ones we wanted to re-run. That's we
[filter-out](https://src.fedoraproject.org/rpms/llvm/blob/rawhide/f/llvm.spec#_2326) a lot of `libomp` and `libarcher` tests.

This changes allows us to set `LIT_OPTS="-Dmaximum_retries_per_test=12"` when running the "check-openmp" build target. Then any test in the `openmp` directory will at most be re-run 12 times. 12 is just an example here and it is up to the caller of the build target to define this.

Please note, that only adds the possibility to re-run openmp tests. It does not actually do it until the caller specifies `-Dmaximum_retries_per_test=<INT>` either on a call to `lit` or in `LIT_OPTS`.

Downstream ticket: https://issues.redhat.com/browse/LLVM-145

>From 8c097a400e013b525edbf1577503253d4d5c4f14 Mon Sep 17 00:00:00 2001
From: Konrad Kleine <kkleine at redhat.com>
Date: Wed, 28 May 2025 20:17:29 +0000
Subject: [PATCH] [openmp] Allow to re-run tests

When packaging LLVM we've seen arbitrary tests failing. It happened
sporadically and most of the times the test do work if they are run a
second time on the next day.

The tests themselves were always different and we didn't know ahead
of time which ones we wanted to re-run. That's we
[filter-out](https://src.fedoraproject.org/rpms/llvm/blob/rawhide/f/llvm.spec#_2326)
a lot of `libomp` and `libarcher` tests.

This changes allows us to set `LIT_OPTS="-Dmaximum_retries_per_test=12"`
when running the "check-openmp" build target. Then any test in the `openmp`
directory will at most be re-run 12 times. 12 is just an example here and
it is up to the caller of the build target to define this.

Please note, that only adds the possibility to re-run openmp tests. It
does not actually do it until the caller specifies
`-Dmaximum_retries_per_test=<INT>` either on a call to `lit` or in
`LIT_OPTS`.

Downstream ticket: https://issues.redhat.com/browse/LLVM-145
---
 openmp/libompd/test/lit.cfg          | 4 ++++
 openmp/runtime/test/lit.cfg          | 4 ++++
 openmp/tools/archer/tests/lit.cfg    | 4 ++++
 openmp/tools/multiplex/tests/lit.cfg | 3 +++
 4 files changed, 15 insertions(+)

diff --git a/openmp/libompd/test/lit.cfg b/openmp/libompd/test/lit.cfg
index df881d026c899..a25ed218ab990 100644
--- a/openmp/libompd/test/lit.cfg
+++ b/openmp/libompd/test/lit.cfg
@@ -81,3 +81,7 @@ config.substitutions.append(("%ompt-tool",
 
 config.substitutions.append(("FileCheck", config.test_filecheck))
 
+maximum_retries_per_test = lit_config.params.get("maximum_retries_per_test", None)
+if maximum_retries_per_test is not None:
+    config.test_retry_attempts = int(maximum_retries_per_test)
+
diff --git a/openmp/runtime/test/lit.cfg b/openmp/runtime/test/lit.cfg
index cfbd2c5d418b5..bf8a2fa30b6ad 100644
--- a/openmp/runtime/test/lit.cfg
+++ b/openmp/runtime/test/lit.cfg
@@ -226,3 +226,7 @@ if config.has_ompt:
         config.substitutions.append(("%no-as-needed-flag", "-Wl,--no-as-needed"))
 else:
     config.substitutions.append(("FileCheck", config.test_filecheck))
+
+maximum_retries_per_test = lit_config.params.get("maximum_retries_per_test", None)
+if maximum_retries_per_test is not None:
+    config.test_retry_attempts = int(maximum_retries_per_test)
diff --git a/openmp/tools/archer/tests/lit.cfg b/openmp/tools/archer/tests/lit.cfg
index f8fbcad752a4c..1cdb1399a38d4 100644
--- a/openmp/tools/archer/tests/lit.cfg
+++ b/openmp/tools/archer/tests/lit.cfg
@@ -138,3 +138,7 @@ elif config.operating_system == 'Darwin':
     config.substitutions.append(("%preload-tool", "env DYLD_INSERT_LIBRARIES=%T/tool.so"))
 else:
     config.substitutions.append(("%preload-tool", "env LD_PRELOAD=%T/tool.so"))
+
+maximum_retries_per_test = lit_config.params.get("maximum_retries_per_test", None)
+if maximum_retries_per_test is not None:
+    config.test_retry_attempts = int(maximum_retries_per_test)
diff --git a/openmp/tools/multiplex/tests/lit.cfg b/openmp/tools/multiplex/tests/lit.cfg
index 459250582ae95..de7e3ac755bc5 100644
--- a/openmp/tools/multiplex/tests/lit.cfg
+++ b/openmp/tools/multiplex/tests/lit.cfg
@@ -103,3 +103,6 @@ config.substitutions.append(("%clang", config.test_c_compiler))
 config.substitutions.append(("%openmp_flag", config.test_openmp_flags))
 config.substitutions.append(("%cflags", config.test_flags))
 
+maximum_retries_per_test = lit_config.params.get("maximum_retries_per_test", None)
+if maximum_retries_per_test is not None:
+    config.test_retry_attempts = int(maximum_retries_per_test)
\ No newline at end of file