[PATCH] D84760: Revert: "[lit] Remove --repeat option, which wasn't that useful."

[Jon Roelofs via Phabricator via llvm-commits]

jroelofs created this revision.
jroelofs added a reviewer: ddunbar.
Herald added subscribers: llvm-commits, delcypher.
Herald added a project: LLVM.
jroelofs requested review of this revision.

This reverts commit b647d5d21dd8fa22b6ff849563809b7763dcf57e.

Repeating tests this way isn't all that useful for gathering timing information, but it is quite useful for re-running flaky tests (for example when running a git-bisect script).


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D84760

Files:
  llvm/docs/CommandGuide/lit.rst
  llvm/utils/lit/lit/Test.py
  llvm/utils/lit/lit/TestRunner.py
  llvm/utils/lit/lit/cl_arguments.py
  llvm/utils/lit/lit/main.py


Index: llvm/utils/lit/lit/main.py
===================================================================
--- llvm/utils/lit/lit/main.py
+++ llvm/utils/lit/lit/main.py
@@ -66,6 +66,11 @@
 
     determine_order(discovered_tests, opts.order)
 
+    if opts.repeatTests:
+        discovered_tests = [t.copyWithIndex(i)
+                            for t in discovered_tests
+                            for i in range(opts.repeatTests)]
+
     selected_tests = [t for t in discovered_tests if
                       opts.filter.search(t.getFullName())]
     if not selected_tests:
Index: llvm/utils/lit/lit/cl_arguments.py
===================================================================
--- llvm/utils/lit/lit/cl_arguments.py
+++ llvm/utils/lit/lit/cl_arguments.py
@@ -163,6 +163,11 @@
     debug_group.add_argument("--show-tests",
             help="Show all discovered tests and exit",
             action="store_true")
+    debug_group.add_argument("--repeat",
+            dest="repeatTests",
+            metavar="N",
+            help="Repeat tests N times",
+            type=_positive_int)
     debug_group.add_argument("--show-used-features",
             help="Show all features used in the test suite (in XFAIL, UNSUPPORTED and REQUIRES) and exit",
             action="store_true")
Index: llvm/utils/lit/lit/TestRunner.py
===================================================================
--- llvm/utils/lit/lit/TestRunner.py
+++ llvm/utils/lit/lit/TestRunner.py
@@ -1061,6 +1061,8 @@
     execdir,execbase = os.path.split(execpath)
     tmpDir = os.path.join(execdir, 'Output')
     tmpBase = os.path.join(tmpDir, execbase)
+    if test.index is not None:
+        tmpBase += '_%d' % test.index
     return tmpDir, tmpBase
 
 def colonNormalizePath(path):
Index: llvm/utils/lit/lit/Test.py
===================================================================
--- llvm/utils/lit/lit/Test.py
+++ llvm/utils/lit/lit/Test.py
@@ -244,6 +244,15 @@
         # The test result, once complete.
         self.result = None
 
+        # The repeat index of this test, or None.
+        self.index = None
+
+    def copyWithIndex(self, index):
+        import copy
+        res = copy.copy(self)
+        res.index = index
+        return res
+
     def setResult(self, result):
         assert self.result is None, "result already set"
         assert isinstance(result, Result), "unexpected result type"
Index: llvm/docs/CommandGuide/lit.rst
===================================================================
--- llvm/docs/CommandGuide/lit.rst
+++ llvm/docs/CommandGuide/lit.rst
@@ -224,6 +224,11 @@
 
  List all of the discovered tests and exit.
 
+.. option:: --repeat=N
+
+ Run each test ``N`` times.  Currently this is primarily useful for catching
+ failures in flaky tests.
+
 EXIT STATUS
 -----------
 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D84760.281244.patch
Type: text/x-patch
Size: 2811 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20200728/c096fde4/attachment.bin>