[llvm] 161e826 - [lit] Add "early_tests" config option

[David Zarzycki via llvm-commits]

Author: David Zarzycki
Date: 2021-02-17T06:32:04-05:00
New Revision: 161e826c586eefad06751aa35e2c0b8070a70bb1

URL: https://github.com/llvm/llvm-project/commit/161e826c586eefad06751aa35e2c0b8070a70bb1
DIFF: https://github.com/llvm/llvm-project/commit/161e826c586eefad06751aa35e2c0b8070a70bb1.diff

LOG: [lit] Add "early_tests" config option

With enough cores, the slowest tests can significantly change the total testing time if they happen to run late. With this change, a test suite can improve performance (for high-end systems) by listing just a few of the slowest tests up front.

Reviewed By: jdenny, jhenderson

Differential Revision: https://reviews.llvm.org/D96594

Added: 
    llvm/utils/lit/tests/Inputs/early-tests/aaa.txt
    llvm/utils/lit/tests/Inputs/early-tests/bbb.txt
    llvm/utils/lit/tests/Inputs/early-tests/lit.cfg
    llvm/utils/lit/tests/Inputs/early-tests/subdir/ccc.txt
    llvm/utils/lit/tests/early-tests.py

Modified: 
    llvm/docs/CommandGuide/lit.rst
    llvm/utils/lit/lit/Test.py
    llvm/utils/lit/lit/TestingConfig.py

Removed: 
    


################################################################################
diff  --git a/llvm/docs/CommandGuide/lit.rst b/llvm/docs/CommandGuide/lit.rst
index 3820cc446c3c..9a8135359967 100644
--- a/llvm/docs/CommandGuide/lit.rst
+++ b/llvm/docs/CommandGuide/lit.rst
@@ -397,6 +397,13 @@ executed, two important global variables are predefined:
  **root** The root configuration.  This is the top-most :program:`lit` configuration in
  the project.
 
+ **is_early** Whether the test suite as a whole should be given a head start
+ before other test suites run.
+
+ **early_tests** An explicit set of '/' separated test paths that should be
+ given a head start before other tests run. For example, the top five or so
+ slowest tests. See also: `--time-tests`
+
  **pipefail** Normally a test using a shell pipe fails if any of the commands
  on the pipe fail. If this is not desired, setting this variable to false
  makes the test fail only if the last command in the pipe fails.

diff  --git a/llvm/utils/lit/lit/Test.py b/llvm/utils/lit/lit/Test.py
index 59fefbc7f089..ce87cfa8abb5 100644
--- a/llvm/utils/lit/lit/Test.py
+++ b/llvm/utils/lit/lit/Test.py
@@ -404,4 +404,6 @@ def isEarlyTest(self):
         This can be used for test suites with long running tests to maximize
         parallelism or where it is desirable to surface their failures early.
         """
+        if '/'.join(self.path_in_suite) in self.suite.config.early_tests:
+            return True
         return self.suite.config.is_early

diff  --git a/llvm/utils/lit/lit/TestingConfig.py b/llvm/utils/lit/lit/TestingConfig.py
index d40ee37636f3..fafc754c1bc1 100644
--- a/llvm/utils/lit/lit/TestingConfig.py
+++ b/llvm/utils/lit/lit/TestingConfig.py
@@ -127,6 +127,8 @@ def __init__(self, parent, name, suffixes, test_format,
         self.limit_to_features = set(limit_to_features)
         # Whether the suite should be tested early in a given run.
         self.is_early = bool(is_early)
+        # List of tests to run early.
+        self.early_tests = {}
         self.parallelism_group = parallelism_group
         self._recursiveExpansionLimit = None
 

diff  --git a/llvm/utils/lit/tests/Inputs/early-tests/aaa.txt b/llvm/utils/lit/tests/Inputs/early-tests/aaa.txt
new file mode 100644
index 000000000000..b80b60b7a279
--- /dev/null
+++ b/llvm/utils/lit/tests/Inputs/early-tests/aaa.txt
@@ -0,0 +1 @@
+# RUN: true

diff  --git a/llvm/utils/lit/tests/Inputs/early-tests/bbb.txt b/llvm/utils/lit/tests/Inputs/early-tests/bbb.txt
new file mode 100644
index 000000000000..b80b60b7a279
--- /dev/null
+++ b/llvm/utils/lit/tests/Inputs/early-tests/bbb.txt
@@ -0,0 +1 @@
+# RUN: true

diff  --git a/llvm/utils/lit/tests/Inputs/early-tests/lit.cfg b/llvm/utils/lit/tests/Inputs/early-tests/lit.cfg
new file mode 100644
index 000000000000..db030510c249
--- /dev/null
+++ b/llvm/utils/lit/tests/Inputs/early-tests/lit.cfg
@@ -0,0 +1,7 @@
+import lit.formats
+config.name = 'early-tests'
+config.suffixes = ['.txt']
+config.test_format = lit.formats.ShTest()
+config.test_source_root = None
+config.test_exec_root = None
+config.early_tests = { "subdir/ccc.txt" }

diff  --git a/llvm/utils/lit/tests/Inputs/early-tests/subdir/ccc.txt b/llvm/utils/lit/tests/Inputs/early-tests/subdir/ccc.txt
new file mode 100644
index 000000000000..b80b60b7a279
--- /dev/null
+++ b/llvm/utils/lit/tests/Inputs/early-tests/subdir/ccc.txt
@@ -0,0 +1 @@
+# RUN: true

diff  --git a/llvm/utils/lit/tests/early-tests.py b/llvm/utils/lit/tests/early-tests.py
new file mode 100644
index 000000000000..b2ca9ac0a97d
--- /dev/null
+++ b/llvm/utils/lit/tests/early-tests.py
@@ -0,0 +1,9 @@
+## Check that we can run tests early.
+
+# RUN: %{lit} -j1 %{inputs}/early-tests | FileCheck %s
+
+# CHECK:     -- Testing: 3 tests, 1 workers --
+# CHECK-NEXT: PASS: early-tests :: subdir/ccc.txt
+# CHECK-NEXT: PASS: early-tests :: aaa.txt
+# CHECK-NEXT: PASS: early-tests :: bbb.txt
+# CHECK:     Passed: 3