[llvm] 2a5aa81 - [lit] Add --ignore-fail
Joel E. Denny via llvm-commits
llvm-commits at lists.llvm.org
Wed Feb 24 10:12:54 PST 2021
Author: Joel E. Denny
Date: 2021-02-24T13:10:27-05:00
New Revision: 2a5aa81739d31959600978c0f11332823010a754
URL: https://github.com/llvm/llvm-project/commit/2a5aa81739d31959600978c0f11332823010a754
DIFF: https://github.com/llvm/llvm-project/commit/2a5aa81739d31959600978c0f11332823010a754.diff
LOG: [lit] Add --ignore-fail
For some build configurations, `check-all` calls lit multiple times to
run multiple lit test suites. Most recently, I've found this to be
true when configuring openmp as part of `LLVM_ENABLE_RUNTIMES`, but
this is not the first time.
If one test suite fails, none of the remaining test suites run, so you
cannot determine if your patch has broken them. It can then be
frustrating to try to determine which `check-` targets will run the
remaining tests without getting stuck on the failing tests.
When such cases arise, it is probably best to adjust the cmake
configuration for `check-all` to run all test suites as part of one
lit invocation. Because that fix will likely not be implemented and
land immediately, this patch introduces `--ignore-fail` to serve as a
workaround for developers trying to see test results until it does
land:
```
$ LIT_OPTS=--ignore-fail ninja check-all
```
One problem with `--ignore-fail` is that it makes it challenging to
detect test failures in a script, perhaps in CI. This problem should
serve as motivation to actually fix the cmake configuration instead of
continuing to use `--ignore-fail` indefinitely.
Reviewed By: jhenderson, thopre
Differential Revision: https://reviews.llvm.org/D96371
Added:
llvm/utils/lit/tests/Inputs/ignore-fail/fail.txt
llvm/utils/lit/tests/Inputs/ignore-fail/lit.cfg
llvm/utils/lit/tests/Inputs/ignore-fail/unresolved.txt
llvm/utils/lit/tests/Inputs/ignore-fail/xfail.txt
llvm/utils/lit/tests/Inputs/ignore-fail/xpass.txt
llvm/utils/lit/tests/ignore-fail.py
Modified:
llvm/docs/CommandGuide/lit.rst
llvm/utils/lit/lit/cl_arguments.py
llvm/utils/lit/lit/main.py
Removed:
################################################################################
diff --git a/llvm/docs/CommandGuide/lit.rst b/llvm/docs/CommandGuide/lit.rst
index c071a3dbc4d0..7e61a276765b 100644
--- a/llvm/docs/CommandGuide/lit.rst
+++ b/llvm/docs/CommandGuide/lit.rst
@@ -154,6 +154,10 @@ EXECUTION OPTIONS
suite take the most time to execute. Note that this option is most useful
with ``-j 1``.
+.. option:: --ignore-fail
+
+ Exit with status zero even if some tests fail.
+
.. option:: --no-indirectly-run-check
Do not error if a test would not be run if the user had specified the
diff --git a/llvm/utils/lit/lit/cl_arguments.py b/llvm/utils/lit/lit/cl_arguments.py
index 62c46705867e..4d829659ea18 100644
--- a/llvm/utils/lit/lit/cl_arguments.py
+++ b/llvm/utils/lit/lit/cl_arguments.py
@@ -130,6 +130,10 @@ def parse_args():
execution_group.add_argument("--allow-empty-runs",
help="Do not fail the run if all tests are filtered out",
action="store_true")
+ execution_group.add_argument("--ignore-fail",
+ dest="ignoreFail",
+ action="store_true",
+ help="Exit with status zero even if some tests fail")
execution_group.add_argument("--no-indirectly-run-check",
dest="indirectlyRunCheck",
help="Do not error if a test would not be run if the user had "
diff --git a/llvm/utils/lit/lit/main.py b/llvm/utils/lit/lit/main.py
index 33ce1109b9ee..3f265446be2e 100755
--- a/llvm/utils/lit/lit/main.py
+++ b/llvm/utils/lit/lit/main.py
@@ -122,8 +122,11 @@ def main(builtin_params={}):
has_failure = any(t.isFailure() for t in discovered_tests)
if has_failure:
- sys.exit(1)
-
+ if opts.ignoreFail:
+ sys.stderr.write("\nExiting with status 0 instead of 1 because "
+ "'--ignore-fail' was specified.\n")
+ else:
+ sys.exit(1)
def create_params(builtin_params, user_params):
def parse(p):
diff --git a/llvm/utils/lit/tests/Inputs/ignore-fail/fail.txt b/llvm/utils/lit/tests/Inputs/ignore-fail/fail.txt
new file mode 100644
index 000000000000..15eb81a5f5e9
--- /dev/null
+++ b/llvm/utils/lit/tests/Inputs/ignore-fail/fail.txt
@@ -0,0 +1 @@
+RUN: false
diff --git a/llvm/utils/lit/tests/Inputs/ignore-fail/lit.cfg b/llvm/utils/lit/tests/Inputs/ignore-fail/lit.cfg
new file mode 100644
index 000000000000..d90883376416
--- /dev/null
+++ b/llvm/utils/lit/tests/Inputs/ignore-fail/lit.cfg
@@ -0,0 +1,6 @@
+import lit.formats
+config.name = 'ignore-fail'
+config.suffixes = ['.txt']
+config.test_format = lit.formats.ShTest()
+config.test_source_root = None
+config.test_exec_root = None
diff --git a/llvm/utils/lit/tests/Inputs/ignore-fail/unresolved.txt b/llvm/utils/lit/tests/Inputs/ignore-fail/unresolved.txt
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/llvm/utils/lit/tests/Inputs/ignore-fail/xfail.txt b/llvm/utils/lit/tests/Inputs/ignore-fail/xfail.txt
new file mode 100644
index 000000000000..6814cda40148
--- /dev/null
+++ b/llvm/utils/lit/tests/Inputs/ignore-fail/xfail.txt
@@ -0,0 +1,2 @@
+RUN: false
+XFAIL: *
diff --git a/llvm/utils/lit/tests/Inputs/ignore-fail/xpass.txt b/llvm/utils/lit/tests/Inputs/ignore-fail/xpass.txt
new file mode 100644
index 000000000000..66b8a6a5a187
--- /dev/null
+++ b/llvm/utils/lit/tests/Inputs/ignore-fail/xpass.txt
@@ -0,0 +1,2 @@
+RUN: true
+XFAIL: *
diff --git a/llvm/utils/lit/tests/ignore-fail.py b/llvm/utils/lit/tests/ignore-fail.py
new file mode 100644
index 000000000000..135e29baa5a6
--- /dev/null
+++ b/llvm/utils/lit/tests/ignore-fail.py
@@ -0,0 +1,19 @@
+# Check that --ignore-fail produces exit status 0 despite various kinds of
+# test failures but doesn't otherwise suppress those failures.
+
+# RUN: not %{lit} -j 1 %{inputs}/ignore-fail | FileCheck %s
+# RUN: %{lit} -j 1 --ignore-fail %{inputs}/ignore-fail | FileCheck %s
+
+# END.
+
+# CHECK: FAIL: ignore-fail :: fail.txt
+# CHECK: UNRESOLVED: ignore-fail :: unresolved.txt
+# CHECK: XFAIL: ignore-fail :: xfail.txt
+# CHECK: XPASS: ignore-fail :: xpass.txt
+
+# CHECK: Testing Time:
+# CHECK-NEXT: Expectedly Failed : 1
+# CHECK-NEXT: Unresolved : 1
+# CHECK-NEXT: Failed : 1
+# CHECK-NEXT: Unexpectedly Passed: 1
+# CHECK-NOT: {{.}}
More information about the llvm-commits
mailing list