[PATCH] D96371: [lit] Add --ignore-fail

[Joel E. Denny via Phabricator via llvm-commits]

jdenny created this revision.
jdenny added reviewers: yln, thopre, probinson, jhenderson.
Herald added a subscriber: delcypher.
jdenny requested review of this revision.
Herald added a reviewer: jdoerfert.
Herald added a subscriber: sstefan1.
Herald added a project: LLVM.

For some build configurations, check-all runs multiple separate lit 
test suites.  Most recently, I've found this to be true when
configuring openmp as part of `LLVM_ENABLE_RUNTIMES`, but this is not 
the first time.

If one test suite fails, none of the remaining test suites run, so you 
cannot determine if your patch has broken them.  This is annoying
enough on the command line, where it can be challenging to realize
that some test suites didn't run or to figure out which ones,
especially as LLVM's build configuration evolves.  It's even worse in
CI bots where most people do not have privileges for running
additional test suites.

With this patch applied, you can make sure all test suites run as
follows:

  $ LIT_OPTS=--ignore-fail ninja check-all

Of course, you then cannot use the check-all exit status to detect
failures, which is problematic for CI.  Fortunately, you can still
grep the output or a generated report, such as the one produced by
`-o`.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D96371

Files:
  llvm/utils/lit/lit/cl_arguments.py
  llvm/utils/lit/lit/main.py
  llvm/utils/lit/tests/Inputs/ignore-fail/fail.txt
  llvm/utils/lit/tests/Inputs/ignore-fail/lit.cfg
  llvm/utils/lit/tests/Inputs/ignore-fail/unresolved.txt
  llvm/utils/lit/tests/Inputs/ignore-fail/xfail.txt
  llvm/utils/lit/tests/Inputs/ignore-fail/xpass.txt
  llvm/utils/lit/tests/ignore-fail.py


Index: llvm/utils/lit/tests/ignore-fail.py
===================================================================
--- /dev/null
+++ llvm/utils/lit/tests/ignore-fail.py
@@ -0,0 +1,19 @@
+# Check that --ignore-fail produces exit status 0 despite various kinds of
+# test failures but doesn't otherwise suppress those failures.
+
+# RUN: not %{lit} -j 1 %{inputs}/ignore-fail | FileCheck %s
+# RUN: %{lit} -j 1 --ignore-fail %{inputs}/ignore-fail | FileCheck %s
+
+# END.
+
+# CHECK: FAIL: ignore-fail :: fail.txt
+# CHECK: UNRESOLVED: ignore-fail :: unresolved.txt
+# CHECK: XFAIL: ignore-fail :: xfail.txt
+# CHECK: XPASS: ignore-fail :: xpass.txt
+
+#      CHECK: Testing Time:
+# CHECK-NEXT:   Expectedly Failed : 1
+# CHECK-NEXT:   Unresolved : 1
+# CHECK-NEXT:   Failed : 1
+# CHECK-NEXT:   Unexpectedly Passed: 1
+#  CHECK-NOT: {{.}}
Index: llvm/utils/lit/tests/Inputs/ignore-fail/xpass.txt
===================================================================
--- /dev/null
+++ llvm/utils/lit/tests/Inputs/ignore-fail/xpass.txt
@@ -0,0 +1,2 @@
+RUN: true
+XFAIL: *
Index: llvm/utils/lit/tests/Inputs/ignore-fail/xfail.txt
===================================================================
--- /dev/null
+++ llvm/utils/lit/tests/Inputs/ignore-fail/xfail.txt
@@ -0,0 +1,2 @@
+RUN: false
+XFAIL: *
Index: llvm/utils/lit/tests/Inputs/ignore-fail/lit.cfg
===================================================================
--- /dev/null
+++ llvm/utils/lit/tests/Inputs/ignore-fail/lit.cfg
@@ -0,0 +1,6 @@
+import lit.formats
+config.name = 'ignore-fail'
+config.suffixes = ['.txt']
+config.test_format = lit.formats.ShTest()
+config.test_source_root = None
+config.test_exec_root = None
Index: llvm/utils/lit/tests/Inputs/ignore-fail/fail.txt
===================================================================
--- /dev/null
+++ llvm/utils/lit/tests/Inputs/ignore-fail/fail.txt
@@ -0,0 +1 @@
+RUN: false
Index: llvm/utils/lit/lit/main.py
===================================================================
--- llvm/utils/lit/lit/main.py
+++ llvm/utils/lit/lit/main.py
@@ -118,8 +118,11 @@
 
     has_failure = any(t.isFailure() for t in discovered_tests)
     if has_failure:
-        sys.exit(1)
-
+        if opts.ignoreFail:
+            sys.stderr.write("\nExiting with status 0 instead of 1 because "
+                             "'--ignore-fail' was specified.\n")
+        else:
+            sys.exit(1)
 
 def create_params(builtin_params, user_params):
     def parse(p):
Index: llvm/utils/lit/lit/cl_arguments.py
===================================================================
--- llvm/utils/lit/lit/cl_arguments.py
+++ llvm/utils/lit/lit/cl_arguments.py
@@ -130,6 +130,10 @@
     execution_group.add_argument("--allow-empty-runs",
             help="Do not fail the run if all tests are filtered out",
             action="store_true")
+    execution_group.add_argument("--ignore-fail",
+            dest="ignoreFail",
+            action="store_true",
+            help="Exit with status zero even if some tests fail")
     execution_group.add_argument("--no-indirectly-run-check",
             dest="indirectlyRunCheck",
             help="Do not error if a test would not be run if the user had "


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D96371.322499.patch
Type: text/x-patch
Size: 3212 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20210209/44da5cc3/attachment.bin>