[Lldb-commits] [PATCH] D55835: [dotest] Consider unexpected passes as failures.

Jonas Devlieghere via Phabricator via lldb-commits lldb-commits at lists.llvm.org
Tue Dec 18 09:47:22 PST 2018

JDevlieghere created this revision.
JDevlieghere added reviewers: labath, davide, jingham, clayborg.
JDevlieghere added a project: LLDB.

Unexpected successes should be considered failures because they can hide regressions when not addressed. When a test is fixed and not re-enabled, it can easily regress without us noticing.

I couldn't find a good way to make this change other than changing it in the unittest2 framework. I know this is less than optimal but since we have the dependency checked in and the change is pretty fundamental to the framework I think it's not unreasonable.




Index: third_party/Python/module/unittest2/unittest2/result.py
--- third_party/Python/module/unittest2/unittest2/result.py
+++ third_party/Python/module/unittest2/unittest2/result.py
@@ -148,7 +148,9 @@
     def wasSuccessful(self):
         "Tells whether or not this result was a success"
-        return (len(self.failures) + len(self.errors) == 0)
+        return (len(self.failures) +
+                len(self.errors) +
+                len(self.unexpectedSuccesses) == 0)
     def stop(self):
         "Indicates that the tests should be aborted"
Index: lit/Suite/lldbtest.py
--- lit/Suite/lldbtest.py
+++ lit/Suite/lldbtest.py
@@ -94,11 +94,10 @@
         if exitCode:
-            return lit.Test.FAIL, out + err
-        unexpected_test_line = 'XPASS'
-        if unexpected_test_line in out or unexpected_test_line in err:
-            return lit.Test.XPASS, ''
+            if 'FAIL' in out or 'FAIL' in err:
+                return lit.Test.FAIL, out + err
+            if 'XPASS' in out or 'XPASS' in err:
+                return lit.Test.XPASS, out + err
         passing_test_line = 'RESULT: PASSED'
         if passing_test_line not in out and passing_test_line not in err:

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D55835.178712.patch
Type: text/x-patch
Size: 1388 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/lldb-commits/attachments/20181218/07f1eed8/attachment.bin>

More information about the lldb-commits mailing list