[Lldb-commits] [lldb] r349818 - [dotest] Consider unexpected passes as failures.

[Jonas Devlieghere via lldb-commits]

Author: jdevlieghere
Date: Thu Dec 20 12:44:23 2018
New Revision: 349818

URL: http://llvm.org/viewvc/llvm-project?rev=349818&view=rev
Log:
[dotest] Consider unexpected passes as failures.

Unexpected successes should be considered failures because they can hide
regressions when not addressed. When a test is fixed and not re-enabled,
it can easily regress without us noticing.

I couldn't find a good way to make this change other than changing it in
the unittest2 framework. I know this is less than optimal but since we
have the dependency checked in and the change is pretty fundamental to
the framework I think it's not unreasonable.

Differential revision: https://reviews.llvm.org/D55835

Modified:
    lldb/trunk/lit/Suite/lldbtest.py
    lldb/trunk/third_party/Python/module/unittest2/unittest2/result.py

Modified: lldb/trunk/lit/Suite/lldbtest.py
URL: http://llvm.org/viewvc/llvm-project/lldb/trunk/lit/Suite/lldbtest.py?rev=349818&r1=349817&r2=349818&view=diff
==============================================================================
--- lldb/trunk/lit/Suite/lldbtest.py (original)
+++ lldb/trunk/lit/Suite/lldbtest.py Thu Dec 20 12:44:23 2018
@@ -94,11 +94,10 @@ class LLDBTest(TestFormat):
                 litConfig.maxIndividualTestTime))
 
         if exitCode:
-            return lit.Test.FAIL, out + err
-
-        unexpected_test_line = 'XPASS'
-        if unexpected_test_line in out or unexpected_test_line in err:
-            return lit.Test.XPASS, ''
+            if 'FAIL:' in out or 'FAIL:' in err:
+                return lit.Test.FAIL, out + err
+            if 'XPASS:' in out or 'XPASS:' in err:
+                return lit.Test.XPASS, out + err
 
         passing_test_line = 'RESULT: PASSED'
         if passing_test_line not in out and passing_test_line not in err:

Modified: lldb/trunk/third_party/Python/module/unittest2/unittest2/result.py
URL: http://llvm.org/viewvc/llvm-project/lldb/trunk/third_party/Python/module/unittest2/unittest2/result.py?rev=349818&r1=349817&r2=349818&view=diff
==============================================================================
--- lldb/trunk/third_party/Python/module/unittest2/unittest2/result.py (original)
+++ lldb/trunk/third_party/Python/module/unittest2/unittest2/result.py Thu Dec 20 12:44:23 2018
@@ -148,7 +148,9 @@ class TestResult(unittest.TestResult):
 
     def wasSuccessful(self):
         "Tells whether or not this result was a success"
-        return (len(self.failures) + len(self.errors) == 0)
+        return (len(self.failures) +
+                len(self.errors) +
+                len(self.unexpectedSuccesses) == 0)
 
     def stop(self):
         "Indicates that the tests should be aborted"