[Lldb-commits] [PATCH] D127258: [lldb] Mark API tests as XFAIL if they have expected failures and no passing tests

[Jonas Devlieghere via Phabricator via lldb-commits]

JDevlieghere updated this revision to Diff 434984.
JDevlieghere added a comment.

I was in the process of writing a comment saying that we could be even more accurate by parsing the number of tests with a particular result code from the dotest output but that that seemed overkill before I changed my mind mid-sentence and implemented that instead.


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D127258/new/

https://reviews.llvm.org/D127258

Files:
  lldb/test/API/lldbtest.py


Index: lldb/test/API/lldbtest.py
===================================================================
--- lldb/test/API/lldbtest.py
+++ lldb/test/API/lldbtest.py
@@ -3,6 +3,7 @@
 import tempfile
 import subprocess
 import sys
+import re
 import platform
 
 import lit.Test
@@ -88,20 +89,31 @@
         if timeoutInfo:
             return lit.Test.TIMEOUT, output
 
-        if exitCode:
-            if 'XPASS:' in out or 'XPASS:' in err:
-                return lit.Test.XPASS, output
-
-            # Otherwise this is just a failure.
-            return lit.Test.FAIL, output
-
-        has_unsupported_tests = 'UNSUPPORTED:' in out or 'UNSUPPORTED:' in err
-        has_passing_tests = 'PASS:' in out or 'PASS:' in err
-        if has_unsupported_tests and not has_passing_tests:
-            return lit.Test.UNSUPPORTED, output
+        # Parse the dotest output from stderr.
+        result_regex = r"\((\d+) passes, (\d+) failures, (\d+) errors, (\d+) skipped, (\d+) expected failures, (\d+) unexpected successes\)"
+        results = re.search(result_regex, err)
 
-        passing_test_line = 'RESULT: PASSED'
-        if passing_test_line not in out and passing_test_line not in err:
+        # If parsing fails mark this test as unresolved.
+        if not results:
             return lit.Test.UNRESOLVED, output
 
-        return lit.Test.PASS, output
+        passes = int(results.group(1))
+        failures = int(results.group(2))
+        errors = int(results.group(3))
+        skipped = int(results.group(4))
+        expected_failures = int(results.group(5))
+        unexpected_successes = int(results.group(6))
+
+        if exitCode:
+            lit_results = [(failures, lit.Test.FAIL),
+                           (errors, lit.Test.UNRESOLVED),
+                           (unexpected_successes, lit.Test.XPASS)]
+        else:
+            lit_results = [(passes, lit.Test.PASS),
+                           (skipped, lit.Test.UNSUPPORTED),
+                           (expected_failures, lit.Test.XFAIL)]
+
+        # Sort the lit result tuples by their number of occurrences. Only look
+        # at the first element and rely on the sorting being stable for ties.
+        lit_results.sort(reverse=True, key=lambda tup: tup[0])
+        return lit_results[0][1], output


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D127258.434984.patch
Type: text/x-patch
Size: 2299 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/lldb-commits/attachments/20220607/27b889dc/attachment.bin>