[Lldb-commits] [lldb] [lldb] change how unsupported tests are aggregated (PR #185675)

[via lldb-commits]

llvmbot wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-lldb

Author: Charles Zablit (charles-zablit)

<details>
<summary>Changes</summary>

Change how the results of multiple non failing tests (UNSUPPORTED, PASS or XFAIL) are aggregated into one result.

Currently if 2 tests are UNSUPPORTED and 1 test is PASS or XFAIL, the aggregated result will be UNSUPPORTED.

This is confusing on Windows because a lot of API tests try to run `dwo`, `dsym` and `dwarf` variants, whereas we only run the `dwarf` variant on Windows. In swiftlang/llvm-project, we are seeing results like the following when running the lldb-swift target:

```
Total Discovered Tests: 2548
  Excluded         : 2009 (78.85%)
  Unsupported      :  462 (18.13%)
  Passed           :    6 (0.24%)
  Expectedly Failed:   19 (0.75%)
  Unresolved       :    6 (0.24%)
  Failed           :   46 (1.81%)
```

With this patch, a group of test is only UNRESOLVED if all the subtests are UNRESOLVED.

The results become:

```
Total Discovered Tests: 2548
  Excluded         : 2009 (78.85%)
  Unsupported      :  257 (10.09%)
  Passed           :    6 (0.24%)
  Expectedly Failed:  224 (8.79%)
  Unresolved       :    6 (0.24%)
  Failed           :   46 (1.81%)
```

---
Full diff: https://github.com/llvm/llvm-project/pull/185675.diff


1 Files Affected:

- (modified) lldb/test/API/lldbtest.py (+3-4) 


``````````diff
diff --git a/lldb/test/API/lldbtest.py b/lldb/test/API/lldbtest.py
index d6b79ebc2c434..e5e4618bf2a3f 100644
--- a/lldb/test/API/lldbtest.py
+++ b/lldb/test/API/lldbtest.py
@@ -138,12 +138,11 @@ def execute(self, test, litConfig):
                 (unexpected_successes, lit.Test.XPASS),
             ]
         else:
-            # Mark this test as PASS if at least one test passed.
-            if passes > 0:
-                return lit.Test.PASS, output
+            # Only mark the test as unsupported if they are all unsupported.
+            if passes == 0 and expected_failures == 0:
+                return lit.Test.UNSUPPORTED
             lit_results = [
                 (passes, lit.Test.PASS),
-                (skipped, lit.Test.UNSUPPORTED),
                 (expected_failures, lit.Test.XFAIL),
             ]
 

``````````

</details>


https://github.com/llvm/llvm-project/pull/185675