[llvm] [lit] Support wildcard in --xfail-not option (PR #151191)

[via llvm-commits]

llvmbot wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-testing-tools

Author: Mircea Trofin (mtrofin)

<details>
<summary>Changes</summary>

See the related issue. We want to set up a build bot where `opt` runs with `-enable-profcheck`, which inserts `MD_prof` before running the rest of the pipeline requested from `opt`, and then validates resulting profile information (more info in the RFC linked by the aforementioned issue)

In that setup, we will also ignore `FileCheck`: while the profile info inserted is, currently, equivalent to the profile info a pass would observe via `BranchProbabilityInfo`/`BlockFrequencyInfo`, (1) we may want to change that, and (2) some tests are quite sensitive to the output IR, and break if, for instance, extra metadata is present (which it would be due to `-enable-profcheck`). Since we're just interested in profile consistency on the upcoming bot, ignoring `FileCheck` is simpler and sufficient. However, this has the effect of passing XFAIL tests. Rather than listing them all, the alternative is to just ignore passing XFAIL.

This PR adds support for that. It is intentionally unsophisticated - e.g. doesn't support "free range" regex - because the motivating scenario doesn't justify that; we can expand that if needed later.

Issue #<!-- -->147390

---
Full diff: https://github.com/llvm/llvm-project/pull/151191.diff


2 Files Affected:

- (modified) llvm/utils/lit/lit/main.py (+5-1) 
- (modified) llvm/utils/lit/tests/xfail-cl.py (+10) 


``````````diff
diff --git a/llvm/utils/lit/lit/main.py b/llvm/utils/lit/lit/main.py
index 0939838b78ceb..2f22c7ef1d053 100755
--- a/llvm/utils/lit/lit/main.py
+++ b/llvm/utils/lit/lit/main.py
@@ -238,7 +238,11 @@ def mark_xfail(selected_tests, opts):
         test_full_name = t.getFullName()
         if test_file in opts.xfail or test_full_name in opts.xfail:
             t.xfails += "*"
-        if test_file in opts.xfail_not or test_full_name in opts.xfail_not:
+        if (
+            test_file in opts.xfail_not
+            or test_full_name in opts.xfail_not
+            or opts.xfail_not == ["*"]
+        ):
             t.xfail_not = True
 
 
diff --git a/llvm/utils/lit/tests/xfail-cl.py b/llvm/utils/lit/tests/xfail-cl.py
index ef1bb0414cfea..b82fb87847c41 100644
--- a/llvm/utils/lit/tests/xfail-cl.py
+++ b/llvm/utils/lit/tests/xfail-cl.py
@@ -5,11 +5,21 @@
 # RUN:   %{inputs}/xfail-cl \
 # RUN: | FileCheck --check-prefix=CHECK-FILTER %s
 
+# RUN: %{lit} --xfail 'false.txt;false2.txt;top-level-suite :: b :: test.txt' \
+# RUN:   --xfail-not '*' \
+# RUN:   %{inputs}/xfail-cl \
+# RUN: | FileCheck --check-prefix=CHECK-FILTER %s
+
 # RUN: env LIT_XFAIL='false.txt;false2.txt;top-level-suite :: b :: test.txt' \
 # RUN:   LIT_XFAIL_NOT='true-xfail.txt;top-level-suite :: a :: test-xfail.txt' \
 # RUN: %{lit} %{inputs}/xfail-cl \
 # RUN: | FileCheck --check-prefix=CHECK-FILTER %s
 
+# RUN: env LIT_XFAIL='false.txt;false2.txt;top-level-suite :: b :: test.txt' \
+# RUN:   LIT_XFAIL_NOT='*' \
+# RUN: %{lit} %{inputs}/xfail-cl \
+# RUN: | FileCheck --check-prefix=CHECK-FILTER %s
+
 # Check that --xfail-not and LIT_XFAIL_NOT always have precedence.
 
 # RUN: env LIT_XFAIL=true-xfail.txt \

``````````

</details>


https://github.com/llvm/llvm-project/pull/151191