[llvm] [lit] Fix `progress-bar.py` flaky test (PR #193741)

[via llvm-commits]

llvmbot wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-testing-tools

Author: Jonathan Thackray (jthackray)

<details>
<summary>Changes</summary>

The `lit` progress-bar test was checking for exact progress-bar text
like `0.. 10.. 20.. 30.. 40..`, but that output is not stable from run
to run.

The progress bar is based on timing estimates, including saved timings
from earlier runs, so small timing differences can change which numbers
get printed.

That made the test flaky: FileCheck could match the wrong Testing: line
and then fail later, even though the actual test failures were correct.

Relax the test to verify ordering instead of exact progress-bar
contents. The updated checks assert that the test header is printed, that
each failing test result is separated by a Testing: progress-bar line,
and that the final failed-tests summary lists all expected tests in order.

This preserves coverage for the simple progress-bar path while removing
dependence on unstable timing-derived percentages.

Fixed #<!-- -->72022

---
Full diff: https://github.com/llvm/llvm-project/pull/193741.diff


1 Files Affected:

- (modified) llvm/utils/lit/tests/progress-bar.py (+16-9) 


``````````diff
diff --git a/llvm/utils/lit/tests/progress-bar.py b/llvm/utils/lit/tests/progress-bar.py
index 89f4cba60d6ae..7b78535231784 100644
--- a/llvm/utils/lit/tests/progress-bar.py
+++ b/llvm/utils/lit/tests/progress-bar.py
@@ -3,12 +3,19 @@
 # RUN: not %{lit} -s %{inputs}/progress-bar > %t.out
 # RUN: FileCheck < %t.out %s
 #
-# CHECK: Testing:
-# CHECK: FAIL: progress-bar :: test-1.txt (1 of 4)
-# CHECK: Testing:  0.. 10.. 20
-# CHECK: FAIL: progress-bar :: test-2.txt (2 of 4)
-# CHECK: Testing:  0.. 10.. 20.. 30.. 40..
-# CHECK: FAIL: progress-bar :: test-3.txt (3 of 4)
-# CHECK: Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70
-# CHECK: FAIL: progress-bar :: test-4.txt (4 of 4)
-# CHECK: Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
+# CHECK: -- Testing: 4 tests, 1 workers --
+# CHECK-NEXT: Testing:
+# CHECK-NEXT: FAIL: progress-bar :: test-1.txt (1 of 4)
+# CHECK-NEXT: Testing:
+# CHECK-NEXT: FAIL: progress-bar :: test-2.txt (2 of 4)
+# CHECK-NEXT: Testing:
+# CHECK-NEXT: FAIL: progress-bar :: test-3.txt (3 of 4)
+# CHECK-NEXT: Testing:
+# CHECK-NEXT: FAIL: progress-bar :: test-4.txt (4 of 4)
+# CHECK-NEXT: Testing:
+# CHECK-NEXT: ********************
+# CHECK-NEXT: Failed Tests (4):
+# CHECK-NEXT:   progress-bar :: test-1.txt
+# CHECK-NEXT:   progress-bar :: test-2.txt
+# CHECK-NEXT:   progress-bar :: test-3.txt
+# CHECK-NEXT:   progress-bar :: test-4.txt

``````````

</details>


https://github.com/llvm/llvm-project/pull/193741