[llvm] [lit] Fix `progress-bar.py` flaky test (PR #193741)

[Jonathan Thackray via llvm-commits]

https://github.com/jthackray created https://github.com/llvm/llvm-project/pull/193741

The `lit` progress-bar test was checking for exact progress-bar text
like `0.. 10.. 20.. 30.. 40..`, but that output is not stable from run
to run.

The progress bar is based on timing estimates, including saved timings
from earlier runs, so small timing differences can change which numbers
get printed.

That made the test flaky: FileCheck could match the wrong Testing: line
and then fail later, even though the actual test failures were correct.

Relax the test to verify ordering instead of exact progress-bar
contents. The updated checks assert that the test header is printed, that
each failing test result is separated by a Testing: progress-bar line,
and that the final failed-tests summary lists all expected tests in order.

This preserves coverage for the simple progress-bar path while removing
dependence on unstable timing-derived percentages.

Fixed #72022

>From 0d6652ad39b33f9df432e3e9781d79ec61e517d5 Mon Sep 17 00:00:00 2001
From: Jonathan Thackray <jonathan.thackray at arm.com>
Date: Thu, 23 Apr 2026 14:29:22 +0100
Subject: [PATCH] [lit] Fix `progress-bar.py` flaky test

The `lit` progress-bar test was checking for exact progress-bar text
like `0.. 10.. 20.. 30.. 40..`, but that output is not stable from run
to run.

The progress bar is based on timing estimates, including saved timings
from earlier runs, so small timing differences can change which numbers
get printed.

That made the test flaky: FileCheck could match the wrong Testing: line
and then fail later, even though the actual test failures were correct.

Relax the test to verify ordering instead of exact progress-bar
contents. The updated checks assert that the test header is printed, that
each failing test result is separated by a Testing: progress-bar line,
and that the final failed-tests summary lists all expected tests in order.

This preserves coverage for the simple progress-bar path while removing
dependence on unstable timing-derived percentages.

Fixed #72022
---
 llvm/utils/lit/tests/progress-bar.py | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)

diff --git a/llvm/utils/lit/tests/progress-bar.py b/llvm/utils/lit/tests/progress-bar.py
index 89f4cba60d6ae..7b78535231784 100644
--- a/llvm/utils/lit/tests/progress-bar.py
+++ b/llvm/utils/lit/tests/progress-bar.py
@@ -3,12 +3,19 @@
 # RUN: not %{lit} -s %{inputs}/progress-bar > %t.out
 # RUN: FileCheck < %t.out %s
 #
-# CHECK: Testing:
-# CHECK: FAIL: progress-bar :: test-1.txt (1 of 4)
-# CHECK: Testing:  0.. 10.. 20
-# CHECK: FAIL: progress-bar :: test-2.txt (2 of 4)
-# CHECK: Testing:  0.. 10.. 20.. 30.. 40..
-# CHECK: FAIL: progress-bar :: test-3.txt (3 of 4)
-# CHECK: Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70
-# CHECK: FAIL: progress-bar :: test-4.txt (4 of 4)
-# CHECK: Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
+# CHECK: -- Testing: 4 tests, 1 workers --
+# CHECK-NEXT: Testing:
+# CHECK-NEXT: FAIL: progress-bar :: test-1.txt (1 of 4)
+# CHECK-NEXT: Testing:
+# CHECK-NEXT: FAIL: progress-bar :: test-2.txt (2 of 4)
+# CHECK-NEXT: Testing:
+# CHECK-NEXT: FAIL: progress-bar :: test-3.txt (3 of 4)
+# CHECK-NEXT: Testing:
+# CHECK-NEXT: FAIL: progress-bar :: test-4.txt (4 of 4)
+# CHECK-NEXT: Testing:
+# CHECK-NEXT: ********************
+# CHECK-NEXT: Failed Tests (4):
+# CHECK-NEXT:   progress-bar :: test-1.txt
+# CHECK-NEXT:   progress-bar :: test-2.txt
+# CHECK-NEXT:   progress-bar :: test-3.txt
+# CHECK-NEXT:   progress-bar :: test-4.txt