[llvm] [LoopUtils] prevent negative estimated trip count result (PR #195610)

[Arda Serdar Pektezol via llvm-commits]

https://github.com/pektezol created https://github.com/llvm/llvm-project/pull/195610

`getLoopEstimatedTripCount` will return nullopt whenever the `llvm.loop.estimated_trip_count` is set to 0, but not for negative values. The linked issue contains `AlreadyPeeled` count of 1, with the `llvm.loop.estimated_trip_count` set to -1 in the metadata -- causing the `if (*EstimatedTripCount + AlreadyPeeled <= MaxPeelCount)` case in `LoopPeel` to trigger. After triggering, it peels for INT_MAX which causes the current hang with `opt`.

Fixes #173169


>From 01184b4e012cae898b154b662a8d00f92ead6e22 Mon Sep 17 00:00:00 2001
From: Arda Serdar Pektezol <arda at pektezol.dev>
Date: Mon, 4 May 2026 11:18:01 +0300
Subject: [PATCH 1/2] [LoopUtils] prevent negative estimated trip count result

---
 llvm/lib/Transforms/Utils/LoopUtils.cpp       |  4 +--
 .../LoopUnroll/negative-trip-count.ll         | 33 +++++++++++++++++++
 2 files changed, 35 insertions(+), 2 deletions(-)
 create mode 100644 llvm/test/Transforms/LoopUnroll/negative-trip-count.ll

diff --git a/llvm/lib/Transforms/Utils/LoopUtils.cpp b/llvm/lib/Transforms/Utils/LoopUtils.cpp
index 03b693974e5fe..8c990707a88d7 100644
--- a/llvm/lib/Transforms/Utils/LoopUtils.cpp
+++ b/llvm/lib/Transforms/Utils/LoopUtils.cpp
@@ -937,14 +937,14 @@ llvm::getLoopEstimatedTripCount(Loop *L,
   // reached, it is the start of an iteration.  Consequently, some passes
   // historically assume that llvm::getLoopEstimatedTripCount always returns a
   // positive count or std::nullopt.  Thus, return std::nullopt when
-  // llvm.loop.estimated_trip_count is 0.
+  // llvm.loop.estimated_trip_count is 0 or negative.
   if (auto TC = getOptionalIntLoopAttribute(L, LLVMLoopEstimatedTripCount)) {
     LLVM_DEBUG(dbgs() << "getLoopEstimatedTripCount: "
                       << LLVMLoopEstimatedTripCount << " metadata has trip "
                       << "count of " << *TC
                       << (*TC == 0 ? " (returning std::nullopt)" : "")
                       << " for " << DbgLoop(L) << "\n");
-    return *TC == 0 ? std::nullopt : std::optional(*TC);
+    return *TC <= 0 ? std::nullopt : std::optional(*TC);
   }
 
   // Estimate the trip count from latch branch weights.
diff --git a/llvm/test/Transforms/LoopUnroll/negative-trip-count.ll b/llvm/test/Transforms/LoopUnroll/negative-trip-count.ll
new file mode 100644
index 0000000000000..d02975ad5dde1
--- /dev/null
+++ b/llvm/test/Transforms/LoopUnroll/negative-trip-count.ll
@@ -0,0 +1,33 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt -passes=loop-unroll -S < %s | FileCheck %s
+
+define void @PR173169() !prof !0 {
+; CHECK-LABEL: define void @PR173169(
+; CHECK-SAME: ) !prof [[PROF0:![0-9]+]] {
+; CHECK-NEXT:  [[ENTRY:.*:]]
+; CHECK-NEXT:    br label %[[LOOP:.*]]
+; CHECK:       [[LOOP]]:
+; CHECK-NEXT:    br i1 true, label %[[LOOP]], label %[[EXIT:.*]], !llvm.loop [[LOOP1:![0-9]+]]
+; CHECK:       [[EXIT]]:
+; CHECK-NEXT:    ret void
+;
+entry:
+  br label %loop
+
+loop:                                             ; preds = %loop, %entry
+  br i1 true, label %loop, label %exit, !llvm.loop !1
+
+exit:                                             ; preds = %loop
+  ret void
+}
+
+!0 = !{!"function_entry_count", i64 1000}
+!1 = distinct !{!1, !2, !3}
+!2 = !{!"llvm.loop.peeled.count", i32 1}
+!3 = !{!"llvm.loop.estimated_trip_count", i32 -1}
+;.
+; CHECK: [[PROF0]] = !{!"function_entry_count", i64 1000}
+; CHECK: [[LOOP1]] = distinct !{[[LOOP1]], [[META2:![0-9]+]], [[META3:![0-9]+]]}
+; CHECK: [[META2]] = !{!"llvm.loop.peeled.count", i32 1}
+; CHECK: [[META3]] = !{!"llvm.loop.estimated_trip_count", i32 -1}
+;.

>From 96c72b8ef978cb22d6d524dbfdc069a5b1a0546d Mon Sep 17 00:00:00 2001
From: Arda Serdar Pektezol <arda at pektezol.dev>
Date: Mon, 4 May 2026 11:30:17 +0300
Subject: [PATCH 2/2] [NFC][LoopUtils] update debug message for negative trip
 count

---
 llvm/lib/Transforms/Utils/LoopUtils.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/llvm/lib/Transforms/Utils/LoopUtils.cpp b/llvm/lib/Transforms/Utils/LoopUtils.cpp
index 8c990707a88d7..d513ef657ba38 100644
--- a/llvm/lib/Transforms/Utils/LoopUtils.cpp
+++ b/llvm/lib/Transforms/Utils/LoopUtils.cpp
@@ -942,7 +942,7 @@ llvm::getLoopEstimatedTripCount(Loop *L,
     LLVM_DEBUG(dbgs() << "getLoopEstimatedTripCount: "
                       << LLVMLoopEstimatedTripCount << " metadata has trip "
                       << "count of " << *TC
-                      << (*TC == 0 ? " (returning std::nullopt)" : "")
+                      << (*TC <= 0 ? " (returning std::nullopt)" : "")
                       << " for " << DbgLoop(L) << "\n");
     return *TC <= 0 ? std::nullopt : std::optional(*TC);
   }