[llvm] [LoopUnroll] Introduce UnrollMaxIterations as a hard cap on how many iterations we try to unroll (PR #78648)

[Nikita Popov via llvm-commits]

================
@@ -1288,6 +1293,14 @@ tryToUnrollLoop(Loop *L, DominatorTree &DT, LoopInfo *LI, ScalarEvolution &SE,
     return LoopUnrollResult::Unmodified;
   }
 
+  // Certain cases with UBSAN can cause trip count to be calculated as INT_MAX,
+  // Block unrolling at a reasonable limit so that the compiler doesn't hang
+  // trying to unroll the loop. See PR77842
+  if (UP.Count > UnrollMaxIterations) {
+    LLVM_DEBUG(dbgs() << "Won't unroll; trip count is too large\n");
+    return LoopUnrollResult::Unmodified;
+  }
----------------
nikic wrote:

This works, but I don't think it fits well into the way the unroll count is currently computed. Rather than checking this after the fact, we should integrate this into the unroll count calculation. In particular, inside https://github.com/llvm/llvm-project/blob/88b1087035fd397996837e35d579d808d6b3f28c/llvm/lib/Transforms/Scalar/LoopUnrollPass.cpp#L784-L785 we should apply a pragma-specific iteration limit. This will also ensure that even if we decline to perform a full unroll to an unreasonable count, we can still chose to perform a partial or runtime unroll to a lower iteration count.

https://github.com/llvm/llvm-project/pull/78648