[llvm] [LV] Support tail-folded epilogue loops (PR #208764)

[Florian Hahn via llvm-commits]

================
@@ -45,20 +45,61 @@ void VPlanTransforms::replaceWideCanonicalIVWithWideIV(
   if (!LoopRegion)
     return;
 
-  auto *WideCanIV =
-      findUserOf<VPWidenCanonicalIVRecipe>(LoopRegion->getCanonicalIV());
-  if (!WideCanIV)
+  VPWidenCanonicalIVRecipe *WideCanIV = nullptr;
+  VPIRValue *StartValue = nullptr;
+  // Find VPWidenCanonicalIVRecipe among the canonical IV's users,
+  // matching Case 1 where it's a direct user of canonicalIV:
+  // <x1> vector loop: {
+  //   vp<%4> = CANONICAL-IV
+  //
+  //   vector.body:
+  //     EMIT vp<%6> = WIDEN-CANONICAL-INDUCTION nuw vp<%4>
+  //     ..
+  // }
+  // or Case 2 (indirect use through an epilogue resume-value add):
+  // <x1> vector loop: {
+  //   vp<%5> = CANONICAL-IV
+  //
+  //   vec.epilog.vector.body:
+  //     EMIT vp<%7> = add vp<%5>, ir<%vec.epilog.resume.val>
+  //     EMIT vp<%8> = WIDEN-CANONICAL-INDUCTION nuw vp<%7>
+  //     ..
+  //}
+  auto *IV = LoopRegion->getCanonicalIV();
+  for (auto *User : IV->users()) {
+    if (isa<VPWidenCanonicalIVRecipe>(User)) {
+      WideCanIV = cast<VPWidenCanonicalIVRecipe>(User);
+      StartValue = Plan.getZero(WideCanIV->getScalarType());
+      break;
+    }
+    if (auto *UI = dyn_cast<VPInstruction>(User)) {
+      auto *It = find_if(UI->users(), IsaPred<VPWidenCanonicalIVRecipe>);
+      if (It != UI->user_end()) {
+        WideCanIV = cast<VPWidenCanonicalIVRecipe>(*It);
+        if (!match(UI, m_c_Add(m_Specific(IV), m_VPIRValue(StartValue))) ||
----------------
fhahn wrote:

I think there may be another crash: https://llvm.godbolt.org/z/TzvEscWfs

Given the amount of code needed to make other styles working, it may help to limit to `DataAndControlFlow`

https://github.com/llvm/llvm-project/pull/208764