[llvm] [LV] Support tail-folded epilogue loops (PR #208764)

[David Sherwood via llvm-commits]

================
@@ -45,20 +45,44 @@ void VPlanTransforms::replaceWideCanonicalIVWithWideIV(
   if (!LoopRegion)
     return;
 
-  auto *WideCanIV =
-      findUserOf<VPWidenCanonicalIVRecipe>(LoopRegion->getCanonicalIV());
+  VPWidenCanonicalIVRecipe *WideCanIV = nullptr;
+  VPIRValue *StartValue = nullptr;
+  // VPWidenCanonicalIVRecipe is either a direct user of CanonicalIV or
+  // Add (CanonicalIV, resumeValue) (like the case for tail-folded epilogue).
+  for (auto *User : LoopRegion->getCanonicalIV()->users()) {
+    if (isa<VPWidenCanonicalIVRecipe>(User)) {
+      WideCanIV = cast<VPWidenCanonicalIVRecipe>(User);
+      StartValue = Plan.getZero(WideCanIV->getScalarType());
+      break;
+    }
+    if (isa<VPInstruction>(User)) {
+      auto *UserInstr = cast<VPInstruction>(User);
+      auto *It = find_if(UserInstr->users(), IsaPred<VPWidenCanonicalIVRecipe>);
+      if (It != UserInstr->user_end()) {
+        WideCanIV = cast<VPWidenCanonicalIVRecipe>(*It);
+        match(UserInstr, m_Add(m_VPValue(), m_VPIRValue(StartValue)));
+        assert(StartValue &&
+               "WIDEN-CANONICAL-INDUCTION is only expected to be reached "
+               "through the canonical IV directly, or through a single 'add "
+               "CanonicalIV, StartValue' introduced for epilogue-loop resume "
+               "values; found a different pattern here");
+        if (!StartValue)
----------------
david-arm wrote:

I added a call to `llvm_unreachable` here and no tests failed, including when building the LLVM test suite. Could you add one please? If the code cannot be tested, then perhaps it can be deleted?

https://github.com/llvm/llvm-project/pull/208764