[llvm] [VPlan] Convert EVL loops to variable-length stepping after dissolution (PR #147222)

[Luke Lau via llvm-commits]

================
@@ -2357,6 +2357,60 @@ bool VPlanTransforms::tryAddExplicitVectorLength(
   return true;
 }
 
+void VPlanTransforms::simplifyEVLIVs(VPlan &Plan) {
+  using namespace llvm::VPlanPatternMatch;
+  // Find EVL loop entries by locating VPEVLBasedIVPHIRecipe
+  // There should be only one EVL PHI in the entire plan
+  VPEVLBasedIVPHIRecipe *EVLPhi = nullptr;
+
+  for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(
+           vp_depth_first_shallow(Plan.getEntry())))
+    for (VPRecipeBase &R : VPBB->phis())
+      if (auto *PhiR = dyn_cast<VPEVLBasedIVPHIRecipe>(&R)) {
+        assert(!EVLPhi && "Found multiple EVL PHIs. Only one expected");
+        EVLPhi = PhiR;
+      }
+
+  // Early return if no EVL PHI is found
+  if (!EVLPhi)
+    return;
+
+  VPBasicBlock *Entry = EVLPhi->getParent();
+  VPValue *EVLIncrement = EVLPhi->getBackedgeValue();
+
+  // Convert EVLPhi to concrete recipe.
+  auto *ScalarR =
+      VPBuilder(EVLPhi).createScalarPhi({EVLPhi->getStartValue(), EVLIncrement},
+                                        EVLPhi->getDebugLoc(), "evl.based.iv");
+  EVLPhi->replaceAllUsesWith(ScalarR);
+  EVLPhi->eraseFromParent();
+
+  // Find the latch-exiting block and convert to variable-length stepping.
+  // Before: (branch-on-count CanonicalIVInc, VectorTripCount)
+  // After: (branch-on-count EVLIVInc, TripCount)
+  auto Range =
+      VPBlockUtils::blocksOnly<VPBasicBlock>(vp_depth_first_shallow(Entry));
+  auto It = find_if(Range, [&Entry](VPBasicBlock *VPBB) {
+    return any_of(VPBB->successors(),
+                  [&Entry](VPBlockBase *Succ) { return Succ == Entry; });
+  });
+  assert((It != Range.end()) && "LatchExiting is not found");
+  VPBasicBlock *LatchExiting = *It;
+  auto *LatchExitingBr = cast<VPInstruction>(LatchExiting->getTerminator());
+  VPValue *ScalarIVInc;
+  assert(LatchExitingBr &&
+         match(LatchExitingBr,
+               m_BranchOnCount(m_VPValue(ScalarIVInc),
+                               m_Specific(&Plan.getVectorTripCount()))) &&
+         "Unexpected terminator in EVL loop");
----------------
lukel97 wrote:

I tried this PR out locally on SPEC CPU 2017 and hit this assertion building 502.gcc_r, it looks like sometimes the branch condition might be simplified?

```
  WIDEN vp.store vp<%6>, ir<0>, vp<%4>
  EMIT-SCALAR vp<%7> = zext vp<%4> to i64
  EMIT vp<%index.evl.next> = add vp<%7>, vp<%evl.based.iv>
  EMIT vp<%index.next> = add vp<%index>, vp<%0>
  EMIT branch-on-cond ir<true>
Successor(s): middle.block, vector.body
...
Assertion failed: (LatchExitingBr && match(LatchExitingBr, m_BranchOnCount(m_VPValue(ScalarIVInc), m_Specific(&Plan.getVectorTripCount()))) && "Unexpected terminator in EVL loop"), function simplifyEVLIVs, file VPlanTransforms.cpp, line 2406.
```

Maybe we need to relax the assertion?

https://github.com/llvm/llvm-project/pull/147222