[llvm] [LV] Support tail-folded epilogue loops (PR #208764)

[Kiran Chandramohan via llvm-commits]

================
@@ -7847,6 +7962,39 @@ static void connectEpilogueVectorLoop(VPlan &EpiPlan, Loop *L,
   for (PHINode &Phi : make_early_inc_range(VecEpiloguePreHeader->phis()))
     if (Phi.use_empty())
       Phi.eraseFromParent();
+
+  if (IsEpilogueTFEnabled) {
+    // The epilogue vector loop is tail-folded, so it can safely handle
+    // any remaining trip count, including zero, via masking.
+    // vec.epilog.iter.check's own min-iters check was therefore built with a
+    // compile-time-known-false condition (see
+    // addMinimumVectorEpilogueIterationCheck) that never needs to bail out to
+    // a scalar remainder. Fold it into an unconditional branch into the
+    // vector epilogue preheader.
+    auto *Br =
+        cast<CondBrInst>(VecEpilogueIterationCountCheck->getTerminator());
+    [[maybe_unused]] auto *CondC = dyn_cast<ConstantInt>(Br->getCondition());
+    assert(CondC && CondC->isZero() &&
+           "expected vec.epilog.iter.check's branch condition to be a "
+           "compile-time false constant when the epilogue is tail-folded");
+    BasicBlock *DeadSucc = Br->getSuccessor(0);
+    UncondBrInst::Create(VecEpiloguePreHeader, Br->getIterator());
+    Br->eraseFromParent();
+    DTU.applyUpdates(
+        {{DominatorTree::Delete, VecEpilogueIterationCountCheck, DeadSucc}});
+
+    if (!SCEVCheckBlock && !MemCheckBlock) {
----------------
kiranchandramohan wrote:

>  It would be good to reduce the size of the diff in this PR where possible.

Feel free to ignore, only do if it is worthwhile. Just based on some codex analysis.
-> NFC patch for cost model plumbing
-> NFC patch that extracts forced epilogue VPlan construction into a dedicated planning helper
-> Tail folded vector epilogues for simple loops (rejecting reductions and fixed order recurrences)
-> Enable tail folded vector epilogues for reduction loops
-> Enable fixed-order recurrences and specialized liveouts

https://github.com/llvm/llvm-project/pull/208764