[llvm] [LV] codegen for tail-folded epilogue loop (PR #208764)

[Florian Hahn via llvm-commits]

================
@@ -5593,6 +5597,60 @@ void LoopVectorizationPlanner::plan(ElementCount UserVF, unsigned UserIC) {
   LLVM_DEBUG(printPlans(dbgs()));
 }
 
+bool LoopVectorizationPlanner::planForEpilogueTF() {
+  if (VPlans.empty()) {
+    LLVM_DEBUG(dbgs() << "LV: no vplans have been built for main loop VF, bail "
+                         "out of epilogue tail-folding\n");
+    return false;
+  }
+
+  EnabledCM->ValuesToIgnore.insert_range(DefaultCM->ValuesToIgnore);
+  EnabledCM->VecValuesToIgnore.insert_range(DefaultCM->VecValuesToIgnore);
+
+  FixedScalableVFPair MaxFactors =
+      EnabledCM->computeMaxVF(EpilogueVectorizationForceVF, /*UserIC*/ 1);
+  if (!MaxFactors || !EnabledCM->foldTailByMasking()) {
+    // Cases that should not to be vectorized // or tail-folded.
+    reportVectorizationInfo("This case of epilogue loop can't be tail-folded",
+                            "InvalidTailFoldedEpilogue", ORE, OrigLoop);
+    return false;
+  }
+
+  auto VPlan1 = tryToBuildVPlan1();
+
+  if (!useMaskedInterleavedAccesses(TTI)) {
+    LLVM_DEBUG(
+        dbgs() << "LV: Invalidate all interleaved groups due to fold-tail by "
+                  "masking which requires masked-interleaved support.\n");
+    if (EnabledCM->InterleaveInfo.invalidateGroups())
+      // Invalidating interleave groups also requires invalidating all decisions
+      // based on them, which includes widening decisions and uniform and scalar
+      // values.
+      EnabledCM->invalidateCostModelingDecisions();
+  }
+  Legal->prepareToFoldTailByMasking();
+
+  // Collect the instructions (and their associated costs) that will be more
+  // profitable to scalarize.
+  EnabledCM->collectNonVectorizedAndSetWideningDecisions(
+      EpilogueVectorizationForceVF);
+
+  // Expecting only 2 vplans as epilogue TF is applied only for forced VFs.
+  assert(VPlans.size() == 2 &&
+         "For tail-folded epilogue, VPlans size is expected to be 2");
+  // Remove the last vplan, which should be the epilogue plan to replace it by
+  // the tail-folded vplan:
+  assert(VPlans.back()->getSingleVF() == EpilogueVectorizationForceVF &&
+         "For tail-folded epilogue, last vplan is expected to have "
+         "EpilogueUserVF");
+  VPlans.pop_back();
+  buildVPlans(*VPlan1, EpilogueVectorizationForceVF,
+              EpilogueVectorizationForceVF);
+
+  cost(*VPlans.back(), EpilogueVectorizationForceVF, /*RU=*/nullptr);
----------------
fhahn wrote:

Shouldn't the result of this be used?

https://github.com/llvm/llvm-project/pull/208764