[llvm] [LV] Don't require scalar epilogue for unsupported IAG with tail (PR #96544)

[Kolya Panchenko via llvm-commits]

================
@@ -1448,29 +1448,58 @@ class LoopVectorizationCostModel {
   /// Returns true if \p I is a memory instruction in an interleaved-group
   /// of memory accesses that can be vectorized with wide vector loads/stores
   /// and shuffles.
-  bool interleavedAccessCanBeWidened(Instruction *I, ElementCount VF);
+  bool interleavedAccessCanBeWidened(Instruction *I, ElementCount VF) const;
 
   /// Check if \p Instr belongs to any interleaved access group.
-  bool isAccessInterleaved(Instruction *Instr) {
+  bool isAccessInterleaved(Instruction *Instr) const {
     return InterleaveInfo.isInterleaved(Instr);
   }
 
   /// Get the interleaved access group that \p Instr belongs to.
   const InterleaveGroup<Instruction> *
-  getInterleavedAccessGroup(Instruction *Instr) {
+  getInterleavedAccessGroup(Instruction *Instr) const {
     return InterleaveInfo.getInterleaveGroup(Instr);
   }
 
   /// Returns true if we're required to use a scalar epilogue for at least
   /// the final iteration of the original loop.
-  bool requiresScalarEpilogue(bool IsVectorizing) const {
-    if (!isScalarEpilogueAllowed())
+  bool requiresScalarEpilogue(ElementCount VF) const {
+    if (!isScalarEpilogueAllowed()) {
+      LLVM_DEBUG(dbgs() << "LV: Loop with VF = " << VF
+                        << " does not require scalar epilogue\n");
       return false;
+    }
     // If we might exit from anywhere but the latch, must run the exiting
     // iteration in scalar form.
-    if (TheLoop->getExitingBlock() != TheLoop->getLoopLatch())
+    if (TheLoop->getExitingBlock() != TheLoop->getLoopLatch()) {
+      LLVM_DEBUG(dbgs() << "LV: Loop with VF = " << VF
+                        << " requires scalar epilogue: multiple exists\n");
       return true;
-    return IsVectorizing && InterleaveInfo.requiresScalarEpilogue();
+    }
+    if (VF.isVector()) {
+      if (InterleaveInfo.requiresScalarEpilogue()) {
+        // Make sure interleaved groups that require scalar epilogue will be
+        // widened.
+        for (auto *G : InterleaveInfo.getInterleaveGroups()) {
+          if (!G->requiresScalarEpilogue())
+            continue;
+
+          Instruction *I = G->getMember(0);
+          InstWidening Decision = getWideningDecision(I, VF);
+          if (Decision == CM_Interleave ||
+              (Decision == CM_Unknown &&
+               interleavedAccessCanBeWidened(G->getMember(0), VF))) {
----------------
nikolaypanchenko wrote:

There's no expectation when `requiresScalarEpilogue` is called: it can be called before decision is made and after. The latter check makes sure `requiresScalarEpilogue` returns the same value when decision is not made. Precondition `decision != cm_unknown` is necessary as `interleavedAccessCanBeWidened` is expected to be called with unknown decision.

https://github.com/llvm/llvm-project/pull/96544