[llvm] [LV] Add tests for excluded cases in epiloge tail-folding (PR #217364)
David Sherwood via llvm-commits
llvm-commits at lists.llvm.org
Mon Aug 24 05:57:25 PDT 2026
================
@@ -7223,23 +7225,64 @@ getEpilogueTailLowering(const LoopVectorizationCostModel &MainCM, const Loop *L,
if (!EnableEpilogueVectorization) {
reportVectorizationInfo(
"Options conflict, epilogue vectorization is disallowed while "
- "epilogue tail-folding allowed!\n",
+ "epilogue tail-folding allowed!",
"UnsupportedEpilogueTailFoldingPolicy", ORE, L);
return CM_EpilogueAllowed;
}
+ if (!Hints.getWidth() || !hasForcedEpilogueVF()) {
+ reportVectorizationInfo("For now, Epilogue tail-folding can't be "
+ "applied without forced main/epilogue loop VF",
+ "UnsupportedEpilogueTailFoldingPolicy", ORE, L);
+ return CM_EpilogueAllowed;
+ }
+
+ if (!L->isInnermost()) {
+ reportVectorizationInfo(
+ "Epilogue tail-folding is not supported for outer loop",
+ "InvalidTailFoldedEpilogue", ORE, L);
+ return CM_EpilogueAllowed;
+ }
+
// If scalar epilogue is explicitly required, we can't apply TF.
if (MainCM.requiresScalarEpilogue(/*IsVectorizing*/ true)) {
- LLVM_DEBUG(dbgs() << "LV: Epilogue tail-folding can't be applied because "
- "scalar epilogue is required\n"
- "LV: Fall back to a normal epilogue\n");
+ reportVectorizationInfo(
+ "Epilogue tail-folding can't be applied because scalar epilogue is "
+ "required. Fall back to a normal epilogue",
+ "InvalidTailFoldedEpilogue", ORE, L);
return CM_EpilogueAllowed;
}
// If having epilogue is NOT allowed, then no epilogue to apply TF for.
if (!MainCM.isEpilogueAllowed()) {
- LLVM_DEBUG(dbgs() << "LV: No epilogue to apply tail-folding for.\n"
- "LV: Fall back to a normal epilogue\n");
+ reportVectorizationInfo("No epilogue to apply tail-folding for. Fall back "
+ "to a normal epilogue",
+ "InvalidTailFoldedEpilogue", ORE, L);
+ return CM_EpilogueAllowed;
+ }
+
+ // For now epilogue TF is not supported for some kinds of reductions that
+ // results in more overhead
+ bool HasReductions = !LVL.getReductionVars().empty();
+ bool HasSelectCmpReductions =
+ HasReductions &&
+ any_of(LVL.getReductionVars(), [](auto &Reduction) -> bool {
+ const RecurrenceDescriptor &RdxDesc = Reduction.second;
+ RecurKind RK = RdxDesc.getRecurrenceKind();
+ return RecurrenceDescriptor::isAnyOfRecurrenceKind(RK) ||
+ RecurrenceDescriptor::isFindIVRecurrenceKind(RK);
+ });
+ if (HasSelectCmpReductions) {
+ reportVectorizationInfo(
+ "Epilogue tail-folding is not supported yet for select-cmp Reductions",
+ "InvalidTailFoldedEpilogue", ORE, L);
+ return CM_EpilogueAllowed;
+ }
+
+ if (L->getExitingBlock() != L->getLoopLatch()) {
----------------
david-arm wrote:
Unfortunately, this isn't a strong enough test due to https://github.com/llvm/llvm-project/pull/205109, which may land soon. You probably need something like:
```
if (L->getExitingBlock() != L->getLoopLatch() ||
Legal->hasUncountableEarlyExit())
```
Can you also add a test similar to the one in llvm/test/Transforms/LoopVectorize/early_exit_combined_exits_epilogue.ll added by that PR?
https://github.com/llvm/llvm-project/pull/217364
More information about the llvm-commits
mailing list