[llvm] [LoopVectorize] Improve Vectorization of Low Trip Count Loops (PR #195823)
Jack Styles via llvm-commits
llvm-commits at lists.llvm.org
Tue Jul 21 01:54:28 PDT 2026
================
@@ -5790,14 +5814,78 @@ LoopVectorizationPlanner::computeBestVF() {
return {VectorizationFactor::Disabled(), nullptr};
// If there is a single VPlan with a single VF, return it directly.
VPlan &FirstPlan = *VPlans[0];
+ auto IsUnprofitableOneScalarTail =
+ [&](const VectorizationFactor &CurrentFactor, bool HasTail,
+ bool ForceVectorization, const ElementCount &ExactTC,
+ const VectorizationFactor &ScalarFactor,
+ const InstructionCost &ScalarCost, unsigned int UserIC) {
+ if (ForceVectorization || !HasTail || !ExactTC.isFixed() ||
+ CurrentFactor.Width.isScalable())
+ return false;
+
+ unsigned TC = ExactTC.getFixedValue();
+ if (TC == 0 || TC > TTI.getMinTripCountTailFoldingThreshold())
+ return false;
+
+ unsigned EstimatedWidth = estimateElementCount(
+ CurrentFactor.Width, Config.getVScaleForTuning());
+ if (TC != (EstimatedWidth * UserIC) + 1)
+ return false;
+
+ InstructionCost VectorCost =
+ getCostForKnownTripCount(CurrentFactor, TC, HasTail);
+ InstructionCost ScalarCostForTC =
+ getCostForKnownTripCount(ScalarFactor, TC, /*HasTail=*/false);
+ // Be conservative for the one-scalar-tail shape. It introduces extra
+ // control flow and a scalar epilogue for a single element, so require
+ // the vectorized form to save at least one scalar iteration.
+ InstructionCost AdjustedVectorCost = VectorCost + ScalarCost;
----------------
Stylie777 wrote:
Thats correct, because we want VectorCost to reflect the cost for the vector iteration(s) and the one remaining scalar cost (I have added a comment here to explain).
Ok, understood. Apologies I got how the costs are done mixed up.
I still think we should be making clear savings. There does seem to be an issue with the costing of `smin/smax/umin/umax` intrinsics when the type needs promoting, but even with that fixed (see, INSERT PR NUMBER FOR SMIN FIX) the costs for `tc3_smin_i8_reject` still just accept the vector cost over the scalar cost (29 vector < 30 scalar) so the function will get vectorised, which as we have already discussed we know is slower for performance as per the micro benchmark.
https://github.com/llvm/llvm-project/pull/195823
More information about the llvm-commits
mailing list