[llvm] [LoopVectorize] Improve Vectorization of Low Trip Count Loops (PR #195823)

[Jack Styles via llvm-commits]

================
@@ -5806,6 +5830,81 @@ InstructionCost LoopVectorizationPlanner::cost(VPlan &Plan, ElementCount VF,
   return Cost;
 }
 
+bool LoopVectorizationPlanner::isProfitableOneScalarTail(
+    const VectorizationFactor &CurrentFactor, const ElementCount &ExactTC,
+    unsigned int UserIC) {
+  if (!ExactTC.isFixed() || CurrentFactor.Width.isScalable())
+    return true;
+
+  unsigned TC = ExactTC.getFixedValue();
+  if (TC == 0 || TC > TTI.getMinTripCountTailFoldingThreshold())
+    return true;
+
+  unsigned EstimatedWidth =
+      estimateElementCount(CurrentFactor.Width, Config.getVScaleForTuning());
+  if (TC % (EstimatedWidth * UserIC) != 1)
+    return true;
+
+  // On certain Instructions or Intrinsics, where Type Promotion is used
----------------
Stylie777 wrote:

The cost model changes stopped vectorisation for loops with TC=3, which is where the worst performance regressions would have been, but TC=5 would always have Scalar costs of ~50 (for the smin intrinsic example) which, even with adjusted costs, the vectorised loops would always cost lower. Really, the cost model does need adjusting regardless because the type promotion is not factored in when we cost the vectorised intrinsic, and its cost is certainly not `1`. I used `6` as this was the value needed to stop TC=3 loops from vectorising.

That is a good point and if so, this could be made target specific with a target hook (where default returns true to continue the vectorisation).

https://github.com/llvm/llvm-project/pull/195823