[llvm] [VPlan] Fix header masks in EVL tail folding (PR #150202)

[Luke Lau via llvm-commits]

================
@@ -2265,6 +2269,27 @@ static void transformRecipestoEVLRecipes(VPlan &Plan, VPValue &EVL) {
     for (VPValue *Op : PossiblyDead)
       recursivelyDeleteDeadRecipes(Op);
   }
+
+  // Replace header masks with a mask equivalent to predicating by EVL:
+  //
+  // icmp ule widen-canonical-iv backedge-taken-count
+  // ->
+  // icmp ult step-vector, EVL
+  Type *EVLType = TypeInfo.inferScalarType(&EVL);
+  for (VPValue *HeaderMask : collectAllHeaderMasks(Plan)) {
+    if (HeaderMask->users().empty())
+      continue;
+    VPRecipeBase *EVLR = EVL.getDefiningRecipe();
+    VPBuilder Builder(Plan.getVectorPreheader());
+    VPValue *StepVector =
+        Builder.createNaryOp(VPInstruction::StepVector, {}, EVLType);
+    Builder.setInsertPoint(EVLR->getParent(), std::next(EVLR->getIterator()));
+    VPValue *EVLMask = Builder.createICmp(
----------------
lukel97 wrote:

I originally left it for LICM, but then I realised that this happens after VPlanTransforms::optimize so we won't get VPlan LICM and it'll affect the cost model. But now I see that VPInstruction::StepVector doesn't actually have a cost so I guess it should be ok.

https://github.com/llvm/llvm-project/pull/150202