[all-commits] [llvm/llvm-project] b663e5: [VPlan] Fix header masks in EVL tail folding (#150...

Luke Lau via All-commits all-commits at lists.llvm.org
Tue Jul 29 20:32:01 PDT 2025


  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: b663e563cce2bcf6cf7e15799f0ab1cfc56a8361
      https://github.com/llvm/llvm-project/commit/b663e563cce2bcf6cf7e15799f0ab1cfc56a8361
  Author: Luke Lau <luke at igalia.com>
  Date:   2025-07-30 (Wed, 30 Jul 2025)

  Changed paths:
    M llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
    M llvm/lib/Transforms/Vectorize/VPlanVerifier.cpp
    M llvm/test/Transforms/LoopVectorize/RISCV/pr88802.ll
    M llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-cond-reduction.ll
    M llvm/test/Transforms/LoopVectorize/RISCV/uniform-load-store.ll

  Log Message:
  -----------
  [VPlan] Fix header masks in EVL tail folding (#150202)

With EVL tail folding, the EVL may not always be VF on the
second-to-last iteration.

Recipes that have been converted to VP intrinsics via optimizeMaskToEVL
account for this, but recipes that are left behind will still use the
old header mask which may end up having a different vector length.

This is effectively the same as #95368, and fixes this by converting
header masks from icmp ule wide-canonical-iv, backedge-trip-count ->
icmp ult step-vector, evl. Without it, recipes that fall through
optimizeMaskToEVL may use the wrong vector length, e.g. in #150074 and
#149981.

We really need to split off optimizeMaskToEVL into
VPlanTransforms::optimize and move transformRecipestoEVLRecipes into
tryToBuildVPlanWithVPRecipes, so we don't mix up what is needed for
correctness and what is needed to optimize away the mask computations.
We should be able to still generate a correct albeit suboptimal VPlan
without running optimizeMaskToEVL. I've added a TODO for this, which I
think we can do after #148274

Fixes #150197



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications


More information about the All-commits mailing list