[PATCH] D145927: [LV] Fix the combination of predicated epilogs and DataAndControlFlow

[Dave Green via Phabricator via llvm-commits]

dmgreen created this revision.
dmgreen added reviewers: bmahjour, fhahn, Ayal, SjoerdMeijer, sdesmalen, david-arm.
Herald added subscribers: luke, StephenFan, frasercrmck, luismarques, apazos, sameer.abuasal, s.egerton, Jim, jocewei, PkmX, the_o, brucehoult, MartinMosbeck, rogfer01, edward-jones, zzheng, jrtc27, niosHD, sabuasal, simoncook, johnrusso, bollu, rbar, asb, hiraditya.
Herald added a project: All.
dmgreen requested review of this revision.
Herald added subscribers: pcwang-thead, vkmr, MaskRay.
Herald added a project: LLVM.

Currently DataAndControlFlow will insert a active.lane.mask at the start of the loop, assuming that the initial value should always be 0. This is incorrect for epilogues where the initial value will be the CanonicalIVStartVPValue. This patch attempts to create a vplan value for CanonicalIVStartVPValue, in the same line as VPValues like BackedgeTakenCount and TripCount. This can then be replaced by the correct value for the main body or the epilog loop in prepareToExecute. In the main loop it will always be 0. For epilogs it will use the %vec.epilog.resume.val value from the phi from the middle block.

That allows the initial value for the active.lane.mask in DataAndControlFlow to be replaced with the correct value when building the plan with the vplan, in the same way as the VPCanonicalIVPHIRecipe.


https://reviews.llvm.org/D145927

Files:
  llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
  llvm/lib/Transforms/Vectorize/VPlan.cpp
  llvm/lib/Transforms/Vectorize/VPlan.h
  llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
  llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
  llvm/test/Transforms/LoopVectorize/AArch64/epilog-predicated.ll
  llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-forced.ll
  llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-gep.ll
  llvm/test/Transforms/LoopVectorize/AArch64/synthesize-mask-for-call.ll
  llvm/test/Transforms/LoopVectorize/AArch64/widen-call-with-intrinsic-or-libfunc.ll
  llvm/test/Transforms/LoopVectorize/RISCV/riscv-vector-reverse.ll
  llvm/test/Transforms/LoopVectorize/first-order-recurrence-chains-vplan.ll
  llvm/test/Transforms/LoopVectorize/first-order-recurrence-sink-replicate-region.ll
  llvm/test/Transforms/LoopVectorize/icmp-uniforms.ll
  llvm/test/Transforms/LoopVectorize/interleave-and-scalarize-only.ll
  llvm/test/Transforms/LoopVectorize/vplan-dot-printing.ll
  llvm/test/Transforms/LoopVectorize/vplan-iv-transforms.ll
  llvm/test/Transforms/LoopVectorize/vplan-printing.ll
  llvm/test/Transforms/LoopVectorize/vplan-sink-scalars-and-merge-vf1.ll
  llvm/test/Transforms/LoopVectorize/vplan-sink-scalars-and-merge.ll
  llvm/unittests/Transforms/Vectorize/VPlanVerifierTest.cpp

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D145927.504600.patch
Type: text/x-patch
Size: 98609 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20230313/76448384/attachment-0001.bin>