[PATCH] D142015: [LV] Plan with and without FoldTailByMasking

[Dave Green via Phabricator via llvm-commits]

dmgreen created this revision.
dmgreen added reviewers: fhahn, Ayal, SjoerdMeijer, sdesmalen, david-arm, bmahjour.
Herald added subscribers: luke, StephenFan, frasercrmck, luismarques, apazos, sameer.abuasal, s.egerton, Jim, jocewei, PkmX, the_o, brucehoult, MartinMosbeck, rogfer01, edward-jones, zzheng, jrtc27, niosHD, sabuasal, simoncook, johnrusso, bollu, rbar, asb, hiraditya, nemanjai.
Herald added a project: All.
dmgreen requested review of this revision.
Herald added subscribers: pcwang-thead, vkmr, MaskRay.
Herald added a project: LLVM.

Currently the loop vectorizer has a single parameter in the CostModel that controls FoldTailByMasking. It is set fairly early, and can't be changed later meaning we need to pick between tail folding and non tail folding before we have done much cost modelling. This patch aims to alter that so that there is not a single parameter, moving it eventually into the vplan, so that we can have plans both with and without FoldTailByMasking, that can be costed against one another and the best one picked for vectorization.

A lot of the changes are fairly mechanical and attempted to be non-disruptive to keep the patch simpler, but there are still a fair number of changes. The important parts are:

- FoldTailByMasking is removed from CostModel. It now has a MayFoldTailByMasking variable to hold whether FoldTailByMasking VPlans can be created.
- A number of maps in the cost model like InstsToScalarize/Uniforms/Scalars/ForcedScalars were made conditional on the pair of (FoldTailByMasking, VF), so that they still contain the information required in both FoldTailByMasking=true and FoldTailByMasking=false cases. The semi-random name VectorizationScheme was used to describe the pair of (FoldTailByMasking, VF). Alternative suggestions welcome.
- We then create vplans with both FoldTailByMasking=false and FoldTailByMasking=true if we MayFoldTailByMasking. The VPlan from then stores whether it is FoldTailByMasking. For VF=1 only non-predicated plans are created.
- Tail folded and non tail folded vplans then need to be able to be costed against one another. This patch makes FoldTailByMasking win on a tie if the costs are equal, which will be more consistent with the vectorization before this patch.
- Epilogue loops for the moment are always use FoldTailByMasking=false. This can hopefully be changed in the near future to allow predicated epilogues for unpredicated loop bodies.

Overall this has the effect of allowing us to model and cost tail folded and non tail folded vplans against one another, and should in the future allow us to generate predicated epilogues for unpredicated vector loops.


https://reviews.llvm.org/D142015

Files:
  llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
  llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
  llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
  llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
  llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h
  llvm/lib/Transforms/Vectorize/VPlan.cpp
  llvm/lib/Transforms/Vectorize/VPlan.h
  llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-forced.ll
  llvm/test/Transforms/LoopVectorize/ARM/tail-folding-reduces-vf.ll
  llvm/test/Transforms/LoopVectorize/PowerPC/reg-usage.ll
  llvm/test/Transforms/LoopVectorize/RISCV/riscv-vector-reverse.ll
  llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.small-tc.ll
  llvm/test/Transforms/LoopVectorize/first-order-recurrence-sink-replicate-region.ll
  llvm/test/Transforms/LoopVectorize/icmp-uniforms.ll
  llvm/test/Transforms/LoopVectorize/tail-folding-vectorization-factor-1.ll
  llvm/test/Transforms/LoopVectorize/vplan-sink-scalars-and-merge.ll

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D142015.490112.patch
Type: text/x-patch
Size: 161012 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20230118/e3dd5c94/attachment-0001.bin>