[PATCH] D116928: [LoopVectorize] Support epilogue vectorisation of loops with reductions

[Kerry McLaughlin via Phabricator via llvm-commits]

kmclaughlin created this revision.
kmclaughlin added reviewers: sdesmalen, david-arm, bmahjour, fhahn, dmgreen.
Herald added a subscriber: hiraditya.
kmclaughlin requested review of this revision.
Herald added a project: LLVM.
Herald added a subscriber: llvm-commits.

isCandidateForEpilogueVectorization will currently return false for loops
which contain reductions. This patch removes this restriction and makes
the following changes to support epilogue vectorisation with reductions:

- `fixReduction`: If fixReduction is being called during vectorisation of the epilogue, the phi node it creates (bc.merge.rdx) will need to additionally carry incoming values from the middle block of the main loop.

- `createEpilogueVectorizedLoopSkeleton`: The incoming values of the phi created by fixReduction are updated after the vec.epilog.iter.check block is added. The phi is also moved to the preheader of the epilogue.

- `processLoop`: The start value of any VPReductionPHIRecipes are updated before vectorising the epilogue loop. The getResumeInstr function added to the ILV will return the resume instruction from the main loop scalar preheader.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D116928

Files:
  llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
  llvm/test/Transforms/LoopVectorize/AArch64/sve-epilog-vect-inloop-reductions.ll
  llvm/test/Transforms/LoopVectorize/AArch64/sve-epilog-vect-reductions.ll
  llvm/test/Transforms/LoopVectorize/AArch64/sve-epilog-vect-strict-reductions.ll
  llvm/test/Transforms/LoopVectorize/X86/invariant-store-vectorization.ll
  llvm/test/Transforms/LoopVectorize/X86/pr42674.ll
  llvm/test/Transforms/LoopVectorize/epilog-vectorization-reductions.ll
  llvm/test/Transforms/LoopVectorize/optimal-epilog-vectorization-limitations.ll

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D116928.398562.patch
Type: text/x-patch
Size: 114981 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20220110/fb2d48b0/attachment.bin>