[all-commits] [llvm/llvm-project] 8082ab: [LoopVectorize] Support epilogue vectorisation of ...

[kmclaughlin-arm via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 8082ab2fc391cb88ab142d3ce8f63e6eb8641a23
      https://github.com/llvm/llvm-project/commit/8082ab2fc391cb88ab142d3ce8f63e6eb8641a23
  Author: Kerry McLaughlin <kerry.mclaughlin at arm.com>
  Date:   2022-01-24 (Mon, 24 Jan 2022)

  Changed paths:
    M llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
    A llvm/test/Transforms/LoopVectorize/AArch64/sve-epilog-vect-inloop-reductions.ll
    A llvm/test/Transforms/LoopVectorize/AArch64/sve-epilog-vect-reductions.ll
    A llvm/test/Transforms/LoopVectorize/AArch64/sve-epilog-vect-strict-reductions.ll
    M llvm/test/Transforms/LoopVectorize/X86/invariant-store-vectorization.ll
    M llvm/test/Transforms/LoopVectorize/X86/pr35432.ll
    M llvm/test/Transforms/LoopVectorize/X86/pr42674.ll
    A llvm/test/Transforms/LoopVectorize/epilog-vectorization-reductions.ll
    M llvm/test/Transforms/LoopVectorize/optimal-epilog-vectorization-limitations.ll

  Log Message:
  -----------
  [LoopVectorize] Support epilogue vectorisation of loops with reductions

isCandidateForEpilogueVectorization will currently return false for loops
which contain reductions. This patch removes this restriction and makes
the following changes to support epilogue vectorisation with reductions:

- `fixReduction`: If fixReduction is being called during vectorisation of the
    epilogue, the phi node it creates will need to additionally carry incoming
     values from the middle block of the main loop.

- `createEpilogueVectorizedLoopSkeleton`: The incoming values of the phi
    created by fixReduction are updated after the vec.epilog.iter.check block
    is added. The phi is also moved to the preheader of the epilogue.

- `processLoop`: The start value of any VPReductionPHIRecipes are updated before
    vectorising the epilogue loop. The getResumeInstr function added to the ILV
    will return the resume instruction associated with the recurrence descriptor.

Reviewed By: sdesmalen

Differential Revision: https://reviews.llvm.org/D116928