[PATCH] D98435: [LoopVectorize] Add strict in-order reduction support for fixed-width vectorization

[Kerry McLaughlin via Phabricator via llvm-commits]

kmclaughlin created this revision.
kmclaughlin added reviewers: sdesmalen, dmgreen, fhahn, david-arm, peterwaller-arm.
Herald added a subscriber: hiraditya.
kmclaughlin requested review of this revision.
Herald added a project: LLVM.
Herald added a subscriber: llvm-commits.

Previously we could only vectorize FP reductions if fast math was enabled, as this allows us to
reorder FP operations. However, it may still be beneficial to vectorize the loop by moving
the reduction inside the vectorized loop and making sure that the scalar reduction value
be an input to the horizontal reduction, e.g:

  %phi = phi float [ 0.0, %entry ], [ %reduction, %vector_body ]
  %load = load <8 x float>
  %reduction = call float @llvm.vector.reduce.fadd.v8f32(float %phi, <8 x float> %load)

For loops with conditional ordered reductions, we need to insert identity values for the lanes
that are masked out since the llvm.vector.reduce intrinsics do not take a predicate mask, e.g:

  for(int i=0; i<N; ++i)
    if (values[i] > 0.0)
      sum += values[i];

This patch adds a new flag (IsOrdered) to RecurrenceDescriptor and makes use of the changes added
by D75069 <https://reviews.llvm.org/D75069> as much as possible, which already teaches the vectorizer about in-loop reductions.
For now in-order reduction support is off by default and controlled with the `-enable-strict-reducs` flag.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D98435

Files:
  llvm/include/llvm/Analysis/IVDescriptors.h
  llvm/include/llvm/Transforms/Utils/LoopUtils.h
  llvm/lib/Analysis/IVDescriptors.cpp
  llvm/lib/Transforms/Utils/LoopUtils.cpp
  llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
  llvm/test/Transforms/LoopVectorize/AArch64/strict-fadd.ll

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D98435.329992.patch
Type: text/x-patch
Size: 20446 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20210311/d566aebd/attachment.bin>