[llvm] [LV][VPlan] Add initial support for CSA vectorization (PR #121222)
via llvm-commits
llvm-commits at lists.llvm.org
Fri Dec 27 11:03:46 PST 2024
llvmbot wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-llvm-transforms
Author: Michael Maitland (michaelmaitland)
<details>
<summary>Changes</summary>
This patch adds initial support for CSA vectorization LLVM. This new class
can be characterized by vectorization of assignment to a scalar in a loop,
such that the assignment is conditional from the perspective of its use.
An assignment is conditional in a loop if a value may or may not be assigned
in the loop body.
For example:
```
int t = init_val;
for (int i = 0; i < N; i++) {
if (cond[i])
t = a[i];
}
s = t; // use t
```
Using pseudo-LLVM code this can be vectorized as
```
vector.ph:
...
%t = %init_val
%init.mask = <all-false-vec>
%init.data = <poison-vec> ; uninitialized
vector.body:
...
%mask.phi = phi [%init.mask, %vector.ph], [%new.mask, %vector.body]
%data.phi = phi [%data.mask, %vector.ph], [%new.mask, %vector.body]
%cond.vec = <widened-cmp> ...
%a.vec = <widened-load> %a, %i
%b = <any-lane-active> %cond.vec
%new.mask = select %b, %cond.vec, %mask.phi
%new.data = select %b, %a.vec, %data.phi
...
middle.block:
%s = <extract-last-active-lane> %new.mask, %new.data
```
On each iteration, we track whether any lane in the widened condition was active,
and if it was take the current mask and data as the new mask and data vector.
Then at the end of the loop, the scalar can be extracted only once.
This transformation works the same way for integer, pointer, and floating point
conditional assignment, since the transformation does not require inspection
of the data being assigned.
In the vectorization of a CSA, we will be introducing recipes into the vector
preheader, the vector body, and the middle block. Recipes that are introduced
into the preheader and middle block are executed only one time, and recipes
that are in the vector body will be possibly executed multiple times. The more
times that the vector body is executed, the less of an impact the preheader
and middle block cost have on the overall cost of a CSA.
A detailed explanation of the concept can be found [here](https://discourse.llvm.org/t/vectorization-of-conditional-scalar-assignment-csa/80964).
This patch is further tested in https://github.com/llvm/llvm-test-suite/pull/155.
This patch contains only the non-EVL related code. The is based on the larger
patch of https://github.com/llvm/llvm-project/pull/106560, which contained both
EVL and non-EVL related parts.
---
Patch is 232.25 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/121222.diff
18 Files Affected:
- (modified) llvm/include/llvm/Analysis/IVDescriptors.h (+67-1)
- (modified) llvm/include/llvm/Analysis/TargetTransformInfo.h (+9)
- (modified) llvm/include/llvm/Analysis/TargetTransformInfoImpl.h (+2)
- (modified) llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h (+27)
- (modified) llvm/lib/Analysis/IVDescriptors.cpp (+59-1)
- (modified) llvm/lib/Analysis/TargetTransformInfo.cpp (+5)
- (modified) llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp (+5)
- (modified) llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h (+4)
- (modified) llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp (+37-4)
- (modified) llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h (+22-2)
- (modified) llvm/lib/Transforms/Vectorize/LoopVectorize.cpp (+136-9)
- (modified) llvm/lib/Transforms/Vectorize/VPlan.cpp (+1-1)
- (modified) llvm/lib/Transforms/Vectorize/VPlan.h (+164-1)
- (modified) llvm/lib/Transforms/Vectorize/VPlanAnalysis.cpp (+9-1)
- (modified) llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp (+238)
- (modified) llvm/lib/Transforms/Vectorize/VPlanValue.h (+3)
- (modified) llvm/lib/Transforms/Vectorize/VPlanVerifier.cpp (+1-1)
- (added) llvm/test/Transforms/LoopVectorize/RISCV/conditional-scalar-assignment.ll (+3021)
``````````diff
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index e8041e22b031ce..b085f4bd173afc 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -6,7 +6,8 @@
//
//===----------------------------------------------------------------------===//
//
-// This file "describes" induction and recurrence variables.
+// This file "describes" induction, recurrence, and conditional scalar
+// assignment variables.
//
//===----------------------------------------------------------------------===//
@@ -423,6 +424,71 @@ class InductionDescriptor {
SmallVector<Instruction *, 2> RedundantCasts;
};
+/// A Conditional Scalar Assignment is an assignment from an initial
+/// scalar that may or may not occur.
+class ConditionalScalarAssignmentDescriptor {
+ /// If the conditional assignment occurs inside a loop, then Phi chooses
+ /// the value of the assignment from the entry block or the loop body block.
+ PHINode *Phi = nullptr;
+
+ /// The initial value of the ConditionalScalarAssignment. If the condition
+ /// guarding the assignment is not met, then the assignment retains this
+ /// value.
+ Value *InitScalar = nullptr;
+
+ /// The Instruction that conditionally assigned to inside the loop.
+ Instruction *Assignment = nullptr;
+
+ /// Create a ConditionalScalarAssignmentDescriptor that models a valid
+ /// conditional scalar assignment with its members initialized correctly.
+ ConditionalScalarAssignmentDescriptor(PHINode *Phi, Instruction *Assignment,
+ Value *InitScalar)
+ : Phi(Phi), InitScalar(InitScalar), Assignment(Assignment) {}
+
+public:
+ /// Create a ConditionalScalarAssignmentDescriptor that models an invalid
+ /// ConditionalScalarAssignment.
+ ConditionalScalarAssignmentDescriptor() = default;
+
+ /// If Phi is the root of a ConditionalScalarAssignment, set
+ /// ConditionalScalarAssignmentDesc as the ConditionalScalarAssignment rooted
+ /// by Phi. Otherwise, return a false, leaving ConditionalScalarAssignmentDesc
+ /// unmodified.
+ static bool isConditionalScalarAssignmentPhi(
+ PHINode *Phi, Loop *TheLoop,
+ ConditionalScalarAssignmentDescriptor &Desc);
+
+ operator bool() const { return isValid(); }
+
+ /// Returns whether SI is the Assignment in ConditionalScalarAssignment
+ static bool isConditionalScalarAssignmentSelect(
+ ConditionalScalarAssignmentDescriptor Desc, SelectInst *SI) {
+ return Desc.getAssignment() == SI;
+ }
+
+ /// Return whether this ConditionalScalarAssignmentDescriptor models a valid
+ /// ConditionalScalarAssignment.
+ bool isValid() const { return Phi && InitScalar && Assignment; }
+
+ /// Return the PHI that roots this ConditionalScalarAssignment.
+ PHINode *getPhi() const { return Phi; }
+
+ /// Return the initial value of the ConditionalScalarAssignment. This is the
+ /// value if the conditional assignment does not occur.
+ Value *getInitScalar() const { return InitScalar; }
+
+ /// The Instruction that is used after the loop
+ Instruction *getAssignment() const { return Assignment; }
+
+ /// Return the condition that this ConditionalScalarAssignment is conditional
+ /// upon.
+ Value *getCond() const {
+ if (auto *SI = dyn_cast_or_null<SelectInst>(Assignment))
+ return SI->getCondition();
+ return nullptr;
+ }
+};
+
} // end namespace llvm
#endif // LLVM_ANALYSIS_IVDESCRIPTORS_H
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfo.h b/llvm/include/llvm/Analysis/TargetTransformInfo.h
index c6b846f96f1622..b41dbb6582f76f 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfo.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfo.h
@@ -1852,6 +1852,10 @@ class TargetTransformInfo {
: EVLParamStrategy(EVLParamStrategy), OpStrategy(OpStrategy) {}
};
+ /// \returns true if the loop vectorizer should vectorize conditional
+ /// scalar assignments for the target.
+ bool enableConditionalScalarAssignmentVectorization() const;
+
/// \returns How the target needs this vector-predicated operation to be
/// transformed.
VPLegalization getVPLegalizationStrategy(const VPIntrinsic &PI) const;
@@ -2305,6 +2309,7 @@ class TargetTransformInfo::Concept {
SmallVectorImpl<Use *> &OpsToSink) const = 0;
virtual bool isVectorShiftByScalarCheap(Type *Ty) const = 0;
+ virtual bool enableConditionalScalarAssignmentVectorization() const = 0;
virtual VPLegalization
getVPLegalizationStrategy(const VPIntrinsic &PI) const = 0;
virtual bool hasArmWideBranch(bool Thumb) const = 0;
@@ -3130,6 +3135,10 @@ class TargetTransformInfo::Model final : public TargetTransformInfo::Concept {
return Impl.isVectorShiftByScalarCheap(Ty);
}
+ bool enableConditionalScalarAssignmentVectorization() const override {
+ return Impl.enableConditionalScalarAssignmentVectorization();
+ }
+
VPLegalization
getVPLegalizationStrategy(const VPIntrinsic &PI) const override {
return Impl.getVPLegalizationStrategy(PI);
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
index 5fa0c46ad292d8..0cb081d48991f2 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
@@ -1030,6 +1030,8 @@ class TargetTransformInfoImplBase {
bool isVectorShiftByScalarCheap(Type *Ty) const { return false; }
+ bool enableConditionalScalarAssignmentVectorization() const { return false; }
+
TargetTransformInfo::VPLegalization
getVPLegalizationStrategy(const VPIntrinsic &PI) const {
return TargetTransformInfo::VPLegalization(
diff --git a/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h b/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
index fbe80eddbae07a..48c9077d653278 100644
--- a/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
+++ b/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
@@ -269,6 +269,12 @@ class LoopVectorizationLegality {
/// induction descriptor.
using InductionList = MapVector<PHINode *, InductionDescriptor>;
+ /// ConditionalScalarAssignmentList contains the
+ /// ConditionalScalarAssignmentDescriptors for all the conditional scalar
+ /// assignments that were found in the loop, rooted by their phis.
+ using ConditionalScalarAssignmentList =
+ MapVector<PHINode *, ConditionalScalarAssignmentDescriptor>;
+
/// RecurrenceSet contains the phi nodes that are recurrences other than
/// inductions and reductions.
using RecurrenceSet = SmallPtrSet<const PHINode *, 8>;
@@ -321,6 +327,18 @@ class LoopVectorizationLegality {
/// Returns True if V is a Phi node of an induction variable in this loop.
bool isInductionPhi(const Value *V) const;
+ /// Returns the conditional scalar assignments found in the loop.
+ const ConditionalScalarAssignmentList &
+ getConditionalScalarAssignments() const {
+ return ConditionalScalarAssignments;
+ }
+
+ /// Returns true if Phi is the root of a conditional scalar assignments in the
+ /// loop.
+ bool isConditionalScalarAssignmentPhi(PHINode *Phi) const {
+ return ConditionalScalarAssignments.count(Phi) != 0;
+ }
+
/// Returns a pointer to the induction descriptor, if \p Phi is an integer or
/// floating point induction.
const InductionDescriptor *getIntOrFpInductionDescriptor(PHINode *Phi) const;
@@ -550,6 +568,12 @@ class LoopVectorizationLegality {
void addInductionPhi(PHINode *Phi, const InductionDescriptor &ID,
SmallPtrSetImpl<Value *> &AllowedExit);
+ /// Updates the vetorization state by adding \p Phi to the
+ /// ConditionalScalarAssignment list.
+ void addConditionalScalarAssignmentPhi(
+ PHINode *Phi, const ConditionalScalarAssignmentDescriptor &Desc,
+ SmallPtrSetImpl<Value *> &AllowedExit);
+
/// The loop that we evaluate.
Loop *TheLoop;
@@ -594,6 +618,9 @@ class LoopVectorizationLegality {
/// variables can be pointers.
InductionList Inductions;
+ /// Holds the conditional scalar assignments
+ ConditionalScalarAssignmentList ConditionalScalarAssignments;
+
/// Holds all the casts that participate in the update chain of the induction
/// variables, and that have been proven to be redundant (possibly under a
/// runtime guard). These casts can be ignored when creating the vectorized
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index f74ede4450ce52..259af79f6cdba5 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -6,7 +6,8 @@
//
//===----------------------------------------------------------------------===//
//
-// This file "describes" induction and recurrence variables.
+// This file "describes" induction, recurrence, and conditional scalar
+// assignment variables.
//
//===----------------------------------------------------------------------===//
@@ -1570,3 +1571,60 @@ bool InductionDescriptor::isInductionPHI(
D = InductionDescriptor(StartValue, IK_PtrInduction, Step);
return true;
}
+
+/// Return ConditionalScalarAssignmentDescriptor that describes a
+/// ConditionalScalarAssignment that matches one of these patterns:
+/// phi loop_inv, (select cmp, value, phi)
+/// phi loop_inv, (select cmp, phi, value)
+/// phi (select cmp, value, phi), loop_inv
+/// phi (select cmp, phi, value), loop_inv
+/// If the ConditionalScalarAssignment does not match any of these paterns,
+/// return a ConditionalScalarAssignmentDescriptor that describes an
+/// InvalidConditionalScalarAssignment.
+bool ConditionalScalarAssignmentDescriptor::isConditionalScalarAssignmentPhi(
+ PHINode *Phi, Loop *TheLoop, ConditionalScalarAssignmentDescriptor &Desc) {
+
+ // Must be a scalar.
+ Type *Type = Phi->getType();
+ if (!Type->isIntegerTy() && !Type->isFloatingPointTy() &&
+ !Type->isPointerTy())
+ return false;
+
+ // Match phi loop_inv, (select cmp, value, phi)
+ // or phi loop_inv, (select cmp, phi, value)
+ // or phi (select cmp, value, phi), loop_inv
+ // or phi (select cmp, phi, value), loop_inv
+ if (Phi->getNumIncomingValues() != 2)
+ return false;
+ auto SelectInstIt = find_if(Phi->incoming_values(), [&Phi](const Use &U) {
+ return match(U.get(), m_Select(m_Value(), m_Specific(Phi), m_Value())) ||
+ match(U.get(), m_Select(m_Value(), m_Value(), m_Specific(Phi)));
+ });
+ if (SelectInstIt == Phi->incoming_values().end())
+ return false;
+ auto LoopInvIt = find_if(Phi->incoming_values(), [&](Use &U) {
+ return U.get() != *SelectInstIt && TheLoop->isLoopInvariant(U.get());
+ });
+ if (LoopInvIt == Phi->incoming_values().end())
+ return false;
+
+ // Phi or Sel must be used only outside the loop,
+ // excluding if Phi use Sel or Sel use Phi
+ auto IsOnlyUsedOutsideLoop = [&](Value *V, Value *Ignore) {
+ return all_of(V->users(), [Ignore, TheLoop](User *U) {
+ if (U == Ignore)
+ return true;
+ if (auto *I = dyn_cast<Instruction>(U))
+ return !TheLoop->contains(I);
+ return true;
+ });
+ };
+ Instruction *Select = cast<SelectInst>(SelectInstIt->get());
+ Value *LoopInv = LoopInvIt->get();
+ if (!IsOnlyUsedOutsideLoop(Phi, Select) ||
+ !IsOnlyUsedOutsideLoop(Select, Phi))
+ return false;
+
+ Desc = ConditionalScalarAssignmentDescriptor(Phi, Select, LoopInv);
+ return true;
+}
diff --git a/llvm/lib/Analysis/TargetTransformInfo.cpp b/llvm/lib/Analysis/TargetTransformInfo.cpp
index c62e40db0c5775..2468227be4a0da 100644
--- a/llvm/lib/Analysis/TargetTransformInfo.cpp
+++ b/llvm/lib/Analysis/TargetTransformInfo.cpp
@@ -1373,6 +1373,11 @@ bool TargetTransformInfo::preferEpilogueVectorization() const {
return TTIImpl->preferEpilogueVectorization();
}
+bool TargetTransformInfo::enableConditionalScalarAssignmentVectorization()
+ const {
+ return TTIImpl->enableConditionalScalarAssignmentVectorization();
+}
+
TargetTransformInfo::VPLegalization
TargetTransformInfo::getVPLegalizationStrategy(const VPIntrinsic &VPI) const {
return TTIImpl->getVPLegalizationStrategy(VPI);
diff --git a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
index 49192bd6380223..e469c0225cb0a3 100644
--- a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
+++ b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
@@ -2361,6 +2361,11 @@ bool RISCVTTIImpl::isLegalMaskedExpandLoad(Type *DataTy, Align Alignment) {
return true;
}
+bool RISCVTTIImpl::enableConditionalScalarAssignmentVectorization() const {
+ return ST->hasVInstructions() &&
+ ST->getProcFamily() == RISCVSubtarget::SiFive7;
+}
+
bool RISCVTTIImpl::isLegalMaskedCompressStore(Type *DataTy, Align Alignment) {
auto *VTy = dyn_cast<VectorType>(DataTy);
if (!VTy || VTy->isScalableTy())
diff --git a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h
index bd90bfed6e2c95..9e1b2cb3f3043f 100644
--- a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h
+++ b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h
@@ -306,6 +306,10 @@ class RISCVTTIImpl : public BasicTTIImplBase<RISCVTTIImpl> {
return TLI->isVScaleKnownToBeAPowerOfTwo();
}
+ /// \returns true if the loop vectorizer should vectorize conditional
+ /// scalar assignments for the target.
+ bool enableConditionalScalarAssignmentVectorization() const;
+
/// \returns How the target needs this vector-predicated operation to be
/// transformed.
TargetTransformInfo::VPLegalization
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
index cb0b4641b6492b..2835d9f385ac5f 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
@@ -83,6 +83,10 @@ static cl::opt<bool> EnableHistogramVectorization(
"enable-histogram-loop-vectorization", cl::init(false), cl::Hidden,
cl::desc("Enables autovectorization of some loops containing histograms"));
+static cl::opt<bool> EnableConditionalScalarAssignment(
+ "enable-csa-vectorization", cl::init(false), cl::Hidden,
+ cl::desc("Control whether loop vectorization is enabled"));
+
/// Maximum vectorization interleave count.
static const unsigned MaxInterleaveFactor = 16;
@@ -749,6 +753,18 @@ bool LoopVectorizationLegality::setupOuterLoopInductions() {
return llvm::all_of(Header->phis(), IsSupportedPhi);
}
+void LoopVectorizationLegality::addConditionalScalarAssignmentPhi(
+ PHINode *Phi, const ConditionalScalarAssignmentDescriptor &Desc,
+ SmallPtrSetImpl<Value *> &AllowedExit) {
+ assert(Desc.isValid() &&
+ "Expected Valid ConditionalScalarAssignmentDescriptor");
+ LLVM_DEBUG(
+ dbgs() << "LV: found legal conditional scalar assignment opportunity"
+ << *Phi << "\n");
+ AllowedExit.insert(Phi);
+ ConditionalScalarAssignments.insert({Phi, Desc});
+}
+
/// Checks if a function is scalarizable according to the TLI, in
/// the sense that it should be vectorized and then expanded in
/// multiple scalar calls. This is represented in the
@@ -866,14 +882,27 @@ bool LoopVectorizationLegality::canVectorizeInstrs() {
continue;
}
- // As a last resort, coerce the PHI to a AddRec expression
- // and re-try classifying it a an induction PHI.
+ // Try to coerce the PHI to a AddRec expression and re-try classifying
+ // it a an induction PHI.
if (InductionDescriptor::isInductionPHI(Phi, TheLoop, PSE, ID, true) &&
!IsDisallowedStridedPointerInduction(ID)) {
addInductionPhi(Phi, ID, AllowedExit);
continue;
}
+ // Check if the PHI can be classified as a conditional scalar assignment
+ // PHI.
+ if (EnableConditionalScalarAssignment ||
+ (TTI->enableConditionalScalarAssignmentVectorization() &&
+ EnableConditionalScalarAssignment.getNumOccurrences() == 0)) {
+ ConditionalScalarAssignmentDescriptor Desc;
+ if (ConditionalScalarAssignmentDescriptor::
+ isConditionalScalarAssignmentPhi(Phi, TheLoop, Desc)) {
+ addConditionalScalarAssignmentPhi(Phi, Desc, AllowedExit);
+ continue;
+ }
+ }
+
reportVectorizationFailure("Found an unidentified PHI",
"value that could not be identified as "
"reduction is used outside the loop",
@@ -1844,11 +1873,15 @@ bool LoopVectorizationLegality::canFoldTailByMasking() const {
for (const auto &Reduction : getReductionVars())
ReductionLiveOuts.insert(Reduction.second.getLoopExitInstr());
+ SmallPtrSet<const Value *, 8> CSALiveOuts;
+ for (const auto &CSA : getConditionalScalarAssignments())
+ CSALiveOuts.insert(CSA.second.getAssignment());
+
// TODO: handle non-reduction outside users when tail is folded by masking.
for (auto *AE : AllowedExit) {
// Check that all users of allowed exit values are inside the loop or
- // are the live-out of a reduction.
- if (ReductionLiveOuts.count(AE))
+ // are the live-out of a reduction or conditional scalar assignment.
+ if (ReductionLiveOuts.count(AE) || CSALiveOuts.count(AE))
continue;
for (User *U : AE->users()) {
Instruction *UI = cast<Instruction>(U);
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
index 650a4859780da2..6eba380ceb7a12 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
@@ -174,8 +174,8 @@ class VPBuilder {
new VPInstruction(Opcode, Operands, WrapFlags, DL, Name));
}
- VPValue *createNot(VPValue *Operand, DebugLoc DL = {},
- const Twine &Name = "") {
+ VPInstruction *createNot(VPValue *Operand, DebugLoc DL = {},
+ const Twine &Name = "") {
return createInstruction(VPInstruction::Not, {Operand}, DL, Name);
}
@@ -257,6 +257,26 @@ class VPBuilder {
FPBinOp ? FPBinOp->getFastMathFlags() : FastMathFlags()));
}
+ VPInstruction *createConditionalScalarAssignmentMaskPhi(VPValue *InitMask,
+ DebugLoc DL,
+ const Twine &Name) {
+ return createInstruction(VPInstruction::ConditionalScalarAssignmentMaskPhi,
+ {InitMask}, DL, Name);
+ }
+
+ VPInstruction *createAnyOf(VPValue *Cond, DebugLoc DL, const Twine &Name) {
+ return createInstruction(VPInstruction::AnyOf, {Cond}, DL, Name);
+ }
+
+ VPInstruction *createConditionalScalarAssignmentMaskSel(VPValue *Cond,
+ VPValue *MaskPhi,
+ VPValue *AnyOf,
+ DebugLoc DL,
+ const Twine &Name) {
+ return createInstruction(VPInstruction::ConditionalScalarAssignmentMaskSel,
+ {Cond, MaskPhi, AnyOf}, DL, Name);
+ }
+
//===--------------------------------------------------------------------===//
// RAII helpers.
//===--------------------------------------------------------------------===//
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 355ff40ce770e7..7102e0437caf5c 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -174,6 +174,8 @@ const char LLVMLoopVectorizeFollowupEpilogue[] =
STATISTIC(LoopsVectorized, "Number of loops vectorized");
STATISTIC(LoopsAnalyzed, "Number of loops analyzed for vectorization");
STATISTIC(LoopsEpilogueVectorized, "Number of epilogues vectorized");
+STATISTIC(ConditionalScalarAssignmentsVectorized,
+ "Number of conditional scalar assignments vectorized");
static cl::opt<bool> EnableEpilogueVectorization(
"enable-epilogue-vectorization", cl::init(true), cl::Hidden,
@@ -4635,6 +4637,9 @@ static bool willGenerateVect...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/121222
More information about the llvm-commits
mailing list