[llvm] e976168 - [LoopVectorizer] Inloop vector reductions
Philip Reames via llvm-commits
llvm-commits at lists.llvm.org
Wed Aug 5 10:19:20 PDT 2020
David,
I'm missing something reading your description of the change. What's the
advantage to doing the reduction "in loop" as opposed to the existing
N-lanes in parallel w/post loop reduction? I definitely see that it's
*possible*, but where's the performance advantage?
You might have hinted at this when you said we can vectorize some
patterns with this we can't otherwise. An example doesn't pop to mind,
could you give an illustrative example?
Philip
On 8/5/20 10:14 AM, David Green via llvm-commits wrote:
> Author: David Green
> Date: 2020-08-05T18:14:05+01:00
> New Revision: e9761688e41cb979a1fa6a79eb18145a75104933
>
> URL: https://github.com/llvm/llvm-project/commit/e9761688e41cb979a1fa6a79eb18145a75104933
> DIFF: https://github.com/llvm/llvm-project/commit/e9761688e41cb979a1fa6a79eb18145a75104933.diff
>
> LOG: [LoopVectorizer] Inloop vector reductions
>
> Arm MVE has multiple instructions such as VMLAVA.s8, which (in this
> case) can take two 128bit vectors, sign extend the inputs to i32,
> multiplying them together and sum the result into a 32bit general
> purpose register. So taking 16 i8's as inputs, they can multiply and
> accumulate the result into a single i32 without any rounding/truncating
> along the way. There are also reduction instructions for plain integer
> add and min/max, and operations that sum into a pair of 32bit registers
> together treated as a 64bit integer (even though MVE does not have a
> plain 64bit addition instruction). So giving the vectorizer the ability
> to use these instructions both enables us to vectorize at higher
> bitwidths, and to vectorize things we previously could not.
>
> In order to do that we need a way to represent that the reduction
> operation, specified with a llvm.experimental.vector.reduce when
> vectorizing for Arm, occurs inside the loop not after it like most
> reductions. This patch attempts to do that, teaching the vectorizer
> about in-loop reductions. It does this through a vplan recipe
> representing the reductions that the original chain of reduction
> operations is replaced by. Cost modelling is currently just done through
> a prefersInloopReduction TTI hook (which follows in a later patch).
>
> Differential Revision: https://reviews.llvm.org/D75069
>
> Added:
>
>
> Modified:
> llvm/include/llvm/Analysis/IVDescriptors.h
> llvm/lib/Analysis/IVDescriptors.cpp
> llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
> llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
> llvm/lib/Transforms/Vectorize/VPlan.cpp
> llvm/lib/Transforms/Vectorize/VPlan.h
> llvm/test/Transforms/LoopVectorize/reduction-inloop-uf4.ll
> llvm/test/Transforms/LoopVectorize/reduction-inloop.ll
>
> Removed:
>
>
>
> ################################################################################
> diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
> index 1bae83d13c7a..5e66ae3014ba 100644
> --- a/llvm/include/llvm/Analysis/IVDescriptors.h
> +++ b/llvm/include/llvm/Analysis/IVDescriptors.h
> @@ -220,6 +220,11 @@ class RecurrenceDescriptor {
> /// Returns true if all source operands of the recurrence are SExtInsts.
> bool isSigned() { return IsSigned; }
>
> + /// Attempts to find a chain of operations from Phi to LoopExitInst that can
> + /// be treated as a set of reductions instructions for in-loop reductions.
> + SmallVector<Instruction *, 4> getReductionOpChain(PHINode *Phi,
> + Loop *L) const;
> +
> private:
> // The starting value of the recurrence.
> // It does not have to be zero!
>
> diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
> index 6686848d75c9..ce18243805a6 100644
> --- a/llvm/lib/Analysis/IVDescriptors.cpp
> +++ b/llvm/lib/Analysis/IVDescriptors.cpp
> @@ -437,6 +437,8 @@ bool RecurrenceDescriptor::AddReductionVar(PHINode *Phi, RecurrenceKind Kind,
> // instructions that are a part of the reduction. The vectorizer cost
> // model could then apply the recurrence type to these instructions,
> // without needing a white list of instructions to ignore.
> + // This may also be useful for the inloop reductions, if it can be
> + // kept simple enough.
> collectCastsToIgnore(TheLoop, ExitInstruction, RecurrenceType, CastInsts);
> }
>
> @@ -796,6 +798,76 @@ unsigned RecurrenceDescriptor::getRecurrenceBinOp(RecurrenceKind Kind) {
> }
> }
>
> +SmallVector<Instruction *, 4>
> +RecurrenceDescriptor::getReductionOpChain(PHINode *Phi, Loop *L) const {
> + SmallVector<Instruction *, 8> ReductionOperations;
> + unsigned RedOp = getRecurrenceBinOp(Kind);
> +
> + // Search down from the Phi to the LoopExitInstr, looking for instructions
> + // with a single user of the correct type for the reduction.
> +
> + // Note that we check that the type of the operand is correct for each item in
> + // the chain, including the last (the loop exit value). This can come up from
> + // sub, which would otherwise be treated as an add reduction. MinMax also need
> + // to check for a pair of icmp/select, for which we use getNextInstruction and
> + // isCorrectOpcode functions to step the right number of instruction, and
> + // check the icmp/select pair.
> + // FIXME: We also do not attempt to look through Phi/Select's yet, which might
> + // be part of the reduction chain, or attempt to looks through And's to find a
> + // smaller bitwidth. Subs are also currently not allowed (which are usually
> + // treated as part of a add reduction) as they are expected to generally be
> + // more expensive than out-of-loop reductions, and need to be costed more
> + // carefully.
> + unsigned ExpectedUses = 1;
> + if (RedOp == Instruction::ICmp || RedOp == Instruction::FCmp)
> + ExpectedUses = 2;
> +
> + auto getNextInstruction = [&](Instruction *Cur) {
> + if (RedOp == Instruction::ICmp || RedOp == Instruction::FCmp) {
> + // We are expecting a icmp/select pair, which we go to the next select
> + // instruction if we can. We already know that Cur has 2 uses.
> + if (isa<SelectInst>(*Cur->user_begin()))
> + return cast<Instruction>(*Cur->user_begin());
> + else
> + return cast<Instruction>(*std::next(Cur->user_begin()));
> + }
> + return cast<Instruction>(*Cur->user_begin());
> + };
> + auto isCorrectOpcode = [&](Instruction *Cur) {
> + if (RedOp == Instruction::ICmp || RedOp == Instruction::FCmp) {
> + Value *LHS, *RHS;
> + return SelectPatternResult::isMinOrMax(
> + matchSelectPattern(Cur, LHS, RHS).Flavor);
> + }
> + return Cur->getOpcode() == RedOp;
> + };
> +
> + // The loop exit instruction we check first (as a quick test) but add last. We
> + // check the opcode is correct (and dont allow them to be Subs) and that they
> + // have expected to have the expected number of uses. They will have one use
> + // from the phi and one from a LCSSA value, no matter the type.
> + if (!isCorrectOpcode(LoopExitInstr) || !LoopExitInstr->hasNUses(2))
> + return {};
> +
> + // Check that the Phi has one (or two for min/max) uses.
> + if (!Phi->hasNUses(ExpectedUses))
> + return {};
> + Instruction *Cur = getNextInstruction(Phi);
> +
> + // Each other instruction in the chain should have the expected number of uses
> + // and be the correct opcode.
> + while (Cur != LoopExitInstr) {
> + if (!isCorrectOpcode(Cur) || !Cur->hasNUses(ExpectedUses))
> + return {};
> +
> + ReductionOperations.push_back(Cur);
> + Cur = getNextInstruction(Cur);
> + }
> +
> + ReductionOperations.push_back(Cur);
> + return ReductionOperations;
> +}
> +
> InductionDescriptor::InductionDescriptor(Value *Start, InductionKind K,
> const SCEV *Step, BinaryOperator *BOp,
> SmallVectorImpl<Instruction *> *Casts)
>
> diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
> index 8dd06983cd84..ecf6c8402cd6 100644
> --- a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
> +++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
> @@ -34,6 +34,7 @@ namespace llvm {
> class LoopVectorizationLegality;
> class LoopVectorizationCostModel;
> class PredicatedScalarEvolution;
> +class VPRecipeBuilder;
>
> /// VPlan-based builder utility analogous to IRBuilder.
> class VPBuilder {
> @@ -294,6 +295,13 @@ class LoopVectorizationPlanner {
> /// according to the information gathered by Legal when it checked if it is
> /// legal to vectorize the loop. This method creates VPlans using VPRecipes.
> void buildVPlansWithVPRecipes(unsigned MinVF, unsigned MaxVF);
> +
> + /// Adjust the recipes for any inloop reductions. The chain of instructions
> + /// leading from the loop exit instr to the phi need to be converted to
> + /// reductions, with one operand being vector and the other being the scalar
> + /// reduction chain.
> + void adjustRecipesForInLoopReductions(VPlanPtr &Plan,
> + VPRecipeBuilder &RecipeBuilder);
> };
>
> } // namespace llvm
>
> diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
> index 7bf846d2a617..3c96a9556dd8 100644
> --- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
> +++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
> @@ -265,6 +265,12 @@ static cl::opt<unsigned> MaxNestedScalarReductionIC(
> cl::desc("The maximum interleave count to use when interleaving a scalar "
> "reduction in a nested loop."));
>
> +static cl::opt<bool>
> + PreferInLoopReductions("prefer-inloop-reductions", cl::init(false),
> + cl::Hidden,
> + cl::desc("Prefer in-loop vector reductions, "
> + "overriding the targets preference."));
> +
> cl::opt<bool> EnableVPlanNativePath(
> "enable-vplan-native-path", cl::init(false), cl::Hidden,
> cl::desc("Enable VPlan-native vectorization path with "
> @@ -1071,6 +1077,10 @@ class LoopVectorizationCostModel {
> /// Collect values we want to ignore in the cost model.
> void collectValuesToIgnore();
>
> + /// Split reductions into those that happen in the loop, and those that happen
> + /// outside. In loop reductions are collected into InLoopReductionChains.
> + void collectInLoopReductions();
> +
> /// \returns The smallest bitwidth each instruction can be represented with.
> /// The vector equivalents of these instructions should be truncated to this
> /// type.
> @@ -1338,6 +1348,22 @@ class LoopVectorizationCostModel {
> return foldTailByMasking() || Legal->blockNeedsPredication(BB);
> }
>
> + /// A SmallMapVector to store the InLoop reduction op chains, mapping phi
> + /// nodes to the chain of instructions representing the reductions. Uses a
> + /// MapVector to ensure deterministic iteration order.
> + using ReductionChainMap =
> + SmallMapVector<PHINode *, SmallVector<Instruction *, 4>, 4>;
> +
> + /// Return the chain of instructions representing an inloop reduction.
> + const ReductionChainMap &getInLoopReductionChains() const {
> + return InLoopReductionChains;
> + }
> +
> + /// Returns true if the Phi is part of an inloop reduction.
> + bool isInLoopReduction(PHINode *Phi) const {
> + return InLoopReductionChains.count(Phi);
> + }
> +
> /// Estimate cost of an intrinsic call instruction CI if it were vectorized
> /// with factor VF. Return the cost of the instruction, including
> /// scalarization overhead if it's needed.
> @@ -1466,6 +1492,11 @@ class LoopVectorizationCostModel {
> /// scalarized.
> DenseMap<unsigned, SmallPtrSet<Instruction *, 4>> ForcedScalars;
>
> + /// PHINodes of the reductions that should be expanded in-loop along with
> + /// their associated chains of reduction operations, in program order from top
> + /// (PHI) to bottom
> + ReductionChainMap InLoopReductionChains;
> +
> /// Returns the expected
> diff erence in cost from scalarizing the expression
> /// feeding a predicated instruction \p PredInst. The instructions to
> /// scalarize and their scalar costs are collected in \p ScalarCosts. A
> @@ -3795,6 +3826,7 @@ void InnerLoopVectorizer::fixReduction(PHINode *Phi) {
> RecurrenceDescriptor::MinMaxRecurrenceKind MinMaxKind =
> RdxDesc.getMinMaxRecurrenceKind();
> setDebugLocFromInst(Builder, ReductionStartValue);
> + bool IsInLoopReductionPhi = Cost->isInLoopReduction(Phi);
>
> // We need to generate a reduction vector from the incoming scalar.
> // To do so, we need to generate the 'identity' vector and override
> @@ -3812,7 +3844,7 @@ void InnerLoopVectorizer::fixReduction(PHINode *Phi) {
> if (RK == RecurrenceDescriptor::RK_IntegerMinMax ||
> RK == RecurrenceDescriptor::RK_FloatMinMax) {
> // MinMax reduction have the start value as their identify.
> - if (VF == 1) {
> + if (VF == 1 || IsInLoopReductionPhi) {
> VectorStart = Identity = ReductionStartValue;
> } else {
> VectorStart = Identity =
> @@ -3822,7 +3854,7 @@ void InnerLoopVectorizer::fixReduction(PHINode *Phi) {
> // Handle other reduction kinds:
> Constant *Iden = RecurrenceDescriptor::getRecurrenceIdentity(
> RK, VecTy->getScalarType());
> - if (VF == 1) {
> + if (VF == 1 || IsInLoopReductionPhi) {
> Identity = Iden;
> // This vector is the Identity vector where the first element is the
> // incoming scalar reduction.
> @@ -3890,6 +3922,7 @@ void InnerLoopVectorizer::fixReduction(PHINode *Phi) {
> // then extend the loop exit value to enable InstCombine to evaluate the
> // entire expression in the smaller type.
> if (VF > 1 && Phi->getType() != RdxDesc.getRecurrenceType()) {
> + assert(!IsInLoopReductionPhi && "Unexpected truncated inloop reduction!");
> Type *RdxVecTy = FixedVectorType::get(RdxDesc.getRecurrenceType(), VF);
> Builder.SetInsertPoint(
> LI->getLoopFor(LoopVectorBody)->getLoopLatch()->getTerminator());
> @@ -3940,7 +3973,9 @@ void InnerLoopVectorizer::fixReduction(PHINode *Phi) {
> RdxPart);
> }
>
> - if (VF > 1) {
> + // Create the reduction after the loop. Note that inloop reductions create the
> + // target reduction in the loop using a Reduction recipe.
> + if (VF > 1 && !IsInLoopReductionPhi) {
> bool NoNaN = Legal->hasFunNoNaNAttr();
> ReducedPartRdx =
> createTargetReduction(Builder, TTI, RdxDesc, ReducedPartRdx, NoNaN);
> @@ -4236,8 +4271,9 @@ void InnerLoopVectorizer::widenPHIInstruction(Instruction *PN, unsigned UF,
> if (Legal->isReductionVariable(P) || Legal->isFirstOrderRecurrence(P)) {
> for (unsigned Part = 0; Part < UF; ++Part) {
> // This is phase one of vectorizing PHIs.
> + bool ScalarPHI = (VF == 1) || Cost->isInLoopReduction(cast<PHINode>(PN));
> Type *VecTy =
> - (VF == 1) ? PN->getType() : FixedVectorType::get(PN->getType(), VF);
> + ScalarPHI ? PN->getType() : FixedVectorType::get(PN->getType(), VF);
> Value *EntryPart = PHINode::Create(
> VecTy, 2, "vec.phi", &*LoopVectorBody->getFirstInsertionPt());
> VectorLoopValueMap.setVectorValue(P, Part, EntryPart);
> @@ -6644,6 +6680,34 @@ void LoopVectorizationCostModel::collectValuesToIgnore() {
> }
> }
>
> +void LoopVectorizationCostModel::collectInLoopReductions() {
> + // For the moment, without predicated reduction instructions, we do not
> + // support inloop reductions whilst folding the tail, and hence in those cases
> + // all reductions are currently out of the loop.
> + if (!PreferInLoopReductions || foldTailByMasking())
> + return;
> +
> + for (auto &Reduction : Legal->getReductionVars()) {
> + PHINode *Phi = Reduction.first;
> + RecurrenceDescriptor &RdxDesc = Reduction.second;
> +
> + // We don't collect reductions that are type promoted (yet).
> + if (RdxDesc.getRecurrenceType() != Phi->getType())
> + continue;
> +
> + // Check that we can correctly put the reductions into the loop, by
> + // finding the chain of operations that leads from the phi to the loop
> + // exit value.
> + SmallVector<Instruction *, 4> ReductionOperations =
> + RdxDesc.getReductionOpChain(Phi, TheLoop);
> + bool InLoop = !ReductionOperations.empty();
> + if (InLoop)
> + InLoopReductionChains[Phi] = ReductionOperations;
> + LLVM_DEBUG(dbgs() << "LV: Using " << (InLoop ? "inloop" : "out of loop")
> + << " reduction for phi: " << *Phi << "\n");
> + }
> +}
> +
> // TODO: we could return a pair of values that specify the max VF and
> // min VF, to be used in `buildVPlans(MinVF, MaxVF)` instead of
> // `buildVPlans(VF, VF)`. We cannot do it because VPLAN at the moment
> @@ -6723,6 +6787,7 @@ Optional<VectorizationFactor> LoopVectorizationPlanner::plan(unsigned UserVF,
> // Collect the instructions (and their associated costs) that will be more
> // profitable to scalarize.
> CM.selectUserVectorizationFactor(UserVF);
> + CM.collectInLoopReductions();
> buildVPlansWithVPRecipes(UserVF, UserVF);
> LLVM_DEBUG(printPlans(dbgs()));
> return {{UserVF, 0}};
> @@ -6741,6 +6806,8 @@ Optional<VectorizationFactor> LoopVectorizationPlanner::plan(unsigned UserVF,
> CM.collectInstsToScalarize(VF);
> }
>
> + CM.collectInLoopReductions();
> +
> buildVPlansWithVPRecipes(1, MaxVF);
> LLVM_DEBUG(printPlans(dbgs()));
> if (MaxVF == 1)
> @@ -7379,6 +7446,23 @@ VPlanPtr LoopVectorizationPlanner::buildVPlanWithVPRecipes(
> RecipeBuilder.recordRecipeOf(Entry.first);
> RecipeBuilder.recordRecipeOf(Entry.second);
> }
> + for (auto &Reduction : CM.getInLoopReductionChains()) {
> + PHINode *Phi = Reduction.first;
> + RecurrenceDescriptor::RecurrenceKind Kind =
> + Legal->getReductionVars()[Phi].getRecurrenceKind();
> + const SmallVector<Instruction *, 4> &ReductionOperations = Reduction.second;
> +
> + RecipeBuilder.recordRecipeOf(Phi);
> + for (auto &R : ReductionOperations) {
> + RecipeBuilder.recordRecipeOf(R);
> + // For min/max reducitons, where we have a pair of icmp/select, we also
> + // need to record the ICmp recipe, so it can be removed later.
> + if (Kind == RecurrenceDescriptor::RK_IntegerMinMax ||
> + Kind == RecurrenceDescriptor::RK_FloatMinMax) {
> + RecipeBuilder.recordRecipeOf(cast<Instruction>(R->getOperand(0)));
> + }
> + }
> + }
>
> // For each interleave group which is relevant for this (possibly trimmed)
> // Range, add it to the set of groups to be later applied to the VPlan and add
> @@ -7491,12 +7575,18 @@ VPlanPtr LoopVectorizationPlanner::buildVPlanWithVPRecipes(
> }
> }
>
> + // Adjust the recipes for any inloop reductions.
> + if (Range.Start > 1)
> + adjustRecipesForInLoopReductions(Plan, RecipeBuilder);
> +
> // Finally, if tail is folded by masking, introduce selects between the phi
> // and the live-out instruction of each reduction, at the end of the latch.
> if (CM.foldTailByMasking() && !Legal->getReductionVars().empty()) {
> Builder.setInsertPoint(VPBB);
> auto *Cond = RecipeBuilder.createBlockInMask(OrigLoop->getHeader(), Plan);
> for (auto &Reduction : Legal->getReductionVars()) {
> + assert(!CM.isInLoopReduction(Reduction.first) &&
> + "Didn't expect inloop tail folded reduction yet!");
> VPValue *Phi = Plan->getVPValue(Reduction.first);
> VPValue *Red = Plan->getVPValue(Reduction.second.getLoopExitInstr());
> Builder.createNaryOp(Instruction::Select, {Cond, Red, Phi});
> @@ -7552,6 +7642,60 @@ VPlanPtr LoopVectorizationPlanner::buildVPlan(VFRange &Range) {
> return Plan;
> }
>
> +// Adjust the recipes for any inloop reductions. The chain of instructions
> +// leading from the loop exit instr to the phi need to be converted to
> +// reductions, with one operand being vector and the other being the scalar
> +// reduction chain.
> +void LoopVectorizationPlanner::adjustRecipesForInLoopReductions(
> + VPlanPtr &Plan, VPRecipeBuilder &RecipeBuilder) {
> + for (auto &Reduction : CM.getInLoopReductionChains()) {
> + PHINode *Phi = Reduction.first;
> + RecurrenceDescriptor &RdxDesc = Legal->getReductionVars()[Phi];
> + const SmallVector<Instruction *, 4> &ReductionOperations = Reduction.second;
> +
> + // ReductionOperations are orders top-down from the phi's use to the
> + // LoopExitValue. We keep a track of the previous item (the Chain) to tell
> + // which of the two operands will remain scalar and which will be reduced.
> + // For minmax the chain will be the select instructions.
> + Instruction *Chain = Phi;
> + for (Instruction *R : ReductionOperations) {
> + VPRecipeBase *WidenRecipe = RecipeBuilder.getRecipe(R);
> + RecurrenceDescriptor::RecurrenceKind Kind = RdxDesc.getRecurrenceKind();
> +
> + VPValue *ChainOp = Plan->getVPValue(Chain);
> + unsigned FirstOpId;
> + if (Kind == RecurrenceDescriptor::RK_IntegerMinMax ||
> + Kind == RecurrenceDescriptor::RK_FloatMinMax) {
> + assert(WidenRecipe->getVPRecipeID() == VPRecipeBase::VPWidenSelectSC &&
> + "Expected to replace a VPWidenSelectSC");
> + FirstOpId = 1;
> + } else {
> + assert(WidenRecipe->getVPRecipeID() == VPRecipeBase::VPWidenSC &&
> + "Expected to replace a VPWidenSC");
> + FirstOpId = 0;
> + }
> + unsigned VecOpId =
> + R->getOperand(FirstOpId) == Chain ? FirstOpId + 1 : FirstOpId;
> + VPValue *VecOp = Plan->getVPValue(R->getOperand(VecOpId));
> +
> + VPReductionRecipe *RedRecipe = new VPReductionRecipe(
> + &RdxDesc, R, ChainOp, VecOp, Legal->hasFunNoNaNAttr(), TTI);
> + WidenRecipe->getParent()->insert(RedRecipe, WidenRecipe->getIterator());
> + WidenRecipe->removeFromParent();
> +
> + if (Kind == RecurrenceDescriptor::RK_IntegerMinMax ||
> + Kind == RecurrenceDescriptor::RK_FloatMinMax) {
> + VPRecipeBase *CompareRecipe =
> + RecipeBuilder.getRecipe(cast<Instruction>(R->getOperand(0)));
> + assert(CompareRecipe->getVPRecipeID() == VPRecipeBase::VPWidenSC &&
> + "Expected to replace a VPWidenSC");
> + CompareRecipe->removeFromParent();
> + }
> + Chain = R;
> + }
> + }
> +}
> +
> Value* LoopVectorizationPlanner::VPCallbackILV::
> getOrCreateVectorValues(Value *V, unsigned Part) {
> return ILV.getOrCreateVectorValue(V, Part);
> @@ -7648,6 +7792,28 @@ void VPInterleaveRecipe::execute(VPTransformState &State) {
> State.ILV->vectorizeInterleaveGroup(IG, State, getAddr(), getMask());
> }
>
> +void VPReductionRecipe::execute(VPTransformState &State) {
> + assert(!State.Instance && "Reduction being replicated.");
> + for (unsigned Part = 0; Part < State.UF; ++Part) {
> + unsigned Kind = RdxDesc->getRecurrenceKind();
> + Value *NewVecOp = State.get(VecOp, Part);
> + Value *NewRed =
> + createTargetReduction(State.Builder, TTI, *RdxDesc, NewVecOp, NoNaN);
> + Value *PrevInChain = State.get(ChainOp, Part);
> + Value *NextInChain;
> + if (Kind == RecurrenceDescriptor::RK_IntegerMinMax ||
> + Kind == RecurrenceDescriptor::RK_FloatMinMax) {
> + NextInChain =
> + createMinMaxOp(State.Builder, RdxDesc->getMinMaxRecurrenceKind(),
> + NewRed, PrevInChain);
> + } else {
> + NextInChain = State.Builder.CreateBinOp(
> + (Instruction::BinaryOps)I->getOpcode(), NewRed, PrevInChain);
> + }
> + State.ValueMap.setVectorValue(I, Part, NextInChain);
> + }
> +}
> +
> void VPReplicateRecipe::execute(VPTransformState &State) {
> if (State.Instance) { // Generate a single instance.
> State.ILV->scalarizeInstruction(Ingredient, User, *State.Instance,
>
> diff --git a/llvm/lib/Transforms/Vectorize/VPlan.cpp b/llvm/lib/Transforms/Vectorize/VPlan.cpp
> index f5f28a3bffa1..302a4845e9a8 100644
> --- a/llvm/lib/Transforms/Vectorize/VPlan.cpp
> +++ b/llvm/lib/Transforms/Vectorize/VPlan.cpp
> @@ -800,6 +800,15 @@ void VPBlendRecipe::print(raw_ostream &O, const Twine &Indent,
> }
> }
>
> +void VPReductionRecipe::print(raw_ostream &O, const Twine &Indent,
> + VPSlotTracker &SlotTracker) const {
> + O << "\"REDUCE of" << *I << " as ";
> + ChainOp->printAsOperand(O, SlotTracker);
> + O << " + reduce(";
> + VecOp->printAsOperand(O, SlotTracker);
> + O << ")";
> +}
> +
> void VPReplicateRecipe::print(raw_ostream &O, const Twine &Indent,
> VPSlotTracker &SlotTracker) const {
> O << "\"" << (IsUniform ? "CLONE " : "REPLICATE ")
>
> diff --git a/llvm/lib/Transforms/Vectorize/VPlan.h b/llvm/lib/Transforms/Vectorize/VPlan.h
> index f07c94e7a3c7..54700cb48839 100644
> --- a/llvm/lib/Transforms/Vectorize/VPlan.h
> +++ b/llvm/lib/Transforms/Vectorize/VPlan.h
> @@ -54,6 +54,7 @@ class InnerLoopVectorizer;
> template <class T> class InterleaveGroup;
> class LoopInfo;
> class raw_ostream;
> +class RecurrenceDescriptor;
> class Value;
> class VPBasicBlock;
> class VPRegionBlock;
> @@ -618,6 +619,7 @@ class VPRecipeBase : public ilist_node_with_parent<VPRecipeBase, VPBasicBlock> {
> VPInstructionSC,
> VPInterleaveSC,
> VPPredInstPHISC,
> + VPReductionSC,
> VPReplicateSC,
> VPWidenCallSC,
> VPWidenCanonicalIVSC,
> @@ -1032,6 +1034,43 @@ class VPInterleaveRecipe : public VPRecipeBase {
> const InterleaveGroup<Instruction> *getInterleaveGroup() { return IG; }
> };
>
> +/// A recipe to represent inloop reduction operations, performing a reduction on
> +/// a vector operand into a scalar value, and adding the result to a chain.
> +class VPReductionRecipe : public VPRecipeBase {
> + /// The recurrence decriptor for the reduction in question.
> + RecurrenceDescriptor *RdxDesc;
> + /// The original instruction being converted to a reduction.
> + Instruction *I;
> + /// The VPValue of the vector value to be reduced.
> + VPValue *VecOp;
> + /// The VPValue of the scalar Chain being accumulated.
> + VPValue *ChainOp;
> + /// Fast math flags to use for the resulting reduction operation.
> + bool NoNaN;
> + /// Pointer to the TTI, needed to create the target reduction
> + const TargetTransformInfo *TTI;
> +
> +public:
> + VPReductionRecipe(RecurrenceDescriptor *R, Instruction *I, VPValue *ChainOp,
> + VPValue *VecOp, bool NoNaN, const TargetTransformInfo *TTI)
> + : VPRecipeBase(VPReductionSC), RdxDesc(R), I(I), VecOp(VecOp),
> + ChainOp(ChainOp), NoNaN(NoNaN), TTI(TTI) {}
> +
> + ~VPReductionRecipe() override = default;
> +
> + /// Method to support type inquiry through isa, cast, and dyn_cast.
> + static inline bool classof(const VPRecipeBase *V) {
> + return V->getVPRecipeID() == VPRecipeBase::VPReductionSC;
> + }
> +
> + /// Generate the reduction in the loop
> + void execute(VPTransformState &State) override;
> +
> + /// Print the recipe.
> + void print(raw_ostream &O, const Twine &Indent,
> + VPSlotTracker &SlotTracker) const override;
> +};
> +
> /// VPReplicateRecipe replicates a given instruction producing multiple scalar
> /// copies of the original scalar type, one per lane, instead of producing a
> /// single copy of widened type for all lanes. If the instruction is known to be
>
> diff --git a/llvm/test/Transforms/LoopVectorize/reduction-inloop-uf4.ll b/llvm/test/Transforms/LoopVectorize/reduction-inloop-uf4.ll
> index 013e08de44f0..a4f9ac13664a 100644
> --- a/llvm/test/Transforms/LoopVectorize/reduction-inloop-uf4.ll
> +++ b/llvm/test/Transforms/LoopVectorize/reduction-inloop-uf4.ll
> @@ -1,5 +1,5 @@
> ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
> -; RUN: opt < %s -loop-vectorize -force-vector-interleave=4 -force-vector-width=4 -force-reduction-intrinsics -dce -instcombine -S | FileCheck %s
> +; RUN: opt < %s -loop-vectorize -force-vector-interleave=4 -force-vector-width=4 -prefer-inloop-reductions -force-reduction-intrinsics -dce -instcombine -S | FileCheck %s
>
> target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
>
> @@ -11,10 +11,10 @@ define i32 @reduction_sum_single(i32* noalias nocapture %A) {
> ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
> ; CHECK: vector.body:
> ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP8:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI1:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP9:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI2:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP10:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI3:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP11:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[TMP9:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI1:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[TMP11:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI2:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[TMP13:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI3:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[TMP15:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP1]], align 4
> @@ -27,25 +27,28 @@ define i32 @reduction_sum_single(i32* noalias nocapture %A) {
> ; CHECK-NEXT: [[TMP6:%.*]] = getelementptr inbounds i32, i32* [[TMP0]], i64 12
> ; CHECK-NEXT: [[TMP7:%.*]] = bitcast i32* [[TMP6]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD6:%.*]] = load <4 x i32>, <4 x i32>* [[TMP7]], align 4
> -; CHECK-NEXT: [[TMP8]] = add <4 x i32> [[VEC_PHI]], [[WIDE_LOAD]]
> -; CHECK-NEXT: [[TMP9]] = add <4 x i32> [[VEC_PHI1]], [[WIDE_LOAD4]]
> -; CHECK-NEXT: [[TMP10]] = add <4 x i32> [[VEC_PHI2]], [[WIDE_LOAD5]]
> -; CHECK-NEXT: [[TMP11]] = add <4 x i32> [[VEC_PHI3]], [[WIDE_LOAD6]]
> +; CHECK-NEXT: [[TMP8:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[WIDE_LOAD]])
> +; CHECK-NEXT: [[TMP9]] = add i32 [[TMP8]], [[VEC_PHI]]
> +; CHECK-NEXT: [[TMP10:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[WIDE_LOAD4]])
> +; CHECK-NEXT: [[TMP11]] = add i32 [[TMP10]], [[VEC_PHI1]]
> +; CHECK-NEXT: [[TMP12:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[WIDE_LOAD5]])
> +; CHECK-NEXT: [[TMP13]] = add i32 [[TMP12]], [[VEC_PHI2]]
> +; CHECK-NEXT: [[TMP14:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[WIDE_LOAD6]])
> +; CHECK-NEXT: [[TMP15]] = add i32 [[TMP14]], [[VEC_PHI3]]
> ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 16
> -; CHECK-NEXT: [[TMP12:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> -; CHECK-NEXT: br i1 [[TMP12]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !0
> +; CHECK-NEXT: [[TMP16:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> +; CHECK-NEXT: br i1 [[TMP16]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !0
> ; CHECK: middle.block:
> -; CHECK-NEXT: [[BIN_RDX:%.*]] = add <4 x i32> [[TMP9]], [[TMP8]]
> -; CHECK-NEXT: [[BIN_RDX7:%.*]] = add <4 x i32> [[TMP10]], [[BIN_RDX]]
> -; CHECK-NEXT: [[BIN_RDX8:%.*]] = add <4 x i32> [[TMP11]], [[BIN_RDX7]]
> -; CHECK-NEXT: [[TMP13:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[BIN_RDX8]])
> +; CHECK-NEXT: [[BIN_RDX:%.*]] = add i32 [[TMP11]], [[TMP9]]
> +; CHECK-NEXT: [[BIN_RDX7:%.*]] = add i32 [[TMP13]], [[BIN_RDX]]
> +; CHECK-NEXT: [[BIN_RDX8:%.*]] = add i32 [[TMP15]], [[BIN_RDX7]]
> ; CHECK-NEXT: br i1 true, label [[DOT_CRIT_EDGE:%.*]], label [[SCALAR_PH]]
> ; CHECK: scalar.ph:
> ; CHECK-NEXT: br label [[DOTLR_PH:%.*]]
> ; CHECK: .lr.ph:
> ; CHECK-NEXT: br i1 undef, label [[DOT_CRIT_EDGE]], label [[DOTLR_PH]], !llvm.loop !2
> ; CHECK: ._crit_edge:
> -; CHECK-NEXT: [[SUM_0_LCSSA:%.*]] = phi i32 [ undef, [[DOTLR_PH]] ], [ [[TMP13]], [[MIDDLE_BLOCK]] ]
> +; CHECK-NEXT: [[SUM_0_LCSSA:%.*]] = phi i32 [ undef, [[DOTLR_PH]] ], [ [[BIN_RDX8]], [[MIDDLE_BLOCK]] ]
> ; CHECK-NEXT: ret i32 [[SUM_0_LCSSA]]
> ;
> entry:
>
> diff --git a/llvm/test/Transforms/LoopVectorize/reduction-inloop.ll b/llvm/test/Transforms/LoopVectorize/reduction-inloop.ll
> index a6747fac8055..3994d64f91d3 100644
> --- a/llvm/test/Transforms/LoopVectorize/reduction-inloop.ll
> +++ b/llvm/test/Transforms/LoopVectorize/reduction-inloop.ll
> @@ -1,5 +1,5 @@
> ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
> -; RUN: opt < %s -loop-vectorize -force-vector-interleave=1 -force-vector-width=4 -force-reduction-intrinsics -dce -instcombine -S | FileCheck %s
> +; RUN: opt < %s -loop-vectorize -force-vector-interleave=1 -force-vector-width=4 -prefer-inloop-reductions -force-reduction-intrinsics -dce -instcombine -S | FileCheck %s
>
> target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
>
> @@ -11,23 +11,23 @@ define i32 @reduction_sum_single(i32* noalias nocapture %A) {
> ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
> ; CHECK: vector.body:
> ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP2:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[TMP3:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP1]], align 4
> -; CHECK-NEXT: [[TMP2]] = add <4 x i32> [[VEC_PHI]], [[WIDE_LOAD]]
> +; CHECK-NEXT: [[TMP2:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[WIDE_LOAD]])
> +; CHECK-NEXT: [[TMP3]] = add i32 [[TMP2]], [[VEC_PHI]]
> ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 4
> -; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> -; CHECK-NEXT: br i1 [[TMP3]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !0
> +; CHECK-NEXT: [[TMP4:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> +; CHECK-NEXT: br i1 [[TMP4]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !0
> ; CHECK: middle.block:
> -; CHECK-NEXT: [[TMP4:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[TMP2]])
> ; CHECK-NEXT: br i1 true, label [[DOT_CRIT_EDGE:%.*]], label [[SCALAR_PH]]
> ; CHECK: scalar.ph:
> ; CHECK-NEXT: br label [[DOTLR_PH:%.*]]
> ; CHECK: .lr.ph:
> ; CHECK-NEXT: br i1 undef, label [[DOT_CRIT_EDGE]], label [[DOTLR_PH]], !llvm.loop !2
> ; CHECK: ._crit_edge:
> -; CHECK-NEXT: [[SUM_0_LCSSA:%.*]] = phi i32 [ undef, [[DOTLR_PH]] ], [ [[TMP4]], [[MIDDLE_BLOCK]] ]
> +; CHECK-NEXT: [[SUM_0_LCSSA:%.*]] = phi i32 [ undef, [[DOTLR_PH]] ], [ [[TMP3]], [[MIDDLE_BLOCK]] ]
> ; CHECK-NEXT: ret i32 [[SUM_0_LCSSA]]
> ;
> entry:
> @@ -57,7 +57,7 @@ define i32 @reduction_sum(i32* noalias nocapture %A, i32* noalias nocapture %B)
> ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
> ; CHECK: vector.body:
> ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP6:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[TMP9:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[VEC_IND2:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT3:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
> @@ -65,22 +65,24 @@ define i32 @reduction_sum(i32* noalias nocapture %A, i32* noalias nocapture %B)
> ; CHECK-NEXT: [[TMP2:%.*]] = getelementptr inbounds i32, i32* [[B:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP3:%.*]] = bitcast i32* [[TMP2]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4
> -; CHECK-NEXT: [[TMP4:%.*]] = add <4 x i32> [[VEC_PHI]], [[VEC_IND2]]
> -; CHECK-NEXT: [[TMP5:%.*]] = add <4 x i32> [[TMP4]], [[WIDE_LOAD]]
> -; CHECK-NEXT: [[TMP6]] = add <4 x i32> [[TMP5]], [[WIDE_LOAD1]]
> +; CHECK-NEXT: [[TMP4:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[VEC_IND2]])
> +; CHECK-NEXT: [[TMP5:%.*]] = add i32 [[TMP4]], [[VEC_PHI]]
> +; CHECK-NEXT: [[TMP6:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[WIDE_LOAD]])
> +; CHECK-NEXT: [[TMP7:%.*]] = add i32 [[TMP6]], [[TMP5]]
> +; CHECK-NEXT: [[TMP8:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[WIDE_LOAD1]])
> +; CHECK-NEXT: [[TMP9]] = add i32 [[TMP8]], [[TMP7]]
> ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 4
> ; CHECK-NEXT: [[VEC_IND_NEXT3]] = add <4 x i32> [[VEC_IND2]], <i32 4, i32 4, i32 4, i32 4>
> -; CHECK-NEXT: [[TMP7:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> -; CHECK-NEXT: br i1 [[TMP7]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !4
> +; CHECK-NEXT: [[TMP10:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> +; CHECK-NEXT: br i1 [[TMP10]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !4
> ; CHECK: middle.block:
> -; CHECK-NEXT: [[TMP8:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[TMP6]])
> ; CHECK-NEXT: br i1 true, label [[DOT_CRIT_EDGE:%.*]], label [[SCALAR_PH]]
> ; CHECK: scalar.ph:
> ; CHECK-NEXT: br label [[DOTLR_PH:%.*]]
> ; CHECK: .lr.ph:
> ; CHECK-NEXT: br i1 undef, label [[DOT_CRIT_EDGE]], label [[DOTLR_PH]], !llvm.loop !5
> ; CHECK: ._crit_edge:
> -; CHECK-NEXT: [[SUM_0_LCSSA:%.*]] = phi i32 [ undef, [[DOTLR_PH]] ], [ [[TMP8]], [[MIDDLE_BLOCK]] ]
> +; CHECK-NEXT: [[SUM_0_LCSSA:%.*]] = phi i32 [ undef, [[DOTLR_PH]] ], [ [[TMP9]], [[MIDDLE_BLOCK]] ]
> ; CHECK-NEXT: ret i32 [[SUM_0_LCSSA]]
> ;
> entry:
> @@ -115,24 +117,24 @@ define i32 @reduction_sum_const(i32* noalias nocapture %A) {
> ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
> ; CHECK: vector.body:
> ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP3:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[TMP4:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP1]], align 4
> -; CHECK-NEXT: [[TMP2:%.*]] = add <4 x i32> [[VEC_PHI]], [[WIDE_LOAD]]
> -; CHECK-NEXT: [[TMP3]] = add <4 x i32> [[TMP2]], <i32 3, i32 3, i32 3, i32 3>
> +; CHECK-NEXT: [[TMP2:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[WIDE_LOAD]])
> +; CHECK-NEXT: [[TMP3:%.*]] = add i32 [[TMP2]], [[VEC_PHI]]
> +; CHECK-NEXT: [[TMP4]] = add i32 [[TMP3]], 12
> ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 4
> -; CHECK-NEXT: [[TMP4:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> -; CHECK-NEXT: br i1 [[TMP4]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !6
> +; CHECK-NEXT: [[TMP5:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> +; CHECK-NEXT: br i1 [[TMP5]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !6
> ; CHECK: middle.block:
> -; CHECK-NEXT: [[TMP5:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[TMP3]])
> ; CHECK-NEXT: br i1 true, label [[DOT_CRIT_EDGE:%.*]], label [[SCALAR_PH]]
> ; CHECK: scalar.ph:
> ; CHECK-NEXT: br label [[DOTLR_PH:%.*]]
> ; CHECK: .lr.ph:
> ; CHECK-NEXT: br i1 undef, label [[DOT_CRIT_EDGE]], label [[DOTLR_PH]], !llvm.loop !7
> ; CHECK: ._crit_edge:
> -; CHECK-NEXT: [[SUM_0_LCSSA:%.*]] = phi i32 [ undef, [[DOTLR_PH]] ], [ [[TMP5]], [[MIDDLE_BLOCK]] ]
> +; CHECK-NEXT: [[SUM_0_LCSSA:%.*]] = phi i32 [ undef, [[DOTLR_PH]] ], [ [[TMP4]], [[MIDDLE_BLOCK]] ]
> ; CHECK-NEXT: ret i32 [[SUM_0_LCSSA]]
> ;
> entry:
> @@ -163,7 +165,7 @@ define i32 @reduction_prod(i32* noalias nocapture %A, i32* noalias nocapture %B)
> ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
> ; CHECK: vector.body:
> ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ <i32 1, i32 1, i32 1, i32 1>, [[VECTOR_PH]] ], [ [[TMP6:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI:%.*]] = phi i32 [ 1, [[VECTOR_PH]] ], [ [[TMP9:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[VEC_IND2:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT3:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
> @@ -171,22 +173,24 @@ define i32 @reduction_prod(i32* noalias nocapture %A, i32* noalias nocapture %B)
> ; CHECK-NEXT: [[TMP2:%.*]] = getelementptr inbounds i32, i32* [[B:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP3:%.*]] = bitcast i32* [[TMP2]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4
> -; CHECK-NEXT: [[TMP4:%.*]] = mul <4 x i32> [[VEC_PHI]], [[VEC_IND2]]
> -; CHECK-NEXT: [[TMP5:%.*]] = mul <4 x i32> [[TMP4]], [[WIDE_LOAD]]
> -; CHECK-NEXT: [[TMP6]] = mul <4 x i32> [[TMP5]], [[WIDE_LOAD1]]
> +; CHECK-NEXT: [[TMP4:%.*]] = call i32 @llvm.experimental.vector.reduce.mul.v4i32(<4 x i32> [[VEC_IND2]])
> +; CHECK-NEXT: [[TMP5:%.*]] = mul i32 [[TMP4]], [[VEC_PHI]]
> +; CHECK-NEXT: [[TMP6:%.*]] = call i32 @llvm.experimental.vector.reduce.mul.v4i32(<4 x i32> [[WIDE_LOAD]])
> +; CHECK-NEXT: [[TMP7:%.*]] = mul i32 [[TMP6]], [[TMP5]]
> +; CHECK-NEXT: [[TMP8:%.*]] = call i32 @llvm.experimental.vector.reduce.mul.v4i32(<4 x i32> [[WIDE_LOAD1]])
> +; CHECK-NEXT: [[TMP9]] = mul i32 [[TMP8]], [[TMP7]]
> ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 4
> ; CHECK-NEXT: [[VEC_IND_NEXT3]] = add <4 x i32> [[VEC_IND2]], <i32 4, i32 4, i32 4, i32 4>
> -; CHECK-NEXT: [[TMP7:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> -; CHECK-NEXT: br i1 [[TMP7]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !8
> +; CHECK-NEXT: [[TMP10:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> +; CHECK-NEXT: br i1 [[TMP10]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !8
> ; CHECK: middle.block:
> -; CHECK-NEXT: [[TMP8:%.*]] = call i32 @llvm.experimental.vector.reduce.mul.v4i32(<4 x i32> [[TMP6]])
> ; CHECK-NEXT: br i1 true, label [[DOT_CRIT_EDGE:%.*]], label [[SCALAR_PH]]
> ; CHECK: scalar.ph:
> ; CHECK-NEXT: br label [[DOTLR_PH:%.*]]
> ; CHECK: .lr.ph:
> ; CHECK-NEXT: br i1 undef, label [[DOT_CRIT_EDGE]], label [[DOTLR_PH]], !llvm.loop !9
> ; CHECK: ._crit_edge:
> -; CHECK-NEXT: [[PROD_0_LCSSA:%.*]] = phi i32 [ undef, [[DOTLR_PH]] ], [ [[TMP8]], [[MIDDLE_BLOCK]] ]
> +; CHECK-NEXT: [[PROD_0_LCSSA:%.*]] = phi i32 [ undef, [[DOTLR_PH]] ], [ [[TMP9]], [[MIDDLE_BLOCK]] ]
> ; CHECK-NEXT: ret i32 [[PROD_0_LCSSA]]
> ;
> entry:
> @@ -221,7 +225,7 @@ define i32 @reduction_mix(i32* noalias nocapture %A, i32* noalias nocapture %B)
> ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
> ; CHECK: vector.body:
> ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP6:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[TMP8:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[VEC_IND2:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT3:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
> @@ -230,14 +234,15 @@ define i32 @reduction_mix(i32* noalias nocapture %A, i32* noalias nocapture %B)
> ; CHECK-NEXT: [[TMP3:%.*]] = bitcast i32* [[TMP2]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4
> ; CHECK-NEXT: [[TMP4:%.*]] = mul nsw <4 x i32> [[WIDE_LOAD1]], [[WIDE_LOAD]]
> -; CHECK-NEXT: [[TMP5:%.*]] = add <4 x i32> [[VEC_PHI]], [[VEC_IND2]]
> -; CHECK-NEXT: [[TMP6]] = add <4 x i32> [[TMP5]], [[TMP4]]
> +; CHECK-NEXT: [[TMP5:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[VEC_IND2]])
> +; CHECK-NEXT: [[TMP6:%.*]] = add i32 [[TMP5]], [[VEC_PHI]]
> +; CHECK-NEXT: [[TMP7:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[TMP4]])
> +; CHECK-NEXT: [[TMP8]] = add i32 [[TMP7]], [[TMP6]]
> ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 4
> ; CHECK-NEXT: [[VEC_IND_NEXT3]] = add <4 x i32> [[VEC_IND2]], <i32 4, i32 4, i32 4, i32 4>
> -; CHECK-NEXT: [[TMP7:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> -; CHECK-NEXT: br i1 [[TMP7]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !10
> +; CHECK-NEXT: [[TMP9:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> +; CHECK-NEXT: br i1 [[TMP9]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !10
> ; CHECK: middle.block:
> -; CHECK-NEXT: [[TMP8:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[TMP6]])
> ; CHECK-NEXT: br i1 true, label [[DOT_CRIT_EDGE:%.*]], label [[SCALAR_PH]]
> ; CHECK: scalar.ph:
> ; CHECK-NEXT: br label [[DOTLR_PH:%.*]]
> @@ -279,20 +284,21 @@ define i32 @reduction_mul(i32* noalias nocapture %A, i32* noalias nocapture %B)
> ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
> ; CHECK: vector.body:
> ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ <i32 19, i32 1, i32 1, i32 1>, [[VECTOR_PH]] ], [ [[TMP5:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI:%.*]] = phi i32 [ 19, [[VECTOR_PH]] ], [ [[TMP7:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP1]], align 4
> ; CHECK-NEXT: [[TMP2:%.*]] = getelementptr inbounds i32, i32* [[B:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP3:%.*]] = bitcast i32* [[TMP2]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4
> -; CHECK-NEXT: [[TMP4:%.*]] = mul <4 x i32> [[VEC_PHI]], [[WIDE_LOAD]]
> -; CHECK-NEXT: [[TMP5]] = mul <4 x i32> [[TMP4]], [[WIDE_LOAD1]]
> +; CHECK-NEXT: [[TMP4:%.*]] = call i32 @llvm.experimental.vector.reduce.mul.v4i32(<4 x i32> [[WIDE_LOAD]])
> +; CHECK-NEXT: [[TMP5:%.*]] = mul i32 [[TMP4]], [[VEC_PHI]]
> +; CHECK-NEXT: [[TMP6:%.*]] = call i32 @llvm.experimental.vector.reduce.mul.v4i32(<4 x i32> [[WIDE_LOAD1]])
> +; CHECK-NEXT: [[TMP7]] = mul i32 [[TMP6]], [[TMP5]]
> ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 4
> -; CHECK-NEXT: [[TMP6:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> -; CHECK-NEXT: br i1 [[TMP6]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !12
> +; CHECK-NEXT: [[TMP8:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> +; CHECK-NEXT: br i1 [[TMP8]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !12
> ; CHECK: middle.block:
> -; CHECK-NEXT: [[TMP7:%.*]] = call i32 @llvm.experimental.vector.reduce.mul.v4i32(<4 x i32> [[TMP5]])
> ; CHECK-NEXT: br i1 true, label [[DOT_CRIT_EDGE:%.*]], label [[SCALAR_PH]]
> ; CHECK: scalar.ph:
> ; CHECK-NEXT: br label [[DOTLR_PH:%.*]]
> @@ -332,7 +338,7 @@ define i32 @start_at_non_zero(i32* nocapture %in, i32* nocapture %coeff, i32* no
> ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
> ; CHECK: vector.body:
> ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ <i32 120, i32 0, i32 0, i32 0>, [[VECTOR_PH]] ], [ [[TMP5:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI:%.*]] = phi i32 [ 120, [[VECTOR_PH]] ], [ [[TMP6:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[IN:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP1]], align 4
> @@ -340,19 +346,19 @@ define i32 @start_at_non_zero(i32* nocapture %in, i32* nocapture %coeff, i32* no
> ; CHECK-NEXT: [[TMP3:%.*]] = bitcast i32* [[TMP2]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4
> ; CHECK-NEXT: [[TMP4:%.*]] = mul nsw <4 x i32> [[WIDE_LOAD1]], [[WIDE_LOAD]]
> -; CHECK-NEXT: [[TMP5]] = add <4 x i32> [[TMP4]], [[VEC_PHI]]
> +; CHECK-NEXT: [[TMP5:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[TMP4]])
> +; CHECK-NEXT: [[TMP6]] = add i32 [[TMP5]], [[VEC_PHI]]
> ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 4
> -; CHECK-NEXT: [[TMP6:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> -; CHECK-NEXT: br i1 [[TMP6]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !14
> +; CHECK-NEXT: [[TMP7:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> +; CHECK-NEXT: br i1 [[TMP7]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !14
> ; CHECK: middle.block:
> -; CHECK-NEXT: [[TMP7:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[TMP5]])
> ; CHECK-NEXT: br i1 true, label [[FOR_END:%.*]], label [[SCALAR_PH]]
> ; CHECK: scalar.ph:
> ; CHECK-NEXT: br label [[FOR_BODY:%.*]]
> ; CHECK: for.body:
> ; CHECK-NEXT: br i1 undef, label [[FOR_END]], label [[FOR_BODY]], !llvm.loop !15
> ; CHECK: for.end:
> -; CHECK-NEXT: [[SUM_0_LCSSA:%.*]] = phi i32 [ undef, [[FOR_BODY]] ], [ [[TMP7]], [[MIDDLE_BLOCK]] ]
> +; CHECK-NEXT: [[SUM_0_LCSSA:%.*]] = phi i32 [ undef, [[FOR_BODY]] ], [ [[TMP6]], [[MIDDLE_BLOCK]] ]
> ; CHECK-NEXT: ret i32 [[SUM_0_LCSSA]]
> ;
> entry:
> @@ -385,20 +391,21 @@ define i32 @reduction_and(i32* nocapture %A, i32* nocapture %B) {
> ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
> ; CHECK: vector.body:
> ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ <i32 -1, i32 -1, i32 -1, i32 -1>, [[VECTOR_PH]] ], [ [[TMP5:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI:%.*]] = phi i32 [ -1, [[VECTOR_PH]] ], [ [[TMP7:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP1]], align 4
> ; CHECK-NEXT: [[TMP2:%.*]] = getelementptr inbounds i32, i32* [[B:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP3:%.*]] = bitcast i32* [[TMP2]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4
> -; CHECK-NEXT: [[TMP4:%.*]] = and <4 x i32> [[VEC_PHI]], [[WIDE_LOAD]]
> -; CHECK-NEXT: [[TMP5]] = and <4 x i32> [[TMP4]], [[WIDE_LOAD1]]
> +; CHECK-NEXT: [[TMP4:%.*]] = call i32 @llvm.experimental.vector.reduce.and.v4i32(<4 x i32> [[WIDE_LOAD]])
> +; CHECK-NEXT: [[TMP5:%.*]] = and i32 [[TMP4]], [[VEC_PHI]]
> +; CHECK-NEXT: [[TMP6:%.*]] = call i32 @llvm.experimental.vector.reduce.and.v4i32(<4 x i32> [[WIDE_LOAD1]])
> +; CHECK-NEXT: [[TMP7]] = and i32 [[TMP6]], [[TMP5]]
> ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 4
> -; CHECK-NEXT: [[TMP6:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> -; CHECK-NEXT: br i1 [[TMP6]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !16
> +; CHECK-NEXT: [[TMP8:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> +; CHECK-NEXT: br i1 [[TMP8]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !16
> ; CHECK: middle.block:
> -; CHECK-NEXT: [[TMP7:%.*]] = call i32 @llvm.experimental.vector.reduce.and.v4i32(<4 x i32> [[TMP5]])
> ; CHECK-NEXT: br i1 true, label [[FOR_END:%.*]], label [[SCALAR_PH]]
> ; CHECK: scalar.ph:
> ; CHECK-NEXT: br label [[FOR_BODY:%.*]]
> @@ -438,7 +445,7 @@ define i32 @reduction_or(i32* nocapture %A, i32* nocapture %B) {
> ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
> ; CHECK: vector.body:
> ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP5:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[TMP6:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP1]], align 4
> @@ -446,19 +453,19 @@ define i32 @reduction_or(i32* nocapture %A, i32* nocapture %B) {
> ; CHECK-NEXT: [[TMP3:%.*]] = bitcast i32* [[TMP2]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4
> ; CHECK-NEXT: [[TMP4:%.*]] = add nsw <4 x i32> [[WIDE_LOAD1]], [[WIDE_LOAD]]
> -; CHECK-NEXT: [[TMP5]] = or <4 x i32> [[TMP4]], [[VEC_PHI]]
> +; CHECK-NEXT: [[TMP5:%.*]] = call i32 @llvm.experimental.vector.reduce.or.v4i32(<4 x i32> [[TMP4]])
> +; CHECK-NEXT: [[TMP6]] = or i32 [[TMP5]], [[VEC_PHI]]
> ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 4
> -; CHECK-NEXT: [[TMP6:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> -; CHECK-NEXT: br i1 [[TMP6]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !18
> +; CHECK-NEXT: [[TMP7:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> +; CHECK-NEXT: br i1 [[TMP7]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !18
> ; CHECK: middle.block:
> -; CHECK-NEXT: [[TMP7:%.*]] = call i32 @llvm.experimental.vector.reduce.or.v4i32(<4 x i32> [[TMP5]])
> ; CHECK-NEXT: br i1 true, label [[FOR_END:%.*]], label [[SCALAR_PH]]
> ; CHECK: scalar.ph:
> ; CHECK-NEXT: br label [[FOR_BODY:%.*]]
> ; CHECK: for.body:
> ; CHECK-NEXT: br i1 undef, label [[FOR_END]], label [[FOR_BODY]], !llvm.loop !19
> ; CHECK: for.end:
> -; CHECK-NEXT: [[RESULT_0_LCSSA:%.*]] = phi i32 [ undef, [[FOR_BODY]] ], [ [[TMP7]], [[MIDDLE_BLOCK]] ]
> +; CHECK-NEXT: [[RESULT_0_LCSSA:%.*]] = phi i32 [ undef, [[FOR_BODY]] ], [ [[TMP6]], [[MIDDLE_BLOCK]] ]
> ; CHECK-NEXT: ret i32 [[RESULT_0_LCSSA]]
> ;
> entry:
> @@ -491,7 +498,7 @@ define i32 @reduction_xor(i32* nocapture %A, i32* nocapture %B) {
> ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
> ; CHECK: vector.body:
> ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP5:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[TMP6:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP1]], align 4
> @@ -499,19 +506,19 @@ define i32 @reduction_xor(i32* nocapture %A, i32* nocapture %B) {
> ; CHECK-NEXT: [[TMP3:%.*]] = bitcast i32* [[TMP2]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4
> ; CHECK-NEXT: [[TMP4:%.*]] = add nsw <4 x i32> [[WIDE_LOAD1]], [[WIDE_LOAD]]
> -; CHECK-NEXT: [[TMP5]] = xor <4 x i32> [[TMP4]], [[VEC_PHI]]
> +; CHECK-NEXT: [[TMP5:%.*]] = call i32 @llvm.experimental.vector.reduce.xor.v4i32(<4 x i32> [[TMP4]])
> +; CHECK-NEXT: [[TMP6]] = xor i32 [[TMP5]], [[VEC_PHI]]
> ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 4
> -; CHECK-NEXT: [[TMP6:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> -; CHECK-NEXT: br i1 [[TMP6]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !20
> +; CHECK-NEXT: [[TMP7:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> +; CHECK-NEXT: br i1 [[TMP7]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !20
> ; CHECK: middle.block:
> -; CHECK-NEXT: [[TMP7:%.*]] = call i32 @llvm.experimental.vector.reduce.xor.v4i32(<4 x i32> [[TMP5]])
> ; CHECK-NEXT: br i1 true, label [[FOR_END:%.*]], label [[SCALAR_PH]]
> ; CHECK: scalar.ph:
> ; CHECK-NEXT: br label [[FOR_BODY:%.*]]
> ; CHECK: for.body:
> ; CHECK-NEXT: br i1 undef, label [[FOR_END]], label [[FOR_BODY]], !llvm.loop !21
> ; CHECK: for.end:
> -; CHECK-NEXT: [[RESULT_0_LCSSA:%.*]] = phi i32 [ undef, [[FOR_BODY]] ], [ [[TMP7]], [[MIDDLE_BLOCK]] ]
> +; CHECK-NEXT: [[RESULT_0_LCSSA:%.*]] = phi i32 [ undef, [[FOR_BODY]] ], [ [[TMP6]], [[MIDDLE_BLOCK]] ]
> ; CHECK-NEXT: ret i32 [[RESULT_0_LCSSA]]
> ;
> entry:
> @@ -544,20 +551,21 @@ define float @reduction_fadd(float* nocapture %A, float* nocapture %B) {
> ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
> ; CHECK: vector.body:
> ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x float> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP5:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI:%.*]] = phi float [ 0.000000e+00, [[VECTOR_PH]] ], [ [[TMP7:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds float, float* [[A:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP1:%.*]] = bitcast float* [[TMP0]] to <4 x float>*
> ; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x float>, <4 x float>* [[TMP1]], align 4
> ; CHECK-NEXT: [[TMP2:%.*]] = getelementptr inbounds float, float* [[B:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP3:%.*]] = bitcast float* [[TMP2]] to <4 x float>*
> ; CHECK-NEXT: [[WIDE_LOAD1:%.*]] = load <4 x float>, <4 x float>* [[TMP3]], align 4
> -; CHECK-NEXT: [[TMP4:%.*]] = fadd fast <4 x float> [[VEC_PHI]], [[WIDE_LOAD]]
> -; CHECK-NEXT: [[TMP5]] = fadd fast <4 x float> [[TMP4]], [[WIDE_LOAD1]]
> +; CHECK-NEXT: [[TMP4:%.*]] = call fast float @llvm.experimental.vector.reduce.v2.fadd.f32.v4f32(float 0.000000e+00, <4 x float> [[WIDE_LOAD]])
> +; CHECK-NEXT: [[TMP5:%.*]] = fadd float [[TMP4]], [[VEC_PHI]]
> +; CHECK-NEXT: [[TMP6:%.*]] = call fast float @llvm.experimental.vector.reduce.v2.fadd.f32.v4f32(float 0.000000e+00, <4 x float> [[WIDE_LOAD1]])
> +; CHECK-NEXT: [[TMP7]] = fadd float [[TMP6]], [[TMP5]]
> ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 4
> -; CHECK-NEXT: [[TMP6:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> -; CHECK-NEXT: br i1 [[TMP6]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !22
> +; CHECK-NEXT: [[TMP8:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> +; CHECK-NEXT: br i1 [[TMP8]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !22
> ; CHECK: middle.block:
> -; CHECK-NEXT: [[TMP7:%.*]] = call fast float @llvm.experimental.vector.reduce.v2.fadd.f32.v4f32(float 0.000000e+00, <4 x float> [[TMP5]])
> ; CHECK-NEXT: br i1 true, label [[FOR_END:%.*]], label [[SCALAR_PH]]
> ; CHECK: scalar.ph:
> ; CHECK-NEXT: br label [[FOR_BODY:%.*]]
> @@ -597,20 +605,21 @@ define float @reduction_fmul(float* nocapture %A, float* nocapture %B) {
> ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
> ; CHECK: vector.body:
> ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x float> [ <float 0.000000e+00, float 1.000000e+00, float 1.000000e+00, float 1.000000e+00>, [[VECTOR_PH]] ], [ [[TMP5:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI:%.*]] = phi float [ 0.000000e+00, [[VECTOR_PH]] ], [ [[TMP7:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds float, float* [[A:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP1:%.*]] = bitcast float* [[TMP0]] to <4 x float>*
> ; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x float>, <4 x float>* [[TMP1]], align 4
> ; CHECK-NEXT: [[TMP2:%.*]] = getelementptr inbounds float, float* [[B:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP3:%.*]] = bitcast float* [[TMP2]] to <4 x float>*
> ; CHECK-NEXT: [[WIDE_LOAD1:%.*]] = load <4 x float>, <4 x float>* [[TMP3]], align 4
> -; CHECK-NEXT: [[TMP4:%.*]] = fmul fast <4 x float> [[VEC_PHI]], [[WIDE_LOAD]]
> -; CHECK-NEXT: [[TMP5]] = fmul fast <4 x float> [[TMP4]], [[WIDE_LOAD1]]
> +; CHECK-NEXT: [[TMP4:%.*]] = call fast float @llvm.experimental.vector.reduce.v2.fmul.f32.v4f32(float 1.000000e+00, <4 x float> [[WIDE_LOAD]])
> +; CHECK-NEXT: [[TMP5:%.*]] = fmul float [[TMP4]], [[VEC_PHI]]
> +; CHECK-NEXT: [[TMP6:%.*]] = call fast float @llvm.experimental.vector.reduce.v2.fmul.f32.v4f32(float 1.000000e+00, <4 x float> [[WIDE_LOAD1]])
> +; CHECK-NEXT: [[TMP7]] = fmul float [[TMP6]], [[TMP5]]
> ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 4
> -; CHECK-NEXT: [[TMP6:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> -; CHECK-NEXT: br i1 [[TMP6]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !24
> +; CHECK-NEXT: [[TMP8:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> +; CHECK-NEXT: br i1 [[TMP8]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !24
> ; CHECK: middle.block:
> -; CHECK-NEXT: [[TMP7:%.*]] = call fast float @llvm.experimental.vector.reduce.v2.fmul.f32.v4f32(float 1.000000e+00, <4 x float> [[TMP5]])
> ; CHECK-NEXT: br i1 true, label [[FOR_END:%.*]], label [[SCALAR_PH]]
> ; CHECK: scalar.ph:
> ; CHECK-NEXT: br label [[FOR_BODY:%.*]]
> @@ -650,24 +659,24 @@ define i32 @reduction_min(i32* nocapture %A, i32* nocapture %B) {
> ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
> ; CHECK: vector.body:
> ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ <i32 1000, i32 1000, i32 1000, i32 1000>, [[VECTOR_PH]] ], [ [[TMP3:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI:%.*]] = phi i32 [ 1000, [[VECTOR_PH]] ], [ [[RDX_MINMAX_SELECT:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP1]], align 4
> -; CHECK-NEXT: [[TMP2:%.*]] = icmp slt <4 x i32> [[VEC_PHI]], [[WIDE_LOAD]]
> -; CHECK-NEXT: [[TMP3]] = select <4 x i1> [[TMP2]], <4 x i32> [[VEC_PHI]], <4 x i32> [[WIDE_LOAD]]
> +; CHECK-NEXT: [[TMP2:%.*]] = call i32 @llvm.experimental.vector.reduce.smin.v4i32(<4 x i32> [[WIDE_LOAD]])
> +; CHECK-NEXT: [[RDX_MINMAX_CMP:%.*]] = icmp slt i32 [[TMP2]], [[VEC_PHI]]
> +; CHECK-NEXT: [[RDX_MINMAX_SELECT]] = select i1 [[RDX_MINMAX_CMP]], i32 [[TMP2]], i32 [[VEC_PHI]]
> ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 4
> -; CHECK-NEXT: [[TMP4:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> -; CHECK-NEXT: br i1 [[TMP4]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !26
> +; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> +; CHECK-NEXT: br i1 [[TMP3]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !26
> ; CHECK: middle.block:
> -; CHECK-NEXT: [[TMP5:%.*]] = call i32 @llvm.experimental.vector.reduce.smin.v4i32(<4 x i32> [[TMP3]])
> ; CHECK-NEXT: br i1 true, label [[FOR_END:%.*]], label [[SCALAR_PH]]
> ; CHECK: scalar.ph:
> ; CHECK-NEXT: br label [[FOR_BODY:%.*]]
> ; CHECK: for.body:
> ; CHECK-NEXT: br i1 undef, label [[FOR_END]], label [[FOR_BODY]], !llvm.loop !27
> ; CHECK: for.end:
> -; CHECK-NEXT: [[RESULT_0_LCSSA:%.*]] = phi i32 [ undef, [[FOR_BODY]] ], [ [[TMP5]], [[MIDDLE_BLOCK]] ]
> +; CHECK-NEXT: [[RESULT_0_LCSSA:%.*]] = phi i32 [ undef, [[FOR_BODY]] ], [ [[RDX_MINMAX_SELECT]], [[MIDDLE_BLOCK]] ]
> ; CHECK-NEXT: ret i32 [[RESULT_0_LCSSA]]
> ;
> entry:
> @@ -698,24 +707,24 @@ define i32 @reduction_max(i32* nocapture %A, i32* nocapture %B) {
> ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
> ; CHECK: vector.body:
> ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ <i32 1000, i32 1000, i32 1000, i32 1000>, [[VECTOR_PH]] ], [ [[TMP3:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI:%.*]] = phi i32 [ 1000, [[VECTOR_PH]] ], [ [[RDX_MINMAX_SELECT:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP1]], align 4
> -; CHECK-NEXT: [[TMP2:%.*]] = icmp ugt <4 x i32> [[VEC_PHI]], [[WIDE_LOAD]]
> -; CHECK-NEXT: [[TMP3]] = select <4 x i1> [[TMP2]], <4 x i32> [[VEC_PHI]], <4 x i32> [[WIDE_LOAD]]
> +; CHECK-NEXT: [[TMP2:%.*]] = call i32 @llvm.experimental.vector.reduce.umax.v4i32(<4 x i32> [[WIDE_LOAD]])
> +; CHECK-NEXT: [[RDX_MINMAX_CMP:%.*]] = icmp ugt i32 [[TMP2]], [[VEC_PHI]]
> +; CHECK-NEXT: [[RDX_MINMAX_SELECT]] = select i1 [[RDX_MINMAX_CMP]], i32 [[TMP2]], i32 [[VEC_PHI]]
> ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 4
> -; CHECK-NEXT: [[TMP4:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> -; CHECK-NEXT: br i1 [[TMP4]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !28
> +; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> +; CHECK-NEXT: br i1 [[TMP3]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !28
> ; CHECK: middle.block:
> -; CHECK-NEXT: [[TMP5:%.*]] = call i32 @llvm.experimental.vector.reduce.umax.v4i32(<4 x i32> [[TMP3]])
> ; CHECK-NEXT: br i1 true, label [[FOR_END:%.*]], label [[SCALAR_PH]]
> ; CHECK: scalar.ph:
> ; CHECK-NEXT: br label [[FOR_BODY:%.*]]
> ; CHECK: for.body:
> ; CHECK-NEXT: br i1 undef, label [[FOR_END]], label [[FOR_BODY]], !llvm.loop !29
> ; CHECK: for.end:
> -; CHECK-NEXT: [[RESULT_0_LCSSA:%.*]] = phi i32 [ undef, [[FOR_BODY]] ], [ [[TMP5]], [[MIDDLE_BLOCK]] ]
> +; CHECK-NEXT: [[RESULT_0_LCSSA:%.*]] = phi i32 [ undef, [[FOR_BODY]] ], [ [[RDX_MINMAX_SELECT]], [[MIDDLE_BLOCK]] ]
> ; CHECK-NEXT: ret i32 [[RESULT_0_LCSSA]]
> ;
> entry:
> @@ -933,7 +942,7 @@ define i32 @reduction_predicated(i32* noalias nocapture %A, i32* noalias nocaptu
> ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
> ; CHECK: vector.body:
> ; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP6:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[VEC_PHI:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[TMP9:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[VEC_IND2:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT3:%.*]], [[VECTOR_BODY]] ]
> ; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
> @@ -941,22 +950,24 @@ define i32 @reduction_predicated(i32* noalias nocapture %A, i32* noalias nocaptu
> ; CHECK-NEXT: [[TMP2:%.*]] = getelementptr inbounds i32, i32* [[B:%.*]], i64 [[INDEX]]
> ; CHECK-NEXT: [[TMP3:%.*]] = bitcast i32* [[TMP2]] to <4 x i32>*
> ; CHECK-NEXT: [[WIDE_LOAD1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4
> -; CHECK-NEXT: [[TMP4:%.*]] = add <4 x i32> [[VEC_PHI]], [[VEC_IND2]]
> -; CHECK-NEXT: [[TMP5:%.*]] = add <4 x i32> [[TMP4]], [[WIDE_LOAD]]
> -; CHECK-NEXT: [[TMP6]] = add <4 x i32> [[TMP5]], [[WIDE_LOAD1]]
> +; CHECK-NEXT: [[TMP4:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[VEC_IND2]])
> +; CHECK-NEXT: [[TMP5:%.*]] = add i32 [[TMP4]], [[VEC_PHI]]
> +; CHECK-NEXT: [[TMP6:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[WIDE_LOAD]])
> +; CHECK-NEXT: [[TMP7:%.*]] = add i32 [[TMP6]], [[TMP5]]
> +; CHECK-NEXT: [[TMP8:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[WIDE_LOAD1]])
> +; CHECK-NEXT: [[TMP9]] = add i32 [[TMP8]], [[TMP7]]
> ; CHECK-NEXT: [[INDEX_NEXT]] = add i64 [[INDEX]], 4
> ; CHECK-NEXT: [[VEC_IND_NEXT3]] = add <4 x i32> [[VEC_IND2]], <i32 4, i32 4, i32 4, i32 4>
> -; CHECK-NEXT: [[TMP7:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> -; CHECK-NEXT: br i1 [[TMP7]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !34
> +; CHECK-NEXT: [[TMP10:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
> +; CHECK-NEXT: br i1 [[TMP10]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !34
> ; CHECK: middle.block:
> -; CHECK-NEXT: [[TMP8:%.*]] = call i32 @llvm.experimental.vector.reduce.add.v4i32(<4 x i32> [[TMP6]])
> ; CHECK-NEXT: br i1 true, label [[DOT_CRIT_EDGE:%.*]], label [[SCALAR_PH]]
> ; CHECK: scalar.ph:
> ; CHECK-NEXT: br label [[DOTLR_PH:%.*]]
> ; CHECK: .lr.ph:
> ; CHECK-NEXT: br i1 undef, label [[DOT_CRIT_EDGE]], label [[DOTLR_PH]], !llvm.loop !35
> ; CHECK: ._crit_edge:
> -; CHECK-NEXT: [[SUM_0_LCSSA:%.*]] = phi i32 [ undef, [[DOTLR_PH]] ], [ [[TMP8]], [[MIDDLE_BLOCK]] ]
> +; CHECK-NEXT: [[SUM_0_LCSSA:%.*]] = phi i32 [ undef, [[DOTLR_PH]] ], [ [[TMP9]], [[MIDDLE_BLOCK]] ]
> ; CHECK-NEXT: ret i32 [[SUM_0_LCSSA]]
> ;
> entry:
> @@ -1041,18 +1052,19 @@ define i8 @reduction_and_trunc(i8* noalias nocapture %A) {
> ; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
> ; CHECK: vector.body:
> ; CHECK-NEXT: [[INDEX:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[VEC_PHI:%.*]] = phi <4 x i32> [ <i32 255, i32 -1, i32 -1, i32 -1>, [[VECTOR_PH]] ], [ [[TMP4:%.*]], [[VECTOR_BODY]] ]
> -; CHECK-NEXT: [[TMP0:%.*]] = sext i32 [[INDEX]] to i64
> -; CHECK-NEXT: [[TMP1:%.*]] = getelementptr inbounds i8, i8* [[A:%.*]], i64 [[TMP0]]
> -; CHECK-NEXT: [[TMP2:%.*]] = bitcast i8* [[TMP1]] to <4 x i8>*
> -; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i8>, <4 x i8>* [[TMP2]], align 4
> -; CHECK-NEXT: [[TMP3:%.*]] = zext <4 x i8> [[WIDE_LOAD]] to <4 x i32>
> -; CHECK-NEXT: [[TMP4]] = and <4 x i32> [[VEC_PHI]], [[TMP3]]
> +; CHECK-NEXT: [[VEC_PHI:%.*]] = phi i32 [ 255, [[VECTOR_PH]] ], [ [[TMP6:%.*]], [[VECTOR_BODY]] ]
> +; CHECK-NEXT: [[TMP0:%.*]] = and i32 [[VEC_PHI]], 255
> +; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[INDEX]] to i64
> +; CHECK-NEXT: [[TMP2:%.*]] = getelementptr inbounds i8, i8* [[A:%.*]], i64 [[TMP1]]
> +; CHECK-NEXT: [[TMP3:%.*]] = bitcast i8* [[TMP2]] to <4 x i8>*
> +; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i8>, <4 x i8>* [[TMP3]], align 4
> +; CHECK-NEXT: [[TMP4:%.*]] = zext <4 x i8> [[WIDE_LOAD]] to <4 x i32>
> +; CHECK-NEXT: [[TMP5:%.*]] = call i32 @llvm.experimental.vector.reduce.and.v4i32(<4 x i32> [[TMP4]])
> +; CHECK-NEXT: [[TMP6]] = and i32 [[TMP5]], [[TMP0]]
> ; CHECK-NEXT: [[INDEX_NEXT]] = add i32 [[INDEX]], 4
> -; CHECK-NEXT: [[TMP5:%.*]] = icmp eq i32 [[INDEX_NEXT]], 256
> -; CHECK-NEXT: br i1 [[TMP5]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !38
> +; CHECK-NEXT: [[TMP7:%.*]] = icmp eq i32 [[INDEX_NEXT]], 256
> +; CHECK-NEXT: br i1 [[TMP7]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop !38
> ; CHECK: middle.block:
> -; CHECK-NEXT: [[TMP6:%.*]] = call i32 @llvm.experimental.vector.reduce.and.v4i32(<4 x i32> [[TMP4]])
> ; CHECK-NEXT: br i1 true, label [[DOT_CRIT_EDGE:%.*]], label [[SCALAR_PH]]
> ; CHECK: scalar.ph:
> ; CHECK-NEXT: br label [[DOTLR_PH:%.*]]
>
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
More information about the llvm-commits
mailing list