<div dir="ltr">I think this caused a miscompile: <a href="http://llvm.org/PR34635">http://llvm.org/PR34635</a><div><br></div><div>I've reverted it for now in r313409.</div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Sep 8, 2017 at 3:52 PM Alexey Bataev via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div><span></span></div>
<div>
<div>Galina, thanks. Will fix it ASAP<br>
<br>
Best regards,
<div>Alexey Bataev</div>
</div>
<div><br>
8 сент. 2017 г., в 17:56, Galina Kistanova <<a href="mailto:gkistanova@gmail.com" target="_blank">gkistanova@gmail.com</a>> написал(а):<br>
<br>
</div></div></div><div dir="auto"><div>
<blockquote type="cite">
<div>
<div dir="ltr">Hello Alexey,<br>
<br>
It looks like this commit added warnings to one of our builders:<br>
<a href="http://lab.llvm.org:8011/builders/ubuntu-gcc7.1-werror/builds/1263" target="_blank">http://lab.llvm.org:8011/builders/ubuntu-gcc7.1-werror/builds/1263</a><br>
<br>
...<br>
FAILED: /usr/local/gcc-7.1/bin/g++-7.1 -DGTEST_HAS_RTTI=0 -D_DEBUG -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ilib/Transforms/Vectorize -I/home/buildslave/am1i-slv2/ubuntu-gcc7.1-werror/llvm/lib/Transforms/Vectorize
-Iinclude -I/home/buildslave/am1i-slv2/ubuntu-gcc7.1-werror/llvm/include -Wno-noexcept-type -fPIC -fvisibility-inlines-hidden -Werror -Werror=date-time -std=c++11 -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic
-Wno-long-long -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment -ffunction-sections -fdata-sections -O3 -fPIC -UNDEBUG -fno-exceptions -fno-rtti -MD -MT lib/Transforms/Vectorize/CMakeFiles/LLVMVectorize.dir/SLPVectorizer.cpp.o -MF lib/Transforms/Vectorize/CMakeFiles/LLVMVectorize.dir/SLPVectorizer.cpp.o.d
-o lib/Transforms/Vectorize/CMakeFiles/LLVMVectorize.dir/SLPVectorizer.cpp.o -c /home/buildslave/am1i-slv2/ubuntu-gcc7.1-werror/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp<br>
/home/buildslave/am1i-slv2/ubuntu-gcc7.1-werror/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp: In member function ‘unsigned int {anonymous}::HorizontalReduction::OperationData::getRequiredNumberOfUses() const’:<br>
/home/buildslave/am1i-slv2/ubuntu-gcc7.1-werror/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:4733:5: error: control reaches end of non-void function [-Werror=return-type]<br>
}<br>
^<br>
/home/buildslave/am1i-slv2/ubuntu-gcc7.1-werror/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp: In member function ‘unsigned int {anonymous}::HorizontalReduction::OperationData::getNumberOfOperands() const’:<br>
/home/buildslave/am1i-slv2/ubuntu-gcc7.1-werror/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:4716:5: error: control reaches end of non-void function [-Werror=return-type]<br>
}<br>
^<br>
/home/buildslave/am1i-slv2/ubuntu-gcc7.1-werror/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp: In member function ‘int {anonymous}::HorizontalReduction::getReductionCost(llvm::TargetTransformInfo*, llvm::Value*, unsigned int)’:<br>
/home/buildslave/am1i-slv2/ubuntu-gcc7.1-werror/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:5183:18: error: this statement may fall through [-Werror=implicit-fallthrough=]<br>
IsUnsigned = false;<br>
~~~~~~~~~~~^~~~~~~<br>
/home/buildslave/am1i-slv2/ubuntu-gcc7.1-werror/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:5184:5: note: here<br>
case RK_UMin:<br>
^~~~<br>
cc1plus: all warnings being treated as errors<br>
<br>
<br>
Please have a look?<br>
<br>
Thanks<br>
<br>
Galina<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Sep 8, 2017 at 6:49 AM, Alexey Bataev via llvm-commits
<span dir="ltr"><<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Author: abataev<br>
Date: Fri Sep 8 06:49:36 2017<br>
New Revision: 312791<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=312791&view=rev" rel="noreferrer" target="_blank">
http://llvm.org/viewvc/llvm-project?rev=312791&view=rev</a><br>
Log:<br>
[SLP] Support for horizontal min/max reduction.<br>
<br>
SLP vectorizer supports horizontal reductions for Add/FAdd binary<br>
operations. Patch adds support for horizontal min/max reductions.<br>
Function getReductionCost() is split to getArithmeticReductionCost() for<br>
binary operation reductions and getMinMaxReductionCost() for min/max<br>
reductions.<br>
Patch fixes PR26956.<br>
<br>
Differential revision: <a href="https://reviews.llvm.org/D27846" rel="noreferrer" target="_blank">
https://reviews.llvm.org/D27846</a><br>
<br>
Modified:<br>
llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h<br>
llvm/trunk/include/llvm/Analysis/TargetTransformInfoImpl.h<br>
llvm/trunk/include/llvm/CodeGen/BasicTTIImpl.h<br>
llvm/trunk/lib/Analysis/CostModel.cpp<br>
llvm/trunk/lib/Analysis/TargetTransformInfo.cpp<br>
llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp<br>
llvm/trunk/lib/Target/X86/X86TargetTransformInfo.h<br>
llvm/trunk/lib/Transforms/Vectorize/SLPVectorizer.cpp<br>
llvm/trunk/test/Transforms/SLPVectorizer/X86/horizontal-list.ll<br>
llvm/trunk/test/Transforms/SLPVectorizer/X86/horizontal-minmax.ll<br>
<br>
Modified: llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h?rev=312791&r1=312790&r2=312791&view=diff" rel="noreferrer" target="_blank">
http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h?rev=312791&r1=312790&r2=312791&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h (original)<br>
+++ llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h Fri Sep 8 06:49:36 2017<br>
@@ -732,6 +732,8 @@ public:<br>
/// ((v0+v2), (v1+v3), undef, undef)<br>
int getArithmeticReductionCost(unsigned Opcode, Type *Ty,<br>
bool IsPairwiseForm) const;<br>
+ int getMinMaxReductionCost(Type *Ty, Type *CondTy, bool IsPairwiseForm,<br>
+ bool IsUnsigned) const;<br>
<br>
/// \returns The cost of Intrinsic instructions. Analyses the real arguments.<br>
/// Three cases are handled: 1. scalar instruction 2. vector instruction<br>
@@ -998,6 +1000,8 @@ public:<br>
unsigned AddressSpace) = 0;<br>
virtual int getArithmeticReductionCost(unsigned Opcode, Type *Ty,<br>
bool IsPairwiseForm) = 0;<br>
+ virtual int getMinMaxReductionCost(Type *Ty, Type *CondTy,<br>
+ bool IsPairwiseForm, bool IsUnsigned) = 0;<br>
virtual int getIntrinsicInstrCost(Intrinsic::ID ID, Type *RetTy,<br>
ArrayRef<Type *> Tys, FastMathFlags FMF,<br>
unsigned ScalarizationCostPassed) = 0;<br>
@@ -1309,6 +1313,10 @@ public:<br>
bool IsPairwiseForm) override {<br>
return Impl.getArithmeticReductionCost(Opcode, Ty, IsPairwiseForm);<br>
}<br>
+ int getMinMaxReductionCost(Type *Ty, Type *CondTy,<br>
+ bool IsPairwiseForm, bool IsUnsigned) override {<br>
+ return Impl.getMinMaxReductionCost(Ty, CondTy, IsPairwiseForm, IsUnsigned);<br>
+ }<br>
int getIntrinsicInstrCost(Intrinsic::ID ID, Type *RetTy, ArrayRef<Type *> Tys,<br>
FastMathFlags FMF, unsigned ScalarizationCostPassed) override {<br>
return Impl.getIntrinsicInstrCost(ID, RetTy, Tys, FMF,<br>
<br>
Modified: llvm/trunk/include/llvm/Analysis/TargetTransformInfoImpl.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Analysis/TargetTransformInfoImpl.h?rev=312791&r1=312790&r2=312791&view=diff" rel="noreferrer" target="_blank">
http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Analysis/TargetTransformInfoImpl.h?rev=312791&r1=312790&r2=312791&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/include/llvm/Analysis/TargetTransformInfoImpl.h (original)<br>
+++ llvm/trunk/include/llvm/Analysis/TargetTransformInfoImpl.h Fri Sep 8 06:49:36 2017<br>
@@ -451,6 +451,8 @@ public:<br>
<br>
unsigned getArithmeticReductionCost(unsigned, Type *, bool) { return 1; }<br>
<br>
+ unsigned getMinMaxReductionCost(Type *, Type *, bool, bool) { return 1; }<br>
+<br>
unsigned getCostOfKeepingLiveOverCall(ArrayRef<Type *> Tys) { return 0; }<br>
<br>
bool getTgtMemIntrinsic(IntrinsicInst *Inst, MemIntrinsicInfo &Info) {<br>
<br>
Modified: llvm/trunk/include/llvm/CodeGen/BasicTTIImpl.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/BasicTTIImpl.h?rev=312791&r1=312790&r2=312791&view=diff" rel="noreferrer" target="_blank">
http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/BasicTTIImpl.h?rev=312791&r1=312790&r2=312791&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/include/llvm/CodeGen/BasicTTIImpl.h (original)<br>
+++ llvm/trunk/include/llvm/CodeGen/BasicTTIImpl.h Fri Sep 8 06:49:36 2017<br>
@@ -1166,6 +1166,66 @@ public:<br>
return ShuffleCost + ArithCost + getScalarizationOverhead(Ty, false, true);<br>
}<br>
<br>
+ /// Try to calculate op costs for min/max reduction operations.<br>
+ /// \param CondTy Conditional type for the Select instruction.<br>
+ unsigned getMinMaxReductionCost(Type *Ty, Type *CondTy, bool IsPairwise,<br>
+ bool) {<br>
+ assert(Ty->isVectorTy() && "Expect a vector type");<br>
+ Type *ScalarTy = Ty->getVectorElementType();<br>
+ Type *ScalarCondTy = CondTy->getVectorElementType();<br>
+ unsigned NumVecElts = Ty->getVectorNumElements();<br>
+ unsigned NumReduxLevels = Log2_32(NumVecElts);<br>
+ unsigned CmpOpcode;<br>
+ if (Ty->isFPOrFPVectorTy()) {<br>
+ CmpOpcode = Instruction::FCmp;<br>
+ } else {<br>
+ assert(Ty->isIntOrIntVectorTy() &&<br>
+ "expecting floating point or integer type for min/max reduction");<br>
+ CmpOpcode = Instruction::ICmp;<br>
+ }<br>
+ unsigned MinMaxCost = 0;<br>
+ unsigned ShuffleCost = 0;<br>
+ auto *ConcreteTTI = static_cast<T *>(this);<br>
+ std::pair<unsigned, MVT> LT =<br>
+ ConcreteTTI->getTLI()->getTypeLegalizationCost(DL, Ty);<br>
+ unsigned LongVectorCount = 0;<br>
+ unsigned MVTLen =<br>
+ LT.second.isVector() ? LT.second.getVectorNumElements() : 1;<br>
+ while (NumVecElts > MVTLen) {<br>
+ NumVecElts /= 2;<br>
+ // Assume the pairwise shuffles add a cost.<br>
+ ShuffleCost += (IsPairwise + 1) *<br>
+ ConcreteTTI->getShuffleCost(TTI::SK_ExtractSubvector, Ty,<br>
+ NumVecElts, Ty);<br>
+ MinMaxCost +=<br>
+ ConcreteTTI->getCmpSelInstrCost(CmpOpcode, Ty, CondTy, nullptr) +<br>
+ ConcreteTTI->getCmpSelInstrCost(Instruction::Select, Ty, CondTy,<br>
+ nullptr);<br>
+ Ty = VectorType::get(ScalarTy, NumVecElts);<br>
+ CondTy = VectorType::get(ScalarCondTy, NumVecElts);<br>
+ ++LongVectorCount;<br>
+ }<br>
+ // The minimal length of the vector is limited by the real length of vector<br>
+ // operations performed on the current platform. That's why several final<br>
+ // reduction opertions are perfomed on the vectors with the same<br>
+ // architecture-dependent length.<br>
+ ShuffleCost += (NumReduxLevels - LongVectorCount) * (IsPairwise + 1) *<br>
+ ConcreteTTI->getShuffleCost(TTI::SK_ExtractSubvector, Ty,<br>
+ NumVecElts, Ty);<br>
+ MinMaxCost +=<br>
+ (NumReduxLevels - LongVectorCount) *<br>
+ (ConcreteTTI->getCmpSelInstrCost(CmpOpcode, Ty, CondTy, nullptr) +<br>
+ ConcreteTTI->getCmpSelInstrCost(Instruction::Select, Ty, CondTy,<br>
+ nullptr));<br>
+ // Need 3 extractelement instructions for scalarization + an additional<br>
+ // scalar select instruction.<br>
+ return ShuffleCost + MinMaxCost +<br>
+ 3 * getScalarizationOverhead(Ty, /*Insert=*/false,<br>
+ /*Extract=*/true) +<br>
+ ConcreteTTI->getCmpSelInstrCost(Instruction::Select, ScalarTy,<br>
+ ScalarCondTy, nullptr);<br>
+ }<br>
+<br>
unsigned getVectorSplitCost() { return 1; }<br>
<br>
/// @}<br>
<br>
Modified: llvm/trunk/lib/Analysis/CostModel.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Analysis/CostModel.cpp?rev=312791&r1=312790&r2=312791&view=diff" rel="noreferrer" target="_blank">
http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Analysis/CostModel.cpp?rev=312791&r1=312790&r2=312791&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Analysis/CostModel.cpp (original)<br>
+++ llvm/trunk/lib/Analysis/CostModel.cpp Fri Sep 8 06:49:36 2017<br>
@@ -186,26 +186,56 @@ static bool matchPairwiseShuffleMask(Shu<br>
}<br>
<br>
namespace {<br>
+/// Kind of the reduction data.<br>
+enum ReductionKind {<br>
+ RK_None, /// Not a reduction.<br>
+ RK_Arithmetic, /// Binary reduction data.<br>
+ RK_MinMax, /// Min/max reduction data.<br>
+ RK_UnsignedMinMax, /// Unsigned min/max reduction data.<br>
+};<br>
/// Contains opcode + LHS/RHS parts of the reduction operations.<br>
struct ReductionData {<br>
- explicit ReductionData() = default;<br>
- ReductionData(unsigned Opcode, Value *LHS, Value *RHS)<br>
- : Opcode(Opcode), LHS(LHS), RHS(RHS) {}<br>
+ ReductionData() = delete;<br>
+ ReductionData(ReductionKind Kind, unsigned Opcode, Value *LHS, Value *RHS)<br>
+ : Opcode(Opcode), LHS(LHS), RHS(RHS), Kind(Kind) {<br>
+ assert(Kind != RK_None && "expected binary or min/max reduction only.");<br>
+ }<br>
unsigned Opcode = 0;<br>
Value *LHS = nullptr;<br>
Value *RHS = nullptr;<br>
+ ReductionKind Kind = RK_None;<br>
+ bool hasSameData(ReductionData &RD) const {<br>
+ return Kind == RD.Kind && Opcode == RD.Opcode;<br>
+ }<br>
};<br>
} // namespace<br>
<br>
static Optional<ReductionData> getReductionData(Instruction *I) {<br>
Value *L, *R;<br>
if (m_BinOp(m_Value(L), m_Value(R)).match(I))<br>
- return ReductionData(I->getOpcode(), L, R);<br>
+ return ReductionData(RK_Arithmetic, I->getOpcode(), L, R);<br>
+ if (auto *SI = dyn_cast<SelectInst>(I)) {<br>
+ if (m_SMin(m_Value(L), m_Value(R)).match(SI) ||<br>
+ m_SMax(m_Value(L), m_Value(R)).match(SI) ||<br>
+ m_OrdFMin(m_Value(L), m_Value(R)).match(SI) ||<br>
+ m_OrdFMax(m_Value(L), m_Value(R)).match(SI) ||<br>
+ m_UnordFMin(m_Value(L), m_Value(R)).match(SI) ||<br>
+ m_UnordFMax(m_Value(L), m_Value(R)).match(SI)) {<br>
+ auto *CI = cast<CmpInst>(SI->getCondition());<br>
+ return ReductionData(RK_MinMax, CI->getOpcode(), L, R);<br>
+ }<br>
+ if (m_UMin(m_Value(L), m_Value(R)).match(SI) ||<br>
+ m_UMax(m_Value(L), m_Value(R)).match(SI)) {<br>
+ auto *CI = cast<CmpInst>(SI->getCondition());<br>
+ return ReductionData(RK_UnsignedMinMax, CI->getOpcode(), L, R);<br>
+ }<br>
+ }<br>
return llvm::None;<br>
}<br>
<br>
-static bool matchPairwiseReductionAtLevel(Instruction *I, unsigned Level,<br>
- unsigned NumLevels) {<br>
+static ReductionKind matchPairwiseReductionAtLevel(Instruction *I,<br>
+ unsigned Level,<br>
+ unsigned NumLevels) {<br>
// Match one level of pairwise operations.<br>
// %rdx.shuf.0.0 = shufflevector <4 x float> %rdx, <4 x float> undef,<br>
// <4 x i32> <i32 0, i32 2 , i32 undef, i32 undef><br>
@@ -213,24 +243,24 @@ static bool matchPairwiseReductionAtLeve<br>
// <4 x i32> <i32 1, i32 3, i32 undef, i32 undef><br>
// %bin.rdx.0 = fadd <4 x float> %rdx.shuf.0.0, %rdx.shuf.0.1<br>
if (!I)<br>
- return false;<br>
+ return RK_None;<br>
<br>
assert(I->getType()->isVectorTy() && "Expecting a vector type");<br>
<br>
Optional<ReductionData> RD = getReductionData(I);<br>
if (!RD)<br>
- return false;<br>
+ return RK_None;<br>
<br>
ShuffleVectorInst *LS = dyn_cast<ShuffleVectorInst>(RD->LHS);<br>
if (!LS && Level)<br>
- return false;<br>
+ return RK_None;<br>
ShuffleVectorInst *RS = dyn_cast<ShuffleVectorInst>(RD->RHS);<br>
if (!RS && Level)<br>
- return false;<br>
+ return RK_None;<br>
<br>
// On level 0 we can omit one shufflevector instruction.<br>
if (!Level && !RS && !LS)<br>
- return false;<br>
+ return RK_None;<br>
<br>
// Shuffle inputs must match.<br>
Value *NextLevelOpL = LS ? LS->getOperand(0) : nullptr;<br>
@@ -239,7 +269,7 @@ static bool matchPairwiseReductionAtLeve<br>
if (NextLevelOpR && NextLevelOpL) {<br>
// If we have two shuffles their operands must match.<br>
if (NextLevelOpL != NextLevelOpR)<br>
- return false;<br>
+ return RK_None;<br>
<br>
NextLevelOp = NextLevelOpL;<br>
} else if (Level == 0 && (NextLevelOpR || NextLevelOpL)) {<br>
@@ -250,45 +280,47 @@ static bool matchPairwiseReductionAtLeve<br>
// %NextLevelOpL = shufflevector %R, <1, undef ...><br>
// %BinOp = fadd %NextLevelOpL, %R<br>
if (NextLevelOpL && NextLevelOpL != RD->RHS)<br>
- return false;<br>
+ return RK_None;<br>
else if (NextLevelOpR && NextLevelOpR != RD->LHS)<br>
- return false;<br>
+ return RK_None;<br>
<br>
NextLevelOp = NextLevelOpL ? RD->RHS : RD->LHS;<br>
- } else<br>
- return false;<br>
+ } else {<br>
+ return RK_None;<br>
+ }<br>
<br>
// Check that the next levels binary operation exists and matches with the<br>
// current one.<br>
if (Level + 1 != NumLevels) {<br>
Optional<ReductionData> NextLevelRD =<br>
getReductionData(cast<Instruction>(NextLevelOp));<br>
- if (!NextLevelRD || RD->Opcode != NextLevelRD->Opcode)<br>
- return false;<br>
+ if (!NextLevelRD || !RD->hasSameData(*NextLevelRD))<br>
+ return RK_None;<br>
}<br>
<br>
// Shuffle mask for pairwise operation must match.<br>
if (matchPairwiseShuffleMask(LS, /*IsLeft=*/true, Level)) {<br>
if (!matchPairwiseShuffleMask(RS, /*IsLeft=*/false, Level))<br>
- return false;<br>
+ return RK_None;<br>
} else if (matchPairwiseShuffleMask(RS, /*IsLeft=*/true, Level)) {<br>
if (!matchPairwiseShuffleMask(LS, /*IsLeft=*/false, Level))<br>
- return false;<br>
- } else<br>
- return false;<br>
+ return RK_None;<br>
+ } else {<br>
+ return RK_None;<br>
+ }<br>
<br>
if (++Level == NumLevels)<br>
- return true;<br>
+ return RD->Kind;<br>
<br>
// Match next level.<br>
return matchPairwiseReductionAtLevel(cast<Instruction>(NextLevelOp), Level,<br>
NumLevels);<br>
}<br>
<br>
-static bool matchPairwiseReduction(const ExtractElementInst *ReduxRoot,<br>
- unsigned &Opcode, Type *&Ty) {<br>
+static ReductionKind matchPairwiseReduction(const ExtractElementInst *ReduxRoot,<br>
+ unsigned &Opcode, Type *&Ty) {<br>
if (!EnableReduxCost)<br>
- return false;<br>
+ return RK_None;<br>
<br>
// Need to extract the first element.<br>
ConstantInt *CI = dyn_cast<ConstantInt>(ReduxRoot->getOperand(1));<br>
@@ -296,19 +328,19 @@ static bool matchPairwiseReduction(const<br>
if (CI)<br>
Idx = CI->getZExtValue();<br>
if (Idx != 0)<br>
- return false;<br>
+ return RK_None;<br>
<br>
auto *RdxStart = dyn_cast<Instruction>(ReduxRoot->getOperand(0));<br>
if (!RdxStart)<br>
- return false;<br>
+ return RK_None;<br>
Optional<ReductionData> RD = getReductionData(RdxStart);<br>
if (!RD)<br>
- return false;<br>
+ return RK_None;<br>
<br>
Type *VecTy = RdxStart->getType();<br>
unsigned NumVecElems = VecTy->getVectorNumElements();<br>
if (!isPowerOf2_32(NumVecElems))<br>
- return false;<br>
+ return RK_None;<br>
<br>
// We look for a sequence of shuffle,shuffle,add triples like the following<br>
// that builds a pairwise reduction tree.<br>
@@ -328,13 +360,14 @@ static bool matchPairwiseReduction(const<br>
// <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef><br>
// %bin.rdx8 = fadd <4 x float> %rdx.shuf.1.0, %rdx.shuf.1.1<br>
// %r = extractelement <4 x float> %bin.rdx8, i32 0<br>
- if (!matchPairwiseReductionAtLevel(RdxStart, 0, Log2_32(NumVecElems)))<br>
- return false;<br>
+ if (matchPairwiseReductionAtLevel(RdxStart, 0, Log2_32(NumVecElems)) ==<br>
+ RK_None)<br>
+ return RK_None;<br>
<br>
Opcode = RD->Opcode;<br>
Ty = VecTy;<br>
<br>
- return true;<br>
+ return RD->Kind;<br>
}<br>
<br>
static std::pair<Value *, ShuffleVectorInst *><br>
@@ -348,10 +381,11 @@ getShuffleAndOtherOprd(Value *L, Value *<br>
return std::make_pair(L, S);<br>
}<br>
<br>
-static bool matchVectorSplittingReduction(const ExtractElementInst *ReduxRoot,<br>
- unsigned &Opcode, Type *&Ty) {<br>
+static ReductionKind<br>
+matchVectorSplittingReduction(const ExtractElementInst *ReduxRoot,<br>
+ unsigned &Opcode, Type *&Ty) {<br>
if (!EnableReduxCost)<br>
- return false;<br>
+ return RK_None;<br>
<br>
// Need to extract the first element.<br>
ConstantInt *CI = dyn_cast<ConstantInt>(ReduxRoot->getOperand(1));<br>
@@ -359,19 +393,19 @@ static bool matchVectorSplittingReductio<br>
if (CI)<br>
Idx = CI->getZExtValue();<br>
if (Idx != 0)<br>
- return false;<br>
+ return RK_None;<br>
<br>
auto *RdxStart = dyn_cast<Instruction>(ReduxRoot->getOperand(0));<br>
if (!RdxStart)<br>
- return false;<br>
+ return RK_None;<br>
Optional<ReductionData> RD = getReductionData(RdxStart);<br>
if (!RD)<br>
- return false;<br>
+ return RK_None;<br>
<br>
Type *VecTy = ReduxRoot->getOperand(0)->getType();<br>
unsigned NumVecElems = VecTy->getVectorNumElements();<br>
if (!isPowerOf2_32(NumVecElems))<br>
- return false;<br>
+ return RK_None;<br>
<br>
// We look for a sequence of shuffles and adds like the following matching one<br>
// fadd, shuffle vector pair at a time.<br>
@@ -391,10 +425,10 @@ static bool matchVectorSplittingReductio<br>
while (NumVecElemsRemain - 1) {<br>
// Check for the right reduction operation.<br>
if (!RdxOp)<br>
- return false;<br>
+ return RK_None;<br>
Optional<ReductionData> RDLevel = getReductionData(RdxOp);<br>
- if (!RDLevel || RDLevel->Opcode != RD->Opcode)<br>
- return false;<br>
+ if (!RDLevel || !RDLevel->hasSameData(*RD))<br>
+ return RK_None;<br>
<br>
Value *NextRdxOp;<br>
ShuffleVectorInst *Shuffle;<br>
@@ -403,9 +437,9 @@ static bool matchVectorSplittingReductio<br>
<br>
// Check the current reduction operation and the shuffle use the same value.<br>
if (Shuffle == nullptr)<br>
- return false;<br>
+ return RK_None;<br>
if (Shuffle->getOperand(0) != NextRdxOp)<br>
- return false;<br>
+ return RK_None;<br>
<br>
// Check that shuffle masks matches.<br>
for (unsigned j = 0; j != MaskStart; ++j)<br>
@@ -415,7 +449,7 @@ static bool matchVectorSplittingReductio<br>
<br>
SmallVector<int, 16> Mask = Shuffle->getShuffleMask();<br>
if (ShuffleMask != Mask)<br>
- return false;<br>
+ return RK_None;<br>
<br>
RdxOp = dyn_cast<Instruction>(NextRdxOp);<br>
NumVecElemsRemain /= 2;<br>
@@ -424,7 +458,7 @@ static bool matchVectorSplittingReductio<br>
<br>
Opcode = RD->Opcode;<br>
Ty = VecTy;<br>
- return true;<br>
+ return RD->Kind;<br>
}<br>
<br>
unsigned CostModelAnalysis::getInstructionCost(const Instruction *I) const {<br>
@@ -519,13 +553,36 @@ unsigned CostModelAnalysis::getInstructi<br>
unsigned ReduxOpCode;<br>
Type *ReduxType;<br>
<br>
- if (matchVectorSplittingReduction(EEI, ReduxOpCode, ReduxType)) {<br>
+ switch (matchVectorSplittingReduction(EEI, ReduxOpCode, ReduxType)) {<br>
+ case RK_Arithmetic:<br>
return TTI->getArithmeticReductionCost(ReduxOpCode, ReduxType,<br>
/*IsPairwiseForm=*/false);<br>
+ case RK_MinMax:<br>
+ return TTI->getMinMaxReductionCost(<br>
+ ReduxType, CmpInst::makeCmpResultType(ReduxType),<br>
+ /*IsPairwiseForm=*/false, /*IsUnsigned=*/false);<br>
+ case RK_UnsignedMinMax:<br>
+ return TTI->getMinMaxReductionCost(<br>
+ ReduxType, CmpInst::makeCmpResultType(ReduxType),<br>
+ /*IsPairwiseForm=*/false, /*IsUnsigned=*/true);<br>
+ case RK_None:<br>
+ break;<br>
}<br>
- if (matchPairwiseReduction(EEI, ReduxOpCode, ReduxType)) {<br>
+<br>
+ switch (matchPairwiseReduction(EEI, ReduxOpCode, ReduxType)) {<br>
+ case RK_Arithmetic:<br>
return TTI->getArithmeticReductionCost(ReduxOpCode, ReduxType,<br>
/*IsPairwiseForm=*/true);<br>
+ case RK_MinMax:<br>
+ return TTI->getMinMaxReductionCost(<br>
+ ReduxType, CmpInst::makeCmpResultType(ReduxType),<br>
+ /*IsPairwiseForm=*/true, /*IsUnsigned=*/false);<br>
+ case RK_UnsignedMinMax:<br>
+ return TTI->getMinMaxReductionCost(<br>
+ ReduxType, CmpInst::makeCmpResultType(ReduxType),<br>
+ /*IsPairwiseForm=*/true, /*IsUnsigned=*/true);<br>
+ case RK_None:<br>
+ break;<br>
}<br>
<br>
return TTI->getVectorInstrCost(I->getOpcode(),<br>
<br>
Modified: llvm/trunk/lib/Analysis/TargetTransformInfo.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Analysis/TargetTransformInfo.cpp?rev=312791&r1=312790&r2=312791&view=diff" rel="noreferrer" target="_blank">
http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Analysis/TargetTransformInfo.cpp?rev=312791&r1=312790&r2=312791&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Analysis/TargetTransformInfo.cpp (original)<br>
+++ llvm/trunk/lib/Analysis/TargetTransformInfo.cpp Fri Sep 8 06:49:36 2017<br>
@@ -484,6 +484,15 @@ int TargetTransformInfo::getArithmeticRe<br>
return Cost;<br>
}<br>
<br>
+int TargetTransformInfo::getMinMaxReductionCost(Type *Ty, Type *CondTy,<br>
+ bool IsPairwiseForm,<br>
+ bool IsUnsigned) const {<br>
+ int Cost =<br>
+ TTIImpl->getMinMaxReductionCost(Ty, CondTy, IsPairwiseForm, IsUnsigned);<br>
+ assert(Cost >= 0 && "TTI should not produce negative costs!");<br>
+ return Cost;<br>
+}<br>
+<br>
unsigned<br>
TargetTransformInfo::getCostOfKeepingLiveOverCall(ArrayRef<Type *> Tys) const {<br>
return TTIImpl->getCostOfKeepingLiveOverCall(Tys);<br>
<br>
Modified: llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp?rev=312791&r1=312790&r2=312791&view=diff" rel="noreferrer" target="_blank">
http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp?rev=312791&r1=312790&r2=312791&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp (original)<br>
+++ llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp Fri Sep 8 06:49:36 2017<br>
@@ -1999,6 +1999,152 @@ int X86TTIImpl::getArithmeticReductionCo<br>
return BaseT::getArithmeticReductionCost(Opcode, ValTy, IsPairwise);<br>
}<br>
<br>
+int X86TTIImpl::getMinMaxReductionCost(Type *ValTy, Type *CondTy,<br>
+ bool IsPairwise, bool IsUnsigned) {<br>
+ std::pair<int, MVT> LT = TLI->getTypeLegalizationCost(DL, ValTy);<br>
+<br>
+ MVT MTy = LT.second;<br>
+<br>
+ int ISD;<br>
+ if (ValTy->isIntOrIntVectorTy()) {<br>
+ ISD = IsUnsigned ? ISD::UMIN : ISD::SMIN;<br>
+ } else {<br>
+ assert(ValTy->isFPOrFPVectorTy() &&<br>
+ "Expected float point or integer vector type.");<br>
+ ISD = ISD::FMINNUM;<br>
+ }<br>
+<br>
+ // We use the Intel Architecture Code Analyzer(IACA) to measure the throughput<br>
+ // and make it as the cost.<br>
+<br>
+ static const CostTblEntry SSE42CostTblPairWise[] = {<br>
+ {ISD::FMINNUM, MVT::v2f64, 3},<br>
+ {ISD::FMINNUM, MVT::v4f32, 2},<br>
+ {ISD::SMIN, MVT::v2i64, 7}, // The data reported by the IACA is "6.8"<br>
+ {ISD::UMIN, MVT::v2i64, 8}, // The data reported by the IACA is "8.6"<br>
+ {ISD::SMIN, MVT::v4i32, 1}, // The data reported by the IACA is "1.5"<br>
+ {ISD::UMIN, MVT::v4i32, 2}, // The data reported by the IACA is "1.8"<br>
+ {ISD::SMIN, MVT::v8i16, 2},<br>
+ {ISD::UMIN, MVT::v8i16, 2},<br>
+ };<br>
+<br>
+ static const CostTblEntry AVX1CostTblPairWise[] = {<br>
+ {ISD::FMINNUM, MVT::v4f32, 1},<br>
+ {ISD::FMINNUM, MVT::v4f64, 1},<br>
+ {ISD::FMINNUM, MVT::v8f32, 2},<br>
+ {ISD::SMIN, MVT::v2i64, 3},<br>
+ {ISD::UMIN, MVT::v2i64, 3},<br>
+ {ISD::SMIN, MVT::v4i32, 1},<br>
+ {ISD::UMIN, MVT::v4i32, 1},<br>
+ {ISD::SMIN, MVT::v8i16, 1},<br>
+ {ISD::UMIN, MVT::v8i16, 1},<br>
+ {ISD::SMIN, MVT::v8i32, 3},<br>
+ {ISD::UMIN, MVT::v8i32, 3},<br>
+ };<br>
+<br>
+ static const CostTblEntry AVX2CostTblPairWise[] = {<br>
+ {ISD::SMIN, MVT::v4i64, 2},<br>
+ {ISD::UMIN, MVT::v4i64, 2},<br>
+ {ISD::SMIN, MVT::v8i32, 1},<br>
+ {ISD::UMIN, MVT::v8i32, 1},<br>
+ {ISD::SMIN, MVT::v16i16, 1},<br>
+ {ISD::UMIN, MVT::v16i16, 1},<br>
+ {ISD::SMIN, MVT::v32i8, 2},<br>
+ {ISD::UMIN, MVT::v32i8, 2},<br>
+ };<br>
+<br>
+ static const CostTblEntry AVX512CostTblPairWise[] = {<br>
+ {ISD::FMINNUM, MVT::v8f64, 1},<br>
+ {ISD::FMINNUM, MVT::v16f32, 2},<br>
+ {ISD::SMIN, MVT::v8i64, 2},<br>
+ {ISD::UMIN, MVT::v8i64, 2},<br>
+ {ISD::SMIN, MVT::v16i32, 1},<br>
+ {ISD::UMIN, MVT::v16i32, 1},<br>
+ };<br>
+<br>
+ static const CostTblEntry SSE42CostTblNoPairWise[] = {<br>
+ {ISD::FMINNUM, MVT::v2f64, 3},<br>
+ {ISD::FMINNUM, MVT::v4f32, 3},<br>
+ {ISD::SMIN, MVT::v2i64, 7}, // The data reported by the IACA is "6.8"<br>
+ {ISD::UMIN, MVT::v2i64, 9}, // The data reported by the IACA is "8.6"<br>
+ {ISD::SMIN, MVT::v4i32, 1}, // The data reported by the IACA is "1.5"<br>
+ {ISD::UMIN, MVT::v4i32, 2}, // The data reported by the IACA is "1.8"<br>
+ {ISD::SMIN, MVT::v8i16, 1}, // The data reported by the IACA is "1.5"<br>
+ {ISD::UMIN, MVT::v8i16, 2}, // The data reported by the IACA is "1.8"<br>
+ };<br>
+<br>
+ static const CostTblEntry AVX1CostTblNoPairWise[] = {<br>
+ {ISD::FMINNUM, MVT::v4f32, 1},<br>
+ {ISD::FMINNUM, MVT::v4f64, 1},<br>
+ {ISD::FMINNUM, MVT::v8f32, 1},<br>
+ {ISD::SMIN, MVT::v2i64, 3},<br>
+ {ISD::UMIN, MVT::v2i64, 3},<br>
+ {ISD::SMIN, MVT::v4i32, 1},<br>
+ {ISD::UMIN, MVT::v4i32, 1},<br>
+ {ISD::SMIN, MVT::v8i16, 1},<br>
+ {ISD::UMIN, MVT::v8i16, 1},<br>
+ {ISD::SMIN, MVT::v8i32, 2},<br>
+ {ISD::UMIN, MVT::v8i32, 2},<br>
+ };<br>
+<br>
+ static const CostTblEntry AVX2CostTblNoPairWise[] = {<br>
+ {ISD::SMIN, MVT::v4i64, 1},<br>
+ {ISD::UMIN, MVT::v4i64, 1},<br>
+ {ISD::SMIN, MVT::v8i32, 1},<br>
+ {ISD::UMIN, MVT::v8i32, 1},<br>
+ {ISD::SMIN, MVT::v16i16, 1},<br>
+ {ISD::UMIN, MVT::v16i16, 1},<br>
+ {ISD::SMIN, MVT::v32i8, 1},<br>
+ {ISD::UMIN, MVT::v32i8, 1},<br>
+ };<br>
+<br>
+ static const CostTblEntry AVX512CostTblNoPairWise[] = {<br>
+ {ISD::FMINNUM, MVT::v8f64, 1},<br>
+ {ISD::FMINNUM, MVT::v16f32, 2},<br>
+ {ISD::SMIN, MVT::v8i64, 1},<br>
+ {ISD::UMIN, MVT::v8i64, 1},<br>
+ {ISD::SMIN, MVT::v16i32, 1},<br>
+ {ISD::UMIN, MVT::v16i32, 1},<br>
+ };<br>
+<br>
+ if (IsPairwise) {<br>
+ if (ST->hasAVX512())<br>
+ if (const auto *Entry = CostTableLookup(AVX512CostTblPairWise, ISD, MTy))<br>
+ return LT.first * Entry->Cost;<br>
+<br>
+ if (ST->hasAVX2())<br>
+ if (const auto *Entry = CostTableLookup(AVX2CostTblPairWise, ISD, MTy))<br>
+ return LT.first * Entry->Cost;<br>
+<br>
+ if (ST->hasAVX())<br>
+ if (const auto *Entry = CostTableLookup(AVX1CostTblPairWise, ISD, MTy))<br>
+ return LT.first * Entry->Cost;<br>
+<br>
+ if (ST->hasSSE42())<br>
+ if (const auto *Entry = CostTableLookup(SSE42CostTblPairWise, ISD, MTy))<br>
+ return LT.first * Entry->Cost;<br>
+ } else {<br>
+ if (ST->hasAVX512())<br>
+ if (const auto *Entry =<br>
+ CostTableLookup(AVX512CostTblNoPairWise, ISD, MTy))<br>
+ return LT.first * Entry->Cost;<br>
+<br>
+ if (ST->hasAVX2())<br>
+ if (const auto *Entry = CostTableLookup(AVX2CostTblNoPairWise, ISD, MTy))<br>
+ return LT.first * Entry->Cost;<br>
+<br>
+ if (ST->hasAVX())<br>
+ if (const auto *Entry = CostTableLookup(AVX1CostTblNoPairWise, ISD, MTy))<br>
+ return LT.first * Entry->Cost;<br>
+<br>
+ if (ST->hasSSE42())<br>
+ if (const auto *Entry = CostTableLookup(SSE42CostTblNoPairWise, ISD, MTy))<br>
+ return LT.first * Entry->Cost;<br>
+ }<br>
+<br>
+ return BaseT::getMinMaxReductionCost(ValTy, CondTy, IsPairwise, IsUnsigned);<br>
+}<br>
+<br>
/// \brief Calculate the cost of materializing a 64-bit value. This helper<br>
/// method might only calculate a fraction of a larger immediate. Therefore it<br>
/// is valid to return a cost of ZERO.<br>
<br>
Modified: llvm/trunk/lib/Target/X86/X86TargetTransformInfo.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86TargetTransformInfo.h?rev=312791&r1=312790&r2=312791&view=diff" rel="noreferrer" target="_blank">
http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86TargetTransformInfo.h?rev=312791&r1=312790&r2=312791&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Target/X86/X86TargetTransformInfo.h (original)<br>
+++ llvm/trunk/lib/Target/X86/X86TargetTransformInfo.h Fri Sep 8 06:49:36 2017<br>
@@ -96,6 +96,9 @@ public:<br>
int getArithmeticReductionCost(unsigned Opcode, Type *Ty,<br>
bool IsPairwiseForm);<br>
<br>
+ int getMinMaxReductionCost(Type *Ty, Type *CondTy, bool IsPairwiseForm,<br>
+ bool IsUnsigned);<br>
+<br>
int getInterleavedMemoryOpCost(unsigned Opcode, Type *VecTy,<br>
unsigned Factor, ArrayRef<unsigned> Indices,<br>
unsigned Alignment, unsigned AddressSpace);<br>
<br>
Modified: llvm/trunk/lib/Transforms/Vectorize/SLPVectorizer.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Vectorize/SLPVectorizer.cpp?rev=312791&r1=312790&r2=312791&view=diff" rel="noreferrer" target="_blank">
http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Vectorize/SLPVectorizer.cpp?rev=312791&r1=312790&r2=312791&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Transforms/Vectorize/SLPVectorizer.cpp (original)<br>
+++ llvm/trunk/lib/Transforms/Vectorize/SLPVectorizer.cpp Fri Sep 8 06:49:36 2017<br>
@@ -4627,11 +4627,17 @@ class HorizontalReduction {<br>
// Use map vector to make stable output.<br>
MapVector<Instruction *, Value *> ExtraArgs;<br>
<br>
+ /// Kind of the reduction data.<br>
+ enum ReductionKind {<br>
+ RK_None, /// Not a reduction.<br>
+ RK_Arithmetic, /// Binary reduction data.<br>
+ RK_Min, /// Minimum reduction data.<br>
+ RK_UMin, /// Unsigned minimum reduction data.<br>
+ RK_Max, /// Maximum reduction data.<br>
+ RK_UMax, /// Unsigned maximum reduction data.<br>
+ };<br>
/// Contains info about operation, like its opcode, left and right operands.<br>
- struct OperationData {<br>
- /// true if the operation is a reduced value, false if reduction operation.<br>
- bool IsReducedValue = false;<br>
-<br>
+ class OperationData {<br>
/// Opcode of the instruction.<br>
unsigned Opcode = 0;<br>
<br>
@@ -4640,12 +4646,21 @@ class HorizontalReduction {<br>
<br>
/// Right operand of the reduction operation.<br>
Value *RHS = nullptr;<br>
+ /// Kind of the reduction operation.<br>
+ ReductionKind Kind = RK_None;<br>
+ /// True if float point min/max reduction has no NaNs.<br>
+ bool NoNaN = false;<br>
<br>
/// Checks if the reduction operation can be vectorized.<br>
bool isVectorizable() const {<br>
return LHS && RHS &&<br>
- // We currently only support adds.<br>
- (Opcode == Instruction::Add || Opcode == Instruction::FAdd);<br>
+ // We currently only support adds && min/max reductions.<br>
+ ((Kind == RK_Arithmetic &&<br>
+ (Opcode == Instruction::Add || Opcode == Instruction::FAdd)) ||<br>
+ ((Opcode == Instruction::ICmp || Opcode == Instruction::FCmp) &&<br>
+ (Kind == RK_Min || Kind == RK_Max)) ||<br>
+ (Opcode == Instruction::ICmp &&<br>
+ (Kind == RK_UMin || Kind == RK_UMax)));<br>
}<br>
<br>
public:<br>
@@ -4653,43 +4668,90 @@ class HorizontalReduction {<br>
<br>
/// Construction for reduced values. They are identified by opcode only and<br>
/// don't have associated LHS/RHS values.<br>
- explicit OperationData(Value *V) : IsReducedValue(true) {<br>
+ explicit OperationData(Value *V) : Kind(RK_None) {<br>
if (auto *I = dyn_cast<Instruction>(V))<br>
Opcode = I->getOpcode();<br>
}<br>
<br>
- /// Constructor for binary reduction operations with opcode and its left and<br>
+ /// Constructor for reduction operations with opcode and its left and<br>
/// right operands.<br>
- OperationData(unsigned Opcode, Value *LHS, Value *RHS)<br>
- : Opcode(Opcode), LHS(LHS), RHS(RHS) {}<br>
-<br>
+ OperationData(unsigned Opcode, Value *LHS, Value *RHS, ReductionKind Kind,<br>
+ bool NoNaN = false)<br>
+ : Opcode(Opcode), LHS(LHS), RHS(RHS), Kind(Kind), NoNaN(NoNaN) {<br>
+ assert(Kind != RK_None && "One of the reduction operations is expected.");<br>
+ }<br>
explicit operator bool() const { return Opcode; }<br>
<br>
/// Get the index of the first operand.<br>
unsigned getFirstOperandIndex() const {<br>
assert(!!*this && "The opcode is not set.");<br>
+ switch (Kind) {<br>
+ case RK_Min:<br>
+ case RK_UMin:<br>
+ case RK_Max:<br>
+ case RK_UMax:<br>
+ return 1;<br>
+ case RK_Arithmetic:<br>
+ case RK_None:<br>
+ break;<br>
+ }<br>
return 0;<br>
}<br>
<br>
/// Total number of operands in the reduction operation.<br>
unsigned getNumberOfOperands() const {<br>
- assert(!IsReducedValue && !!*this && LHS && RHS &&<br>
+ assert(Kind != RK_None && !!*this && LHS && RHS &&<br>
"Expected reduction operation.");<br>
- return 2;<br>
+ switch (Kind) {<br>
+ case RK_Arithmetic:<br>
+ return 2;<br>
+ case RK_Min:<br>
+ case RK_UMin:<br>
+ case RK_Max:<br>
+ case RK_UMax:<br>
+ return 3;<br>
+ case RK_None:<br>
+ llvm_unreachable("Reduction kind is not set");<br>
+ }<br>
}<br>
<br>
/// Expected number of uses for reduction operations/reduced values.<br>
unsigned getRequiredNumberOfUses() const {<br>
- assert(!IsReducedValue && !!*this && LHS && RHS &&<br>
+ assert(Kind != RK_None && !!*this && LHS && RHS &&<br>
"Expected reduction operation.");<br>
- return 1;<br>
+ switch (Kind) {<br>
+ case RK_Arithmetic:<br>
+ return 1;<br>
+ case RK_Min:<br>
+ case RK_UMin:<br>
+ case RK_Max:<br>
+ case RK_UMax:<br>
+ return 2;<br>
+ case RK_None:<br>
+ llvm_unreachable("Reduction kind is not set");<br>
+ }<br>
}<br>
<br>
/// Checks if instruction is associative and can be vectorized.<br>
bool isAssociative(Instruction *I) const {<br>
- assert(!IsReducedValue && *this && LHS && RHS &&<br>
+ assert(Kind != RK_None && *this && LHS && RHS &&<br>
"Expected reduction operation.");<br>
- return I->isAssociative();<br>
+ switch (Kind) {<br>
+ case RK_Arithmetic:<br>
+ return I->isAssociative();<br>
+ case RK_Min:<br>
+ case RK_Max:<br>
+ return Opcode == Instruction::ICmp ||<br>
+ cast<Instruction>(I->getOperand(0))->hasUnsafeAlgebra();<br>
+ case RK_UMin:<br>
+ case RK_UMax:<br>
+ assert(Opcode == Instruction::ICmp &&<br>
+ "Only integer compare operation is expected.");<br>
+ return true;<br>
+ case RK_None:<br>
+ break;<br>
+ }<br>
+ llvm_unreachable("Reduction kind is not set");<br>
}<br>
<br>
/// Checks if the reduction operation can be vectorized.<br>
@@ -4700,18 +4762,17 @@ class HorizontalReduction {<br>
/// Checks if two operation data are both a reduction op or both a reduced<br>
/// value.<br>
bool operator==(const OperationData &OD) {<br>
- assert(((IsReducedValue != OD.IsReducedValue) ||<br>
- ((!LHS == !OD.LHS) && (!RHS == !OD.RHS))) &&<br>
+ assert(((Kind != OD.Kind) || ((!LHS == !OD.LHS) && (!RHS == !OD.RHS))) &&<br>
"One of the comparing operations is incorrect.");<br>
- return this == &OD ||<br>
- (IsReducedValue == OD.IsReducedValue && Opcode == OD.Opcode);<br>
+ return this == &OD || (Kind == OD.Kind && Opcode == OD.Opcode);<br>
}<br>
bool operator!=(const OperationData &OD) { return !(*this == OD); }<br>
void clear() {<br>
- IsReducedValue = false;<br>
Opcode = 0;<br>
LHS = nullptr;<br>
RHS = nullptr;<br>
+ Kind = RK_None;<br>
+ NoNaN = false;<br>
}<br>
<br>
/// Get the opcode of the reduction operation.<br>
@@ -4720,16 +4781,81 @@ class HorizontalReduction {<br>
return Opcode;<br>
}<br>
<br>
+ /// Get kind of reduction data.<br>
+ ReductionKind getKind() const { return Kind; }<br>
Value *getLHS() const { return LHS; }<br>
Value *getRHS() const { return RHS; }<br>
+ Type *getConditionType() const {<br>
+ switch (Kind) {<br>
+ case RK_Arithmetic:<br>
+ return nullptr;<br>
+ case RK_Min:<br>
+ case RK_Max:<br>
+ case RK_UMin:<br>
+ case RK_UMax:<br>
+ return CmpInst::makeCmpResultType(LHS->getType());<br>
+ case RK_None:<br>
+ break;<br>
+ }<br>
+ llvm_unreachable("Reduction kind is not set");<br>
+ }<br>
<br>
/// Creates reduction operation with the current opcode.<br>
Value *createOp(IRBuilder<> &Builder, const Twine &Name = "") const {<br>
- assert(!IsReducedValue &&<br>
- (Opcode == Instruction::FAdd || Opcode == Instruction::Add) &&<br>
- "Expected add|fadd reduction operation.");<br>
- return Builder.CreateBinOp((Instruction::BinaryOps)Opcode, LHS, RHS,<br>
- Name);<br>
+ assert(isVectorizable() &&<br>
+ "Expected add|fadd or min/max reduction operation.");<br>
+ Value *Cmp;<br>
+ switch (Kind) {<br>
+ case RK_Arithmetic:<br>
+ return Builder.CreateBinOp((Instruction::BinaryOps)Opcode, LHS, RHS,<br>
+ Name);<br>
+ case RK_Min:<br>
+ Cmp = Opcode == Instruction::ICmp ? Builder.CreateICmpSLT(LHS, RHS)<br>
+ : Builder.CreateFCmpOLT(LHS, RHS);<br>
+ break;<br>
+ case RK_Max:<br>
+ Cmp = Opcode == Instruction::ICmp ? Builder.CreateICmpSGT(LHS, RHS)<br>
+ : Builder.CreateFCmpOGT(LHS, RHS);<br>
+ break;<br>
+ case RK_UMin:<br>
+ assert(Opcode == Instruction::ICmp && "Expected integer types.");<br>
+ Cmp = Builder.CreateICmpULT(LHS, RHS);<br>
+ break;<br>
+ case RK_UMax:<br>
+ assert(Opcode == Instruction::ICmp && "Expected integer types.");<br>
+ Cmp = Builder.CreateICmpUGT(LHS, RHS);<br>
+ break;<br>
+ case RK_None:<br>
+ llvm_unreachable("Unknown reduction operation.");<br>
+ }<br>
+ return Builder.CreateSelect(Cmp, LHS, RHS, Name);<br>
+ }<br>
+ TargetTransformInfo::ReductionFlags getFlags() const {<br>
+ TargetTransformInfo::ReductionFlags Flags;<br>
+ Flags.NoNaN = NoNaN;<br>
+ switch (Kind) {<br>
+ case RK_Arithmetic:<br>
+ break;<br>
+ case RK_Min:<br>
+ Flags.IsSigned = Opcode == Instruction::ICmp;<br>
+ Flags.IsMaxOp = false;<br>
+ break;<br>
+ case RK_Max:<br>
+ Flags.IsSigned = Opcode == Instruction::ICmp;<br>
+ Flags.IsMaxOp = true;<br>
+ break;<br>
+ case RK_UMin:<br>
+ Flags.IsSigned = false;<br>
+ Flags.IsMaxOp = false;<br>
+ break;<br>
+ case RK_UMax:<br>
+ Flags.IsSigned = false;<br>
+ Flags.IsMaxOp = true;<br>
+ break;<br>
+ case RK_None:<br>
+ llvm_unreachable("Reduction kind is not set");<br>
+ }<br>
+ return Flags;<br>
}<br>
};<br>
<br>
@@ -4771,8 +4897,32 @@ class HorizontalReduction {<br>
<br>
Value *LHS;<br>
Value *RHS;<br>
- if (m_BinOp(m_Value(LHS), m_Value(RHS)).match(V))<br>
- return OperationData(cast<BinaryOperator>(V)->getOpcode(), LHS, RHS);<br>
+ if (m_BinOp(m_Value(LHS), m_Value(RHS)).match(V)) {<br>
+ return OperationData(cast<BinaryOperator>(V)->getOpcode(), LHS, RHS,<br>
+ RK_Arithmetic);<br>
+ }<br>
+ if (auto *Select = dyn_cast<SelectInst>(V)) {<br>
+ // Look for a min/max pattern.<br>
+ if (m_UMin(m_Value(LHS), m_Value(RHS)).match(Select)) {<br>
+ return OperationData(Instruction::ICmp, LHS, RHS, RK_UMin);<br>
+ } else if (m_SMin(m_Value(LHS), m_Value(RHS)).match(Select)) {<br>
+ return OperationData(Instruction::ICmp, LHS, RHS, RK_Min);<br>
+ } else if (m_OrdFMin(m_Value(LHS), m_Value(RHS)).match(Select) ||<br>
+ m_UnordFMin(m_Value(LHS), m_Value(RHS)).match(Select)) {<br>
+ return OperationData(<br>
+ Instruction::FCmp, LHS, RHS, RK_Min,<br>
+ cast<Instruction>(Select->getCondition())->hasNoNaNs());<br>
+ } else if (m_UMax(m_Value(LHS), m_Value(RHS)).match(Select)) {<br>
+ return OperationData(Instruction::ICmp, LHS, RHS, RK_UMax);<br>
+ } else if (m_SMax(m_Value(LHS), m_Value(RHS)).match(Select)) {<br>
+ return OperationData(Instruction::ICmp, LHS, RHS, RK_Max);<br>
+ } else if (m_OrdFMax(m_Value(LHS), m_Value(RHS)).match(Select) ||<br>
+ m_UnordFMax(m_Value(LHS), m_Value(RHS)).match(Select)) {<br>
+ return OperationData(<br>
+ Instruction::FCmp, LHS, RHS, RK_Max,<br>
+ cast<Instruction>(Select->getCondition())->hasNoNaNs());<br>
+ }<br>
+ }<br>
return OperationData(V);<br>
}<br>
<br>
@@ -4965,8 +5115,9 @@ public:<br>
if (VectorizedTree) {<br>
Builder.SetCurrentDebugLocation(Loc);<br>
OperationData VectReductionData(ReductionData.getOpcode(),<br>
- VectorizedTree, ReducedSubTree);<br>
- VectorizedTree = VectReductionData.createOp(Builder, "bin.rdx");<br>
+ VectorizedTree, ReducedSubTree,<br>
+ ReductionData.getKind());<br>
+ VectorizedTree = VectReductionData.createOp(Builder, "op.rdx");<br>
propagateIRFlags(VectorizedTree, ReductionOps);<br>
} else<br>
VectorizedTree = ReducedSubTree;<br>
@@ -4980,7 +5131,8 @@ public:<br>
auto *I = cast<Instruction>(ReducedVals[i]);<br>
Builder.SetCurrentDebugLocation(I->getDebugLoc());<br>
OperationData VectReductionData(ReductionData.getOpcode(),<br>
- VectorizedTree, I);<br>
+ VectorizedTree, I,<br>
+ ReductionData.getKind());<br>
VectorizedTree = VectReductionData.createOp(Builder);<br>
propagateIRFlags(VectorizedTree, ReductionOps);<br>
}<br>
@@ -4991,8 +5143,9 @@ public:<br>
for (auto *I : Pair.second) {<br>
Builder.SetCurrentDebugLocation(I->getDebugLoc());<br>
OperationData VectReductionData(ReductionData.getOpcode(),<br>
- VectorizedTree, Pair.first);<br>
- VectorizedTree = VectReductionData.createOp(Builder, "bin.extra");<br>
+ VectorizedTree, Pair.first,<br>
+ ReductionData.getKind());<br>
+ VectorizedTree = VectReductionData.createOp(Builder, "op.extra");<br>
propagateIRFlags(VectorizedTree, I);<br>
}<br>
}<br>
@@ -5013,19 +5166,58 @@ private:<br>
Type *ScalarTy = FirstReducedVal->getType();<br>
Type *VecTy = VectorType::get(ScalarTy, ReduxWidth);<br>
<br>
- int PairwiseRdxCost =<br>
- TTI->getArithmeticReductionCost(ReductionData.getOpcode(), VecTy,<br>
- /*IsPairwiseForm=*/true);<br>
- int SplittingRdxCost =<br>
- TTI->getArithmeticReductionCost(ReductionData.getOpcode(), VecTy,<br>
- /*IsPairwiseForm=*/false);<br>
+ int PairwiseRdxCost;<br>
+ int SplittingRdxCost;<br>
+ bool IsUnsigned = true;<br>
+ switch (ReductionData.getKind()) {<br>
+ case RK_Arithmetic:<br>
+ PairwiseRdxCost =<br>
+ TTI->getArithmeticReductionCost(ReductionData.getOpcode(), VecTy,<br>
+ /*IsPairwiseForm=*/true);<br>
+ SplittingRdxCost =<br>
+ TTI->getArithmeticReductionCost(ReductionData.getOpcode(), VecTy,<br>
+ /*IsPairwiseForm=*/false);<br>
+ break;<br>
+ case RK_Min:<br>
+ case RK_Max:<br>
+ IsUnsigned = false;<br>
+ case RK_UMin:<br>
+ case RK_UMax: {<br>
+ Type *VecCondTy = CmpInst::makeCmpResultType(VecTy);<br>
+ PairwiseRdxCost =<br>
+ TTI->getMinMaxReductionCost(VecTy, VecCondTy,<br>
+ /*IsPairwiseForm=*/true, IsUnsigned);<br>
+ SplittingRdxCost =<br>
+ TTI->getMinMaxReductionCost(VecTy, VecCondTy,<br>
+ /*IsPairwiseForm=*/false, IsUnsigned);<br>
+ break;<br>
+ }<br>
+ case RK_None:<br>
+ llvm_unreachable("Expected arithmetic or min/max reduction operation");<br>
+ }<br>
<br>
IsPairwiseReduction = PairwiseRdxCost < SplittingRdxCost;<br>
int VecReduxCost = IsPairwiseReduction ? PairwiseRdxCost : SplittingRdxCost;<br>
<br>
- int ScalarReduxCost =<br>
- (ReduxWidth - 1) *<br>
- TTI->getArithmeticInstrCost(ReductionData.getOpcode(), ScalarTy);<br>
+ int ScalarReduxCost;<br>
+ switch (ReductionData.getKind()) {<br>
+ case RK_Arithmetic:<br>
+ ScalarReduxCost =<br>
+ TTI->getArithmeticInstrCost(ReductionData.getOpcode(), ScalarTy);<br>
+ break;<br>
+ case RK_Min:<br>
+ case RK_Max:<br>
+ case RK_UMin:<br>
+ case RK_UMax:<br>
+ ScalarReduxCost =<br>
+ TTI->getCmpSelInstrCost(ReductionData.getOpcode(), ScalarTy) +<br>
+ TTI->getCmpSelInstrCost(Instruction::Select, ScalarTy,<br>
+ CmpInst::makeCmpResultType(ScalarTy));<br>
+ break;<br>
+ case RK_None:<br>
+ llvm_unreachable("Expected arithmetic or min/max reduction operation");<br>
+ }<br>
+ ScalarReduxCost *= (ReduxWidth - 1);<br>
<br>
DEBUG(dbgs() << "SLP: Adding cost " << VecReduxCost - ScalarReduxCost<br>
<< " for reduction that starts with " << *FirstReducedVal<br>
@@ -5047,7 +5239,7 @@ private:<br>
if (!IsPairwiseReduction)<br>
return createSimpleTargetReduction(<br>
Builder, TTI, ReductionData.getOpcode(), VectorizedValue,<br>
- TargetTransformInfo::ReductionFlags(), RedOps);<br>
+ ReductionData.getFlags(), RedOps);<br>
<br>
Value *TmpVec = VectorizedValue;<br>
for (unsigned i = ReduxWidth / 2; i != 0; i >>= 1) {<br>
@@ -5062,8 +5254,8 @@ private:<br>
TmpVec, UndefValue::get(TmpVec->getType()), (RightMask),<br>
"rdx.shuf.r");<br>
OperationData VectReductionData(ReductionData.getOpcode(), LeftShuf,<br>
- RightShuf);<br>
- TmpVec = VectReductionData.createOp(Builder, "bin.rdx");<br>
+ RightShuf, ReductionData.getKind());<br>
+ TmpVec = VectReductionData.createOp(Builder, "op.rdx");<br>
propagateIRFlags(TmpVec, RedOps);<br>
}<br>
<br>
@@ -5224,9 +5416,11 @@ static bool tryToVectorizeHorReductionOr<br>
auto *Inst = dyn_cast<Instruction>(V);<br>
if (!Inst)<br>
continue;<br>
- if (auto *BI = dyn_cast<BinaryOperator>(Inst)) {<br>
+ auto *BI = dyn_cast<BinaryOperator>(Inst);<br>
+ auto *SI = dyn_cast<SelectInst>(Inst);<br>
+ if (BI || SI) {<br>
HorizontalReduction HorRdx;<br>
- if (HorRdx.matchAssociativeReduction(P, BI)) {<br>
+ if (HorRdx.matchAssociativeReduction(P, Inst)) {<br>
if (HorRdx.tryToReduce(R, TTI)) {<br>
Res = true;<br>
// Set P to nullptr to avoid re-analysis of phi node in<br>
@@ -5235,7 +5429,7 @@ static bool tryToVectorizeHorReductionOr<br>
continue;<br>
}<br>
}<br>
- if (P) {<br>
+ if (P && BI) {<br>
Inst = dyn_cast<Instruction>(BI->getOperand(0));<br>
if (Inst == P)<br>
Inst = dyn_cast<Instruction>(BI->getOperand(1));<br>
<br>
Modified: llvm/trunk/test/Transforms/SLPVectorizer/X86/horizontal-list.ll<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/horizontal-list.ll?rev=312791&r1=312790&r2=312791&view=diff" rel="noreferrer" target="_blank">
http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/horizontal-list.ll?rev=312791&r1=312790&r2=312791&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/test/Transforms/SLPVectorizer/X86/horizontal-list.ll (original)<br>
+++ llvm/trunk/test/Transforms/SLPVectorizer/X86/horizontal-list.ll Fri Sep 8 06:49:36 2017<br>
@@ -117,11 +117,11 @@ define float @bazz() {<br>
; CHECK-NEXT: [[RDX_SHUF3:%.*]] = shufflevector <8 x float> [[BIN_RDX2]], <8 x float> undef, <8 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef><br>
; CHECK-NEXT: [[BIN_RDX4:%.*]] = fadd fast <8 x float> [[BIN_RDX2]], [[RDX_SHUF3]]<br>
; CHECK-NEXT: [[TMP4:%.*]] = extractelement <8 x float> [[BIN_RDX4]], i32 0<br>
-; CHECK-NEXT: [[BIN_EXTRA:%.*]] = fadd fast float [[TMP4]], [[CONV]]<br>
-; CHECK-NEXT: [[BIN_EXTRA5:%.*]] = fadd fast float [[BIN_EXTRA]], [[CONV6]]<br>
+; CHECK-NEXT: [[OP_EXTRA:%.*]] = fadd fast float [[TMP4]], [[CONV]]<br>
+; CHECK-NEXT: [[OP_EXTRA5:%.*]] = fadd fast float [[OP_EXTRA]], [[CONV6]]<br>
; CHECK-NEXT: [[ADD19_3:%.*]] = fadd fast float undef, [[ADD19_2]]<br>
-; CHECK-NEXT: store float [[BIN_EXTRA5]], float* @res, align 4<br>
-; CHECK-NEXT: ret float [[BIN_EXTRA5]]<br>
+; CHECK-NEXT: store float [[OP_EXTRA5]], float* @res, align 4<br>
+; CHECK-NEXT: ret float [[OP_EXTRA5]]<br>
;<br>
; THRESHOLD-LABEL: @bazz(<br>
; THRESHOLD-NEXT: entry:<br>
@@ -148,11 +148,11 @@ define float @bazz() {<br>
; THRESHOLD-NEXT: [[RDX_SHUF3:%.*]] = shufflevector <8 x float> [[BIN_RDX2]], <8 x float> undef, <8 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef><br>
; THRESHOLD-NEXT: [[BIN_RDX4:%.*]] = fadd fast <8 x float> [[BIN_RDX2]], [[RDX_SHUF3]]<br>
; THRESHOLD-NEXT: [[TMP4:%.*]] = extractelement <8 x float> [[BIN_RDX4]], i32 0<br>
-; THRESHOLD-NEXT: [[BIN_EXTRA:%.*]] = fadd fast float [[TMP4]], [[CONV]]<br>
-; THRESHOLD-NEXT: [[BIN_EXTRA5:%.*]] = fadd fast float [[BIN_EXTRA]], [[CONV6]]<br>
+; THRESHOLD-NEXT: [[OP_EXTRA:%.*]] = fadd fast float [[TMP4]], [[CONV]]<br>
+; THRESHOLD-NEXT: [[OP_EXTRA5:%.*]] = fadd fast float [[OP_EXTRA]], [[CONV6]]<br>
; THRESHOLD-NEXT: [[ADD19_3:%.*]] = fadd fast float undef, [[ADD19_2]]<br>
-; THRESHOLD-NEXT: store float [[BIN_EXTRA5]], float* @res, align 4<br>
-; THRESHOLD-NEXT: ret float [[BIN_EXTRA5]]<br>
+; THRESHOLD-NEXT: store float [[OP_EXTRA5]], float* @res, align 4<br>
+; THRESHOLD-NEXT: ret float [[OP_EXTRA5]]<br>
;<br>
entry:<br>
%0 = load i32, i32* @n, align 4<br>
@@ -327,47 +327,53 @@ entry:<br>
define float @bar() {<br>
; CHECK-LABEL: @bar(<br>
; CHECK-NEXT: entry:<br>
-; CHECK-NEXT: [[TMP0:%.*]] = load <2 x float>, <2 x float>* bitcast ([20 x float]* @arr to <2 x float>*), align 16<br>
-; CHECK-NEXT: [[TMP1:%.*]] = load <2 x float>, <2 x float>* bitcast ([20 x float]* @arr1 to <2 x float>*), align 16<br>
-; CHECK-NEXT: [[TMP2:%.*]] = fmul fast <2 x float> [[TMP1]], [[TMP0]]<br>
-; CHECK-NEXT: [[TMP3:%.*]] = extractelement <2 x float> [[TMP2]], i32 0<br>
-; CHECK-NEXT: [[TMP4:%.*]] = extractelement <2 x float> [[TMP2]], i32 1<br>
+; CHECK-NEXT: [[TMP0:%.*]] = load <4 x float>, <4 x float>* bitcast ([20 x float]* @arr to <4 x float>*), align 16<br>
+; CHECK-NEXT: [[TMP1:%.*]] = load <4 x float>, <4 x float>* bitcast ([20 x float]* @arr1 to <4 x float>*), align 16<br>
+; CHECK-NEXT: [[TMP2:%.*]] = fmul fast <4 x float> [[TMP1]], [[TMP0]]<br>
+; CHECK-NEXT: [[TMP3:%.*]] = extractelement <4 x float> [[TMP2]], i32 0<br>
+; CHECK-NEXT: [[TMP4:%.*]] = extractelement <4 x float> [[TMP2]], i32 1<br>
; CHECK-NEXT: [[CMP4:%.*]] = fcmp fast ogt float [[TMP3]], [[TMP4]]<br>
-; CHECK-NEXT: [[MAX_0_MUL3:%.*]] = select i1 [[CMP4]], float [[TMP3]], float [[TMP4]]<br>
-; CHECK-NEXT: [[TMP5:%.*]] = load float, float* getelementptr inbounds ([20 x float], [20 x float]* @arr, i64 0, i64 2), align 8<br>
-; CHECK-NEXT: [[TMP6:%.*]] = load float, float* getelementptr inbounds ([20 x float], [20 x float]* @arr1, i64 0, i64 2), align 8<br>
-; CHECK-NEXT: [[MUL3_1:%.*]] = fmul fast float [[TMP6]], [[TMP5]]<br>
-; CHECK-NEXT: [[CMP4_1:%.*]] = fcmp fast ogt float [[MAX_0_MUL3]], [[MUL3_1]]<br>
-; CHECK-NEXT: [[MAX_0_MUL3_1:%.*]] = select i1 [[CMP4_1]], float [[MAX_0_MUL3]], float [[MUL3_1]]<br>
-; CHECK-NEXT: [[TMP7:%.*]] = load float, float* getelementptr inbounds ([20 x float], [20 x float]* @arr, i64 0, i64 3), align 4<br>
-; CHECK-NEXT: [[TMP8:%.*]] = load float, float* getelementptr inbounds ([20 x float], [20 x float]* @arr1, i64 0, i64 3), align 4<br>
-; CHECK-NEXT: [[MUL3_2:%.*]] = fmul fast float [[TMP8]], [[TMP7]]<br>
-; CHECK-NEXT: [[CMP4_2:%.*]] = fcmp fast ogt float [[MAX_0_MUL3_1]], [[MUL3_2]]<br>
-; CHECK-NEXT: [[MAX_0_MUL3_2:%.*]] = select i1 [[CMP4_2]], float [[MAX_0_MUL3_1]], float [[MUL3_2]]<br>
-; CHECK-NEXT: store float [[MAX_0_MUL3_2]], float* @res, align 4<br>
-; CHECK-NEXT: ret float [[MAX_0_MUL3_2]]<br>
+; CHECK-NEXT: [[MAX_0_MUL3:%.*]] = select i1 [[CMP4]], float undef, float undef<br>
+; CHECK-NEXT: [[TMP5:%.*]] = extractelement <4 x float> [[TMP2]], i32 2<br>
+; CHECK-NEXT: [[CMP4_1:%.*]] = fcmp fast ogt float [[MAX_0_MUL3]], [[TMP5]]<br>
+; CHECK-NEXT: [[MAX_0_MUL3_1:%.*]] = select i1 [[CMP4_1]], float [[MAX_0_MUL3]], float undef<br>
+; CHECK-NEXT: [[TMP6:%.*]] = extractelement <4 x float> [[TMP2]], i32 3<br>
+; CHECK-NEXT: [[CMP4_2:%.*]] = fcmp fast ogt float [[MAX_0_MUL3_1]], [[TMP6]]<br>
+; CHECK-NEXT: [[RDX_SHUF:%.*]] = shufflevector <4 x float> [[TMP2]], <4 x float> undef, <4 x i32> <i32 2, i32 3, i32 undef, i32 undef><br>
+; CHECK-NEXT: [[RDX_MINMAX_CMP:%.*]] = fcmp fast ogt <4 x float> [[TMP2]], [[RDX_SHUF]]<br>
+; CHECK-NEXT: [[RDX_MINMAX_SELECT:%.*]] = select <4 x i1> [[RDX_MINMAX_CMP]], <4 x float> [[TMP2]], <4 x float> [[RDX_SHUF]]<br>
+; CHECK-NEXT: [[RDX_SHUF1:%.*]] = shufflevector <4 x float> [[RDX_MINMAX_SELECT]], <4 x float> undef, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef><br>
+; CHECK-NEXT: [[RDX_MINMAX_CMP2:%.*]] = fcmp fast ogt <4 x float> [[RDX_MINMAX_SELECT]], [[RDX_SHUF1]]<br>
+; CHECK-NEXT: [[RDX_MINMAX_SELECT3:%.*]] = select <4 x i1> [[RDX_MINMAX_CMP2]], <4 x float> [[RDX_MINMAX_SELECT]], <4 x float> [[RDX_SHUF1]]<br>
+; CHECK-NEXT: [[TMP7:%.*]] = extractelement <4 x float> [[RDX_MINMAX_SELECT3]], i32 0<br>
+; CHECK-NEXT: [[MAX_0_MUL3_2:%.*]] = select i1 [[CMP4_2]], float [[MAX_0_MUL3_1]], float undef<br>
+; CHECK-NEXT: store float [[TMP7]], float* @res, align 4<br>
+; CHECK-NEXT: ret float [[TMP7]]<br>
;<br>
; THRESHOLD-LABEL: @bar(<br>
; THRESHOLD-NEXT: entry:<br>
-; THRESHOLD-NEXT: [[TMP0:%.*]] = load <2 x float>, <2 x float>* bitcast ([20 x float]* @arr to <2 x float>*), align 16<br>
-; THRESHOLD-NEXT: [[TMP1:%.*]] = load <2 x float>, <2 x float>* bitcast ([20 x float]* @arr1 to <2 x float>*), align 16<br>
-; THRESHOLD-NEXT: [[TMP2:%.*]] = fmul fast <2 x float> [[TMP1]], [[TMP0]]<br>
-; THRESHOLD-NEXT: [[TMP3:%.*]] = extractelement <2 x float> [[TMP2]], i32 0<br>
-; THRESHOLD-NEXT: [[TMP4:%.*]] = extractelement <2 x float> [[TMP2]], i32 1<br>
+; THRESHOLD-NEXT: [[TMP0:%.*]] = load <4 x float>, <4 x float>* bitcast ([20 x float]* @arr to <4 x float>*), align 16<br>
+; THRESHOLD-NEXT: [[TMP1:%.*]] = load <4 x float>, <4 x float>* bitcast ([20 x float]* @arr1 to <4 x float>*), align 16<br>
+; THRESHOLD-NEXT: [[TMP2:%.*]] = fmul fast <4 x float> [[TMP1]], [[TMP0]]<br>
+; THRESHOLD-NEXT: [[TMP3:%.*]] = extractelement <4 x float> [[TMP2]], i32 0<br>
+; THRESHOLD-NEXT: [[TMP4:%.*]] = extractelement <4 x float> [[TMP2]], i32 1<br>
; THRESHOLD-NEXT: [[CMP4:%.*]] = fcmp fast ogt float [[TMP3]], [[TMP4]]<br>
-; THRESHOLD-NEXT: [[MAX_0_MUL3:%.*]] = select i1 [[CMP4]], float [[TMP3]], float [[TMP4]]<br>
-; THRESHOLD-NEXT: [[TMP5:%.*]] = load float, float* getelementptr inbounds ([20 x float], [20 x float]* @arr, i64 0, i64 2), align 8<br>
-; THRESHOLD-NEXT: [[TMP6:%.*]] = load float, float* getelementptr inbounds ([20 x float], [20 x float]* @arr1, i64 0, i64 2), align 8<br>
-; THRESHOLD-NEXT: [[MUL3_1:%.*]] = fmul fast float [[TMP6]], [[TMP5]]<br>
-; THRESHOLD-NEXT: [[CMP4_1:%.*]] = fcmp fast ogt float [[MAX_0_MUL3]], [[MUL3_1]]<br>
-; THRESHOLD-NEXT: [[MAX_0_MUL3_1:%.*]] = select i1 [[CMP4_1]], float [[MAX_0_MUL3]], float [[MUL3_1]]<br>
-; THRESHOLD-NEXT: [[TMP7:%.*]] = load float, float* getelementptr inbounds ([20 x float], [20 x float]* @arr, i64 0, i64 3), align 4<br>
-; THRESHOLD-NEXT: [[TMP8:%.*]] = load float, float* getelementptr inbounds ([20 x float], [20 x float]* @arr1, i64 0, i64 3), align 4<br>
-; THRESHOLD-NEXT: [[MUL3_2:%.*]] = fmul fast float [[TMP8]], [[TMP7]]<br>
-; THRESHOLD-NEXT: [[CMP4_2:%.*]] = fcmp fast ogt float [[MAX_0_MUL3_1]], [[MUL3_2]]<br>
-; THRESHOLD-NEXT: [[MAX_0_MUL3_2:%.*]] = select i1 [[CMP4_2]], float [[MAX_0_MUL3_1]], float [[MUL3_2]]<br>
-; THRESHOLD-NEXT: store float [[MAX_0_MUL3_2]], float* @res, align 4<br>
-; THRESHOLD-NEXT: ret float [[MAX_0_MUL3_2]]<br>
+; THRESHOLD-NEXT: [[MAX_0_MUL3:%.*]] = select i1 [[CMP4]], float undef, float undef<br>
+; THRESHOLD-NEXT: [[TMP5:%.*]] = extractelement <4 x float> [[TMP2]], i32 2<br>
+; THRESHOLD-NEXT: [[CMP4_1:%.*]] = fcmp fast ogt float [[MAX_0_MUL3]], [[TMP5]]<br>
+; THRESHOLD-NEXT: [[MAX_0_MUL3_1:%.*]] = select i1 [[CMP4_1]], float [[MAX_0_MUL3]], float undef<br>
+; THRESHOLD-NEXT: [[TMP6:%.*]] = extractelement <4 x float> [[TMP2]], i32 3<br>
+; THRESHOLD-NEXT: [[CMP4_2:%.*]] = fcmp fast ogt float [[MAX_0_MUL3_1]], [[TMP6]]<br>
+; THRESHOLD-NEXT: [[RDX_SHUF:%.*]] = shufflevector <4 x float> [[TMP2]], <4 x float> undef, <4 x i32> <i32 2, i32 3, i32 undef, i32 undef><br>
+; THRESHOLD-NEXT: [[RDX_MINMAX_CMP:%.*]] = fcmp fast ogt <4 x float> [[TMP2]], [[RDX_SHUF]]<br>
+; THRESHOLD-NEXT: [[RDX_MINMAX_SELECT:%.*]] = select <4 x i1> [[RDX_MINMAX_CMP]], <4 x float> [[TMP2]], <4 x float> [[RDX_SHUF]]<br>
+; THRESHOLD-NEXT: [[RDX_SHUF1:%.*]] = shufflevector <4 x float> [[RDX_MINMAX_SELECT]], <4 x float> undef, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef><br>
+; THRESHOLD-NEXT: [[RDX_MINMAX_CMP2:%.*]] = fcmp fast ogt <4 x float> [[RDX_MINMAX_SELECT]], [[RDX_SHUF1]]<br>
+; THRESHOLD-NEXT: [[RDX_MINMAX_SELECT3:%.*]] = select <4 x i1> [[RDX_MINMAX_CMP2]], <4 x float> [[RDX_MINMAX_SELECT]], <4 x float> [[RDX_SHUF1]]<br>
+; THRESHOLD-NEXT: [[TMP7:%.*]] = extractelement <4 x float> [[RDX_MINMAX_SELECT3]], i32 0<br>
+; THRESHOLD-NEXT: [[MAX_0_MUL3_2:%.*]] = select i1 [[CMP4_2]], float [[MAX_0_MUL3_1]], float undef<br>
+; THRESHOLD-NEXT: store float [[TMP7]], float* @res, align 4<br>
+; THRESHOLD-NEXT: ret float [[TMP7]]<br>
;<br>
entry:<br>
%0 = load float, float* getelementptr inbounds ([20 x float], [20 x float]* @arr, i64 0, i64 0), align 16<br>
@@ -512,9 +518,9 @@ define float @f(float* nocapture readonl<br>
; CHECK-NEXT: [[RDX_SHUF15:%.*]] = shufflevector <16 x float> [[BIN_RDX14]], <16 x float> undef, <16 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef,
i32 undef, i32 undef><br>
; CHECK-NEXT: [[BIN_RDX16:%.*]] = fadd fast <16 x float> [[BIN_RDX14]], [[RDX_SHUF15]]<br>
; CHECK-NEXT: [[TMP5:%.*]] = extractelement <16 x float> [[BIN_RDX16]], i32 0<br>
-; CHECK-NEXT: [[BIN_RDX17:%.*]] = fadd fast float [[TMP4]], [[TMP5]]<br>
+; CHECK-NEXT: [[OP_RDX:%.*]] = fadd fast float [[TMP4]], [[TMP5]]<br>
; CHECK-NEXT: [[ADD_47:%.*]] = fadd fast float undef, [[ADD_46]]<br>
-; CHECK-NEXT: ret float [[BIN_RDX17]]<br>
+; CHECK-NEXT: ret float [[OP_RDX]]<br>
;<br>
; THRESHOLD-LABEL: @f(<br>
; THRESHOLD-NEXT: entry:<br>
@@ -635,9 +641,9 @@ define float @f(float* nocapture readonl<br>
; THRESHOLD-NEXT: [[RDX_SHUF15:%.*]] = shufflevector <16 x float> [[BIN_RDX14]], <16 x float> undef, <16 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32
undef, i32 undef, i32 undef><br>
; THRESHOLD-NEXT: [[BIN_RDX16:%.*]] = fadd fast <16 x float> [[BIN_RDX14]], [[RDX_SHUF15]]<br>
; THRESHOLD-NEXT: [[TMP5:%.*]] = extractelement <16 x float> [[BIN_RDX16]], i32 0<br>
-; THRESHOLD-NEXT: [[BIN_RDX17:%.*]] = fadd fast float [[TMP4]], [[TMP5]]<br>
+; THRESHOLD-NEXT: [[OP_RDX:%.*]] = fadd fast float [[TMP4]], [[TMP5]]<br>
; THRESHOLD-NEXT: [[ADD_47:%.*]] = fadd fast float undef, [[ADD_46]]<br>
-; THRESHOLD-NEXT: ret float [[BIN_RDX17]]<br>
+; THRESHOLD-NEXT: ret float [[OP_RDX]]<br>
;<br>
entry:<br>
%0 = load float, float* %x, align 4<br>
@@ -865,9 +871,9 @@ define float @f1(float* nocapture readon<br>
; CHECK-NEXT: [[RDX_SHUF7:%.*]] = shufflevector <32 x float> [[BIN_RDX6]], <32 x float> undef, <32 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef,
i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef><br>
; CHECK-NEXT: [[BIN_RDX8:%.*]] = fadd fast <32 x float> [[BIN_RDX6]], [[RDX_SHUF7]]<br>
; CHECK-NEXT: [[TMP2:%.*]] = extractelement <32 x float> [[BIN_RDX8]], i32 0<br>
-; CHECK-NEXT: [[BIN_EXTRA:%.*]] = fadd fast float [[TMP2]], [[CONV]]<br>
+; CHECK-NEXT: [[OP_EXTRA:%.*]] = fadd fast float [[TMP2]], [[CONV]]<br>
; CHECK-NEXT: [[ADD_31:%.*]] = fadd fast float undef, [[ADD_30]]<br>
-; CHECK-NEXT: ret float [[BIN_EXTRA]]<br>
+; CHECK-NEXT: ret float [[OP_EXTRA]]<br>
;<br>
; THRESHOLD-LABEL: @f1(<br>
; THRESHOLD-NEXT: entry:<br>
@@ -948,9 +954,9 @@ define float @f1(float* nocapture readon<br>
; THRESHOLD-NEXT: [[RDX_SHUF7:%.*]] = shufflevector <32 x float> [[BIN_RDX6]], <32 x float> undef, <32 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32
undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef><br>
; THRESHOLD-NEXT: [[BIN_RDX8:%.*]] = fadd fast <32 x float> [[BIN_RDX6]], [[RDX_SHUF7]]<br>
; THRESHOLD-NEXT: [[TMP2:%.*]] = extractelement <32 x float> [[BIN_RDX8]], i32 0<br>
-; THRESHOLD-NEXT: [[BIN_EXTRA:%.*]] = fadd fast float [[TMP2]], [[CONV]]<br>
+; THRESHOLD-NEXT: [[OP_EXTRA:%.*]] = fadd fast float [[TMP2]], [[CONV]]<br>
; THRESHOLD-NEXT: [[ADD_31:%.*]] = fadd fast float undef, [[ADD_30]]<br>
-; THRESHOLD-NEXT: ret float [[BIN_EXTRA]]<br>
+; THRESHOLD-NEXT: ret float [[OP_EXTRA]]<br>
;<br>
entry:<br>
%rem = srem i32 %a, %b<br>
@@ -1138,14 +1144,14 @@ define float @loadadd31(float* nocapture<br>
; CHECK-NEXT: [[RDX_SHUF11:%.*]] = shufflevector <8 x float> [[BIN_RDX10]], <8 x float> undef, <8 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef><br>
; CHECK-NEXT: [[BIN_RDX12:%.*]] = fadd fast <8 x float> [[BIN_RDX10]], [[RDX_SHUF11]]<br>
; CHECK-NEXT: [[TMP9:%.*]] = extractelement <8 x float> [[BIN_RDX12]], i32 0<br>
-; CHECK-NEXT: [[BIN_RDX13:%.*]] = fadd fast float [[TMP8]], [[TMP9]]<br>
-; CHECK-NEXT: [[RDX_SHUF14:%.*]] = shufflevector <4 x float> [[TMP3]], <4 x float> undef, <4 x i32> <i32 2, i32 3, i32 undef, i32 undef><br>
-; CHECK-NEXT: [[BIN_RDX15:%.*]] = fadd fast <4 x float> [[TMP3]], [[RDX_SHUF14]]<br>
-; CHECK-NEXT: [[RDX_SHUF16:%.*]] = shufflevector <4 x float> [[BIN_RDX15]], <4 x float> undef, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef><br>
-; CHECK-NEXT: [[BIN_RDX17:%.*]] = fadd fast <4 x float> [[BIN_RDX15]], [[RDX_SHUF16]]<br>
-; CHECK-NEXT: [[TMP10:%.*]] = extractelement <4 x float> [[BIN_RDX17]], i32 0<br>
-; CHECK-NEXT: [[BIN_RDX18:%.*]] = fadd fast float [[BIN_RDX13]], [[TMP10]]<br>
-; CHECK-NEXT: [[TMP11:%.*]] = fadd fast float [[BIN_RDX18]], [[TMP1]]<br>
+; CHECK-NEXT: [[OP_RDX:%.*]] = fadd fast float [[TMP8]], [[TMP9]]<br>
+; CHECK-NEXT: [[RDX_SHUF13:%.*]] = shufflevector <4 x float> [[TMP3]], <4 x float> undef, <4 x i32> <i32 2, i32 3, i32 undef, i32 undef><br>
+; CHECK-NEXT: [[BIN_RDX14:%.*]] = fadd fast <4 x float> [[TMP3]], [[RDX_SHUF13]]<br>
+; CHECK-NEXT: [[RDX_SHUF15:%.*]] = shufflevector <4 x float> [[BIN_RDX14]], <4 x float> undef, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef><br>
+; CHECK-NEXT: [[BIN_RDX16:%.*]] = fadd fast <4 x float> [[BIN_RDX14]], [[RDX_SHUF15]]<br>
+; CHECK-NEXT: [[TMP10:%.*]] = extractelement <4 x float> [[BIN_RDX16]], i32 0<br>
+; CHECK-NEXT: [[OP_RDX17:%.*]] = fadd fast float [[OP_RDX]], [[TMP10]]<br>
+; CHECK-NEXT: [[TMP11:%.*]] = fadd fast float [[OP_RDX17]], [[TMP1]]<br>
; CHECK-NEXT: [[TMP12:%.*]] = fadd fast float [[TMP11]], [[TMP0]]<br>
; CHECK-NEXT: [[ADD_29:%.*]] = fadd fast float undef, [[ADD_28]]<br>
; CHECK-NEXT: ret float [[TMP12]]<br>
@@ -1234,14 +1240,14 @@ define float @loadadd31(float* nocapture<br>
; THRESHOLD-NEXT: [[RDX_SHUF11:%.*]] = shufflevector <8 x float> [[BIN_RDX10]], <8 x float> undef, <8 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef><br>
; THRESHOLD-NEXT: [[BIN_RDX12:%.*]] = fadd fast <8 x float> [[BIN_RDX10]], [[RDX_SHUF11]]<br>
; THRESHOLD-NEXT: [[TMP9:%.*]] = extractelement <8 x float> [[BIN_RDX12]], i32 0<br>
-; THRESHOLD-NEXT: [[BIN_RDX13:%.*]] = fadd fast float [[TMP8]], [[TMP9]]<br>
-; THRESHOLD-NEXT: [[RDX_SHUF14:%.*]] = shufflevector <4 x float> [[TMP3]], <4 x float> undef, <4 x i32> <i32 2, i32 3, i32 undef, i32 undef><br>
-; THRESHOLD-NEXT: [[BIN_RDX15:%.*]] = fadd fast <4 x float> [[TMP3]], [[RDX_SHUF14]]<br>
-; THRESHOLD-NEXT: [[RDX_SHUF16:%.*]] = shufflevector <4 x float> [[BIN_RDX15]], <4 x float> undef, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef><br>
-; THRESHOLD-NEXT: [[BIN_RDX17:%.*]] = fadd fast <4 x float> [[BIN_RDX15]], [[RDX_SHUF16]]<br>
-; THRESHOLD-NEXT: [[TMP10:%.*]] = extractelement <4 x float> [[BIN_RDX17]], i32 0<br>
-; THRESHOLD-NEXT: [</blockquote></div></div></div></blockquote></div></div></blockquote></div>