[llvm] r185774 - SLPVectorizer: Implement DCE as part of vectorization.
Benjamin Kramer
benny.kra at gmail.com
Mon Jul 8 03:00:40 PDT 2013
On 08.07.2013, at 09:53, Alexey Samsonov <samsonov at google.com> wrote:
Looks like it wasn't fixed - I still observe the same error.
Would it make sense to have BuilderLocGuard store an AssertingVH so it
complains loudly when the insertion point is deleted while the guard still
has a reference to it?
- Ben
On Sun, Jul 7, 2013 at 6:59 PM, Nadav Rotem <nrotem at apple.com> wrote:
>
> Thanks for catching this Alexey! This should be fixed in r185777.
>
> On Jul 7, 2013, at 5:22 AM, Alexey Samsonov <samsonov at google.com> wrote:
>
> Hi Nadav!
>
> Our ASan bootstrap bot reports a problem with this:
> =================================================================
> ==9987==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x60b00000ae90 at pc 0x15b0917 bp 0x7fff39e016b0 sp 0x7fff39e016a8
> READ of size 8 at 0x60b00000ae90 thread T0
> #0 0x15b0916 in getParent /build/llvm/include/llvm/IR/Instruction.h:53
> #1 0x15b0916 in SetInsertPoint
> /build/llvm/include/llvm/IR/IRBuilder.h:90
> #2 0x15b0916 in ~BuilderLocGuard
> /build/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:62
> #3 0x15b0916 in (anonymous
> namespace)::BoUpSLP::vectorizeTree((anonymous
> namespace)::BoUpSLP::TreeEntry*)
> /build/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:1217
> #4 0x15aad0c in (anonymous namespace)::BoUpSLP::vectorizeTree()
> /build/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:1220
> #5 0x15a44c3 in vectorizeStoreChain
> /build/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:1512
> #6 0x15a44c3 in vectorizeStores
> /build/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:1582
> #7 0x15a44c3 in vectorizeStoreChains
> /build/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:1781
> #8 0x15a44c3 in (anonymous
> namespace)::SLPVectorizer::runOnFunction(llvm::Function&)
> /build/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:1425
> #9 0x2b029a2 in llvm::FPPassManager::runOnFunction(llvm::Function&)
> /build/llvm/lib/IR/PassManager.cpp:1530
> #10 0x2b02f75 in llvm::FPPassManager::runOnModule(llvm::Module&)
> /build/llvm/lib/IR/PassManager.cpp:1550
> #11 0x2b037a0 in llvm::MPPassManager::runOnModule(llvm::Module&)
> /build/llvm/lib/IR/PassManager.cpp:1608
> #12 0x2b049a3 in llvm::PassManagerImpl::run(llvm::Module&)
> /build/llvm/lib/IR/PassManager.cpp:1703
> #13 0x2b04e1f in llvm::PassManager::run(llvm::Module&)
> /build/llvm/lib/IR/PassManager.cpp:1738
> #14 0x61dc7d in main /build/llvm/tools/opt/opt.cpp:826
> #15 0x7fac4bb4476c (/lib/x86_64-linux-gnu/libc.so.6+0x2176c)
> #16 0x60ecec in _start (/build/llvm_build_asan/bin/opt+0x60ecec)
> 0x60b00000ae90 is located 96 bytes inside of 112-byte region
> [0x60b00000ae30,0x60b00000aea0)
> freed by thread T0 here:
> #0 0x5f96c5 in operator delete(void*)
> /build/llvm/projects/compiler-rt/lib/asan/asan_new_delete.cc:83
> #1 0x15ac453 in (anonymous namespace)::BoUpSLP::vectorizeTree()
> /build/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:1288
> #2 0x15a44c3 in vectorizeStoreChain
> /build/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:1512
> #3 0x15a44c3 in vectorizeStores
> /build/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:1582
> #4 0x15a44c3 in vectorizeStoreChains
> /build/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:1781
> #5 0x15a44c3 in (anonymous
> namespace)::SLPVectorizer::runOnFunction(llvm::Function&)
> /build/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:1425
> #6 0x2b029a2 in llvm::FPPassManager::runOnFunction(llvm::Function&)
> /build/llvm/lib/IR/PassManager.cpp:1530
> #7 0x2b02f75 in llvm::FPPassManager::runOnModule(llvm::Module&)
> /build/llvm/lib/IR/PassManager.cpp:1550
> #8 0x2b037a0 in llvm::MPPassManager::runOnModule(llvm::Module&)
> /build/llvm/lib/IR/PassManager.cpp:1608
> #9 0x2b049a3 in llvm::PassManagerImpl::run(llvm::Module&)
> /build/llvm/lib/IR/PassManager.cpp:1703
> #10 0x2b04e1f in llvm::PassManager::run(llvm::Module&)
> /build/llvm/lib/IR/PassManager.cpp:1738
> #11 0x61dc7d in main /build/llvm/tools/opt/opt.cpp:826
> #12 0x7fac4bb4476c (/lib/x86_64-linux-gnu/libc.so.6+0x2176c)
> previously allocated by thread T0 here:
> #0 0x5f9405 in operator new(unsigned long)
> /build/llvm/projects/compiler-rt/lib/asan/asan_new_delete.cc:52
> #1 0x2b2ed73 in llvm::User::operator new(unsigned long, unsigned int)
> /build/llvm/lib/IR/User.cpp:60
> #2 0x14a4391 in operator new
> /build/llvm/include/llvm/IR/InstrTypes.h:105
> #3 0x14a4391 in llvm::LLParser::ParseLoad(llvm::Instruction*&,
> llvm::LLParser::PerFunctionState&)
> /build/llvm/lib/AsmParser/LLParser.cpp:4096
> #4 0x149380c in llvm::LLParser::ParseInstruction(llvm::Instruction*&,
> llvm::BasicBlock*, llvm::LLParser::PerFunctionState&)
> /build/llvm/lib/AsmParser/LLParser.cpp:3316
> #5 0x14927b8 in
> llvm::LLParser::ParseBasicBlock(llvm::LLParser::PerFunctionState&)
> /build/llvm/lib/AsmParser/LLParser.cpp:3190
> #6 0x146af2f in llvm::LLParser::ParseFunctionBody(llvm::Function&)
> /build/llvm/lib/AsmParser/LLParser.cpp:3143
> #7 0x14584c0 in ParseDefine /build/llvm/lib/AsmParser/LLParser.cpp:424
> #8 0x14584c0 in llvm::LLParser::ParseTopLevelEntities()
> /build/llvm/lib/AsmParser/LLParser.cpp:226
> #9 0x145813e in llvm::LLParser::Run()
> /build/llvm/lib/AsmParser/LLParser.cpp:41
> #10 0x1449c59 in llvm::ParseAssembly(llvm::MemoryBuffer*,
> llvm::Module*, llvm::SMDiagnostic&, llvm::LLVMContext&)
> /build/llvm/lib/AsmParser/Parser.cpp:38
> #11 0x11be978 in llvm::ParseIR(llvm::MemoryBuffer*,
> llvm::SMDiagnostic&, llvm::LLVMContext&)
> /build/llvm/lib/IRReader/IRReader.cpp:76
> #12 0x11bf290 in llvm::ParseIRFile(std::string const&,
> llvm::SMDiagnostic&, llvm::LLVMContext&)
> /build/llvm/lib/IRReader/IRReader.cpp:88
> #13 0x618e4c in main /build/llvm/tools/opt/opt.cpp:595
> #14 0x7fac4bb4476c (/lib/x86_64-linux-gnu/libc.so.6+0x2176c)
> SUMMARY: AddressSanitizer: heap-use-after-free
> /build/llvm/include/llvm/IR/Instruction.h:53 getParent
>
>
> On Sun, Jul 7, 2013 at 10:57 AM, Nadav Rotem <nrotem at apple.com> wrote:
>
>> Author: nadav
>> Date: Sun Jul 7 01:57:07 2013
>> New Revision: 185774
>>
>> URL: http://llvm.org/viewvc/llvm-project?rev=185774&view=rev
>> Log:
>> SLPVectorizer: Implement DCE as part of vectorization.
>>
>> This is a complete re-write if the bottom-up vectorization class.
>> Before this commit we scanned the instruction tree 3 times. First in
>> search of merge points for the trees. Second, for estimating the cost. And
>> finally for vectorization.
>> There was a lot of code duplication and adding the DCE exposed bugs. The
>> new design is simpler and DCE was a part of the design.
>> In this implementation we build the tree once. After that we estimate the
>> cost by scanning the different entries in the constructed tree (in any
>> order). The vectorization phase also works on the built tree.
>>
>>
>> Added:
>> llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_7zip.ll
>> llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_bullet.ll
>> llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_bullet2.ll
>> llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_dequeue.ll
>> llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_flop7.ll
>> llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_lame.ll
>> llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_lencod.ll
>> llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_lencod2.ll
>> llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_mandeltext.ll
>> llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_rc4.ll
>> llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_sim4b1.ll
>> llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_smallpt.ll
>> llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_smallpt2.ll
>> Modified:
>> llvm/trunk/lib/Transforms/Vectorize/SLPVectorizer.cpp
>> llvm/trunk/test/Transforms/SLPVectorizer/X86/diamond.ll
>> llvm/trunk/test/Transforms/SLPVectorizer/X86/long_chains.ll
>> llvm/trunk/test/Transforms/SLPVectorizer/X86/saxpy.ll
>>
>> Modified: llvm/trunk/lib/Transforms/Vectorize/SLPVectorizer.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Vectorize/SLPVectorizer.cpp?rev=185774&r1=185773&r2=185774&view=diff
>>
>> ==============================================================================
>> --- llvm/trunk/lib/Transforms/Vectorize/SLPVectorizer.cpp (original)
>> +++ llvm/trunk/lib/Transforms/Vectorize/SLPVectorizer.cpp Sun Jul 7
>> 01:57:07 2013
>> @@ -59,7 +59,7 @@ static const unsigned RecursionMaxDepth
>> class BuilderLocGuard {
>> public:
>> BuilderLocGuard(IRBuilder<> &B) : Builder(B), Loc(B.GetInsertPoint())
>> {}
>> - ~BuilderLocGuard() { Builder.SetInsertPoint(Loc); }
>> + ~BuilderLocGuard() { if (Loc) Builder.SetInsertPoint(Loc); }
>>
>> private:
>> // Prevent copying.
>> @@ -91,6 +91,7 @@ struct BlockNumbering {
>> }
>>
>> int getIndex(Instruction *I) {
>> + assert(I->getParent() == BB && "Invalid instruction");
>> if (!Valid)
>> numberInstructions();
>> assert(InstrIdx.count(I) && "Unknown instruction");
>> @@ -117,72 +118,169 @@ private:
>> std::vector<Instruction *> InstrVec;
>> };
>>
>> -class FuncSLP {
>> +/// \returns the parent basic block if all of the instructions in \p VL
>> +/// are in the same block or null otherwise.
>> +static BasicBlock *getSameBlock(ArrayRef<Value *> VL) {
>> + Instruction *I0 = dyn_cast<Instruction>(VL[0]);
>> + if (!I0)
>> + return 0;
>> + BasicBlock *BB = I0->getParent();
>> + for (int i = 1, e = VL.size(); i < e; i++) {
>> + Instruction *I = dyn_cast<Instruction>(VL[i]);
>> + if (!I)
>> + return 0;
>> +
>> + if (BB != I->getParent())
>> + return 0;
>> + }
>> + return BB;
>> +}
>> +
>> +/// \returns True if all of the values in \p VL are constants.
>> +static bool allConstant(ArrayRef<Value *> VL) {
>> + for (unsigned i = 0, e = VL.size(); i < e; ++i)
>> + if (!isa<Constant>(VL[i]))
>> + return false;
>> + return true;
>> +}
>> +
>> +/// \returns True if all of the values in \p VL are identical.
>> +static bool isSplat(ArrayRef<Value *> VL) {
>> + for (unsigned i = 1, e = VL.size(); i < e; ++i)
>> + if (VL[i] != VL[0])
>> + return false;
>> + return true;
>> +}
>> +
>> +/// \returns The opcode if all of the Instructions in \p VL have the same
>> +/// opcode, or zero.
>> +static unsigned getSameOpcode(ArrayRef<Value *> VL) {
>> + Instruction *I0 = dyn_cast<Instruction>(VL[0]);
>> + if (!I0)
>> + return 0;
>> + unsigned Opcode = I0->getOpcode();
>> + for (int i = 1, e = VL.size(); i < e; i++) {
>> + Instruction *I = dyn_cast<Instruction>(VL[i]);
>> + if (!I || Opcode != I->getOpcode())
>> + return 0;
>> + }
>> + return Opcode;
>> +}
>> +
>> +/// \returns The type that all of the values in \p VL have or null if
>> there
>> +/// are different types.
>> +static Type* getSameType(ArrayRef<Value *> VL) {
>> + Type *Ty = VL[0]->getType();
>> + for (int i = 1, e = VL.size(); i < e; i++)
>> + if (VL[0]->getType() != Ty)
>> + return 0;
>> +
>> + return Ty;
>> +}
>> +
>> +/// \returns True if the ExtractElement instructions in VL can be
>> vectorized
>> +/// to use the original vector.
>> +static bool CanReuseExtract(ArrayRef<Value *> VL) {
>> + assert(Instruction::ExtractElement == getSameOpcode(VL) && "Invalid
>> opcode");
>> + // Check if all of the extracts come from the same vector and from the
>> + // correct offset.
>> + Value *VL0 = VL[0];
>> + ExtractElementInst *E0 = cast<ExtractElementInst>(VL0);
>> + Value *Vec = E0->getOperand(0);
>> +
>> + // We have to extract from the same vector type.
>> + unsigned NElts = Vec->getType()->getVectorNumElements();
>> +
>> + if (NElts != VL.size())
>> + return false;
>> +
>> + // Check that all of the indices extract from the correct offset.
>> + ConstantInt *CI = dyn_cast<ConstantInt>(E0->getOperand(1));
>> + if (!CI || CI->getZExtValue())
>> + return false;
>> +
>> + for (unsigned i = 1, e = VL.size(); i < e; ++i) {
>> + ExtractElementInst *E = cast<ExtractElementInst>(VL[i]);
>> + ConstantInt *CI = dyn_cast<ConstantInt>(E->getOperand(1));
>> +
>> + if (!CI || CI->getZExtValue() != i || E->getOperand(0) != Vec)
>> + return false;
>> + }
>> +
>> + return true;
>> +}
>> +
>> +/// Bottom Up SLP Vectorizer.
>> +class BoUpSLP {
>> +public:
>> typedef SmallVector<Value *, 8> ValueList;
>> typedef SmallVector<Instruction *, 16> InstrList;
>> typedef SmallPtrSet<Value *, 16> ValueSet;
>> typedef SmallVector<StoreInst *, 8> StoreList;
>>
>> -public:
>> - static const int MAX_COST = INT_MIN;
>> -
>> - FuncSLP(Function *Func, ScalarEvolution *Se, DataLayout *Dl,
>> - TargetTransformInfo *Tti, AliasAnalysis *Aa, LoopInfo *Li,
>> + BoUpSLP(Function *Func, ScalarEvolution *Se, DataLayout *Dl,
>> + TargetTransformInfo *Tti, AliasAnalysis *Aa, LoopInfo *Li,
>> DominatorTree *Dt) :
>> F(Func), SE(Se), DL(Dl), TTI(Tti), AA(Aa), LI(Li), DT(Dt),
>> Builder(Se->getContext()) {
>> - for (Function::iterator it = F->begin(), e = F->end(); it != e;
>> ++it) {
>> - BasicBlock *BB = it;
>> - BlocksNumbers[BB] = BlockNumbering(BB);
>> + // Setup the block numbering utility for all of the blocks in the
>> + // function.
>> + for (Function::iterator it = F->begin(), e = F->end(); it != e;
>> ++it) {
>> + BasicBlock *BB = it;
>> + BlocksNumbers[BB] = BlockNumbering(BB);
>> + }
>> }
>> - }
>> -
>> - /// \brief Take the pointer operand from the Load/Store instruction.
>> - /// \returns NULL if this is not a valid Load/Store instruction.
>> - static Value *getPointerOperand(Value *I);
>> -
>> - /// \brief Take the address space operand from the Load/Store
>> instruction.
>> - /// \returns -1 if this is not a valid Load/Store instruction.
>> - static unsigned getAddressSpaceOperand(Value *I);
>> -
>> - /// \returns true if the memory operations A and B are consecutive.
>> - bool isConsecutiveAccess(Value *A, Value *B);
>>
>> /// \brief Vectorize the tree that starts with the elements in \p VL.
>> - /// \returns the vectorized value.
>> - Value *vectorizeTree(ArrayRef<Value *> VL);
>> + void vectorizeTree();
>>
>> /// \returns the vectorization cost of the subtree that starts at \p
>> VL.
>> /// A negative number means that this is profitable.
>> - int getTreeCost(ArrayRef<Value *> VL);
>> + int getTreeCost();
>> +
>> + /// Construct a vectorizable tree that starts at \p Roots.
>> + void buildTree(ArrayRef<Value *> Roots);
>> +
>> + /// Clear the internal data structures that are created by 'buildTree'.
>> + void deleteTree() {
>> + VectorizableTree.clear();
>> + ScalarToTreeEntry.clear();
>> + MustGather.clear();
>> + MemBarrierIgnoreList.clear();
>> + }
>>
>> /// \returns the scalarization cost for this list of values. Assuming
>> that
>> /// this subtree gets vectorized, we may need to extract the values
>> from the
>> /// roots. This method calculates the cost of extracting the values.
>> int getGatherCost(ArrayRef<Value *> VL);
>>
>> - /// \brief Attempts to order and vectorize a sequence of stores. This
>> - /// function does a quadratic scan of the given stores.
>> - /// \returns true if the basic block was modified.
>> - bool vectorizeStores(ArrayRef<StoreInst *> Stores, int costThreshold);
>> -
>> - /// \brief Vectorize a group of scalars into a vector tree.
>> - /// \returns the vectorized value.
>> - Value *vectorizeArith(ArrayRef<Value *> Operands);
>> -
>> - /// \brief This method contains the recursive part of getTreeCost.
>> - int getTreeCost_rec(ArrayRef<Value *> VL, unsigned Depth);
>> -
>> - /// \brief This recursive method looks for vectorization hazards such
>> as
>> - /// values that are used by multiple users and checks that values are
>> used
>> - /// by only one vector lane. It updates the variables LaneMap,
>> MultiUserVals.
>> - void getTreeUses_rec(ArrayRef<Value *> VL, unsigned Depth);
>> + /// \returns true if the memory operations A and B are consecutive.
>> + bool isConsecutiveAccess(Value *A, Value *B);
>> +
>> + /// \brief Perform LICM and CSE on the newly generated gather
>> sequences.
>> + void optimizeGatherSequence();
>> +private:
>> + struct TreeEntry;
>> +
>> + /// \returns the cost of the vectorizable entry.
>> + int getEntryCost(TreeEntry *E);
>> +
>> + /// This is the recursive part of buildTree.
>> + void buildTree_rec(ArrayRef<Value *> Roots, unsigned Depth);
>>
>> - /// \brief This method contains the recursive part of vectorizeTree.
>> - Value *vectorizeTree_rec(ArrayRef<Value *> VL);
>> + /// Vectorizer a single entry in the tree.
>> + Value *vectorizeTree(TreeEntry *E);
>>
>> - /// \brief Vectorize a sorted sequence of stores.
>> - bool vectorizeStoreChain(ArrayRef<Value *> Chain, int CostThreshold);
>> + /// Vectorizer a single entry in the tree, starting in \p VL.
>> + Value *vectorizeTree(ArrayRef<Value *> VL);
>> +
>> + /// \brief Take the pointer operand from the Load/Store instruction.
>> + /// \returns NULL if this is not a valid Load/Store instruction.
>> + static Value *getPointerOperand(Value *I);
>> +
>> + /// \brief Take the address space operand from the Load/Store
>> instruction.
>> + /// \returns -1 if this is not a valid Load/Store instruction.
>> + static unsigned getAddressSpaceOperand(Value *I);
>>
>> /// \returns the scalarization cost for this type. Scalarization in
>> this
>> /// context means the creation of vectors from a group of scalars.
>> @@ -211,57 +309,65 @@ public:
>> /// \returns a vector from a collection of scalars in \p VL.
>> Value *Gather(ArrayRef<Value *> VL, VectorType *Ty);
>>
>> - /// \brief Perform LICM and CSE on the newly generated gather
>> sequences.
>> - void optimizeGatherSequence();
>> + struct TreeEntry {
>> + TreeEntry() : Scalars(), VectorizedValue(0), LastScalarIndex(0),
>> + NeedToGather(0) {}
>> +
>> + /// \returns true if the scalars in VL are equal to this entry.
>> + bool isSame(ArrayRef<Value *> VL) {
>> + assert(VL.size() == Scalars.size() && "Invalid size");
>> + for (int i = 0, e = VL.size(); i != e; ++i)
>> + if (VL[i] != Scalars[i])
>> + return false;
>> + return true;
>> + }
>>
>> - bool needToGatherAny(ArrayRef<Value *> VL) {
>> - for (int i = 0, e = VL.size(); i < e; ++i)
>> - if (MustGather.count(VL[i]))
>> - return true;
>> - return false;
>> - }
>> + /// A vector of scalars.
>> + ValueList Scalars;
>> +
>> + /// The Scalars are vectorized into this value. It is initialized to
>> Null.
>> + Value *VectorizedValue;
>>
>> - void forgetNumbering() {
>> - for (Function::iterator it = F->begin(), e = F->end(); it != e; ++it)
>> - BlocksNumbers[it].forget();
>> + /// The index in the basic block of the last scalar.
>> + int LastScalarIndex;
>> +
>> + /// Do we need to gather this sequence ?
>> + bool NeedToGather;
>> + };
>> +
>> + /// Create a new VectorizableTree entry.
>> + TreeEntry *newTreeEntry(ArrayRef<Value *> VL, bool Vectorized) {
>> + VectorizableTree.push_back(TreeEntry());
>> + int idx = VectorizableTree.size() - 1;
>> + TreeEntry *Last = &VectorizableTree[idx];
>> + Last->Scalars.insert(Last->Scalars.begin(), VL.begin(), VL.end());
>> + Last->NeedToGather = !Vectorized;
>> + if (Vectorized) {
>> + Last->LastScalarIndex = getLastIndex(VL);
>> + for (int i = 0, e = VL.size(); i != e; ++i) {
>> + assert(!ScalarToTreeEntry.count(VL[i]) && "Scalar already in
>> tree!");
>> + ScalarToTreeEntry[VL[i]] = idx;
>> + }
>> + } else {
>> + Last->LastScalarIndex = 0;
>> + MustGather.insert(VL.begin(), VL.end());
>> + }
>> + return Last;
>> }
>>
>> /// -- Vectorization State --
>> + /// Holds all of the tree entries.
>> + std::vector<TreeEntry> VectorizableTree;
>>
>> - /// Maps values in the tree to the vector lanes that uses them. This
>> map must
>> - /// be reset between runs of getCost.
>> - std::map<Value *, int> LaneMap;
>> - /// A list of instructions to ignore while sinking
>> - /// memory instructions. This map must be reset between runs of
>> getCost.
>> - ValueSet MemBarrierIgnoreList;
>> + /// Maps a specific scalar to its tree entry.
>> + SmallDenseMap<Value*, int> ScalarToTreeEntry;
>>
>> - /// Maps between the first scalar to the vector. This map must be reset
>> - /// between runs.
>> - DenseMap<Value *, Value *> VectorizedValues;
>> -
>> - /// Contains values that must be gathered because they are used
>> - /// by multiple lanes, or by users outside the tree.
>> - /// NOTICE: The vectorization methods also use this set.
>> + /// A list of scalars that we found that we need to keep as scalars.
>> ValueSet MustGather;
>>
>> - /// Contains PHINodes that are being processed. We use this data
>> structure
>> - /// to stop cycles in the graph.
>> - ValueSet VisitedPHIs;
>> -
>> - /// Contains a list of values that are used outside the current tree,
>> the
>> - /// first element in the bundle and the insertion point for extracts.
>> This
>> - /// set must be reset between runs.
>> - struct UseInfo{
>> - UseInfo(Instruction *VL0, int I) :
>> - Leader(VL0), LastIndex(I) {}
>> - UseInfo() : Leader(0), LastIndex(0) {}
>> - /// The first element in the bundle.
>> - Instruction *Leader;
>> - /// The insertion index.
>> - int LastIndex;
>> - };
>> - MapVector<Instruction*, UseInfo> MultiUserVals;
>> - SetVector<Instruction*> ExtractedLane;
>> + /// A list of instructions to ignore while sinking
>> + /// memory instructions. This map must be reset between runs of
>> getCost.
>> + ValueSet MemBarrierIgnoreList;
>>
>> /// Holds all of the instructions that we gathered.
>> SetVector<Instruction *> GatherSeq;
>> @@ -281,409 +387,176 @@ public:
>> IRBuilder<> Builder;
>> };
>>
>> -int FuncSLP::getGatherCost(Type *Ty) {
>> - int Cost = 0;
>> - for (unsigned i = 0, e = cast<VectorType>(Ty)->getNumElements(); i <
>> e; ++i)
>> - Cost += TTI->getVectorInstrCost(Instruction::InsertElement, Ty, i);
>> - return Cost;
>> -}
>> -
>> -int FuncSLP::getGatherCost(ArrayRef<Value *> VL) {
>> - // Find the type of the operands in VL.
>> - Type *ScalarTy = VL[0]->getType();
>> - if (StoreInst *SI = dyn_cast<StoreInst>(VL[0]))
>> - ScalarTy = SI->getValueOperand()->getType();
>> - VectorType *VecTy = VectorType::get(ScalarTy, VL.size());
>> - // Find the cost of inserting/extracting values from the vector.
>> - return getGatherCost(VecTy);
>> -}
>> -
>> -AliasAnalysis::Location FuncSLP::getLocation(Instruction *I) {
>> - if (StoreInst *SI = dyn_cast<StoreInst>(I))
>> - return AA->getLocation(SI);
>> - if (LoadInst *LI = dyn_cast<LoadInst>(I))
>> - return AA->getLocation(LI);
>> - return AliasAnalysis::Location();
>> -}
>> -
>> -Value *FuncSLP::getPointerOperand(Value *I) {
>> - if (LoadInst *LI = dyn_cast<LoadInst>(I))
>> - return LI->getPointerOperand();
>> - if (StoreInst *SI = dyn_cast<StoreInst>(I))
>> - return SI->getPointerOperand();
>> - return 0;
>> +void BoUpSLP::buildTree(ArrayRef<Value *> Roots) {
>> + deleteTree();
>> + buildTree_rec(Roots, 0);
>> }
>>
>> -unsigned FuncSLP::getAddressSpaceOperand(Value *I) {
>> - if (LoadInst *L = dyn_cast<LoadInst>(I))
>> - return L->getPointerAddressSpace();
>> - if (StoreInst *S = dyn_cast<StoreInst>(I))
>> - return S->getPointerAddressSpace();
>> - return -1;
>> -}
>>
>> -bool FuncSLP::isConsecutiveAccess(Value *A, Value *B) {
>> - Value *PtrA = getPointerOperand(A);
>> - Value *PtrB = getPointerOperand(B);
>> - unsigned ASA = getAddressSpaceOperand(A);
>> - unsigned ASB = getAddressSpaceOperand(B);
>> +void BoUpSLP::buildTree_rec(ArrayRef<Value *> VL, unsigned Depth) {
>> + bool SameTy = getSameType(VL); (void)SameTy;
>> + assert(SameTy && "Invalid types!");
>>
>> - // Check that the address spaces match and that the pointers are valid.
>> - if (!PtrA || !PtrB || (ASA != ASB))
>> - return false;
>> + if (Depth == RecursionMaxDepth) {
>> + DEBUG(dbgs() << "SLP: Gathering due to max recursion depth.\n");
>> + newTreeEntry(VL, false);
>> + return;
>> + }
>>
>> - // Check that A and B are of the same type.
>> - if (PtrA->getType() != PtrB->getType())
>> - return false;
>> + // Don't handle vectors.
>> + if (VL[0]->getType()->isVectorTy()) {
>> + DEBUG(dbgs() << "SLP: Gathering due to vector type.\n");
>> + newTreeEntry(VL, false);
>> + return;
>> + }
>>
>> - // Calculate the distance.
>> - const SCEV *PtrSCEVA = SE->getSCEV(PtrA);
>> - const SCEV *PtrSCEVB = SE->getSCEV(PtrB);
>> - const SCEV *OffsetSCEV = SE->getMinusSCEV(PtrSCEVA, PtrSCEVB);
>> - const SCEVConstant *ConstOffSCEV = dyn_cast<SCEVConstant>(OffsetSCEV);
>> + if (StoreInst *SI = dyn_cast<StoreInst>(VL[0]))
>> + if (SI->getValueOperand()->getType()->isVectorTy()) {
>> + DEBUG(dbgs() << "SLP: Gathering due to store vector type.\n");
>> + newTreeEntry(VL, false);
>> + return;
>> + }
>>
>> - // Non constant distance.
>> - if (!ConstOffSCEV)
>> - return false;
>> + // If all of the operands are identical or constant we have a simple
>> solution.
>> + if (allConstant(VL) || isSplat(VL) || !getSameBlock(VL) ||
>> + !getSameOpcode(VL)) {
>> + DEBUG(dbgs() << "SLP: Gathering due to C,S,B,O. \n");
>> + newTreeEntry(VL, false);
>> + return;
>> + }
>>
>> - int64_t Offset = ConstOffSCEV->getValue()->getSExtValue();
>> - Type *Ty = cast<PointerType>(PtrA->getType())->getElementType();
>> - // The Instructions are connsecutive if the size of the first
>> load/store is
>> - // the same as the offset.
>> - int64_t Sz = DL->getTypeStoreSize(Ty);
>> - return ((-Offset) == Sz);
>> -}
>> + // We now know that this is a vector of instructions of the same type
>> from
>> + // the same block.
>>
>> -Value *FuncSLP::getSinkBarrier(Instruction *Src, Instruction *Dst) {
>> - assert(Src->getParent() == Dst->getParent() && "Not the same BB");
>> - BasicBlock::iterator I = Src, E = Dst;
>> - /// Scan all of the instruction from SRC to DST and check if
>> - /// the source may alias.
>> - for (++I; I != E; ++I) {
>> - // Ignore store instructions that are marked as 'ignore'.
>> - if (MemBarrierIgnoreList.count(I))
>> - continue;
>> - if (Src->mayWriteToMemory()) /* Write */ {
>> - if (!I->mayReadOrWriteMemory())
>> - continue;
>> - } else /* Read */ {
>> - if (!I->mayWriteToMemory())
>> - continue;
>> + // Check if this is a duplicate of another entry.
>> + if (ScalarToTreeEntry.count(VL[0])) {
>> + int Idx = ScalarToTreeEntry[VL[0]];
>> + TreeEntry *E = &VectorizableTree[Idx];
>> + for (unsigned i = 0, e = VL.size(); i != e; ++i) {
>> + DEBUG(dbgs() << "SLP: \tChecking bundle: " << *VL[i] << ".\n");
>> + if (E->Scalars[i] != VL[i]) {
>> + DEBUG(dbgs() << "SLP: Gathering due to partial overlap.\n");
>> + newTreeEntry(VL, false);
>> + return;
>> + }
>> }
>> - AliasAnalysis::Location A = getLocation(&*I);
>> - AliasAnalysis::Location B = getLocation(Src);
>> -
>> - if (!A.Ptr || !B.Ptr || AA->alias(A, B))
>> - return I;
>> + DEBUG(dbgs() << "SLP: Perfect diamond merge at " << *VL[0] << ".\n");
>> + return;
>> }
>> - return 0;
>> -}
>> -
>> -static BasicBlock *getSameBlock(ArrayRef<Value *> VL) {
>> - BasicBlock *BB = 0;
>> - for (int i = 0, e = VL.size(); i < e; i++) {
>> - Instruction *I = dyn_cast<Instruction>(VL[i]);
>> - if (!I)
>> - return 0;
>>
>> - if (!BB) {
>> - BB = I->getParent();
>> - continue;
>> + // Check that none of the instructions in the bundle are already in
>> the tree.
>> + for (unsigned i = 0, e = VL.size(); i != e; ++i) {
>> + if (ScalarToTreeEntry.count(VL[i])) {
>> + DEBUG(dbgs() << "SLP: The instruction (" << *VL[i] <<
>> + ") is already in tree.\n");
>> + newTreeEntry(VL, false);
>> + return;
>> }
>> + }
>>
>> - if (BB != I->getParent())
>> - return 0;
>> + // If any of the scalars appears in the table OR it is marked as a
>> value that
>> + // needs to stat scalar then we need to gather the scalars.
>> + for (unsigned i = 0, e = VL.size(); i != e; ++i) {
>> + if (ScalarToTreeEntry.count(VL[i]) || MustGather.count(VL[i])) {
>> + DEBUG(dbgs() << "SLP: Gathering due to gathered scalar. \n");
>> + newTreeEntry(VL, false);
>> + return;
>> + }
>> }
>> - return BB;
>> -}
>>
>> -static bool allConstant(ArrayRef<Value *> VL) {
>> - for (unsigned i = 0, e = VL.size(); i < e; ++i)
>> - if (!isa<Constant>(VL[i]))
>> - return false;
>> - return true;
>> -}
>> + // Check that all of the users of the scalars that we want to
>> vectorize are
>> + // schedulable.
>> + Instruction *VL0 = cast<Instruction>(VL[0]);
>> + int MyLastIndex = getLastIndex(VL);
>> + BasicBlock *BB = cast<Instruction>(VL0)->getParent();
>>
>> -static bool isSplat(ArrayRef<Value *> VL) {
>> - for (unsigned i = 1, e = VL.size(); i < e; ++i)
>> - if (VL[i] != VL[0])
>> - return false;
>> - return true;
>> -}
>> + for (unsigned i = 0, e = VL.size(); i != e; ++i) {
>> + Instruction *Scalar = cast<Instruction>(VL[i]);
>> + DEBUG(dbgs() << "SLP: Checking users of " << *Scalar << ". \n");
>> + for (Value::use_iterator U = Scalar->use_begin(), UE =
>> Scalar->use_end();
>> + U != UE; ++U) {
>> + DEBUG(dbgs() << "SLP: \tUser " << **U << ". \n");
>> + Instruction *User = dyn_cast<Instruction>(*U);
>> + if (!User) {
>> + DEBUG(dbgs() << "SLP: Gathering due unknown user. \n");
>> + newTreeEntry(VL, false);
>> + return;
>> + }
>>
>> -static unsigned getSameOpcode(ArrayRef<Value *> VL) {
>> - unsigned Opcode = 0;
>> - for (int i = 0, e = VL.size(); i < e; i++) {
>> - if (Instruction *I = dyn_cast<Instruction>(VL[i])) {
>> - if (!Opcode) {
>> - Opcode = I->getOpcode();
>> + // We don't care if the user is in a different basic block.
>> + BasicBlock *UserBlock = User->getParent();
>> + if (UserBlock != BB) {
>> + DEBUG(dbgs() << "SLP: User from a different basic block "
>> + << *User << ". \n");
>> continue;
>> }
>> - if (Opcode != I->getOpcode())
>> - return 0;
>> - }
>> - }
>> - return Opcode;
>> -}
>>
>> -static bool CanReuseExtract(ArrayRef<Value *> VL, unsigned VF,
>> - VectorType *VecTy) {
>> - assert(Instruction::ExtractElement == getSameOpcode(VL) && "Invalid
>> opcode");
>> - // Check if all of the extracts come from the same vector and from the
>> - // correct offset.
>> - Value *VL0 = VL[0];
>> - ExtractElementInst *E0 = cast<ExtractElementInst>(VL0);
>> - Value *Vec = E0->getOperand(0);
>> -
>> - // We have to extract from the same vector type.
>> - if (Vec->getType() != VecTy)
>> - return false;
>> -
>> - // Check that all of the indices extract from the correct offset.
>> - ConstantInt *CI = dyn_cast<ConstantInt>(E0->getOperand(1));
>> - if (!CI || CI->getZExtValue())
>> - return false;
>> -
>> - for (unsigned i = 1, e = VF; i < e; ++i) {
>> - ExtractElementInst *E = cast<ExtractElementInst>(VL[i]);
>> - ConstantInt *CI = dyn_cast<ConstantInt>(E->getOperand(1));
>> -
>> - if (!CI || CI->getZExtValue() != i || E->getOperand(0) != Vec)
>> - return false;
>> - }
>> -
>> - return true;
>> -}
>> -
>> -void FuncSLP::getTreeUses_rec(ArrayRef<Value *> VL, unsigned Depth) {
>> - if (Depth == RecursionMaxDepth)
>> - return MustGather.insert(VL.begin(), VL.end());
>> -
>> - // Don't handle vectors.
>> - if (VL[0]->getType()->isVectorTy())
>> - return;
>> -
>> - if (StoreInst *SI = dyn_cast<StoreInst>(VL[0]))
>> - if (SI->getValueOperand()->getType()->isVectorTy())
>> - return;
>> -
>> - // If all of the operands are identical or constant we have a simple
>> solution.
>> - if (allConstant(VL) || isSplat(VL) || !getSameBlock(VL))
>> - return MustGather.insert(VL.begin(), VL.end());
>> -
>> - // Stop the scan at unknown IR.
>> - Instruction *VL0 = dyn_cast<Instruction>(VL[0]);
>> - assert(VL0 && "Invalid instruction");
>> -
>> - // Mark instructions with multiple users.
>> - int LastIndex = getLastIndex(VL);
>> - for (unsigned i = 0, e = VL.size(); i < e; ++i) {
>> - if (PHINode *PN = dyn_cast<PHINode>(VL[i])) {
>> - unsigned NumUses = 0;
>> - // Check that PHINodes have only one external (non-self) use.
>> - for (Value::use_iterator U = VL[i]->use_begin(), UE =
>> VL[i]->use_end();
>> - U != UE; ++U) {
>> - // Don't count self uses.
>> - if (*U == PN)
>> - continue;
>> - NumUses++;
>> - }
>> - if (NumUses > 1) {
>> - DEBUG(dbgs() << "SLP: Adding PHI to MultiUserVals "
>> - "because it has " << NumUses << " users:" << *PN << " \n");
>> - UseInfo UI(VL0, 0);
>> - MultiUserVals[PN] = UI;
>> + // If this is a PHINode within this basic block then we can place
>> the
>> + // extract wherever we want.
>> + if (isa<PHINode>(*User)) {
>> + DEBUG(dbgs() << "SLP: \tWe can schedule PHIs:" << *User << ".
>> \n");
>> + continue;
>> }
>> - continue;
>> - }
>> -
>> - Instruction *I = dyn_cast<Instruction>(VL[i]);
>> - // Remember to check if all of the users of this instruction are
>> vectorized
>> - // within our tree. At depth zero we have no local users, only
>> external
>> - // users that we don't care about.
>> - if (Depth && I && I->getNumUses() > 1) {
>> - DEBUG(dbgs() << "SLP: Adding to MultiUserVals "
>> - "because it has " << I->getNumUses() << " users:" << *I << "
>> \n");
>> - UseInfo UI(VL0, LastIndex);
>> - MultiUserVals[I] = UI;
>> - }
>> - }
>> -
>> - // Check that the instruction is only used within one lane.
>> - for (int i = 0, e = VL.size(); i < e; ++i) {
>> - if (LaneMap.count(VL[i]) && LaneMap[VL[i]] != i) {
>> - DEBUG(dbgs() << "SLP: Value used by multiple lanes:" << *VL[i] <<
>> "\n");
>> - return MustGather.insert(VL.begin(), VL.end());
>> - }
>> - // Make this instruction as 'seen' and remember the lane.
>> - LaneMap[VL[i]] = i;
>> - }
>> -
>> - unsigned Opcode = getSameOpcode(VL);
>> - if (!Opcode)
>> - return MustGather.insert(VL.begin(), VL.end());
>>
>> - switch (Opcode) {
>> - case Instruction::PHI: {
>> - PHINode *PH = dyn_cast<PHINode>(VL0);
>> + // Check if this is a safe in-tree user.
>> + if (ScalarToTreeEntry.count(User)) {
>> + int Idx = ScalarToTreeEntry[User];
>> + int VecLocation = VectorizableTree[Idx].LastScalarIndex;
>> + if (VecLocation <= MyLastIndex) {
>> + DEBUG(dbgs() << "SLP: Gathering due to unschedulable vector.
>> \n");
>> + newTreeEntry(VL, false);
>> + return;
>> + }
>> + DEBUG(dbgs() << "SLP: In-tree user (" << *User << ") at #" <<
>> + VecLocation << " vector value (" << *Scalar << ") at #"
>> + << MyLastIndex << ".\n");
>> + continue;
>> + }
>>
>> - // Stop self cycles.
>> - if (VisitedPHIs.count(PH))
>> + // Make sure that we can schedule this unknown user.
>> + BlockNumbering &BN = BlocksNumbers[BB];
>> + int UserIndex = BN.getIndex(User);
>> + if (UserIndex < MyLastIndex) {
>> +
>> + DEBUG(dbgs() << "SLP: Can't schedule extractelement for "
>> + << *User << ". \n");
>> + newTreeEntry(VL, false);
>> return;
>> -
>> - VisitedPHIs.insert(PH);
>> - for (unsigned i = 0, e = PH->getNumIncomingValues(); i < e; ++i) {
>> - ValueList Operands;
>> - // Prepare the operand vector.
>> - for (unsigned j = 0; j < VL.size(); ++j)
>> - Operands.push_back(cast<PHINode>(VL[j])->getIncomingValue(i));
>> -
>> - getTreeUses_rec(Operands, Depth + 1);
>> - }
>> - return;
>> - }
>> - case Instruction::ExtractElement: {
>> - VectorType *VecTy = VectorType::get(VL[0]->getType(), VL.size());
>> - // No need to follow ExtractElements that are going to be optimized
>> away.
>> - if (CanReuseExtract(VL, VL.size(), VecTy))
>> - return;
>> - // Fall through.
>> - }
>> - case Instruction::Load:
>> - return;
>> - case Instruction::ZExt:
>> - case Instruction::SExt:
>> - case Instruction::FPToUI:
>> - case Instruction::FPToSI:
>> - case Instruction::FPExt:
>> - case Instruction::PtrToInt:
>> - case Instruction::IntToPtr:
>> - case Instruction::SIToFP:
>> - case Instruction::UIToFP:
>> - case Instruction::Trunc:
>> - case Instruction::FPTrunc:
>> - case Instruction::BitCast:
>> - case Instruction::Select:
>> - case Instruction::ICmp:
>> - case Instruction::FCmp:
>> - case Instruction::Add:
>> - case Instruction::FAdd:
>> - case Instruction::Sub:
>> - case Instruction::FSub:
>> - case Instruction::Mul:
>> - case Instruction::FMul:
>> - case Instruction::UDiv:
>> - case Instruction::SDiv:
>> - case Instruction::FDiv:
>> - case Instruction::URem:
>> - case Instruction::SRem:
>> - case Instruction::FRem:
>> - case Instruction::Shl:
>> - case Instruction::LShr:
>> - case Instruction::AShr:
>> - case Instruction::And:
>> - case Instruction::Or:
>> - case Instruction::Xor: {
>> - for (unsigned i = 0, e = VL0->getNumOperands(); i < e; ++i) {
>> - ValueList Operands;
>> - // Prepare the operand vector.
>> - for (unsigned j = 0; j < VL.size(); ++j)
>> - Operands.push_back(cast<Instruction>(VL[j])->getOperand(i));
>> -
>> - getTreeUses_rec(Operands, Depth + 1);
>> + }
>> }
>> - return;
>> - }
>> - case Instruction::Store: {
>> - ValueList Operands;
>> - for (unsigned j = 0; j < VL.size(); ++j)
>> - Operands.push_back(cast<Instruction>(VL[j])->getOperand(0));
>> - getTreeUses_rec(Operands, Depth + 1);
>> - return;
>> - }
>> - default:
>> - return MustGather.insert(VL.begin(), VL.end());
>> }
>> -}
>>
>> -int FuncSLP::getLastIndex(ArrayRef<Value *> VL) {
>> - BasicBlock *BB = cast<Instruction>(VL[0])->getParent();
>> - assert(BB == getSameBlock(VL) && BlocksNumbers.count(BB) && "Invalid
>> block");
>> - BlockNumbering &BN = BlocksNumbers[BB];
>> -
>> - int MaxIdx = BN.getIndex(BB->getFirstNonPHI());
>> + // Check that every instructions appears once in this bundle.
>> for (unsigned i = 0, e = VL.size(); i < e; ++i)
>> - MaxIdx = std::max(MaxIdx, BN.getIndex(cast<Instruction>(VL[i])));
>> - return MaxIdx;
>> -}
>> -
>> -Instruction *FuncSLP::getLastInstruction(ArrayRef<Value *> VL) {
>> - BasicBlock *BB = cast<Instruction>(VL[0])->getParent();
>> - assert(BB == getSameBlock(VL) && BlocksNumbers.count(BB) && "Invalid
>> block");
>> - BlockNumbering &BN = BlocksNumbers[BB];
>> -
>> - int MaxIdx = BN.getIndex(cast<Instruction>(VL[0]));
>> - for (unsigned i = 1, e = VL.size(); i < e; ++i)
>> - MaxIdx = std::max(MaxIdx, BN.getIndex(cast<Instruction>(VL[i])));
>> - return BN.getInstruction(MaxIdx);
>> -}
>> -
>> -Instruction *FuncSLP::getInstructionForIndex(unsigned Index, BasicBlock
>> *BB) {
>> - BlockNumbering &BN = BlocksNumbers[BB];
>> - return BN.getInstruction(Index);
>> -}
>> -
>> -int FuncSLP::getFirstUserIndex(ArrayRef<Value *> VL) {
>> - BasicBlock *BB = getSameBlock(VL);
>> - assert(BB && "All instructions must come from the same block");
>> - BlockNumbering &BN = BlocksNumbers[BB];
>> + for (unsigned j = i+1; j < e; ++j)
>> + if (VL[i] == VL[j]) {
>> + DEBUG(dbgs() << "SLP: Scalar used twice in bundle.\n");
>> + newTreeEntry(VL, false);
>> + return;
>> + }
>>
>> - // Find the first user of the values.
>> - int FirstUser = BN.getIndex(BB->getTerminator());
>> + // Check that instructions in this bundle don't reference other
>> instructions.
>> + // The runtime of this check is O(N * N-1 * uses(N)) and a typical N
>> is 4.
>> for (unsigned i = 0, e = VL.size(); i < e; ++i) {
>> for (Value::use_iterator U = VL[i]->use_begin(), UE =
>> VL[i]->use_end();
>> U != UE; ++U) {
>> - Instruction *Instr = dyn_cast<Instruction>(*U);
>> -
>> - if (!Instr || Instr->getParent() != BB)
>> - continue;
>> -
>> - FirstUser = std::min(FirstUser, BN.getIndex(Instr));
>> + for (unsigned j = 0; j < e; ++j) {
>> + if (i != j && *U == VL[j]) {
>> + DEBUG(dbgs() << "SLP: Intra-bundle dependencies!" << **U << ".
>> \n");
>> + newTreeEntry(VL, false);
>> + return;
>> + }
>> + }
>> }
>> }
>> - return FirstUser;
>> -}
>> -
>> -int FuncSLP::getTreeCost_rec(ArrayRef<Value *> VL, unsigned Depth) {
>> - Type *ScalarTy = VL[0]->getType();
>> -
>> - if (StoreInst *SI = dyn_cast<StoreInst>(VL[0]))
>> - ScalarTy = SI->getValueOperand()->getType();
>> -
>> - /// Don't mess with vectors.
>> - if (ScalarTy->isVectorTy())
>> - return FuncSLP::MAX_COST;
>>
>> - if (allConstant(VL))
>> - return 0;
>> -
>> - VectorType *VecTy = VectorType::get(ScalarTy, VL.size());
>> -
>> - if (isSplat(VL))
>> - return TTI->getShuffleCost(TargetTransformInfo::SK_Broadcast, VecTy,
>> 0);
>> -
>> - int GatherCost = getGatherCost(VecTy);
>> - if (Depth == RecursionMaxDepth || needToGatherAny(VL))
>> - return GatherCost;
>> + DEBUG(dbgs() << "SLP: We are able to schedule this bundle.\n");
>>
>> - BasicBlock *BB = getSameBlock(VL);
>> unsigned Opcode = getSameOpcode(VL);
>> - assert(Opcode && BB && "Invalid Instruction Value");
>>
>> // Check if it is safe to sink the loads or the stores.
>> if (Opcode == Instruction::Load || Opcode == Instruction::Store) {
>> - int MaxIdx = getLastIndex(VL);
>> - Instruction *Last = getInstructionForIndex(MaxIdx, BB);
>> + Instruction *Last = getLastInstruction(VL);
>>
>> for (unsigned i = 0, e = VL.size(); i < e; ++i) {
>> if (VL[i] == Last)
>> @@ -691,407 +564,442 @@ int FuncSLP::getTreeCost_rec(ArrayRef<Va
>> Value *Barrier = getSinkBarrier(cast<Instruction>(VL[i]), Last);
>> if (Barrier) {
>> DEBUG(dbgs() << "SLP: Can't sink " << *VL[i] << "\n down to " <<
>> *Last
>> - << "\n because of " << *Barrier << "\n");
>> - return MAX_COST;
>> + << "\n because of " << *Barrier << ". Gathering.\n");
>> + newTreeEntry(VL, false);
>> + return;
>> }
>> }
>> }
>>
>> - // Calculate the extract cost.
>> - unsigned ExternalUserExtractCost = 0;
>> - for (unsigned i = 0, e = VL.size(); i < e; ++i)
>> - if (ExtractedLane.count(cast<Instruction>(VL[i])))
>> - ExternalUserExtractCost +=
>> - TTI->getVectorInstrCost(Instruction::ExtractElement, VecTy, i);
>> -
>> - Instruction *VL0 = cast<Instruction>(VL[0]);
>> switch (Opcode) {
>> - case Instruction::PHI: {
>> - PHINode *PH = dyn_cast<PHINode>(VL0);
>> -
>> - // Stop self cycles.
>> - if (VisitedPHIs.count(PH))
>> - return 0;
>> -
>> - VisitedPHIs.insert(PH);
>> - int TotalCost = 0;
>> - // Calculate the cost of all of the operands.
>> - for (unsigned i = 0, e = PH->getNumIncomingValues(); i < e; ++i) {
>> - ValueList Operands;
>> - // Prepare the operand vector.
>> - for (unsigned j = 0; j < VL.size(); ++j)
>> - Operands.push_back(cast<PHINode>(VL[j])->getIncomingValue(i));
>> + case Instruction::PHI: {
>> + PHINode *PH = dyn_cast<PHINode>(VL0);
>> + newTreeEntry(VL, true);
>> + DEBUG(dbgs() << "SLP: added a vector of PHINodes.\n");
>> +
>> + for (unsigned i = 0, e = PH->getNumIncomingValues(); i < e; ++i) {
>> + ValueList Operands;
>> + // Prepare the operand vector.
>> + for (unsigned j = 0; j < VL.size(); ++j)
>> + Operands.push_back(cast<PHINode>(VL[j])->getIncomingValue(i));
>>
>> - int Cost = getTreeCost_rec(Operands, Depth + 1);
>> - if (Cost == MAX_COST)
>> - return MAX_COST;
>> - TotalCost += TotalCost;
>> + buildTree_rec(Operands, Depth + 1);
>> + }
>> + return;
>> }
>> + case Instruction::ExtractElement: {
>> + bool Reuse = CanReuseExtract(VL);
>> + if (Reuse) {
>> + DEBUG(dbgs() << "SLP: Reusing extract sequence.\n");
>> + }
>> + newTreeEntry(VL, Reuse);
>> + return;
>> + }
>> + case Instruction::Load: {
>> + // Check if the loads are consecutive or of we need to swizzle
>> them.
>> + for (unsigned i = 0, e = VL.size() - 1; i < e; ++i)
>> + if (!isConsecutiveAccess(VL[i], VL[i + 1])) {
>> + newTreeEntry(VL, false);
>> + DEBUG(dbgs() << "SLP: Need to swizzle loads.\n");
>> + return;
>> + }
>>
>> - if (TotalCost > GatherCost) {
>> - MustGather.insert(VL.begin(), VL.end());
>> - return GatherCost;
>> + newTreeEntry(VL, true);
>> + DEBUG(dbgs() << "SLP: added a vector of loads.\n");
>> + return;
>> }
>> + case Instruction::ZExt:
>> + case Instruction::SExt:
>> + case Instruction::FPToUI:
>> + case Instruction::FPToSI:
>> + case Instruction::FPExt:
>> + case Instruction::PtrToInt:
>> + case Instruction::IntToPtr:
>> + case Instruction::SIToFP:
>> + case Instruction::UIToFP:
>> + case Instruction::Trunc:
>> + case Instruction::FPTrunc:
>> + case Instruction::BitCast: {
>> + Type *SrcTy = VL0->getOperand(0)->getType();
>> + for (unsigned i = 0; i < VL.size(); ++i) {
>> + Type *Ty = cast<Instruction>(VL[i])->getOperand(0)->getType();
>> + if (Ty != SrcTy || Ty->isAggregateType() || Ty->isVectorTy()) {
>> + newTreeEntry(VL, false);
>> + DEBUG(dbgs() << "SLP: Gathering casts with different src
>> types.\n");
>> + return;
>> + }
>> + }
>> + newTreeEntry(VL, true);
>> + DEBUG(dbgs() << "SLP: added a vector of casts.\n");
>>
>> - return TotalCost + ExternalUserExtractCost;
>> - }
>> - case Instruction::ExtractElement: {
>> - if (CanReuseExtract(VL, VL.size(), VecTy))
>> - return 0;
>> - return getGatherCost(VecTy);
>> - }
>> - case Instruction::ZExt:
>> - case Instruction::SExt:
>> - case Instruction::FPToUI:
>> - case Instruction::FPToSI:
>> - case Instruction::FPExt:
>> - case Instruction::PtrToInt:
>> - case Instruction::IntToPtr:
>> - case Instruction::SIToFP:
>> - case Instruction::UIToFP:
>> - case Instruction::Trunc:
>> - case Instruction::FPTrunc:
>> - case Instruction::BitCast: {
>> - ValueList Operands;
>> - Type *SrcTy = VL0->getOperand(0)->getType();
>> - // Prepare the operand vector.
>> - for (unsigned j = 0; j < VL.size(); ++j) {
>> - Operands.push_back(cast<Instruction>(VL[j])->getOperand(0));
>> - // Check that the casted type is the same for all users.
>> - if (cast<Instruction>(VL[j])->getOperand(0)->getType() != SrcTy)
>> - return getGatherCost(VecTy);
>> - }
>> -
>> - int Cost = getTreeCost_rec(Operands, Depth + 1);
>> - if (Cost == MAX_COST)
>> - return MAX_COST;
>> -
>> - // Calculate the cost of this instruction.
>> - int ScalarCost = VL.size() * TTI->getCastInstrCost(VL0->getOpcode(),
>> - VL0->getType(),
>> SrcTy);
>> -
>> - VectorType *SrcVecTy = VectorType::get(SrcTy, VL.size());
>> - int VecCost = TTI->getCastInstrCost(VL0->getOpcode(), VecTy,
>> SrcVecTy);
>> - Cost += (VecCost - ScalarCost);
>> + for (unsigned i = 0, e = VL0->getNumOperands(); i < e; ++i) {
>> + ValueList Operands;
>> + // Prepare the operand vector.
>> + for (unsigned j = 0; j < VL.size(); ++j)
>> + Operands.push_back(cast<Instruction>(VL[j])->getOperand(i));
>>
>> - if (Cost > GatherCost) {
>> - MustGather.insert(VL.begin(), VL.end());
>> - return GatherCost;
>> + buildTree_rec(Operands, Depth+1);
>> + }
>> + return;
>> }
>> + case Instruction::ICmp:
>> + case Instruction::FCmp: {
>> + // Check that all of the compares have the same predicate.
>> + CmpInst::Predicate P0 = dyn_cast<CmpInst>(VL0)->getPredicate();
>> + for (unsigned i = 1, e = VL.size(); i < e; ++i) {
>> + CmpInst *Cmp = cast<CmpInst>(VL[i]);
>> + if (Cmp->getPredicate() != P0) {
>> + newTreeEntry(VL, false);
>> + DEBUG(dbgs() << "SLP: Gathering cmp with different
>> predicate.\n");
>> + return;
>> + }
>> + }
>>
>> - return Cost + ExternalUserExtractCost;
>> - }
>> - case Instruction::FCmp:
>> - case Instruction::ICmp: {
>> - // Check that all of the compares have the same predicate.
>> - CmpInst::Predicate P0 = dyn_cast<CmpInst>(VL0)->getPredicate();
>> - for (unsigned i = 1, e = VL.size(); i < e; ++i) {
>> - CmpInst *Cmp = cast<CmpInst>(VL[i]);
>> - if (Cmp->getPredicate() != P0)
>> - return getGatherCost(VecTy);
>> - }
>> - // Fall through.
>> - }
>> - case Instruction::Select:
>> - case Instruction::Add:
>> - case Instruction::FAdd:
>> - case Instruction::Sub:
>> - case Instruction::FSub:
>> - case Instruction::Mul:
>> - case Instruction::FMul:
>> - case Instruction::UDiv:
>> - case Instruction::SDiv:
>> - case Instruction::FDiv:
>> - case Instruction::URem:
>> - case Instruction::SRem:
>> - case Instruction::FRem:
>> - case Instruction::Shl:
>> - case Instruction::LShr:
>> - case Instruction::AShr:
>> - case Instruction::And:
>> - case Instruction::Or:
>> - case Instruction::Xor: {
>> - int TotalCost = 0;
>> - // Calculate the cost of all of the operands.
>> - for (unsigned i = 0, e = VL0->getNumOperands(); i < e; ++i) {
>> - ValueList Operands;
>> - // Prepare the operand vector.
>> - for (unsigned j = 0; j < VL.size(); ++j)
>> - Operands.push_back(cast<Instruction>(VL[j])->getOperand(i));
>> + newTreeEntry(VL, true);
>> + DEBUG(dbgs() << "SLP: added a vector of compares.\n");
>>
>> - int Cost = getTreeCost_rec(Operands, Depth + 1);
>> - if (Cost == MAX_COST)
>> - return MAX_COST;
>> - TotalCost += Cost;
>> - }
>> -
>> - // Calculate the cost of this instruction.
>> - int ScalarCost = 0;
>> - int VecCost = 0;
>> - if (Opcode == Instruction::FCmp || Opcode == Instruction::ICmp ||
>> - Opcode == Instruction::Select) {
>> - VectorType *MaskTy = VectorType::get(Builder.getInt1Ty(),
>> VL.size());
>> - ScalarCost =
>> - VecTy->getNumElements() *
>> - TTI->getCmpSelInstrCost(Opcode, ScalarTy, Builder.getInt1Ty());
>> - VecCost = TTI->getCmpSelInstrCost(Opcode, VecTy, MaskTy);
>> - } else {
>> - ScalarCost = VecTy->getNumElements() *
>> - TTI->getArithmeticInstrCost(Opcode, ScalarTy);
>> - VecCost = TTI->getArithmeticInstrCost(Opcode, VecTy);
>> + for (unsigned i = 0, e = VL0->getNumOperands(); i < e; ++i) {
>> + ValueList Operands;
>> + // Prepare the operand vector.
>> + for (unsigned j = 0; j < VL.size(); ++j)
>> + Operands.push_back(cast<Instruction>(VL[j])->getOperand(i));
>> +
>> + buildTree_rec(Operands, Depth+1);
>> + }
>> + return;
>> }
>> - TotalCost += (VecCost - ScalarCost);
>> + case Instruction::Select:
>> + case Instruction::Add:
>> + case Instruction::FAdd:
>> + case Instruction::Sub:
>> + case Instruction::FSub:
>> + case Instruction::Mul:
>> + case Instruction::FMul:
>> + case Instruction::UDiv:
>> + case Instruction::SDiv:
>> + case Instruction::FDiv:
>> + case Instruction::URem:
>> + case Instruction::SRem:
>> + case Instruction::FRem:
>> + case Instruction::Shl:
>> + case Instruction::LShr:
>> + case Instruction::AShr:
>> + case Instruction::And:
>> + case Instruction::Or:
>> + case Instruction::Xor: {
>> + newTreeEntry(VL, true);
>> + DEBUG(dbgs() << "SLP: added a vector of bin op.\n");
>> +
>> + for (unsigned i = 0, e = VL0->getNumOperands(); i < e; ++i) {
>> + ValueList Operands;
>> + // Prepare the operand vector.
>> + for (unsigned j = 0; j < VL.size(); ++j)
>> + Operands.push_back(cast<Instruction>(VL[j])->getOperand(i));
>>
>> - if (TotalCost > GatherCost) {
>> - MustGather.insert(VL.begin(), VL.end());
>> - return GatherCost;
>> + buildTree_rec(Operands, Depth+1);
>> + }
>> + return;
>> }
>> + case Instruction::Store: {
>> + // Check if the stores are consecutive or of we need to swizzle
>> them.
>> + for (unsigned i = 0, e = VL.size() - 1; i < e; ++i)
>> + if (!isConsecutiveAccess(VL[i], VL[i + 1])) {
>> + newTreeEntry(VL, false);
>> + DEBUG(dbgs() << "SLP: Non consecutive store.\n");
>> + return;
>> + }
>>
>> - return TotalCost + ExternalUserExtractCost;
>> - }
>> - case Instruction::Load: {
>> - // If we are scalarize the loads, add the cost of forming the vector.
>> - for (unsigned i = 0, e = VL.size() - 1; i < e; ++i)
>> - if (!isConsecutiveAccess(VL[i], VL[i + 1]))
>> - return getGatherCost(VecTy);
>> + newTreeEntry(VL, true);
>> + DEBUG(dbgs() << "SLP: added a vector of stores.\n");
>>
>> - // Cost of wide load - cost of scalar loads.
>> - int ScalarLdCost = VecTy->getNumElements() *
>> - TTI->getMemoryOpCost(Instruction::Load, ScalarTy,
>> 1, 0);
>> - int VecLdCost = TTI->getMemoryOpCost(Instruction::Load, ScalarTy, 1,
>> 0);
>> - int TotalCost = VecLdCost - ScalarLdCost;
>> + ValueList Operands;
>> + for (unsigned j = 0; j < VL.size(); ++j)
>> + Operands.push_back(cast<Instruction>(VL[j])->getOperand(0));
>>
>> - if (TotalCost > GatherCost) {
>> - MustGather.insert(VL.begin(), VL.end());
>> - return GatherCost;
>> + // We can ignore these values because we are sinking them down.
>> + MemBarrierIgnoreList.insert(VL.begin(), VL.end());
>> + buildTree_rec(Operands, Depth + 1);
>> + return;
>> }
>> -
>> - return TotalCost + ExternalUserExtractCost;
>> + default:
>> + newTreeEntry(VL, false);
>> + DEBUG(dbgs() << "SLP: Gathering unknown instruction.\n");
>> + return;
>> }
>> - case Instruction::Store: {
>> - // We know that we can merge the stores. Calculate the cost.
>> - int ScalarStCost = VecTy->getNumElements() *
>> - TTI->getMemoryOpCost(Instruction::Store,
>> ScalarTy, 1, 0);
>> - int VecStCost = TTI->getMemoryOpCost(Instruction::Store, ScalarTy,
>> 1, 0);
>> - int StoreCost = VecStCost - ScalarStCost;
>> +}
>>
>> - ValueList Operands;
>> - for (unsigned j = 0; j < VL.size(); ++j) {
>> - Operands.push_back(cast<Instruction>(VL[j])->getOperand(0));
>> - MemBarrierIgnoreList.insert(VL[j]);
>> - }
>> +int BoUpSLP::getEntryCost(TreeEntry *E) {
>> + ArrayRef<Value*> VL = E->Scalars;
>>
>> - int Cost = getTreeCost_rec(Operands, Depth + 1);
>> - if (Cost == MAX_COST)
>> - return MAX_COST;
>> + Type *ScalarTy = VL[0]->getType();
>> + if (StoreInst *SI = dyn_cast<StoreInst>(VL[0]))
>> + ScalarTy = SI->getValueOperand()->getType();
>> + VectorType *VecTy = VectorType::get(ScalarTy, VL.size());
>>
>> - int TotalCost = StoreCost + Cost;
>> - return TotalCost + ExternalUserExtractCost;
>> + if (E->NeedToGather) {
>> + if (allConstant(VL))
>> + return 0;
>> + if (isSplat(VL)) {
>> + return TTI->getShuffleCost(TargetTransformInfo::SK_Broadcast,
>> VecTy, 0);
>> + }
>> + return getGatherCost(E->Scalars);
>> }
>> - default:
>> - // Unable to vectorize unknown instructions.
>> - return getGatherCost(VecTy);
>> +
>> + assert(getSameOpcode(VL) && getSameType(VL) && getSameBlock(VL) &&
>> + "Invalid VL");
>> + Instruction *VL0 = cast<Instruction>(VL[0]);
>> + unsigned Opcode = VL0->getOpcode();
>> + switch (Opcode) {
>> + case Instruction::PHI: {
>> + return 0;
>> + }
>> + case Instruction::ExtractElement: {
>> + if (CanReuseExtract(VL))
>> + return 0;
>> + return getGatherCost(VecTy);
>> + }
>> + case Instruction::ZExt:
>> + case Instruction::SExt:
>> + case Instruction::FPToUI:
>> + case Instruction::FPToSI:
>> + case Instruction::FPExt:
>> + case Instruction::PtrToInt:
>> + case Instruction::IntToPtr:
>> + case Instruction::SIToFP:
>> + case Instruction::UIToFP:
>> + case Instruction::Trunc:
>> + case Instruction::FPTrunc:
>> + case Instruction::BitCast: {
>> + Type *SrcTy = VL0->getOperand(0)->getType();
>> +
>> + // Calculate the cost of this instruction.
>> + int ScalarCost = VL.size() *
>> TTI->getCastInstrCost(VL0->getOpcode(),
>> + VL0->getType(),
>> SrcTy);
>> +
>> + VectorType *SrcVecTy = VectorType::get(SrcTy, VL.size());
>> + int VecCost = TTI->getCastInstrCost(VL0->getOpcode(), VecTy,
>> SrcVecTy);
>> + return VecCost - ScalarCost;
>> + }
>> + case Instruction::FCmp:
>> + case Instruction::ICmp:
>> + case Instruction::Select:
>> + case Instruction::Add:
>> + case Instruction::FAdd:
>> + case Instruction::Sub:
>> + case Instruction::FSub:
>> + case Instruction::Mul:
>> + case Instruction::FMul:
>> + case Instruction::UDiv:
>> + case Instruction::SDiv:
>> + case Instruction::FDiv:
>> + case Instruction::URem:
>> + case Instruction::SRem:
>> + case Instruction::FRem:
>> + case Instruction::Shl:
>> + case Instruction::LShr:
>> + case Instruction::AShr:
>> + case Instruction::And:
>> + case Instruction::Or:
>> + case Instruction::Xor: {
>> + // Calculate the cost of this instruction.
>> + int ScalarCost = 0;
>> + int VecCost = 0;
>> + if (Opcode == Instruction::FCmp || Opcode == Instruction::ICmp ||
>> + Opcode == Instruction::Select) {
>> + VectorType *MaskTy = VectorType::get(Builder.getInt1Ty(),
>> VL.size());
>> + ScalarCost = VecTy->getNumElements() *
>> + TTI->getCmpSelInstrCost(Opcode, ScalarTy, Builder.getInt1Ty());
>> + VecCost = TTI->getCmpSelInstrCost(Opcode, VecTy, MaskTy);
>> + } else {
>> + ScalarCost = VecTy->getNumElements() *
>> + TTI->getArithmeticInstrCost(Opcode, ScalarTy);
>> + VecCost = TTI->getArithmeticInstrCost(Opcode, VecTy);
>> + }
>> + return VecCost - ScalarCost;
>> + }
>> + case Instruction::Load: {
>> + // Cost of wide load - cost of scalar loads.
>> + int ScalarLdCost = VecTy->getNumElements() *
>> + TTI->getMemoryOpCost(Instruction::Load, ScalarTy, 1, 0);
>> + int VecLdCost = TTI->getMemoryOpCost(Instruction::Load, ScalarTy,
>> 1, 0);
>> + return VecLdCost - ScalarLdCost;
>> + }
>> + case Instruction::Store: {
>> + // We know that we can merge the stores. Calculate the cost.
>> + int ScalarStCost = VecTy->getNumElements() *
>> + TTI->getMemoryOpCost(Instruction::Store, ScalarTy, 1, 0);
>> + int VecStCost = TTI->getMemoryOpCost(Instruction::Store, ScalarTy,
>> 1, 0);
>> + return VecStCost - ScalarStCost;
>> + }
>> + default:
>> + llvm_unreachable("Unknown instruction");
>> }
>> }
>>
>> -int FuncSLP::getTreeCost(ArrayRef<Value *> VL) {
>> - // Get rid of the list of stores that were removed, and from the
>> - // lists of instructions with multiple users.
>> - MemBarrierIgnoreList.clear();
>> - LaneMap.clear();
>> - MultiUserVals.clear();
>> - ExtractedLane.clear();
>> - MustGather.clear();
>> - VisitedPHIs.clear();
>> -
>> - if (!getSameBlock(VL))
>> - return MAX_COST;
>> -
>> - // Find the location of the last root.
>> - int LastRootIndex = getLastIndex(VL);
>> - int FirstUserIndex = getFirstUserIndex(VL);
>> -
>> - // Don't vectorize if there are users of the tree roots inside the tree
>> - // itself.
>> - if (LastRootIndex > FirstUserIndex)
>> - return MAX_COST;
>> -
>> - // Scan the tree and find which value is used by which lane, and which
>> values
>> - // must be scalarized.
>> - getTreeUses_rec(VL, 0);
>> -
>> - // Check that instructions with multiple users can be vectorized. Mark
>> - // unsafe instructions.
>> - for (MapVector<Instruction *, UseInfo>::iterator UI =
>> MultiUserVals.begin(),
>> - e = MultiUserVals.end(); UI != e; ++UI) {
>> - Instruction *Scalar = UI->first;
>> -
>> - if (MustGather.count(Scalar))
>> - continue;
>> +int BoUpSLP::getTreeCost() {
>> + int Cost = 0;
>> + DEBUG(dbgs() << "SLP: Calculating cost for tree of size " <<
>> + VectorizableTree.size() << ".\n");
>>
>> - assert(LaneMap.count(Scalar) && "Unknown scalar");
>> - int ScalarLane = LaneMap[Scalar];
>> + for (unsigned i = 0, e = VectorizableTree.size(); i != e; ++i) {
>> + int C = getEntryCost(&VectorizableTree[i]);
>> + DEBUG(dbgs() << "SLP: Adding cost " << C << " for bundle that starts
>> with "
>> + << *VectorizableTree[i].Scalars[0] << " .\n");
>> + Cost += C;
>> + }
>> + DEBUG(dbgs() << "SLP: Total Cost " << Cost << ".\n");
>> + return Cost;
>> +}
>>
>> - bool ExternalUse = false;
>> - // Check that all of the users of this instr are within the tree.
>> - for (Value::use_iterator Usr = Scalar->use_begin(),
>> - UE = Scalar->use_end(); Usr != UE; ++Usr) {
>> - // If this user is within the tree, make sure it is from the same
>> lane.
>> - // Notice that we have both in-tree and out-of-tree users.
>> - if (LaneMap.count(*Usr)) {
>> - if (LaneMap[*Usr] != ScalarLane) {
>> - DEBUG(dbgs() << "SLP: Adding to MustExtract "
>> - "because of an out-of-lane usage.\n");
>> - MustGather.insert(Scalar);
>> - break;
>> - }
>> - continue;
>> - }
>> +int BoUpSLP::getGatherCost(Type *Ty) {
>> + int Cost = 0;
>> + for (unsigned i = 0, e = cast<VectorType>(Ty)->getNumElements(); i <
>> e; ++i)
>> + Cost += TTI->getVectorInstrCost(Instruction::InsertElement, Ty, i);
>> + return Cost;
>> +}
>>
>> - // We have an out-of-tree user. Check if we can place an 'extract'.
>> - Instruction *User = cast<Instruction>(*Usr);
>> - // We care about the order only if the user is in the same block.
>> - if (User->getParent() == Scalar->getParent()) {
>> - int LastLoc = UI->second.LastIndex;
>> - BlockNumbering &BN = BlocksNumbers[User->getParent()];
>> - int UserIdx = BN.getIndex(User);
>> - if (UserIdx <= LastLoc) {
>> - DEBUG(dbgs() << "SLP: Adding to MustExtract because of an
>> external "
>> - "user that we can't schedule.\n");
>> - MustGather.insert(Scalar);
>> - break;
>> - }
>> - }
>> - // We have an external user.
>> - ExternalUse = true;
>> - }
>> +int BoUpSLP::getGatherCost(ArrayRef<Value *> VL) {
>> + // Find the type of the operands in VL.
>> + Type *ScalarTy = VL[0]->getType();
>> + if (StoreInst *SI = dyn_cast<StoreInst>(VL[0]))
>> + ScalarTy = SI->getValueOperand()->getType();
>> + VectorType *VecTy = VectorType::get(ScalarTy, VL.size());
>> + // Find the cost of inserting/extracting values from the vector.
>> + return getGatherCost(VecTy);
>> +}
>>
>> - if (ExternalUse) {
>> - // Items that are left in MultiUserVals are to be extracted.
>> - // ExtractLane is used for the lookup.
>> - ExtractedLane.insert(Scalar);
>> - }
>> +AliasAnalysis::Location BoUpSLP::getLocation(Instruction *I) {
>> + if (StoreInst *SI = dyn_cast<StoreInst>(I))
>> + return AA->getLocation(SI);
>> + if (LoadInst *LI = dyn_cast<LoadInst>(I))
>> + return AA->getLocation(LI);
>> + return AliasAnalysis::Location();
>> +}
>>
>> - }
>> +Value *BoUpSLP::getPointerOperand(Value *I) {
>> + if (LoadInst *LI = dyn_cast<LoadInst>(I))
>> + return LI->getPointerOperand();
>> + if (StoreInst *SI = dyn_cast<StoreInst>(I))
>> + return SI->getPointerOperand();
>> + return 0;
>> +}
>>
>> - // Now calculate the cost of vectorizing the tree.
>> - return getTreeCost_rec(VL, 0);
>> +unsigned BoUpSLP::getAddressSpaceOperand(Value *I) {
>> + if (LoadInst *L = dyn_cast<LoadInst>(I))
>> + return L->getPointerAddressSpace();
>> + if (StoreInst *S = dyn_cast<StoreInst>(I))
>> + return S->getPointerAddressSpace();
>> + return -1;
>> }
>> -bool FuncSLP::vectorizeStoreChain(ArrayRef<Value *> Chain, int
>> CostThreshold) {
>> - unsigned ChainLen = Chain.size();
>> - DEBUG(dbgs() << "SLP: Analyzing a store chain of length " << ChainLen
>> - << "\n");
>> - Type *StoreTy =
>> cast<StoreInst>(Chain[0])->getValueOperand()->getType();
>> - unsigned Sz = DL->getTypeSizeInBits(StoreTy);
>> - unsigned VF = MinVecRegSize / Sz;
>>
>> - if (!isPowerOf2_32(Sz) || VF < 2)
>> +bool BoUpSLP::isConsecutiveAccess(Value *A, Value *B) {
>> + Value *PtrA = getPointerOperand(A);
>> + Value *PtrB = getPointerOperand(B);
>> + unsigned ASA = getAddressSpaceOperand(A);
>> + unsigned ASB = getAddressSpaceOperand(B);
>> +
>> + // Check that the address spaces match and that the pointers are valid.
>> + if (!PtrA || !PtrB || (ASA != ASB))
>> return false;
>>
>> - bool Changed = false;
>> - // Look for profitable vectorizable trees at all offsets, starting at
>> zero.
>> - for (unsigned i = 0, e = ChainLen; i < e; ++i) {
>> - if (i + VF > e)
>> - break;
>> - DEBUG(dbgs() << "SLP: Analyzing " << VF << " stores at offset " << i
>> - << "\n");
>> - ArrayRef<Value *> Operands = Chain.slice(i, VF);
>> + // Check that A and B are of the same type.
>> + if (PtrA->getType() != PtrB->getType())
>> + return false;
>>
>> - int Cost = getTreeCost(Operands);
>> - if (Cost == FuncSLP::MAX_COST)
>> - continue;
>> - DEBUG(dbgs() << "SLP: Found cost=" << Cost << " for VF=" << VF <<
>> "\n");
>> - if (Cost < CostThreshold) {
>> - DEBUG(dbgs() << "SLP: Decided to vectorize cost=" << Cost << "\n");
>> - vectorizeTree(Operands);
>> + // Calculate the distance.
>> + const SCEV *PtrSCEVA = SE->getSCEV(PtrA);
>> + const SCEV *PtrSCEVB = SE->getSCEV(PtrB);
>> + const SCEV *OffsetSCEV = SE->getMinusSCEV(PtrSCEVA, PtrSCEVB);
>> + const SCEVConstant *ConstOffSCEV = dyn_cast<SCEVConstant>(OffsetSCEV);
>>
>> - // Remove the scalar stores.
>> - for (int j = 0, e = VF; j < e; ++j)
>> - cast<Instruction>(Operands[j])->eraseFromParent();
>> + // Non constant distance.
>> + if (!ConstOffSCEV)
>> + return false;
>>
>> - // Move to the next bundle.
>> - i += VF - 1;
>> - Changed = true;
>> + int64_t Offset = ConstOffSCEV->getValue()->getSExtValue();
>> + Type *Ty = cast<PointerType>(PtrA->getType())->getElementType();
>> + // The Instructions are connsecutive if the size of the first
>> load/store is
>> + // the same as the offset.
>> + int64_t Sz = DL->getTypeStoreSize(Ty);
>> + return ((-Offset) == Sz);
>> +}
>> +
>> +Value *BoUpSLP::getSinkBarrier(Instruction *Src, Instruction *Dst) {
>> + assert(Src->getParent() == Dst->getParent() && "Not the same BB");
>> + BasicBlock::iterator I = Src, E = Dst;
>> + /// Scan all of the instruction from SRC to DST and check if
>> + /// the source may alias.
>> + for (++I; I != E; ++I) {
>> + // Ignore store instructions that are marked as 'ignore'.
>> + if (MemBarrierIgnoreList.count(I))
>> + continue;
>> + if (Src->mayWriteToMemory()) /* Write */ {
>> + if (!I->mayReadOrWriteMemory())
>> + continue;
>> + } else /* Read */ {
>> + if (!I->mayWriteToMemory())
>> + continue;
>> }
>> + AliasAnalysis::Location A = getLocation(&*I);
>> + AliasAnalysis::Location B = getLocation(Src);
>> +
>> + if (!A.Ptr || !B.Ptr || AA->alias(A, B))
>> + return I;
>> }
>> + return 0;
>> +}
>>
>> - if (Changed || ChainLen > VF)
>> - return Changed;
>> +int BoUpSLP::getLastIndex(ArrayRef<Value *> VL) {
>> + BasicBlock *BB = cast<Instruction>(VL[0])->getParent();
>> + assert(BB == getSameBlock(VL) && BlocksNumbers.count(BB) && "Invalid
>> block");
>> + BlockNumbering &BN = BlocksNumbers[BB];
>>
>> - // Handle short chains. This helps us catch types such as <3 x float>
>> that
>> - // are smaller than vector size.
>> - int Cost = getTreeCost(Chain);
>> - if (Cost == FuncSLP::MAX_COST)
>> - return false;
>> - if (Cost < CostThreshold) {
>> - DEBUG(dbgs() << "SLP: Found store chain cost = " << Cost
>> - << " for size = " << ChainLen << "\n");
>> - vectorizeTree(Chain);
>> + int MaxIdx = BN.getIndex(BB->getFirstNonPHI());
>> + for (unsigned i = 0, e = VL.size(); i < e; ++i)
>> + MaxIdx = std::max(MaxIdx, BN.getIndex(cast<Instruction>(VL[i])));
>> + return MaxIdx;
>> +}
>>
>> - // Remove all of the scalar stores.
>> - for (int i = 0, e = Chain.size(); i < e; ++i)
>> - cast<Instruction>(Chain[i])->eraseFromParent();
>> +Instruction *BoUpSLP::getLastInstruction(ArrayRef<Value *> VL) {
>> + BasicBlock *BB = cast<Instruction>(VL[0])->getParent();
>> + assert(BB == getSameBlock(VL) && BlocksNumbers.count(BB) && "Invalid
>> block");
>> + BlockNumbering &BN = BlocksNumbers[BB];
>>
>> - return true;
>> - }
>> + int MaxIdx = BN.getIndex(cast<Instruction>(VL[0]));
>> + for (unsigned i = 1, e = VL.size(); i < e; ++i)
>> + MaxIdx = std::max(MaxIdx, BN.getIndex(cast<Instruction>(VL[i])));
>> + Instruction *I = BN.getInstruction(MaxIdx);
>> + assert(I && "bad location");
>> + return I;
>> +}
>>
>> - return false;
>> +Instruction *BoUpSLP::getInstructionForIndex(unsigned Index, BasicBlock
>> *BB) {
>> + BlockNumbering &BN = BlocksNumbers[BB];
>> + return BN.getInstruction(Index);
>> }
>>
>> -bool FuncSLP::vectorizeStores(ArrayRef<StoreInst *> Stores, int
>> costThreshold) {
>> - SetVector<Value *> Heads, Tails;
>> - SmallDenseMap<Value *, Value *> ConsecutiveChain;
>> +int BoUpSLP::getFirstUserIndex(ArrayRef<Value *> VL) {
>> + BasicBlock *BB = getSameBlock(VL);
>> + assert(BB && "All instructions must come from the same block");
>> + BlockNumbering &BN = BlocksNumbers[BB];
>>
>> - // We may run into multiple chains that merge into a single chain. We
>> mark the
>> - // stores that we vectorized so that we don't visit the same store
>> twice.
>> - ValueSet VectorizedStores;
>> - bool Changed = false;
>> + // Find the first user of the values.
>> + int FirstUser = BN.getIndex(BB->getTerminator());
>> + for (unsigned i = 0, e = VL.size(); i < e; ++i) {
>> + for (Value::use_iterator U = VL[i]->use_begin(), UE =
>> VL[i]->use_end();
>> + U != UE; ++U) {
>> + Instruction *Instr = dyn_cast<Instruction>(*U);
>>
>> - // Do a quadratic search on all of the given stores and find
>> - // all of the pairs of loads that follow each other.
>> - for (unsigned i = 0, e = Stores.size(); i < e; ++i)
>> - for (unsigned j = 0; j < e; ++j) {
>> - if (i == j)
>> + if (!Instr || Instr->getParent() != BB)
>> continue;
>>
>> - if (isConsecutiveAccess(Stores[i], Stores[j])) {
>> - Tails.insert(Stores[j]);
>> - Heads.insert(Stores[i]);
>> - ConsecutiveChain[Stores[i]] = Stores[j];
>> - }
>> - }
>> -
>> - // For stores that start but don't end a link in the chain:
>> - for (SetVector<Value *>::iterator it = Heads.begin(), e = Heads.end();
>> - it != e; ++it) {
>> - if (Tails.count(*it))
>> - continue;
>> -
>> - // We found a store instr that starts a chain. Now follow the chain
>> and try
>> - // to vectorize it.
>> - ValueList Operands;
>> - Value *I = *it;
>> - // Collect the chain into a list.
>> - while (Tails.count(I) || Heads.count(I)) {
>> - if (VectorizedStores.count(I))
>> - break;
>> - Operands.push_back(I);
>> - // Move to the next value in the chain.
>> - I = ConsecutiveChain[I];
>> + FirstUser = std::min(FirstUser, BN.getIndex(Instr));
>> }
>> -
>> - bool Vectorized = vectorizeStoreChain(Operands, costThreshold);
>> -
>> - // Mark the vectorized stores so that we don't vectorize them again.
>> - if (Vectorized)
>> - VectorizedStores.insert(Operands.begin(), Operands.end());
>> - Changed |= Vectorized;
>> }
>> -
>> - return Changed;
>> + return FirstUser;
>> }
>>
>> -Value *FuncSLP::Gather(ArrayRef<Value *> VL, VectorType *Ty) {
>> +Value *BoUpSLP::Gather(ArrayRef<Value *> VL, VectorType *Ty) {
>> Value *Vec = UndefValue::get(Ty);
>> // Generate the 'InsertElement' instruction.
>> for (unsigned i = 0; i < Ty->getNumElements(); ++i) {
>> @@ -1103,282 +1011,292 @@ Value *FuncSLP::Gather(ArrayRef<Value *>
>> return Vec;
>> }
>>
>> -Value *FuncSLP::vectorizeTree_rec(ArrayRef<Value *> VL) {
>> - BuilderLocGuard Guard(Builder);
>> +Value *BoUpSLP::vectorizeTree(ArrayRef<Value *> VL) {
>> + if (ScalarToTreeEntry.count(VL[0])) {
>> + int Idx = ScalarToTreeEntry[VL[0]];
>> + TreeEntry *E = &VectorizableTree[Idx];
>> + if (E->isSame(VL))
>> + return vectorizeTree(E);
>> + }
>>
>> Type *ScalarTy = VL[0]->getType();
>> if (StoreInst *SI = dyn_cast<StoreInst>(VL[0]))
>> ScalarTy = SI->getValueOperand()->getType();
>> VectorType *VecTy = VectorType::get(ScalarTy, VL.size());
>>
>> - if (needToGatherAny(VL))
>> - return Gather(VL, VecTy);
>> + return Gather(VL, VecTy);
>> +}
>> +
>> +Value *BoUpSLP::vectorizeTree(TreeEntry *E) {
>> + BuilderLocGuard Guard(Builder);
>> +
>> + if (E->VectorizedValue) {
>> + DEBUG(dbgs() << "SLP: Diamond merged for " << *E->Scalars[0] <<
>> ".\n");
>> + return E->VectorizedValue;
>> + }
>> +
>> + Type *ScalarTy = E->Scalars[0]->getType();
>> + if (StoreInst *SI = dyn_cast<StoreInst>(E->Scalars[0]))
>> + ScalarTy = SI->getValueOperand()->getType();
>> + VectorType *VecTy = VectorType::get(ScalarTy, E->Scalars.size());
>>
>> - if (VectorizedValues.count(VL[0])) {
>> - DEBUG(dbgs() << "SLP: Diamond merged at depth.\n");
>> - return VectorizedValues[VL[0]];
>> + if (E->NeedToGather) {
>> + return Gather(E->Scalars, VecTy);
>> }
>>
>> - Instruction *VL0 = cast<Instruction>(VL[0]);
>> + Instruction *VL0 = cast<Instruction>(E->Scalars[0]);
>> unsigned Opcode = VL0->getOpcode();
>> - assert(Opcode == getSameOpcode(VL) && "Invalid opcode");
>> + assert(Opcode == getSameOpcode(E->Scalars) && "Invalid opcode");
>>
>> switch (Opcode) {
>> - case Instruction::PHI: {
>> - PHINode *PH = dyn_cast<PHINode>(VL0);
>> - Builder.SetInsertPoint(PH->getParent()->getFirstInsertionPt());
>> - PHINode *NewPhi = Builder.CreatePHI(VecTy,
>> PH->getNumIncomingValues());
>> - VectorizedValues[VL0] = NewPhi;
>> + case Instruction::PHI: {
>> + PHINode *PH = dyn_cast<PHINode>(VL0);
>> + Builder.SetInsertPoint(PH->getParent()->getFirstInsertionPt());
>> + PHINode *NewPhi = Builder.CreatePHI(VecTy,
>> PH->getNumIncomingValues());
>> + E->VectorizedValue = NewPhi;
>> +
>> + for (unsigned i = 0, e = PH->getNumIncomingValues(); i < e; ++i) {
>> + ValueList Operands;
>> + BasicBlock *IBB = PH->getIncomingBlock(i);
>> +
>> + // Prepare the operand vector.
>> + for (unsigned j = 0; j < E->Scalars.size(); ++j)
>> + Operands.push_back(cast<PHINode>(E->Scalars[j])->
>> + getIncomingValueForBlock(IBB));
>> +
>> + Builder.SetInsertPoint(IBB->getTerminator());
>> + Value *Vec = vectorizeTree(Operands);
>> + NewPhi->addIncoming(Vec, IBB);
>> + }
>>
>> - for (unsigned i = 0, e = PH->getNumIncomingValues(); i < e; ++i) {
>> - ValueList Operands;
>> - BasicBlock *IBB = PH->getIncomingBlock(i);
>> + assert(NewPhi->getNumIncomingValues() ==
>> PH->getNumIncomingValues() &&
>> + "Invalid number of incoming values");
>> + return NewPhi;
>> + }
>>
>> - // Prepare the operand vector.
>> - for (unsigned j = 0; j < VL.size(); ++j)
>> -
>> Operands.push_back(cast<PHINode>(VL[j])->getIncomingValueForBlock(IBB));
>> + case Instruction::ExtractElement: {
>> + if (CanReuseExtract(E->Scalars)) {
>> + Value *V = VL0->getOperand(0);
>> + E->VectorizedValue = V;
>> + return V;
>> + }
>> + return Gather(E->Scalars, VecTy);
>> + }
>> + case Instruction::ZExt:
>> + case Instruction::SExt:
>> + case Instruction::FPToUI:
>> + case Instruction::FPToSI:
>> + case Instruction::FPExt:
>> + case Instruction::PtrToInt:
>> + case Instruction::IntToPtr:
>> + case Instruction::SIToFP:
>> + case Instruction::UIToFP:
>> + case Instruction::Trunc:
>> + case Instruction::FPTrunc:
>> + case Instruction::BitCast: {
>> + ValueList INVL;
>> + for (int i = 0, e = E->Scalars.size(); i < e; ++i)
>> + INVL.push_back(cast<Instruction>(E->Scalars[i])->getOperand(0));
>> +
>> + Builder.SetInsertPoint(getLastInstruction(E->Scalars));
>> + Value *InVec = vectorizeTree(INVL);
>> + CastInst *CI = dyn_cast<CastInst>(VL0);
>> + Value *V = Builder.CreateCast(CI->getOpcode(), InVec, VecTy);
>> + E->VectorizedValue = V;
>> + return V;
>> + }
>> + case Instruction::FCmp:
>> + case Instruction::ICmp: {
>> + ValueList LHSV, RHSV;
>> + for (int i = 0, e = E->Scalars.size(); i < e; ++i) {
>> + LHSV.push_back(cast<Instruction>(E->Scalars[i])->getOperand(0));
>> + RHSV.push_back(cast<Instruction>(E->Scalars[i])->getOperand(1));
>> + }
>>
>> - Builder.SetInsertPoint(IBB->getTerminator());
>> - Value *Vec = vectorizeTree_rec(Operands);
>> - NewPhi->addIncoming(Vec, IBB);
>> - }
>> -
>> - assert(NewPhi->getNumIncomingValues() == PH->getNumIncomingValues()
>> &&
>> - "Invalid number of incoming values");
>> - return NewPhi;
>> - }
>> -
>> - case Instruction::ExtractElement: {
>> - if (CanReuseExtract(VL, VL.size(), VecTy))
>> - return VL0->getOperand(0);
>> - return Gather(VL, VecTy);
>> - }
>> - case Instruction::ZExt:
>> - case Instruction::SExt:
>> - case Instruction::FPToUI:
>> - case Instruction::FPToSI:
>> - case Instruction::FPExt:
>> - case Instruction::PtrToInt:
>> - case Instruction::IntToPtr:
>> - case Instruction::SIToFP:
>> - case Instruction::UIToFP:
>> - case Instruction::Trunc:
>> - case Instruction::FPTrunc:
>> - case Instruction::BitCast: {
>> - ValueList INVL;
>> - for (int i = 0, e = VL.size(); i < e; ++i)
>> - INVL.push_back(cast<Instruction>(VL[i])->getOperand(0));
>> -
>> - Builder.SetInsertPoint(getLastInstruction(VL));
>> - Value *InVec = vectorizeTree_rec(INVL);
>> - CastInst *CI = dyn_cast<CastInst>(VL0);
>> - Value *V = Builder.CreateCast(CI->getOpcode(), InVec, VecTy);
>> - VectorizedValues[VL0] = V;
>> - return V;
>> - }
>> - case Instruction::FCmp:
>> - case Instruction::ICmp: {
>> - // Check that all of the compares have the same predicate.
>> - CmpInst::Predicate P0 = dyn_cast<CmpInst>(VL0)->getPredicate();
>> - for (unsigned i = 1, e = VL.size(); i < e; ++i) {
>> - CmpInst *Cmp = cast<CmpInst>(VL[i]);
>> - if (Cmp->getPredicate() != P0)
>> - return Gather(VL, VecTy);
>> - }
>> -
>> - ValueList LHSV, RHSV;
>> - for (int i = 0, e = VL.size(); i < e; ++i) {
>> - LHSV.push_back(cast<Instruction>(VL[i])->getOperand(0));
>> - RHSV.push_back(cast<Instruction>(VL[i])->getOperand(1));
>> - }
>> -
>> - Builder.SetInsertPoint(getLastInstruction(VL));
>> - Value *L = vectorizeTree_rec(LHSV);
>> - Value *R = vectorizeTree_rec(RHSV);
>> - Value *V;
>> -
>> - if (Opcode == Instruction::FCmp)
>> - V = Builder.CreateFCmp(P0, L, R);
>> - else
>> - V = Builder.CreateICmp(P0, L, R);
>> -
>> - VectorizedValues[VL0] = V;
>> - return V;
>> - }
>> - case Instruction::Select: {
>> - ValueList TrueVec, FalseVec, CondVec;
>> - for (int i = 0, e = VL.size(); i < e; ++i) {
>> - CondVec.push_back(cast<Instruction>(VL[i])->getOperand(0));
>> - TrueVec.push_back(cast<Instruction>(VL[i])->getOperand(1));
>> - FalseVec.push_back(cast<Instruction>(VL[i])->getOperand(2));
>> - }
>> -
>> - Builder.SetInsertPoint(getLastInstruction(VL));
>> - Value *True = vectorizeTree_rec(TrueVec);
>> - Value *False = vectorizeTree_rec(FalseVec);
>> - Value *Cond = vectorizeTree_rec(CondVec);
>> - Value *V = Builder.CreateSelect(Cond, True, False);
>> - VectorizedValues[VL0] = V;
>> - return V;
>> - }
>> - case Instruction::Add:
>> - case Instruction::FAdd:
>> - case Instruction::Sub:
>> - case Instruction::FSub:
>> - case Instruction::Mul:
>> - case Instruction::FMul:
>> - case Instruction::UDiv:
>> - case Instruction::SDiv:
>> - case Instruction::FDiv:
>> - case Instruction::URem:
>> - case Instruction::SRem:
>> - case Instruction::FRem:
>> - case Instruction::Shl:
>> - case Instruction::LShr:
>> - case Instruction::AShr:
>> - case Instruction::And:
>> - case Instruction::Or:
>> - case Instruction::Xor: {
>> - ValueList LHSVL, RHSVL;
>> - for (int i = 0, e = VL.size(); i < e; ++i) {
>> - LHSVL.push_back(cast<Instruction>(VL[i])->getOperand(0));
>> - RHSVL.push_back(cast<Instruction>(VL[i])->getOperand(1));
>> - }
>> -
>> - Builder.SetInsertPoint(getLastInstruction(VL));
>> - Value *LHS = vectorizeTree_rec(LHSVL);
>> - Value *RHS = vectorizeTree_rec(RHSVL);
>> -
>> - if (LHS == RHS) {
>> - assert((VL0->getOperand(0) == VL0->getOperand(1)) && "Invalid
>> order");
>> - }
>> -
>> - BinaryOperator *BinOp = cast<BinaryOperator>(VL0);
>> - Value *V = Builder.CreateBinOp(BinOp->getOpcode(), LHS, RHS);
>> - VectorizedValues[VL0] = V;
>> - return V;
>> - }
>> - case Instruction::Load: {
>> - // Check if all of the loads are consecutive.
>> - for (unsigned i = 1, e = VL.size(); i < e; ++i)
>> - if (!isConsecutiveAccess(VL[i - 1], VL[i]))
>> - return Gather(VL, VecTy);
>> -
>> - // Loads are inserted at the head of the tree because we don't want
>> to
>> - // sink them all the way down past store instructions.
>> - Builder.SetInsertPoint(getLastInstruction(VL));
>> - LoadInst *LI = cast<LoadInst>(VL0);
>> - Value *VecPtr =
>> - Builder.CreateBitCast(LI->getPointerOperand(),
>> VecTy->getPointerTo());
>> - unsigned Alignment = LI->getAlignment();
>> - LI = Builder.CreateLoad(VecPtr);
>> - LI->setAlignment(Alignment);
>> -
>> - VectorizedValues[VL0] = LI;
>> - return LI;
>> - }
>> - case Instruction::Store: {
>> - StoreInst *SI = cast<StoreInst>(VL0);
>> - unsigned Alignment = SI->getAlignment();
>> -
>> - ValueList ValueOp;
>> - for (int i = 0, e = VL.size(); i < e; ++i)
>> - ValueOp.push_back(cast<StoreInst>(VL[i])->getValueOperand());
>> -
>> - Value *VecValue = vectorizeTree_rec(ValueOp);
>> -
>> - Builder.SetInsertPoint(getLastInstruction(VL));
>> - Value *VecPtr =
>> - Builder.CreateBitCast(SI->getPointerOperand(),
>> VecTy->getPointerTo());
>> - Builder.CreateStore(VecValue, VecPtr)->setAlignment(Alignment);
>> - return 0;
>> - }
>> - default:
>> - return Gather(VL, VecTy);
>> + Builder.SetInsertPoint(getLastInstruction(E->Scalars));
>> + Value *L = vectorizeTree(LHSV);
>> + Value *R = vectorizeTree(RHSV);
>> + Value *V;
>> +
>> + CmpInst::Predicate P0 = dyn_cast<CmpInst>(VL0)->getPredicate();
>> + if (Opcode == Instruction::FCmp)
>> + V = Builder.CreateFCmp(P0, L, R);
>> + else
>> + V = Builder.CreateICmp(P0, L, R);
>> +
>> + E->VectorizedValue = V;
>> + return V;
>> + }
>> + case Instruction::Select: {
>> + ValueList TrueVec, FalseVec, CondVec;
>> + for (int i = 0, e = E->Scalars.size(); i < e; ++i) {
>> +
>> CondVec.push_back(cast<Instruction>(E->Scalars[i])->getOperand(0));
>> +
>> TrueVec.push_back(cast<Instruction>(E->Scalars[i])->getOperand(1));
>> +
>> FalseVec.push_back(cast<Instruction>(E->Scalars[i])->getOperand(2));
>> + }
>> +
>> + Builder.SetInsertPoint(getLastInstruction(E->Scalars));
>> + Value *Cond = vectorizeTree(CondVec);
>> + Value *True = vectorizeTree(TrueVec);
>> + Value *False = vectorizeTree(FalseVec);
>> + Value *V = Builder.CreateSelect(Cond, True, False);
>> + E->VectorizedValue = V;
>> + return V;
>> + }
>> + case Instruction::Add:
>> + case Instruction::FAdd:
>> + case Instruction::Sub:
>> + case Instruction::FSub:
>> + case Instruction::Mul:
>> + case Instruction::FMul:
>> + case Instruction::UDiv:
>> + case Instruction::SDiv:
>> + case Instruction::FDiv:
>> + case Instruction::URem:
>> + case Instruction::SRem:
>> + case Instruction::FRem:
>> + case Instruction::Shl:
>> + case Instruction::LShr:
>> + case Instruction::AShr:
>> + case Instruction::And:
>> + case Instruction::Or:
>> + case Instruction::Xor: {
>> + ValueList LHSVL, RHSVL;
>> + for (int i = 0, e = E->Scalars.size(); i < e; ++i) {
>> + LHSVL.push_back(cast<Instruction>(E->Scalars[i])->getOperand(0));
>> + RHSVL.push_back(cast<Instruction>(E->Scalars[i])->getOperand(1));
>> + }
>> +
>> + Builder.SetInsertPoint(getLastInstruction(E->Scalars));
>> + Value *LHS = vectorizeTree(LHSVL);
>> + Value *RHS = vectorizeTree(RHSVL);
>> +
>> + if (LHS == RHS && isa<Instruction>(LHS)) {
>> + assert((VL0->getOperand(0) == VL0->getOperand(1)) && "Invalid
>> order");
>> + }
>> +
>> + BinaryOperator *BinOp = cast<BinaryOperator>(VL0);
>> + Value *V = Builder.CreateBinOp(BinOp->getOpcode(), LHS, RHS);
>> + E->VectorizedValue = V;
>> + return V;
>> + }
>> + case Instruction::Load: {
>> + // Loads are inserted at the head of the tree because we don't
>> want to
>> + // sink them all the way down past store instructions.
>> + Builder.SetInsertPoint(getLastInstruction(E->Scalars));
>> + LoadInst *LI = cast<LoadInst>(VL0);
>> + Value *VecPtr =
>> + Builder.CreateBitCast(LI->getPointerOperand(),
>> VecTy->getPointerTo());
>> + unsigned Alignment = LI->getAlignment();
>> + LI = Builder.CreateLoad(VecPtr);
>> + LI->setAlignment(Alignment);
>> + E->VectorizedValue = LI;
>> + return LI;
>> + }
>> + case Instruction::Store: {
>> + StoreInst *SI = cast<StoreInst>(VL0);
>> + unsigned Alignment = SI->getAlignment();
>> +
>> + ValueList ValueOp;
>> + for (int i = 0, e = E->Scalars.size(); i < e; ++i)
>> +
>> ValueOp.push_back(cast<StoreInst>(E->Scalars[i])->getValueOperand());
>> +
>> + Builder.SetInsertPoint(getLastInstruction(E->Scalars));
>> + Value *VecValue = vectorizeTree(ValueOp);
>> + Value *VecPtr =
>> + Builder.CreateBitCast(SI->getPointerOperand(),
>> VecTy->getPointerTo());
>> + StoreInst *S = Builder.CreateStore(VecValue, VecPtr);
>> + S->setAlignment(Alignment);
>> + E->VectorizedValue = S;
>> + return S;
>> + }
>> + default:
>> + llvm_unreachable("unknown inst");
>> }
>> + return 0;
>> }
>>
>> -Value *FuncSLP::vectorizeTree(ArrayRef<Value *> VL) {
>> - Builder.SetInsertPoint(getLastInstruction(VL));
>> - Value *V = vectorizeTree_rec(VL);
>> +void BoUpSLP::vectorizeTree() {
>> + vectorizeTree(&VectorizableTree[0]);
>>
>> - DEBUG(dbgs() << "SLP: Placing 'extracts'\n");
>> - for (SetVector<Instruction*>::iterator it = ExtractedLane.begin(), e =
>> - ExtractedLane.end(); it != e; ++it) {
>> - Instruction *Scalar = *it;
>> - DEBUG(dbgs() << "SLP: Looking at " << *Scalar);
>> + // For each vectorized value:
>> + for (int EIdx = 0, EE = VectorizableTree.size(); EIdx < EE; ++EIdx) {
>> + TreeEntry *Entry = &VectorizableTree[EIdx];
>> +
>> + // For each lane:
>> + for (int Lane = 0, LE = Entry->Scalars.size(); Lane != LE; ++Lane) {
>> + Value *Scalar = Entry->Scalars[Lane];
>>
>> - if (!Scalar)
>> - continue;
>> + // No need to handle users of gathered values.
>> + if (Entry->NeedToGather)
>> + continue;
>>
>> - Instruction *Loc = 0;
>> + Value *Vec = Entry->VectorizedValue;
>> + assert(Vec && "Can't find vectorizable value");
>>
>> - assert(MultiUserVals.count(Scalar) && "Can't find the lane to
>> extract");
>> - Instruction *Leader = MultiUserVals[Scalar].Leader;
>> + SmallVector<User*, 16> Users(Scalar->use_begin(),
>> Scalar->use_end());
>>
>> - // This value is gathered so we don't need to extract from anywhere.
>> - if (!VectorizedValues.count(Leader))
>> - continue;
>> + for (SmallVector<User*, 16>::iterator User = Users.begin(),
>> + UE = Users.end(); User != UE; ++User) {
>> + DEBUG(dbgs() << "SLP: \tupdating user " << **User << ".\n");
>> +
>> + bool Gathered = MustGather.count(*User);
>> +
>> + // Skip in-tree scalars that become vectors.
>> + if (ScalarToTreeEntry.count(*User) && !Gathered) {
>> + DEBUG(dbgs() << "SLP: \tUser will be removed soon:" <<
>> + **User << ".\n");
>> + int Idx = ScalarToTreeEntry[*User]; (void) Idx;
>> + assert(!VectorizableTree[Idx].NeedToGather && "bad state ?");
>> + continue;
>> + }
>>
>> - Value *Vec = VectorizedValues[Leader];
>> - if (PHINode *PN = dyn_cast<PHINode>(Vec)) {
>> - Loc = PN->getParent()->getFirstInsertionPt();
>> - } else {
>> - Instruction *I = cast<Instruction>(Vec);
>> - BasicBlock::iterator L = *I;
>> - Loc = ++L;
>> - }
>> + if (!isa<Instruction>(*User))
>> + continue;
>>
>> - Builder.SetInsertPoint(Loc);
>> - assert(LaneMap.count(Scalar) && "Can't find the extracted lane.");
>> - int Lane = LaneMap[Scalar];
>> - Value *Idx = Builder.getInt32(Lane);
>> - Value *Extract = Builder.CreateExtractElement(Vec, Idx);
>> + // Generate extracts for out-of-tree users.
>> + // Find the insertion point for the extractelement lane.
>> + Instruction *Loc = 0;
>> + if (PHINode *PN = dyn_cast<PHINode>(Vec)) {
>> + Loc = PN->getParent()->getFirstInsertionPt();
>> + } else if (Instruction *Iv = dyn_cast<Instruction>(Vec)){
>> + Loc = ++((BasicBlock::iterator)*Iv);
>> + } else {
>> + Loc = F->getEntryBlock().begin();
>> + }
>>
>> - bool Replaced = false;;
>> - for (Value::use_iterator U = Scalar->use_begin(), UE =
>> Scalar->use_end();
>> - U != UE; ++U) {
>> - Instruction *UI = cast<Instruction>(*U);
>> - // No need to replace instructions that are inside our lane map.
>> - if (LaneMap.count(UI))
>> - continue;
>> + Builder.SetInsertPoint(Loc);
>> + Value *Ex = Builder.CreateExtractElement(Vec,
>> Builder.getInt32(Lane));
>> + (*User)->replaceUsesOfWith(Scalar, Ex);
>> + DEBUG(dbgs() << "SLP: \tupdated user:" << **User << ".\n");
>> + }
>>
>> - UI->replaceUsesOfWith(Scalar ,Extract);
>> - Replaced = true;
>> + Type *Ty = Scalar->getType();
>> + if (!Ty->isVoidTy()) {
>> + for (Value::use_iterator User = Scalar->use_begin(), UE =
>> Scalar->use_end();
>> + User != UE; ++User) {
>> + DEBUG(dbgs() << "SLP: \tvalidating user:" << **User << ".\n");
>> + assert(!MustGather.count(*User) &&
>> + "Replacing gathered value with undef");
>> + assert(ScalarToTreeEntry.count(*User) &&
>> + "Replacing out-of-tree value with undef");
>> + }
>> + Value *Undef = UndefValue::get(Ty);
>> + Scalar->replaceAllUsesWith(Undef);
>> + }
>> + DEBUG(dbgs() << "SLP: \tErasing scalar:" << *Scalar << ".\n");
>> + cast<Instruction>(Scalar)->eraseFromParent();
>> }
>> - assert(Replaced && "Must replace at least one outside user");
>> - (void)Replaced;
>> }
>>
>> - // We moved some instructions around. We have to number them again
>> - // before we can do any analysis.
>> - forgetNumbering();
>> -
>> - // Clear the state.
>> - MustGather.clear();
>> - VisitedPHIs.clear();
>> - VectorizedValues.clear();
>> - MemBarrierIgnoreList.clear();
>> - return V;
>> -}
>> -
>> -Value *FuncSLP::vectorizeArith(ArrayRef<Value *> Operands) {
>> - Instruction *LastInst = getLastInstruction(Operands);
>> - Value *Vec = vectorizeTree(Operands);
>> - // After vectorizing the operands we need to generate extractelement
>> - // instructions and replace all of the uses of the scalar values with
>> - // the values that we extracted from the vectorized tree.
>> - Builder.SetInsertPoint(LastInst);
>> - for (unsigned i = 0, e = Operands.size(); i != e; ++i) {
>> - Value *S = Builder.CreateExtractElement(Vec, Builder.getInt32(i));
>> - Operands[i]->replaceAllUsesWith(S);
>> + for (Function::iterator it = F->begin(), e = F->end(); it != e; ++it) {
>> + BlocksNumbers[it].forget();
>> }
>> -
>> - forgetNumbering();
>> - return Vec;
>> }
>>
>> -void FuncSLP::optimizeGatherSequence() {
>> +void BoUpSLP::optimizeGatherSequence() {
>> + DEBUG(dbgs() << "SLP: Optimizing " << GatherSeq.size()
>> + << " gather sequences instructions.\n");
>> // LICM InsertElementInst sequences.
>> for (SetVector<Instruction *>::iterator it = GatherSeq.begin(),
>> e = GatherSeq.end(); it != e; ++it) {
>> @@ -1449,8 +1367,6 @@ void FuncSLP::optimizeGatherSequence() {
>> assert((*v)->getNumUses() == 0 && "Can't remove instructions with
>> uses");
>> (*v)->eraseFromParent();
>> }
>> -
>> - forgetNumbering();
>> }
>>
>> /// The SLPVectorizer Pass.
>> @@ -1492,7 +1408,7 @@ struct SLPVectorizer : public FunctionPa
>>
>> // Use the bollom up slp vectorizer to construct chains that start
>> with
>> // he store instructions.
>> - FuncSLP R(&F, SE, DL, TTI, AA, LI, DT);
>> + BoUpSLP R(&F, SE, DL, TTI, AA, LI, DT);
>>
>> // Scan the blocks in the function in post order.
>> for (po_iterator<BasicBlock*> it = po_begin(&F.getEntryBlock()),
>> @@ -1536,31 +1452,146 @@ private:
>> /// object. We sort the stores to their base objects to reduce the
>> cost of the
>> /// quadratic search on the stores. TODO: We can further reduce this
>> cost
>> /// if we flush the chain creation every time we run into a memory
>> barrier.
>> - unsigned collectStores(BasicBlock *BB, FuncSLP &R);
>> + unsigned collectStores(BasicBlock *BB, BoUpSLP &R);
>>
>> /// \brief Try to vectorize a chain that starts at two arithmetic
>> instrs.
>> - bool tryToVectorizePair(Value *A, Value *B, FuncSLP &R);
>> + bool tryToVectorizePair(Value *A, Value *B, BoUpSLP &R);
>>
>> /// \brief Try to vectorize a list of operands. If \p NeedExtracts is
>> true
>> /// then we calculate the cost of extracting the scalars from the
>> vector.
>> /// \returns true if a value was vectorized.
>> - bool tryToVectorizeList(ArrayRef<Value *> VL, FuncSLP &R, bool
>> NeedExtracts);
>> + bool tryToVectorizeList(ArrayRef<Value *> VL, BoUpSLP &R, bool
>> NeedExtracts);
>>
>> /// \brief Try to vectorize a chain that may start at the operands of
>> \V;
>> - bool tryToVectorize(BinaryOperator *V, FuncSLP &R);
>> + bool tryToVectorize(BinaryOperator *V, BoUpSLP &R);
>>
>> /// \brief Vectorize the stores that were collected in StoreRefs.
>> - bool vectorizeStoreChains(FuncSLP &R);
>> + bool vectorizeStoreChains(BoUpSLP &R);
>>
>> /// \brief Scan the basic block and look for patterns that are likely
>> to start
>> /// a vectorization chain.
>> - bool vectorizeChainsInBlock(BasicBlock *BB, FuncSLP &R);
>> + bool vectorizeChainsInBlock(BasicBlock *BB, BoUpSLP &R);
>> +
>> + bool vectorizeStoreChain(ArrayRef<Value *> Chain, int CostThreshold,
>> + BoUpSLP &R);
>>
>> + bool vectorizeStores(ArrayRef<StoreInst *> Stores, int costThreshold,
>> + BoUpSLP &R);
>> private:
>> StoreListMap StoreRefs;
>> };
>>
>> -unsigned SLPVectorizer::collectStores(BasicBlock *BB, FuncSLP &R) {
>> +bool SLPVectorizer::vectorizeStoreChain(ArrayRef<Value *> Chain,
>> + int CostThreshold, BoUpSLP &R)
>> {
>> + unsigned ChainLen = Chain.size();
>> + DEBUG(dbgs() << "SLP: Analyzing a store chain of length " << ChainLen
>> + << "\n");
>> + Type *StoreTy =
>> cast<StoreInst>(Chain[0])->getValueOperand()->getType();
>> + unsigned Sz = DL->getTypeSizeInBits(StoreTy);
>> + unsigned VF = MinVecRegSize / Sz;
>> +
>> + if (!isPowerOf2_32(Sz) || VF < 2)
>> + return false;
>> +
>> + bool Changed = false;
>> + // Look for profitable vectorizable trees at all offsets, starting at
>> zero.
>> + for (unsigned i = 0, e = ChainLen; i < e; ++i) {
>> + if (i + VF > e)
>> + break;
>> + DEBUG(dbgs() << "SLP: Analyzing " << VF << " stores at offset " << i
>> + << "\n");
>> + ArrayRef<Value *> Operands = Chain.slice(i, VF);
>> +
>> + R.buildTree(Operands);
>> +
>> + int Cost = R.getTreeCost();
>> +
>> + DEBUG(dbgs() << "SLP: Found cost=" << Cost << " for VF=" << VF <<
>> "\n");
>> + if (Cost < CostThreshold) {
>> + DEBUG(dbgs() << "SLP: Decided to vectorize cost=" << Cost << "\n");
>> + R.vectorizeTree();
>> +
>> + // Move to the next bundle.
>> + i += VF - 1;
>> + Changed = true;
>> + }
>> + }
>> +
>> + if (Changed || ChainLen > VF)
>> + return Changed;
>> +
>> + // Handle short chains. This helps us catch types such as <3 x float>
>> that
>> + // are smaller than vector size.
>> + R.buildTree(Chain);
>> +
>> + int Cost = R.getTreeCost();
>> +
>> + if (Cost < CostThreshold) {
>> + DEBUG(dbgs() << "SLP: Found store chain cost = " << Cost
>> + << " for size = " << ChainLen << "\n");
>> + R.vectorizeTree();
>> + return true;
>> + }
>> +
>> + return false;
>> +}
>> +
>> +bool SLPVectorizer::vectorizeStores(ArrayRef<StoreInst *> Stores,
>> + int costThreshold, BoUpSLP &R) {
>> + SetVector<Value *> Heads, Tails;
>> + SmallDenseMap<Value *, Value *> ConsecutiveChain;
>> +
>> + // We may run into multiple chains that merge into a single chain. We
>> mark the
>> + // stores that we vectorized so that we don't visit the same store
>> twice.
>> + BoUpSLP::ValueSet VectorizedStores;
>> + bool Changed = false;
>> +
>> + // Do a quadratic search on all of the given stores and find
>> + // all of the pairs of loads that follow each other.
>> + for (unsigned i = 0, e = Stores.size(); i < e; ++i)
>> + for (unsigned j = 0; j < e; ++j) {
>> + if (i == j)
>> + continue;
>> +
>> + if (R.isConsecutiveAccess(Stores[i], Stores[j])) {
>> + Tails.insert(Stores[j]);
>> + Heads.insert(Stores[i]);
>> + ConsecutiveChain[Stores[i]] = Stores[j];
>> + }
>> + }
>> +
>> + // For stores that start but don't end a link in the chain:
>> + for (SetVector<Value *>::iterator it = Heads.begin(), e = Heads.end();
>> + it != e; ++it) {
>> + if (Tails.count(*it))
>> + continue;
>> +
>> + // We found a store instr that starts a chain. Now follow the chain
>> and try
>> + // to vectorize it.
>> + BoUpSLP::ValueList Operands;
>> + Value *I = *it;
>> + // Collect the chain into a list.
>> + while (Tails.count(I) || Heads.count(I)) {
>> + if (VectorizedStores.count(I))
>> + break;
>> + Operands.push_back(I);
>> + // Move to the next value in the chain.
>> + I = ConsecutiveChain[I];
>> + }
>> +
>> + bool Vectorized = vectorizeStoreChain(Operands, costThreshold, R);
>> +
>> + // Mark the vectorized stores so that we don't vectorize them again.
>> + if (Vectorized)
>> + VectorizedStores.insert(Operands.begin(), Operands.end());
>> + Changed |= Vectorized;
>> + }
>> +
>> + return Changed;
>> +}
>> +
>> +
>> +unsigned SLPVectorizer::collectStores(BasicBlock *BB, BoUpSLP &R) {
>> unsigned count = 0;
>> StoreRefs.clear();
>> for (BasicBlock::iterator it = BB->begin(), e = BB->end(); it != e;
>> ++it) {
>> @@ -1585,14 +1616,14 @@ unsigned SLPVectorizer::collectStores(Ba
>> return count;
>> }
>>
>> -bool SLPVectorizer::tryToVectorizePair(Value *A, Value *B, FuncSLP &R) {
>> +bool SLPVectorizer::tryToVectorizePair(Value *A, Value *B, BoUpSLP &R) {
>> if (!A || !B)
>> return false;
>> Value *VL[] = { A, B };
>> return tryToVectorizeList(VL, R, true);
>> }
>>
>> -bool SLPVectorizer::tryToVectorizeList(ArrayRef<Value *> VL, FuncSLP &R,
>> +bool SLPVectorizer::tryToVectorizeList(ArrayRef<Value *> VL, BoUpSLP &R,
>> bool NeedExtracts) {
>> if (VL.size() < 2)
>> return false;
>> @@ -1615,9 +1646,8 @@ bool SLPVectorizer::tryToVectorizeList(A
>> return 0;
>> }
>>
>> - int Cost = R.getTreeCost(VL);
>> - if (Cost == FuncSLP::MAX_COST)
>> - return false;
>> + R.buildTree(VL);
>> + int Cost = R.getTreeCost();
>>
>> int ExtrCost = NeedExtracts ? R.getGatherCost(VL) : 0;
>> DEBUG(dbgs() << "SLP: Cost of pair:" << Cost
>> @@ -1625,11 +1655,11 @@ bool SLPVectorizer::tryToVectorizeList(A
>> if ((Cost + ExtrCost) >= -SLPCostThreshold)
>> return false;
>> DEBUG(dbgs() << "SLP: Vectorizing pair.\n");
>> - R.vectorizeArith(VL);
>> + R.vectorizeTree();
>> return true;
>> }
>>
>> -bool SLPVectorizer::tryToVectorize(BinaryOperator *V, FuncSLP &R) {
>> +bool SLPVectorizer::tryToVectorize(BinaryOperator *V, BoUpSLP &R) {
>> if (!V)
>> return false;
>>
>> @@ -1669,7 +1699,7 @@ bool SLPVectorizer::tryToVectorize(Binar
>> return 0;
>> }
>>
>> -bool SLPVectorizer::vectorizeChainsInBlock(BasicBlock *BB, FuncSLP &R) {
>> +bool SLPVectorizer::vectorizeChainsInBlock(BasicBlock *BB, BoUpSLP &R) {
>> bool Changed = false;
>> for (BasicBlock::iterator it = BB->begin(), e = BB->end(); it != e;
>> ++it) {
>> if (isa<DbgInfoIntrinsic>(it))
>> @@ -1737,7 +1767,7 @@ bool SLPVectorizer::vectorizeChainsInBlo
>> return Changed;
>> }
>>
>> -bool SLPVectorizer::vectorizeStoreChains(FuncSLP &R) {
>> +bool SLPVectorizer::vectorizeStoreChains(BoUpSLP &R) {
>> bool Changed = false;
>> // Attempt to sort and vectorize each of the store-groups.
>> for (StoreListMap::iterator it = StoreRefs.begin(), e =
>> StoreRefs.end();
>> @@ -1748,7 +1778,7 @@ bool SLPVectorizer::vectorizeStoreChains
>> DEBUG(dbgs() << "SLP: Analyzing a store chain of length "
>> << it->second.size() << ".\n");
>>
>> - Changed |= R.vectorizeStores(it->second, -SLPCostThreshold);
>> + Changed |= vectorizeStores(it->second, -SLPCostThreshold, R);
>> }
>> return Changed;
>> }
>>
>> Added: llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_7zip.ll
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_7zip.ll?rev=185774&view=auto
>>
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_7zip.ll (added)
>> +++ llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_7zip.ll Sun Jul 7
>> 01:57:07 2013
>> @@ -0,0 +1,38 @@
>> +; RUN: opt < %s -basicaa -slp-vectorizer -dce -S
>> -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7
>> +
>> +target datalayout =
>> "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
>> +target triple = "x86_64-apple-macosx10.8.0"
>> +
>> +%struct.CLzmaDec.1.28.55.82.103.124.145.166.181.196.229.259.334 = type {
>> %struct._CLzmaProps.0.27.54.81.102.123.144.165.180.195.228.258.333, i16*,
>> i8*, i8*, i32, i32, i64, i64, i32, i32, i32, [4 x i32], i32, i32, i32, i32,
>> i32, [20 x i8] }
>> +%struct._CLzmaProps.0.27.54.81.102.123.144.165.180.195.228.258.333 =
>> type { i32, i32, i32, i32 }
>> +
>> +define fastcc void
>> @LzmaDec_DecodeReal2(%struct.CLzmaDec.1.28.55.82.103.124.145.166.181.196.229.259.334*
>> %p) {
>> +entry:
>> + %range20.i = getelementptr inbounds
>> %struct.CLzmaDec.1.28.55.82.103.124.145.166.181.196.229.259.334* %p, i64 0,
>> i32 4
>> + %code21.i = getelementptr inbounds
>> %struct.CLzmaDec.1.28.55.82.103.124.145.166.181.196.229.259.334* %p, i64 0,
>> i32 5
>> + br label %do.body66.i
>> +
>> +do.body66.i: ; preds = %do.cond.i,
>> %entry
>> + %range.2.i = phi i32 [ %range.4.i, %do.cond.i ], [ undef, %entry ]
>> + %code.2.i = phi i32 [ %code.4.i, %do.cond.i ], [ undef, %entry ]
>> + %.range.2.i = select i1 undef, i32 undef, i32 %range.2.i
>> + %.code.2.i = select i1 undef, i32 undef, i32 %code.2.i
>> + br i1 undef, label %do.cond.i, label %if.else.i
>> +
>> +if.else.i: ; preds = %do.body66.i
>> + %sub91.i = sub i32 %.range.2.i, undef
>> + %sub92.i = sub i32 %.code.2.i, undef
>> + br label %do.cond.i
>> +
>> +do.cond.i: ; preds = %if.else.i,
>> %do.body66.i
>> + %range.4.i = phi i32 [ %sub91.i, %if.else.i ], [ undef, %do.body66.i ]
>> + %code.4.i = phi i32 [ %sub92.i, %if.else.i ], [ %.code.2.i,
>> %do.body66.i ]
>> + br i1 undef, label %do.body66.i, label %do.end1006.i
>> +
>> +do.end1006.i: ; preds = %do.cond.i
>> + %.range.4.i = select i1 undef, i32 undef, i32 %range.4.i
>> + %.code.4.i = select i1 undef, i32 undef, i32 %code.4.i
>> + store i32 %.range.4.i, i32* %range20.i, align 4
>> + store i32 %.code.4.i, i32* %code21.i, align 4
>> + ret void
>> +}
>>
>> Added: llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_bullet.ll
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_bullet.ll?rev=185774&view=auto
>>
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_bullet.ll (added)
>> +++ llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_bullet.ll Sun Jul
>> 7 01:57:07 2013
>> @@ -0,0 +1,38 @@
>> +; RUN: opt < %s -basicaa -slp-vectorizer -dce -S
>> -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7
>> +
>> +target datalayout =
>> "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
>> +target triple = "x86_64-apple-macosx10.8.0"
>> +
>> +%"struct.btTypedConstraint::btConstraintInfo1.17.157.357.417.477.960" =
>> type { i32, i32 }
>> +
>> +define void
>> @_ZN23btGeneric6DofConstraint8getInfo1EPN17btTypedConstraint17btConstraintInfo1E(%"struct.btTypedConstraint::btConstraintInfo1.17.157.357.417.477.960"*
>> nocapture %info) {
>> +entry:
>> + br i1 undef, label %if.else, label %if.then
>> +
>> +if.then: ; preds = %entry
>> + ret void
>> +
>> +if.else: ; preds = %entry
>> + %m_numConstraintRows4 = getelementptr inbounds
>> %"struct.btTypedConstraint::btConstraintInfo1.17.157.357.417.477.960"*
>> %info, i64 0, i32 0
>> + %nub5 = getelementptr inbounds
>> %"struct.btTypedConstraint::btConstraintInfo1.17.157.357.417.477.960"*
>> %info, i64 0, i32 1
>> + br i1 undef, label %land.lhs.true.i.1, label %if.then7.1
>> +
>> +land.lhs.true.i.1: ; preds = %if.else
>> + br i1 undef, label %for.inc.1, label %if.then7.1
>> +
>> +if.then7.1: ; preds =
>> %land.lhs.true.i.1, %if.else
>> + %inc.1 = add nsw i32 0, 1
>> + store i32 %inc.1, i32* %m_numConstraintRows4, align 4
>> + %dec.1 = add nsw i32 6, -1
>> + store i32 %dec.1, i32* %nub5, align 4
>> + br label %for.inc.1
>> +
>> +for.inc.1: ; preds = %if.then7.1,
>> %land.lhs.true.i.1
>> + %0 = phi i32 [ %dec.1, %if.then7.1 ], [ 6, %land.lhs.true.i.1 ]
>> + %1 = phi i32 [ %inc.1, %if.then7.1 ], [ 0, %land.lhs.true.i.1 ]
>> + %inc.2 = add nsw i32 %1, 1
>> + store i32 %inc.2, i32* %m_numConstraintRows4, align 4
>> + %dec.2 = add nsw i32 %0, -1
>> + store i32 %dec.2, i32* %nub5, align 4
>> + unreachable
>> +}
>>
>> Added: llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_bullet2.ll
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_bullet2.ll?rev=185774&view=auto
>>
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_bullet2.ll (added)
>> +++ llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_bullet2.ll Sun Jul
>> 7 01:57:07 2013
>> @@ -0,0 +1,38 @@
>> +; RUN: opt < %s -basicaa -slp-vectorizer -dce -S
>> -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7
>> +
>> +target datalayout =
>> "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
>> +target triple = "x86_64-apple-macosx10.8.0"
>> +
>> +%class.GIM_TRIANGLE_CALCULATION_CACHE.9.34.69.94.119.144.179.189.264.284.332
>> = type { float, [3 x
>> %class.btVector3.5.30.65.90.115.140.175.185.260.280.330], [3 x
>> %class.btVector3.5.30.65.90.115.140.175.185.260.280.330],
>> %class.btVector4.7.32.67.92.117.142.177.187.262.282.331,
>> %class.btVector4.7.32.67.92.117.142.177.187.262.282.331,
>> %class.btVector3.5.30.65.90.115.140.175.185.260.280.330,
>> %class.btVector3.5.30.65.90.115.140.175.185.260.280.330,
>> %class.btVector3.5.30.65.90.115.140.175.185.260.280.330,
>> %class.btVector3.5.30.65.90.115.140.175.185.260.280.330, [4 x float],
>> float, float, [4 x float], float, float, [16 x
>> %class.btVector3.5.30.65.90.115.140.175.185.260.280.330], [16 x
>> %class.btVector3.5.30.65.90.115.140.175.185.260.280.330], [16 x
>> %class.btVector3.5.30.65.90.115.140.175.185.260.280.330] }
>> +%class.btVector3.5.30.65.90.115.140.175.185.260.280.330 = type { [4 x
>> float] }
>> +%class.btVector4.7.32.67.92.117.142.177.187.262.282.331 = type {
>> %class.btVector3.5.30.65.90.115.140.175.185.260.280.330 }
>> +
>> +define void
>> @_ZN30GIM_TRIANGLE_CALCULATION_CACHE18triangle_collisionERK9btVector3S2_S2_fS2_S2_S2_fR25GIM_TRIANGLE_CONTACT_DATA(%class.GIM_TRIANGLE_CALCULATION_CACHE.9.34.69.94.119.144.179.189.264.284.332*
>> %this) {
>> +entry:
>> + %arrayidx26 = getelementptr inbounds
>> %class.GIM_TRIANGLE_CALCULATION_CACHE.9.34.69.94.119.144.179.189.264.284.332*
>> %this, i64 0, i32 2, i64 0, i32 0, i64 1
>> + %arrayidx36 = getelementptr inbounds
>> %class.GIM_TRIANGLE_CALCULATION_CACHE.9.34.69.94.119.144.179.189.264.284.332*
>> %this, i64 0, i32 2, i64 0, i32 0, i64 2
>> + %0 = load float* %arrayidx36, align 4
>> + %add587 = fadd float undef, undef
>> + %sub600 = fsub float %add587, undef
>> + store float %sub600, float* undef, align 4
>> + %sub613 = fsub float %add587, %sub600
>> + store float %sub613, float* %arrayidx26, align 4
>> + %add626 = fadd float %0, undef
>> + %sub639 = fsub float %add626, undef
>> + %sub652 = fsub float %add626, %sub639
>> + store float %sub652, float* %arrayidx36, align 4
>> + br i1 undef, label %if.else1609, label %if.then1595
>> +
>> +if.then1595: ; preds = %entry
>> + br i1 undef, label %return, label %for.body.lr.ph.i.i1702
>> +
>> +for.body.lr.ph.i.i1702: ; preds = %if.then1595
>> + unreachable
>> +
>> +if.else1609: ; preds = %entry
>> + unreachable
>> +
>> +return: ; preds = %if.then1595
>> + ret void
>> +}
>> +
>>
>> Added: llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_dequeue.ll
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_dequeue.ll?rev=185774&view=auto
>>
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_dequeue.ll (added)
>> +++ llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_dequeue.ll Sun Jul
>> 7 01:57:07 2013
>> @@ -0,0 +1,40 @@
>> +; RUN: opt < %s -basicaa -slp-vectorizer -dce -S
>> -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7
>> +
>> +target datalayout =
>> "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
>> +target triple = "x86_64-apple-macosx10.8.0"
>> +%"struct.std::_Deque_iterator.4.157.174.208.259.276.344.731" = type {
>> double*, double*, double*, double** }
>> +
>> +; Function Attrs: nounwind ssp uwtable
>> +define void
>> @_ZSt6uniqueISt15_Deque_iteratorIdRdPdEET_S4_S4_(%"struct.std::_Deque_iterator.4.157.174.208.259.276.344.731"*
>> %__first, %"struct.std::_Deque_iterator.4.157.174.208.259.276.344.731"*
>> nocapture %__last) {
>> +entry:
>> + %_M_cur2.i.i = getelementptr inbounds
>> %"struct.std::_Deque_iterator.4.157.174.208.259.276.344.731"* %__first, i64
>> 0, i32 0
>> + %0 = load double** %_M_cur2.i.i, align 8
>> + %_M_first3.i.i = getelementptr inbounds
>> %"struct.std::_Deque_iterator.4.157.174.208.259.276.344.731"* %__first, i64
>> 0, i32 1
>> + %_M_cur2.i.i81 = getelementptr inbounds
>> %"struct.std::_Deque_iterator.4.157.174.208.259.276.344.731"* %__last, i64
>> 0, i32 0
>> + %1 = load double** %_M_cur2.i.i81, align 8
>> + %_M_first3.i.i83 = getelementptr inbounds
>> %"struct.std::_Deque_iterator.4.157.174.208.259.276.344.731"* %__last, i64
>> 0, i32 1
>> + %2 = load double** %_M_first3.i.i83, align 8
>> + br i1 undef, label
>> %_ZSt13adjacent_findISt15_Deque_iteratorIdRdPdEET_S4_S4_.exit, label
>> %while.cond.i.preheader
>> +
>> +while.cond.i.preheader: ; preds = %entry
>> + br label %while.cond.i
>> +
>> +while.cond.i: ; preds =
>> %while.body.i, %while.cond.i.preheader
>> + br i1 undef, label
>> %_ZSt13adjacent_findISt15_Deque_iteratorIdRdPdEET_S4_S4_.exit, label
>> %while.body.i
>> +
>> +while.body.i: ; preds = %while.cond.i
>> + br i1 undef, label
>> %_ZSt13adjacent_findISt15_Deque_iteratorIdRdPdEET_S4_S4_.exit, label
>> %while.cond.i
>> +
>> +_ZSt13adjacent_findISt15_Deque_iteratorIdRdPdEET_S4_S4_.exit: ; preds =
>> %while.body.i, %while.cond.i, %entry
>> + %3 = phi double* [ %2, %entry ], [ %2, %while.cond.i ], [ undef,
>> %while.body.i ]
>> + %4 = phi double* [ %0, %entry ], [ %1, %while.cond.i ], [ undef,
>> %while.body.i ]
>> + store double* %4, double** %_M_cur2.i.i, align 8
>> + store double* %3, double** %_M_first3.i.i, align 8
>> + br i1 undef, label %if.then.i55, label %while.cond
>> +
>> +if.then.i55: ; preds =
>> %_ZSt13adjacent_findISt15_Deque_iteratorIdRdPdEET_S4_S4_.exit
>> + br label %while.cond
>> +
>> +while.cond: ; preds = %while.cond,
>> %if.then.i55, %_ZSt13adjacent_findISt15_Deque_iteratorIdRdPdEET_S4_S4_.exit
>> + br label %while.cond
>> +}
>>
>> Added: llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_flop7.ll
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_flop7.ll?rev=185774&view=auto
>>
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_flop7.ll (added)
>> +++ llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_flop7.ll Sun Jul
>> 7 01:57:07 2013
>> @@ -0,0 +1,46 @@
>> +; RUN: opt < %s -basicaa -slp-vectorizer -dce -S
>> -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7
>> +
>> +target datalayout =
>> "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
>> +target triple = "x86_64-apple-macosx10.8.0"
>> +
>> +; Function Attrs: nounwind ssp uwtable
>> +define void @main() #0 {
>> +entry:
>> + br i1 undef, label %while.body, label %while.end
>> +
>> +while.body: ; preds = %entry
>> + unreachable
>> +
>> +while.end: ; preds = %entry
>> + br i1 undef, label %for.end80, label %for.body75.lr.ph
>> +
>> +for.body75.lr.ph: ; preds = %while.end
>> + br label %for.body75
>> +
>> +for.body75: ; preds = %for.body75,
>> %for.body75.lr.ph
>> + br label %for.body75
>> +
>> +for.end80: ; preds = %while.end
>> + br i1 undef, label %for.end300, label %for.body267.lr.ph
>> +
>> +for.body267.lr.ph: ; preds = %for.end80
>> + br label %for.body267
>> +
>> +for.body267: ; preds =
>> %for.body267, %for.body267.lr.ph
>> + %s.71010 = phi double [ 0.000000e+00, %for.body267.lr.ph ], [
>> %add297, %for.body267 ]
>> + %mul269 = fmul double undef, undef
>> + %mul270 = fmul double %mul269, %mul269
>> + %add282 = fadd double undef, undef
>> + %mul283 = fmul double %mul269, %add282
>> + %add293 = fadd double undef, undef
>> + %mul294 = fmul double %mul270, %add293
>> + %add295 = fadd double undef, %mul294
>> + %div296 = fdiv double %mul283, %add295
>> + %add297 = fadd double %s.71010, %div296
>> + br i1 undef, label %for.body267, label %for.end300
>> +
>> +for.end300: ; preds =
>> %for.body267, %for.end80
>> + unreachable
>> +}
>> +
>> +attributes #0 = { nounwind ssp uwtable "less-precise-fpmad"="false"
>> "no-frame-pointer-elim"="false" "no-frame-pointer-elim-non-leaf"="false"
>> "no-infs-fp-math"="false" "no-nans-fp-math"="false"
>> "unsafe-fp-math"="false" "use-soft-float"="false" }
>>
>> Added: llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_lame.ll
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_lame.ll?rev=185774&view=auto
>>
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_lame.ll (added)
>> +++ llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_lame.ll Sun Jul 7
>> 01:57:07 2013
>> @@ -0,0 +1,24 @@
>> +; RUN: opt < %s -basicaa -slp-vectorizer -dce -S
>> -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7
>> +
>> +target datalayout =
>> "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
>> +target triple = "x86_64-apple-macosx10.8.0"
>> +
>> +; Function Attrs: nounwind ssp uwtable
>> +define fastcc void @dct36(double* %inbuf) #0 {
>> +entry:
>> + %arrayidx41 = getelementptr inbounds double* %inbuf, i64 2
>> + %arrayidx44 = getelementptr inbounds double* %inbuf, i64 1
>> + %0 = load double* %arrayidx44, align 8, !tbaa !0
>> + %add46 = fadd double %0, undef
>> + store double %add46, double* %arrayidx41, align 8, !tbaa !0
>> + %1 = load double* %inbuf, align 8, !tbaa !0
>> + %add49 = fadd double %1, %0
>> + store double %add49, double* %arrayidx44, align 8, !tbaa !0
>> + ret void
>> +}
>> +
>> +attributes #0 = { nounwind ssp uwtable "less-precise-fpmad"="false"
>> "no-frame-pointer-elim"="false" "no-frame-pointer-elim-non-leaf"="false"
>> "no-infs-fp-math"="false" "no-nans-fp-math"="false"
>> "unsafe-fp-math"="false" "use-soft-float"="false" }
>> +
>> +!0 = metadata !{metadata !"double", metadata !1}
>> +!1 = metadata !{metadata !"omnipotent char", metadata !2}
>> +!2 = metadata !{metadata !"Simple C/C++ TBAA"}
>>
>> Added: llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_lencod.ll
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_lencod.ll?rev=185774&view=auto
>>
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_lencod.ll (added)
>> +++ llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_lencod.ll Sun Jul
>> 7 01:57:07 2013
>> @@ -0,0 +1,66 @@
>> +; RUN: opt < %s -basicaa -slp-vectorizer -dce -S
>> -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7
>> +
>> +target datalayout =
>> "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
>> +target triple = "x86_64-apple-macosx10.8.0"
>> +
>> +; Function Attrs: nounwind ssp uwtable
>> +define void @RCModelEstimator() {
>> +entry:
>> + br i1 undef, label %for.body.lr.ph, label %for.end.thread
>> +
>> +for.end.thread: ; preds = %entry
>> + unreachable
>> +
>> +for.body.lr.ph: ; preds = %entry
>> + br i1 undef, label %for.end, label %for.body
>> +
>> +for.body: ; preds = %for.body, %
>> for.body.lr.ph
>> + br i1 undef, label %for.end, label %for.body
>> +
>> +for.end: ; preds = %for.body, %
>> for.body.lr.ph
>> + br i1 undef, label %for.body3, label %if.end103
>> +
>> +for.cond14.preheader: ; preds = %for.inc11
>> + br i1 undef, label %for.body16.lr.ph, label %if.end103
>> +
>> +for.body16.lr.ph: ; preds =
>> %for.cond14.preheader
>> + br label %for.body16
>> +
>> +for.body3: ; preds = %for.inc11,
>> %for.end
>> + br i1 undef, label %if.then7, label %for.inc11
>> +
>> +if.then7: ; preds = %for.body3
>> + br label %for.inc11
>> +
>> +for.inc11: ; preds = %if.then7,
>> %for.body3
>> + br i1 false, label %for.cond14.preheader, label %for.body3
>> +
>> +for.body16: ; preds = %for.body16,
>> %for.body16.lr.ph
>> + br i1 undef, label %for.end39, label %for.body16
>> +
>> +for.end39: ; preds = %for.body16
>> + br i1 undef, label %if.end103, label %for.cond45.preheader
>> +
>> +for.cond45.preheader: ; preds = %for.end39
>> + br i1 undef, label %if.then88, label %if.else
>> +
>> +if.then88: ; preds =
>> %for.cond45.preheader
>> + %mul89 = fmul double 0.000000e+00, 0.000000e+00
>> + %mul90 = fmul double 0.000000e+00, 0.000000e+00
>> + %sub91 = fsub double %mul89, %mul90
>> + %div92 = fdiv double %sub91, undef
>> + %mul94 = fmul double 0.000000e+00, 0.000000e+00
>> + %mul95 = fmul double 0.000000e+00, 0.000000e+00
>> + %sub96 = fsub double %mul94, %mul95
>> + %div97 = fdiv double %sub96, undef
>> + br label %if.end103
>> +
>> +if.else: ; preds =
>> %for.cond45.preheader
>> + br label %if.end103
>> +
>> +if.end103: ; preds = %if.else,
>> %if.then88, %for.end39, %for.cond14.preheader, %for.end
>> + %0 = phi double [ 0.000000e+00, %for.end39 ], [ %div97, %if.then88 ],
>> [ 0.000000e+00, %if.else ], [ 0.000000e+00, %for.cond14.preheader ], [
>> 0.000000e+00, %for.end ]
>> + %1 = phi double [ undef, %for.end39 ], [ %div92, %if.then88 ], [
>> undef, %if.else ], [ 0.000000e+00, %for.cond14.preheader ], [ 0.000000e+00,
>> %for.end ]
>> + ret void
>> +}
>> +
>>
>> Added: llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_lencod2.ll
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_lencod2.ll?rev=185774&view=auto
>>
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_lencod2.ll (added)
>> +++ llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_lencod2.ll Sun Jul
>> 7 01:57:07 2013
>> @@ -0,0 +1,23 @@
>> +; RUN: opt < %s -basicaa -slp-vectorizer -dce -S
>> -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7
>> +
>> +target datalayout =
>> "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
>> +target triple = "x86_64-apple-macosx10.8.0"
>> +
>> +; Function Attrs: nounwind ssp uwtable
>> +define void @intrapred_luma() #0 {
>> +entry:
>> + %conv153 = trunc i32 undef to i16
>> + %arrayidx154 = getelementptr inbounds [13 x i16]* undef, i64 0, i64 12
>> + store i16 %conv153, i16* %arrayidx154, align 8, !tbaa !0
>> + %arrayidx155 = getelementptr inbounds [13 x i16]* undef, i64 0, i64 11
>> + store i16 %conv153, i16* %arrayidx155, align 2, !tbaa !0
>> + %arrayidx156 = getelementptr inbounds [13 x i16]* undef, i64 0, i64 10
>> + store i16 %conv153, i16* %arrayidx156, align 4, !tbaa !0
>> + ret void
>> +}
>> +
>> +attributes #0 = { nounwind ssp uwtable "less-precise-fpmad"="false"
>> "no-frame-pointer-elim"="false" "no-frame-pointer-elim-non-leaf"="false"
>> "no-infs-fp-math"="false" "no-nans-fp-math"="false"
>> "unsafe-fp-math"="false" "use-soft-float"="false" }
>> +
>> +!0 = metadata !{metadata !"short", metadata !1}
>> +!1 = metadata !{metadata !"omnipotent char", metadata !2}
>> +!2 = metadata !{metadata !"Simple C/C++ TBAA"}
>>
>> Added: llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_mandeltext.ll
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_mandeltext.ll?rev=185774&view=auto
>>
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_mandeltext.ll
>> (added)
>> +++ llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_mandeltext.ll Sun
>> Jul 7 01:57:07 2013
>> @@ -0,0 +1,53 @@
>> +; RUN: opt < %s -basicaa -slp-vectorizer -dce -S
>> -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7
>> +
>> +target datalayout =
>> "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
>> +target triple = "x86_64-apple-macosx10.8.0"
>> +
>> +define void @main() {
>> +entry:
>> + br label %for.body
>> +
>> +for.body: ; preds = %for.end44,
>> %entry
>> + br label %for.cond4.preheader
>> +
>> +for.cond4.preheader: ; preds = %if.then25,
>> %for.body
>> + br label %for.body6
>> +
>> +for.body6: ; preds = %for.inc21,
>> %for.cond4.preheader
>> + br label %for.body12
>> +
>> +for.body12: ; preds = %if.end,
>> %for.body6
>> + %fZImg.069 = phi double [ undef, %for.body6 ], [ %add19, %if.end ]
>> + %fZReal.068 = phi double [ undef, %for.body6 ], [ %add20, %if.end ]
>> + %mul13 = fmul double %fZReal.068, %fZReal.068
>> + %mul14 = fmul double %fZImg.069, %fZImg.069
>> + %add15 = fadd double %mul13, %mul14
>> + %cmp16 = fcmp ogt double %add15, 4.000000e+00
>> + br i1 %cmp16, label %for.inc21, label %if.end
>> +
>> +if.end: ; preds = %for.body12
>> + %mul18 = fmul double undef, %fZImg.069
>> + %add19 = fadd double undef, %mul18
>> + %sub = fsub double %mul13, %mul14
>> + %add20 = fadd double undef, %sub
>> + br i1 undef, label %for.body12, label %for.inc21
>> +
>> +for.inc21: ; preds = %if.end,
>> %for.body12
>> + br i1 undef, label %for.end23, label %for.body6
>> +
>> +for.end23: ; preds = %for.inc21
>> + br i1 undef, label %if.then25, label %if.then26
>> +
>> +if.then25: ; preds = %for.end23
>> + br i1 undef, label %for.end44, label %for.cond4.preheader
>> +
>> +if.then26: ; preds = %for.end23
>> + unreachable
>> +
>> +for.end44: ; preds = %if.then25
>> + br i1 undef, label %for.end48, label %for.body
>> +
>> +for.end48: ; preds = %for.end44
>> + ret void
>> +}
>> +
>>
>> Added: llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_rc4.ll
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_rc4.ll?rev=185774&view=auto
>>
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_rc4.ll (added)
>> +++ llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_rc4.ll Sun Jul 7
>> 01:57:07 2013
>> @@ -0,0 +1,28 @@
>> +; RUN: opt < %s -basicaa -slp-vectorizer -dce -S
>> -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7
>> +
>> +target datalayout =
>> "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
>> +target triple = "x86_64-apple-macosx10.8.0"
>> +
>> +%struct.rc4_state.0.24 = type { i32, i32, [256 x i32] }
>> +
>> +define void @rc4_crypt(%struct.rc4_state.0.24* nocapture %s) {
>> +entry:
>> + %x1 = getelementptr inbounds %struct.rc4_state.0.24* %s, i64 0, i32 0
>> + %y2 = getelementptr inbounds %struct.rc4_state.0.24* %s, i64 0, i32 1
>> + br i1 undef, label %for.body, label %for.end
>> +
>> +for.body: ; preds = %for.body,
>> %entry
>> + %x.045 = phi i32 [ %conv4, %for.body ], [ undef, %entry ]
>> + %conv4 = and i32 undef, 255
>> + %conv7 = and i32 undef, 255
>> + %idxprom842 = zext i32 %conv7 to i64
>> + br i1 undef, label %for.end, label %for.body
>> +
>> +for.end: ; preds = %for.body,
>> %entry
>> + %x.0.lcssa = phi i32 [ undef, %entry ], [ %conv4, %for.body ]
>> + %y.0.lcssa = phi i32 [ undef, %entry ], [ %conv7, %for.body ]
>> + store i32 %x.0.lcssa, i32* %x1, align 4
>> + store i32 %y.0.lcssa, i32* %y2, align 4
>> + ret void
>> +}
>> +
>>
>> Added: llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_sim4b1.ll
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_sim4b1.ll?rev=185774&view=auto
>>
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_sim4b1.ll (added)
>> +++ llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_sim4b1.ll Sun Jul
>> 7 01:57:07 2013
>> @@ -0,0 +1,113 @@
>> +; RUN: opt < %s -basicaa -slp-vectorizer -dce -S
>> -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7
>> +
>> +target datalayout =
>> "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
>> +target triple = "x86_64-apple-macosx10.8.0"
>> +
>> +%struct._exon_t.12.103.220.363.480.649.740.857.1039.1065.1078.1091.1117.1130.1156.1169.1195.1221.1234.1286.1299.1312.1338.1429.1455.1468.1494.1520.1884.1897.1975.2066.2105.2170.2171
>> = type { i32, i32, i32, i32, i32, i32, [8 x i8] }
>> +
>> +define void @SIM4() {
>> +entry:
>> + br i1 undef, label %return, label %lor.lhs.false
>> +
>> +lor.lhs.false: ; preds = %entry
>> + br i1 undef, label %return, label %if.end
>> +
>> +if.end: ; preds =
>> %lor.lhs.false
>> + br i1 undef, label %for.end605, label %for.body.lr.ph
>> +
>> +for.body.lr.ph: ; preds = %if.end
>> + br label %for.body
>> +
>> +for.body: ; preds = %for.inc603,
>> %for.body.lr.ph
>> + br i1 undef, label %for.inc603, label %if.end12
>> +
>> +if.end12: ; preds = %for.body
>> + br i1 undef, label %land.lhs.true, label %land.lhs.true167
>> +
>> +land.lhs.true: ; preds = %if.end12
>> + br i1 undef, label %if.then17, label %land.lhs.true167
>> +
>> +if.then17: ; preds =
>> %land.lhs.true
>> + br i1 undef, label %if.end98, label %land.rhs.lr.ph
>> +
>> +land.rhs.lr.ph: ; preds = %if.then17
>> + unreachable
>> +
>> +if.end98: ; preds = %if.then17
>> + %from299 = getelementptr inbounds
>> %struct._exon_t.12.103.220.363.480.649.740.857.1039.1065.1078.1091.1117.1130.1156.1169.1195.1221.1234.1286.1299.1312.1338.1429.1455.1468.1494.1520.1884.1897.1975.2066.2105.2170.2171*
>> undef, i64 0, i32 1
>> + br i1 undef, label %land.lhs.true167, label %if.then103
>> +
>> +if.then103: ; preds = %if.end98
>> + %.sub100 = select i1 undef, i32 250, i32 undef
>> + %mul114 = shl nsw i32 %.sub100, 2
>> + %from1115 = getelementptr inbounds
>> %struct._exon_t.12.103.220.363.480.649.740.857.1039.1065.1078.1091.1117.1130.1156.1169.1195.1221.1234.1286.1299.1312.1338.1429.1455.1468.1494.1520.1884.1897.1975.2066.2105.2170.2171*
>> undef, i64 0, i32 0
>> + %cond125 = select i1 undef, i32 undef, i32 %mul114
>> + br label %for.cond.i
>> +
>> +for.cond.i: ; preds =
>> %land.rhs.i874, %if.then103
>> + %row.0.i = phi i32 [ undef, %land.rhs.i874 ], [ %.sub100, %if.then103 ]
>> + %col.0.i = phi i32 [ undef, %land.rhs.i874 ], [ %cond125, %if.then103 ]
>> + br i1 undef, label %land.rhs.i874, label %for.end.i
>> +
>> +land.rhs.i874: ; preds = %for.cond.i
>> + br i1 undef, label %for.cond.i, label %for.end.i
>> +
>> +for.end.i: ; preds =
>> %land.rhs.i874, %for.cond.i
>> + br i1 undef, label %if.then.i, label %if.end.i
>> +
>> +if.then.i: ; preds = %for.end.i
>> + %add14.i = add nsw i32 %row.0.i, undef
>> + %add15.i = add nsw i32 %col.0.i, undef
>> + br label %extend_bw.exit
>> +
>> +if.end.i: ; preds = %for.end.i
>> + %add16.i = add i32 %cond125, %.sub100
>> + %cmp26514.i = icmp slt i32 %add16.i, 0
>> + br i1 %cmp26514.i, label %for.end33.i, label %for.body28.lr.ph.i
>> +
>> +for.body28.lr.ph.i: ; preds = %if.end.i
>> + br label %for.end33.i
>> +
>> +for.end33.i: ; preds =
>> %for.body28.lr.ph.i, %if.end.i
>> + br i1 undef, label %for.end58.i, label %for.body52.lr.ph.i
>> +
>> +for.body52.lr.ph.i: ; preds = %for.end33.i
>> + br label %for.end58.i
>> +
>> +for.end58.i: ; preds =
>> %for.body52.lr.ph.i, %for.end33.i
>> + br label %while.cond260.i
>> +
>> +while.cond260.i: ; preds =
>> %land.rhs263.i, %for.end58.i
>> + br i1 undef, label %land.rhs263.i, label %while.end275.i
>> +
>> +land.rhs263.i: ; preds =
>> %while.cond260.i
>> + br i1 undef, label %while.cond260.i, label %while.end275.i
>> +
>> +while.end275.i: ; preds =
>> %land.rhs263.i, %while.cond260.i
>> + br label %extend_bw.exit
>> +
>> +extend_bw.exit: ; preds =
>> %while.end275.i, %if.then.i
>> + %add14.i1262 = phi i32 [ %add14.i, %if.then.i ], [ undef,
>> %while.end275.i ]
>> + %add15.i1261 = phi i32 [ %add15.i, %if.then.i ], [ undef,
>> %while.end275.i ]
>> + br i1 false, label %if.then157, label %land.lhs.true167
>> +
>> +if.then157: ; preds =
>> %extend_bw.exit
>> + %add158 = add nsw i32 %add14.i1262, 1
>> + store i32 %add158, i32* %from299, align 4
>> + %add160 = add nsw i32 %add15.i1261, 1
>> + store i32 %add160, i32* %from1115, align 4
>> + br label %land.lhs.true167
>> +
>> +land.lhs.true167: ; preds = %if.then157,
>> %extend_bw.exit, %if.end98, %land.lhs.true, %if.end12
>> + unreachable
>> +
>> +for.inc603: ; preds = %for.body
>> + br i1 undef, label %for.body, label %for.end605
>> +
>> +for.end605: ; preds = %for.inc603,
>> %if.end
>> + unreachable
>> +
>> +return: ; preds =
>> %lor.lhs.false, %entry
>> + ret void
>> +}
>> +
>>
>> Added: llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_smallpt.ll
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_smallpt.ll?rev=185774&view=auto
>>
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_smallpt.ll (added)
>> +++ llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_smallpt.ll Sun Jul
>> 7 01:57:07 2013
>> @@ -0,0 +1,65 @@
>> +; RUN: opt < %s -basicaa -slp-vectorizer -dce -S
>> -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7
>> +
>> +target datalayout =
>> "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
>> +target triple = "x86_64-apple-macosx10.8.0"
>> +
>> +%struct.Ray.5.11.53.113.119.137.149.185.329.389.415 = type {
>> %struct.Vec.0.6.48.108.114.132.144.180.324.384.414,
>> %struct.Vec.0.6.48.108.114.132.144.180.324.384.414 }
>> +%struct.Vec.0.6.48.108.114.132.144.180.324.384.414 = type { double,
>> double, double }
>> +
>> +; Function Attrs: ssp uwtable
>> +define void @main() #0 {
>> +entry:
>> + br i1 undef, label %cond.true, label %cond.end
>> +
>> +cond.true: ; preds = %entry
>> + unreachable
>> +
>> +cond.end: ; preds = %entry
>> + br label %invoke.cont
>> +
>> +invoke.cont: ; preds =
>> %invoke.cont, %cond.end
>> + br i1 undef, label %arrayctor.cont, label %invoke.cont
>> +
>> +arrayctor.cont: ; preds = %invoke.cont
>> + %agg.tmp99208.sroa.0.0.idx = getelementptr inbounds
>> %struct.Ray.5.11.53.113.119.137.149.185.329.389.415* undef, i64 0, i32 0,
>> i32 0
>> + %agg.tmp99208.sroa.1.8.idx388 = getelementptr inbounds
>> %struct.Ray.5.11.53.113.119.137.149.185.329.389.415* undef, i64 0, i32 0,
>> i32 1
>> + %agg.tmp101211.sroa.0.0.idx = getelementptr inbounds
>> %struct.Ray.5.11.53.113.119.137.149.185.329.389.415* undef, i64 0, i32 1,
>> i32 0
>> + %agg.tmp101211.sroa.1.8.idx390 = getelementptr inbounds
>> %struct.Ray.5.11.53.113.119.137.149.185.329.389.415* undef, i64 0, i32 1,
>> i32 1
>> + br label %for.cond36.preheader
>> +
>> +for.cond36.preheader: ; preds =
>> %_Z5clampd.exit.1, %arrayctor.cont
>> + br i1 undef, label %for.body42.lr.ph.us, label %_Z5clampd.exit.1
>> +
>> +cond.false51.us: ; preds = %
>> for.body42.lr.ph.us
>> + unreachable
>> +
>> +cond.true48.us: ; preds = %
>> for.body42.lr.ph.us
>> + br i1 undef, label %cond.true63.us, label %cond.false66.us
>> +
>> +cond.false66.us: ; preds = %
>> cond.true48.us
>> + %add.i276.us = fadd double 0.000000e+00, undef
>> + %add.i264.us = fadd double %add.i276.us, 0.000000e+00
>> + %add4.i267.us = fadd double undef, 0xBFA5CC2D1960285F
>> + %mul.i254.us = fmul double %add.i264.us, 1.400000e+02
>> + %mul2.i256.us = fmul double %add4.i267.us, 1.400000e+02
>> + %add.i243.us = fadd double %mul.i254.us, 5.000000e+01
>> + %add4.i246.us = fadd double %mul2.i256.us, 5.200000e+01
>> + %mul.i.i.us = fmul double undef, %add.i264.us
>> + %mul2.i.i.us = fmul double undef, %add4.i267.us
>> + store double %add.i243.us, double* %agg.tmp99208.sroa.0.0.idx, align 8
>> + store double %add4.i246.us, double* %agg.tmp99208.sroa.1.8.idx388,
>> align 8
>> + store double %mul.i.i.us, double* %agg.tmp101211.sroa.0.0.idx, align 8
>> + store double %mul2.i.i.us, double* %agg.tmp101211.sroa.1.8.idx390,
>> align 8
>> + unreachable
>> +
>> +cond.true63.us: ; preds = %
>> cond.true48.us
>> + unreachable
>> +
>> +for.body42.lr.ph.us: ; preds =
>> %for.cond36.preheader
>> + br i1 undef, label %cond.true48.us, label %cond.false51.us
>> +
>> +_Z5clampd.exit.1: ; preds =
>> %for.cond36.preheader
>> + br label %for.cond36.preheader
>> +}
>> +
>> +attributes #0 = { ssp uwtable "less-precise-fpmad"="false"
>> "no-frame-pointer-elim"="false" "no-frame-pointer-elim-non-leaf"="false"
>> "no-infs-fp-math"="false" "no-nans-fp-math"="false"
>> "unsafe-fp-math"="false" "use-soft-float"="false" }
>>
>> Added: llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_smallpt2.ll
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_smallpt2.ll?rev=185774&view=auto
>>
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_smallpt2.ll (added)
>> +++ llvm/trunk/test/Transforms/SLPVectorizer/X86/crash_smallpt2.ll Sun
>> Jul 7 01:57:07 2013
>> @@ -0,0 +1,46 @@
>> +; RUN: opt < %s -basicaa -slp-vectorizer -dce -S
>> -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7
>> +
>> +target datalayout =
>> "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
>> +target triple = "x86_64-apple-macosx10.8.0"
>> +
>> +%struct.Ray.5.11.53.95.137.191.197.203.239.257.263.269.275.281.287.293.383.437.443.455.461.599.601
>> = type {
>> %struct.Vec.0.6.48.90.132.186.192.198.234.252.258.264.270.276.282.288.378.432.438.450.456.594.600,
>> %struct.Vec.0.6.48.90.132.186.192.198.234.252.258.264.270.276.282.288.378.432.438.450.456.594.600
>> }
>> +%struct.Vec.0.6.48.90.132.186.192.198.234.252.258.264.270.276.282.288.378.432.438.450.456.594.600
>> = type { double, double, double }
>> +
>> +; Function Attrs: ssp uwtable
>> +define void @_Z8radianceRK3RayiPt() #0 {
>> +entry:
>> + br i1 undef, label %if.then78, label %if.then38
>> +
>> +if.then38: ; preds = %entry
>> + %mul.i.i790 = fmul double undef, undef
>> + %mul3.i.i792 = fmul double undef, undef
>> + %mul.i764 = fmul double undef, %mul3.i.i792
>> + %mul4.i767 = fmul double undef, undef
>> + %sub.i768 = fsub double %mul.i764, %mul4.i767
>> + %mul6.i770 = fmul double undef, %mul.i.i790
>> + %mul9.i772 = fmul double undef, %mul3.i.i792
>> + %sub10.i773 = fsub double %mul6.i770, %mul9.i772
>> + %mul.i736 = fmul double undef, %sub.i768
>> + %mul2.i738 = fmul double undef, %sub10.i773
>> + %mul.i727 = fmul double undef, %mul.i736
>> + %mul2.i729 = fmul double undef, %mul2.i738
>> + %add.i716 = fadd double undef, %mul.i727
>> + %add4.i719 = fadd double undef, %mul2.i729
>> + %add.i695 = fadd double undef, %add.i716
>> + %add4.i698 = fadd double undef, %add4.i719
>> + %mul.i.i679 = fmul double undef, %add.i695
>> + %mul2.i.i680 = fmul double undef, %add4.i698
>> + %agg.tmp74663.sroa.0.0.idx = getelementptr inbounds
>> %struct.Ray.5.11.53.95.137.191.197.203.239.257.263.269.275.281.287.293.383.437.443.455.461.599.601*
>> undef, i64 0, i32 1, i32 0
>> + store double %mul.i.i679, double* %agg.tmp74663.sroa.0.0.idx, align 8
>> + %agg.tmp74663.sroa.1.8.idx943 = getelementptr inbounds
>> %struct.Ray.5.11.53.95.137.191.197.203.239.257.263.269.275.281.287.293.383.437.443.455.461.599.601*
>> undef, i64 0, i32 1, i32 1
>> + store double %mul2.i.i680, double* %agg.tmp74663.sroa.1.8.idx943,
>> align 8
>> + br label %return
>> +
>> +if.then78: ; preds = %entry
>> + br label %return
>> +
>> +return: ; preds = %if.then78,
>> %if.then38
>> + ret void
>> +}
>> +
>> +attributes #0 = { ssp uwtable "less-precise-fpmad"="false"
>> "no-frame-pointer-elim"="false" "no-frame-pointer-elim-non-leaf"="false"
>> "no-infs-fp-math"="false" "no-nans-fp-math"="false"
>> "unsafe-fp-math"="false" "use-soft-float"="false" }
>>
>> Modified: llvm/trunk/test/Transforms/SLPVectorizer/X86/diamond.ll
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/diamond.ll?rev=185774&r1=185773&r2=185774&view=diff
>>
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/SLPVectorizer/X86/diamond.ll (original)
>> +++ llvm/trunk/test/Transforms/SLPVectorizer/X86/diamond.ll Sun Jul 7
>> 01:57:07 2013
>> @@ -50,7 +50,8 @@ entry:
>> ; }
>>
>> ; CHECK: @extr_user
>> -; CHECK: load i32*
>> +; CHECK: load <4 x i32>
>> +; CHECK-NEXT: extractelement <4 x i32>
>> ; CHECK: store <4 x i32>
>> ; CHECK-NEXT: ret
>> define i32 @extr_user(i32* noalias nocapture %B, i32* noalias nocapture
>> %A, i32 %n, i32 %m) {
>> @@ -79,7 +80,8 @@ entry:
>>
>> ; In this example we have an external user that is not the first element
>> in the vector.
>> ; CHECK: @extr_user1
>> -; CHECK: load i32*
>> +; CHECK: load <4 x i32>
>> +; CHECK-NEXT: extractelement <4 x i32>
>> ; CHECK: store <4 x i32>
>> ; CHECK-NEXT: ret
>> define i32 @extr_user1(i32* noalias nocapture %B, i32* noalias nocapture
>> %A, i32 %n, i32 %m) {
>>
>> Modified: llvm/trunk/test/Transforms/SLPVectorizer/X86/long_chains.ll
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/long_chains.ll?rev=185774&r1=185773&r2=185774&view=diff
>>
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/SLPVectorizer/X86/long_chains.ll (original)
>> +++ llvm/trunk/test/Transforms/SLPVectorizer/X86/long_chains.ll Sun Jul
>> 7 01:57:07 2013
>> @@ -3,12 +3,13 @@
>> target datalayout =
>> "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
>> target triple = "x86_64-apple-macosx10.8.0"
>>
>> +; At this point we can't vectorize only parts of the tree.
>> +
>> ; CHECK: test
>> -; CHECK: sitofp i8
>> -; CHECK-NEXT: sitofp i8
>> -; CHECK-NEXT: insertelement
>> -; CHECK-NEXT: insertelement
>> -; CHECK-NEXT: fmul <2 x double>
>> +; CHECK: insertelement <2 x i8>
>> +; CHECK: insertelement <2 x i8>
>> +; CHECK: sitofp <2 x i8>
>> +; CHECK: fmul <2 x double>
>> ; CHECK: ret
>> define i32 @test(double* nocapture %A, i8* nocapture %B) {
>> entry:
>> @@ -18,7 +19,7 @@ entry:
>> %add = add i8 %0, 3
>> %add4 = add i8 %1, 3
>> %conv6 = sitofp i8 %add to double
>> - %conv7 = sitofp i8 %add4 to double ; <--- This is inefficient. The
>> chain stops here.
>> + %conv7 = sitofp i8 %add4 to double
>> %mul = fmul double %conv6, %conv6
>> %add8 = fadd double %mul, 1.000000e+00
>> %mul9 = fmul double %conv7, %conv7
>>
>> Modified: llvm/trunk/test/Transforms/SLPVectorizer/X86/saxpy.ll
>> URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SLPVectorizer/X86/saxpy.ll?rev=185774&r1=185773&r2=185774&view=diff
>>
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/SLPVectorizer/X86/saxpy.ll (original)
>> +++ llvm/trunk/test/Transforms/SLPVectorizer/X86/saxpy.ll Sun Jul 7
>> 01:57:07 2013
>> @@ -43,3 +43,19 @@ define void @SAXPY(i32* noalias nocaptur
>> ret void
>> }
>>
>> +; Make sure we don't crash on this one.
>> +define void @SAXPY_crash(i32* noalias nocapture %x, i32* noalias
>> nocapture %y, i64 %i) {
>> + %1 = add i64 %i, 1
>> + %2 = getelementptr inbounds i32* %x, i64 %1
>> + %3 = getelementptr inbounds i32* %y, i64 %1
>> + %4 = load i32* %3, align 4
>> + %5 = add nsw i32 undef, %4
>> + store i32 %5, i32* %2, align 4
>> + %6 = add i64 %i, 2
>> + %7 = getelementptr inbounds i32* %x, i64 %6
>> + %8 = getelementptr inbounds i32* %y, i64 %6
>> + %9 = load i32* %8, align 4
>> + %10 = add nsw i32 undef, %9
>> + store i32 %10, i32* %7, align 4
>> + ret void
>> +}
>>
>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>>
>
>
>
> --
> Alexey Samsonov, MSK
>
>
>
--
Alexey Samsonov, MSK
_______________________________________________
llvm-commits mailing list
llvm-commits at cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20130708/a622215f/attachment.html>
More information about the llvm-commits
mailing list