<div dir="ltr">Reverted by r298329</div><br><div class="gmail_quote"><div dir="ltr">On Mon, Mar 20, 2017 at 5:27 PM Vitaly Buka <<a href="mailto:vitalybuka@google.com">vitalybuka@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg">Should we just revert this?</div><br class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Mon, Mar 20, 2017 at 3:05 PM Krzysztof Parzyszek <<a href="mailto:kparzysz@codeaurora.org" class="gmail_msg" target="_blank">kparzysz@codeaurora.org</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Sure. Checking...<br class="gmail_msg">
<br class="gmail_msg">
-Krzysztof<br class="gmail_msg">
<br class="gmail_msg">
On 3/20/2017 4:58 PM, Vitaly Buka wrote:<br class="gmail_msg">
> Could you please take a look?<br class="gmail_msg">
> <a href="http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/3574/steps/check-llvm%20asan/logs/stdio" rel="noreferrer" class="gmail_msg" target="_blank">http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/3574/steps/check-llvm%20asan/logs/stdio</a><br class="gmail_msg">
><br class="gmail_msg">
> Direct leak of 136 byte(s) in 1 object(s) allocated from: #0 0x98a100 in<br class="gmail_msg">
> operator new(unsigned long)<br class="gmail_msg">
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/projects/compiler-rt/lib/asan/asan_new_delete.cc:82<br class="gmail_msg">
> #1 0x3d355e5 in allocateFixedOperandUser<br class="gmail_msg">
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/IR/User.cpp:128:7<br class="gmail_msg">
> #2 0x3d355e5 in llvm::User::operator new(unsigned long, unsigned int)<br class="gmail_msg">
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/IR/User.cpp:146<br class="gmail_msg">
> #3 0x3c3422a in Create<br class="gmail_msg">
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/include/llvm/IR/Instructions.h:1962:23<br class="gmail_msg">
> #4 0x3c3422a in llvm::SelectInst::cloneImpl() const<br class="gmail_msg">
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/IR/Instructions.cpp:3929<br class="gmail_msg">
> #5 0x3c01f42 in llvm::Instruction::clone() const<br class="gmail_msg">
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/include/llvm/IR/Instruction.def:187:1<br class="gmail_msg">
> #6 0x1921b77 in initialize<br class="gmail_msg">
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/Target/Hexagon/HexagonLoopIdiomRecognition.cpp:239:28<br class="gmail_msg">
> #7 0x1921b77 in Context<br class="gmail_msg">
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/Target/Hexagon/HexagonLoopIdiomRecognition.cpp:154<br class="gmail_msg">
> #8 0x1921b77 in (anonymous<br class="gmail_msg">
> namespace)::PolynomialMultiplyRecognize::recognize()<br class="gmail_msg">
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/Target/Hexagon/HexagonLoopIdiomRecognition.cpp:1678<br class="gmail_msg">
> #9 0x191e5b0 in runOnCountableLoop<br class="gmail_msg">
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/Target/Hexagon/HexagonLoopIdiomRecognition.cpp:2234:11<br class="gmail_msg">
> #10 0x191e5b0 in (anonymous<br class="gmail_msg">
> namespace)::HexagonLoopIdiomRecognize::runOnLoop(llvm::Loop*,<br class="gmail_msg">
> llvm::LPPassManager&)<br class="gmail_msg">
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/Target/Hexagon/HexagonLoopIdiomRecognition.cpp:2291<br class="gmail_msg">
> #11 0x2dcf8bc in llvm::LPPassManager::runOnFunction(llvm::Function&)<br class="gmail_msg">
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/Analysis/LoopPass.cpp:203:23<br class="gmail_msg">
> #12 0x3c7b90d in llvm::FPPassManager::runOnFunction(llvm::Function&)<br class="gmail_msg">
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/IR/LegacyPassManager.cpp:1513:27<br class="gmail_msg">
> #13 0x3c7beb2 in llvm::FPPassManager::runOnModule(llvm::Module&)<br class="gmail_msg">
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/IR/LegacyPassManager.cpp:1534:16<br class="gmail_msg">
> #14 0x3c7cd4a in runOnModule<br class="gmail_msg">
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/IR/LegacyPassManager.cpp:1590:27<br class="gmail_msg">
> #15 0x3c7cd4a in llvm::legacy::PassManagerImpl::run(llvm::Module&)<br class="gmail_msg">
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/IR/LegacyPassManager.cpp:1693<br class="gmail_msg">
> #16 0x9b4feb in main<br class="gmail_msg">
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/opt/opt.cpp:722:10<br class="gmail_msg">
> #17 0x7f9b7820482f in __libc_start_main<br class="gmail_msg">
> (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> On Mon, Mar 20, 2017 at 11:25 AM Krzysztof Parzyszek via llvm-commits<br class="gmail_msg">
> <<a href="mailto:llvm-commits@lists.llvm.org" class="gmail_msg" target="_blank">llvm-commits@lists.llvm.org</a> <mailto:<a href="mailto:llvm-commits@lists.llvm.org" class="gmail_msg" target="_blank">llvm-commits@lists.llvm.org</a>>> wrote:<br class="gmail_msg">
><br class="gmail_msg">
> Author: kparzysz<br class="gmail_msg">
> Date: Mon Mar 20 13:12:58 2017<br class="gmail_msg">
> New Revision: 298282<br class="gmail_msg">
><br class="gmail_msg">
> URL: <a href="http://llvm.org/viewvc/llvm-project?rev=298282&view=rev" rel="noreferrer" class="gmail_msg" target="_blank">http://llvm.org/viewvc/llvm-project?rev=298282&view=rev</a><br class="gmail_msg">
> Log:<br class="gmail_msg">
> [Hexagon] Recognize polynomial-modulo loop idiom again<br class="gmail_msg">
><br class="gmail_msg">
> Regain the ability to recognize loops calculating polynomial modulo<br class="gmail_msg">
> operation. This ability has been lost due to some changes in the<br class="gmail_msg">
> preceding optimizations. Add code to preprocess the IR to a form<br class="gmail_msg">
> that the pattern matching code can recognize.<br class="gmail_msg">
><br class="gmail_msg">
> Added:<br class="gmail_msg">
> llvm/trunk/test/CodeGen/Hexagon/loop-idiom/pmpy-mod.ll<br class="gmail_msg">
> Modified:<br class="gmail_msg">
> llvm/trunk/lib/Target/Hexagon/HexagonLoopIdiomRecognition.cpp<br class="gmail_msg">
><br class="gmail_msg">
> Modified: llvm/trunk/lib/Target/Hexagon/HexagonLoopIdiomRecognition.cpp<br class="gmail_msg">
> URL:<br class="gmail_msg">
> <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/Hexagon/HexagonLoopIdiomRecognition.cpp?rev=298282&r1=298281&r2=298282&view=diff" rel="noreferrer" class="gmail_msg" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/Hexagon/HexagonLoopIdiomRecognition.cpp?rev=298282&r1=298281&r2=298282&view=diff</a><br class="gmail_msg">
> ==============================================================================<br class="gmail_msg">
> --- llvm/trunk/lib/Target/Hexagon/HexagonLoopIdiomRecognition.cpp<br class="gmail_msg">
> (original)<br class="gmail_msg">
> +++ llvm/trunk/lib/Target/Hexagon/HexagonLoopIdiomRecognition.cpp<br class="gmail_msg">
> Mon Mar 20 13:12:58 2017<br class="gmail_msg">
> @@ -129,6 +129,342 @@ INITIALIZE_PASS_END(HexagonLoopIdiomReco<br class="gmail_msg">
> "Recognize Hexagon-specific loop idioms", false, false)<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> +namespace {<br class="gmail_msg">
> + struct Simplifier {<br class="gmail_msg">
> + typedef std::function<Value* (Instruction*, LLVMContext&)> Rule;<br class="gmail_msg">
> +<br class="gmail_msg">
> + void addRule(const Rule &R) { Rules.push_back(R); }<br class="gmail_msg">
> +<br class="gmail_msg">
> + private:<br class="gmail_msg">
> + typedef std::deque<Value*> WorkListType;<br class="gmail_msg">
> + typedef std::set<Value*> ValueSetType;<br class="gmail_msg">
> + std::vector<Rule> Rules;<br class="gmail_msg">
> +<br class="gmail_msg">
> + public:<br class="gmail_msg">
> + struct Context {<br class="gmail_msg">
> + typedef DenseMap<Value*,Value*> ValueMapType;<br class="gmail_msg">
> +<br class="gmail_msg">
> + Value *Root;<br class="gmail_msg">
> + ValueSetType Used;<br class="gmail_msg">
> + ValueMapType Clones, Orig;<br class="gmail_msg">
> + LLVMContext &Ctx;<br class="gmail_msg">
> +<br class="gmail_msg">
> + Context(Instruction *Exp)<br class="gmail_msg">
> + : Ctx(Exp->getParent()->getParent()->getContext()) {<br class="gmail_msg">
> + initialize(Exp);<br class="gmail_msg">
> + reset();<br class="gmail_msg">
> + }<br class="gmail_msg">
> + ~Context() { cleanup(); }<br class="gmail_msg">
> + void print(raw_ostream &OS, const Value *V) const;<br class="gmail_msg">
> +<br class="gmail_msg">
> + Value *materialize(BasicBlock *B, BasicBlock::iterator At);<br class="gmail_msg">
> +<br class="gmail_msg">
> + private:<br class="gmail_msg">
> + void initialize(Instruction *Exp);<br class="gmail_msg">
> + void reset();<br class="gmail_msg">
> + void cleanup();<br class="gmail_msg">
> + void cleanup(Value *V);<br class="gmail_msg">
> +<br class="gmail_msg">
> + bool equal(const Instruction *I, const Instruction *J) const;<br class="gmail_msg">
> + Value *find(Value *Tree, Value *Sub) const;<br class="gmail_msg">
> + Value *subst(Value *Tree, Value *OldV, Value *NewV);<br class="gmail_msg">
> + void replace(Value *OldV, Value *NewV);<br class="gmail_msg">
> + void link(Instruction *I, BasicBlock *B, BasicBlock::iterator<br class="gmail_msg">
> At);<br class="gmail_msg">
> +<br class="gmail_msg">
> + friend struct Simplifier;<br class="gmail_msg">
> + };<br class="gmail_msg">
> +<br class="gmail_msg">
> + Value *simplify(Context &C);<br class="gmail_msg">
> + };<br class="gmail_msg">
> +<br class="gmail_msg">
> + struct PE {<br class="gmail_msg">
> + PE(const Simplifier::Context &c, Value *v = nullptr) : C(c),<br class="gmail_msg">
> V(v) {}<br class="gmail_msg">
> + const Simplifier::Context &C;<br class="gmail_msg">
> + const Value *V;<br class="gmail_msg">
> + };<br class="gmail_msg">
> +<br class="gmail_msg">
> + raw_ostream &operator<< (raw_ostream &OS, const PE &P)<br class="gmail_msg">
> LLVM_ATTRIBUTE_USED;<br class="gmail_msg">
> + raw_ostream &operator<< (raw_ostream &OS, const PE &P) {<br class="gmail_msg">
> + P.C.print(OS, P.V ? P.V : P.C.Root);<br class="gmail_msg">
> + return OS;<br class="gmail_msg">
> + }<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +<br class="gmail_msg">
> +void Simplifier::Context::print(raw_ostream &OS, const Value *V)<br class="gmail_msg">
> const {<br class="gmail_msg">
> + const auto *U = dyn_cast<const Instruction>(V);<br class="gmail_msg">
> + if (!U) {<br class="gmail_msg">
> + OS << V << '(' << *V << ')';<br class="gmail_msg">
> + return;<br class="gmail_msg">
> + }<br class="gmail_msg">
> +<br class="gmail_msg">
> + if (U->getParent()) {<br class="gmail_msg">
> + OS << U << '(';<br class="gmail_msg">
> + U->printAsOperand(OS, true);<br class="gmail_msg">
> + OS << ')';<br class="gmail_msg">
> + return;<br class="gmail_msg">
> + }<br class="gmail_msg">
> +<br class="gmail_msg">
> + unsigned N = U->getNumOperands();<br class="gmail_msg">
> + if (N != 0)<br class="gmail_msg">
> + OS << U << '(';<br class="gmail_msg">
> + OS << U->getOpcodeName();<br class="gmail_msg">
> + for (const Value *Op : U->operands()) {<br class="gmail_msg">
> + OS << ' ';<br class="gmail_msg">
> + print(OS, Op);<br class="gmail_msg">
> + }<br class="gmail_msg">
> + if (N != 0)<br class="gmail_msg">
> + OS << ')';<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +<br class="gmail_msg">
> +void Simplifier::Context::initialize(Instruction *Exp) {<br class="gmail_msg">
> + // Perform a deep clone of the expression, set Root to the root<br class="gmail_msg">
> + // of the clone, and build a map from the cloned values to the<br class="gmail_msg">
> + // original ones.<br class="gmail_msg">
> + BasicBlock *Block = Exp->getParent();<br class="gmail_msg">
> + WorkListType Q;<br class="gmail_msg">
> + Q.push_back(Exp);<br class="gmail_msg">
> +<br class="gmail_msg">
> + while (!Q.empty()) {<br class="gmail_msg">
> + Value *V = Q.front();<br class="gmail_msg">
> + Q.pop_front();<br class="gmail_msg">
> + if (Clones.find(V) != Clones.end())<br class="gmail_msg">
> + continue;<br class="gmail_msg">
> + if (Instruction *U = dyn_cast<Instruction>(V)) {<br class="gmail_msg">
> + if (isa<PHINode>(U) || U->getParent() != Block)<br class="gmail_msg">
> + continue;<br class="gmail_msg">
> + for (Value *Op : U->operands())<br class="gmail_msg">
> + Q.push_back(Op);<br class="gmail_msg">
> + Clones.insert({U, U->clone()});<br class="gmail_msg">
> + }<br class="gmail_msg">
> + }<br class="gmail_msg">
> +<br class="gmail_msg">
> + for (std::pair<Value*,Value*> P : Clones) {<br class="gmail_msg">
> + Instruction *U = cast<Instruction>(P.second);<br class="gmail_msg">
> + for (unsigned i = 0, n = U->getNumOperands(); i != n; ++i) {<br class="gmail_msg">
> + auto F = Clones.find(U->getOperand(i));<br class="gmail_msg">
> + if (F != Clones.end())<br class="gmail_msg">
> + U->setOperand(i, F->second);<br class="gmail_msg">
> + }<br class="gmail_msg">
> + Orig.insert({P.second, P.first});<br class="gmail_msg">
> + }<br class="gmail_msg">
> +<br class="gmail_msg">
> + auto R = Clones.find(Exp);<br class="gmail_msg">
> + assert(R != Clones.end());<br class="gmail_msg">
> + Root = R->second;<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +<br class="gmail_msg">
> +void Simplifier::Context::reset() {<br class="gmail_msg">
> + ValueSetType NewUsed;<br class="gmail_msg">
> + WorkListType Q;<br class="gmail_msg">
> + Q.push_back(Root);<br class="gmail_msg">
> +<br class="gmail_msg">
> + while (!Q.empty()) {<br class="gmail_msg">
> + Instruction *U = dyn_cast<Instruction>(Q.front());<br class="gmail_msg">
> + Q.pop_front();<br class="gmail_msg">
> + if (!U || U->getParent())<br class="gmail_msg">
> + continue;<br class="gmail_msg">
> + NewUsed.insert(U);<br class="gmail_msg">
> + for (Value *Op : U->operands())<br class="gmail_msg">
> + Q.push_back(Op);<br class="gmail_msg">
> + }<br class="gmail_msg">
> + for (Value *V : Used)<br class="gmail_msg">
> + if (!NewUsed.count(V))<br class="gmail_msg">
> + cast<Instruction>(V)->dropAllReferences();<br class="gmail_msg">
> + Used = NewUsed;<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +<br class="gmail_msg">
> +Value *Simplifier::Context::subst(Value *Tree, Value *OldV, Value<br class="gmail_msg">
> *NewV) {<br class="gmail_msg">
> + if (Tree == OldV) {<br class="gmail_msg">
> + cleanup(OldV);<br class="gmail_msg">
> + return NewV;<br class="gmail_msg">
> + }<br class="gmail_msg">
> +<br class="gmail_msg">
> + WorkListType Q;<br class="gmail_msg">
> + Q.push_back(Tree);<br class="gmail_msg">
> + while (!Q.empty()) {<br class="gmail_msg">
> + Instruction *U = dyn_cast<Instruction>(Q.front());<br class="gmail_msg">
> + Q.pop_front();<br class="gmail_msg">
> + // If U is not an instruction, or it's not a clone, skip it.<br class="gmail_msg">
> + if (!U || U->getParent())<br class="gmail_msg">
> + continue;<br class="gmail_msg">
> + for (unsigned i = 0, n = U->getNumOperands(); i != n; ++i) {<br class="gmail_msg">
> + Value *Op = U->getOperand(i);<br class="gmail_msg">
> + if (Op == OldV) {<br class="gmail_msg">
> + cleanup(OldV);<br class="gmail_msg">
> + U->setOperand(i, NewV);<br class="gmail_msg">
> + } else {<br class="gmail_msg">
> + Q.push_back(Op);<br class="gmail_msg">
> + }<br class="gmail_msg">
> + }<br class="gmail_msg">
> + }<br class="gmail_msg">
> + return Tree;<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +<br class="gmail_msg">
> +void Simplifier::Context::replace(Value *OldV, Value *NewV) {<br class="gmail_msg">
> + if (Root == OldV) {<br class="gmail_msg">
> + Root = NewV;<br class="gmail_msg">
> + reset();<br class="gmail_msg">
> + return;<br class="gmail_msg">
> + }<br class="gmail_msg">
> +<br class="gmail_msg">
> + // NewV may be a complex tree that has just been created by one<br class="gmail_msg">
> of the<br class="gmail_msg">
> + // transformation rules. We need to make sure that it is commoned<br class="gmail_msg">
> with<br class="gmail_msg">
> + // the existing Root to the maximum extent possible.<br class="gmail_msg">
> + // Identify all subtrees of NewV (including NewV itself) that have<br class="gmail_msg">
> + // equivalent counterparts in Root, and replace those subtrees with<br class="gmail_msg">
> + // these counterparts.<br class="gmail_msg">
> + WorkListType Q;<br class="gmail_msg">
> + Q.push_back(NewV);<br class="gmail_msg">
> + while (!Q.empty()) {<br class="gmail_msg">
> + Value *V = Q.front();<br class="gmail_msg">
> + Q.pop_front();<br class="gmail_msg">
> + Instruction *U = dyn_cast<Instruction>(V);<br class="gmail_msg">
> + if (!U || U->getParent())<br class="gmail_msg">
> + continue;<br class="gmail_msg">
> + if (Value *DupV = find(Root, V)) {<br class="gmail_msg">
> + if (DupV != V)<br class="gmail_msg">
> + NewV = subst(NewV, V, DupV);<br class="gmail_msg">
> + } else {<br class="gmail_msg">
> + for (Value *Op : U->operands())<br class="gmail_msg">
> + Q.push_back(Op);<br class="gmail_msg">
> + }<br class="gmail_msg">
> + }<br class="gmail_msg">
> +<br class="gmail_msg">
> + // Now, simply replace OldV with NewV in Root.<br class="gmail_msg">
> + Root = subst(Root, OldV, NewV);<br class="gmail_msg">
> + reset();<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +<br class="gmail_msg">
> +void Simplifier::Context::cleanup() {<br class="gmail_msg">
> + for (Value *V : Used) {<br class="gmail_msg">
> + Instruction *U = cast<Instruction>(V);<br class="gmail_msg">
> + if (!U->getParent())<br class="gmail_msg">
> + U->dropAllReferences();<br class="gmail_msg">
> + }<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +<br class="gmail_msg">
> +void Simplifier::Context::cleanup(Value *V) {<br class="gmail_msg">
> + if (!isa<Instruction>(V) || cast<Instruction>(V)->getParent() !=<br class="gmail_msg">
> nullptr)<br class="gmail_msg">
> + return;<br class="gmail_msg">
> + WorkListType Q;<br class="gmail_msg">
> + Q.push_back(V);<br class="gmail_msg">
> + while (!Q.empty()) {<br class="gmail_msg">
> + Instruction *U = dyn_cast<Instruction>(Q.front());<br class="gmail_msg">
> + Q.pop_front();<br class="gmail_msg">
> + if (!U || U->getParent() || Used.count(U))<br class="gmail_msg">
> + continue;<br class="gmail_msg">
> + for (Value *Op : U->operands())<br class="gmail_msg">
> + Q.push_back(Op);<br class="gmail_msg">
> + U->dropAllReferences();<br class="gmail_msg">
> + }<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +<br class="gmail_msg">
> +bool Simplifier::Context::equal(const Instruction *I,<br class="gmail_msg">
> + const Instruction *J) const {<br class="gmail_msg">
> + if (I == J)<br class="gmail_msg">
> + return true;<br class="gmail_msg">
> + if (!I->isSameOperationAs(J))<br class="gmail_msg">
> + return false;<br class="gmail_msg">
> + if (isa<PHINode>(I))<br class="gmail_msg">
> + return I->isIdenticalTo(J);<br class="gmail_msg">
> +<br class="gmail_msg">
> + for (unsigned i = 0, n = I->getNumOperands(); i != n; ++i) {<br class="gmail_msg">
> + Value *OpI = I->getOperand(i), *OpJ = J->getOperand(i);<br class="gmail_msg">
> + if (OpI == OpJ)<br class="gmail_msg">
> + continue;<br class="gmail_msg">
> + auto *InI = dyn_cast<const Instruction>(OpI);<br class="gmail_msg">
> + auto *InJ = dyn_cast<const Instruction>(OpJ);<br class="gmail_msg">
> + if (InI && InJ) {<br class="gmail_msg">
> + if (!equal(InI, InJ))<br class="gmail_msg">
> + return false;<br class="gmail_msg">
> + } else if (InI != InJ || !InI)<br class="gmail_msg">
> + return false;<br class="gmail_msg">
> + }<br class="gmail_msg">
> + return true;<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +<br class="gmail_msg">
> +Value *Simplifier::Context::find(Value *Tree, Value *Sub) const {<br class="gmail_msg">
> + Instruction *SubI = dyn_cast<Instruction>(Sub);<br class="gmail_msg">
> + WorkListType Q;<br class="gmail_msg">
> + Q.push_back(Tree);<br class="gmail_msg">
> +<br class="gmail_msg">
> + while (!Q.empty()) {<br class="gmail_msg">
> + Value *V = Q.front();<br class="gmail_msg">
> + Q.pop_front();<br class="gmail_msg">
> + if (V == Sub)<br class="gmail_msg">
> + return V;<br class="gmail_msg">
> + Instruction *U = dyn_cast<Instruction>(V);<br class="gmail_msg">
> + if (!U || U->getParent())<br class="gmail_msg">
> + continue;<br class="gmail_msg">
> + if (SubI && equal(SubI, U))<br class="gmail_msg">
> + return U;<br class="gmail_msg">
> + assert(!isa<PHINode>(U));<br class="gmail_msg">
> + for (Value *Op : U->operands())<br class="gmail_msg">
> + Q.push_back(Op);<br class="gmail_msg">
> + }<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +<br class="gmail_msg">
> +void Simplifier::Context::link(Instruction *I, BasicBlock *B,<br class="gmail_msg">
> + BasicBlock::iterator At) {<br class="gmail_msg">
> + if (I->getParent())<br class="gmail_msg">
> + return;<br class="gmail_msg">
> +<br class="gmail_msg">
> + for (Value *Op : I->operands()) {<br class="gmail_msg">
> + if (Instruction *OpI = dyn_cast<Instruction>(Op))<br class="gmail_msg">
> + link(OpI, B, At);<br class="gmail_msg">
> + }<br class="gmail_msg">
> +<br class="gmail_msg">
> + B->getInstList().insert(At, I);<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +<br class="gmail_msg">
> +Value *Simplifier::Context::materialize(BasicBlock *B,<br class="gmail_msg">
> + BasicBlock::iterator At) {<br class="gmail_msg">
> + if (Instruction *RootI = dyn_cast<Instruction>(Root))<br class="gmail_msg">
> + link(RootI, B, At);<br class="gmail_msg">
> + return Root;<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +<br class="gmail_msg">
> +Value *Simplifier::simplify(Context &C) {<br class="gmail_msg">
> + WorkListType Q;<br class="gmail_msg">
> + Q.push_back(C.Root);<br class="gmail_msg">
> +<br class="gmail_msg">
> + while (!Q.empty()) {<br class="gmail_msg">
> + Instruction *U = dyn_cast<Instruction>(Q.front());<br class="gmail_msg">
> + Q.pop_front();<br class="gmail_msg">
> + if (!U || U->getParent() || !C.Used.count(U))<br class="gmail_msg">
> + continue;<br class="gmail_msg">
> + bool Changed = false;<br class="gmail_msg">
> + for (Rule &R : Rules) {<br class="gmail_msg">
> + Value *W = R(U, C.Ctx);<br class="gmail_msg">
> + if (!W)<br class="gmail_msg">
> + continue;<br class="gmail_msg">
> + Changed = true;<br class="gmail_msg">
> + C.replace(U, W);<br class="gmail_msg">
> + Q.push_back(C.Root);<br class="gmail_msg">
> + break;<br class="gmail_msg">
> + }<br class="gmail_msg">
> + if (!Changed) {<br class="gmail_msg">
> + for (Value *Op : U->operands())<br class="gmail_msg">
> + Q.push_back(Op);<br class="gmail_msg">
> + }<br class="gmail_msg">
> + }<br class="gmail_msg">
> + return C.Root;<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +<br class="gmail_msg">
> //===----------------------------------------------------------------------===//<br class="gmail_msg">
> //<br class="gmail_msg">
> // Implementation of PolynomialMultiplyRecognize<br class="gmail_msg">
> @@ -147,6 +483,14 @@ namespace {<br class="gmail_msg">
> private:<br class="gmail_msg">
> typedef SetVector<Value*> ValueSeq;<br class="gmail_msg">
><br class="gmail_msg">
> + IntegerType *getPmpyType() const {<br class="gmail_msg">
> + LLVMContext &Ctx =<br class="gmail_msg">
> CurLoop->getHeader()->getParent()->getContext();<br class="gmail_msg">
> + return IntegerType::get(Ctx, 32);<br class="gmail_msg">
> + }<br class="gmail_msg">
> + bool isPromotableTo(Value *V, IntegerType *Ty);<br class="gmail_msg">
> + void promoteTo(Instruction *In, IntegerType *DestTy, BasicBlock<br class="gmail_msg">
> *LoopB);<br class="gmail_msg">
> + bool promoteTypes(BasicBlock *LoopB, BasicBlock *ExitB);<br class="gmail_msg">
> +<br class="gmail_msg">
> Value *getCountIV(BasicBlock *BB);<br class="gmail_msg">
> bool findCycle(Value *Out, Value *In, ValueSeq &Cycle);<br class="gmail_msg">
> void classifyCycle(Instruction *DivI, ValueSeq &Cycle, ValueSeq<br class="gmail_msg">
> &Early,<br class="gmail_msg">
> @@ -176,6 +520,9 @@ namespace {<br class="gmail_msg">
> unsigned getInverseMxN(unsigned QP);<br class="gmail_msg">
> Value *generate(BasicBlock::iterator At, ParsedValues &PV);<br class="gmail_msg">
><br class="gmail_msg">
> + void setupSimplifier();<br class="gmail_msg">
> +<br class="gmail_msg">
> + Simplifier Simp;<br class="gmail_msg">
> Loop *CurLoop;<br class="gmail_msg">
> const DataLayout &DL;<br class="gmail_msg">
> const DominatorTree &DT;<br class="gmail_msg">
> @@ -425,7 +772,6 @@ bool PolynomialMultiplyRecognize::scanSe<br class="gmail_msg">
> BasicBlock *LoopB, BasicBlock *PrehB, Value *CIV,<br class="gmail_msg">
> ParsedValues &PV,<br class="gmail_msg">
> bool PreScan) {<br class="gmail_msg">
> using namespace PatternMatch;<br class="gmail_msg">
> -<br class="gmail_msg">
> // The basic pattern for R = P.Q is:<br class="gmail_msg">
> // for i = 0..31<br class="gmail_msg">
> // R = phi (0, R')<br class="gmail_msg">
> @@ -529,6 +875,150 @@ bool PolynomialMultiplyRecognize::scanSe<br class="gmail_msg">
> }<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> +bool PolynomialMultiplyRecognize::isPromotableTo(Value *Val,<br class="gmail_msg">
> + IntegerType *DestTy) {<br class="gmail_msg">
> + IntegerType *T = dyn_cast<IntegerType>(Val->getType());<br class="gmail_msg">
> + if (!T || T->getBitWidth() > DestTy->getBitWidth())<br class="gmail_msg">
> + return false;<br class="gmail_msg">
> + if (T->getBitWidth() == DestTy->getBitWidth())<br class="gmail_msg">
> + return true;<br class="gmail_msg">
> + // Non-instructions are promotable. The reason why an instruction<br class="gmail_msg">
> may not<br class="gmail_msg">
> + // be promotable is that it may produce a different result if its<br class="gmail_msg">
> operands<br class="gmail_msg">
> + // and the result are promoted, for example, it may produce more<br class="gmail_msg">
> non-zero<br class="gmail_msg">
> + // bits. While it would still be possible to represent the proper<br class="gmail_msg">
> result<br class="gmail_msg">
> + // in a wider type, it may require adding additional instructions<br class="gmail_msg">
> (which<br class="gmail_msg">
> + // we don't want to do).<br class="gmail_msg">
> + Instruction *In = dyn_cast<Instruction>(Val);<br class="gmail_msg">
> + if (!In)<br class="gmail_msg">
> + return true;<br class="gmail_msg">
> + // The bitwidth of the source type is smaller than the destination.<br class="gmail_msg">
> + // Check if the individual operation can be promoted.<br class="gmail_msg">
> + switch (In->getOpcode()) {<br class="gmail_msg">
> + case Instruction::PHI:<br class="gmail_msg">
> + case Instruction::ZExt:<br class="gmail_msg">
> + case Instruction::And:<br class="gmail_msg">
> + case Instruction::Or:<br class="gmail_msg">
> + case Instruction::Xor:<br class="gmail_msg">
> + case Instruction::LShr: // Shift right is ok.<br class="gmail_msg">
> + case Instruction::Select:<br class="gmail_msg">
> + return true;<br class="gmail_msg">
> + case Instruction::ICmp:<br class="gmail_msg">
> + if (CmpInst *CI = cast<CmpInst>(In))<br class="gmail_msg">
> + return CI->isEquality() || CI->isUnsigned();<br class="gmail_msg">
> + llvm_unreachable("Cast failed unexpectedly");<br class="gmail_msg">
> + case Instruction::Add:<br class="gmail_msg">
> + return In->hasNoSignedWrap() && In->hasNoUnsignedWrap();<br class="gmail_msg">
> + }<br class="gmail_msg">
> + return false;<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +<br class="gmail_msg">
> +void PolynomialMultiplyRecognize::promoteTo(Instruction *In,<br class="gmail_msg">
> + IntegerType *DestTy, BasicBlock *LoopB) {<br class="gmail_msg">
> + // Leave boolean values alone.<br class="gmail_msg">
> + if (!In->getType()->isIntegerTy(1))<br class="gmail_msg">
> + In->mutateType(DestTy);<br class="gmail_msg">
> + unsigned DestBW = DestTy->getBitWidth();<br class="gmail_msg">
> +<br class="gmail_msg">
> + // Handle PHIs.<br class="gmail_msg">
> + if (PHINode *P = dyn_cast<PHINode>(In)) {<br class="gmail_msg">
> + unsigned N = P->getNumIncomingValues();<br class="gmail_msg">
> + for (unsigned i = 0; i != N; ++i) {<br class="gmail_msg">
> + BasicBlock *InB = P->getIncomingBlock(i);<br class="gmail_msg">
> + if (InB == LoopB)<br class="gmail_msg">
> + continue;<br class="gmail_msg">
> + Value *InV = P->getIncomingValue(i);<br class="gmail_msg">
> + IntegerType *Ty = cast<IntegerType>(InV->getType());<br class="gmail_msg">
> + // Do not promote values in PHI nodes of type i1.<br class="gmail_msg">
> + if (Ty != P->getType()) {<br class="gmail_msg">
> + // If the value type does not match the PHI type, the PHI type<br class="gmail_msg">
> + // must have been promoted.<br class="gmail_msg">
> + assert(Ty->getBitWidth() < DestBW);<br class="gmail_msg">
> + InV = IRBuilder<>(InB->getTerminator()).CreateZExt(InV,<br class="gmail_msg">
> DestTy);<br class="gmail_msg">
> + P->setIncomingValue(i, InV);<br class="gmail_msg">
> + }<br class="gmail_msg">
> + }<br class="gmail_msg">
> + } else if (ZExtInst *Z = dyn_cast<ZExtInst>(In)) {<br class="gmail_msg">
> + Value *Op = Z->getOperand(0);<br class="gmail_msg">
> + if (Op->getType() == Z->getType())<br class="gmail_msg">
> + Z->replaceAllUsesWith(Op);<br class="gmail_msg">
> + Z->eraseFromParent();<br class="gmail_msg">
> + return;<br class="gmail_msg">
> + }<br class="gmail_msg">
> +<br class="gmail_msg">
> + // Promote immediates.<br class="gmail_msg">
> + for (unsigned i = 0, n = In->getNumOperands(); i != n; ++i) {<br class="gmail_msg">
> + if (ConstantInt *CI = dyn_cast<ConstantInt>(In->getOperand(i)))<br class="gmail_msg">
> + if (CI->getType()->getBitWidth() < DestBW)<br class="gmail_msg">
> + In->setOperand(i, ConstantInt::get(DestTy,<br class="gmail_msg">
> CI->getZExtValue()));<br class="gmail_msg">
> + }<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +<br class="gmail_msg">
> +bool PolynomialMultiplyRecognize::promoteTypes(BasicBlock *LoopB,<br class="gmail_msg">
> + BasicBlock *ExitB) {<br class="gmail_msg">
> + assert(LoopB);<br class="gmail_msg">
> + // Skip loops where the exit block has more than one predecessor.<br class="gmail_msg">
> The values<br class="gmail_msg">
> + // coming from the loop block will be promoted to another type,<br class="gmail_msg">
> and so the<br class="gmail_msg">
> + // values coming into the exit block from other predecessors<br class="gmail_msg">
> would also have<br class="gmail_msg">
> + // to be promoted.<br class="gmail_msg">
> + if (!ExitB || (ExitB->getSinglePredecessor() != LoopB))<br class="gmail_msg">
> + return false;<br class="gmail_msg">
> + IntegerType *DestTy = getPmpyType();<br class="gmail_msg">
> + // Check if the exit values have types that are no wider than the<br class="gmail_msg">
> type<br class="gmail_msg">
> + // that we want to promote to.<br class="gmail_msg">
> + unsigned DestBW = DestTy->getBitWidth();<br class="gmail_msg">
> + for (Instruction &In : *ExitB) {<br class="gmail_msg">
> + PHINode *P = dyn_cast<PHINode>(&In);<br class="gmail_msg">
> + if (!P)<br class="gmail_msg">
> + break;<br class="gmail_msg">
> + if (P->getNumIncomingValues() != 1)<br class="gmail_msg">
> + return false;<br class="gmail_msg">
> + assert(P->getIncomingBlock(0) == LoopB);<br class="gmail_msg">
> + IntegerType *T = dyn_cast<IntegerType>(P->getType());<br class="gmail_msg">
> + if (!T || T->getBitWidth() > DestBW)<br class="gmail_msg">
> + return false;<br class="gmail_msg">
> + }<br class="gmail_msg">
> +<br class="gmail_msg">
> + // Check all instructions in the loop.<br class="gmail_msg">
> + for (Instruction &In : *LoopB)<br class="gmail_msg">
> + if (!In.isTerminator() && !isPromotableTo(&In, DestTy))<br class="gmail_msg">
> + return false;<br class="gmail_msg">
> +<br class="gmail_msg">
> + // Perform the promotion.<br class="gmail_msg">
> + std::vector<Instruction*> LoopIns;<br class="gmail_msg">
> + std::transform(LoopB->begin(), LoopB->end(),<br class="gmail_msg">
> std::back_inserter(LoopIns),<br class="gmail_msg">
> + [](Instruction &In) { return &In; });<br class="gmail_msg">
> + for (Instruction *In : LoopIns)<br class="gmail_msg">
> + promoteTo(In, DestTy, LoopB);<br class="gmail_msg">
> +<br class="gmail_msg">
> + // Fix up the PHI nodes in the exit block.<br class="gmail_msg">
> + Instruction *EndI = ExitB->getFirstNonPHI();<br class="gmail_msg">
> + BasicBlock::iterator End = EndI ? EndI->getIterator() : ExitB->end();<br class="gmail_msg">
> + for (auto I = ExitB->begin(); I != End; ++I) {<br class="gmail_msg">
> + PHINode *P = dyn_cast<PHINode>(I);<br class="gmail_msg">
> + if (!P)<br class="gmail_msg">
> + break;<br class="gmail_msg">
> + Type *Ty0 = P->getIncomingValue(0)->getType();<br class="gmail_msg">
> + Type *PTy = P->getType();<br class="gmail_msg">
> + if (PTy != Ty0) {<br class="gmail_msg">
> + assert(Ty0 == DestTy);<br class="gmail_msg">
> + // In order to create the trunc, P must have the promoted type.<br class="gmail_msg">
> + P->mutateType(Ty0);<br class="gmail_msg">
> + Value *T = IRBuilder<>(ExitB, End).CreateTrunc(P, PTy);<br class="gmail_msg">
> + // In order for the RAUW to work, the types of P and T must<br class="gmail_msg">
> match.<br class="gmail_msg">
> + P->mutateType(PTy);<br class="gmail_msg">
> + P->replaceAllUsesWith(T);<br class="gmail_msg">
> + // Final update of the P's type.<br class="gmail_msg">
> + P->mutateType(Ty0);<br class="gmail_msg">
> + cast<Instruction>(T)->setOperand(0, P);<br class="gmail_msg">
> + }<br class="gmail_msg">
> + }<br class="gmail_msg">
> +<br class="gmail_msg">
> + return true;<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +<br class="gmail_msg">
> bool PolynomialMultiplyRecognize::findCycle(Value *Out, Value *In,<br class="gmail_msg">
> ValueSeq &Cycle) {<br class="gmail_msg">
> // Out = ..., In, ...<br class="gmail_msg">
> @@ -699,6 +1189,7 @@ bool PolynomialMultiplyRecognize::keepsH<br class="gmail_msg">
> case Instruction::Select:<br class="gmail_msg">
> case Instruction::ICmp:<br class="gmail_msg">
> case Instruction::PHI:<br class="gmail_msg">
> + case Instruction::ZExt:<br class="gmail_msg">
> return true;<br class="gmail_msg">
> }<br class="gmail_msg">
> }<br class="gmail_msg">
> @@ -985,13 +1476,170 @@ Value *PolynomialMultiplyRecognize::gene<br class="gmail_msg">
> }<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> +void PolynomialMultiplyRecognize::setupSimplifier() {<br class="gmail_msg">
> + Simp.addRule(<br class="gmail_msg">
> + // Sink zext past bitwise operations.<br class="gmail_msg">
> + [](Instruction *I, LLVMContext &Ctx) -> Value* {<br class="gmail_msg">
> + if (I->getOpcode() != Instruction::ZExt)<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + Instruction *T = dyn_cast<Instruction>(I->getOperand(0));<br class="gmail_msg">
> + if (!T)<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + switch (T->getOpcode()) {<br class="gmail_msg">
> + case Instruction::And:<br class="gmail_msg">
> + case Instruction::Or:<br class="gmail_msg">
> + case Instruction::Xor:<br class="gmail_msg">
> + break;<br class="gmail_msg">
> + default:<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + }<br class="gmail_msg">
> + IRBuilder<> B(Ctx);<br class="gmail_msg">
> + return B.CreateBinOp(cast<BinaryOperator>(T)->getOpcode(),<br class="gmail_msg">
> + B.CreateZExt(T->getOperand(0),<br class="gmail_msg">
> I->getType()),<br class="gmail_msg">
> + B.CreateZExt(T->getOperand(1),<br class="gmail_msg">
> I->getType()));<br class="gmail_msg">
> + });<br class="gmail_msg">
> + Simp.addRule(<br class="gmail_msg">
> + // (xor (and x a) (and y a)) -> (and (xor x y) a)<br class="gmail_msg">
> + [](Instruction *I, LLVMContext &Ctx) -> Value* {<br class="gmail_msg">
> + if (I->getOpcode() != Instruction::Xor)<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + Instruction *And0 = dyn_cast<Instruction>(I->getOperand(0));<br class="gmail_msg">
> + Instruction *And1 = dyn_cast<Instruction>(I->getOperand(1));<br class="gmail_msg">
> + if (!And0 || !And1)<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + if (And0->getOpcode() != Instruction::And ||<br class="gmail_msg">
> + And1->getOpcode() != Instruction::And)<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + if (And0->getOperand(1) != And1->getOperand(1))<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + IRBuilder<> B(Ctx);<br class="gmail_msg">
> + return B.CreateAnd(B.CreateXor(And0->getOperand(0),<br class="gmail_msg">
> And1->getOperand(0)),<br class="gmail_msg">
> + And0->getOperand(1));<br class="gmail_msg">
> + });<br class="gmail_msg">
> + Simp.addRule(<br class="gmail_msg">
> + // (Op (select c x y) z) -> (select c (Op x z) (Op y z))<br class="gmail_msg">
> + // (Op x (select c y z)) -> (select c (Op x y) (Op x z))<br class="gmail_msg">
> + [](Instruction *I, LLVMContext &Ctx) -> Value* {<br class="gmail_msg">
> + BinaryOperator *BO = dyn_cast<BinaryOperator>(I);<br class="gmail_msg">
> + if (!BO)<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + Instruction::BinaryOps Op = BO->getOpcode();<br class="gmail_msg">
> + if (SelectInst *Sel = dyn_cast<SelectInst>(BO->getOperand(0))) {<br class="gmail_msg">
> + IRBuilder<> B(Ctx);<br class="gmail_msg">
> + Value *X = Sel->getTrueValue(), *Y = Sel->getFalseValue();<br class="gmail_msg">
> + Value *Z = BO->getOperand(1);<br class="gmail_msg">
> + return B.CreateSelect(Sel->getCondition(),<br class="gmail_msg">
> + B.CreateBinOp(Op, X, Z),<br class="gmail_msg">
> + B.CreateBinOp(Op, Y, Z));<br class="gmail_msg">
> + }<br class="gmail_msg">
> + if (SelectInst *Sel = dyn_cast<SelectInst>(BO->getOperand(1))) {<br class="gmail_msg">
> + IRBuilder<> B(Ctx);<br class="gmail_msg">
> + Value *X = BO->getOperand(0);<br class="gmail_msg">
> + Value *Y = Sel->getTrueValue(), *Z = Sel->getFalseValue();<br class="gmail_msg">
> + return B.CreateSelect(Sel->getCondition(),<br class="gmail_msg">
> + B.CreateBinOp(Op, X, Y),<br class="gmail_msg">
> + B.CreateBinOp(Op, X, Z));<br class="gmail_msg">
> + }<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + });<br class="gmail_msg">
> + Simp.addRule(<br class="gmail_msg">
> + // (select c (select c x y) z) -> (select c x z)<br class="gmail_msg">
> + // (select c x (select c y z)) -> (select c x z)<br class="gmail_msg">
> + [](Instruction *I, LLVMContext &Ctx) -> Value* {<br class="gmail_msg">
> + SelectInst *Sel = dyn_cast<SelectInst>(I);<br class="gmail_msg">
> + if (!Sel)<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + IRBuilder<> B(Ctx);<br class="gmail_msg">
> + Value *C = Sel->getCondition();<br class="gmail_msg">
> + if (SelectInst *Sel0 =<br class="gmail_msg">
> dyn_cast<SelectInst>(Sel->getTrueValue())) {<br class="gmail_msg">
> + if (Sel0->getCondition() == C)<br class="gmail_msg">
> + return B.CreateSelect(C, Sel0->getTrueValue(),<br class="gmail_msg">
> Sel->getFalseValue());<br class="gmail_msg">
> + }<br class="gmail_msg">
> + if (SelectInst *Sel1 =<br class="gmail_msg">
> dyn_cast<SelectInst>(Sel->getFalseValue())) {<br class="gmail_msg">
> + if (Sel1->getCondition() == C)<br class="gmail_msg">
> + return B.CreateSelect(C, Sel->getTrueValue(),<br class="gmail_msg">
> Sel1->getFalseValue());<br class="gmail_msg">
> + }<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + });<br class="gmail_msg">
> + Simp.addRule(<br class="gmail_msg">
> + // (or (lshr x 1) 0x800.0) -> (xor (lshr x 1) 0x800.0)<br class="gmail_msg">
> + [](Instruction *I, LLVMContext &Ctx) -> Value* {<br class="gmail_msg">
> + if (I->getOpcode() != Instruction::Or)<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + Instruction *LShr = dyn_cast<Instruction>(I->getOperand(0));<br class="gmail_msg">
> + if (!LShr || LShr->getOpcode() != Instruction::LShr)<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + ConstantInt *One = dyn_cast<ConstantInt>(LShr->getOperand(1));<br class="gmail_msg">
> + if (!One || One->getZExtValue() != 1)<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + ConstantInt *Msb = dyn_cast<ConstantInt>(I->getOperand(1));<br class="gmail_msg">
> + if (!Msb || Msb->getZExtValue() != Msb->getType()->getSignBit())<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + return IRBuilder<>(Ctx).CreateXor(LShr, Msb);<br class="gmail_msg">
> + });<br class="gmail_msg">
> + Simp.addRule(<br class="gmail_msg">
> + // (lshr (BitOp x y) c) -> (BitOp (lshr x c) (lshr y c))<br class="gmail_msg">
> + [](Instruction *I, LLVMContext &Ctx) -> Value* {<br class="gmail_msg">
> + if (I->getOpcode() != Instruction::LShr)<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + BinaryOperator *BitOp =<br class="gmail_msg">
> dyn_cast<BinaryOperator>(I->getOperand(0));<br class="gmail_msg">
> + if (!BitOp)<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + switch (BitOp->getOpcode()) {<br class="gmail_msg">
> + case Instruction::And:<br class="gmail_msg">
> + case Instruction::Or:<br class="gmail_msg">
> + case Instruction::Xor:<br class="gmail_msg">
> + break;<br class="gmail_msg">
> + default:<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + }<br class="gmail_msg">
> + IRBuilder<> B(Ctx);<br class="gmail_msg">
> + Value *S = I->getOperand(1);<br class="gmail_msg">
> + return B.CreateBinOp(BitOp->getOpcode(),<br class="gmail_msg">
> + B.CreateLShr(BitOp->getOperand(0), S),<br class="gmail_msg">
> + B.CreateLShr(BitOp->getOperand(1), S));<br class="gmail_msg">
> + });<br class="gmail_msg">
> + Simp.addRule(<br class="gmail_msg">
> + // (BitOp1 (BitOp2 x a) b) -> (BitOp2 x (BitOp1 a b))<br class="gmail_msg">
> + [](Instruction *I, LLVMContext &Ctx) -> Value* {<br class="gmail_msg">
> + auto IsBitOp = [](unsigned Op) -> bool {<br class="gmail_msg">
> + switch (Op) {<br class="gmail_msg">
> + case Instruction::And:<br class="gmail_msg">
> + case Instruction::Or:<br class="gmail_msg">
> + case Instruction::Xor:<br class="gmail_msg">
> + return true;<br class="gmail_msg">
> + }<br class="gmail_msg">
> + return false;<br class="gmail_msg">
> + };<br class="gmail_msg">
> + BinaryOperator *BitOp1 = dyn_cast<BinaryOperator>(I);<br class="gmail_msg">
> + if (!BitOp1 || !IsBitOp(BitOp1->getOpcode()))<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + BinaryOperator *BitOp2 =<br class="gmail_msg">
> dyn_cast<BinaryOperator>(BitOp1->getOperand(0));<br class="gmail_msg">
> + if (!BitOp2 || !IsBitOp(BitOp2->getOpcode()))<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + ConstantInt *CA = dyn_cast<ConstantInt>(BitOp2->getOperand(1));<br class="gmail_msg">
> + ConstantInt *CB = dyn_cast<ConstantInt>(BitOp1->getOperand(1));<br class="gmail_msg">
> + if (!CA || !CB)<br class="gmail_msg">
> + return nullptr;<br class="gmail_msg">
> + IRBuilder<> B(Ctx);<br class="gmail_msg">
> + Value *X = BitOp2->getOperand(0);<br class="gmail_msg">
> + return B.CreateBinOp(BitOp2->getOpcode(), X,<br class="gmail_msg">
> + B.CreateBinOp(BitOp1->getOpcode(), CA, CB));<br class="gmail_msg">
> + });<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +<br class="gmail_msg">
> bool PolynomialMultiplyRecognize::recognize() {<br class="gmail_msg">
> + DEBUG(dbgs() << "Starting PolynomialMultiplyRecognize on loop\n"<br class="gmail_msg">
> + << *CurLoop << '\n');<br class="gmail_msg">
> // Restrictions:<br class="gmail_msg">
> // - The loop must consist of a single block.<br class="gmail_msg">
> // - The iteration count must be known at compile-time.<br class="gmail_msg">
> // - The loop must have an induction variable starting from 0, and<br class="gmail_msg">
> // incremented in each iteration of the loop.<br class="gmail_msg">
> BasicBlock *LoopB = CurLoop->getHeader();<br class="gmail_msg">
> + DEBUG(dbgs() << "Loop header:\n" << *LoopB);<br class="gmail_msg">
> +<br class="gmail_msg">
> if (LoopB != CurLoop->getLoopLatch())<br class="gmail_msg">
> return false;<br class="gmail_msg">
> BasicBlock *ExitB = CurLoop->getExitBlock();<br class="gmail_msg">
> @@ -1011,30 +1659,65 @@ bool PolynomialMultiplyRecognize::recogn<br class="gmail_msg">
> Value *CIV = getCountIV(LoopB);<br class="gmail_msg">
> ParsedValues PV;<br class="gmail_msg">
> PV.IterCount = IterCount;<br class="gmail_msg">
> + DEBUG(dbgs() << "Loop IV: " << *CIV << "\nIterCount: " <<<br class="gmail_msg">
> IterCount << '\n');<br class="gmail_msg">
><br class="gmail_msg">
> - // Test function to see if a given select instruction is a part<br class="gmail_msg">
> of the<br class="gmail_msg">
> - // pmpy pattern. The argument PreScan set to "true" indicates<br class="gmail_msg">
> that only<br class="gmail_msg">
> - // a preliminary scan is needed, "false" indicated an exact match.<br class="gmail_msg">
> - auto CouldBePmpy = [this, LoopB, EntryB, CIV, &PV] (bool PreScan)<br class="gmail_msg">
> - -> std::function<bool (Instruction &I)> {<br class="gmail_msg">
> - return [this, LoopB, EntryB, CIV, &PV, PreScan] (Instruction<br class="gmail_msg">
> &I) -> bool {<br class="gmail_msg">
> - if (auto *SelI = dyn_cast<SelectInst>(&I))<br class="gmail_msg">
> - return scanSelect(SelI, LoopB, EntryB, CIV, PV, PreScan);<br class="gmail_msg">
> - return false;<br class="gmail_msg">
> - };<br class="gmail_msg">
> - };<br class="gmail_msg">
> - auto PreF = std::find_if(LoopB->begin(), LoopB->end(),<br class="gmail_msg">
> CouldBePmpy(true));<br class="gmail_msg">
> - if (PreF == LoopB->end())<br class="gmail_msg">
> + setupSimplifier();<br class="gmail_msg">
> +<br class="gmail_msg">
> + // Perform a preliminary scan of select instructions to see if<br class="gmail_msg">
> any of them<br class="gmail_msg">
> + // looks like a generator of the polynomial multiply steps.<br class="gmail_msg">
> Assume that a<br class="gmail_msg">
> + // loop can only contain a single transformable operation, so<br class="gmail_msg">
> stop the<br class="gmail_msg">
> + // traversal after the first reasonable candidate was found.<br class="gmail_msg">
> + // XXX: Currently this approach can modify the loop before being<br class="gmail_msg">
> 100% sure<br class="gmail_msg">
> + // that the transformation can be carried out.<br class="gmail_msg">
> + bool FoundPreScan = false;<br class="gmail_msg">
> + for (Instruction &In : *LoopB) {<br class="gmail_msg">
> + SelectInst *SI = dyn_cast<SelectInst>(&In);<br class="gmail_msg">
> + if (!SI)<br class="gmail_msg">
> + continue;<br class="gmail_msg">
> +<br class="gmail_msg">
> + Simplifier::Context C(SI);<br class="gmail_msg">
> + Value *T = Simp.simplify(C);<br class="gmail_msg">
> + SelectInst *SelI = (T && isa<SelectInst>(T)) ?<br class="gmail_msg">
> cast<SelectInst>(T) : SI;<br class="gmail_msg">
> + DEBUG(dbgs() << "scanSelect(pre-scan): " << PE(C, SelI) << '\n');<br class="gmail_msg">
> + if (scanSelect(SelI, LoopB, EntryB, CIV, PV, true)) {<br class="gmail_msg">
> + FoundPreScan = true;<br class="gmail_msg">
> + if (SelI != SI) {<br class="gmail_msg">
> + Value *NewSel = C.materialize(LoopB, SI->getIterator());<br class="gmail_msg">
> + SI->replaceAllUsesWith(NewSel);<br class="gmail_msg">
> + RecursivelyDeleteTriviallyDeadInstructions(SI, &TLI);<br class="gmail_msg">
> + }<br class="gmail_msg">
> + break;<br class="gmail_msg">
> + }<br class="gmail_msg">
> + }<br class="gmail_msg">
> +<br class="gmail_msg">
> + if (!FoundPreScan) {<br class="gmail_msg">
> + DEBUG(dbgs() << "Have not found candidates for pmpy\n");<br class="gmail_msg">
> return false;<br class="gmail_msg">
> + }<br class="gmail_msg">
><br class="gmail_msg">
> if (!PV.Left) {<br class="gmail_msg">
> + // The right shift version actually only returns the higher bits of<br class="gmail_msg">
> + // the result (each iteration discards the LSB). If we want to<br class="gmail_msg">
> convert it<br class="gmail_msg">
> + // to a left-shifting loop, the working data type must be at<br class="gmail_msg">
> least as<br class="gmail_msg">
> + // wide as the target's pmpy instruction.<br class="gmail_msg">
> + if (!promoteTypes(LoopB, ExitB))<br class="gmail_msg">
> + return false;<br class="gmail_msg">
> convertShiftsToLeft(LoopB, ExitB, IterCount);<br class="gmail_msg">
> cleanupLoopBody(LoopB);<br class="gmail_msg">
> }<br class="gmail_msg">
><br class="gmail_msg">
> - auto PostF = std::find_if(LoopB->begin(), LoopB->end(),<br class="gmail_msg">
> CouldBePmpy(false));<br class="gmail_msg">
> - if (PostF == LoopB->end())<br class="gmail_msg">
> - return false;<br class="gmail_msg">
> + // Scan the loop again, find the generating select instruction.<br class="gmail_msg">
> + bool FoundScan = false;<br class="gmail_msg">
> + for (Instruction &In : *LoopB) {<br class="gmail_msg">
> + SelectInst *SelI = dyn_cast<SelectInst>(&In);<br class="gmail_msg">
> + if (!SelI)<br class="gmail_msg">
> + continue;<br class="gmail_msg">
> + DEBUG(dbgs() << "scanSelect: " << *SelI << '\n');<br class="gmail_msg">
> + FoundScan = scanSelect(SelI, LoopB, EntryB, CIV, PV, false);<br class="gmail_msg">
> + if (FoundScan)<br class="gmail_msg">
> + break;<br class="gmail_msg">
> + }<br class="gmail_msg">
> + assert(FoundScan);<br class="gmail_msg">
><br class="gmail_msg">
> DEBUG({<br class="gmail_msg">
> StringRef PP = (PV.M ? "(P+M)" : "P");<br class="gmail_msg">
><br class="gmail_msg">
> Added: llvm/trunk/test/CodeGen/Hexagon/loop-idiom/pmpy-mod.ll<br class="gmail_msg">
> URL:<br class="gmail_msg">
> <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/CodeGen/Hexagon/loop-idiom/pmpy-mod.ll?rev=298282&view=auto" rel="noreferrer" class="gmail_msg" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/CodeGen/Hexagon/loop-idiom/pmpy-mod.ll?rev=298282&view=auto</a><br class="gmail_msg">
> ==============================================================================<br class="gmail_msg">
> --- llvm/trunk/test/CodeGen/Hexagon/loop-idiom/pmpy-mod.ll (added)<br class="gmail_msg">
> +++ llvm/trunk/test/CodeGen/Hexagon/loop-idiom/pmpy-mod.ll Mon Mar<br class="gmail_msg">
> 20 13:12:58 2017<br class="gmail_msg">
> @@ -0,0 +1,84 @@<br class="gmail_msg">
> +; Run -O2 to make sure that all the usual optimizations do happen<br class="gmail_msg">
> before<br class="gmail_msg">
> +; the Hexagon loop idiom recognition runs. This is to check that we<br class="gmail_msg">
> still<br class="gmail_msg">
> +; get this opportunity regardless of what happens before.<br class="gmail_msg">
> +<br class="gmail_msg">
> +; RUN: opt -O2 -march=hexagon -S < %s | FileCheck %s<br class="gmail_msg">
> +<br class="gmail_msg">
> +target triple = "hexagon"<br class="gmail_msg">
> +target datalayout =<br class="gmail_msg">
> "e-m:e-p:32:32:32-a:0-n16:32-i64:64:64-i32:32:32-i16:16:16-i1:8:8-f32:32:32-f64:64:64-v32:32:32-v64:64:64-v512:512:512-v1024:1024:1024-v2048:2048:2048"<br class="gmail_msg">
> +<br class="gmail_msg">
> +; CHECK-LABEL: define zeroext i16 @pmpy_mod_lsr<br class="gmail_msg">
> +; There need to be two pmpy instructions.<br class="gmail_msg">
> +; CHECK: call i64 @llvm.hexagon.M4.pmpyw<br class="gmail_msg">
> +; CHECK: call i64 @llvm.hexagon.M4.pmpyw<br class="gmail_msg">
> +<br class="gmail_msg">
> +define zeroext i16 @pmpy_mod_lsr(i8 zeroext %a0, i16 zeroext %a1) #0 {<br class="gmail_msg">
> +b2:<br class="gmail_msg">
> + br label %b3<br class="gmail_msg">
> +<br class="gmail_msg">
> +b3: ; preds = %b44, %b2<br class="gmail_msg">
> + %v4 = phi i8 [ %a0, %b2 ], [ %v19, %b44 ]<br class="gmail_msg">
> + %v5 = phi i16 [ %a1, %b2 ], [ %v43, %b44 ]<br class="gmail_msg">
> + %v6 = phi i8 [ 0, %b2 ], [ %v45, %b44 ]<br class="gmail_msg">
> + %v7 = zext i8 %v6 to i32<br class="gmail_msg">
> + %v8 = icmp slt i32 %v7, 8<br class="gmail_msg">
> + br i1 %v8, label %b9, label %b46<br class="gmail_msg">
> +<br class="gmail_msg">
> +b9: ; preds = %b3<br class="gmail_msg">
> + %v10 = zext i8 %v4 to i32<br class="gmail_msg">
> + %v11 = and i32 %v10, 1<br class="gmail_msg">
> + %v12 = trunc i16 %v5 to i8<br class="gmail_msg">
> + %v13 = zext i8 %v12 to i32<br class="gmail_msg">
> + %v14 = and i32 %v13, 1<br class="gmail_msg">
> + %v15 = xor i32 %v11, %v14<br class="gmail_msg">
> + %v16 = trunc i32 %v15 to i8<br class="gmail_msg">
> + %v17 = zext i8 %v4 to i32<br class="gmail_msg">
> + %v18 = ashr i32 %v17, 1<br class="gmail_msg">
> + %v19 = trunc i32 %v18 to i8<br class="gmail_msg">
> + %v20 = zext i8 %v16 to i32<br class="gmail_msg">
> + %v21 = icmp eq i32 %v20, 1<br class="gmail_msg">
> + br i1 %v21, label %b22, label %b26<br class="gmail_msg">
> +<br class="gmail_msg">
> +b22: ; preds = %b9<br class="gmail_msg">
> + %v23 = zext i16 %v5 to i32<br class="gmail_msg">
> + %v24 = xor i32 %v23, 16386<br class="gmail_msg">
> + %v25 = trunc i32 %v24 to i16<br class="gmail_msg">
> + br label %b27<br class="gmail_msg">
> +<br class="gmail_msg">
> +b26: ; preds = %b9<br class="gmail_msg">
> + br label %b27<br class="gmail_msg">
> +<br class="gmail_msg">
> +b27: ; preds = %b26, %b22<br class="gmail_msg">
> + %v28 = phi i16 [ %v25, %b22 ], [ %v5, %b26 ]<br class="gmail_msg">
> + %v29 = phi i8 [ 1, %b22 ], [ 0, %b26 ]<br class="gmail_msg">
> + %v30 = zext i16 %v28 to i32<br class="gmail_msg">
> + %v31 = ashr i32 %v30, 1<br class="gmail_msg">
> + %v32 = trunc i32 %v31 to i16<br class="gmail_msg">
> + %v33 = icmp ne i8 %v29, 0<br class="gmail_msg">
> + br i1 %v33, label %b34, label %b38<br class="gmail_msg">
> +<br class="gmail_msg">
> +b34: ; preds = %b27<br class="gmail_msg">
> + %v35 = zext i16 %v32 to i32<br class="gmail_msg">
> + %v36 = or i32 %v35, 32768<br class="gmail_msg">
> + %v37 = trunc i32 %v36 to i16<br class="gmail_msg">
> + br label %b42<br class="gmail_msg">
> +<br class="gmail_msg">
> +b38: ; preds = %b27<br class="gmail_msg">
> + %v39 = zext i16 %v32 to i32<br class="gmail_msg">
> + %v40 = and i32 %v39, 32767<br class="gmail_msg">
> + %v41 = trunc i32 %v40 to i16<br class="gmail_msg">
> + br label %b42<br class="gmail_msg">
> +<br class="gmail_msg">
> +b42: ; preds = %b38, %b34<br class="gmail_msg">
> + %v43 = phi i16 [ %v37, %b34 ], [ %v41, %b38 ]<br class="gmail_msg">
> + br label %b44<br class="gmail_msg">
> +<br class="gmail_msg">
> +b44: ; preds = %b42<br class="gmail_msg">
> + %v45 = add i8 %v6, 1<br class="gmail_msg">
> + br label %b3<br class="gmail_msg">
> +<br class="gmail_msg">
> +b46: ; preds = %b3<br class="gmail_msg">
> + ret i16 %v5<br class="gmail_msg">
> +}<br class="gmail_msg">
> +<br class="gmail_msg">
> +attributes #0 = { noinline nounwind "target-cpu"="hexagonv5"<br class="gmail_msg">
> "target-features"="-hvx,-hvx-double,-long-calls" }<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> _______________________________________________<br class="gmail_msg">
> llvm-commits mailing list<br class="gmail_msg">
> <a href="mailto:llvm-commits@lists.llvm.org" class="gmail_msg" target="_blank">llvm-commits@lists.llvm.org</a> <mailto:<a href="mailto:llvm-commits@lists.llvm.org" class="gmail_msg" target="_blank">llvm-commits@lists.llvm.org</a>><br class="gmail_msg">
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br class="gmail_msg">
><br class="gmail_msg">
<br class="gmail_msg">
--<br class="gmail_msg">
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,<br class="gmail_msg">
hosted by The Linux Foundation<br class="gmail_msg">
</blockquote></div></blockquote></div>