<div dir="ltr">The compile time regression is still present after that.<br><br><div class="gmail_quote"><div dir="ltr">On Thu, Mar 2, 2017 at 11:56 PM Mikael Holmén via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br class="gmail_msg">
<br class="gmail_msg">
On 03/03/2017 10:28 AM, Chandler Carruth wrote:<br class="gmail_msg">
> I also now have an over 20x compile time regression that this triggers.<br class="gmail_msg">
> Sadly my test case is still huge, but we can reduce it. Especially given<br class="gmail_msg">
> the crasher, i'm going to revert for now.<br class="gmail_msg">
<br class="gmail_msg">
As far as I know, the crash has been fixed already in r296668, see:<br class="gmail_msg">
<br class="gmail_msg">
<a href="http://bugs.llvm.org/show_bug.cgi?id=32108" rel="noreferrer" class="gmail_msg" target="_blank">http://bugs.llvm.org/show_bug.cgi?id=32108</a><br class="gmail_msg">
<br class="gmail_msg">
The regressions you're seeing might be enough for revert anyway, I don't<br class="gmail_msg">
know, but at least the crash seems to be fixed.<br class="gmail_msg">
<br class="gmail_msg">
Regards,<br class="gmail_msg">
Mikael<br class="gmail_msg">
<br class="gmail_msg">
><br class="gmail_msg">
> On Wed, Mar 1, 2017 at 12:48 AM Mikael Holmén via llvm-commits<br class="gmail_msg">
> <<a href="mailto:llvm-commits@lists.llvm.org" class="gmail_msg" target="_blank">llvm-commits@lists.llvm.org</a> <mailto:<a href="mailto:llvm-commits@lists.llvm.org" class="gmail_msg" target="_blank">llvm-commits@lists.llvm.org</a>>> wrote:<br class="gmail_msg">
><br class="gmail_msg">
> Hi Nirav,<br class="gmail_msg">
><br class="gmail_msg">
> The attached (llvm-stress generated and then bugpoint reduced) ll-file<br class="gmail_msg">
> crashes llc with this commit:<br class="gmail_msg">
><br class="gmail_msg">
> llc -march=x86-64 stress_reduced.ll<br class="gmail_msg">
><br class="gmail_msg">
> gives<br class="gmail_msg">
><br class="gmail_msg">
> llc: ../lib/CodeGen/SelectionDAG/SelectionDAG.cpp:746: bool<br class="gmail_msg">
> llvm::SelectionDAG::RemoveNodeFromCSEMaps(llvm::SDNode *): Assertion<br class="gmail_msg">
> `N->getOpcode() != ISD::DELETED_NODE && "DELETED_NODE in CSEMap!"'<br class="gmail_msg">
> failed.<br class="gmail_msg">
> #0 0x0b161bcc llvm::sys::PrintStackTrace(llvm::raw_ostream&)<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/Support/Unix/Signals.inc:402:5<br class="gmail_msg">
> #1 0x0b161dfe PrintStackTraceSignalHandler(void*)<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/Support/Unix/Signals.inc:466:1<br class="gmail_msg">
> #2 0x0b15fdff llvm::sys::RunSignalHandlers()<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/Support/Signals.cpp:45:5<br class="gmail_msg">
> #3 0x0b1625b1 SignalHandler(int)<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/Support/Unix/Signals.inc:256:1<br class="gmail_msg">
> #4 0xf77c6410 0xffffe410 llvm::SelectionDAG::DeleteNode(llvm::SDNode*)<br class="gmail_msg">
> #5 0xf77c6410<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/CodeGen/SelectionDAG/SelectionDAG.cpp:615:26<br class="gmail_msg">
><br class="gmail_msg">
> #6 0xf77c6410 (anonymous<br class="gmail_msg">
> namespace)::DAGCombiner::recursivelyDeleteUnusedNodes(llvm::SDNode*)<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/CodeGen/SelectionDAG/DAGCombiner.cpp:1260:5<br class="gmail_msg">
> #7 0x0aecc079 (anonymous<br class="gmail_msg">
> namespace)::DAGCombiner::Run(llvm::CombineLevel)<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/CodeGen/SelectionDAG/DAGCombiner.cpp:1302:9<br class="gmail_msg">
> #8 0x0ad0cb1f llvm::SelectionDAG::Combine(llvm::CombineLevel,<br class="gmail_msg">
> llvm::AAResults&, llvm::CodeGenOpt::Level)<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/CodeGen/SelectionDAG/DAGCombiner.cpp:16066:3<br class="gmail_msg">
> #9 0x0ad0c044 llvm::SelectionDAGISel::CodeGenAndEmitDAG()<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:871:7<br class="gmail_msg">
> #10 0x0ad0bcd6<br class="gmail_msg">
> llvm::SelectionDAGISel::SelectBasicBlock(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::Instruction,<br class="gmail_msg">
> true, false, void>, false, true>,<br class="gmail_msg">
> llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::Instruction,<br class="gmail_msg">
> true, false, void>, false, true>, bool&)<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:726:1<br class="gmail_msg">
> #11 0x0af36f98<br class="gmail_msg">
> llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&)<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:1689:5<br class="gmail_msg">
> #12 0x0af357a7<br class="gmail_msg">
> llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&)<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:543:7<br class="gmail_msg">
> #13 0x0af3558a (anonymous<br class="gmail_msg">
> namespace)::X86DAGToDAGISel::runOnMachineFunction(llvm::MachineFunction&)<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/Target/X86/X86ISelDAGToDAG.cpp:176:25<br class="gmail_msg">
> #14 0x0af32754 llvm::MachineFunctionPass::runOnFunction(llvm::Function&)<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/CodeGen/MachineFunctionPass.cpp:62:8<br class="gmail_msg">
> #15 0x09936a8d llvm::FPPassManager::runOnFunction(llvm::Function&)<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/IR/LegacyPassManager.cpp:1513:23<br class="gmail_msg">
> #16 0x0a2c0401 llvm::FPPassManager::runOnModule(llvm::Module&)<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/IR/LegacyPassManager.cpp:1534:16<br class="gmail_msg">
> #17 0x0a7f6c23 (anonymous<br class="gmail_msg">
> namespace)::MPPassManager::runOnModule(llvm::Module&)<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/IR/LegacyPassManager.cpp:1590:23<br class="gmail_msg">
> #18 0x0a7f6fca llvm::legacy::PassManagerImpl::run(llvm::Module&)<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/IR/LegacyPassManager.cpp:1693:16<br class="gmail_msg">
> #19 0x0a7f788c llvm::legacy::PassManager::run(llvm::Module&)<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../lib/IR/LegacyPassManager.cpp:1724:3<br class="gmail_msg">
> #20 0x0a7f72dc compileModule(char**, llvm::LLVMContext&)<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../tools/llc/llc.cpp:579:42<br class="gmail_msg">
> #21 0x0a7f7ee6 main<br class="gmail_msg">
> /data/repo/llvm-patch/build-all-Debug/../tools/llc/llc.cpp:331:13<br class="gmail_msg">
> #22 0x0860a90b __libc_start_main<br class="gmail_msg">
> /build/eglibc-X4bnBz/eglibc-2.19/csu/libc-start.c:321:0<br class="gmail_msg">
> build-all-Debug/bin/llc(_ZN4llvm3sys15PrintStackTraceERNS_11raw_ostreamE+0x5c)[0xb161bcc]<br class="gmail_msg">
> build-all-Debug/bin/llc[0xb161dfe]<br class="gmail_msg">
> build-all-Debug/bin/llc(_ZN4llvm3sys17RunSignalHandlersEv+0xbf)[0xb15fdff]<br class="gmail_msg">
> build-all-Debug/bin/llc[0xb1625b1]<br class="gmail_msg">
> [0xf77c6410]<br class="gmail_msg">
> build-all-Debug/bin/llc(_ZN4llvm12SelectionDAG10DeleteNodeEPNS_6SDNodeE+0x39)[0xaecc079]<br class="gmail_msg">
> build-all-Debug/bin/llc[0xad0cb1f]<br class="gmail_msg">
> build-all-Debug/bin/llc[0xad0c044]<br class="gmail_msg">
> build-all-Debug/bin/llc(_ZN4llvm12SelectionDAG7CombineENS_12CombineLevelERNS_9AAResultsENS_10CodeGenOpt5LevelE+0x76)[0xad0bcd6]<br class="gmail_msg">
> build-all-Debug/bin/llc(_ZN4llvm16SelectionDAGISel17CodeGenAndEmitDAGEv+0x17e8)[0xaf36f98]<br class="gmail_msg">
> build-all-Debug/bin/llc(_ZN4llvm16SelectionDAGISel16SelectBasicBlockENS_14ilist_iteratorINS_12ilist_detail12node_optionsINS_11InstructionELb1ELb0EvEELb0ELb1EEES6_Rb+0x137)[0xaf357a7]<br class="gmail_msg">
> build-all-Debug/bin/llc(_ZN4llvm16SelectionDAGISel20SelectAllBasicBlocksERKNS_8FunctionE+0x144a)[0xaf3558a]<br class="gmail_msg">
> build-all-Debug/bin/llc(_ZN4llvm16SelectionDAGISel20runOnMachineFunctionERNS_15MachineFunctionE+0x734)[0xaf32754]<br class="gmail_msg">
> build-all-Debug/bin/llc[0x9936a8d]<br class="gmail_msg">
> build-all-Debug/bin/llc(_ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE+0x251)[0xa2c0401]<br class="gmail_msg">
> build-all-Debug/bin/llc(_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE+0x1b3)[0xa7f6c23]<br class="gmail_msg">
> build-all-Debug/bin/llc(_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE+0xaa)[0xa7f6fca]<br class="gmail_msg">
> build-all-Debug/bin/llc[0xa7f788c]<br class="gmail_msg">
> build-all-Debug/bin/llc(_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE+0x14c)[0xa7f72dc]<br class="gmail_msg">
> build-all-Debug/bin/llc(_ZN4llvm6legacy11PassManager3runERNS_6ModuleE+0x36)[0xa7f7ee6]<br class="gmail_msg">
> build-all-Debug/bin/llc[0x860a90b]<br class="gmail_msg">
> build-all-Debug/bin/llc(main+0x5b0)[0x86084e0]<br class="gmail_msg">
> /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf7493af3]<br class="gmail_msg">
> Stack dump:<br class="gmail_msg">
> 0. Program arguments: build-all-Debug/bin/llc -march=x86-64<br class="gmail_msg">
> stress_reduced.ll<br class="gmail_msg">
> 1. Running pass 'Function Pass Manager' on module<br class="gmail_msg">
> 'stress_reduced.ll'.<br class="gmail_msg">
> 2. Running pass 'X86 DAG->DAG Instruction Selection' on function<br class="gmail_msg">
> '@autogen_SD1794'<br class="gmail_msg">
> Abort<br class="gmail_msg">
><br class="gmail_msg">
> Regards,<br class="gmail_msg">
> Mikael<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> On 02/28/2017 03:24 PM, Nirav Dave via llvm-commits wrote:<br class="gmail_msg">
> > Author: niravd<br class="gmail_msg">
> > Date: Tue Feb 28 08:24:15 2017<br class="gmail_msg">
> > New Revision: 296476<br class="gmail_msg">
> ><br class="gmail_msg">
> > URL: <a href="http://llvm.org/viewvc/llvm-project?rev=296476&view=rev" rel="noreferrer" class="gmail_msg" target="_blank">http://llvm.org/viewvc/llvm-project?rev=296476&view=rev</a><br class="gmail_msg">
> > Log:<br class="gmail_msg">
> > In visitSTORE, always use FindBetterChain, rather than only when<br class="gmail_msg">
> UseAA is enabled.<br class="gmail_msg">
> ><br class="gmail_msg">
> > Recommiting after fixup of 32-bit aliasing sign offset bug in<br class="gmail_msg">
> DAGCombiner.<br class="gmail_msg">
> ><br class="gmail_msg">
> > * Simplify Consecutive Merge Store Candidate Search<br class="gmail_msg">
> ><br class="gmail_msg">
> > Now that address aliasing is much less conservative, push through<br class="gmail_msg">
> > simplified store merging search and chain alias analysis which<br class="gmail_msg">
> only<br class="gmail_msg">
> > checks for parallel stores through the chain subgraph. This is<br class="gmail_msg">
> cleaner<br class="gmail_msg">
> > as the separation of non-interfering loads/stores from the<br class="gmail_msg">
> > store-merging logic.<br class="gmail_msg">
> ><br class="gmail_msg">
> > When merging stores search up the chain through a single load, and<br class="gmail_msg">
> > finds all possible stores by looking down from through a load<br class="gmail_msg">
> and a<br class="gmail_msg">
> > TokenFactor to all stores visited.<br class="gmail_msg">
> ><br class="gmail_msg">
> > This improves the quality of the output SelectionDAG and the<br class="gmail_msg">
> output<br class="gmail_msg">
> > Codegen (save perhaps for some ARM cases where we correctly<br class="gmail_msg">
> constructs<br class="gmail_msg">
> > wider loads, but then promotes them to float operations which<br class="gmail_msg">
> appear<br class="gmail_msg">
> > but requires more expensive constant generation).<br class="gmail_msg">
> ><br class="gmail_msg">
> > Some minor peephole optimizations to deal with improved SubDAG<br class="gmail_msg">
> shapes (listed below)<br class="gmail_msg">
> ><br class="gmail_msg">
> > Additional Minor Changes:<br class="gmail_msg">
> ><br class="gmail_msg">
> > 1. Finishes removing unused AliasLoad code<br class="gmail_msg">
> ><br class="gmail_msg">
> > 2. Unifies the chain aggregation in the merged stores across<br class="gmail_msg">
> code<br class="gmail_msg">
> > paths<br class="gmail_msg">
> ><br class="gmail_msg">
> > 3. Re-add the Store node to the worklist after calling<br class="gmail_msg">
> > SimplifyDemandedBits.<br class="gmail_msg">
> ><br class="gmail_msg">
> > 4. Increase GatherAllAliasesMaxDepth from 6 to 18. That<br class="gmail_msg">
> number is<br class="gmail_msg">
> > arbitrary, but seems sufficient to not cause regressions in<br class="gmail_msg">
> > tests.<br class="gmail_msg">
> ><br class="gmail_msg">
> > 5. Remove Chain dependencies of Memory operations on CopyfromReg<br class="gmail_msg">
> > nodes as these are captured by data dependence<br class="gmail_msg">
> ><br class="gmail_msg">
> > 6. Forward loads-store values through tokenfactors containing<br class="gmail_msg">
> > {CopyToReg,CopyFromReg} Values.<br class="gmail_msg">
> ><br class="gmail_msg">
> > 7. Peephole to convert buildvector of extract_vector_elt to<br class="gmail_msg">
> > extract_subvector if possible (see<br class="gmail_msg">
> > CodeGen/AArch64/store-merge.ll)<br class="gmail_msg">
> ><br class="gmail_msg">
> > 8. Store merging for the ARM target is restricted to 32-bit as<br class="gmail_msg">
> > some in some contexts invalid 64-bit operations are being<br class="gmail_msg">
> > generated. This can be removed once appropriate checks are<br class="gmail_msg">
> > added.<br class="gmail_msg">
> ><br class="gmail_msg">
> > This finishes the change Matt Arsenault started in r246307 and<br class="gmail_msg">
> > jyknight's original patch.<br class="gmail_msg">
> ><br class="gmail_msg">
> > Many tests required some changes as memory operations are now<br class="gmail_msg">
> > reorderable, improving load-store forwarding. One test in<br class="gmail_msg">
> > particular is worth noting:<br class="gmail_msg">
> ><br class="gmail_msg">
> > CodeGen/PowerPC/ppc64-align-long-double.ll - Improved load-store<br class="gmail_msg">
> > forwarding converts a load-store pair into a parallel store and<br class="gmail_msg">
> > a memory-realized bitcast of the same value. However, because we<br class="gmail_msg">
> > lose the sharing of the explicit and implicit store values we<br class="gmail_msg">
> > must create another local store. A similar transformation<br class="gmail_msg">
> > happens before SelectionDAG as well.<br class="gmail_msg">
> ><br class="gmail_msg">
> > Reviewers: arsenm, hfinkel, tstellarAMD, jyknight, nhaehnle<br class="gmail_msg">
> ><br class="gmail_msg">
> > Removed:<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/combiner-aa-0.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/combiner-aa-1.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/pr18023.ll<br class="gmail_msg">
> > Modified:<br class="gmail_msg">
> > llvm/trunk/include/llvm/Target/TargetLowering.h<br class="gmail_msg">
> > llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp<br class="gmail_msg">
> > llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp<br class="gmail_msg">
> > llvm/trunk/lib/Target/AArch64/AArch64ISelLowering.cpp<br class="gmail_msg">
> > llvm/trunk/lib/Target/ARM/ARMISelLowering.h<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/AArch64/argument-blocks.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/AArch64/arm64-abi.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/AArch64/arm64-memset-inline.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/AArch64/arm64-variadic-aapcs.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/AArch64/merge-store.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/AArch64/vector_merge_dep_check.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/AMDGPU/debugger-insert-nops.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/AMDGPU/insert_vector_elt.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/AMDGPU/merge-stores.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/AMDGPU/private-element-size.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/AMDGPU/si-triv-disjoint-mem-access.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/ARM/2012-10-04-AAPCS-byval-align8.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/ARM/alloc-no-stack-realign.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/ARM/gpr-paired-spill.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/ARM/ifcvt10.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/ARM/illegal-bitfield-loadstore.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/ARM/static-addr-hoisting.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/BPF/undef.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/MSP430/Inst16mm.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/Mips/cconv/arguments-float.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/Mips/cconv/arguments-varargs.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/Mips/fastcc.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/Mips/load-store-left-right.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/Mips/micromips-li.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/Mips/mips64-f128-call.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/Mips/mips64-f128.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/Mips/mno-ldc1-sdc1.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/Mips/msa/f16-llvm-ir.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/Mips/msa/i5_ld_st.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/Mips/o32_cc_byval.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/Mips/o32_cc_vararg.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/PowerPC/anon_aggr.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/PowerPC/complex-return.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/PowerPC/jaggedstructs.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/PowerPC/ppc64-align-long-double.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/PowerPC/structsinmem.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/PowerPC/structsinregs.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/SystemZ/unaligned-01.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/Thumb/2010-07-15-debugOrdering.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/Thumb/stack-access.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/2010-09-17-SideEffectsInChain.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/2012-11-28-merge-store-alias.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/MergeConsecutiveStores.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/avx512-mask-op.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/chain_order.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/clear_upper_vector_element_bits.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/copy-eflags.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/dag-merge-fast-accesses.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/dont-trunc-store-double-to-float.ll<br class="gmail_msg">
> ><br class="gmail_msg">
> llvm/trunk/test/CodeGen/X86/extractelement-legalization-store-ordering.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/i256-add.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/i386-shrink-wrapping.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/illegal-bitfield-loadstore.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/live-range-nosubreg.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/longlong-deadload.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/merge-consecutive-loads-128.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/merge-consecutive-loads-256.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/merge-store-partially-alias-loads.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/split-store.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/stores-merging.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/vector-compare-results.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/vector-shuffle-variable-128.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/vector-shuffle-variable-256.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/vectorcall.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/X86/win32-eh.ll<br class="gmail_msg">
> > llvm/trunk/test/CodeGen/XCore/varargs.ll<br class="gmail_msg">
> ><br class="gmail_msg">
> > Modified: llvm/trunk/include/llvm/Target/TargetLowering.h<br class="gmail_msg">
> > URL:<br class="gmail_msg">
> <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Target/TargetLowering.h?rev=296476&r1=296475&r2=296476&view=diff" rel="noreferrer" class="gmail_msg" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Target/TargetLowering.h?rev=296476&r1=296475&r2=296476&view=diff</a><br class="gmail_msg">
> ><br class="gmail_msg">
> ==============================================================================<br class="gmail_msg">
> > --- llvm/trunk/include/llvm/Target/TargetLowering.h (original)<br class="gmail_msg">
> > +++ llvm/trunk/include/llvm/Target/TargetLowering.h Tue Feb 28<br class="gmail_msg">
> 08:24:15 2017<br class="gmail_msg">
> > @@ -363,6 +363,9 @@ public:<br class="gmail_msg">
> > return false;<br class="gmail_msg">
> > }<br class="gmail_msg">
> ><br class="gmail_msg">
> > + /// Returns if it's reasonable to merge stores to MemVT size.<br class="gmail_msg">
> > + virtual bool canMergeStoresTo(EVT MemVT) const { return true; }<br class="gmail_msg">
> > +<br class="gmail_msg">
> > /// \brief Return true if it is cheap to speculate a call to<br class="gmail_msg">
> intrinsic cttz.<br class="gmail_msg">
> > virtual bool isCheapToSpeculateCttz() const {<br class="gmail_msg">
> > return false;<br class="gmail_msg">
> ><br class="gmail_msg">
> > Modified: llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp<br class="gmail_msg">
> > URL:<br class="gmail_msg">
> <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp?rev=296476&r1=296475&r2=296476&view=diff" rel="noreferrer" class="gmail_msg" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp?rev=296476&r1=296475&r2=296476&view=diff</a><br class="gmail_msg">
> ><br class="gmail_msg">
> ==============================================================================<br class="gmail_msg">
> > --- llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp (original)<br class="gmail_msg">
> > +++ llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp Tue Feb 28<br class="gmail_msg">
> 08:24:15 2017<br class="gmail_msg">
> > @@ -53,10 +53,6 @@ STATISTIC(SlicedLoads, "Number of load s<br class="gmail_msg">
> ><br class="gmail_msg">
> > namespace {<br class="gmail_msg">
> > static cl::opt<bool><br class="gmail_msg">
> > - CombinerAA("combiner-alias-analysis", cl::Hidden,<br class="gmail_msg">
> > - cl::desc("Enable DAG combiner alias-analysis<br class="gmail_msg">
> heuristics"));<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - static cl::opt<bool><br class="gmail_msg">
> > CombinerGlobalAA("combiner-global-alias-analysis", cl::Hidden,<br class="gmail_msg">
> > cl::desc("Enable DAG combiner's use of IR alias<br class="gmail_msg">
> analysis"));<br class="gmail_msg">
> ><br class="gmail_msg">
> > @@ -419,15 +415,12 @@ namespace {<br class="gmail_msg">
> > /// Holds a pointer to an LSBaseSDNode as well as information<br class="gmail_msg">
> on where it<br class="gmail_msg">
> > /// is located in a sequence of memory operations connected<br class="gmail_msg">
> by a chain.<br class="gmail_msg">
> > struct MemOpLink {<br class="gmail_msg">
> > - MemOpLink (LSBaseSDNode *N, int64_t Offset, unsigned Seq):<br class="gmail_msg">
> > - MemNode(N), OffsetFromBase(Offset), SequenceNum(Seq) { }<br class="gmail_msg">
> > + MemOpLink(LSBaseSDNode *N, int64_t Offset)<br class="gmail_msg">
> > + : MemNode(N), OffsetFromBase(Offset) {}<br class="gmail_msg">
> > // Ptr to the mem node.<br class="gmail_msg">
> > LSBaseSDNode *MemNode;<br class="gmail_msg">
> > // Offset from the base ptr.<br class="gmail_msg">
> > int64_t OffsetFromBase;<br class="gmail_msg">
> > - // What is the sequence number of this mem node.<br class="gmail_msg">
> > - // Lowest mem operand in the DAG starts at zero.<br class="gmail_msg">
> > - unsigned SequenceNum;<br class="gmail_msg">
> > };<br class="gmail_msg">
> ><br class="gmail_msg">
> > /// This is a helper function for visitMUL to check the<br class="gmail_msg">
> profitability<br class="gmail_msg">
> > @@ -438,12 +431,6 @@ namespace {<br class="gmail_msg">
> > SDValue &AddNode,<br class="gmail_msg">
> > SDValue &ConstNode);<br class="gmail_msg">
> ><br class="gmail_msg">
> > - /// This is a helper function for<br class="gmail_msg">
> MergeStoresOfConstantsOrVecElts. Returns a<br class="gmail_msg">
> > - /// constant build_vector of the stored constant values in<br class="gmail_msg">
> Stores.<br class="gmail_msg">
> > - SDValue getMergedConstantVectorStore(SelectionDAG &DAG, const<br class="gmail_msg">
> SDLoc &SL,<br class="gmail_msg">
> > - ArrayRef<MemOpLink> Stores,<br class="gmail_msg">
> > - SmallVectorImpl<SDValue><br class="gmail_msg">
> &Chains,<br class="gmail_msg">
> > - EVT Ty) const;<br class="gmail_msg">
> ><br class="gmail_msg">
> > /// This is a helper function for visitAND and<br class="gmail_msg">
> visitZERO_EXTEND. Returns<br class="gmail_msg">
> > /// true if the (and (load x) c) pattern matches an extload.<br class="gmail_msg">
> ExtVT returns<br class="gmail_msg">
> > @@ -457,18 +444,15 @@ namespace {<br class="gmail_msg">
> > /// This is a helper function for MergeConsecutiveStores.<br class="gmail_msg">
> When the source<br class="gmail_msg">
> > /// elements of the consecutive stores are all constants or<br class="gmail_msg">
> all extracted<br class="gmail_msg">
> > /// vector elements, try to merge them into one larger store.<br class="gmail_msg">
> > - /// \return number of stores that were merged into a merged<br class="gmail_msg">
> store (always<br class="gmail_msg">
> > - /// a prefix of \p StoreNode).<br class="gmail_msg">
> > - bool MergeStoresOfConstantsOrVecElts(<br class="gmail_msg">
> > - SmallVectorImpl<MemOpLink> &StoreNodes, EVT MemVT,<br class="gmail_msg">
> unsigned NumStores,<br class="gmail_msg">
> > - bool IsConstantSrc, bool UseVector);<br class="gmail_msg">
> > + /// \return True if a merged store was created.<br class="gmail_msg">
> > + bool<br class="gmail_msg">
> MergeStoresOfConstantsOrVecElts(SmallVectorImpl<MemOpLink> &StoreNodes,<br class="gmail_msg">
> > + EVT MemVT, unsigned<br class="gmail_msg">
> NumStores,<br class="gmail_msg">
> > + bool IsConstantSrc, bool<br class="gmail_msg">
> UseVector);<br class="gmail_msg">
> ><br class="gmail_msg">
> > /// This is a helper function for MergeConsecutiveStores.<br class="gmail_msg">
> > /// Stores that may be merged are placed in StoreNodes.<br class="gmail_msg">
> > - /// Loads that may alias with those stores are placed in<br class="gmail_msg">
> AliasLoadNodes.<br class="gmail_msg">
> > - void getStoreMergeAndAliasCandidates(<br class="gmail_msg">
> > - StoreSDNode* St, SmallVectorImpl<MemOpLink> &StoreNodes,<br class="gmail_msg">
> > - SmallVectorImpl<LSBaseSDNode*> &AliasLoadNodes);<br class="gmail_msg">
> > + void getStoreMergeCandidates(StoreSDNode *St,<br class="gmail_msg">
> > + SmallVectorImpl<MemOpLink><br class="gmail_msg">
> &StoreNodes);<br class="gmail_msg">
> ><br class="gmail_msg">
> > /// Helper function for MergeConsecutiveStores. Checks if<br class="gmail_msg">
> > /// Candidate stores have indirect dependency through their<br class="gmail_msg">
> > @@ -480,8 +464,7 @@ namespace {<br class="gmail_msg">
> > /// This optimization uses wide integers or vectors when<br class="gmail_msg">
> possible.<br class="gmail_msg">
> > /// \return number of stores that were merged into a merged<br class="gmail_msg">
> store (the<br class="gmail_msg">
> > /// affected nodes are stored as a prefix in \p StoreNodes).<br class="gmail_msg">
> > - bool MergeConsecutiveStores(StoreSDNode *N,<br class="gmail_msg">
> > - SmallVectorImpl<MemOpLink><br class="gmail_msg">
> &StoreNodes);<br class="gmail_msg">
> > + bool MergeConsecutiveStores(StoreSDNode *N);<br class="gmail_msg">
> ><br class="gmail_msg">
> > /// \brief Try to transform a truncation where C is a constant:<br class="gmail_msg">
> > /// (trunc (and X, C)) -> (and (trunc X), (trunc C))<br class="gmail_msg">
> > @@ -1584,7 +1567,7 @@ SDValue DAGCombiner::visitTokenFactor(SD<br class="gmail_msg">
> > }<br class="gmail_msg">
> ><br class="gmail_msg">
> > SmallVector<SDNode *, 8> TFs; // List of token factors to<br class="gmail_msg">
> visit.<br class="gmail_msg">
> > - SmallVector<SDValue, 8> Ops; // Ops for replacing token factor.<br class="gmail_msg">
> > + SmallVector<SDValue, 8> Ops; // Ops for replacing token<br class="gmail_msg">
> factor.<br class="gmail_msg">
> > SmallPtrSet<SDNode*, 16> SeenOps;<br class="gmail_msg">
> > bool Changed = false; // If we should replace this<br class="gmail_msg">
> token factor.<br class="gmail_msg">
> ><br class="gmail_msg">
> > @@ -1628,6 +1611,86 @@ SDValue DAGCombiner::visitTokenFactor(SD<br class="gmail_msg">
> > }<br class="gmail_msg">
> > }<br class="gmail_msg">
> ><br class="gmail_msg">
> > + // Remove Nodes that are chained to another node in the list. Do so<br class="gmail_msg">
> > + // by walking up chains breath-first stopping when we've seen<br class="gmail_msg">
> > + // another operand. In general we must climb to the EntryNode,<br class="gmail_msg">
> but we can exit<br class="gmail_msg">
> > + // early if we find all remaining work is associated with just<br class="gmail_msg">
> one operand as<br class="gmail_msg">
> > + // no further pruning is possible.<br class="gmail_msg">
> > +<br class="gmail_msg">
> > + // List of nodes to search through and original Ops from which<br class="gmail_msg">
> they originate.<br class="gmail_msg">
> > + SmallVector<std::pair<SDNode *, unsigned>, 8> Worklist;<br class="gmail_msg">
> > + SmallVector<unsigned, 8> OpWorkCount; // Count of work for each Op.<br class="gmail_msg">
> > + SmallPtrSet<SDNode *, 16> SeenChains;<br class="gmail_msg">
> > + bool DidPruneOps = false;<br class="gmail_msg">
> > +<br class="gmail_msg">
> > + unsigned NumLeftToConsider = 0;<br class="gmail_msg">
> > + for (const SDValue &Op : Ops) {<br class="gmail_msg">
> > + Worklist.push_back(std::make_pair(Op.getNode(),<br class="gmail_msg">
> NumLeftToConsider++));<br class="gmail_msg">
> > + OpWorkCount.push_back(1);<br class="gmail_msg">
> > + }<br class="gmail_msg">
> > +<br class="gmail_msg">
> > + auto AddToWorklist = [&](unsigned CurIdx, SDNode *Op, unsigned<br class="gmail_msg">
> OpNumber) {<br class="gmail_msg">
> > + // If this is an Op, we can remove the op from the list.<br class="gmail_msg">
> Remark any<br class="gmail_msg">
> > + // search associated with it as from the current OpNumber.<br class="gmail_msg">
> > + if (SeenOps.count(Op) != 0) {<br class="gmail_msg">
> > + Changed = true;<br class="gmail_msg">
> > + DidPruneOps = true;<br class="gmail_msg">
> > + unsigned OrigOpNumber = 0;<br class="gmail_msg">
> > + while (Ops[OrigOpNumber].getNode() != Op && OrigOpNumber <<br class="gmail_msg">
> Ops.size())<br class="gmail_msg">
> > + OrigOpNumber++;<br class="gmail_msg">
> > + assert((OrigOpNumber != Ops.size()) &&<br class="gmail_msg">
> > + "expected to find TokenFactor Operand");<br class="gmail_msg">
> > + // Re-mark worklist from OrigOpNumber to OpNumber<br class="gmail_msg">
> > + for (unsigned i = CurIdx + 1; i < Worklist.size(); ++i) {<br class="gmail_msg">
> > + if (Worklist[i].second == OrigOpNumber) {<br class="gmail_msg">
> > + Worklist[i].second = OpNumber;<br class="gmail_msg">
> > + }<br class="gmail_msg">
> > + }<br class="gmail_msg">
> > + OpWorkCount[OpNumber] += OpWorkCount[OrigOpNumber];<br class="gmail_msg">
> > + OpWorkCount[OrigOpNumber] = 0;<br class="gmail_msg">
> > + NumLeftToConsider--;<br class="gmail_msg">
> > + }<br class="gmail_msg">
> > + // Add if it's a new chain<br class="gmail_msg">
> > + if (SeenChains.insert(Op).second) {<br class="gmail_msg">
> > + OpWorkCount[OpNumber]++;<br class="gmail_msg">
> > + Worklist.push_back(std::make_pair(Op, OpNumber));<br class="gmail_msg">
> > + }<br class="gmail_msg">
> > + };<br class="gmail_msg">
> > +<br class="gmail_msg">
> > + for (unsigned i = 0; i < Worklist.size(); ++i) {<br class="gmail_msg">
> > + // We need at least be consider at least 2 Ops to prune.<br class="gmail_msg">
> > + if (NumLeftToConsider <= 1)<br class="gmail_msg">
> > + break;<br class="gmail_msg">
> > + auto CurNode = Worklist[i].first;<br class="gmail_msg">
> > + auto CurOpNumber = Worklist[i].second;<br class="gmail_msg">
> > + assert((OpWorkCount[CurOpNumber] > 0) &&<br class="gmail_msg">
> > + "Node should not appear in worklist");<br class="gmail_msg">
> > + switch (CurNode->getOpcode()) {<br class="gmail_msg">
> > + case ISD::EntryToken:<br class="gmail_msg">
> > + // Hitting EntryToken is the only way for the search to<br class="gmail_msg">
> terminate without<br class="gmail_msg">
> > + // hitting<br class="gmail_msg">
> > + // another operand's search. Prevent us from marking this<br class="gmail_msg">
> operand<br class="gmail_msg">
> > + // considered.<br class="gmail_msg">
> > + NumLeftToConsider++;<br class="gmail_msg">
> > + break;<br class="gmail_msg">
> > + case ISD::TokenFactor:<br class="gmail_msg">
> > + for (const SDValue &Op : CurNode->op_values())<br class="gmail_msg">
> > + AddToWorklist(i, Op.getNode(), CurOpNumber);<br class="gmail_msg">
> > + break;<br class="gmail_msg">
> > + case ISD::CopyFromReg:<br class="gmail_msg">
> > + case ISD::CopyToReg:<br class="gmail_msg">
> > + AddToWorklist(i, CurNode->getOperand(0).getNode(),<br class="gmail_msg">
> CurOpNumber);<br class="gmail_msg">
> > + break;<br class="gmail_msg">
> > + default:<br class="gmail_msg">
> > + if (auto *MemNode = dyn_cast<MemSDNode>(CurNode))<br class="gmail_msg">
> > + AddToWorklist(i, MemNode->getChain().getNode(), CurOpNumber);<br class="gmail_msg">
> > + break;<br class="gmail_msg">
> > + }<br class="gmail_msg">
> > + OpWorkCount[CurOpNumber]--;<br class="gmail_msg">
> > + if (OpWorkCount[CurOpNumber] == 0)<br class="gmail_msg">
> > + NumLeftToConsider--;<br class="gmail_msg">
> > + }<br class="gmail_msg">
> > +<br class="gmail_msg">
> > SDValue Result;<br class="gmail_msg">
> ><br class="gmail_msg">
> > // If we've changed things around then replace token factor.<br class="gmail_msg">
> > @@ -1636,15 +1699,22 @@ SDValue DAGCombiner::visitTokenFactor(SD<br class="gmail_msg">
> > // The entry token is the only possible outcome.<br class="gmail_msg">
> > Result = DAG.getEntryNode();<br class="gmail_msg">
> > } else {<br class="gmail_msg">
> > - // New and improved token factor.<br class="gmail_msg">
> > - Result = DAG.getNode(ISD::TokenFactor, SDLoc(N),<br class="gmail_msg">
> MVT::Other, Ops);<br class="gmail_msg">
> > + if (DidPruneOps) {<br class="gmail_msg">
> > + SmallVector<SDValue, 8> PrunedOps;<br class="gmail_msg">
> > + //<br class="gmail_msg">
> > + for (const SDValue &Op : Ops) {<br class="gmail_msg">
> > + if (SeenChains.count(Op.getNode()) == 0)<br class="gmail_msg">
> > + PrunedOps.push_back(Op);<br class="gmail_msg">
> > + }<br class="gmail_msg">
> > + Result = DAG.getNode(ISD::TokenFactor, SDLoc(N),<br class="gmail_msg">
> MVT::Other, PrunedOps);<br class="gmail_msg">
> > + } else {<br class="gmail_msg">
> > + Result = DAG.getNode(ISD::TokenFactor, SDLoc(N),<br class="gmail_msg">
> MVT::Other, Ops);<br class="gmail_msg">
> > + }<br class="gmail_msg">
> > }<br class="gmail_msg">
> ><br class="gmail_msg">
> > - // Add users to worklist if AA is enabled, since it may introduce<br class="gmail_msg">
> > - // a lot of new chained token factors while removing memory deps.<br class="gmail_msg">
> > - bool UseAA = CombinerAA.getNumOccurrences() > 0 ? CombinerAA<br class="gmail_msg">
> > - : DAG.getSubtarget().useAA();<br class="gmail_msg">
> > - return CombineTo(N, Result, UseAA /*add to worklist*/);<br class="gmail_msg">
> > + // Add users to worklist, since we may introduce a lot of new<br class="gmail_msg">
> > + // chained token factors while removing memory deps.<br class="gmail_msg">
> > + return CombineTo(N, Result, true /*add to worklist*/);<br class="gmail_msg">
> > }<br class="gmail_msg">
> ><br class="gmail_msg">
> > return Result;<br class="gmail_msg">
> > @@ -6583,6 +6653,9 @@ SDValue DAGCombiner::CombineExtLoad(SDNo<br class="gmail_msg">
> > SDValue NewChain = DAG.getNode(ISD::TokenFactor, DL,<br class="gmail_msg">
> MVT::Other, Chains);<br class="gmail_msg">
> > SDValue NewValue = DAG.getNode(ISD::CONCAT_VECTORS, DL, DstVT,<br class="gmail_msg">
> Loads);<br class="gmail_msg">
> ><br class="gmail_msg">
> > + // Simplify TF.<br class="gmail_msg">
> > + AddToWorklist(NewChain.getNode());<br class="gmail_msg">
> > +<br class="gmail_msg">
> > CombineTo(N, NewValue);<br class="gmail_msg">
> ><br class="gmail_msg">
> > // Replace uses of the original load (before extension)<br class="gmail_msg">
> > @@ -10761,11 +10834,12 @@ SDValue DAGCombiner::visitLOAD(SDNode *N<br class="gmail_msg">
> > // TODO: Handle TRUNCSTORE/LOADEXT<br class="gmail_msg">
> > if (OptLevel != CodeGenOpt::None &&<br class="gmail_msg">
> > ISD::isNormalLoad(N) && !LD->isVolatile()) {<br class="gmail_msg">
> > + // We can forward a direct store or a store off of a tokenfactor.<br class="gmail_msg">
> > if (ISD::isNON_TRUNCStore(Chain.getNode())) {<br class="gmail_msg">
> > StoreSDNode *PrevST = cast<StoreSDNode>(Chain);<br class="gmail_msg">
> > if (PrevST->getBasePtr() == Ptr &&<br class="gmail_msg">
> > PrevST->getValue().getValueType() == N->getValueType(0))<br class="gmail_msg">
> > - return CombineTo(N, Chain.getOperand(1), Chain);<br class="gmail_msg">
> > + return CombineTo(N, PrevST->getOperand(1), Chain);<br class="gmail_msg">
> > }<br class="gmail_msg">
> > }<br class="gmail_msg">
> ><br class="gmail_msg">
> > @@ -10783,14 +10857,7 @@ SDValue DAGCombiner::visitLOAD(SDNode *N<br class="gmail_msg">
> > }<br class="gmail_msg">
> > }<br class="gmail_msg">
> ><br class="gmail_msg">
> > - bool UseAA = CombinerAA.getNumOccurrences() > 0 ? CombinerAA<br class="gmail_msg">
> > - :<br class="gmail_msg">
> DAG.getSubtarget().useAA();<br class="gmail_msg">
> > -#ifndef NDEBUG<br class="gmail_msg">
> > - if (CombinerAAOnlyFunc.getNumOccurrences() &&<br class="gmail_msg">
> > - CombinerAAOnlyFunc != DAG.getMachineFunction().getName())<br class="gmail_msg">
> > - UseAA = false;<br class="gmail_msg">
> > -#endif<br class="gmail_msg">
> > - if (UseAA && LD->isUnindexed()) {<br class="gmail_msg">
> > + if (LD->isUnindexed()) {<br class="gmail_msg">
> > // Walk up chain skipping non-aliasing memory nodes.<br class="gmail_msg">
> > SDValue BetterChain = FindBetterChain(N, Chain);<br class="gmail_msg">
> ><br class="gmail_msg">
> > @@ -11372,6 +11439,7 @@ bool DAGCombiner::SliceUpLoad(SDNode *N)<br class="gmail_msg">
> > SDValue Chain = DAG.getNode(ISD::TokenFactor, SDLoc(LD),<br class="gmail_msg">
> MVT::Other,<br class="gmail_msg">
> > ArgChains);<br class="gmail_msg">
> > DAG.ReplaceAllUsesOfValueWith(SDValue(N, 1), Chain);<br class="gmail_msg">
> > + AddToWorklist(Chain.getNode());<br class="gmail_msg">
> > return true;<br class="gmail_msg">
> > }<br class="gmail_msg">
> ><br class="gmail_msg">
> > @@ -11765,20 +11833,6 @@ bool DAGCombiner::isMulAddWithConstProfi<br class="gmail_msg">
> > return false;<br class="gmail_msg">
> > }<br class="gmail_msg">
> ><br class="gmail_msg">
> > -SDValue DAGCombiner::getMergedConstantVectorStore(<br class="gmail_msg">
> > - SelectionDAG &DAG, const SDLoc &SL, ArrayRef<MemOpLink> Stores,<br class="gmail_msg">
> > - SmallVectorImpl<SDValue> &Chains, EVT Ty) const {<br class="gmail_msg">
> > - SmallVector<SDValue, 8> BuildVector;<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - for (unsigned I = 0, E = Ty.getVectorNumElements(); I != E; ++I) {<br class="gmail_msg">
> > - StoreSDNode *St = cast<StoreSDNode>(Stores[I].MemNode);<br class="gmail_msg">
> > - Chains.push_back(St->getChain());<br class="gmail_msg">
> > - BuildVector.push_back(St->getValue());<br class="gmail_msg">
> > - }<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - return DAG.getBuildVector(Ty, SL, BuildVector);<br class="gmail_msg">
> > -}<br class="gmail_msg">
> > -<br class="gmail_msg">
> > bool DAGCombiner::MergeStoresOfConstantsOrVecElts(<br class="gmail_msg">
> > SmallVectorImpl<MemOpLink> &StoreNodes, EVT MemVT,<br class="gmail_msg">
> > unsigned NumStores, bool IsConstantSrc, bool<br class="gmail_msg">
> UseVector) {<br class="gmail_msg">
> > @@ -11787,22 +11841,8 @@ bool DAGCombiner::MergeStoresOfConstants<br class="gmail_msg">
> > return false;<br class="gmail_msg">
> ><br class="gmail_msg">
> > int64_t ElementSizeBytes = MemVT.getSizeInBits() / 8;<br class="gmail_msg">
> > - LSBaseSDNode *FirstInChain = StoreNodes[0].MemNode;<br class="gmail_msg">
> > - unsigned LatestNodeUsed = 0;<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - for (unsigned i=0; i < NumStores; ++i) {<br class="gmail_msg">
> > - // Find a chain for the new wide-store operand. Notice that some<br class="gmail_msg">
> > - // of the store nodes that we found may not be selected for<br class="gmail_msg">
> inclusion<br class="gmail_msg">
> > - // in the wide store. The chain we use needs to be the chain<br class="gmail_msg">
> of the<br class="gmail_msg">
> > - // latest store node which is *used* and replaced by the wide<br class="gmail_msg">
> store.<br class="gmail_msg">
> > - if (StoreNodes[i].SequenceNum <<br class="gmail_msg">
> StoreNodes[LatestNodeUsed].SequenceNum)<br class="gmail_msg">
> > - LatestNodeUsed = i;<br class="gmail_msg">
> > - }<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - SmallVector<SDValue, 8> Chains;<br class="gmail_msg">
> ><br class="gmail_msg">
> > // The latest Node in the DAG.<br class="gmail_msg">
> > - LSBaseSDNode *LatestOp = StoreNodes[LatestNodeUsed].MemNode;<br class="gmail_msg">
> > SDLoc DL(StoreNodes[0].MemNode);<br class="gmail_msg">
> ><br class="gmail_msg">
> > SDValue StoredVal;<br class="gmail_msg">
> > @@ -11818,7 +11858,18 @@ bool DAGCombiner::MergeStoresOfConstants<br class="gmail_msg">
> > assert(TLI.isTypeLegal(Ty) && "Illegal vector store");<br class="gmail_msg">
> ><br class="gmail_msg">
> > if (IsConstantSrc) {<br class="gmail_msg">
> > - StoredVal = getMergedConstantVectorStore(DAG, DL,<br class="gmail_msg">
> StoreNodes, Chains, Ty);<br class="gmail_msg">
> > + SmallVector<SDValue, 8> BuildVector;<br class="gmail_msg">
> > + for (unsigned I = 0, E = Ty.getVectorNumElements(); I != E;<br class="gmail_msg">
> ++I) {<br class="gmail_msg">
> > + StoreSDNode *St = cast<StoreSDNode>(StoreNodes[I].MemNode);<br class="gmail_msg">
> > + SDValue Val = St->getValue();<br class="gmail_msg">
> > + if (MemVT.getScalarType().isInteger())<br class="gmail_msg">
> > + if (auto *CFP = dyn_cast<ConstantFPSDNode>(St->getValue()))<br class="gmail_msg">
> > + Val = DAG.getConstant(<br class="gmail_msg">
> > +<br class="gmail_msg">
> (uint32_t)CFP->getValueAPF().bitcastToAPInt().getZExtValue(),<br class="gmail_msg">
> > + SDLoc(CFP), MemVT);<br class="gmail_msg">
> > + BuildVector.push_back(Val);<br class="gmail_msg">
> > + }<br class="gmail_msg">
> > + StoredVal = DAG.getBuildVector(Ty, DL, BuildVector);<br class="gmail_msg">
> > } else {<br class="gmail_msg">
> > SmallVector<SDValue, 8> Ops;<br class="gmail_msg">
> > for (unsigned i = 0; i < NumStores; ++i) {<br class="gmail_msg">
> > @@ -11828,7 +11879,6 @@ bool DAGCombiner::MergeStoresOfConstants<br class="gmail_msg">
> > if (Val.getValueType() != MemVT)<br class="gmail_msg">
> > return false;<br class="gmail_msg">
> > Ops.push_back(Val);<br class="gmail_msg">
> > - Chains.push_back(St->getChain());<br class="gmail_msg">
> > }<br class="gmail_msg">
> ><br class="gmail_msg">
> > // Build the extracted vector elements back into a vector.<br class="gmail_msg">
> > @@ -11848,7 +11898,6 @@ bool DAGCombiner::MergeStoresOfConstants<br class="gmail_msg">
> > for (unsigned i = 0; i < NumStores; ++i) {<br class="gmail_msg">
> > unsigned Idx = IsLE ? (NumStores - 1 - i) : i;<br class="gmail_msg">
> > StoreSDNode *St = cast<StoreSDNode>(StoreNodes[Idx].MemNode);<br class="gmail_msg">
> > - Chains.push_back(St->getChain());<br class="gmail_msg">
> ><br class="gmail_msg">
> > SDValue Val = St->getValue();<br class="gmail_msg">
> > StoreInt <<= ElementSizeBytes * 8;<br class="gmail_msg">
> > @@ -11866,54 +11915,36 @@ bool DAGCombiner::MergeStoresOfConstants<br class="gmail_msg">
> > StoredVal = DAG.getConstant(StoreInt, DL, StoreTy);<br class="gmail_msg">
> > }<br class="gmail_msg">
> ><br class="gmail_msg">
> > - assert(!Chains.empty());<br class="gmail_msg">
> > + SmallVector<SDValue, 8> Chains;<br class="gmail_msg">
> > +<br class="gmail_msg">
> > + // Gather all Chains we're inheriting. As generally all chains are<br class="gmail_msg">
> > + // equal, do minor check to remove obvious redundancies.<br class="gmail_msg">
> > + Chains.push_back(StoreNodes[0].MemNode->getChain());<br class="gmail_msg">
> > + for (unsigned i = 1; i < NumStores; ++i)<br class="gmail_msg">
> > + if (StoreNodes[0].MemNode->getChain() !=<br class="gmail_msg">
> StoreNodes[i].MemNode->getChain())<br class="gmail_msg">
> > + Chains.push_back(StoreNodes[i].MemNode->getChain());<br class="gmail_msg">
> ><br class="gmail_msg">
> > + LSBaseSDNode *FirstInChain = StoreNodes[0].MemNode;<br class="gmail_msg">
> > SDValue NewChain = DAG.getNode(ISD::TokenFactor, DL,<br class="gmail_msg">
> MVT::Other, Chains);<br class="gmail_msg">
> > SDValue NewStore = DAG.getStore(NewChain, DL, StoredVal,<br class="gmail_msg">
> > FirstInChain->getBasePtr(),<br class="gmail_msg">
> > FirstInChain->getPointerInfo(),<br class="gmail_msg">
> > FirstInChain->getAlignment());<br class="gmail_msg">
> ><br class="gmail_msg">
> > - bool UseAA = CombinerAA.getNumOccurrences() > 0 ? CombinerAA<br class="gmail_msg">
> > - :<br class="gmail_msg">
> DAG.getSubtarget().useAA();<br class="gmail_msg">
> > - if (UseAA) {<br class="gmail_msg">
> > - // Replace all merged stores with the new store.<br class="gmail_msg">
> > - for (unsigned i = 0; i < NumStores; ++i)<br class="gmail_msg">
> > - CombineTo(StoreNodes[i].MemNode, NewStore);<br class="gmail_msg">
> > - } else {<br class="gmail_msg">
> > - // Replace the last store with the new store.<br class="gmail_msg">
> > - CombineTo(LatestOp, NewStore);<br class="gmail_msg">
> > - // Erase all other stores.<br class="gmail_msg">
> > - for (unsigned i = 0; i < NumStores; ++i) {<br class="gmail_msg">
> > - if (StoreNodes[i].MemNode == LatestOp)<br class="gmail_msg">
> > - continue;<br class="gmail_msg">
> > - StoreSDNode *St = cast<StoreSDNode>(StoreNodes[i].MemNode);<br class="gmail_msg">
> > - // ReplaceAllUsesWith will replace all uses that existed<br class="gmail_msg">
> when it was<br class="gmail_msg">
> > - // called, but graph optimizations may cause new ones to<br class="gmail_msg">
> appear. For<br class="gmail_msg">
> > - // example, the case in pr14333 looks like<br class="gmail_msg">
> > - //<br class="gmail_msg">
> > - // St's chain -> St -> another store -> X<br class="gmail_msg">
> > - //<br class="gmail_msg">
> > - // And the only difference from St to the other store is<br class="gmail_msg">
> the chain.<br class="gmail_msg">
> > - // When we change it's chain to be St's chain they become<br class="gmail_msg">
> identical,<br class="gmail_msg">
> > - // get CSEed and the net result is that X is now a use of St.<br class="gmail_msg">
> > - // Since we know that St is redundant, just iterate.<br class="gmail_msg">
> > - while (!St->use_empty())<br class="gmail_msg">
> > - DAG.ReplaceAllUsesWith(SDValue(St, 0), St->getChain());<br class="gmail_msg">
> > - deleteAndRecombine(St);<br class="gmail_msg">
> > - }<br class="gmail_msg">
> > - }<br class="gmail_msg">
> > + // Replace all merged stores with the new store.<br class="gmail_msg">
> > + for (unsigned i = 0; i < NumStores; ++i)<br class="gmail_msg">
> > + CombineTo(StoreNodes[i].MemNode, NewStore);<br class="gmail_msg">
> ><br class="gmail_msg">
> > - StoreNodes.erase(StoreNodes.begin() + NumStores, StoreNodes.end());<br class="gmail_msg">
> > + AddToWorklist(NewChain.getNode());<br class="gmail_msg">
> > return true;<br class="gmail_msg">
> > }<br class="gmail_msg">
> ><br class="gmail_msg">
> > -void DAGCombiner::getStoreMergeAndAliasCandidates(<br class="gmail_msg">
> > - StoreSDNode* St, SmallVectorImpl<MemOpLink> &StoreNodes,<br class="gmail_msg">
> > - SmallVectorImpl<LSBaseSDNode*> &AliasLoadNodes) {<br class="gmail_msg">
> > +void DAGCombiner::getStoreMergeCandidates(<br class="gmail_msg">
> > + StoreSDNode *St, SmallVectorImpl<MemOpLink> &StoreNodes) {<br class="gmail_msg">
> > // This holds the base pointer, index, and the offset in bytes<br class="gmail_msg">
> from the base<br class="gmail_msg">
> > // pointer.<br class="gmail_msg">
> > BaseIndexOffset BasePtr =<br class="gmail_msg">
> BaseIndexOffset::match(St->getBasePtr(), DAG);<br class="gmail_msg">
> > + EVT MemVT = St->getMemoryVT();<br class="gmail_msg">
> ><br class="gmail_msg">
> > // We must have a base and an offset.<br class="gmail_msg">
> > if (!BasePtr.Base.getNode())<br class="gmail_msg">
> > @@ -11923,104 +11954,70 @@ void DAGCombiner::getStoreMergeAndAliasC<br class="gmail_msg">
> > if (BasePtr.Base.isUndef())<br class="gmail_msg">
> > return;<br class="gmail_msg">
> ><br class="gmail_msg">
> > - // Walk up the chain and look for nodes with offsets from the same<br class="gmail_msg">
> > - // base pointer. Stop when reaching an instruction with a<br class="gmail_msg">
> different kind<br class="gmail_msg">
> > - // or instruction which has a different base pointer.<br class="gmail_msg">
> > - EVT MemVT = St->getMemoryVT();<br class="gmail_msg">
> > - unsigned Seq = 0;<br class="gmail_msg">
> > - StoreSDNode *Index = St;<br class="gmail_msg">
> > -<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - bool UseAA = CombinerAA.getNumOccurrences() > 0 ? CombinerAA<br class="gmail_msg">
> > - :<br class="gmail_msg">
> DAG.getSubtarget().useAA();<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - if (UseAA) {<br class="gmail_msg">
> > - // Look at other users of the same chain. Stores on the same<br class="gmail_msg">
> chain do not<br class="gmail_msg">
> > - // alias. If combiner-aa is enabled, non-aliasing stores are<br class="gmail_msg">
> canonicalized<br class="gmail_msg">
> > - // to be on the same chain, so don't bother looking at<br class="gmail_msg">
> adjacent chains.<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - SDValue Chain = St->getChain();<br class="gmail_msg">
> > - for (auto I = Chain->use_begin(), E = Chain->use_end(); I !=<br class="gmail_msg">
> E; ++I) {<br class="gmail_msg">
> > - if (StoreSDNode *OtherST = dyn_cast<StoreSDNode>(*I)) {<br class="gmail_msg">
> > - if (I.getOperandNo() != 0)<br class="gmail_msg">
> > - continue;<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - if (OtherST->isVolatile() || OtherST->isIndexed())<br class="gmail_msg">
> > - continue;<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - if (OtherST->getMemoryVT() != MemVT)<br class="gmail_msg">
> > - continue;<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - BaseIndexOffset Ptr =<br class="gmail_msg">
> BaseIndexOffset::match(OtherST->getBasePtr(), DAG);<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - if (Ptr.equalBaseIndex(BasePtr))<br class="gmail_msg">
> > - StoreNodes.push_back(MemOpLink(OtherST, Ptr.Offset,<br class="gmail_msg">
> Seq++));<br class="gmail_msg">
> > - }<br class="gmail_msg">
> > - }<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - return;<br class="gmail_msg">
> > - }<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - while (Index) {<br class="gmail_msg">
> > - // If the chain has more than one use, then we can't reorder<br class="gmail_msg">
> the mem ops.<br class="gmail_msg">
> > - if (Index != St && !SDValue(Index, 0)->hasOneUse())<br class="gmail_msg">
> > - break;<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - // Find the base pointer and offset for this memory node.<br class="gmail_msg">
> > - BaseIndexOffset Ptr =<br class="gmail_msg">
> BaseIndexOffset::match(Index->getBasePtr(), DAG);<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - // Check that the base pointer is the same as the original one.<br class="gmail_msg">
> > - if (!Ptr.equalBaseIndex(BasePtr))<br class="gmail_msg">
> > - break;<br class="gmail_msg">
> > + // We looking for a root node which is an ancestor to all mergable<br class="gmail_msg">
> > + // stores. We search up through a load, to our root and then down<br class="gmail_msg">
> > + // through all children. For instance we will find Store{1,2,3} if<br class="gmail_msg">
> > + // St is Store1, Store2. or Store3 where the root is not a load<br class="gmail_msg">
> > + // which always true for nonvolatile ops. TODO: Expand<br class="gmail_msg">
> > + // the search to find all valid candidates through multiple<br class="gmail_msg">
> layers of loads.<br class="gmail_msg">
> > + //<br class="gmail_msg">
> > + // Root<br class="gmail_msg">
> > + // |-------|-------|<br class="gmail_msg">
> > + // Load Load Store3<br class="gmail_msg">
> > + // | |<br class="gmail_msg">
> > + // Store1 Store2<br class="gmail_msg">
> > + //<br class="gmail_msg">
> > + // FIXME: We should be able to climb and<br class="gmail_msg">
> > + // descend TokenFactors to find candidates as well.<br class="gmail_msg">
> ><br class="gmail_msg">
> > - // The memory operands must not be volatile.<br class="gmail_msg">
> > - if (Index->isVolatile() || Index->isIndexed())<br class="gmail_msg">
> > - break;<br class="gmail_msg">
> > + SDNode *RootNode = (St->getChain()).getNode();<br class="gmail_msg">
> ><br class="gmail_msg">
> > - // No truncation.<br class="gmail_msg">
> > - if (Index->isTruncatingStore())<br class="gmail_msg">
> > - break;<br class="gmail_msg">
> > + // Set of Parents of Candidates<br class="gmail_msg">
> > + std::set<SDNode *> CandidateParents;<br class="gmail_msg">
> ><br class="gmail_msg">
> > - // The stored memory type must be the same.<br class="gmail_msg">
> > - if (Index->getMemoryVT() != MemVT)<br class="gmail_msg">
> > - break;<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - // We do not allow under-aligned stores in order to prevent<br class="gmail_msg">
> > - // overriding stores. NOTE: this is a bad hack. Alignment SHOULD<br class="gmail_msg">
> > - // be irrelevant here; what MATTERS is that we not move memory<br class="gmail_msg">
> > - // operations that potentially overlap past each-other.<br class="gmail_msg">
> > - if (Index->getAlignment() < MemVT.getStoreSize())<br class="gmail_msg">
> > - break;<br class="gmail_msg">
> > + if (LoadSDNode *Ldn = dyn_cast<LoadSDNode>(RootNode)) {<br class="gmail_msg">
> > + RootNode = Ldn->getChain().getNode();<br class="gmail_msg">
> > + for (auto I = RootNode->use_begin(), E = RootNode->use_end();<br class="gmail_msg">
> I != E; ++I)<br class="gmail_msg">
> > + if (I.getOperandNo() == 0 && isa<LoadSDNode>(*I)) // walk<br class="gmail_msg">
> down chain<br class="gmail_msg">
> > + CandidateParents.insert(*I);<br class="gmail_msg">
> > + } else<br class="gmail_msg">
> > + CandidateParents.insert(RootNode);<br class="gmail_msg">
> ><br class="gmail_msg">
> > - // We found a potential memory operand to merge.<br class="gmail_msg">
> > - StoreNodes.push_back(MemOpLink(Index, Ptr.Offset, Seq++));<br class="gmail_msg">
> > + bool IsLoadSrc = isa<LoadSDNode>(St->getValue());<br class="gmail_msg">
> > + bool IsConstantSrc = isa<ConstantSDNode>(St->getValue()) ||<br class="gmail_msg">
> > + isa<ConstantFPSDNode>(St->getValue());<br class="gmail_msg">
> > + bool IsExtractVecSrc =<br class="gmail_msg">
> > + (St->getValue().getOpcode() == ISD::EXTRACT_VECTOR_ELT ||<br class="gmail_msg">
> > + St->getValue().getOpcode() == ISD::EXTRACT_SUBVECTOR);<br class="gmail_msg">
> > + auto CorrectValueKind = [&](StoreSDNode *Other) -> bool {<br class="gmail_msg">
> > + if (IsLoadSrc)<br class="gmail_msg">
> > + return isa<LoadSDNode>(Other->getValue());<br class="gmail_msg">
> > + if (IsConstantSrc)<br class="gmail_msg">
> > + return (isa<ConstantSDNode>(Other->getValue()) ||<br class="gmail_msg">
> > + isa<ConstantFPSDNode>(Other->getValue()));<br class="gmail_msg">
> > + if (IsExtractVecSrc)<br class="gmail_msg">
> > + return (Other->getValue().getOpcode() ==<br class="gmail_msg">
> ISD::EXTRACT_VECTOR_ELT ||<br class="gmail_msg">
> > + Other->getValue().getOpcode() ==<br class="gmail_msg">
> ISD::EXTRACT_SUBVECTOR);<br class="gmail_msg">
> > + return false;<br class="gmail_msg">
> > + };<br class="gmail_msg">
> ><br class="gmail_msg">
> > - // Find the next memory operand in the chain. If the next<br class="gmail_msg">
> operand in the<br class="gmail_msg">
> > - // chain is a store then move up and continue the scan with<br class="gmail_msg">
> the next<br class="gmail_msg">
> > - // memory operand. If the next operand is a load save it and<br class="gmail_msg">
> use alias<br class="gmail_msg">
> > - // information to check if it interferes with anything.<br class="gmail_msg">
> > - SDNode *NextInChain = Index->getChain().getNode();<br class="gmail_msg">
> > - while (1) {<br class="gmail_msg">
> > - if (StoreSDNode *STn = dyn_cast<StoreSDNode>(NextInChain)) {<br class="gmail_msg">
> > - // We found a store node. Use it for the next iteration.<br class="gmail_msg">
> > - Index = STn;<br class="gmail_msg">
> > - break;<br class="gmail_msg">
> > - } else if (LoadSDNode *Ldn =<br class="gmail_msg">
> dyn_cast<LoadSDNode>(NextInChain)) {<br class="gmail_msg">
> > - if (Ldn->isVolatile()) {<br class="gmail_msg">
> > - Index = nullptr;<br class="gmail_msg">
> > - break;<br class="gmail_msg">
> > + // check all parents of mergable children<br class="gmail_msg">
> > + for (auto P = CandidateParents.begin(); P !=<br class="gmail_msg">
> CandidateParents.end(); ++P)<br class="gmail_msg">
> > + for (auto I = (*P)->use_begin(), E = (*P)->use_end(); I != E;<br class="gmail_msg">
> ++I)<br class="gmail_msg">
> > + if (I.getOperandNo() == 0)<br class="gmail_msg">
> > + if (StoreSDNode *OtherST = dyn_cast<StoreSDNode>(*I)) {<br class="gmail_msg">
> > + if (OtherST->isVolatile() || OtherST->isIndexed())<br class="gmail_msg">
> > + continue;<br class="gmail_msg">
> > + // We can merge constant floats to equivalent integers<br class="gmail_msg">
> > + if (OtherST->getMemoryVT() != MemVT)<br class="gmail_msg">
> > + if (!(MemVT.isInteger() &&<br class="gmail_msg">
> MemVT.bitsEq(OtherST->getMemoryVT()) &&<br class="gmail_msg">
> > + isa<ConstantFPSDNode>(OtherST->getValue())))<br class="gmail_msg">
> > + continue;<br class="gmail_msg">
> > + BaseIndexOffset Ptr =<br class="gmail_msg">
> > + BaseIndexOffset::match(OtherST->getBasePtr(), DAG);<br class="gmail_msg">
> > + if (Ptr.equalBaseIndex(BasePtr) &&<br class="gmail_msg">
> CorrectValueKind(OtherST))<br class="gmail_msg">
> > + StoreNodes.push_back(MemOpLink(OtherST, Ptr.Offset));<br class="gmail_msg">
> > }<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - // Save the load node for later. Continue the scan.<br class="gmail_msg">
> > - AliasLoadNodes.push_back(Ldn);<br class="gmail_msg">
> > - NextInChain = Ldn->getChain().getNode();<br class="gmail_msg">
> > - continue;<br class="gmail_msg">
> > - } else {<br class="gmail_msg">
> > - Index = nullptr;<br class="gmail_msg">
> > - break;<br class="gmail_msg">
> > - }<br class="gmail_msg">
> > - }<br class="gmail_msg">
> > - }<br class="gmail_msg">
> > }<br class="gmail_msg">
> ><br class="gmail_msg">
> > // We need to check that merging these stores does not cause a loop<br class="gmail_msg">
> > @@ -12047,8 +12044,7 @@ bool DAGCombiner::checkMergeStoreCandida<br class="gmail_msg">
> > return true;<br class="gmail_msg">
> > }<br class="gmail_msg">
> ><br class="gmail_msg">
> > -bool DAGCombiner::MergeConsecutiveStores(<br class="gmail_msg">
> > - StoreSDNode* St, SmallVectorImpl<MemOpLink> &StoreNodes) {<br class="gmail_msg">
> > +bool DAGCombiner::MergeConsecutiveStores(StoreSDNode *St) {<br class="gmail_msg">
> > if (OptLevel == CodeGenOpt::None)<br class="gmail_msg">
> > return false;<br class="gmail_msg">
> ><br class="gmail_msg">
> > @@ -12082,145 +12078,136 @@ bool DAGCombiner::MergeConsecutiveStores<br class="gmail_msg">
> > if (MemVT.isVector() && IsLoadSrc)<br class="gmail_msg">
> > return false;<br class="gmail_msg">
> ><br class="gmail_msg">
> > - // Only look at ends of store sequences.<br class="gmail_msg">
> > - SDValue Chain = SDValue(St, 0);<br class="gmail_msg">
> > - if (Chain->hasOneUse() && Chain->use_begin()->getOpcode() ==<br class="gmail_msg">
> ISD::STORE)<br class="gmail_msg">
> > - return false;<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - // Save the LoadSDNodes that we find in the chain.<br class="gmail_msg">
> > - // We need to make sure that these nodes do not interfere with<br class="gmail_msg">
> > - // any of the store nodes.<br class="gmail_msg">
> > - SmallVector<LSBaseSDNode*, 8> AliasLoadNodes;<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - getStoreMergeAndAliasCandidates(St, StoreNodes, AliasLoadNodes);<br class="gmail_msg">
> > + SmallVector<MemOpLink, 8> StoreNodes;<br class="gmail_msg">
> > + // Find potential store merge candidates by searching through<br class="gmail_msg">
> chain sub-DAG<br class="gmail_msg">
> > + getStoreMergeCandidates(St, StoreNodes);<br class="gmail_msg">
> ><br class="gmail_msg">
> > // Check if there is anything to merge.<br class="gmail_msg">
> > if (StoreNodes.size() < 2)<br class="gmail_msg">
> > return false;<br class="gmail_msg">
> ><br class="gmail_msg">
> > - // only do dependence check in AA case<br class="gmail_msg">
> > - bool UseAA = CombinerAA.getNumOccurrences() > 0 ? CombinerAA<br class="gmail_msg">
> > - :<br class="gmail_msg">
> DAG.getSubtarget().useAA();<br class="gmail_msg">
> > - if (UseAA && !checkMergeStoreCandidatesForDependencies(StoreNodes))<br class="gmail_msg">
> > + // Check that we can merge these candidates without causing a cycle<br class="gmail_msg">
> > + if (!checkMergeStoreCandidatesForDependencies(StoreNodes))<br class="gmail_msg">
> > return false;<br class="gmail_msg">
> ><br class="gmail_msg">
> > // Sort the memory operands according to their distance from the<br class="gmail_msg">
> > - // base pointer. As a secondary criteria: make sure stores coming<br class="gmail_msg">
> > - // later in the code come first in the list. This is important for<br class="gmail_msg">
> > - // the non-UseAA case, because we're merging stores into the FINAL<br class="gmail_msg">
> > - // store along a chain which potentially contains aliasing stores.<br class="gmail_msg">
> > - // Thus, if there are multiple stores to the same address, the last<br class="gmail_msg">
> > - // one can be considered for merging but not the others.<br class="gmail_msg">
> > + // base pointer.<br class="gmail_msg">
> > std::sort(StoreNodes.begin(), StoreNodes.end(),<br class="gmail_msg">
> > [](MemOpLink LHS, MemOpLink RHS) {<br class="gmail_msg">
> > - return LHS.OffsetFromBase < RHS.OffsetFromBase ||<br class="gmail_msg">
> > - (LHS.OffsetFromBase == RHS.OffsetFromBase &&<br class="gmail_msg">
> > - LHS.SequenceNum < RHS.SequenceNum);<br class="gmail_msg">
> > - });<br class="gmail_msg">
> > + return LHS.OffsetFromBase < RHS.OffsetFromBase;<br class="gmail_msg">
> > + });<br class="gmail_msg">
> ><br class="gmail_msg">
> > // Scan the memory operations on the chain and find the first<br class="gmail_msg">
> non-consecutive<br class="gmail_msg">
> > // store memory address.<br class="gmail_msg">
> > - unsigned LastConsecutiveStore = 0;<br class="gmail_msg">
> > + unsigned NumConsecutiveStores = 0;<br class="gmail_msg">
> > int64_t StartAddress = StoreNodes[0].OffsetFromBase;<br class="gmail_msg">
> > - for (unsigned i = 0, e = StoreNodes.size(); i < e; ++i) {<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - // Check that the addresses are consecutive starting from the<br class="gmail_msg">
> second<br class="gmail_msg">
> > - // element in the list of stores.<br class="gmail_msg">
> > - if (i > 0) {<br class="gmail_msg">
> > - int64_t CurrAddress = StoreNodes[i].OffsetFromBase;<br class="gmail_msg">
> > - if (CurrAddress - StartAddress != (ElementSizeBytes * i))<br class="gmail_msg">
> > - break;<br class="gmail_msg">
> > - }<br class="gmail_msg">
> ><br class="gmail_msg">
> > - // Check if this store interferes with any of the loads that<br class="gmail_msg">
> we found.<br class="gmail_msg">
> > - // If we find a load that alias with this store. Stop the<br class="gmail_msg">
> sequence.<br class="gmail_msg">
> > - if (any_of(AliasLoadNodes, [&](LSBaseSDNode *Ldn) {<br class="gmail_msg">
> > - return isAlias(Ldn, StoreNodes[i].MemNode);<br class="gmail_msg">
> > - }))<br class="gmail_msg">
> > + // Check that the addresses are consecutive starting from the<br class="gmail_msg">
> second<br class="gmail_msg">
> > + // element in the list of stores.<br class="gmail_msg">
> > + for (unsigned i = 1, e = StoreNodes.size(); i < e; ++i) {<br class="gmail_msg">
> > + int64_t CurrAddress = StoreNodes[i].OffsetFromBase;<br class="gmail_msg">
> > + if (CurrAddress - StartAddress != (ElementSizeBytes * i))<br class="gmail_msg">
> > break;<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - // Mark this node as useful.<br class="gmail_msg">
> > - LastConsecutiveStore = i;<br class="gmail_msg">
> > + NumConsecutiveStores = i + 1;<br class="gmail_msg">
> > }<br class="gmail_msg">
> ><br class="gmail_msg">
> > + if (NumConsecutiveStores < 2)<br class="gmail_msg">
> > + return false;<br class="gmail_msg">
> > +<br class="gmail_msg">
> > // The node with the lowest store address.<br class="gmail_msg">
> > - LSBaseSDNode *FirstInChain = StoreNodes[0].MemNode;<br class="gmail_msg">
> > - unsigned FirstStoreAS = FirstInChain->getAddressSpace();<br class="gmail_msg">
> > - unsigned FirstStoreAlign = FirstInChain->getAlignment();<br class="gmail_msg">
> > LLVMContext &Context = *DAG.getContext();<br class="gmail_msg">
> > const DataLayout &DL = DAG.getDataLayout();<br class="gmail_msg">
> ><br class="gmail_msg">
> > // Store the constants into memory as one consecutive store.<br class="gmail_msg">
> > if (IsConstantSrc) {<br class="gmail_msg">
> > - unsigned LastLegalType = 0;<br class="gmail_msg">
> > - unsigned LastLegalVectorType = 0;<br class="gmail_msg">
> > - bool NonZero = false;<br class="gmail_msg">
> > - for (unsigned i=0; i<LastConsecutiveStore+1; ++i) {<br class="gmail_msg">
> > - StoreSDNode *St = cast<StoreSDNode>(StoreNodes[i].MemNode);<br class="gmail_msg">
> > - SDValue StoredVal = St->getValue();<br class="gmail_msg">
> > -<br class="gmail_msg">
> > - if (ConstantSDNode *C = dyn_cast<ConstantSDNode>(StoredVal)) {<br class="gmail_msg">
> > - NonZero |= !C->isNullValue();<br class="gmail_msg">
> > - } else if (ConstantFPSDNode *C =<br class="gmail_msg">
> dyn_cast<ConstantFPSDNode>(StoredVal)) {<br class="gmail_msg">
> > - NonZero |= !C->getConstantFPValue()->isNullValue();<br class="gmail_msg">
> > - } else {<br class="gmail_msg">
> > - // Non-constant.<br class="gmail_msg">
> > - break;<br class="gmail_msg">
> > - }<br class="gmail_msg">
> > + bool RV = false;<br class="gmail_msg">
> > + while (NumConsecutiveStores > 1) {<br class="gmail_msg">
> > + LSBaseSDNode *FirstInChain = StoreNodes[0].MemNode;<br class="gmail_msg">
> > + unsigned FirstStoreAS = FirstInChain->getAddressSpace();<br class="gmail_msg">
> > + unsigned FirstStoreAlign = FirstInChain->getAlignment();<br class="gmail_msg">
> > + unsigned LastLegalType = 0;<br class="gmail_msg">
> > + unsigned LastLegalVectorType = 0;<br class="gmail_msg">
> > + bool NonZero = false;<br class="gmail_msg">
> > + for (unsigned i = 0; i < NumConsecutiveStores; ++i) {<br class="gmail_msg">
> > + StoreSDNode *ST = cast<StoreSDNode>(StoreNodes[i].MemNode);<br class="gmail_msg">
> > + SDValue StoredVal = ST->getValue();<br class="gmail_msg">
> > +<br class="gmail_msg">
> > + if (ConstantSDNode *C =<br class="gmail_msg">
> dyn_cast<ConstantSDNode>(StoredVal)) {<br class="gmail_msg">
> > + NonZero |= !C->isNullValue();<br class="gmail_msg">
> > + } else if (ConstantFPSDNode *C =<br class="gmail_msg">
> > + dyn_cast<ConstantFPSDNode>(StoredVal)) {<br class="gmail_msg">
> > + NonZero |= !C->getConstantFPValue()->isNullValue();<br class="gmail_msg">
> > + } else {<br class="gmail_msg">
> > + // Non-constant.<br class="gmail_msg">
> > + break;<br class="gmail_msg">
> > + }<br class="gmail_msg">
> ><br class="gmail_msg">
> > - // Find a legal type for the constant store.<br class="gmail_msg">
> > - unsigned SizeInBits = (i+1) * ElementSizeBytes * 8;<br class="gmail_msg">
> > - EVT StoreTy = EVT::getIntegerVT(Context, SizeInBits);<br class="gmail_msg">
> > - bool IsFast;<br class="gmail_msg">
> > - if (TLI.isTypeLegal(StoreTy) &&<br class="gmail_msg">
> > - TLI.allowsMemoryAccess(Context, DL, StoreTy, FirstStoreAS,<br class="gmail_msg">
> > - FirstStoreAlign, &IsFast) &&<br class="gmail_msg">
> IsFast) {<br class="gmail_msg">
> > - LastLegalType = i+1;<br class="gmail_msg">
> > - // Or check whether a truncstore is legal.<br class="gmail_msg">
> > - } else if (TLI.getTypeAction(Context, StoreTy) ==<br class="gmail_msg">
> > - TargetLowering::TypePromoteInteger) {<br class="gmail_msg">
> > - EVT LegalizedStoredValueTy =<br class="gmail_msg">
> > - TLI.getTypeToTransformTo(Context,<br class="gmail_msg">
> StoredVal.getValueType());<br class="gmail_msg">
> > - if (TLI.isTruncStoreLegal(LegalizedStoredValueTy, StoreTy) &&<br class="gmail_msg">
> > - TLI.allowsMemoryAccess(Context, DL,<br class="gmail_msg">
> LegalizedStoredValueTy,<br class="gmail_msg">
> > - FirstStoreAS, FirstStoreAlign,<br class="gmail_msg">
> &IsFast) &&<br class="gmail_msg">
> > + // Find a legal type for the constant store.<br class="gmail_msg">
> > + unsigned SizeInBits = (i + 1) * ElementSizeBytes * 8;<br class="gmail_msg">
> > + EVT StoreTy = EVT::getIntegerVT(Context, SizeInBits);<br class="gmail_msg">
> > + bool IsFast = false;<br class="gmail_msg">
> > + if (TLI.isTypeLegal(StoreTy) &&<br class="gmail_msg">
> > + TLI.allowsMemoryAccess(Context, DL, StoreTy,<br class="gmail_msg">
> FirstStoreAS,<br class="gmail_msg">
> > + FirstStoreAlign, &IsFast) &&<br class="gmail_msg">
> > IsFast) {<br class="gmail_msg">
> > LastLegalType = i + 1;<br class="gmail_msg">
> > + // Or check whether a truncstore is legal.<br class="gmail_msg">
> > + } else if (TLI.getTypeAction(Context, StoreTy) ==<br class="gmail_msg">
> > + TargetLowering::TypePromoteInteger) {<br class="gmail_msg">
> > + EVT LegalizedStoredValueTy =<br class="gmail_msg">
> > + TLI.getTypeToTransformTo(Context,<br class="gmail_msg">
> StoredVal.getValueType());<br class="gmail_msg">
> > + if (TLI.isTruncStoreLegal(LegalizedStoredValueTy,<br class="gmail_msg">
> StoreTy) &&<br class="gmail_msg">
> > + TLI.allowsMemoryAccess(Context, DL,<br class="gmail_msg">
> LegalizedStoredValueTy,<br class="gmail_msg">
> > + FirstStoreAS,<br class="gmail_msg">
> FirstStoreAlign, &IsFast) &&<br class="gmail_msg">
> > + IsFast) {<br class="gmail_msg">
> > + LastLegalType = i + 1;<br class="gmail_msg">
> > + }<br class="gmail_msg">
> > }<br class="gmail_msg">
> > - }<br class="gmail_msg">
> ><br class="gmail_msg">
> > - // We only use vectors if the constant is known to be zero<br class="gmail_msg">
> or the target<br class="gmail_msg">
> > - // allows it and the function is not marked with the<br class="gmail_msg">
> noimplicitfloat<br class="gmail_msg">
> > - // attribute.<br class="gmail_msg">
> > - if ((!NonZero || TLI.storeOfVectorConstantIsCheap(MemVT, i+1,<br class="gmail_msg">
> > -<br class="gmail_msg">
> FirstStoreAS)) &&<br class="gmail_msg">
> > - !NoVectors) {<br class="gmail_msg">
> > - </blockquote></div></div>