<div dir="auto">I'll revert it tonight. It should be easy to add that.Can you send me a test case that experiences the problem?<div dir="auto"><br></div><div dir="auto">Nirav</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mar 16, 2017 16:19, "Aditya Nandakumar" <<a href="mailto:aditya_nandakumar@apple.com">aditya_nandakumar@apple.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Hi Nirav<div><br></div><div>This patch is affecting our internal backends (large instruction count regressions). I haven’t completely gone through your patch but form what I see, the problem seems to be that we don’t handle</div><div>descending into TokenFactors (in getStoreMergeCandidates).</div><div>I also see a relevant FIXME which matches what I observe as missing. I have the relevant DAG dump from before and after this change below.</div><div>Before:</div><div><div><br></div><div> t17: i64 = add t6, Constant:i64<4></div><div> t18: ch = store<ST1[%dst.gep2.i105.4](<wbr>align=2)> t15, Constant:i8<0>, t17, undef:i64</div><div> t20: i64 = add t6, Constant:i64<5></div><div> t21: ch = store<ST1[%dst.gep2.i105.5]> t18, Constant:i8<0>, t20, undef:i64</div></div><div><br></div><div>After:</div><div> t17: i64 = add t6, Constant:i64<4><br> t18: ch = store<ST1[%dst.gep2.i105.4](<wbr>align=2)> t15, Constant:i8<0>, t17, undef:i64<br> t20: i64 = add t6, Constant:i64<5><br> t50: ch = store<ST1[%dst.gep2.i105.5]> t0, Constant:i8<0>, t20, undef:i64<br> t51: ch = TokenFactor t18, t50</div><div><br></div><div>Clearly we need to handle TokenFactors for getStoreMergeCandidates.</div><div><br></div><div>Would it be possible to revert this patch and commit it again once you handle TokenFactors? Do you have an ETA for the TokenFactors handling ?</div><div><br></div><div>Thanks</div><div>Aditya</div><div><div><blockquote type="cite"><div>On Mar 13, 2017, at 6:50 PM, Nirav Davé via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>> wrote:</div><br class="m_-8394487157821945359Apple-interchange-newline"><div><div dir="ltr"><div>Yes. It'll be in presently.<br></div><div><br></div><div>Thanks, </div><div><br></div><div>-Nirav</div><div><br></div><div class="gmail_extra"><div class="gmail_quote">On Mon, Mar 13, 2017 at 9:23 PM, Craig Topper <span dir="ltr"><<a href="mailto:craig.topper@gmail.com" target="_blank">craig.topper@gmail.com</a>></span> wrote:</div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Will you also be restoring my fix for i256-add.ll?</div><div class="gmail_extra"><br clear="all"><div><div class="m_-8394487157821945359m_4108594283957026219gmail_signature" data-smartmail="gmail_signature">~Craig</div></div>
<br><div class="gmail_quote">On Mon, Mar 13, 2017 at 5:34 PM, Nirav Dave via llvm-commits <span dir="ltr"><<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Author: niravd<br>
Date: Mon Mar 13 19:34:14 2017<br>
New Revision: 297695<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=297695&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-pr<wbr>oject?rev=297695&view=rev</a><br>
Log:<br>
In visitSTORE, always use FindBetterChain, rather than only when UseAA is enabled.<br>
<br>
Recommiting with compiler time improvements<br>
<br>
Recommitting after fixup of 32-bit aliasing sign offset bug in DAGCombiner.<br>
<br>
* Simplify Consecutive Merge Store Candidate Search<br>
<br>
Now that address aliasing is much less conservative, push through<br>
simplified store merging search and chain alias analysis which only<br>
checks for parallel stores through the chain subgraph. This is cleaner<br>
as the separation of non-interfering loads/stores from the<br>
store-merging logic.<br>
<br>
When merging stores search up the chain through a single load, and<br>
finds all possible stores by looking down from through a load and a<br>
TokenFactor to all stores visited.<br>
<br>
This improves the quality of the output SelectionDAG and the output<br>
Codegen (save perhaps for some ARM cases where we correctly constructs<br>
wider loads, but then promotes them to float operations which appear<br>
but requires more expensive constant generation).<br>
<br>
Some minor peephole optimizations to deal with improved SubDAG shapes (listed below)<br>
<br>
Additional Minor Changes:<br>
<br>
1. Finishes removing unused AliasLoad code<br>
<br>
2. Unifies the chain aggregation in the merged stores across code<br>
paths<br>
<br>
3. Re-add the Store node to the worklist after calling<br>
SimplifyDemandedBits.<br>
<br>
4. Increase GatherAllAliasesMaxDepth from 6 to 18. That number is<br>
arbitrary, but seems sufficient to not cause regressions in<br>
tests.<br>
<br>
5. Remove Chain dependencies of Memory operations on CopyfromReg<br>
nodes as these are captured by data dependence<br>
<br>
6. Forward loads-store values through tokenfactors containing<br>
{CopyToReg,CopyFromReg} Values.<br>
<br>
7. Peephole to convert buildvector of extract_vector_elt to<br>
extract_subvector if possible (see<br>
CodeGen/AArch64/store-merge.l<wbr>l)<br>
<br>
8. Store merging for the ARM target is restricted to 32-bit as<br>
some in some contexts invalid 64-bit operations are being<br>
generated. This can be removed once appropriate checks are<br>
added.<br>
<br>
This finishes the change Matt Arsenault started in r246307 and<br>
jyknight's original patch.<br>
<br>
Many tests required some changes as memory operations are now<br>
reorderable, improving load-store forwarding. One test in<br>
particular is worth noting:<br>
<br>
CodeGen/PowerPC/ppc64-align-lo<wbr>ng-double.ll - Improved load-store<br>
forwarding converts a load-store pair into a parallel store and<br>
a memory-realized bitcast of the same value. However, because we<br>
lose the sharing of the explicit and implicit store values we<br>
must create another local store. A similar transformation<br>
happens before SelectionDAG as well.<br>
<br>
Reviewers: arsenm, hfinkel, tstellarAMD, jyknight, nhaehnle<br>
<br>
Added:<br>
llvm/trunk/test/CodeGen/X86/pr<wbr>32108.ll<br>
Removed:<br>
llvm/trunk/test/CodeGen/X86/co<wbr>mbiner-aa-0.ll<br>
llvm/trunk/test/CodeGen/X86/co<wbr>mbiner-aa-1.ll<br>
llvm/trunk/test/CodeGen/X86/pr<wbr>18023.ll<br>
Modified:<br>
llvm/trunk/include/llvm/Target<wbr>/TargetLowering.h<br>
llvm/trunk/lib/CodeGen/Selecti<wbr>onDAG/DAGCombiner.cpp<br>
llvm/trunk/lib/CodeGen/TargetL<wbr>oweringBase.cpp<br>
llvm/trunk/lib/Target/AArch64/<wbr>AArch64ISelLowering.cpp<br>
llvm/trunk/lib/Target/ARM/ARMI<wbr>SelLowering.h<br>
llvm/trunk/test/CodeGen/AArch6<wbr>4/argument-blocks.ll<br>
llvm/trunk/test/CodeGen/AArch6<wbr>4/arm64-abi.ll<br>
llvm/trunk/test/CodeGen/AArch6<wbr>4/arm64-memset-inline.ll<br>
llvm/trunk/test/CodeGen/AArch6<wbr>4/arm64-variadic-aapcs.ll<br>
llvm/trunk/test/CodeGen/AArch6<wbr>4/merge-store.ll<br>
llvm/trunk/test/CodeGen/AArch6<wbr>4/vector_merge_dep_check.ll<br>
llvm/trunk/test/CodeGen/AMDGPU<wbr>/debugger-insert-nops.ll<br>
llvm/trunk/test/CodeGen/AMDGPU<wbr>/insert_vector_elt.ll<br>
llvm/trunk/test/CodeGen/AMDGPU<wbr>/merge-stores.ll<br>
llvm/trunk/test/CodeGen/AMDGPU<wbr>/private-element-size.ll<br>
llvm/trunk/test/CodeGen/AMDGPU<wbr>/si-triv-disjoint-mem-access.l<wbr>l<br>
llvm/trunk/test/CodeGen/ARM/20<wbr>12-10-04-AAPCS-byval-align8.ll<br>
llvm/trunk/test/CodeGen/ARM/al<wbr>loc-no-stack-realign.ll<br>
llvm/trunk/test/CodeGen/ARM/gp<wbr>r-paired-spill.ll<br>
llvm/trunk/test/CodeGen/ARM/if<wbr>cvt10.ll<br>
llvm/trunk/test/CodeGen/ARM/il<wbr>legal-bitfield-loadstore.ll<br>
llvm/trunk/test/CodeGen/ARM/st<wbr>atic-addr-hoisting.ll<br>
llvm/trunk/test/CodeGen/BPF/un<wbr>def.ll<br>
llvm/trunk/test/CodeGen/MSP430<wbr>/Inst16mm.ll<br>
llvm/trunk/test/CodeGen/Mips/c<wbr>conv/arguments-float.ll<br>
llvm/trunk/test/CodeGen/Mips/c<wbr>conv/arguments-varargs.ll<br>
llvm/trunk/test/CodeGen/Mips/f<wbr>astcc.ll<br>
llvm/trunk/test/CodeGen/Mips/l<wbr>oad-store-left-right.ll<br>
llvm/trunk/test/CodeGen/Mips/m<wbr>icromips-li.ll<br>
llvm/trunk/test/CodeGen/Mips/m<wbr>ips64-f128-call.ll<br>
llvm/trunk/test/CodeGen/Mips/m<wbr>ips64-f128.ll<br>
llvm/trunk/test/CodeGen/Mips/m<wbr>no-ldc1-sdc1.ll<br>
llvm/trunk/test/CodeGen/Mips/m<wbr>sa/f16-llvm-ir.ll<br>
llvm/trunk/test/CodeGen/Mips/m<wbr>sa/i5_ld_st.ll<br>
llvm/trunk/test/CodeGen/Mips/o<wbr>32_cc_byval.ll<br>
llvm/trunk/test/CodeGen/Mips/o<wbr>32_cc_vararg.ll<br>
llvm/trunk/test/CodeGen/PowerP<wbr>C/anon_aggr.ll<br>
llvm/trunk/test/CodeGen/PowerP<wbr>C/complex-return.ll<br>
llvm/trunk/test/CodeGen/PowerP<wbr>C/jaggedstructs.ll<br>
llvm/trunk/test/CodeGen/PowerP<wbr>C/ppc64-align-long-double.ll<br>
llvm/trunk/test/CodeGen/PowerP<wbr>C/structsinmem.ll<br>
llvm/trunk/test/CodeGen/PowerP<wbr>C/structsinregs.ll<br>
llvm/trunk/test/CodeGen/System<wbr>Z/unaligned-01.ll<br>
llvm/trunk/test/CodeGen/Thumb/<wbr>2010-07-15-debugOrdering.ll<br>
llvm/trunk/test/CodeGen/Thumb/<wbr>stack-access.ll<br>
llvm/trunk/test/CodeGen/X86/20<wbr>10-09-17-SideEffectsInChain.ll<br>
llvm/trunk/test/CodeGen/X86/20<wbr>12-11-28-merge-store-alias.ll<br>
llvm/trunk/test/CodeGen/X86/Me<wbr>rgeConsecutiveStores.ll<br>
llvm/trunk/test/CodeGen/X86/av<wbr>x-vbroadcast.ll<br>
llvm/trunk/test/CodeGen/X86/av<wbr>x512-mask-op.ll<br>
llvm/trunk/test/CodeGen/X86/ch<wbr>ain_order.ll<br>
llvm/trunk/test/CodeGen/X86/cl<wbr>ear_upper_vector_element_bits.<wbr>ll<br>
llvm/trunk/test/CodeGen/X86/co<wbr>py-eflags.ll<br>
llvm/trunk/test/CodeGen/X86/da<wbr>g-merge-fast-accesses.ll<br>
llvm/trunk/test/CodeGen/X86/do<wbr>nt-trunc-store-double-to-float<wbr>.ll<br>
llvm/trunk/test/CodeGen/X86/ex<wbr>tractelement-legalization-stor<wbr>e-ordering.ll<br>
llvm/trunk/test/CodeGen/X86/i2<wbr>56-add.ll<br>
llvm/trunk/test/CodeGen/X86/i3<wbr>86-shrink-wrapping.ll<br>
llvm/trunk/test/CodeGen/X86/li<wbr>ve-range-nosubreg.ll<br>
llvm/trunk/test/CodeGen/X86/lo<wbr>nglong-deadload.ll<br>
llvm/trunk/test/CodeGen/X86/me<wbr>rge-consecutive-loads-128.ll<br>
llvm/trunk/test/CodeGen/X86/me<wbr>rge-consecutive-loads-256.ll<br>
llvm/trunk/test/CodeGen/X86/me<wbr>rge-store-partially-alias-load<wbr>s.ll<br>
llvm/trunk/test/CodeGen/X86/sp<wbr>lit-store.ll<br>
llvm/trunk/test/CodeGen/X86/st<wbr>ores-merging.ll<br>
llvm/trunk/test/CodeGen/X86/ve<wbr>ctor-compare-results.ll<br>
llvm/trunk/test/CodeGen/X86/ve<wbr>ctor-shuffle-variable-128.ll<br>
llvm/trunk/test/CodeGen/X86/ve<wbr>ctor-shuffle-variable-256.ll<br>
llvm/trunk/test/CodeGen/X86/ve<wbr>ctorcall.ll<br>
llvm/trunk/test/CodeGen/X86/wi<wbr>n32-eh.ll<br>
llvm/trunk/test/CodeGen/XCore/<wbr>varargs.ll<br>
<br>
Modified: llvm/trunk/include/llvm/Target<wbr>/TargetLowering.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Target/TargetLowering.h?rev=297695&r1=297694&r2=297695&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-pr<wbr>oject/llvm/trunk/include/llvm/<wbr>Target/TargetLowering.h?rev=29<wbr>7695&r1=297694&r2=297695&view=<wbr>diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/include/llvm/Target<wbr>/TargetLowering.h (original)<br>
+++ llvm/trunk/include/llvm/Target<wbr>/TargetLowering.h Mon Mar 13 19:34:14 2017<br>
@@ -363,6 +363,9 @@ public:<br>
return false;<br>
}<br>
<br>
+ /// Returns if it's reasonable to merge stores to MemVT size.<br>
+ virtual bool canMergeStoresTo(EVT MemVT) const { return true; }<br>
+<br>
/// \brief Return true if it is cheap to speculate a call to intrinsic cttz.<br>
virtual bool isCheapToSpeculateCttz() const {<br>
return false;<br>
<br>
Modified: llvm/trunk/lib/CodeGen/Selecti<wbr>onDAG/DAGCombiner.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp?rev=297695&r1=297694&r2=297695&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-pr<wbr>oject/llvm/trunk/lib/CodeGen/S<wbr>electionDAG/DAGCombiner.cpp?re<wbr>v=297695&r1=297694&r2=297695&<wbr>view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/lib/CodeGen/Selecti<wbr>onDAG/DAGCombiner.cpp (original)<br>
+++ llvm/trunk/lib/CodeGen/Selecti<wbr>onDAG/DAGCombiner.cpp Mon Mar 13 19:34:14 2017<br>
@@ -53,10 +53,6 @@ STATISTIC(SlicedLoads, "Number of load s<br>
<br>
namespace {<br>
static cl::opt<bool><br>
- CombinerAA("combiner-alias-ana<wbr>lysis", cl::Hidden,<br>
- cl::desc("Enable DAG combiner alias-analysis heuristics"));<br>
-<br>
- static cl::opt<bool><br>
CombinerGlobalAA("combiner-gl<wbr>obal-alias-analysis", cl::Hidden,<br>
cl::desc("Enable DAG combiner's use of IR alias analysis"));<br>
<br>
@@ -133,6 +129,9 @@ namespace {<br>
/// Add to the worklist making sure its instance is at the back (next to be<br>
/// processed.)<br>
void AddToWorklist(SDNode *N) {<br>
+ assert(N->getOpcode() != ISD::DELETED_NODE &&<br>
+ "Deleted Node added to Worklist");<br>
+<br>
// Skip handle nodes as they can't usefully be combined and confuse the<br>
// zero-use deletion strategy.<br>
if (N->getOpcode() == ISD::HANDLENODE)<br>
@@ -177,6 +176,7 @@ namespace {<br>
void CommitTargetLoweringOpt(const TargetLowering::TargetLowering<wbr>Opt &TLO);<br>
<br>
private:<br>
+ unsigned MaximumLegalStoreInBits;<br>
<br>
/// Check the specified integer node value to see if it can be simplified or<br>
/// if things it uses can be simplified by bit propagation.<br>
@@ -422,15 +422,12 @@ namespace {<br>
/// Holds a pointer to an LSBaseSDNode as well as information on where it<br>
/// is located in a sequence of memory operations connected by a chain.<br>
struct MemOpLink {<br>
- MemOpLink (LSBaseSDNode *N, int64_t Offset, unsigned Seq):<br>
- MemNode(N), OffsetFromBase(Offset), SequenceNum(Seq) { }<br>
+ MemOpLink(LSBaseSDNode *N, int64_t Offset)<br>
+ : MemNode(N), OffsetFromBase(Offset) {}<br>
// Ptr to the mem node.<br>
LSBaseSDNode *MemNode;<br>
// Offset from the base ptr.<br>
int64_t OffsetFromBase;<br>
- // What is the sequence number of this mem node.<br>
- // Lowest mem operand in the DAG starts at zero.<br>
- unsigned SequenceNum;<br>
};<br>
<br>
/// This is a helper function for visitMUL to check the profitability<br>
@@ -441,12 +438,6 @@ namespace {<br>
SDValue &AddNode,<br>
SDValue &ConstNode);<br>
<br>
- /// This is a helper function for MergeStoresOfConstantsOrVecElt<wbr>s. Returns a<br>
- /// constant build_vector of the stored constant values in Stores.<br>
- SDValue getMergedConstantVectorStore(S<wbr>electionDAG &DAG, const SDLoc &SL,<br>
- ArrayRef<MemOpLink> Stores,<br>
- SmallVectorImpl<SDValue> &Chains,<br>
- EVT Ty) const;<br>
<br>
/// This is a helper function for visitAND and visitZERO_EXTEND. Returns<br>
/// true if the (and (load x) c) pattern matches an extload. ExtVT returns<br>
@@ -460,18 +451,15 @@ namespace {<br>
/// This is a helper function for MergeConsecutiveStores. When the source<br>
/// elements of the consecutive stores are all constants or all extracted<br>
/// vector elements, try to merge them into one larger store.<br>
- /// \return number of stores that were merged into a merged store (always<br>
- /// a prefix of \p StoreNode).<br>
- bool MergeStoresOfConstantsOrVecElt<wbr>s(<br>
- SmallVectorImpl<MemOpLink> &StoreNodes, EVT MemVT, unsigned NumStores,<br>
- bool IsConstantSrc, bool UseVector);<br>
+ /// \return True if a merged store was created.<br>
+ bool MergeStoresOfConstantsOrVecElt<wbr>s(SmallVectorImpl<MemOpLink> &StoreNodes,<br>
+ EVT MemVT, unsigned NumStores,<br>
+ bool IsConstantSrc, bool UseVector);<br>
<br>
/// This is a helper function for MergeConsecutiveStores.<br>
/// Stores that may be merged are placed in StoreNodes.<br>
- /// Loads that may alias with those stores are placed in AliasLoadNodes.<br>
- void getStoreMergeAndAliasCandidate<wbr>s(<br>
- StoreSDNode* St, SmallVectorImpl<MemOpLink> &StoreNodes,<br>
- SmallVectorImpl<LSBaseSDNode*> &AliasLoadNodes);<br>
+ void getStoreMergeCandidates(StoreS<wbr>DNode *St,<br>
+ SmallVectorImpl<MemOpLink> &StoreNodes);<br>
<br>
/// Helper function for MergeConsecutiveStores. Checks if<br>
/// Candidate stores have indirect dependency through their<br>
@@ -483,8 +471,7 @@ namespace {<br>
/// This optimization uses wide integers or vectors when possible.<br>
/// \return number of stores that were merged into a merged store (the<br>
/// affected nodes are stored as a prefix in \p StoreNodes).<br>
- bool MergeConsecutiveStores(StoreSD<wbr>Node *N,<br>
- SmallVectorImpl<MemOpLink> &StoreNodes);<br>
+ bool MergeConsecutiveStores(StoreSD<wbr>Node *N);<br>
<br>
/// \brief Try to transform a truncation where C is a constant:<br>
/// (trunc (and X, C)) -> (and (trunc X), (trunc C))<br>
@@ -499,6 +486,13 @@ namespace {<br>
: DAG(D), TLI(D.getTargetLoweringInfo())<wbr>, Level(BeforeLegalizeTypes),<br>
OptLevel(OL), LegalOperations(false), LegalTypes(false), AA(A) {<br>
ForCodeSize = DAG.getMachineFunction().getFu<wbr>nction()->optForSize();<br>
+<br>
+ MaximumLegalStoreInBits = 0;<br>
+ for (MVT VT : MVT::all_valuetypes())<br>
+ if (EVT(VT).isSimple() && VT != MVT::Other &&<br>
+ TLI.isTypeLegal(EVT(VT)) &&<br>
+ VT.getSizeInBits() >= MaximumLegalStoreInBits)<br>
+ MaximumLegalStoreInBits = VT.getSizeInBits();<br>
}<br>
<br>
/// Runs the dag combiner on all nodes in the work list<br>
@@ -1589,7 +1583,7 @@ SDValue DAGCombiner::visitTokenFactor(<wbr>SD<br>
}<br>
<br>
SmallVector<SDNode *, 8> TFs; // List of token factors to visit.<br>
- SmallVector<SDValue, 8> Ops; // Ops for replacing token factor.<br>
+ SmallVector<SDValue, 8> Ops; // Ops for replacing token factor.<br>
SmallPtrSet<SDNode*, 16> SeenOps;<br>
bool Changed = false; // If we should replace this token factor.<br>
<br>
@@ -1633,6 +1627,86 @@ SDValue DAGCombiner::visitTokenFactor(<wbr>SD<br>
}<br>
}<br>
<br>
+ // Remove Nodes that are chained to another node in the list. Do so<br>
+ // by walking up chains breath-first stopping when we've seen<br>
+ // another operand. In general we must climb to the EntryNode, but we can exit<br>
+ // early if we find all remaining work is associated with just one operand as<br>
+ // no further pruning is possible.<br>
+<br>
+ // List of nodes to search through and original Ops from which they originate.<br>
+ SmallVector<std::pair<SDNode *, unsigned>, 8> Worklist;<br>
+ SmallVector<unsigned, 8> OpWorkCount; // Count of work for each Op.<br>
+ SmallPtrSet<SDNode *, 16> SeenChains;<br>
+ bool DidPruneOps = false;<br>
+<br>
+ unsigned NumLeftToConsider = 0;<br>
+ for (const SDValue &Op : Ops) {<br>
+ Worklist.push_back(std::make_p<wbr>air(Op.getNode(), NumLeftToConsider++));<br>
+ OpWorkCount.push_back(1);<br>
+ }<br>
+<br>
+ auto AddToWorklist = [&](unsigned CurIdx, SDNode *Op, unsigned OpNumber) {<br>
+ // If this is an Op, we can remove the op from the list. Remark any<br>
+ // search associated with it as from the current OpNumber.<br>
+ if (SeenOps.count(Op) != 0) {<br>
+ Changed = true;<br>
+ DidPruneOps = true;<br>
+ unsigned OrigOpNumber = 0;<br>
+ while (Ops[OrigOpNumber].getNode() != Op && OrigOpNumber < Ops.size())<br>
+ OrigOpNumber++;<br>
+ assert((OrigOpNumber != Ops.size()) &&<br>
+ "expected to find TokenFactor Operand");<br>
+ // Re-mark worklist from OrigOpNumber to OpNumber<br>
+ for (unsigned i = CurIdx + 1; i < Worklist.size(); ++i) {<br>
+ if (Worklist[i].second == OrigOpNumber) {<br>
+ Worklist[i].second = OpNumber;<br>
+ }<br>
+ }<br>
+ OpWorkCount[OpNumber] += OpWorkCount[OrigOpNumber];<br>
+ OpWorkCount[OrigOpNumber] = 0;<br>
+ NumLeftToConsider--;<br>
+ }<br>
+ // Add if it's a new chain<br>
+ if (SeenChains.insert(Op).second) {<br>
+ OpWorkCount[OpNumber]++;<br>
+ Worklist.push_back(std::make_p<wbr>air(Op, OpNumber));<br>
+ }<br>
+ };<br>
+<br>
+ for (unsigned i = 0; i < Worklist.size() && i < 1024; ++i) {<br>
+ // We need at least be consider at least 2 Ops to prune.<br>
+ if (NumLeftToConsider <= 1)<br>
+ break;<br>
+ auto CurNode = Worklist[i].first;<br>
+ auto CurOpNumber = Worklist[i].second;<br>
+ assert((OpWorkCount[CurOpNumbe<wbr>r] > 0) &&<br>
+ "Node should not appear in worklist");<br>
+ switch (CurNode->getOpcode()) {<br>
+ case ISD::EntryToken:<br>
+ // Hitting EntryToken is the only way for the search to terminate without<br>
+ // hitting<br>
+ // another operand's search. Prevent us from marking this operand<br>
+ // considered.<br>
+ NumLeftToConsider++;<br>
+ break;<br>
+ case ISD::TokenFactor:<br>
+ for (const SDValue &Op : CurNode->op_values())<br>
+ AddToWorklist(i, Op.getNode(), CurOpNumber);<br>
+ break;<br>
+ case ISD::CopyFromReg:<br>
+ case ISD::CopyToReg:<br>
+ AddToWorklist(i, CurNode->getOperand(0).getNode<wbr>(), CurOpNumber);<br>
+ break;<br>
+ default:<br>
+ if (auto *MemNode = dyn_cast<MemSDNode>(CurNode))<br>
+ AddToWorklist(i, MemNode->getChain().getNode(), CurOpNumber);<br>
+ break;<br>
+ }<br>
+ OpWorkCount[CurOpNumber]--;<br>
+ if (OpWorkCount[CurOpNumber] == 0)<br>
+ NumLeftToConsider--;<br>
+ }<br>
+<br>
SDValue Result;<br>
<br>
// If we've changed things around then replace token factor.<br>
@@ -1641,15 +1715,22 @@ SDValue DAGCombiner::visitTokenFactor(<wbr>SD<br>
// The entry token is the only possible outcome.<br>
Result = DAG.getEntryNode();<br>
} else {<br>
- // New and improved token factor.<br>
- Result = DAG.getNode(ISD::TokenFactor, SDLoc(N), MVT::Other, Ops);<br>
+ if (DidPruneOps) {<br>
+ SmallVector<SDValue, 8> PrunedOps;<br>
+ //<br>
+ for (const SDValue &Op : Ops) {<br>
+ if (SeenChains.count(Op.getNode()<wbr>) == 0)<br>
+ PrunedOps.push_back(Op);<br>
+ }<br>
+ Result = DAG.getNode(ISD::TokenFactor, SDLoc(N), MVT::Other, PrunedOps);<br>
+ } else {<br>
+ Result = DAG.getNode(ISD::TokenFactor, SDLoc(N), MVT::Other, Ops);<br>
+ }<br>
}<br>
<br>
- // Add users to worklist if AA is enabled, since it may introduce<br>
- // a lot of new chained token factors while removing memory deps.<br>
- bool UseAA = CombinerAA.getNumOccurrences() > 0 ? CombinerAA<br>
- : DAG.getSubtarget().useAA();<br>
- return CombineTo(N, Result, UseAA /*add to worklist*/);<br>
+ // Add users to worklist, since we may introduce a lot of new<br>
+ // chained token factors while removing memory deps.<br>
+ return CombineTo(N, Result, true /*add to worklist*/);<br>
}<br>
<br>
return Result;<br>
@@ -6792,6 +6873,9 @@ SDValue DAGCombiner::CombineExtLoad(SD<wbr>No<br>
SDValue NewChain = DAG.getNode(ISD::TokenFactor, DL, MVT::Other, Chains);<br>
SDValue NewValue = DAG.getNode(ISD::CONCAT_VECTOR<wbr>S, DL, DstVT, Loads);<br>
<br>
+ // Simplify TF.<br>
+ AddToWorklist(NewChain.getNode<wbr>());<br>
+<br>
CombineTo(N, NewValue);<br>
<br>
// Replace uses of the original load (before extension)<br>
@@ -10947,7 +11031,7 @@ SDValue DAGCombiner::visitLOAD(SDNode *N<br>
dbgs() << "\n");<br>
WorklistRemover DeadNodes(*this);<br>
DAG.ReplaceAllUsesOfValueWith<wbr>(SDValue(N, 1), Chain);<br>
-<br>
+ AddUsersToWorklist(Chain.getNo<wbr>de());<br>
if (N->use_empty())<br>
deleteAndRecombine(N);<br>
<br>
@@ -11000,7 +11084,7 @@ SDValue DAGCombiner::visitLOAD(SDNode *N<br>
StoreSDNode *PrevST = cast<StoreSDNode>(Chain);<br>
if (PrevST->getBasePtr() == Ptr &&<br>
PrevST->getValue().getValueTy<wbr>pe() == N->getValueType(0))<br>
- return CombineTo(N, Chain.getOperand(1), Chain);<br>
+ return CombineTo(N, PrevST->getOperand(1), Chain);<br>
}<br>
}<br>
<br>
@@ -11018,14 +11102,7 @@ SDValue DAGCombiner::visitLOAD(SDNode *N<br>
}<br>
}<br>
<br>
- bool UseAA = CombinerAA.getNumOccurrences() > 0 ? CombinerAA<br>
- : DAG.getSubtarget().useAA();<br>
-#ifndef NDEBUG<br>
- if (CombinerAAOnlyFunc.getNumOccu<wbr>rrences() &&<br>
- CombinerAAOnlyFunc != DAG.getMachineFunction().getNa<wbr>me())<br>
- UseAA = false;<br>
-#endif<br>
- if (UseAA && LD->isUnindexed()) {<br>
+ if (LD->isUnindexed()) {<br>
// Walk up chain skipping non-aliasing memory nodes.<br>
SDValue BetterChain = FindBetterChain(N, Chain);<br>
<br>
@@ -11607,6 +11684,7 @@ bool DAGCombiner::SliceUpLoad(SDNod<wbr>e *N)<br>
SDValue Chain = DAG.getNode(ISD::TokenFactor, SDLoc(LD), MVT::Other,<br>
ArgChains);<br>
DAG.ReplaceAllUsesOfValueWith<wbr>(SDValue(N, 1), Chain);<br>
+ AddToWorklist(Chain.getNode())<wbr>;<br>
return true;<br>
}<br>
<br>
@@ -12000,20 +12078,6 @@ bool DAGCombiner::isMulAddWithConst<wbr>Profi<br>
return false;<br>
}<br>
<br>
-SDValue DAGCombiner::getMergedConstant<wbr>VectorStore(<br>
- SelectionDAG &DAG, const SDLoc &SL, ArrayRef<MemOpLink> Stores,<br>
- SmallVectorImpl<SDValue> &Chains, EVT Ty) const {<br>
- SmallVector<SDValue, 8> BuildVector;<br>
-<br>
- for (unsigned I = 0, E = Ty.getVectorNumElements(); I != E; ++I) {<br>
- StoreSDNode *St = cast<StoreSDNode>(Stores[I].Me<wbr>mNode);<br>
- Chains.push_back(St->getChain(<wbr>));<br>
- BuildVector.push_back(St->getV<wbr>alue());<br>
- }<br>
-<br>
- return DAG.getBuildVector(Ty, SL, BuildVector);<br>
-}<br>
-<br>
bool DAGCombiner::MergeStoresOfCons<wbr>tantsOrVecElts(<br>
SmallVectorImpl<MemOpLink> &StoreNodes, EVT MemVT,<br>
unsigned NumStores, bool IsConstantSrc, bool UseVector) {<br>
@@ -12022,22 +12086,8 @@ bool DAGCombiner::MergeStoresOfCons<wbr>tants<br>
return false;<br>
<br>
int64_t ElementSizeBytes = MemVT.getSizeInBits() / 8;<br>
- LSBaseSDNode *FirstInChain = StoreNodes[0].MemNode;<br>
- unsigned LatestNodeUsed = 0;<br>
-<br>
- for (unsigned i=0; i < NumStores; ++i) {<br>
- // Find a chain for the new wide-store operand. Notice that some<br>
- // of the store nodes that we found may not be selected for inclusion<br>
- // in the wide store. The chain we use needs to be the chain of the<br>
- // latest store node which is *used* and replaced by the wide store.<br>
- if (StoreNodes[i].SequenceNum < StoreNodes[LatestNodeUsed].Seq<wbr>uenceNum)<br>
- LatestNodeUsed = i;<br>
- }<br>
-<br>
- SmallVector<SDValue, 8> Chains;<br>
<br>
// The latest Node in the DAG.<br>
- LSBaseSDNode *LatestOp = StoreNodes[LatestNodeUsed].Mem<wbr>Node;<br>
SDLoc DL(StoreNodes[0].MemNode);<br>
<br>
SDValue StoredVal;<br>
@@ -12053,7 +12103,18 @@ bool DAGCombiner::MergeStoresOfCons<wbr>tants<br>
assert(TLI.isTypeLegal(Ty) && "Illegal vector store");<br>
<br>
if (IsConstantSrc) {<br>
- StoredVal = getMergedConstantVectorStore(D<wbr>AG, DL, StoreNodes, Chains, Ty);<br>
+ SmallVector<SDValue, 8> BuildVector;<br>
+ for (unsigned I = 0, E = Ty.getVectorNumElements(); I != E; ++I) {<br>
+ StoreSDNode *St = cast<StoreSDNode>(StoreNodes[I<wbr>].MemNode);<br>
+ SDValue Val = St->getValue();<br>
+ if (MemVT.getScalarType().isInteg<wbr>er())<br>
+ if (auto *CFP = dyn_cast<ConstantFPSDNode>(St-<wbr>>getValue()))<br>
+ Val = DAG.getConstant(<br>
+ (uint32_t)CFP->getValueAPF().b<wbr>itcastToAPInt().getZExtValue()<wbr>,<br>
+ SDLoc(CFP), MemVT);<br>
+ BuildVector.push_back(Val);<br>
+ }<br>
+ StoredVal = DAG.getBuildVector(Ty, DL, BuildVector);<br>
} else {<br>
SmallVector<SDValue, 8> Ops;<br>
for (unsigned i = 0; i < NumStores; ++i) {<br>
@@ -12063,7 +12124,6 @@ bool DAGCombiner::MergeStoresOfCons<wbr>tants<br>
if (Val.getValueType() != MemVT)<br>
return false;<br>
Ops.push_back(Val);<br>
- Chains.push_back(St->getChain(<wbr>));<br>
}<br>
<br>
// Build the extracted vector elements back into a vector.<br>
@@ -12083,7 +12143,6 @@ bool DAGCombiner::MergeStoresOfCons<wbr>tants<br>
for (unsigned i = 0; i < NumStores; ++i) {<br>
unsigned Idx = IsLE ? (NumStores - 1 - i) : i;<br>
StoreSDNode *St = cast<StoreSDNode>(StoreNodes[I<wbr>dx].MemNode);<br>
- Chains.push_back(St->getChain(<wbr>));<br>
<br>
SDValue Val = St->getValue();<br>
StoreInt <<= ElementSizeBytes * 8;<br>
@@ -12101,54 +12160,36 @@ bool DAGCombiner::MergeStoresOfCons<wbr>tants<br>
StoredVal = DAG.getConstant(StoreInt, DL, StoreTy);<br>
}<br>
<br>
- assert(!Chains.empty());<br>
+ SmallVector<SDValue, 8> Chains;<br>
+<br>
+ // Gather all Chains we're inheriting. As generally all chains are<br>
+ // equal, do minor check to remove obvious redundancies.<br>
+ Chains.push_back(StoreNodes[0]<wbr>.MemNode->getChain());<br>
+ for (unsigned i = 1; i < NumStores; ++i)<br>
+ if (StoreNodes[0].MemNode->getCha<wbr>in() != StoreNodes[i].MemNode->getChai<wbr>n())<br>
+ Chains.push_back(StoreNodes[i]<wbr>.MemNode->getChain());<br>
<br>
+ LSBaseSDNode *FirstInChain = StoreNodes[0].MemNode;<br>
SDValue NewChain = DAG.getNode(ISD::TokenFactor, DL, MVT::Other, Chains);<br>
SDValue NewStore = DAG.getStore(NewChain, DL, StoredVal,<br>
FirstInChain->getBasePtr(),<br>
FirstInChain->getPointerInfo(<wbr>),<br>
FirstInChain->getAlignment())<wbr>;<br>
<br>
- bool UseAA = CombinerAA.getNumOccurrences() > 0 ? CombinerAA<br>
- : DAG.getSubtarget().useAA();<br>
- if (UseAA) {<br>
- // Replace all merged stores with the new store.<br>
- for (unsigned i = 0; i < NumStores; ++i)<br>
- CombineTo(StoreNodes[i].MemNod<wbr>e, NewStore);<br>
- } else {<br>
- // Replace the last store with the new store.<br>
- CombineTo(LatestOp, NewStore);<br>
- // Erase all other stores.<br>
- for (unsigned i = 0; i < NumStores; ++i) {<br>
- if (StoreNodes[i].MemNode == LatestOp)<br>
- continue;<br>
- StoreSDNode *St = cast<StoreSDNode>(StoreNodes[i<wbr>].MemNode);<br>
- // ReplaceAllUsesWith will replace all uses that existed when it was<br>
- // called, but graph optimizations may cause new ones to appear. For<br>
- // example, the case in pr14333 looks like<br>
- //<br>
- // St's chain -> St -> another store -> X<br>
- //<br>
- // And the only difference from St to the other store is the chain.<br>
- // When we change it's chain to be St's chain they become identical,<br>
- // get CSEed and the net result is that X is now a use of St.<br>
- // Since we know that St is redundant, just iterate.<br>
- while (!St->use_empty())<br>
- DAG.ReplaceAllUsesWith(SDValue<wbr>(St, 0), St->getChain());<br>
- deleteAndRecombine(St);<br>
- }<br>
- }<br>
+ // Replace all merged stores with the new store.<br>
+ for (unsigned i = 0; i < NumStores; ++i)<br>
+ CombineTo(StoreNodes[i].MemNod<wbr>e, NewStore);<br>
<br>
- StoreNodes.erase(<a href="http://StoreNodes.be" target="_blank">StoreNodes.be</a><wbr>gin() + NumStores, StoreNodes.end());<br>
+ AddToWorklist(NewChain.getNode<wbr>());<br>
return true;<br>
}<br>
<br>
-void DAGCombiner::getStoreMergeAndA<wbr>liasCandidates(<br>
- StoreSDNode* St, SmallVectorImpl<MemOpLink> &StoreNodes,<br>
- SmallVectorImpl<LSBaseSDNode*> &AliasLoadNodes) {<br>
+void DAGCombiner::getStoreMergeCand<wbr>idates(<br>
+ StoreSDNode *St, SmallVectorImpl<MemOpLink> &StoreNodes) {<br>
// This holds the base pointer, index, and the offset in bytes from the base<br>
// pointer.<br>
BaseIndexOffset BasePtr = BaseIndexOffset::match(St->get<wbr>BasePtr(), DAG);<br>
+ EVT MemVT = St->getMemoryVT();<br>
<br>
// We must have a base and an offset.<br>
if (!BasePtr.Base.getNode())<br>
@@ -12158,104 +12199,70 @@ void DAGCombiner::getStoreMergeAndA<wbr>liasC<br>
if (BasePtr.Base.isUndef())<br>
return;<br>
<br>
- // Walk up the chain and look for nodes with offsets from the same<br>
- // base pointer. Stop when reaching an instruction with a different kind<br>
- // or instruction which has a different base pointer.<br>
- EVT MemVT = St->getMemoryVT();<br>
- unsigned Seq = 0;<br>
- StoreSDNode *Index = St;<br>
-<br>
-<br>
- bool UseAA = CombinerAA.getNumOccurrences() > 0 ? CombinerAA<br>
- : DAG.getSubtarget().useAA();<br>
-<br>
- if (UseAA) {<br>
- // Look at other users of the same chain. Stores on the same chain do not<br>
- // alias. If combiner-aa is enabled, non-aliasing stores are canonicalized<br>
- // to be on the same chain, so don't bother looking at adjacent chains.<br>
-<br>
- SDValue Chain = St->getChain();<br>
- for (auto I = Chain->use_begin(), E = Chain->use_end(); I != E; ++I) {<br>
- if (StoreSDNode *OtherST = dyn_cast<StoreSDNode>(*I)) {<br>
- if (I.getOperandNo() != 0)<br>
- continue;<br>
-<br>
- if (OtherST->isVolatile() || OtherST->isIndexed())<br>
- continue;<br>
-<br>
- if (OtherST->getMemoryVT() != MemVT)<br>
- continue;<br>
-<br>
- BaseIndexOffset Ptr = BaseIndexOffset::match(OtherST<wbr>->getBasePtr(), DAG);<br>
-<br>
- if (Ptr.equalBaseIndex(BasePtr))<br>
- StoreNodes.push_back(MemOpLink<wbr>(OtherST, Ptr.Offset, Seq++));<br>
- }<br>
- }<br>
-<br>
- return;<br>
- }<br>
-<br>
- while (Index) {<br>
- // If the chain has more than one use, then we can't reorder the mem ops.<br>
- if (Index != St && !SDValue(Index, 0)->hasOneUse())<br>
- break;<br>
-<br>
- // Find the base pointer and offset for this memory node.<br>
- BaseIndexOffset Ptr = BaseIndexOffset::match(Index-><wbr>getBasePtr(), DAG);<br>
-<br>
- // Check that the base pointer is the same as the original one.<br>
- if (!Ptr.equalBaseIndex(BasePtr))<br>
- break;<br>
+ // We looking for a root node which is an ancestor to all mergable<br>
+ // stores. We search up through a load, to our root and then down<br>
+ // through all children. For instance we will find Store{1,2,3} if<br>
+ // St is Store1, Store2. or Store3 where the root is not a load<br>
+ // which always true for nonvolatile ops. TODO: Expand<br>
+ // the search to find all valid candidates through multiple layers of loads.<br>
+ //<br>
+ // Root<br>
+ // |-------|-------|<br>
+ // Load Load Store3<br>
+ // | |<br>
+ // Store1 Store2<br>
+ //<br>
+ // FIXME: We should be able to climb and<br>
+ // descend TokenFactors to find candidates as well.<br>
<br>
- // The memory operands must not be volatile.<br>
- if (Index->isVolatile() || Index->isIndexed())<br>
- break;<br>
+ SDNode *RootNode = (St->getChain()).getNode();<br>
<br>
- // No truncation.<br>
- if (Index->isTruncatingStore())<br>
- break;<br>
+ // Set of Parents of Candidates<br>
+ std::set<SDNode *> CandidateParents;<br>
<br>
- // The stored memory type must be the same.<br>
- if (Index->getMemoryVT() != MemVT)<br>
- break;<br>
-<br>
- // We do not allow under-aligned stores in order to prevent<br>
- // overriding stores. NOTE: this is a bad hack. Alignment SHOULD<br>
- // be irrelevant here; what MATTERS is that we not move memory<br>
- // operations that potentially overlap past each-other.<br>
- if (Index->getAlignment() < MemVT.getStoreSize())<br>
- break;<br>
+ if (LoadSDNode *Ldn = dyn_cast<LoadSDNode>(RootNode)<wbr>) {<br>
+ RootNode = Ldn->getChain().getNode();<br>
+ for (auto I = RootNode->use_begin(), E = RootNode->use_end(); I != E; ++I)<br>
+ if (I.getOperandNo() == 0 && isa<LoadSDNode>(*I)) // walk down chain<br>
+ CandidateParents.insert(*I);<br>
+ } else<br>
+ CandidateParents.insert(RootNo<wbr>de);<br>
<br>
- // We found a potential memory operand to merge.<br>
- StoreNodes.push_back(MemOpLink<wbr>(Index, Ptr.Offset, Seq++));<br>
+ bool IsLoadSrc = isa<LoadSDNode>(St->getValue()<wbr>);<br>
+ bool IsConstantSrc = isa<ConstantSDNode>(St->getVal<wbr>ue()) ||<br>
+ isa<ConstantFPSDNode>(St->get<wbr>Value());<br>
+ bool IsExtractVecSrc =<br>
+ (St->getValue().getOpcode() == ISD::EXTRACT_VECTOR_ELT ||<br>
+ St->getValue().getOpcode() == ISD::EXTRACT_SUBVECTOR);<br>
+ auto CorrectValueKind = [&](StoreSDNode *Other) -> bool {<br>
+ if (IsLoadSrc)<br>
+ return isa<LoadSDNode>(Other->getValu<wbr>e());<br>
+ if (IsConstantSrc)<br>
+ return (isa<ConstantSDNode>(Other->ge<wbr>tValue()) ||<br>
+ isa<ConstantFPSDNode>(Other->g<wbr>etValue()));<br>
+ if (IsExtractVecSrc)<br>
+ return (Other->getValue().getOpcode() == ISD::EXTRACT_VECTOR_ELT ||<br>
+ Other->getValue().getOpcode() == ISD::EXTRACT_SUBVECTOR);<br>
+ return false;<br>
+ };<br>
<br>
- // Find the next memory operand in the chain. If the next operand in the<br>
- // chain is a store then move up and continue the scan with the next<br>
- // memory operand. If the next operand is a load save it and use alias<br>
- // information to check if it interferes with anything.<br>
- SDNode *NextInChain = Index->getChain().getNode();<br>
- while (1) {<br>
- if (StoreSDNode *STn = dyn_cast<StoreSDNode>(NextInCh<wbr>ain)) {<br>
- // We found a store node. Use it for the next iteration.<br>
- Index = STn;<br>
- break;<br>
- } else if (LoadSDNode *Ldn = dyn_cast<LoadSDNode>(NextInCha<wbr>in)) {<br>
- if (Ldn->isVolatile()) {<br>
- Index = nullptr;<br>
- break;<br>
+ // check all parents of mergable children<br>
+ for (auto P = CandidateParents.begin(); P != CandidateParents.end(); ++P)<br>
+ for (auto I = (*P)->use_begin(), E = (*P)->use_end(); I != E; ++I)<br>
+ if (I.getOperandNo() == 0)<br>
+ if (StoreSDNode *OtherST = dyn_cast<StoreSDNode>(*I)) {<br>
+ if (OtherST->isVolatile() || OtherST->isIndexed())<br>
+ continue;<br>
+ // We can merge constant floats to equivalent integers<br>
+ if (OtherST->getMemoryVT() != MemVT)<br>
+ if (!(MemVT.isInteger() && MemVT.bitsEq(OtherST->getMemor<wbr>yVT()) &&<br>
+ isa<ConstantFPSDNode>(OtherST-<wbr>>getValue())))<br>
+ continue;<br>
+ BaseIndexOffset Ptr =<br>
+ BaseIndexOffset::match(OtherST<wbr>->getBasePtr(), DAG);<br>
+ if (Ptr.equalBaseIndex(BasePtr) && CorrectValueKind(OtherST))<br>
+ StoreNodes.push_back(MemOpLink<wbr>(OtherST, Ptr.Offset));<br>
}<br>
-<br>
- // Save the load node for later. Continue the scan.<br>
- AliasLoadNodes.push_back(Ldn);<br>
- NextInChain = Ldn->getChain().getNode();<br>
- continue;<br>
- } else {<br>
- Index = nullptr;<br>
- break;<br>
- }<br>
- }<br>
- }<br>
}<br>
<br>
// We need to check that merging these stores does not cause a loop<br>
@@ -12282,13 +12289,16 @@ bool DAGCombiner::checkMergeStoreCa<wbr>ndida<br>
return true;<br>
}<br>
<br>
-bool DAGCombiner::MergeConsecutiveS<wbr>tores(<br>
- StoreSDNode* St, SmallVectorImpl<MemOpLink> &StoreNodes) {<br>
+bool DAGCombiner::MergeConsecutiveS<wbr>tores(StoreSDNode *St) {<br>
if (OptLevel == CodeGenOpt::None)<br>
return false;<br>
<br>
EVT MemVT = St->getMemoryVT();<br>
int64_t ElementSizeBytes = MemVT.getSizeInBits() / 8;<br>
+<br>
+ if (MemVT.getSizeInBits() * 2 > MaximumLegalStoreInBits)<br>
+ return false;<br>
+<br>
bool NoVectors = DAG.getMachineFunction().getFu<wbr>nction()->hasFnAttribute(<br>
Attribute::NoImplicitFloat);<br>
<br>
@@ -12317,145 +12327,136 @@ bool DAGCombiner::MergeConsecutiveS<wbr>tores<br>
if (MemVT.isVector() && IsLoadSrc)<br>
return false;<br>
<br>
- // Only look at ends of store sequences.<br>
- SDValue Chain = SDValue(St, 0);<br>
- if (Chain->hasOneUse() && Chain->use_begin()->getOpcode(<wbr>) == ISD::STORE)<br>
- return false;<br>
-<br>
- // Save the LoadSDNodes that we find in the chain.<br>
- // We need to make sure that these nodes do not interfere with<br>
- // any of the store nodes.<br>
- SmallVector<LSBaseSDNode*, 8> AliasLoadNodes;<br>
-<br>
- getStoreMergeAndAliasCandidate<wbr>s(St, StoreNodes, AliasLoadNodes);<br>
+ SmallVector<MemOpLink, 8> StoreNodes;<br>
+ // Find potential store merge candidates by searching through chain sub-DAG<br>
+ getStoreMergeCandidates(St, StoreNodes);<br>
<br>
// Check if there is anything to merge.<br>
if (StoreNodes.size() < 2)<br>
return false;<br>
<br>
- // only do dependence check in AA case<br>
- bool UseAA = CombinerAA.getNumOccurrences() > 0 ? CombinerAA<br>
- : DAG.getSubtarget().useAA();<br>
- if (UseAA && !checkMergeStoreCandidatesForD<wbr>ependencies(StoreNodes))<br>
+ // Check that we can merge these candidates without causing a cycle<br>
+ if (!checkMergeStoreCandidatesFor<wbr>Dependencies(StoreNodes))<br>
return false;<br>
<br>
// Sort the memory operands according to their distance from the<br>
- // base pointer. As a secondary criteria: make sure stores coming<br>
- // later in the code come first in the list. This is important for<br>
- // the non-UseAA case, because we're merging stores into the FINAL<br>
- // store along a chain which potentially contains aliasing stores.<br>
- // Thus, if there are multiple stores to the same address, the last<br>
- // one can be considered for merging but not the others.<br>
+ // base pointer.<br>
std::sort(StoreNodes.begin(), StoreNodes.end(),<br>
[](MemOpLink LHS, MemOpLink RHS) {<br>
- return LHS.OffsetFromBase < RHS.OffsetFromBase ||<br>
- (LHS.OffsetFromBase == RHS.OffsetFromBase &&<br>
- LHS.SequenceNum < RHS.SequenceNum);<br>
- });<br>
+ return LHS.OffsetFromBase < RHS.OffsetFromBase;<br>
+ });<br>
<br>
// Scan the memory operations on the chain and find the first non-consecutive<br>
// store memory address.<br>
- unsigned LastConsecutiveStore = 0;<br>
+ unsigned NumConsecutiveStores = 0;<br>
int64_t StartAddress = StoreNodes[0].OffsetFromBase;<br>
- for (unsigned i = 0, e = StoreNodes.size(); i < e; ++i) {<br>
-<br>
- // Check that the addresses are consecutive starting from the second<br>
- // element in the list of stores.<br>
- if (i > 0) {<br>
- int64_t CurrAddress = StoreNodes[i].OffsetFromBase;<br>
- if (CurrAddress - StartAddress != (ElementSizeBytes * i))<br>
- break;<br>
- }<br>
<br>
- // Check if this store interferes with any of the loads that we found.<br>
- // If we find a load that alias with this store. Stop the sequence.<br>
- if (any_of(AliasLoadNodes, [&](LSBaseSDNode *Ldn) {<br>
- return isAlias(Ldn, StoreNodes[i].MemNode);<br>
- }))<br>
+ // Check that the addresses are consecutive starting from the second<br>
+ // element in the list of stores.<br>
+ for (unsigned i = 1, e = StoreNodes.size(); i < e; ++i) {<br>
+ int64_t CurrAddress = StoreNodes[i].OffsetFromBase;<br>
+ if (CurrAddress - StartAddress != (ElementSizeBytes * i))<br>
break;<br>
-<br>
- // Mark this node as useful.<br>
- LastConsecutiveStore = i;<br>
+ NumConsecutiveStores = i + 1;<br>
}<br>
<br>
+ if (NumConsecutiveStores < 2)<br>
+ return false;<br>
+<br>
// The node with the lowest store address.<br>
- LSBaseSDNode *FirstInChain = StoreNodes[0].MemNode;<br>
- unsigned FirstStoreAS = FirstInChain->getAddressSpace(<wbr>);<br>
- unsigned FirstStoreAlign = FirstInChain->getAlignment();<br>
LLVMContext &Context = *DAG.getContext();<br>
const DataLayout &DL = DAG.getDataLayout();<br>
<br>
// Store the constants into memory as one consecutive store.<br>
if (IsConstantSrc) {<br>
- unsigned LastLegalType = 0;<br>
- unsigned LastLegalVectorType = 0;<br>
- bool NonZero = false;<br>
- for (unsigned i=0; i<LastConsecutiveStore+1; ++i) {<br>
- StoreSDNode *St = cast<StoreSDNode>(StoreNodes[i<wbr>].MemNode);<br>
- SDValue StoredVal = St->getValue();<br>
-<br>
- if (ConstantSDNode *C = dyn_cast<ConstantSDNode>(Store<wbr>dVal)) {<br>
- NonZero |= !C->isNullValue();<br>
- } else if (ConstantFPSDNode *C = dyn_cast<ConstantFPSDNode>(Sto<wbr>redVal)) {<br>
- NonZero |= !C->getConstantFPValue()->isNu<wbr>llValue();<br>
- } else {<br>
- // Non-constant.<br>
- break;<br>
- }<br>
+ bool RV = false;<br>
+ while (NumConsecutiveStores > 1) {<br>
+ LSBaseSDNode *FirstInChain = StoreNodes[0].MemNode;<br>
+ unsigned FirstStoreAS = FirstInChain->getAddressSpace(<wbr>);<br>
+ unsigned FirstStoreAlign = FirstInChain->getAlignment();<br>
+ unsigned LastLegalType = 0;<br>
+ unsigned LastLegalVectorType = 0;<br>
+ bool NonZero = false;<br>
+ for (unsigned i = 0; i < NumConsecutiveStores; ++i) {<br>
+ StoreSDNode *ST = cast<StoreSDNode>(StoreNodes[i<wbr>].MemNode);<br>
+ SDValue StoredVal = ST->getValue();<br>
+<br>
+ if (ConstantSDNode *C = dyn_cast<ConstantSDNode>(Store<wbr>dVal)) {<br>
+ NonZero |= !C->isNullValue();<br>
+ } else if (ConstantFPSDNode *C =<br>
+ dyn_cast<ConstantFPSDNode>(St<wbr>oredVal)) {<br>
+ NonZero |= !C->getConstantFPValue()->isNu<wbr>llValue();<br>
+ } else {<br>
+ // Non-constant.<br>
+ break;<br>
+ }<br>
<br>
- // Find a legal type for the constant store.<br>
- unsigned SizeInBits = (i+1) * ElementSizeBytes * 8;<br>
- EVT StoreTy = EVT::getIntegerVT(Context, SizeInBits);<br>
- bool IsFast;<br>
- if (TLI.isTypeLegal(StoreTy) &&<br>
- TLI.allowsMemoryAccess(Context<wbr>, DL, StoreTy, FirstStoreAS,<br>
- FirstStoreAlign, &IsFast) && IsFast) {<br>
- LastLegalType = i+1;<br>
- // Or check whether a truncstore is legal.<br>
- } else if (TLI.getTypeAction(Context, StoreTy) ==<br>
- TargetLowering::TypePromoteIn<wbr>teger) {<br>
- EVT LegalizedStoredValueTy =<br>
- TLI.getTypeToTransformTo(Conte<wbr>xt, StoredVal.getValueType());<br>
- if (TLI.isTruncStoreLegal(Legaliz<wbr>edStoredValueTy, StoreTy) &&<br>
- TLI.allowsMemoryAccess(Context<wbr>, DL, LegalizedStoredValueTy,<br>
- FirstStoreAS, FirstStoreAlign, &IsFast) &&<br>
+ // Find a legal type for the constant store.<br>
+ unsigned SizeInBits = (i + 1) * ElementSizeBytes * 8;<br>
+ EVT StoreTy = EVT::getIntegerVT(Context, SizeInBits);<br>
+ bool IsFast = false;<br>
+ if (TLI.isTypeLegal(StoreTy) &&<br>
+ TLI.allowsMemoryAccess(Context<wbr>, DL, StoreTy, FirstStoreAS,<br>
+ FirstStoreAlign, &IsFast) &&<br>
IsFast) {<br>
LastLegalType = i + 1;<br>
+ // Or check whether a truncstore is legal.<br>
+ } else if (TLI.getTypeAction(Context, StoreTy) ==<br>
+ TargetLowering::TypePromoteIn<wbr>teger) {<br>
+ EVT LegalizedStoredValueTy =<br>
+ TLI.getTypeToTransformTo(Conte<wbr>xt, StoredVal.getValueType());<br>
+ if (TLI.isTruncStoreLegal(Legaliz<wbr>edStoredValueTy, StoreTy) &&<br>
+ TLI.allowsMemoryAccess(Context<wbr>, DL, LegalizedStoredValueTy,<br>
+ FirstStoreAS, FirstStoreAlign, &IsFast) &&<br>
+ IsFast) {<br>
+ LastLegalType = i + 1;<br>
+ }<br>
}<br>
- }<br>
<br>
- // We only use vectors if the constant is known to be zero or the target<br>
- // allows it and the function is not marked with the noimplicitfloat<br>
- // attribute.<br>
- if ((!NonZero || TLI.storeOfVectorConstantIsChe<wbr>ap(MemVT, i+1,<br>
- FirstStoreAS)) &&<br>
- !NoVectors) {<br>
- // Find a legal type for the vector store.<br>
- EVT Ty = EVT::getVectorVT(Context, MemVT, i+1);<br>
- if (TLI.isTypeLegal(Ty) &&<br>
- TLI.allowsMemoryAccess(Context<wbr>, DL, Ty, FirstStoreAS,<br>
- FirstStoreAlign, &IsFast) && IsFast)<br>
- LastLegalVectorType = i + 1;<br>
+ // We only use vectors if the constant is known to be zero or the target<br>
+ // allows it and the function is not marked with the noimplicitfloat<br>
+ // attribute.<br>
+ if ((!NonZero ||<br>
+ TLI.storeOfVectorConstantIsCh<wbr>eap(MemVT, i + 1, FirstStoreAS)) &&<br>
+ !NoVectors) {<br>
+ // Find a legal type for the vector store.<br>
+ EVT Ty = EVT::getVectorVT(Context, MemVT, i + 1);<br>
+ if (TLI.isTypeLegal(Ty) && TLI.canMergeStoresTo(Ty) &&<br>
+ TLI.allowsMemoryAccess(Context<wbr>, DL, Ty, FirstStoreAS,<br>
+ FirstStoreAlign, &IsFast) &&<br>
+ IsFast)<br>
+ LastLegalVectorType = i + 1;<br>
+ }<br>
}<br>
- }<br>
<br>
- // Check if we found a legal integer type to store.<br>
- if (LastLegalType == 0 && LastLegalVectorType == 0)<br>
- return false;<br>
+ // Check if we found a legal integer type that creates a meaningful merge.<br>
+ if (LastLegalType < 2 && LastLegalVectorType < 2)<br>
+ break;<br>
<br>
- bool UseVector = (LastLegalVectorType > LastLegalType) && !NoVectors;<br>
- unsigned NumElem = UseVector ? LastLegalVectorType : LastLegalType;<br>
+ bool UseVector = (LastLegalVectorType > LastLegalType) && !NoVectors;<br>
+ unsigned NumElem = (UseVector) ? LastLegalVectorType : LastLegalType;<br>
<br>
- return MergeStoresOfConstantsOrVecElt<wbr>s(StoreNodes, MemVT, NumElem,<br>
- true, UseVector);<br>
+ bool Merged = MergeStoresOfConstantsOrVecElt<wbr>s(StoreNodes, MemVT, NumElem,<br>
+ true, UseVector);<br>
+ if (!Merged)<br>
+ break;<br>
+ // Remove merged stores for next iteration.<br>
+ StoreNodes.erase(<a href="http://StoreNodes.be" target="_blank">StoreNodes.be</a><wbr>gin(), StoreNodes.begin() + NumElem);<br>
+ RV = true;<br>
+ NumConsecutiveStores -= NumElem;<br>
+ }<br>
+ return RV;<br>
}<br>
<br>
// When extracting multiple vector elements, try to store them<br>
// in one vector store rather than a sequence of scalar stores.<br>
if (IsExtractVecSrc) {<br>
+ LSBaseSDNode *FirstInChain = StoreNodes[0].MemNode;<br>
+ unsigned FirstStoreAS = FirstInChain->getAddressSpace(<wbr>);<br>
+ unsigned FirstStoreAlign = FirstInChain->getAlignment();<br>
unsigned NumStoresToMerge = 0;<br>
bool IsVec = MemVT.isVector();<br>
- for (unsigned i = 0; i < LastConsecutiveStore + 1; ++i) {<br>
+ for (unsigned i = 0; i < NumConsecutiveStores; ++i) {<br>
StoreSDNode *St = cast<StoreSDNode>(StoreNodes[i<wbr>].MemNode);<br>
unsigned StoreValOpcode = St->getValue().getOpcode();<br>
// This restriction could be loosened.<br>
@@ -12495,7 +12496,7 @@ bool DAGCombiner::MergeConsecutiveS<wbr>tores<br>
// Find acceptable loads. Loads need to have the same chain (token factor),<br>
// must not be zext, volatile, indexed, and they must be consecutive.<br>
BaseIndexOffset LdBasePtr;<br>
- for (unsigned i=0; i<LastConsecutiveStore+1; ++i) {<br>
+ for (unsigned i = 0; i < NumConsecutiveStores; ++i) {<br>
StoreSDNode *St = cast<StoreSDNode>(StoreNodes[i<wbr>].MemNode);<br>
LoadSDNode *Ld = dyn_cast<LoadSDNode>(St->getVa<wbr>lue());<br>
if (!Ld) break;<br>
@@ -12528,7 +12529,7 @@ bool DAGCombiner::MergeConsecutiveS<wbr>tores<br>
}<br>
<br>
// We found a potential memory operand to merge.<br>
- LoadNodes.push_back(MemOpLink(<wbr>Ld, LdPtr.Offset, 0));<br>
+ LoadNodes.push_back(MemOpLink(<wbr>Ld, LdPtr.Offset));<br>
}<br>
<br>
if (LoadNodes.size() < 2)<br>
@@ -12540,7 +12541,9 @@ bool DAGCombiner::MergeConsecutiveS<wbr>tores<br>
if (LoadNodes.size() == 2 && TLI.hasPairedLoad(MemVT, RequiredAlignment) &&<br>
St->getAlignment() >= RequiredAlignment)<br>
return false;<br>
-<br>
+ LSBaseSDNode *FirstInChain = StoreNodes[0].MemNode;<br>
+ unsigned FirstStoreAS = FirstInChain->getAddressSpace(<wbr>);<br>
+ unsigned FirstStoreAlign = FirstInChain->getAlignment();<br>
LoadSDNode *FirstLoad = cast<LoadSDNode>(LoadNodes[0].<wbr>MemNode);<br>
unsigned FirstLoadAS = FirstLoad->getAddressSpace();<br>
unsigned FirstLoadAlign = FirstLoad->getAlignment();<br>
@@ -12609,30 +12612,19 @@ bool DAGCombiner::MergeConsecutiveS<wbr>tores<br>
<br>
// We add +1 here because the LastXXX variables refer to location while<br>
// the NumElem refers to array/index size.<br>
- unsigned NumElem = std::min(LastConsecutiveStore, LastConsecutiveLoad) + 1;<br>
+ unsigned NumElem = std::min(NumConsecutiveStores, LastConsecutiveLoad + 1);<br>
NumElem = std::min(LastLegalType, NumElem);<br>
<br>
if (NumElem < 2)<br>
return false;<br>
<br>
- // Collect the chains from all merged stores.<br>
+ // Collect the chains from all merged stores. Because the common case<br>
+ // all chains are the same, check if we match the first Chain.<br>
SmallVector<SDValue, 8> MergeStoreChains;<br>
MergeStoreChains.push_back(St<wbr>oreNodes[0].MemNode->getChain(<wbr>));<br>
-<br>
- // The latest Node in the DAG.<br>
- unsigned LatestNodeUsed = 0;<br>
- for (unsigned i=1; i<NumElem; ++i) {<br>
- // Find a chain for the new wide-store operand. Notice that some<br>
- // of the store nodes that we found may not be selected for inclusion<br>
- // in the wide store. The chain we use needs to be the chain of the<br>
- // latest store node which is *used* and replaced by the wide store.<br>
- if (StoreNodes[i].SequenceNum < StoreNodes[LatestNodeUsed].Seq<wbr>uenceNum)<br>
- LatestNodeUsed = i;<br>
-<br>
- MergeStoreChains.push_back(Sto<wbr>reNodes[i].MemNode->getChain()<wbr>);<br>
- }<br>
-<br>
- LSBaseSDNode *LatestOp = StoreNodes[LatestNodeUsed].Mem<wbr>Node;<br>
+ for (unsigned i = 1; i < NumElem; ++i)<br>
+ if (StoreNodes[0].MemNode->getCha<wbr>in() != StoreNodes[i].MemNode->getChai<wbr>n())<br>
+ MergeStoreChains.push_back(Sto<wbr>reNodes[i].MemNode->getChain()<wbr>);<br>
<br>
// Find if it is better to use vectors or integers to load and store<br>
// to memory.<br>
@@ -12656,6 +12648,8 @@ bool DAGCombiner::MergeConsecutiveS<wbr>tores<br>
SDValue NewStoreChain =<br>
DAG.getNode(ISD::TokenFactor, StoreDL, MVT::Other, MergeStoreChains);<br>
<br>
+ AddToWorklist(NewStoreChain.ge<wbr>tNode());<br>
+<br>
SDValue NewStore =<br>
DAG.getStore(NewStoreChain, StoreDL, NewLoad, FirstInChain->getBasePtr(),<br>
FirstInChain->getPointerInfo()<wbr>, FirstStoreAlign);<br>
@@ -12667,25 +12661,9 @@ bool DAGCombiner::MergeConsecutiveS<wbr>tores<br>
SDValue(NewLoad.getNode(), 1));<br>
}<br>
<br>
- if (UseAA) {<br>
- // Replace the all stores with the new store.<br>
- for (unsigned i = 0; i < NumElem; ++i)<br>
- CombineTo(StoreNodes[i].MemNod<wbr>e, NewStore);<br>
- } else {<br>
- // Replace the last store with the new store.<br>
- CombineTo(LatestOp, NewStore);<br>
- // Erase all other stores.<br>
- for (unsigned i = 0; i < NumElem; ++i) {<br>
- // Remove all Store nodes.<br>
- if (StoreNodes[i].MemNode == LatestOp)<br>
- continue;<br>
- StoreSDNode *St = cast<StoreSDNode>(StoreNodes[i<wbr>].MemNode);<br>
- DAG.ReplaceAllUsesOfValueWith(<wbr>SDValue(St, 0), St->getChain());<br>
- deleteAndRecombine(St);<br>
- }<br>
- }<br>
-<br>
- StoreNodes.erase(<a href="http://StoreNodes.be" target="_blank">StoreNodes.be</a><wbr>gin() + NumElem, StoreNodes.end());<br>
+ // Replace the all stores with the new store.<br>
+ for (unsigned i = 0; i < NumElem; ++i)<br>
+ CombineTo(StoreNodes[i].MemNod<wbr>e, NewStore);<br>
return true;<br>
}<br>
<br>
@@ -12842,19 +12820,7 @@ SDValue DAGCombiner::visitSTORE(SDNode *<br>
if (SDValue NewST = TransformFPLoadStorePair(N))<br>
return NewST;<br>
<br>
- bool UseAA = CombinerAA.getNumOccurrences() > 0 ? CombinerAA<br>
- : DAG.getSubtarget().useAA();<br>
-#ifndef NDEBUG<br>
- if (CombinerAAOnlyFunc.getNumOccu<wbr>rrences() &&<br>
- CombinerAAOnlyFunc != DAG.getMachineFunction().getNa<wbr>me())<br>
- UseAA = false;<br>
-#endif<br>
- if (UseAA && ST->isUnindexed()) {<br>
- // FIXME: We should do this even without AA enabled. AA will just allow<br>
- // FindBetterChain to work in more situations. The problem with this is that<br>
- // any combine that expects memory operations to be on consecutive chains<br>
- // first needs to be updated to look for users of the same chain.<br>
-<br>
+ if (ST->isUnindexed()) {<br>
// Walk up chain skipping non-aliasing memory nodes, on this store and any<br>
// adjacent stores.<br>
if (findBetterNeighborChains(ST)) {<br>
@@ -12888,8 +12854,15 @@ SDValue DAGCombiner::visitSTORE(SDNode *<br>
if (SimplifyDemandedBits(<br>
Value,<br>
APInt::getLowBitsSet(Value.ge<wbr>tScalarValueSizeInBits(),<br>
- ST->getMemoryVT().getScalarSi<wbr>zeInBits())))<br>
+ ST->getMemoryVT().getScalarSi<wbr>zeInBits()))) {<br>
+ // Re-visit the store if anything changed and the store hasn't been merged<br>
+ // with another node (N is deleted) SimplifyDemandedBits will add Value's<br>
+ // node back to the worklist if necessary, but we also need to re-visit<br>
+ // the Store node itself.<br>
+ if (N->getOpcode() != ISD::DELETED_NODE)<br>
+ AddToWorklist(N);<br>
return SDValue(N, 0);<br>
+ }<br>
}<br>
<br>
// If this is a load followed by a store to the same location, then the store<br>
@@ -12933,15 +12906,12 @@ SDValue DAGCombiner::visitSTORE(SDNode *<br>
// There can be multiple store sequences on the same chain.<br>
// Keep trying to merge store sequences until we are unable to do so<br>
// or until we merge the last store on the chain.<br>
- SmallVector<MemOpLink, 8> StoreNodes;<br>
- bool Changed = MergeConsecutiveStores(ST, StoreNodes);<br>
+ bool Changed = MergeConsecutiveStores(ST);<br>
if (!Changed) break;<br>
-<br>
- if (any_of(StoreNodes,<br>
- [ST](const MemOpLink &Link) { return Link.MemNode == ST; })) {<br>
- // ST has been merged and no longer exists.<br>
+ // Return N as merge only uses CombineTo and no worklist clean<br>
+ // up is necessary.<br>
+ if (N->getOpcode() == ISD::DELETED_NODE || !isa<StoreSDNode>(N))<br>
return SDValue(N, 0);<br>
- }<br>
}<br>
}<br>
<br>
@@ -12950,7 +12920,7 @@ SDValue DAGCombiner::visitSTORE(SDNode *<br>
// Make sure to do this only after attempting to merge stores in order to<br>
// avoid changing the types of some subset of stores due to visit order,<br>
// preventing their merging.<br>
- if (isa<ConstantFPSDNode>(Value)) {<br>
+ if (isa<ConstantFPSDNode>(ST->get<wbr>Value())) {<br>
if (SDValue NewSt = replaceStoreOfFPConstant(ST))<br>
return NewSt;<br>
}<br>
@@ -13887,6 +13857,35 @@ SDValue DAGCombiner::visitBUILD_VECTOR<wbr>(S<br>
if (ISD::allOperandsUndef(N))<br>
return DAG.getUNDEF(VT);<br>
<br>
+ // Check if we can express BUILD VECTOR via subvector extract.<br>
+ if (!LegalTypes && (N->getNumOperands() > 1)) {<br>
+ SDValue Op0 = N->getOperand(0);<br>
+ auto checkElem = [&](SDValue Op) -> uint64_t {<br>
+ if ((Op.getOpcode() == ISD::EXTRACT_VECTOR_ELT) &&<br>
+ (Op0.getOperand(0) == Op.getOperand(0)))<br>
+ if (auto CNode = dyn_cast<ConstantSDNode>(Op.ge<wbr>tOperand(1)))<br>
+ return CNode->getZExtValue();<br>
+ return -1;<br>
+ };<br>
+<br>
+ int Offset = checkElem(Op0);<br>
+ for (unsigned i = 0; i < N->getNumOperands(); ++i) {<br>
+ if (Offset + i != checkElem(N->getOperand(i))) {<br>
+ Offset = -1;<br>
+ break;<br>
+ }<br>
+ }<br>
+<br>
+ if ((Offset == 0) &&<br>
+ (Op0.getOperand(0).getValueTyp<wbr>e() == N->getValueType(0)))<br>
+ return Op0.getOperand(0);<br>
+ if ((Offset != -1) &&<br>
+ ((Offset % N->getValueType(0).getVectorNu<wbr>mElements()) ==<br>
+ 0)) // IDX must be multiple of output size.<br>
+ return DAG.getNode(ISD::EXTRACT_SUBVE<wbr>CTOR, SDLoc(N), N->getValueType(0),<br>
+ Op0.getOperand(0), Op0.getOperand(1));<br>
+ }<br>
+<br>
if (SDValue V = reduceBuildVecExtToExtBuildVec<wbr>(N))<br>
return V;<br>
<br>
@@ -15983,7 +15982,7 @@ static bool FindBaseOffset(SDValue Ptr,<br>
if (Base.getOpcode() == ISD::ADD) {<br>
if (ConstantSDNode *C = dyn_cast<ConstantSDNode>(Base.<wbr>getOperand(1))) {<br>
Base = Base.getOperand(0);<br>
- Offset += C->getZExtValue();<br>
+ Offset += C->getSExtValue();<br>
}<br>
}<br>
<br>
@@ -16180,6 +16179,12 @@ void DAGCombiner::GatherAllAliases(<wbr>SDNod<br>
++Depth;<br>
break;<br>
<br>
+ case ISD::CopyFromReg:<br>
+ // Forward past CopyFromReg.<br>
+ Chains.push_back(Chain.getOper<wbr>and(0));<br>
+ ++Depth;<br>
+ break;<br>
+<br>
default:<br>
// For all other instructions we will just have to take what we can get.<br>
Aliases.push_back(Chain);<br>
@@ -16208,6 +16213,18 @@ SDValue DAGCombiner::FindBetterChain(S<wbr>DN<br>
return DAG.getNode(ISD::TokenFactor, SDLoc(N), MVT::Other, Aliases);<br>
}<br>
<br>
+// This function tries to collect a bunch of potentially interesting<br>
+// nodes to improve the chains of, all at once. This might seem<br>
+// redundant, as this function gets called when visiting every store<br>
+// node, so why not let the work be done on each store as it's visited?<br>
+//<br>
+// I believe this is mainly important because MergeConsecutiveStores<br>
+// is unable to deal with merging stores of different sizes, so unless<br>
+// we improve the chains of all the potential candidates up-front<br>
+// before running MergeConsecutiveStores, it might only see some of<br>
+// the nodes that will eventually be candidates, and then not be able<br>
+// to go from a partially-merged state to the desired final<br>
+// fully-merged state.<br>
bool DAGCombiner::findBetterNeighbo<wbr>rChains(StoreSDNode *St) {<br>
// This holds the base pointer, index, and the offset in bytes from the base<br>
// pointer.<br>
@@ -16243,10 +16260,8 @@ bool DAGCombiner::findBetterNeighbo<wbr>rChai<br>
if (!Ptr.equalBaseIndex(BasePtr))<br>
break;<br>
<br>
- // Find the next memory operand in the chain. If the next operand in the<br>
- // chain is a store then move up and continue the scan with the next<br>
- // memory operand. If the next operand is a load save it and use alias<br>
- // information to check if it interferes with anything.<br>
+ // Walk up the chain to find the next store node, ignoring any<br>
+ // intermediate loads. Any other kind of node will halt the loop.<br>
SDNode *NextInChain = Index->getChain().getNode();<br>
while (true) {<br>
if (StoreSDNode *STn = dyn_cast<StoreSDNode>(NextInCh<wbr>ain)) {<br>
@@ -16265,9 +16280,14 @@ bool DAGCombiner::findBetterNeighbo<wbr>rChai<br>
Index = nullptr;<br>
break;<br>
}<br>
- }<br>
+ } // end while<br>
}<br>
<br>
+ // At this point, ChainedStores lists all of the Store nodes<br>
+ // reachable by iterating up through chain nodes matching the above<br>
+ // conditions. For each such store identified, try to find an<br>
+ // earlier chain to attach the store to which won't violate the<br>
+ // required ordering.<br>
bool MadeChangeToSt = false;<br>
SmallVector<std::pair<StoreSD<wbr>Node *, SDValue>, 8> BetterChains;<br>
<br>
<br>
Modified: llvm/trunk/lib/CodeGen/TargetL<wbr>oweringBase.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp?rev=297695&r1=297694&r2=297695&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-pr<wbr>oject/llvm/trunk/lib/CodeGen/T<wbr>argetLoweringBase.cpp?rev=2976<wbr>95&r1=297694&r2=297695&view=<wbr>diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/lib/CodeGen/TargetL<wbr>oweringBase.cpp (original)<br>
+++ llvm/trunk/lib/CodeGen/TargetL<wbr>oweringBase.cpp Mon Mar </blockquote></div></div></blockquote></div></div></div></div></blockquote></div></div></div>...</blockquote></div></div>