<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Thanks Elena!<div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jan 8, 2015, at 5:34 AM, Demikhovsky, Elena <<a href="mailto:elena.demikhovsky@intel.com" class="">elena.demikhovsky@intel.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">Hi,<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""> </span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">I fixed the bug in 225441.<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""> </span></div><div class=""><div style="margin: 0cm 0cm 0.0001pt 36pt; font-size: 12pt; font-family: 'Times New Roman', serif; text-indent: -18pt;" class=""><span style="font-family: Calibri, sans-serif; color: rgb(49, 132, 155);" class=""><span class="">-<span style="font-style: normal; font-variant: normal; font-weight: normal; font-size: 7pt; line-height: normal; font-family: 'Times New Roman';" class=""> <span class="Apple-converted-space"> </span></span></span></span><span dir="LTR" class=""></span><b class=""><i class=""><span style="color: rgb(49, 132, 155);" class=""> Elena<o:p class=""></o:p></span></i></b></div></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""> </span></div><div class=""><div style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0cm 0cm;" class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><b class=""><span style="font-size: 10pt; font-family: Tahoma, sans-serif;" class="">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif;" class=""><span class="Apple-converted-space"> </span>Steven Wu [<a href="mailto:stevenwu@apple.com" style="color: purple; text-decoration: underline;" class="">mailto:stevenwu@apple.com</a>]<span class="Apple-converted-space"> </span><br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Wednesday, January 07, 2015 19:48<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span>Demikhovsky, Elena<br class=""><b class="">Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:llvm-commits@cs.uiuc.edu" style="color: purple; text-decoration: underline;" class="">llvm-commits@cs.uiuc.edu</a><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [llvm] r223348 - Masked Load / Store Intrinsics - the CodeGen part.<o:p class=""></o:p></span></div></div></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">Thanks in advance.<o:p class=""></o:p></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">Steven<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">On Jan 7, 2015, at 2:56 AM, Demikhovsky, Elena <<a href="mailto:elena.demikhovsky@intel.com" style="color: purple; text-decoration: underline;" class="">elena.demikhovsky@intel.com</a>> wrote:<o:p class=""></o:p></div></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div><div class=""><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">Hi Steven,</span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""> </span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">I can reproduce the failure. I’ll fix it by tomorrow.</span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">Thanks.</span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""> </span><o:p class=""></o:p></div></div><div class=""><div style="margin-left: 36pt;" class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; text-indent: -18pt;" class=""><span style="font-family: Calibri, sans-serif; color: rgb(49, 132, 155);" class="">-</span><span style="font-size: 7pt; color: rgb(49, 132, 155);" class=""> <span class="apple-converted-space"> </span></span><b class=""><i class=""><span style="color: rgb(49, 132, 155);" class=""> Elena</span></i></b><o:p class=""></o:p></div></div></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""> </span><o:p class=""></o:p></div></div><div class=""><div style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0cm 0cm;" class=""><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><b class=""><span style="font-size: 10pt; font-family: Tahoma, sans-serif;" class="">From:</span></b><span class="apple-converted-space"><span style="font-size: 10pt; font-family: Tahoma, sans-serif;" class=""> </span></span><span style="font-size: 10pt; font-family: Tahoma, sans-serif;" class="">Steven Wu [<a href="mailto:stevenwu@apple.com" style="color: purple; text-decoration: underline;" class="">mailto:stevenwu@apple.com</a>]<span class="apple-converted-space"> </span><br class=""><b class="">Sent:</b><span class="apple-converted-space"> </span>Wednesday, January 07, 2015 03:05<br class=""><b class="">To:</b><span class="apple-converted-space"> </span>Demikhovsky, Elena<br class=""><b class="">Cc:</b><span class="apple-converted-space"> </span><a href="mailto:llvm-commits@cs.uiuc.edu" style="color: purple; text-decoration: underline;" class="">llvm-commits@cs.uiuc.edu</a><br class=""><b class="">Subject:</b><span class="apple-converted-space"> </span>Re: [llvm] r223348 - Masked Load / Store Intrinsics - the CodeGen part.</span><o:p class=""></o:p></div></div></div></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div class=""><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 10pt;" class="">Hi Elena<br class=""><br class="">I saw the masked load/store intrinsics broke one of our bots. It causes assertion failure in LegalizeIntegerTypes. I attached a reduced test case in the email.<br class="">You can reproduce with:<br class="">llc -mtriple x86_64-apple-macosx10.10.0 -O0 bugpoint-reduced-simplified.ll<br class=""><br class="">Let me know if you want a PR for it or you need more information.<br class=""><br class="">Thanks<br class=""><br class="">Steven</span><o:p class=""></o:p></div></div></div></div><div class=""><div class=""><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 10pt;" class=""><br class=""><br class="">> On Dec 4, 2014, at 1:40 AM, Elena Demikhovsky <<a href="mailto:elena.demikhovsky@intel.com" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">elena.demikhovsky@intel.com</span></a>> wrote:<br class="">><span class="apple-converted-space"> </span><br class="">> Author: delena<br class="">> Date: Thu Dec 4 03:40:44 2014<br class="">> New Revision: 223348<br class="">><span class="apple-converted-space"> </span><br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project?rev=223348&view=rev" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project?rev=223348&view=rev</span></a><br class="">> Log:<br class="">> Masked Load / Store Intrinsics - the CodeGen part.<br class="">> I'm recommiting the codegen part of the patch.<br class="">> The vectorizer part will be send to review again.<br class="">><span class="apple-converted-space"> </span><br class="">> Masked Vector Load and Store Intrinsics.<br class="">> Introduced new target-independent intrinsics in order to support masked vector loads and stores. The loop vectorizer optimizes loops containing conditional memory accesses by generating these intrinsics for existing targets AVX2 and AVX-512. The vectorizer asks the target about availability of masked vector loads and stores.<br class="">> Added SDNodes for masked operations and lowering patterns for X86 code generator.<br class="">> Examples:<br class="">> <16 x i32> @llvm.masked.load.v16i32(i8* %addr, <16 x i32> %passthru, i32 4 /* align */, <16 x i1> %mask)<br class="">> declare void @llvm.masked.store.v8f64(i8* %addr, <8 x double> %value, i32 4, <8 x i1> %mask)<br class="">><span class="apple-converted-space"> </span><br class="">> Scalarizer for other targets (not AVX2/AVX-512) will be done in a separate patch.<br class="">><span class="apple-converted-space"> </span><br class="">><span class="apple-converted-space"> </span><a href="http://reviews.llvm.org/D6191" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://reviews.llvm.org/D6191</span></a><br class="">><span class="apple-converted-space"> </span><br class="">><span class="apple-converted-space"> </span><br class="">> Added:<br class="">> llvm/trunk/test/CodeGen/X86/masked_memop.ll<br class="">> Modified:<br class="">> llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h<br class="">> llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h<br class="">> llvm/trunk/include/llvm/CodeGen/SelectionDAG.h<br class="">> llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h<br class="">> llvm/trunk/include/llvm/IR/IRBuilder.h<br class="">> llvm/trunk/include/llvm/IR/Intrinsics.h<br class="">> llvm/trunk/include/llvm/IR/Intrinsics.td<br class="">> llvm/trunk/include/llvm/Target/TargetSelectionDAG.td<br class="">> llvm/trunk/lib/Analysis/TargetTransformInfo.cpp<br class="">> llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp<br class="">> llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp<br class="">> llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeTypes.h<br class="">> llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp<br class="">> llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp<br class="">> llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp<br class="">> llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h<br class="">> llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp<br class="">> llvm/trunk/lib/IR/Function.cpp<br class="">> llvm/trunk/lib/IR/IRBuilder.cpp<br class="">> llvm/trunk/lib/IR/Verifier.cpp<br class="">> llvm/trunk/lib/Target/X86/X86ISelLowering.cpp<br class="">> llvm/trunk/lib/Target/X86/X86InstrAVX512.td<br class="">> llvm/trunk/lib/Target/X86/X86InstrSSE.td<br class="">> llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp<br class="">> llvm/trunk/utils/TableGen/CodeGenTarget.cpp<br class="">> llvm/trunk/utils/TableGen/IntrinsicEmitter.cpp<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h (original)<br class="">> +++ llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h Thu Dec 4 03:40:44 2014<br class="">> @@ -270,6 +270,13 @@ public:<br class="">> int64_t BaseOffset, bool HasBaseReg,<br class="">> int64_t Scale) const;<br class="">><span class="apple-converted-space"> </span><br class="">> + /// \brief Return true if the target works with masked instruction<br class="">> + /// AVX2 allows masks for consecutive load and store for i32 and i64 elements.<br class="">> + /// AVX-512 architecture will also allow masks for non-consecutive memory<br class="">> + /// accesses.<br class="">> + virtual bool isLegalPredicatedStore(Type *DataType, int Consecutive) const;<br class="">> + virtual bool isLegalPredicatedLoad (Type *DataType, int Consecutive) const;<br class="">> +<br class="">> /// \brief Return the cost of the scaling factor used in the addressing<br class="">> /// mode represented by AM for this target, for a load/store<br class="">> /// of the specified type.<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h (original)<br class="">> +++ llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h Thu Dec 4 03:40:44 2014<br class="">> @@ -675,6 +675,9 @@ namespace ISD {<br class="">> ATOMIC_LOAD_UMIN,<br class="">> ATOMIC_LOAD_UMAX,<br class="">><span class="apple-converted-space"> </span><br class="">> + // Masked load and store<br class="">> + MLOAD, MSTORE,<br class="">> +<br class="">> /// This corresponds to the llvm.lifetime.* intrinsics. The first operand<br class="">> /// is the chain and the second operand is the alloca pointer.<br class="">> LIFETIME_START, LIFETIME_END,<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/include/llvm/CodeGen/SelectionDAG.h<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/SelectionDAG.h?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/SelectionDAG.h?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/include/llvm/CodeGen/SelectionDAG.h (original)<br class="">> +++ llvm/trunk/include/llvm/CodeGen/SelectionDAG.h Thu Dec 4 03:40:44 2014<br class="">> @@ -866,6 +866,10 @@ public:<br class="">> SDValue getIndexedStore(SDValue OrigStoe, SDLoc dl, SDValue Base,<br class="">> SDValue Offset, ISD::MemIndexedMode AM);<br class="">><span class="apple-converted-space"> </span><br class="">> + SDValue getMaskedLoad(EVT VT, SDLoc dl, SDValue Chain, SDValue Ptr,<br class="">> + SDValue Mask, SDValue Src0, MachineMemOperand *MMO);<br class="">> + SDValue getMaskedStore(SDValue Chain, SDLoc dl, SDValue Val,<br class="">> + SDValue Ptr, SDValue Mask, MachineMemOperand *MMO);<br class="">> /// getSrcValue - Construct a node to track a Value* through the backend.<br class="">> SDValue getSrcValue(const Value *v);<br class="">><span class="apple-converted-space"> </span><br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h (original)<br class="">> +++ llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h Thu Dec 4 03:40:44 2014<br class="">> @@ -1177,6 +1177,8 @@ public:<br class="">> N->getOpcode() == ISD::ATOMIC_LOAD_UMAX ||<br class="">> N->getOpcode() == ISD::ATOMIC_LOAD ||<br class="">> N->getOpcode() == ISD::ATOMIC_STORE ||<br class="">> + N->getOpcode() == ISD::MLOAD ||<br class="">> + N->getOpcode() == ISD::MSTORE ||<br class="">> N->isMemIntrinsic() ||<br class="">> N->isTargetMemoryOpcode();<br class="">> }<br class="">> @@ -1926,6 +1928,72 @@ public:<br class="">> }<br class="">> };<br class="">><span class="apple-converted-space"> </span><br class="">> +/// MaskedLoadStoreSDNode - This is a base class is used to represent MLOAD and<br class="">> +/// MSTORE nodes<br class="">> +///<br class="">> +class MaskedLoadStoreSDNode : public MemSDNode {<br class="">> + // Operands<br class="">> + SDUse Ops[4];<br class="">> +public:<br class="">> + friend class SelectionDAG;<br class="">> + MaskedLoadStoreSDNode(ISD::NodeType NodeTy, unsigned Order, DebugLoc dl,<br class="">> + SDValue *Operands, unsigned numOperands,<span class="apple-converted-space"> </span><br class="">> + SDVTList VTs, EVT MemVT, MachineMemOperand *MMO)<br class="">> + : MemSDNode(NodeTy, Order, dl, VTs, MemVT, MMO) {<br class="">> + InitOperands(Ops, Operands, numOperands);<br class="">> + }<br class="">> +<br class="">> + // In the both nodes address is Op1, mask is Op2:<br class="">> + // MaskedLoadSDNode (Chain, ptr, mask, src0), src0 is a passthru value<br class="">> + // MaskedStoreSDNode (Chain, ptr, mask, data)<br class="">> + // Mask is a vector of i1 elements<br class="">> + const SDValue &getBasePtr() const { return getOperand(1); }<br class="">> + const SDValue &getMask() const { return getOperand(2); }<br class="">> +<br class="">> + static bool classof(const SDNode *N) {<br class="">> + return N->getOpcode() == ISD::MLOAD ||<br class="">> + N->getOpcode() == ISD::MSTORE;<br class="">> + }<br class="">> +};<br class="">> +<br class="">> +/// MaskedLoadSDNode - This class is used to represent an MLOAD node<br class="">> +///<br class="">> +class MaskedLoadSDNode : public MaskedLoadStoreSDNode {<br class="">> +public:<br class="">> + friend class SelectionDAG;<br class="">> + MaskedLoadSDNode(unsigned Order, DebugLoc dl,<br class="">> + SDValue *Operands, unsigned numOperands,<span class="apple-converted-space"> </span><br class="">> + SDVTList VTs, EVT MemVT, MachineMemOperand *MMO)<br class="">> + : MaskedLoadStoreSDNode(ISD::MLOAD, Order, dl, Operands, numOperands,<br class="">> + VTs, MemVT, MMO)<span class="apple-converted-space"> </span><br class="">> + {}<br class="">> +<br class="">> + const SDValue &getSrc0() const { return getOperand(3); }<br class="">> + static bool classof(const SDNode *N) {<br class="">> + return N->getOpcode() == ISD::MLOAD;<br class="">> + }<br class="">> +};<br class="">> +<br class="">> +/// MaskedStoreSDNode - This class is used to represent an MSTORE node<br class="">> +///<br class="">> +class MaskedStoreSDNode : public MaskedLoadStoreSDNode {<br class="">> +<br class="">> +public:<br class="">> + friend class SelectionDAG;<br class="">> + MaskedStoreSDNode(unsigned Order, DebugLoc dl,<br class="">> + SDValue *Operands, unsigned numOperands,<span class="apple-converted-space"> </span><br class="">> + SDVTList VTs, EVT MemVT, MachineMemOperand *MMO)<br class="">> + : MaskedLoadStoreSDNode(ISD::MSTORE, Order, dl, Operands, numOperands,<br class="">> + VTs, MemVT, MMO)<span class="apple-converted-space"> </span><br class="">> + {}<br class="">> +<br class="">> + const SDValue &getData() const { return getOperand(3); }<br class="">> +<br class="">> + static bool classof(const SDNode *N) {<br class="">> + return N->getOpcode() == ISD::MSTORE;<br class="">> + }<br class="">> +};<br class="">> +<br class="">> /// MachineSDNode - An SDNode that represents everything that will be needed<br class="">> /// to construct a MachineInstr. These nodes are created during the<br class="">> /// instruction selection proper phase.<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/include/llvm/IR/IRBuilder.h<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/IRBuilder.h?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/IRBuilder.h?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/include/llvm/IR/IRBuilder.h (original)<br class="">> +++ llvm/trunk/include/llvm/IR/IRBuilder.h Thu Dec 4 03:40:44 2014<br class="">> @@ -429,11 +429,22 @@ public:<br class="">> /// If the pointer isn't i8* it will be converted.<br class="">> CallInst *CreateLifetimeEnd(Value *Ptr, ConstantInt *Size = nullptr);<br class="">><span class="apple-converted-space"> </span><br class="">> + /// \brief Create a call to Masked Load intrinsic<br class="">> + CallInst *CreateMaskedLoad(ArrayRef<Value *> Ops);<br class="">> +<br class="">> + /// \brief Create a call to Masked Store intrinsic<br class="">> + CallInst *CreateMaskedStore(ArrayRef<Value *> Ops);<br class="">> +<br class="">> /// \brief Create an assume intrinsic call that allows the optimizer to<br class="">> /// assume that the provided condition will be true.<br class="">> CallInst *CreateAssumption(Value *Cond);<br class="">><span class="apple-converted-space"> </span><br class="">> private:<br class="">> + /// \brief Create a call to a masked intrinsic with given Id.<br class="">> + /// Masked intrinsic has only one overloaded type - data type.<br class="">> + CallInst *CreateMaskedIntrinsic(unsigned Id, ArrayRef<Value *> Ops,<br class="">> + Type *DataTy);<br class="">> +<br class="">> Value *getCastedInt8PtrValue(Value *Ptr);<br class="">> };<br class="">><span class="apple-converted-space"> </span><br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/include/llvm/IR/Intrinsics.h<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/Intrinsics.h?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/Intrinsics.h?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/include/llvm/IR/Intrinsics.h (original)<br class="">> +++ llvm/trunk/include/llvm/IR/Intrinsics.h Thu Dec 4 03:40:44 2014<br class="">> @@ -76,7 +76,8 @@ namespace Intrinsic {<br class="">> enum IITDescriptorKind {<br class="">> Void, VarArg, MMX, Metadata, Half, Float, Double,<br class="">> Integer, Vector, Pointer, Struct,<br class="">> - Argument, ExtendArgument, TruncArgument, HalfVecArgument<br class="">> + Argument, ExtendArgument, TruncArgument, HalfVecArgument,<br class="">> + SameVecWidthArgument<br class="">> } Kind;<br class="">><span class="apple-converted-space"> </span><br class="">> union {<br class="">> @@ -96,13 +97,15 @@ namespace Intrinsic {<br class="">> };<br class="">> unsigned getArgumentNumber() const {<br class="">> assert(Kind == Argument || Kind == ExtendArgument ||<br class="">> - Kind == TruncArgument || Kind == HalfVecArgument);<br class="">> + Kind == TruncArgument || Kind == HalfVecArgument ||<br class="">> + Kind == SameVecWidthArgument);<br class="">> return Argument_Info >> 2;<br class="">> }<br class="">> ArgKind getArgumentKind() const {<br class="">> assert(Kind == Argument || Kind == ExtendArgument ||<br class="">> - Kind == TruncArgument || Kind == HalfVecArgument);<br class="">> - return (ArgKind)(Argument_Info&3);<br class="">> + Kind == TruncArgument || Kind == HalfVecArgument ||<br class="">> + Kind == SameVecWidthArgument);<br class="">> + return (ArgKind)(Argument_Info & 3);<br class="">> }<br class="">><span class="apple-converted-space"> </span><br class="">> static IITDescriptor get(IITDescriptorKind K, unsigned Field) {<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/include/llvm/IR/Intrinsics.td<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/Intrinsics.td?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/Intrinsics.td?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/include/llvm/IR/Intrinsics.td (original)<br class="">> +++ llvm/trunk/include/llvm/IR/Intrinsics.td Thu Dec 4 03:40:44 2014<br class="">> @@ -112,6 +112,10 @@ class LLVMMatchType<int num><br class="">> // the intrinsic is overloaded, so the matched type should be declared as iAny.<br class="">> class LLVMExtendedType<int num> : LLVMMatchType<num>;<br class="">> class LLVMTruncatedType<int num> : LLVMMatchType<num>;<br class="">> +class LLVMVectorSameWidth<int num, LLVMType elty><br class="">> + : LLVMMatchType<num> {<br class="">> + ValueType ElTy = elty.VT;<br class="">> +}<br class="">><span class="apple-converted-space"> </span><br class="">> // Match the type of another intrinsic parameter that is expected to be a<br class="">> // vector type, but change the element count to be half as many<br class="">> @@ -555,6 +559,17 @@ def int_convertuu : Intrinsic<[llvm_any<br class="">> def int_clear_cache : Intrinsic<[], [llvm_ptr_ty, llvm_ptr_ty],<br class="">> [], "llvm.clear_cache">;<br class="">><span class="apple-converted-space"> </span><br class="">> +//===-------------------------- Masked Intrinsics -------------------------===//<br class="">> +//<br class="">> +def int_masked_store : Intrinsic<[], [llvm_ptr_ty, llvm_anyvector_ty,<br class="">> + llvm_i32_ty,<br class="">> + LLVMVectorSameWidth<0, llvm_i1_ty>],<br class="">> + [IntrReadWriteArgMem]>;<br class="">> +<br class="">> +def int_masked_load : Intrinsic<[llvm_anyvector_ty],<br class="">> + [llvm_ptr_ty, LLVMMatchType<0>, llvm_i32_ty,<br class="">> + LLVMVectorSameWidth<0, llvm_i1_ty>],<br class="">> + [IntrReadArgMem]>;<br class="">> //===----------------------------------------------------------------------===//<br class="">> // Target-specific intrinsics<br class="">> //===----------------------------------------------------------------------===//<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/include/llvm/Target/TargetSelectionDAG.td<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Target/TargetSelectionDAG.td?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Target/TargetSelectionDAG.td?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/include/llvm/Target/TargetSelectionDAG.td (original)<br class="">> +++ llvm/trunk/include/llvm/Target/TargetSelectionDAG.td Thu Dec 4 03:40:44 2014<br class="">> @@ -188,6 +188,14 @@ def SDTIStore : SDTypeProfile<1, 3, [<br class="">> SDTCisSameAs<0, 2>, SDTCisPtrTy<0>, SDTCisPtrTy<3><br class="">> ]>;<br class="">><span class="apple-converted-space"> </span><br class="">> +def SDTMaskedStore: SDTypeProfile<0, 3, [ // masked store<br class="">> + SDTCisPtrTy<0>, SDTCisVec<1>, SDTCisVec<2><br class="">> +]>;<br class="">> +<br class="">> +def SDTMaskedLoad: SDTypeProfile<1, 3, [ // masked load<br class="">> + SDTCisVec<0>, SDTCisPtrTy<1>, SDTCisVec<2>, SDTCisSameAs<0, 3><br class="">> +]>;<br class="">> +<br class="">> def SDTVecShuffle : SDTypeProfile<1, 2, [<br class="">> SDTCisSameAs<0, 1>, SDTCisSameAs<1, 2><br class="">> ]>;<br class="">> @@ -454,6 +462,11 @@ def atomic_load : SDNode<"ISD::ATOM<br class="">> def atomic_store : SDNode<"ISD::ATOMIC_STORE", SDTAtomicStore,<br class="">> [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;<br class="">><span class="apple-converted-space"> </span><br class="">> +def masked_store : SDNode<"ISD::MSTORE", SDTMaskedStore,<br class="">> + [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;<br class="">> +def masked_load : SDNode<"ISD::MLOAD", SDTMaskedLoad,<br class="">> + [SDNPHasChain, SDNPMayLoad, SDNPMemOperand]>;<br class="">> +<br class="">> // Do not use ld, st directly. Use load, extload, sextload, zextload, store,<br class="">> // and truncst (see below).<br class="">> def ld : SDNode<"ISD::LOAD" , SDTLoad,<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/lib/Analysis/TargetTransformInfo.cpp<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Analysis/TargetTransformInfo.cpp?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Analysis/TargetTransformInfo.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/lib/Analysis/TargetTransformInfo.cpp (original)<br class="">> +++ llvm/trunk/lib/Analysis/TargetTransformInfo.cpp Thu Dec 4 03:40:44 2014<br class="">> @@ -101,6 +101,17 @@ bool TargetTransformInfo::isLegalICmpImm<br class="">> return PrevTTI->isLegalICmpImmediate(Imm);<br class="">> }<br class="">><span class="apple-converted-space"> </span><br class="">> +bool TargetTransformInfo::isLegalPredicatedLoad(Type *DataType,<br class="">> + int Consecutive) const {<br class="">> + return false;<br class="">> +}<br class="">> +<br class="">> +bool TargetTransformInfo::isLegalPredicatedStore(Type *DataType,<br class="">> + int Consecutive) const {<br class="">> + return false;<br class="">> +}<br class="">> +<br class="">> +<br class="">> bool TargetTransformInfo::isLegalAddressingMode(Type *Ty, GlobalValue *BaseGV,<br class="">> int64_t BaseOffset,<br class="">> bool HasBaseReg,<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp (original)<br class="">> +++ llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp Thu Dec 4 03:40:44 2014<br class="">> @@ -303,6 +303,8 @@ namespace {<br class="">> SDValue visitEXTRACT_SUBVECTOR(SDNode *N);<br class="">> SDValue visitVECTOR_SHUFFLE(SDNode *N);<br class="">> SDValue visitINSERT_SUBVECTOR(SDNode *N);<br class="">> + SDValue visitMLOAD(SDNode *N);<br class="">> + SDValue visitMSTORE(SDNode *N);<br class="">><span class="apple-converted-space"> </span><br class="">> SDValue XformToShuffleWithZero(SDNode *N);<br class="">> SDValue ReassociateOps(unsigned Opc, SDLoc DL, SDValue LHS, SDValue RHS);<br class="">> @@ -1351,6 +1353,8 @@ SDValue DAGCombiner::visit(SDNode *N) {<br class="">> case ISD::EXTRACT_SUBVECTOR: return visitEXTRACT_SUBVECTOR(N);<br class="">> case ISD::VECTOR_SHUFFLE: return visitVECTOR_SHUFFLE(N);<br class="">> case ISD::INSERT_SUBVECTOR: return visitINSERT_SUBVECTOR(N);<br class="">> + case ISD::MLOAD: return visitMLOAD(N);<br class="">> + case ISD::MSTORE: return visitMSTORE(N);<br class="">> }<br class="">> return SDValue();<br class="">> }<br class="">> @@ -4771,6 +4775,162 @@ static SDValue ConvertSelectToConcatVect<br class="">> TopHalf->isNullValue() ? RHS->getOperand(1) : LHS->getOperand(1));<br class="">> }<br class="">><span class="apple-converted-space"> </span><br class="">> +SDValue DAGCombiner::visitMSTORE(SDNode *N) {<br class="">> +<br class="">> + if (Level >= AfterLegalizeTypes)<br class="">> + return SDValue();<br class="">> +<br class="">> + MaskedStoreSDNode *MST = dyn_cast<MaskedStoreSDNode>(N);<br class="">> + SDValue Mask = MST->getMask();<br class="">> + SDValue Data = MST->getData();<br class="">> + SDLoc DL(N);<br class="">> +<br class="">> + // If the MSTORE data type requires splitting and the mask is provided by a<br class="">> + // SETCC, then split both nodes and its operands before legalization. This<br class="">> + // prevents the type legalizer from unrolling SETCC into scalar comparisons<br class="">> + // and enables future optimizations (e.g. min/max pattern matching on X86).<br class="">> + if (Mask.getOpcode() == ISD::SETCC) {<br class="">> +<br class="">> + // Check if any splitting is required.<br class="">> + if (TLI.getTypeAction(*DAG.getContext(), Data.getValueType()) !=<br class="">> + TargetLowering::TypeSplitVector)<br class="">> + return SDValue();<br class="">> +<br class="">> + SDValue MaskLo, MaskHi, Lo, Hi;<br class="">> + std::tie(MaskLo, MaskHi) = SplitVSETCC(Mask.getNode(), DAG);<br class="">> +<br class="">> + EVT LoVT, HiVT;<br class="">> + std::tie(LoVT, HiVT) = DAG.GetSplitDestVTs(MST->getValueType(0));<br class="">> +<br class="">> + SDValue Chain = MST->getChain();<br class="">> + SDValue Ptr = MST->getBasePtr();<br class="">> +<br class="">> + EVT MemoryVT = MST->getMemoryVT();<br class="">> + unsigned Alignment = MST->getOriginalAlignment();<br class="">> +<br class="">> + // if Alignment is equal to the vector size,<br class="">> + // take the half of it for the second part<br class="">> + unsigned SecondHalfAlignment =<br class="">> + (Alignment == Data->getValueType(0).getSizeInBits()/8) ?<br class="">> + Alignment/2 : Alignment;<br class="">> +<br class="">> + EVT LoMemVT, HiMemVT;<br class="">> + std::tie(LoMemVT, HiMemVT) = DAG.GetSplitDestVTs(MemoryVT);<br class="">> +<br class="">> + SDValue DataLo, DataHi;<br class="">> + std::tie(DataLo, DataHi) = DAG.SplitVector(Data, DL);<br class="">> +<br class="">> + MachineMemOperand *MMO = DAG.getMachineFunction().<br class="">> + getMachineMemOperand(MST->getPointerInfo(),<span class="apple-converted-space"> </span><br class="">> + MachineMemOperand::MOStore, LoMemVT.getStoreSize(),<br class="">> + Alignment, MST->getAAInfo(), MST->getRanges());<br class="">> +<br class="">> + Lo = DAG.getMaskedStore(Chain, DL, DataLo, Ptr, MaskLo, MMO);<br class="">> +<br class="">> + unsigned IncrementSize = LoMemVT.getSizeInBits()/8;<br class="">> + Ptr = DAG.getNode(ISD::ADD, DL, Ptr.getValueType(), Ptr,<br class="">> + DAG.getConstant(IncrementSize, Ptr.getValueType()));<br class="">> +<br class="">> + MMO = DAG.getMachineFunction().<br class="">> + getMachineMemOperand(MST->getPointerInfo(),<span class="apple-converted-space"> </span><br class="">> + MachineMemOperand::MOStore, HiMemVT.getStoreSize(),<br class="">> + SecondHalfAlignment, MST->getAAInfo(),<br class="">> + MST->getRanges());<br class="">> +<br class="">> + Hi = DAG.getMaskedStore(Chain, DL, DataHi, Ptr, MaskHi, MMO);<br class="">> +<br class="">> + AddToWorklist(Lo.getNode());<br class="">> + AddToWorklist(Hi.getNode());<br class="">> +<br class="">> + return DAG.getNode(ISD::TokenFactor, DL, MVT::Other, Lo, Hi);<br class="">> + }<br class="">> + return SDValue();<br class="">> +}<br class="">> +<br class="">> +SDValue DAGCombiner::visitMLOAD(SDNode *N) {<br class="">> +<br class="">> + if (Level >= AfterLegalizeTypes)<br class="">> + return SDValue();<br class="">> +<br class="">> + MaskedLoadSDNode *MLD = dyn_cast<MaskedLoadSDNode>(N);<br class="">> + SDValue Mask = MLD->getMask();<br class="">> + SDLoc DL(N);<br class="">> +<br class="">> + // If the MLOAD result requires splitting and the mask is provided by a<br class="">> + // SETCC, then split both nodes and its operands before legalization. This<br class="">> + // prevents the type legalizer from unrolling SETCC into scalar comparisons<br class="">> + // and enables future optimizations (e.g. min/max pattern matching on X86).<br class="">> +<br class="">> + if (Mask.getOpcode() == ISD::SETCC) {<br class="">> + EVT VT = N->getValueType(0);<br class="">> +<br class="">> + // Check if any splitting is required.<br class="">> + if (TLI.getTypeAction(*DAG.getContext(), VT) !=<br class="">> + TargetLowering::TypeSplitVector)<br class="">> + return SDValue();<br class="">> +<br class="">> + SDValue MaskLo, MaskHi, Lo, Hi;<br class="">> + std::tie(MaskLo, MaskHi) = SplitVSETCC(Mask.getNode(), DAG);<br class="">> +<br class="">> + SDValue Src0 = MLD->getSrc0();<br class="">> + SDValue Src0Lo, Src0Hi;<br class="">> + std::tie(Src0Lo, Src0Hi) = DAG.SplitVector(Src0, DL);<br class="">> +<br class="">> + EVT LoVT, HiVT;<br class="">> + std::tie(LoVT, HiVT) = DAG.GetSplitDestVTs(MLD->getValueType(0));<br class="">> +<br class="">> + SDValue Chain = MLD->getChain();<br class="">> + SDValue Ptr = MLD->getBasePtr();<br class="">> + EVT MemoryVT = MLD->getMemoryVT();<br class="">> + unsigned Alignment = MLD->getOriginalAlignment();<br class="">> +<br class="">> + // if Alignment is equal to the vector size,<br class="">> + // take the half of it for the second part<br class="">> + unsigned SecondHalfAlignment =<br class="">> + (Alignment == MLD->getValueType(0).getSizeInBits()/8) ?<br class="">> + Alignment/2 : Alignment;<br class="">> +<br class="">> + EVT LoMemVT, HiMemVT;<br class="">> + std::tie(LoMemVT, HiMemVT) = DAG.GetSplitDestVTs(MemoryVT);<br class="">> +<br class="">> + MachineMemOperand *MMO = DAG.getMachineFunction().<br class="">> + getMachineMemOperand(MLD->getPointerInfo(),<span class="apple-converted-space"> </span><br class="">> + MachineMemOperand::MOLoad, LoMemVT.getStoreSize(),<br class="">> + Alignment, MLD->getAAInfo(), MLD->getRanges());<br class="">> +<br class="">> + Lo = DAG.getMaskedLoad(LoVT, DL, Chain, Ptr, MaskLo, Src0Lo, MMO);<br class="">> +<br class="">> + unsigned IncrementSize = LoMemVT.getSizeInBits()/8;<br class="">> + Ptr = DAG.getNode(ISD::ADD, DL, Ptr.getValueType(), Ptr,<br class="">> + DAG.getConstant(IncrementSize, Ptr.getValueType()));<br class="">> +<br class="">> + MMO = DAG.getMachineFunction().<br class="">> + getMachineMemOperand(MLD->getPointerInfo(),<span class="apple-converted-space"> </span><br class="">> + MachineMemOperand::MOLoad, HiMemVT.getStoreSize(),<br class="">> + SecondHalfAlignment, MLD->getAAInfo(), MLD->getRanges());<br class="">> +<br class="">> + Hi = DAG.getMaskedLoad(HiVT, DL, Chain, Ptr, MaskHi, Src0Hi, MMO);<br class="">> +<br class="">> + AddToWorklist(Lo.getNode());<br class="">> + AddToWorklist(Hi.getNode());<br class="">> +<br class="">> + // Build a factor node to remember that this load is independent of the<br class="">> + // other one.<br class="">> + Chain = DAG.getNode(ISD::TokenFactor, DL, MVT::Other, Lo.getValue(1),<br class="">> + Hi.getValue(1));<br class="">> +<br class="">> + // Legalized the chain result - switch anything that used the old chain to<br class="">> + // use the new one.<br class="">> + DAG.ReplaceAllUsesOfValueWith(SDValue(MLD, 1), Chain);<br class="">> +<br class="">> + SDValue LoadRes = DAG.getNode(ISD::CONCAT_VECTORS, DL, VT, Lo, Hi);<br class="">> +<br class="">> + SDValue RetOps[] = { LoadRes, Chain };<br class="">> + return DAG.getMergeValues(RetOps, DL);<br class="">> + }<br class="">> + return SDValue();<br class="">> +}<br class="">> +<br class="">> SDValue DAGCombiner::visitVSELECT(SDNode *N) {<br class="">> SDValue N0 = N->getOperand(0);<br class="">> SDValue N1 = N->getOperand(1);<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp (original)<br class="">> +++ llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp Thu Dec 4 03:40:44 2014<br class="">> @@ -825,6 +825,10 @@ bool DAGTypeLegalizer::PromoteIntegerOpe<br class="">> case ISD::SINT_TO_FP: Res = PromoteIntOp_SINT_TO_FP(N); break;<br class="">> case ISD::STORE: Res = PromoteIntOp_STORE(cast<StoreSDNode>(N),<br class="">> OpNo); break;<br class="">> + case ISD::MSTORE: Res = PromoteIntOp_MSTORE(cast<MaskedStoreSDNode>(N),<br class="">> + OpNo); break;<br class="">> + case ISD::MLOAD: Res = PromoteIntOp_MLOAD(cast<MaskedLoadSDNode>(N),<br class="">> + OpNo); break;<br class="">> case ISD::TRUNCATE: Res = PromoteIntOp_TRUNCATE(N); break;<br class="">> case ISD::FP16_TO_FP:<br class="">> case ISD::UINT_TO_FP: Res = PromoteIntOp_UINT_TO_FP(N); break;<br class="">> @@ -1091,6 +1095,25 @@ SDValue DAGTypeLegalizer::PromoteIntOp_S<br class="">> N->getMemoryVT(), N->getMemOperand());<br class="">> }<br class="">><span class="apple-converted-space"> </span><br class="">> +SDValue DAGTypeLegalizer::PromoteIntOp_MSTORE(MaskedStoreSDNode *N, unsigned OpNo){<br class="">> +<br class="">> + assert(OpNo == 2 && "Only know how to promote the mask!");<br class="">> + EVT DataVT = N->getOperand(3).getValueType();<br class="">> + SDValue Mask = PromoteTargetBoolean(N->getOperand(OpNo), DataVT);<br class="">> + SmallVector<SDValue, 4> NewOps(N->op_begin(), N->op_end());<br class="">> + NewOps[OpNo] = Mask;<br class="">> + return SDValue(DAG.UpdateNodeOperands(N, NewOps), 0);<br class="">> +}<br class="">> +<br class="">> +SDValue DAGTypeLegalizer::PromoteIntOp_MLOAD(MaskedLoadSDNode *N, unsigned OpNo){<br class="">> + assert(OpNo == 2 && "Only know how to promote the mask!");<br class="">> + EVT DataVT = N->getValueType(0);<br class="">> + SDValue Mask = PromoteTargetBoolean(N->getOperand(OpNo), DataVT);<br class="">> + SmallVector<SDValue, 4> NewOps(N->op_begin(), N->op_end());<br class="">> + NewOps[OpNo] = Mask;<br class="">> + return SDValue(DAG.UpdateNodeOperands(N, NewOps), 0);<br class="">> +}<br class="">> +<br class="">> SDValue DAGTypeLegalizer::PromoteIntOp_TRUNCATE(SDNode *N) {<br class="">> SDValue Op = GetPromotedInteger(N->getOperand(0));<br class="">> return DAG.getNode(ISD::TRUNCATE, SDLoc(N), N->getValueType(0), Op);<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeTypes.h<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeTypes.h?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeTypes.h?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeTypes.h (original)<br class="">> +++ llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeTypes.h Thu Dec 4 03:40:44 2014<br class="">> @@ -285,6 +285,8 @@ private:<br class="">> SDValue PromoteIntOp_TRUNCATE(SDNode *N);<br class="">> SDValue PromoteIntOp_UINT_TO_FP(SDNode *N);<br class="">> SDValue PromoteIntOp_ZERO_EXTEND(SDNode *N);<br class="">> + SDValue PromoteIntOp_MSTORE(MaskedStoreSDNode *N, unsigned OpNo);<br class="">> + SDValue PromoteIntOp_MLOAD(MaskedLoadSDNode *N, unsigned OpNo);<br class="">><span class="apple-converted-space"> </span><br class="">> void PromoteSetCCOperands(SDValue &LHS,SDValue &RHS, ISD::CondCode Code);<br class="">><span class="apple-converted-space"> </span><br class="">> @@ -578,6 +580,7 @@ private:<br class="">> void SplitVecRes_FPOWI(SDNode *N, SDValue &Lo, SDValue &Hi);<br class="">> void SplitVecRes_INSERT_VECTOR_ELT(SDNode *N, SDValue &Lo, SDValue &Hi);<br class="">> void SplitVecRes_LOAD(LoadSDNode *N, SDValue &Lo, SDValue &Hi);<br class="">> + void SplitVecRes_MLOAD(MaskedLoadSDNode *N, SDValue &Lo, SDValue &Hi);<br class="">> void SplitVecRes_SCALAR_TO_VECTOR(SDNode *N, SDValue &Lo, SDValue &Hi);<br class="">> void SplitVecRes_SIGN_EXTEND_INREG(SDNode *N, SDValue &Lo, SDValue &Hi);<br class="">> void SplitVecRes_SETCC(SDNode *N, SDValue &Lo, SDValue &Hi);<br class="">> @@ -594,6 +597,7 @@ private:<br class="">> SDValue SplitVecOp_EXTRACT_SUBVECTOR(SDNode *N);<br class="">> SDValue SplitVecOp_EXTRACT_VECTOR_ELT(SDNode *N);<br class="">> SDValue SplitVecOp_STORE(StoreSDNode *N, unsigned OpNo);<br class="">> + SDValue SplitVecOp_MSTORE(MaskedStoreSDNode *N, unsigned OpNo);<br class="">> SDValue SplitVecOp_CONCAT_VECTORS(SDNode *N);<br class="">> SDValue SplitVecOp_TRUNCATE(SDNode *N);<br class="">> SDValue SplitVecOp_VSETCC(SDNode *N);<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp (original)<br class="">> +++ llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp Thu Dec 4 03:40:44 2014<br class="">> @@ -597,6 +597,9 @@ void DAGTypeLegalizer::SplitVectorResult<br class="">> case ISD::LOAD:<br class="">> SplitVecRes_LOAD(cast<LoadSDNode>(N), Lo, Hi);<br class="">> break;<br class="">> + case ISD::MLOAD:<br class="">> + SplitVecRes_MLOAD(cast<MaskedLoadSDNode>(N), Lo, Hi);<br class="">> + break;<br class="">> case ISD::SETCC:<br class="">> SplitVecRes_SETCC(N, Lo, Hi);<br class="">> break;<br class="">> @@ -979,6 +982,64 @@ void DAGTypeLegalizer::SplitVecRes_LOAD(<br class="">> ReplaceValueWith(SDValue(LD, 1), Ch);<br class="">> }<br class="">><span class="apple-converted-space"> </span><br class="">> +void DAGTypeLegalizer::SplitVecRes_MLOAD(MaskedLoadSDNode *MLD,<br class="">> + SDValue &Lo, SDValue &Hi) {<br class="">> + EVT LoVT, HiVT;<br class="">> + SDLoc dl(MLD);<br class="">> + std::tie(LoVT, HiVT) = DAG.GetSplitDestVTs(MLD->getValueType(0));<br class="">> +<br class="">> + SDValue Ch = MLD->getChain();<br class="">> + SDValue Ptr = MLD->getBasePtr();<br class="">> + SDValue Mask = MLD->getMask();<br class="">> + unsigned Alignment = MLD->getOriginalAlignment();<br class="">> +<br class="">> + // if Alignment is equal to the vector size,<br class="">> + // take the half of it for the second part<br class="">> + unsigned SecondHalfAlignment =<br class="">> + (Alignment == MLD->getValueType(0).getSizeInBits()/8) ?<br class="">> + Alignment/2 : Alignment;<br class="">> +<br class="">> + SDValue MaskLo, MaskHi;<br class="">> + std::tie(MaskLo, MaskHi) = DAG.SplitVector(Mask, dl);<br class="">> +<br class="">> + EVT MemoryVT = MLD->getMemoryVT();<br class="">> + EVT LoMemVT, HiMemVT;<br class="">> + std::tie(LoMemVT, HiMemVT) = DAG.GetSplitDestVTs(MemoryVT);<br class="">> +<br class="">> + SDValue Src0 = MLD->getSrc0();<br class="">> + SDValue Src0Lo, Src0Hi;<br class="">> + std::tie(Src0Lo, Src0Hi) = DAG.SplitVector(Src0, dl);<br class="">> +<br class="">> + MachineMemOperand *MMO = DAG.getMachineFunction().<br class="">> + getMachineMemOperand(MLD->getPointerInfo(),<span class="apple-converted-space"> </span><br class="">> + MachineMemOperand::MOLoad, LoMemVT.getStoreSize(),<br class="">> + Alignment, MLD->getAAInfo(), MLD->getRanges());<br class="">> +<br class="">> + Lo = DAG.getMaskedLoad(LoVT, dl, Ch, Ptr, MaskLo, Src0Lo, MMO);<br class="">> +<br class="">> + unsigned IncrementSize = LoMemVT.getSizeInBits()/8;<br class="">> + Ptr = DAG.getNode(ISD::ADD, dl, Ptr.getValueType(), Ptr,<br class="">> + DAG.getConstant(IncrementSize, Ptr.getValueType()));<br class="">> +<br class="">> + MMO = DAG.getMachineFunction().<br class="">> + getMachineMemOperand(MLD->getPointerInfo(),<span class="apple-converted-space"> </span><br class="">> + MachineMemOperand::MOLoad, HiMemVT.getStoreSize(),<br class="">> + SecondHalfAlignment, MLD->getAAInfo(), MLD->getRanges());<br class="">> +<br class="">> + Hi = DAG.getMaskedLoad(HiVT, dl, Ch, Ptr, MaskHi, Src0Hi, MMO);<br class="">> +<br class="">> +<br class="">> + // Build a factor node to remember that this load is independent of the<br class="">> + // other one.<br class="">> + Ch = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, Lo.getValue(1),<br class="">> + Hi.getValue(1));<br class="">> +<br class="">> + // Legalized the chain result - switch anything that used the old chain to<br class="">> + // use the new one.<br class="">> + ReplaceValueWith(SDValue(MLD, 1), Ch);<br class="">> +<br class="">> +}<br class="">> +<br class="">> void DAGTypeLegalizer::SplitVecRes_SETCC(SDNode *N, SDValue &Lo, SDValue &Hi) {<br class="">> assert(N->getValueType(0).isVector() &&<br class="">> N->getOperand(0).getValueType().isVector() &&<br class="">> @@ -1234,6 +1295,9 @@ bool DAGTypeLegalizer::SplitVectorOperan<br class="">> case ISD::STORE:<br class="">> Res = SplitVecOp_STORE(cast<StoreSDNode>(N), OpNo);<br class="">> break;<br class="">> + case ISD::MSTORE:<br class="">> + Res = SplitVecOp_MSTORE(cast<MaskedStoreSDNode>(N), OpNo);<br class="">> + break;<br class="">> case ISD::VSELECT:<br class="">> Res = SplitVecOp_VSELECT(N, OpNo);<br class="">> break;<br class="">> @@ -1395,6 +1459,56 @@ SDValue DAGTypeLegalizer::SplitVecOp_EXT<br class="">> MachinePointerInfo(), EltVT, false, false, false, 0);<br class="">> }<br class="">><span class="apple-converted-space"> </span><br class="">> +SDValue DAGTypeLegalizer::SplitVecOp_MSTORE(MaskedStoreSDNode *N,<br class="">> + unsigned OpNo) {<br class="">> + SDValue Ch = N->getChain();<br class="">> + SDValue Ptr = N->getBasePtr();<br class="">> + SDValue Mask = N->getMask();<br class="">> + SDValue Data = N->getData();<br class="">> + EVT MemoryVT = N->getMemoryVT();<br class="">> + unsigned Alignment = N->getOriginalAlignment();<br class="">> + SDLoc DL(N);<br class="">> + <span class="apple-converted-space"> </span><br class="">> + EVT LoMemVT, HiMemVT;<br class="">> + std::tie(LoMemVT, HiMemVT) = DAG.GetSplitDestVTs(MemoryVT);<br class="">> +<br class="">> + SDValue DataLo, DataHi;<br class="">> + GetSplitVector(Data, DataLo, DataHi);<br class="">> + SDValue MaskLo, MaskHi;<br class="">> + GetSplitVector(Mask, MaskLo, MaskHi);<br class="">> +<br class="">> + // if Alignment is equal to the vector size,<br class="">> + // take the half of it for the second part<br class="">> + unsigned SecondHalfAlignment =<br class="">> + (Alignment == Data->getValueType(0).getSizeInBits()/8) ?<br class="">> + Alignment/2 : Alignment;<br class="">> +<br class="">> + SDValue Lo, Hi;<br class="">> + MachineMemOperand *MMO = DAG.getMachineFunction().<br class="">> + getMachineMemOperand(N->getPointerInfo(),<span class="apple-converted-space"> </span><br class="">> + MachineMemOperand::MOStore, LoMemVT.getStoreSize(),<br class="">> + Alignment, N->getAAInfo(), N->getRanges());<br class="">> +<br class="">> + Lo = DAG.getMaskedStore(Ch, DL, DataLo, Ptr, MaskLo, MMO);<br class="">> +<br class="">> + unsigned IncrementSize = LoMemVT.getSizeInBits()/8;<br class="">> + Ptr = DAG.getNode(ISD::ADD, DL, Ptr.getValueType(), Ptr,<br class="">> + DAG.getConstant(IncrementSize, Ptr.getValueType()));<br class="">> +<br class="">> + MMO = DAG.getMachineFunction().<br class="">> + getMachineMemOperand(N->getPointerInfo(),<span class="apple-converted-space"> </span><br class="">> + MachineMemOperand::MOStore, HiMemVT.getStoreSize(),<br class="">> + SecondHalfAlignment, N->getAAInfo(), N->getRanges());<br class="">> +<br class="">> + Hi = DAG.getMaskedStore(Ch, DL, DataHi, Ptr, MaskHi, MMO);<br class="">> +<br class="">> +<br class="">> + // Build a factor node to remember that this store is independent of the<br class="">> + // other one.<br class="">> + return DAG.getNode(ISD::TokenFactor, DL, MVT::Other, Lo, Hi);<br class="">> +<br class="">> +}<br class="">> +<br class="">> SDValue DAGTypeLegalizer::SplitVecOp_STORE(StoreSDNode *N, unsigned OpNo) {<br class="">> assert(N->isUnindexed() && "Indexed store of vector?");<br class="">> assert(OpNo == 1 && "Can only split the stored value");<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp (original)<br class="">> +++ llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp Thu Dec 4 03:40:44 2014<br class="">> @@ -4917,6 +4917,60 @@ SelectionDAG::getIndexedStore(SDValue Or<br class="">> return SDValue(N, 0);<br class="">> }<br class="">><span class="apple-converted-space"> </span><br class="">> +SDValue<br class="">> +SelectionDAG::getMaskedLoad(EVT VT, SDLoc dl, SDValue Chain,<br class="">> + SDValue Ptr, SDValue Mask, SDValue Src0,<br class="">> + MachineMemOperand *MMO) {<br class="">> +<br class="">> + SDVTList VTs = getVTList(VT, MVT::Other);<br class="">> + SDValue Ops[] = { Chain, Ptr, Mask, Src0 };<br class="">> + FoldingSetNodeID ID;<br class="">> + AddNodeIDNode(ID, ISD::MLOAD, VTs, Ops);<br class="">> + ID.AddInteger(VT.getRawBits());<br class="">> + ID.AddInteger(encodeMemSDNodeFlags(ISD::NON_EXTLOAD, ISD::UNINDEXED,<br class="">> + MMO->isVolatile(),<br class="">> + MMO->isNonTemporal(),<br class="">> + MMO->isInvariant()));<br class="">> + ID.AddInteger(MMO->getPointerInfo().getAddrSpace());<br class="">> + void *IP = nullptr;<br class="">> + if (SDNode *E = CSEMap.FindNodeOrInsertPos(ID, IP)) {<br class="">> + cast<MaskedLoadSDNode>(E)->refineAlignment(MMO);<br class="">> + return SDValue(E, 0);<br class="">> + }<br class="">> + SDNode *N = new (NodeAllocator) MaskedLoadSDNode(dl.getIROrder(),<br class="">> + dl.getDebugLoc(), Ops, 4, VTs,<br class="">> + VT, MMO);<br class="">> + CSEMap.InsertNode(N, IP);<br class="">> + InsertNode(N);<br class="">> + return SDValue(N, 0);<br class="">> +}<br class="">> +<br class="">> +SDValue SelectionDAG::getMaskedStore(SDValue Chain, SDLoc dl, SDValue Val,<br class="">> + SDValue Ptr, SDValue Mask, MachineMemOperand *MMO) {<br class="">> + assert(Chain.getValueType() == MVT::Other &&<br class="">> + "Invalid chain type");<br class="">> + EVT VT = Val.getValueType();<br class="">> + SDVTList VTs = getVTList(MVT::Other);<br class="">> + SDValue Ops[] = { Chain, Ptr, Mask, Val };<br class="">> + FoldingSetNodeID ID;<br class="">> + AddNodeIDNode(ID, ISD::MSTORE, VTs, Ops);<br class="">> + ID.AddInteger(VT.getRawBits());<br class="">> + ID.AddInteger(encodeMemSDNodeFlags(false, ISD::UNINDEXED, MMO->isVolatile(),<br class="">> + MMO->isNonTemporal(), MMO->isInvariant()));<br class="">> + ID.AddInteger(MMO->getPointerInfo().getAddrSpace());<br class="">> + void *IP = nullptr;<br class="">> + if (SDNode *E = CSEMap.FindNodeOrInsertPos(ID, IP)) {<br class="">> + cast<MaskedStoreSDNode>(E)->refineAlignment(MMO);<br class="">> + return SDValue(E, 0);<br class="">> + }<br class="">> + SDNode *N = new (NodeAllocator) MaskedStoreSDNode(dl.getIROrder(),<br class="">> + dl.getDebugLoc(), Ops, 4,<br class="">> + VTs, VT, MMO);<br class="">> + CSEMap.InsertNode(N, IP);<br class="">> + InsertNode(N);<br class="">> + return SDValue(N, 0);<br class="">> +}<br class="">> +<br class="">> SDValue SelectionDAG::getVAArg(EVT VT, SDLoc dl,<br class="">> SDValue Chain, SDValue Ptr,<br class="">> SDValue SV,<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp (original)<br class="">> +++ llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp Thu Dec 4 03:40:44 2014<br class="">> @@ -3617,6 +3617,70 @@ void SelectionDAGBuilder::visitStore(con<br class="">> DAG.setRoot(StoreNode);<br class="">> }<br class="">><span class="apple-converted-space"> </span><br class="">> +void SelectionDAGBuilder::visitMaskedStore(const CallInst &I) {<br class="">> + SDLoc sdl = getCurSDLoc();<br class="">> +<br class="">> + Value *PtrOperand = I.getArgOperand(0);<br class="">> + SDValue Ptr = getValue(PtrOperand);<br class="">> + SDValue Src0 = getValue(I.getArgOperand(1));<br class="">> + SDValue Mask = getValue(I.getArgOperand(3));<br class="">> + EVT VT = Src0.getValueType();<br class="">> + unsigned Alignment = (cast<ConstantInt>(I.getArgOperand(2)))->getZExtValue();<br class="">> + if (!Alignment)<br class="">> + Alignment = DAG.getEVTAlignment(VT);<br class="">> +<br class="">> + AAMDNodes AAInfo;<br class="">> + I.getAAMetadata(AAInfo);<br class="">> +<br class="">> + MachineMemOperand *MMO =<br class="">> + DAG.getMachineFunction().<br class="">> + getMachineMemOperand(MachinePointerInfo(PtrOperand),<br class="">> + MachineMemOperand::MOStore, VT.getStoreSize(),<br class="">> + Alignment, AAInfo);<br class="">> + SDValue StoreNode = DAG.getMaskedStore(getRoot(), sdl, Src0, Ptr, Mask, MMO);<br class="">> + DAG.setRoot(StoreNode);<br class="">> + setValue(&I, StoreNode);<br class="">> +}<br class="">> +<br class="">> +void SelectionDAGBuilder::visitMaskedLoad(const CallInst &I) {<br class="">> + SDLoc sdl = getCurSDLoc();<br class="">> +<br class="">> + Value *PtrOperand = I.getArgOperand(0);<br class="">> + SDValue Ptr = getValue(PtrOperand);<br class="">> + SDValue Src0 = getValue(I.getArgOperand(1));<br class="">> + SDValue Mask = getValue(I.getArgOperand(3));<br class="">> +<br class="">> + const TargetLowering &TLI = DAG.getTargetLoweringInfo();<br class="">> + EVT VT = TLI.getValueType(I.getType());<br class="">> + unsigned Alignment = (cast<ConstantInt>(I.getArgOperand(2)))->getZExtValue();<br class="">> + if (!Alignment)<br class="">> + Alignment = DAG.getEVTAlignment(VT);<br class="">> +<br class="">> + AAMDNodes AAInfo;<br class="">> + I.getAAMetadata(AAInfo);<br class="">> + const MDNode *Ranges = I.getMetadata(LLVMContext::MD_range);<br class="">> +<br class="">> + SDValue InChain = DAG.getRoot();<br class="">> + if (AA->pointsToConstantMemory(<br class="">> + AliasAnalysis::Location(PtrOperand,<br class="">> + AA->getTypeStoreSize(I.getType()),<br class="">> + AAInfo))) {<br class="">> + // Do not serialize (non-volatile) loads of constant memory with anything.<br class="">> + InChain = DAG.getEntryNode();<br class="">> + }<br class="">> +<br class="">> + MachineMemOperand *MMO =<br class="">> + DAG.getMachineFunction().<br class="">> + getMachineMemOperand(MachinePointerInfo(PtrOperand),<br class="">> + MachineMemOperand::MOLoad, VT.getStoreSize(),<br class="">> + Alignment, AAInfo, Ranges);<br class="">> +<br class="">> + SDValue Load = DAG.getMaskedLoad(VT, sdl, InChain, Ptr, Mask, Src0, MMO);<br class="">> + SDValue OutChain = Load.getValue(1);<br class="">> + DAG.setRoot(OutChain);<br class="">> + setValue(&I, Load);<br class="">> +}<br class="">> +<br class="">> void SelectionDAGBuilder::visitAtomicCmpXchg(const AtomicCmpXchgInst &I) {<br class="">> SDLoc dl = getCurSDLoc();<br class="">> AtomicOrdering SuccessOrder = I.getSuccessOrdering();<br class="">> @@ -4918,6 +4982,12 @@ SelectionDAGBuilder::visitIntrinsicCall(<br class="">> return nullptr;<br class="">> }<br class="">><span class="apple-converted-space"> </span><br class="">> + case Intrinsic::masked_load:<br class="">> + visitMaskedLoad(I);<br class="">> + return nullptr;<br class="">> + case Intrinsic::masked_store:<br class="">> + visitMaskedStore(I);<br class="">> + return nullptr;<br class="">> case Intrinsic::x86_mmx_pslli_w:<br class="">> case Intrinsic::x86_mmx_pslli_d:<br class="">> case Intrinsic::x86_mmx_pslli_q:<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h (original)<br class="">> +++ llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h Thu Dec 4 03:40:44 2014<br class="">> @@ -769,6 +769,8 @@ private:<br class="">> void visitAlloca(const AllocaInst &I);<br class="">> void visitLoad(const LoadInst &I);<br class="">> void visitStore(const StoreInst &I);<br class="">> + void visitMaskedLoad(const CallInst &I);<br class="">> + void visitMaskedStore(const CallInst &I);<br class="">> void visitAtomicCmpXchg(const AtomicCmpXchgInst &I);<br class="">> void visitAtomicRMW(const AtomicRMWInst &I);<br class="">> void visitFence(const FenceInst &I);<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp (original)<br class="">> +++ llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp Thu Dec 4 03:40:44 2014<br class="">> @@ -269,6 +269,8 @@ std::string SDNode::getOperationName(con<br class="">> // Other operators<br class="">> case ISD::LOAD: return "load";<br class="">> case ISD::STORE: return "store";<br class="">> + case ISD::MLOAD: return "masked_load";<br class="">> + case ISD::MSTORE: return "masked_store";<br class="">> case ISD::VAARG: return "vaarg";<br class="">> case ISD::VACOPY: return "vacopy";<br class="">> case ISD::VAEND: return "vaend";<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/lib/IR/Function.cpp<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/IR/Function.cpp?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/IR/Function.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/lib/IR/Function.cpp (original)<br class="">> +++ llvm/trunk/lib/IR/Function.cpp Thu Dec 4 03:40:44 2014<br class="">> @@ -551,7 +551,8 @@ enum IIT_Info {<br class="">> IIT_ANYPTR = 26,<br class="">> IIT_V1 = 27,<br class="">> IIT_VARARG = 28,<br class="">> - IIT_HALF_VEC_ARG = 29<br class="">> + IIT_HALF_VEC_ARG = 29,<br class="">> + IIT_SAME_VEC_WIDTH_ARG = 30<br class="">> };<br class="">><span class="apple-converted-space"> </span><br class="">><span class="apple-converted-space"> </span><br class="">> @@ -659,6 +660,12 @@ static void DecodeIITType(unsigned &Next<br class="">> ArgInfo));<br class="">> return;<br class="">> }<br class="">> + case IIT_SAME_VEC_WIDTH_ARG: {<br class="">> + unsigned ArgInfo = (NextElt == Infos.size() ? 0 : Infos[NextElt++]);<br class="">> + OutputTable.push_back(IITDescriptor::get(IITDescriptor::SameVecWidthArgument,<br class="">> + ArgInfo));<br class="">> + return;<br class="">> + }<br class="">> case IIT_EMPTYSTRUCT:<br class="">> OutputTable.push_back(IITDescriptor::get(IITDescriptor::Struct, 0));<br class="">> return;<br class="">> @@ -766,7 +773,14 @@ static Type *DecodeFixedType(ArrayRef<In<br class="">> case IITDescriptor::HalfVecArgument:<br class="">> return VectorType::getHalfElementsVectorType(cast<VectorType>(<br class="">> Tys[D.getArgumentNumber()]));<br class="">> - }<br class="">> + case IITDescriptor::SameVecWidthArgument:<br class="">> + Type *EltTy = DecodeFixedType(Infos, Tys, Context);<br class="">> + Type *Ty = Tys[D.getArgumentNumber()];<br class="">> + if (VectorType *VTy = dyn_cast<VectorType>(Ty)) {<br class="">> + return VectorType::get(EltTy, VTy->getNumElements());<br class="">> + }<br class="">> + llvm_unreachable("unhandled");<br class="">> + }<br class="">> llvm_unreachable("unhandled");<br class="">> }<br class="">><span class="apple-converted-space"> </span><br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/lib/IR/IRBuilder.cpp<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/IR/IRBuilder.cpp?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/IR/IRBuilder.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/lib/IR/IRBuilder.cpp (original)<br class="">> +++ llvm/trunk/lib/IR/IRBuilder.cpp Thu Dec 4 03:40:44 2014<br class="">> @@ -183,3 +183,29 @@ CallInst *IRBuilderBase::CreateAssumptio<br class="">> return createCallHelper(FnAssume, Ops, this);<br class="">> }<br class="">><span class="apple-converted-space"> </span><br class="">> +/// Create a call to a Masked Load intrinsic.<br class="">> +/// Ops - an array of operands.<br class="">> +CallInst *IRBuilderBase::CreateMaskedLoad(ArrayRef<Value *> Ops) {<br class="">> + // The only one overloaded type - the type of passthru value in this case<br class="">> + Type *DataTy = Ops[1]->getType();<br class="">> + return CreateMaskedIntrinsic(Intrinsic::masked_load, Ops, DataTy);<br class="">> +}<br class="">> +<br class="">> +/// Create a call to a Masked Store intrinsic.<br class="">> +/// Ops - an array of operands.<br class="">> +CallInst *IRBuilderBase::CreateMaskedStore(ArrayRef<Value *> Ops) {<br class="">> + // DataTy - type of the data to be stored - the only one overloaded type<br class="">> + Type *DataTy = Ops[1]->getType();<br class="">> + return CreateMaskedIntrinsic(Intrinsic::masked_store, Ops, DataTy);<br class="">> +}<br class="">> +<br class="">> +/// Create a call to a Masked intrinsic, with given intrinsic Id,<br class="">> +/// an array of operands - Ops, and one overloaded type - DataTy<br class="">> +CallInst *IRBuilderBase::CreateMaskedIntrinsic(unsigned Id,<br class="">> + ArrayRef<Value *> Ops,<br class="">> + Type *DataTy) {<br class="">> + Module *M = BB->getParent()->getParent();<br class="">> + Type *OverloadedTypes[] = { DataTy };<br class="">> + Value *TheFn = Intrinsic::getDeclaration(M, (Intrinsic::ID)Id, OverloadedTypes);<br class="">> + return createCallHelper(TheFn, Ops, this);<br class="">> +}<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/lib/IR/Verifier.cpp<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/IR/Verifier.cpp?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/IR/Verifier.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/lib/IR/Verifier.cpp (original)<br class="">> +++ llvm/trunk/lib/IR/Verifier.cpp Thu Dec 4 03:40:44 2014<br class="">> @@ -2406,6 +2406,19 @@ bool Verifier::VerifyIntrinsicType(Type<br class="">> !isa<VectorType>(ArgTys[D.getArgumentNumber()]) ||<br class="">> VectorType::getHalfElementsVectorType(<br class="">> cast<VectorType>(ArgTys[D.getArgumentNumber()])) != Ty;<br class="">> + case IITDescriptor::SameVecWidthArgument: {<br class="">> + if (D.getArgumentNumber() >= ArgTys.size())<br class="">> + return true;<br class="">> + VectorType * ReferenceType =<br class="">> + dyn_cast<VectorType>(ArgTys[D.getArgumentNumber()]);<br class="">> + VectorType *ThisArgType = dyn_cast<VectorType>(Ty);<br class="">> + if (!ThisArgType || !ReferenceType ||<span class="apple-converted-space"> </span><br class="">> + (ReferenceType->getVectorNumElements() !=<br class="">> + ThisArgType->getVectorNumElements()))<br class="">> + return true;<br class="">> + return VerifyIntrinsicType(ThisArgType->getVectorElementType(),<br class="">> + Infos, ArgTys);<br class="">> + }<br class="">> }<br class="">> llvm_unreachable("unhandled");<br class="">> }<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/lib/Target/X86/X86ISelLowering.cpp<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86ISelLowering.cpp?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86ISelLowering.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/lib/Target/X86/X86ISelLowering.cpp (original)<br class="">> +++ llvm/trunk/lib/Target/X86/X86ISelLowering.cpp Thu Dec 4 03:40:44 2014<br class="">> @@ -1319,13 +1319,21 @@ void X86TargetLowering::resetOperationAc<br class="">><span class="apple-converted-space"> </span><br class="">> // Extract subvector is special because the value type<br class="">> // (result) is 128-bit but the source is 256-bit wide.<br class="">> - if (VT.is128BitVector())<br class="">> + if (VT.is128BitVector()) {<br class="">> + if (VT.getScalarSizeInBits() >= 32) {<br class="">> + setOperationAction(ISD::MLOAD, VT, Custom);<br class="">> + setOperationAction(ISD::MSTORE, VT, Custom);<br class="">> + }<br class="">> setOperationAction(ISD::EXTRACT_SUBVECTOR, VT, Custom);<br class="">> -<br class="">> + }<br class="">> // Do not attempt to custom lower other non-256-bit vectors<br class="">> if (!VT.is256BitVector())<br class="">> continue;<br class="">><span class="apple-converted-space"> </span><br class="">> + if (VT.getScalarSizeInBits() >= 32) {<br class="">> + setOperationAction(ISD::MLOAD, VT, Legal);<br class="">> + setOperationAction(ISD::MSTORE, VT, Legal);<br class="">> + }<br class="">> setOperationAction(ISD::BUILD_VECTOR, VT, Custom);<br class="">> setOperationAction(ISD::VECTOR_SHUFFLE, VT, Custom);<br class="">> setOperationAction(ISD::INSERT_VECTOR_ELT, VT, Custom);<br class="">> @@ -1492,9 +1500,13 @@ void X86TargetLowering::resetOperationAc<br class="">> unsigned EltSize = VT.getVectorElementType().getSizeInBits();<br class="">> // Extract subvector is special because the value type<br class="">> // (result) is 256/128-bit but the source is 512-bit wide.<br class="">> - if (VT.is128BitVector() || VT.is256BitVector())<br class="">> + if (VT.is128BitVector() || VT.is256BitVector()) {<br class="">> setOperationAction(ISD::EXTRACT_SUBVECTOR, VT, Custom);<br class="">> -<br class="">> + if ( EltSize >= 32) {<br class="">> + setOperationAction(ISD::MLOAD, VT, Legal);<br class="">> + setOperationAction(ISD::MSTORE, VT, Legal);<br class="">> + }<br class="">> + }<br class="">> if (VT.getVectorElementType() == MVT::i1)<br class="">> setOperationAction(ISD::EXTRACT_SUBVECTOR, VT, Legal);<br class="">><span class="apple-converted-space"> </span><br class="">> @@ -1510,6 +1522,8 @@ void X86TargetLowering::resetOperationAc<br class="">> setOperationAction(ISD::EXTRACT_VECTOR_ELT, VT, Custom);<br class="">> setOperationAction(ISD::SCALAR_TO_VECTOR, VT, Custom);<br class="">> setOperationAction(ISD::INSERT_SUBVECTOR, VT, Custom);<br class="">> + setOperationAction(ISD::MLOAD, VT, Legal);<br class="">> + setOperationAction(ISD::MSTORE, VT, Legal);<br class="">> }<br class="">> }<br class="">> for (int i = MVT::v32i8; i != MVT::v8i64; ++i) {<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/lib/Target/X86/X86InstrAVX512.td<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86InstrAVX512.td?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86InstrAVX512.td?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/lib/Target/X86/X86InstrAVX512.td (original)<br class="">> +++ llvm/trunk/lib/Target/X86/X86InstrAVX512.td Thu Dec 4 03:40:44 2014<br class="">> @@ -2122,6 +2122,41 @@ def: Pat<(int_x86_avx512_mask_storeu_pd_<br class="">> (VMOVUPDZmrk addr:$ptr, (v8i1 (COPY_TO_REGCLASS GR8:$mask, VK8WM)),<br class="">> VR512:$src)>;<br class="">><span class="apple-converted-space"> </span><br class="">> +def: Pat<(masked_store addr:$ptr, VK8WM:$mask, (v8f32 VR256:$src)),<br class="">> + (VMOVUPSZmrk addr:$ptr,<br class="">> + (v16i1 (COPY_TO_REGCLASS VK8WM:$mask, VK16WM)),<br class="">> + (INSERT_SUBREG (v16f32 (IMPLICIT_DEF)), VR256:$src, sub_ymm))>;<br class="">> +<br class="">> +def: Pat<(v8f32 (masked_load addr:$ptr, VK8WM:$mask, undef)),<br class="">> + (v8f32 (EXTRACT_SUBREG (v16f32 (VMOVUPSZrmkz<span class="apple-converted-space"> </span><br class="">> + (v16i1 (COPY_TO_REGCLASS VK8WM:$mask, VK16WM)), addr:$ptr)), sub_ymm))>;<br class="">> +<br class="">> +def: Pat<(masked_store addr:$ptr, VK16WM:$mask, (v16f32 VR512:$src)),<br class="">> + (VMOVUPSZmrk addr:$ptr, VK16WM:$mask, VR512:$src)>;<br class="">> +<br class="">> +def: Pat<(masked_store addr:$ptr, VK8WM:$mask, (v8f64 VR512:$src)),<br class="">> + (VMOVUPDZmrk addr:$ptr, VK8WM:$mask, VR512:$src)>;<br class="">> +<br class="">> +def: Pat<(v16f32 (masked_load addr:$ptr, VK16WM:$mask, undef)),<br class="">> + (VMOVUPSZrmkz VK16WM:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v16f32 (masked_load addr:$ptr, VK16WM:$mask,<br class="">> + (bc_v16f32 (v16i32 immAllZerosV)))),<br class="">> + (VMOVUPSZrmkz VK16WM:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v16f32 (masked_load addr:$ptr, VK16WM:$mask, (v16f32 VR512:$src0))),<br class="">> + (VMOVUPSZrmk VR512:$src0, VK16WM:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v8f64 (masked_load addr:$ptr, VK8WM:$mask, undef)),<br class="">> + (VMOVUPDZrmkz VK8WM:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v8f64 (masked_load addr:$ptr, VK8WM:$mask,<br class="">> + (bc_v8f64 (v16i32 immAllZerosV)))),<br class="">> + (VMOVUPDZrmkz VK8WM:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v8f64 (masked_load addr:$ptr, VK8WM:$mask, (v8f64 VR512:$src0))),<br class="">> + (VMOVUPDZrmk VR512:$src0, VK8WM:$mask, addr:$ptr)>;<br class="">> +<br class="">> defm VMOVDQA32 : avx512_load_vl<0x6F, "vmovdqa32", "alignedload", "i", "32",<br class="">> "16", "8", "4", SSEPackedInt, HasAVX512>,<br class="">> avx512_store_vl<0x7F, "vmovdqa32", "alignedstore",<br class="">> @@ -2196,6 +2231,46 @@ def : Pat<(v16i32 (vselect VK16WM:$mask,<br class="">> (VMOVDQU32Zrrkz (KNOTWrr VK16WM:$mask), VR512:$src)>;<br class="">> }<br class="">><span class="apple-converted-space"> </span><br class="">> +def: Pat<(v16i32 (masked_load addr:$ptr, VK16WM:$mask, (v16i32 immAllZerosV))),<br class="">> + (VMOVDQU32Zrmkz VK16WM:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v16i32 (masked_load addr:$ptr, VK16WM:$mask, undef)),<br class="">> + (VMOVDQU32Zrmkz VK16WM:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v16i32 (masked_load addr:$ptr, VK16WM:$mask, (v16i32 VR512:$src0))),<br class="">> + (VMOVDQU32Zrmk VR512:$src0, VK16WM:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v8i64 (masked_load addr:$ptr, VK8WM:$mask,<br class="">> + (bc_v8i64 (v16i32 immAllZerosV)))),<br class="">> + (VMOVDQU64Zrmkz VK8WM:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v8i64 (masked_load addr:$ptr, VK8WM:$mask, undef)),<br class="">> + (VMOVDQU64Zrmkz VK8WM:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v8i64 (masked_load addr:$ptr, VK8WM:$mask, (v8i64 VR512:$src0))),<br class="">> + (VMOVDQU64Zrmk VR512:$src0, VK8WM:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(masked_store addr:$ptr, VK16WM:$mask, (v16i32 VR512:$src)),<br class="">> + (VMOVDQU32Zmrk addr:$ptr, VK16WM:$mask, VR512:$src)>;<br class="">> +<br class="">> +def: Pat<(masked_store addr:$ptr, VK8WM:$mask, (v8i64 VR512:$src)),<br class="">> + (VMOVDQU64Zmrk addr:$ptr, VK8WM:$mask, VR512:$src)>;<br class="">> +<br class="">> +// SKX replacement<br class="">> +def: Pat<(masked_store addr:$ptr, VK8WM:$mask, (v8i32 VR256:$src)),<br class="">> + (VMOVDQU32Z256mrk addr:$ptr, VK8WM:$mask, VR256:$src)>;<br class="">> +<br class="">> +// KNL replacement<br class="">> +def: Pat<(masked_store addr:$ptr, VK8WM:$mask, (v8i32 VR256:$src)),<br class="">> + (VMOVDQU32Zmrk addr:$ptr,<br class="">> + (v16i1 (COPY_TO_REGCLASS VK8WM:$mask, VK16WM)),<br class="">> + (INSERT_SUBREG (v16i32 (IMPLICIT_DEF)), VR256:$src, sub_ymm))>;<br class="">> +<br class="">> +def: Pat<(v8i32 (masked_load addr:$ptr, VK8WM:$mask, undef)),<br class="">> + (v8i32 (EXTRACT_SUBREG (v16i32 (VMOVDQU32Zrmkz<span class="apple-converted-space"> </span><br class="">> + (v16i1 (COPY_TO_REGCLASS VK8WM:$mask, VK16WM)), addr:$ptr)), sub_ymm))>;<br class="">> +<br class="">> +<br class="">> // Move Int Doubleword to Packed Double Int<br class="">> //<br class="">> def VMOVDI2PDIZrr : AVX512BI<0x6E, MRMSrcReg, (outs VR128X:$dst), (ins GR32:$src),<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/lib/Target/X86/X86InstrSSE.td<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86InstrSSE.td?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86InstrSSE.td?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/lib/Target/X86/X86InstrSSE.td (original)<br class="">> +++ llvm/trunk/lib/Target/X86/X86InstrSSE.td Thu Dec 4 03:40:44 2014<br class="">> @@ -9260,6 +9260,61 @@ defm VPMASKMOVQ : avx2_pmovmask<"vpmaskm<br class="">> int_x86_avx2_maskstore_q,<br class="">> int_x86_avx2_maskstore_q_256>, VEX_W;<br class="">><span class="apple-converted-space"> </span><br class="">> +def: Pat<(masked_store addr:$ptr, (v8i32 VR256:$mask), (v8f32 VR256:$src)),<br class="">> + (VPMASKMOVDYmr addr:$ptr, VR256:$mask, VR256:$src)>;<br class="">> +<br class="">> +def: Pat<(masked_store addr:$ptr, (v8i32 VR256:$mask), (v8i32 VR256:$src)),<br class="">> + (VPMASKMOVDYmr addr:$ptr, VR256:$mask, VR256:$src)>;<br class="">> +<br class="">> +def: Pat<(v8f32 (masked_load addr:$ptr, (v8i32 VR256:$mask), undef)),<br class="">> + (VPMASKMOVDYrm VR256:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v8f32 (masked_load addr:$ptr, (v8i32 VR256:$mask),<br class="">> + (bc_v8f32 (v8i32 immAllZerosV)))),<br class="">> + (VPMASKMOVDYrm VR256:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v8f32 (masked_load addr:$ptr, (v8i32 VR256:$mask), (v8f32 VR256:$src0))),<br class="">> + (VBLENDVPSYrr VR256:$src0, (VPMASKMOVDYrm VR256:$mask, addr:$ptr),<br class="">> + VR256:$mask)>;<br class="">> +<br class="">> +def: Pat<(v8i32 (masked_load addr:$ptr, (v8i32 VR256:$mask), undef)),<br class="">> + (VPMASKMOVDYrm VR256:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v8i32 (masked_load addr:$ptr, (v8i32 VR256:$mask), (v8i32 immAllZerosV))),<br class="">> + (VPMASKMOVDYrm VR256:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v8i32 (masked_load addr:$ptr, (v8i32 VR256:$mask), (v8i32 VR256:$src0))),<br class="">> + (VBLENDVPSYrr VR256:$src0, (VPMASKMOVDYrm VR256:$mask, addr:$ptr),<br class="">> + VR256:$mask)>;<br class="">> +<br class="">> +def: Pat<(masked_store addr:$ptr, (v4i64 VR256:$mask), (v4f64 VR256:$src)),<br class="">> + (VPMASKMOVQYmr addr:$ptr, VR256:$mask, VR256:$src)>;<br class="">> +<br class="">> +def: Pat<(masked_store addr:$ptr, (v4i64 VR256:$mask), (v4i64 VR256:$src)),<br class="">> + (VPMASKMOVQYmr addr:$ptr, VR256:$mask, VR256:$src)>;<br class="">> +<br class="">> +def: Pat<(v4f64 (masked_load addr:$ptr, (v4i64 VR256:$mask), undef)),<br class="">> + (VPMASKMOVQYrm VR256:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v4f64 (masked_load addr:$ptr, (v4i64 VR256:$mask),<br class="">> + (v4f64 immAllZerosV))),<br class="">> + (VPMASKMOVQYrm VR256:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v4f64 (masked_load addr:$ptr, (v4i64 VR256:$mask), (v4f64 VR256:$src0))),<br class="">> + (VBLENDVPDYrr VR256:$src0, (VPMASKMOVQYrm VR256:$mask, addr:$ptr),<br class="">> + VR256:$mask)>;<br class="">> +<br class="">> +def: Pat<(v4i64 (masked_load addr:$ptr, (v4i64 VR256:$mask), undef)),<br class="">> + (VPMASKMOVQYrm VR256:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v4i64 (masked_load addr:$ptr, (v4i64 VR256:$mask),<br class="">> + (bc_v4i64 (v8i32 immAllZerosV)))),<br class="">> + (VPMASKMOVQYrm VR256:$mask, addr:$ptr)>;<br class="">> +<br class="">> +def: Pat<(v4i64 (masked_load addr:$ptr, (v4i64 VR256:$mask), (v4i64 VR256:$src0))),<br class="">> + (VBLENDVPDYrr VR256:$src0, (VPMASKMOVQYrm VR256:$mask, addr:$ptr),<br class="">> + VR256:$mask)>;<br class="">> +<br class="">><span class="apple-converted-space"> </span><br class="">> //===----------------------------------------------------------------------===//<br class="">> // Variable Bit Shifts<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp (original)<br class="">> +++ llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp Thu Dec 4 03:40:44 2014<br class="">> @@ -111,6 +111,8 @@ public:<br class="">> Type *Ty) const override;<br class="">> unsigned getIntImmCost(Intrinsic::ID IID, unsigned Idx, const APInt &Imm,<br class="">> Type *Ty) const override;<br class="">> + bool isLegalPredicatedLoad (Type *DataType, int Consecutive) const override;<br class="">> + bool isLegalPredicatedStore(Type *DataType, int Consecutive) const override;<br class="">><span class="apple-converted-space"> </span><br class="">> /// @}<br class="">> };<br class="">> @@ -1156,3 +1158,19 @@ unsigned X86TTI::getIntImmCost(Intrinsic<br class="">> }<br class="">> return X86TTI::getIntImmCost(Imm, Ty);<br class="">> }<br class="">> +<br class="">> +bool X86TTI::isLegalPredicatedLoad(Type *DataType, int Consecutive) const {<br class="">> + int ScalarWidth = DataType->getScalarSizeInBits();<br class="">> + <span class="apple-converted-space"> </span><br class="">> + // Todo: AVX512 allows gather/scatter, works with strided and random as well<br class="">> + if ((ScalarWidth < 32) || (Consecutive == 0))<br class="">> + return false;<br class="">> + if (ST->hasAVX512() || ST->hasAVX2())<span class="apple-converted-space"> </span><br class="">> + return true;<br class="">> + return false;<br class="">> +}<br class="">> +<br class="">> +bool X86TTI::isLegalPredicatedStore(Type *DataType, int Consecutive) const {<br class="">> + return isLegalPredicatedLoad(DataType, Consecutive);<br class="">> +}<br class="">> +<br class="">><span class="apple-converted-space"> </span><br class="">> Added: llvm/trunk/test/CodeGen/X86/masked_memop.ll<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/CodeGen/X86/masked_memop.ll?rev=223348&view=auto" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/CodeGen/X86/masked_memop.ll?rev=223348&view=auto</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/test/CodeGen/X86/masked_memop.ll (added)<br class="">> +++ llvm/trunk/test/CodeGen/X86/masked_memop.ll Thu Dec 4 03:40:44 2014<br class="">> @@ -0,0 +1,73 @@<br class="">> +; RUN: llc -mtriple=x86_64-apple-darwin -mcpu=knl < %s | FileCheck %s -check-prefix=AVX512<br class="">> +; RUN: llc -mtriple=x86_64-apple-darwin -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2<br class="">> +<br class="">> +; AVX512-LABEL: test1<br class="">> +; AVX512: vmovdqu32 (%rdi), %zmm0 {%k1} {z}<br class="">> +<br class="">> +; AVX2-LABEL: test1<br class="">> +; AVX2: vpmaskmovd 32(%rdi)<br class="">> +; AVX2: vpmaskmovd (%rdi)<br class="">> +; AVX2-NOT: blend<br class="">> +<br class="">> +define <16 x i32> @test1(<16 x i32> %trigger, i8* %addr) {<br class="">> + %mask = icmp eq <16 x i32> %trigger, zeroinitializer<br class="">> + %res = call <16 x i32> @llvm.masked.load.v16i32(i8* %addr, <16 x i32>undef, i32 4, <16 x i1>%mask)<br class="">> + ret <16 x i32> %res<br class="">> +}<br class="">> +<br class="">> +; AVX512-LABEL: test2<br class="">> +; AVX512: vmovdqu32 (%rdi), %zmm0 {%k1} {z}<br class="">> +<br class="">> +; AVX2-LABEL: test2<br class="">> +; AVX2: vpmaskmovd {{.*}}(%rdi)<br class="">> +; AVX2: vpmaskmovd {{.*}}(%rdi)<br class="">> +; AVX2-NOT: blend<br class="">> +define <16 x i32> @test2(<16 x i32> %trigger, i8* %addr) {<br class="">> + %mask = icmp eq <16 x i32> %trigger, zeroinitializer<br class="">> + %res = call <16 x i32> @llvm.masked.load.v16i32(i8* %addr, <16 x i32>zeroinitializer, i32 4, <16 x i1>%mask)<br class="">> + ret <16 x i32> %res<br class="">> +}<br class="">> +<br class="">> +; AVX512-LABEL: test3<br class="">> +; AVX512: vmovdqu32 %zmm1, (%rdi) {%k1}<br class="">> +<br class="">> +define void @test3(<16 x i32> %trigger, i8* %addr, <16 x i32> %val) {<br class="">> + %mask = icmp eq <16 x i32> %trigger, zeroinitializer<br class="">> + call void @llvm.masked.store.v16i32(i8* %addr, <16 x i32>%val, i32 4, <16 x i1>%mask)<br class="">> + ret void<br class="">> +}<br class="">> +<br class="">> +; AVX512-LABEL: test4<br class="">> +; AVX512: vmovups (%rdi), %zmm{{.*{%k[1-7]}}}<br class="">> +<br class="">> +; AVX2-LABEL: test4<br class="">> +; AVX2: vpmaskmovd {{.*}}(%rdi)<br class="">> +; AVX2: vpmaskmovd {{.*}}(%rdi)<br class="">> +; AVX2: blend<br class="">> +define <16 x float> @test4(<16 x i32> %trigger, i8* %addr, <16 x float> %dst) {<br class="">> + %mask = icmp eq <16 x i32> %trigger, zeroinitializer<br class="">> + %res = call <16 x float> @llvm.masked.load.v16f32(i8* %addr, <16 x float>%dst, i32 4, <16 x i1>%mask)<br class="">> + ret <16 x float> %res<br class="">> +}<br class="">> +<br class="">> +; AVX512-LABEL: test5<br class="">> +; AVX512: vmovupd (%rdi), %zmm1 {%k1}<br class="">> +<br class="">> +; AVX2-LABEL: test5<br class="">> +; AVX2: vpmaskmovq<br class="">> +; AVX2: vblendvpd<br class="">> +; AVX2: vpmaskmovq <span class="apple-converted-space"> </span><br class="">> +; AVX2: vblendvpd<br class="">> +define <8 x double> @test5(<8 x i32> %trigger, i8* %addr, <8 x double> %dst) {<br class="">> + %mask = icmp eq <8 x i32> %trigger, zeroinitializer<br class="">> + %res = call <8 x double> @llvm.masked.load.v8f64(i8* %addr, <8 x double>%dst, i32 4, <8 x i1>%mask)<br class="">> + ret <8 x double> %res<br class="">> +}<br class="">> +<br class="">> +declare <16 x i32> @llvm.masked.load.v16i32(i8*, <16 x i32>, i32, <16 x i1>)<span class="apple-converted-space"> </span><br class="">> +declare void @llvm.masked.store.v16i32(i8*, <16 x i32>, i32, <16 x i1>)<span class="apple-converted-space"> </span><br class="">> +declare <16 x float> @llvm.masked.load.v16f32(i8*, <16 x float>, i32, <16 x i1>)<span class="apple-converted-space"> </span><br class="">> +declare void @llvm.masked.store.v16f32(i8*, <16 x float>, i32, <16 x i1>)<span class="apple-converted-space"> </span><br class="">> +declare <8 x double> @llvm.masked.load.v8f64(i8*, <8 x double>, i32, <8 x i1>)<span class="apple-converted-space"> </span><br class="">> +declare void @llvm.masked.store.v8f64(i8*, <8 x double>, i32, <8 x i1>)<span class="apple-converted-space"> </span><br class="">> +<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/utils/TableGen/CodeGenTarget.cpp<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/TableGen/CodeGenTarget.cpp?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/TableGen/CodeGenTarget.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/utils/TableGen/CodeGenTarget.cpp (original)<br class="">> +++ llvm/trunk/utils/TableGen/CodeGenTarget.cpp Thu Dec 4 03:40:44 2014<br class="">> @@ -534,7 +534,8 @@ CodeGenIntrinsic::CodeGenIntrinsic(Recor<br class="">> // variants with iAny types; otherwise, if the intrinsic is not<br class="">> // overloaded, all the types can be specified directly.<br class="">> assert(((!TyEl->isSubClassOf("LLVMExtendedType") &&<br class="">> - !TyEl->isSubClassOf("LLVMTruncatedType")) ||<br class="">> + !TyEl->isSubClassOf("LLVMTruncatedType") &&<br class="">> + !TyEl->isSubClassOf("LLVMVectorSameWidth")) ||<br class="">> VT == MVT::iAny || VT == MVT::vAny) &&<br class="">> "Expected iAny or vAny type");<br class="">> } else<br class="">><span class="apple-converted-space"> </span><br class="">> Modified: llvm/trunk/utils/TableGen/IntrinsicEmitter.cpp<br class="">> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/TableGen/IntrinsicEmitter.cpp?rev=223348&r1=223347&r2=223348&view=diff" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/TableGen/IntrinsicEmitter.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br class="">> ==============================================================================<br class="">> --- llvm/trunk/utils/TableGen/IntrinsicEmitter.cpp (original)<br class="">> +++ llvm/trunk/utils/TableGen/IntrinsicEmitter.cpp Thu Dec 4 03:40:44 2014<br class="">> @@ -257,7 +257,8 @@ enum IIT_Info {<br class="">> IIT_ANYPTR = 26,<br class="">> IIT_V1 = 27,<br class="">> IIT_VARARG = 28,<br class="">> - IIT_HALF_VEC_ARG = 29<br class="">> + IIT_HALF_VEC_ARG = 29,<br class="">> + IIT_SAME_VEC_WIDTH_ARG = 30<br class="">> };<br class="">><span class="apple-converted-space"> </span><br class="">><span class="apple-converted-space"> </span><br class="">> @@ -305,6 +306,13 @@ static void EncodeFixedType(Record *R, s<br class="">> Sig.push_back(IIT_TRUNC_ARG);<br class="">> else if (R->isSubClassOf("LLVMHalfElementsVectorType"))<br class="">> Sig.push_back(IIT_HALF_VEC_ARG);<br class="">> + else if (R->isSubClassOf("LLVMVectorSameWidth")) {<br class="">> + Sig.push_back(IIT_SAME_VEC_WIDTH_ARG);<br class="">> + Sig.push_back((Number << 2) | ArgCodes[Number]);<br class="">> + MVT::SimpleValueType VT = getValueType(R->getValueAsDef("ElTy"));<br class="">> + EncodeFixedValueType(VT, Sig);<br class="">> + return;<br class="">> + }<br class="">> else<br class="">> Sig.push_back(IIT_ARG);<br class="">> return Sig.push_back((Number << 2) | ArgCodes[Number]);<br class="">><span class="apple-converted-space"> </span><br class="">><span class="apple-converted-space"> </span><br class="">> _______________________________________________<br class="">> llvm-commits mailing list<br class="">><span class="apple-converted-space"> </span><a href="mailto:llvm-commits@cs.uiuc.edu" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">llvm-commits@cs.uiuc.edu</span></a><br class="">><span class="apple-converted-space"> </span><a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</span></a></span><o:p class=""></o:p></p></div></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; orphans: auto; text-align: start; widows: auto; -webkit-text-stroke-width: 0px; word-spacing: 0px;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">---------------------------------------------------------------------<br class="">Intel Israel (74) Limited<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; orphans: auto; text-align: start; widows: auto; -webkit-text-stroke-width: 0px; word-spacing: 0px;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">This e-mail and any attachments may contain confidential material for<br class="">the sole use of the intended recipient(s). Any review or distribution<br class="">by others is strictly prohibited. If you are not the intended<br class="">recipient, please contact the sender and delete all copies.<o:p class=""></o:p></span></div></div></blockquote></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div></div><p style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">---------------------------------------------------------------------<br class="">Intel Israel (74) Limited</p><p style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">This e-mail and any attachments may contain confidential material for<br class="">the sole use of the intended recipient(s). Any review or distribution<br class="">by others is strictly prohibited. If you are not the intended<br class="">recipient, please contact the sender and delete all copies.</p></div></blockquote></div><br class=""></div></body></html>