<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:799957587;
mso-list-type:hybrid;
mso-list-template-ids:-1214880770 44585902 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I fixed the bug in 225441.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="font-family:"Calibri","sans-serif";color:#31849B"><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span dir="LTR"></span><b><i><span style="color:#31849B"> Elena<o:p></o:p></span></i></b></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Steven Wu [mailto:stevenwu@apple.com]
<br>
<b>Sent:</b> Wednesday, January 07, 2015 19:48<br>
<b>To:</b> Demikhovsky, Elena<br>
<b>Cc:</b> llvm-commits@cs.uiuc.edu<br>
<b>Subject:</b> Re: [llvm] r223348 - Masked Load / Store Intrinsics - the CodeGen part.<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks in advance.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Steven<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Jan 7, 2015, at 2:56 AM, Demikhovsky, Elena <<a href="mailto:elena.demikhovsky@intel.com">elena.demikhovsky@intel.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Steven,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I can reproduce the failure. I’ll fix it by tomorrow.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<div style="margin-left:36.0pt">
<p class="MsoNormal" style="text-indent:-18.0pt"><span style="font-family:"Calibri","sans-serif";color:#31849B">-</span><span style="font-size:7.0pt;color:#31849B"> <span class="apple-converted-space"> </span></span><b><i><span style="color:#31849B"> Elena</span></i></b><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span class="apple-converted-space"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span></span><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">Steven
Wu [<a href="mailto:stevenwu@apple.com">mailto:stevenwu@apple.com</a>]<span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Wednesday, January 07, 2015 03:05<br>
<b>To:</b><span class="apple-converted-space"> </span>Demikhovsky, Elena<br>
<b>Cc:</b><span class="apple-converted-space"> </span><a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [llvm] r223348 - Masked Load / Store Intrinsics - the CodeGen part.</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt">Hi Elena<br>
<br>
I saw the masked load/store intrinsics broke one of our bots. It causes assertion failure in LegalizeIntegerTypes. I attached a reduced test case in the email.<br>
You can reproduce with:<br>
llc -mtriple x86_64-apple-macosx10.10.0 -O0 bugpoint-reduced-simplified.ll<br>
<br>
Let me know if you want a PR for it or you need more information.<br>
<br>
Thanks<br>
<br>
Steven</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.0pt"><br>
<br>
> On Dec 4, 2014, at 1:40 AM, Elena Demikhovsky <<a href="mailto:elena.demikhovsky@intel.com"><span style="color:purple">elena.demikhovsky@intel.com</span></a>> wrote:<br>
><span class="apple-converted-space"> </span><br>
> Author: delena<br>
> Date: Thu Dec 4 03:40:44 2014<br>
> New Revision: 223348<br>
><span class="apple-converted-space"> </span><br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project?rev=223348&view=rev"><span style="color:purple">http://llvm.org/viewvc/llvm-project?rev=223348&view=rev</span></a><br>
> Log:<br>
> Masked Load / Store Intrinsics - the CodeGen part.<br>
> I'm recommiting the codegen part of the patch.<br>
> The vectorizer part will be send to review again.<br>
><span class="apple-converted-space"> </span><br>
> Masked Vector Load and Store Intrinsics.<br>
> Introduced new target-independent intrinsics in order to support masked vector loads and stores. The loop vectorizer optimizes loops containing conditional memory accesses by generating these intrinsics for existing targets AVX2 and AVX-512. The vectorizer
asks the target about availability of masked vector loads and stores.<br>
> Added SDNodes for masked operations and lowering patterns for X86 code generator.<br>
> Examples:<br>
> <16 x i32> @llvm.masked.load.v16i32(i8* %addr, <16 x i32> %passthru, i32 4 /* align */, <16 x i1> %mask)<br>
> declare void @llvm.masked.store.v8f64(i8* %addr, <8 x double> %value, i32 4, <8 x i1> %mask)<br>
><span class="apple-converted-space"> </span><br>
> Scalarizer for other targets (not AVX2/AVX-512) will be done in a separate patch.<br>
><span class="apple-converted-space"> </span><br>
><span class="apple-converted-space"> </span><a href="http://reviews.llvm.org/D6191"><span style="color:purple">http://reviews.llvm.org/D6191</span></a><br>
><span class="apple-converted-space"> </span><br>
><span class="apple-converted-space"> </span><br>
> Added:<br>
> llvm/trunk/test/CodeGen/X86/masked_memop.ll<br>
> Modified:<br>
> llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h<br>
> llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h<br>
> llvm/trunk/include/llvm/CodeGen/SelectionDAG.h<br>
> llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h<br>
> llvm/trunk/include/llvm/IR/IRBuilder.h<br>
> llvm/trunk/include/llvm/IR/Intrinsics.h<br>
> llvm/trunk/include/llvm/IR/Intrinsics.td<br>
> llvm/trunk/include/llvm/Target/TargetSelectionDAG.td<br>
> llvm/trunk/lib/Analysis/TargetTransformInfo.cpp<br>
> llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp<br>
> llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp<br>
> llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeTypes.h<br>
> llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp<br>
> llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp<br>
> llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp<br>
> llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h<br>
> llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp<br>
> llvm/trunk/lib/IR/Function.cpp<br>
> llvm/trunk/lib/IR/IRBuilder.cpp<br>
> llvm/trunk/lib/IR/Verifier.cpp<br>
> llvm/trunk/lib/Target/X86/X86ISelLowering.cpp<br>
> llvm/trunk/lib/Target/X86/X86InstrAVX512.td<br>
> llvm/trunk/lib/Target/X86/X86InstrSSE.td<br>
> llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp<br>
> llvm/trunk/utils/TableGen/CodeGenTarget.cpp<br>
> llvm/trunk/utils/TableGen/IntrinsicEmitter.cpp<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h (original)<br>
> +++ llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h Thu Dec 4 03:40:44 2014<br>
> @@ -270,6 +270,13 @@ public:<br>
> int64_t BaseOffset, bool HasBaseReg,<br>
> int64_t Scale) const;<br>
><span class="apple-converted-space"> </span><br>
> + /// \brief Return true if the target works with masked instruction<br>
> + /// AVX2 allows masks for consecutive load and store for i32 and i64 elements.<br>
> + /// AVX-512 architecture will also allow masks for non-consecutive memory<br>
> + /// accesses.<br>
> + virtual bool isLegalPredicatedStore(Type *DataType, int Consecutive) const;<br>
> + virtual bool isLegalPredicatedLoad (Type *DataType, int Consecutive) const;<br>
> +<br>
> /// \brief Return the cost of the scaling factor used in the addressing<br>
> /// mode represented by AM for this target, for a load/store<br>
> /// of the specified type.<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h (original)<br>
> +++ llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h Thu Dec 4 03:40:44 2014<br>
> @@ -675,6 +675,9 @@ namespace ISD {<br>
> ATOMIC_LOAD_UMIN,<br>
> ATOMIC_LOAD_UMAX,<br>
><span class="apple-converted-space"> </span><br>
> + // Masked load and store<br>
> + MLOAD, MSTORE,<br>
> +<br>
> /// This corresponds to the llvm.lifetime.* intrinsics. The first operand<br>
> /// is the chain and the second operand is the alloca pointer.<br>
> LIFETIME_START, LIFETIME_END,<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/include/llvm/CodeGen/SelectionDAG.h<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/SelectionDAG.h?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/SelectionDAG.h?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/include/llvm/CodeGen/SelectionDAG.h (original)<br>
> +++ llvm/trunk/include/llvm/CodeGen/SelectionDAG.h Thu Dec 4 03:40:44 2014<br>
> @@ -866,6 +866,10 @@ public:<br>
> SDValue getIndexedStore(SDValue OrigStoe, SDLoc dl, SDValue Base,<br>
> SDValue Offset, ISD::MemIndexedMode AM);<br>
><span class="apple-converted-space"> </span><br>
> + SDValue getMaskedLoad(EVT VT, SDLoc dl, SDValue Chain, SDValue Ptr,<br>
> + SDValue Mask, SDValue Src0, MachineMemOperand *MMO);<br>
> + SDValue getMaskedStore(SDValue Chain, SDLoc dl, SDValue Val,<br>
> + SDValue Ptr, SDValue Mask, MachineMemOperand *MMO);<br>
> /// getSrcValue - Construct a node to track a Value* through the backend.<br>
> SDValue getSrcValue(const Value *v);<br>
><span class="apple-converted-space"> </span><br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h (original)<br>
> +++ llvm/trunk/include/llvm/CodeGen/SelectionDAGNodes.h Thu Dec 4 03:40:44 2014<br>
> @@ -1177,6 +1177,8 @@ public:<br>
> N->getOpcode() == ISD::ATOMIC_LOAD_UMAX ||<br>
> N->getOpcode() == ISD::ATOMIC_LOAD ||<br>
> N->getOpcode() == ISD::ATOMIC_STORE ||<br>
> + N->getOpcode() == ISD::MLOAD ||<br>
> + N->getOpcode() == ISD::MSTORE ||<br>
> N->isMemIntrinsic() ||<br>
> N->isTargetMemoryOpcode();<br>
> }<br>
> @@ -1926,6 +1928,72 @@ public:<br>
> }<br>
> };<br>
><span class="apple-converted-space"> </span><br>
> +/// MaskedLoadStoreSDNode - This is a base class is used to represent MLOAD and<br>
> +/// MSTORE nodes<br>
> +///<br>
> +class MaskedLoadStoreSDNode : public MemSDNode {<br>
> + // Operands<br>
> + SDUse Ops[4];<br>
> +public:<br>
> + friend class SelectionDAG;<br>
> + MaskedLoadStoreSDNode(ISD::NodeType NodeTy, unsigned Order, DebugLoc dl,<br>
> + SDValue *Operands, unsigned numOperands,<span class="apple-converted-space"> </span><br>
> + SDVTList VTs, EVT MemVT, MachineMemOperand *MMO)<br>
> + : MemSDNode(NodeTy, Order, dl, VTs, MemVT, MMO) {<br>
> + InitOperands(Ops, Operands, numOperands);<br>
> + }<br>
> +<br>
> + // In the both nodes address is Op1, mask is Op2:<br>
> + // MaskedLoadSDNode (Chain, ptr, mask, src0), src0 is a passthru value<br>
> + // MaskedStoreSDNode (Chain, ptr, mask, data)<br>
> + // Mask is a vector of i1 elements<br>
> + const SDValue &getBasePtr() const { return getOperand(1); }<br>
> + const SDValue &getMask() const { return getOperand(2); }<br>
> +<br>
> + static bool classof(const SDNode *N) {<br>
> + return N->getOpcode() == ISD::MLOAD ||<br>
> + N->getOpcode() == ISD::MSTORE;<br>
> + }<br>
> +};<br>
> +<br>
> +/// MaskedLoadSDNode - This class is used to represent an MLOAD node<br>
> +///<br>
> +class MaskedLoadSDNode : public MaskedLoadStoreSDNode {<br>
> +public:<br>
> + friend class SelectionDAG;<br>
> + MaskedLoadSDNode(unsigned Order, DebugLoc dl,<br>
> + SDValue *Operands, unsigned numOperands,<span class="apple-converted-space"> </span><br>
> + SDVTList VTs, EVT MemVT, MachineMemOperand *MMO)<br>
> + : MaskedLoadStoreSDNode(ISD::MLOAD, Order, dl, Operands, numOperands,<br>
> + VTs, MemVT, MMO)<span class="apple-converted-space"> </span><br>
> + {}<br>
> +<br>
> + const SDValue &getSrc0() const { return getOperand(3); }<br>
> + static bool classof(const SDNode *N) {<br>
> + return N->getOpcode() == ISD::MLOAD;<br>
> + }<br>
> +};<br>
> +<br>
> +/// MaskedStoreSDNode - This class is used to represent an MSTORE node<br>
> +///<br>
> +class MaskedStoreSDNode : public MaskedLoadStoreSDNode {<br>
> +<br>
> +public:<br>
> + friend class SelectionDAG;<br>
> + MaskedStoreSDNode(unsigned Order, DebugLoc dl,<br>
> + SDValue *Operands, unsigned numOperands,<span class="apple-converted-space"> </span><br>
> + SDVTList VTs, EVT MemVT, MachineMemOperand *MMO)<br>
> + : MaskedLoadStoreSDNode(ISD::MSTORE, Order, dl, Operands, numOperands,<br>
> + VTs, MemVT, MMO)<span class="apple-converted-space"> </span><br>
> + {}<br>
> +<br>
> + const SDValue &getData() const { return getOperand(3); }<br>
> +<br>
> + static bool classof(const SDNode *N) {<br>
> + return N->getOpcode() == ISD::MSTORE;<br>
> + }<br>
> +};<br>
> +<br>
> /// MachineSDNode - An SDNode that represents everything that will be needed<br>
> /// to construct a MachineInstr. These nodes are created during the<br>
> /// instruction selection proper phase.<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/include/llvm/IR/IRBuilder.h<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/IRBuilder.h?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/IRBuilder.h?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/include/llvm/IR/IRBuilder.h (original)<br>
> +++ llvm/trunk/include/llvm/IR/IRBuilder.h Thu Dec 4 03:40:44 2014<br>
> @@ -429,11 +429,22 @@ public:<br>
> /// If the pointer isn't i8* it will be converted.<br>
> CallInst *CreateLifetimeEnd(Value *Ptr, ConstantInt *Size = nullptr);<br>
><span class="apple-converted-space"> </span><br>
> + /// \brief Create a call to Masked Load intrinsic<br>
> + CallInst *CreateMaskedLoad(ArrayRef<Value *> Ops);<br>
> +<br>
> + /// \brief Create a call to Masked Store intrinsic<br>
> + CallInst *CreateMaskedStore(ArrayRef<Value *> Ops);<br>
> +<br>
> /// \brief Create an assume intrinsic call that allows the optimizer to<br>
> /// assume that the provided condition will be true.<br>
> CallInst *CreateAssumption(Value *Cond);<br>
><span class="apple-converted-space"> </span><br>
> private:<br>
> + /// \brief Create a call to a masked intrinsic with given Id.<br>
> + /// Masked intrinsic has only one overloaded type - data type.<br>
> + CallInst *CreateMaskedIntrinsic(unsigned Id, ArrayRef<Value *> Ops,<br>
> + Type *DataTy);<br>
> +<br>
> Value *getCastedInt8PtrValue(Value *Ptr);<br>
> };<br>
><span class="apple-converted-space"> </span><br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/include/llvm/IR/Intrinsics.h<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/Intrinsics.h?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/Intrinsics.h?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/include/llvm/IR/Intrinsics.h (original)<br>
> +++ llvm/trunk/include/llvm/IR/Intrinsics.h Thu Dec 4 03:40:44 2014<br>
> @@ -76,7 +76,8 @@ namespace Intrinsic {<br>
> enum IITDescriptorKind {<br>
> Void, VarArg, MMX, Metadata, Half, Float, Double,<br>
> Integer, Vector, Pointer, Struct,<br>
> - Argument, ExtendArgument, TruncArgument, HalfVecArgument<br>
> + Argument, ExtendArgument, TruncArgument, HalfVecArgument,<br>
> + SameVecWidthArgument<br>
> } Kind;<br>
><span class="apple-converted-space"> </span><br>
> union {<br>
> @@ -96,13 +97,15 @@ namespace Intrinsic {<br>
> };<br>
> unsigned getArgumentNumber() const {<br>
> assert(Kind == Argument || Kind == ExtendArgument ||<br>
> - Kind == TruncArgument || Kind == HalfVecArgument);<br>
> + Kind == TruncArgument || Kind == HalfVecArgument ||<br>
> + Kind == SameVecWidthArgument);<br>
> return Argument_Info >> 2;<br>
> }<br>
> ArgKind getArgumentKind() const {<br>
> assert(Kind == Argument || Kind == ExtendArgument ||<br>
> - Kind == TruncArgument || Kind == HalfVecArgument);<br>
> - return (ArgKind)(Argument_Info&3);<br>
> + Kind == TruncArgument || Kind == HalfVecArgument ||<br>
> + Kind == SameVecWidthArgument);<br>
> + return (ArgKind)(Argument_Info & 3);<br>
> }<br>
><span class="apple-converted-space"> </span><br>
> static IITDescriptor get(IITDescriptorKind K, unsigned Field) {<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/include/llvm/IR/Intrinsics.td<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/Intrinsics.td?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/Intrinsics.td?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/include/llvm/IR/Intrinsics.td (original)<br>
> +++ llvm/trunk/include/llvm/IR/Intrinsics.td Thu Dec 4 03:40:44 2014<br>
> @@ -112,6 +112,10 @@ class LLVMMatchType<int num><br>
> // the intrinsic is overloaded, so the matched type should be declared as iAny.<br>
> class LLVMExtendedType<int num> : LLVMMatchType<num>;<br>
> class LLVMTruncatedType<int num> : LLVMMatchType<num>;<br>
> +class LLVMVectorSameWidth<int num, LLVMType elty><br>
> + : LLVMMatchType<num> {<br>
> + ValueType ElTy = elty.VT;<br>
> +}<br>
><span class="apple-converted-space"> </span><br>
> // Match the type of another intrinsic parameter that is expected to be a<br>
> // vector type, but change the element count to be half as many<br>
> @@ -555,6 +559,17 @@ def int_convertuu : Intrinsic<[llvm_any<br>
> def int_clear_cache : Intrinsic<[], [llvm_ptr_ty, llvm_ptr_ty],<br>
> [], "llvm.clear_cache">;<br>
><span class="apple-converted-space"> </span><br>
> +//===-------------------------- Masked Intrinsics -------------------------===//<br>
> +//<br>
> +def int_masked_store : Intrinsic<[], [llvm_ptr_ty, llvm_anyvector_ty,<br>
> + llvm_i32_ty,<br>
> + LLVMVectorSameWidth<0, llvm_i1_ty>],<br>
> + [IntrReadWriteArgMem]>;<br>
> +<br>
> +def int_masked_load : Intrinsic<[llvm_anyvector_ty],<br>
> + [llvm_ptr_ty, LLVMMatchType<0>, llvm_i32_ty,<br>
> + LLVMVectorSameWidth<0, llvm_i1_ty>],<br>
> + [IntrReadArgMem]>;<br>
> //===----------------------------------------------------------------------===//<br>
> // Target-specific intrinsics<br>
> //===----------------------------------------------------------------------===//<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/include/llvm/Target/TargetSelectionDAG.td<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Target/TargetSelectionDAG.td?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Target/TargetSelectionDAG.td?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/include/llvm/Target/TargetSelectionDAG.td (original)<br>
> +++ llvm/trunk/include/llvm/Target/TargetSelectionDAG.td Thu Dec 4 03:40:44 2014<br>
> @@ -188,6 +188,14 @@ def SDTIStore : SDTypeProfile<1, 3, [<br>
> SDTCisSameAs<0, 2>, SDTCisPtrTy<0>, SDTCisPtrTy<3><br>
> ]>;<br>
><span class="apple-converted-space"> </span><br>
> +def SDTMaskedStore: SDTypeProfile<0, 3, [ // masked store<br>
> + SDTCisPtrTy<0>, SDTCisVec<1>, SDTCisVec<2><br>
> +]>;<br>
> +<br>
> +def SDTMaskedLoad: SDTypeProfile<1, 3, [ // masked load<br>
> + SDTCisVec<0>, SDTCisPtrTy<1>, SDTCisVec<2>, SDTCisSameAs<0, 3><br>
> +]>;<br>
> +<br>
> def SDTVecShuffle : SDTypeProfile<1, 2, [<br>
> SDTCisSameAs<0, 1>, SDTCisSameAs<1, 2><br>
> ]>;<br>
> @@ -454,6 +462,11 @@ def atomic_load : SDNode<"ISD::ATOM<br>
> def atomic_store : SDNode<"ISD::ATOMIC_STORE", SDTAtomicStore,<br>
> [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;<br>
><span class="apple-converted-space"> </span><br>
> +def masked_store : SDNode<"ISD::MSTORE", SDTMaskedStore,<br>
> + [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;<br>
> +def masked_load : SDNode<"ISD::MLOAD", SDTMaskedLoad,<br>
> + [SDNPHasChain, SDNPMayLoad, SDNPMemOperand]>;<br>
> +<br>
> // Do not use ld, st directly. Use load, extload, sextload, zextload, store,<br>
> // and truncst (see below).<br>
> def ld : SDNode<"ISD::LOAD" , SDTLoad,<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/lib/Analysis/TargetTransformInfo.cpp<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Analysis/TargetTransformInfo.cpp?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Analysis/TargetTransformInfo.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/Analysis/TargetTransformInfo.cpp (original)<br>
> +++ llvm/trunk/lib/Analysis/TargetTransformInfo.cpp Thu Dec 4 03:40:44 2014<br>
> @@ -101,6 +101,17 @@ bool TargetTransformInfo::isLegalICmpImm<br>
> return PrevTTI->isLegalICmpImmediate(Imm);<br>
> }<br>
><span class="apple-converted-space"> </span><br>
> +bool TargetTransformInfo::isLegalPredicatedLoad(Type *DataType,<br>
> + int Consecutive) const {<br>
> + return false;<br>
> +}<br>
> +<br>
> +bool TargetTransformInfo::isLegalPredicatedStore(Type *DataType,<br>
> + int Consecutive) const {<br>
> + return false;<br>
> +}<br>
> +<br>
> +<br>
> bool TargetTransformInfo::isLegalAddressingMode(Type *Ty, GlobalValue *BaseGV,<br>
> int64_t BaseOffset,<br>
> bool HasBaseReg,<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp (original)<br>
> +++ llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp Thu Dec 4 03:40:44 2014<br>
> @@ -303,6 +303,8 @@ namespace {<br>
> SDValue visitEXTRACT_SUBVECTOR(SDNode *N);<br>
> SDValue visitVECTOR_SHUFFLE(SDNode *N);<br>
> SDValue visitINSERT_SUBVECTOR(SDNode *N);<br>
> + SDValue visitMLOAD(SDNode *N);<br>
> + SDValue visitMSTORE(SDNode *N);<br>
><span class="apple-converted-space"> </span><br>
> SDValue XformToShuffleWithZero(SDNode *N);<br>
> SDValue ReassociateOps(unsigned Opc, SDLoc DL, SDValue LHS, SDValue RHS);<br>
> @@ -1351,6 +1353,8 @@ SDValue DAGCombiner::visit(SDNode *N) {<br>
> case ISD::EXTRACT_SUBVECTOR: return visitEXTRACT_SUBVECTOR(N);<br>
> case ISD::VECTOR_SHUFFLE: return visitVECTOR_SHUFFLE(N);<br>
> case ISD::INSERT_SUBVECTOR: return visitINSERT_SUBVECTOR(N);<br>
> + case ISD::MLOAD: return visitMLOAD(N);<br>
> + case ISD::MSTORE: return visitMSTORE(N);<br>
> }<br>
> return SDValue();<br>
> }<br>
> @@ -4771,6 +4775,162 @@ static SDValue ConvertSelectToConcatVect<br>
> TopHalf->isNullValue() ? RHS->getOperand(1) : LHS->getOperand(1));<br>
> }<br>
><span class="apple-converted-space"> </span><br>
> +SDValue DAGCombiner::visitMSTORE(SDNode *N) {<br>
> +<br>
> + if (Level >= AfterLegalizeTypes)<br>
> + return SDValue();<br>
> +<br>
> + MaskedStoreSDNode *MST = dyn_cast<MaskedStoreSDNode>(N);<br>
> + SDValue Mask = MST->getMask();<br>
> + SDValue Data = MST->getData();<br>
> + SDLoc DL(N);<br>
> +<br>
> + // If the MSTORE data type requires splitting and the mask is provided by a<br>
> + // SETCC, then split both nodes and its operands before legalization. This<br>
> + // prevents the type legalizer from unrolling SETCC into scalar comparisons<br>
> + // and enables future optimizations (e.g. min/max pattern matching on X86).<br>
> + if (Mask.getOpcode() == ISD::SETCC) {<br>
> +<br>
> + // Check if any splitting is required.<br>
> + if (TLI.getTypeAction(*DAG.getContext(), Data.getValueType()) !=<br>
> + TargetLowering::TypeSplitVector)<br>
> + return SDValue();<br>
> +<br>
> + SDValue MaskLo, MaskHi, Lo, Hi;<br>
> + std::tie(MaskLo, MaskHi) = SplitVSETCC(Mask.getNode(), DAG);<br>
> +<br>
> + EVT LoVT, HiVT;<br>
> + std::tie(LoVT, HiVT) = DAG.GetSplitDestVTs(MST->getValueType(0));<br>
> +<br>
> + SDValue Chain = MST->getChain();<br>
> + SDValue Ptr = MST->getBasePtr();<br>
> +<br>
> + EVT MemoryVT = MST->getMemoryVT();<br>
> + unsigned Alignment = MST->getOriginalAlignment();<br>
> +<br>
> + // if Alignment is equal to the vector size,<br>
> + // take the half of it for the second part<br>
> + unsigned SecondHalfAlignment =<br>
> + (Alignment == Data->getValueType(0).getSizeInBits()/8) ?<br>
> + Alignment/2 : Alignment;<br>
> +<br>
> + EVT LoMemVT, HiMemVT;<br>
> + std::tie(LoMemVT, HiMemVT) = DAG.GetSplitDestVTs(MemoryVT);<br>
> +<br>
> + SDValue DataLo, DataHi;<br>
> + std::tie(DataLo, DataHi) = DAG.SplitVector(Data, DL);<br>
> +<br>
> + MachineMemOperand *MMO = DAG.getMachineFunction().<br>
> + getMachineMemOperand(MST->getPointerInfo(),<span class="apple-converted-space"> </span><br>
> + MachineMemOperand::MOStore, LoMemVT.getStoreSize(),<br>
> + Alignment, MST->getAAInfo(), MST->getRanges());<br>
> +<br>
> + Lo = DAG.getMaskedStore(Chain, DL, DataLo, Ptr, MaskLo, MMO);<br>
> +<br>
> + unsigned IncrementSize = LoMemVT.getSizeInBits()/8;<br>
> + Ptr = DAG.getNode(ISD::ADD, DL, Ptr.getValueType(), Ptr,<br>
> + DAG.getConstant(IncrementSize, Ptr.getValueType()));<br>
> +<br>
> + MMO = DAG.getMachineFunction().<br>
> + getMachineMemOperand(MST->getPointerInfo(),<span class="apple-converted-space"> </span><br>
> + MachineMemOperand::MOStore, HiMemVT.getStoreSize(),<br>
> + SecondHalfAlignment, MST->getAAInfo(),<br>
> + MST->getRanges());<br>
> +<br>
> + Hi = DAG.getMaskedStore(Chain, DL, DataHi, Ptr, MaskHi, MMO);<br>
> +<br>
> + AddToWorklist(Lo.getNode());<br>
> + AddToWorklist(Hi.getNode());<br>
> +<br>
> + return DAG.getNode(ISD::TokenFactor, DL, MVT::Other, Lo, Hi);<br>
> + }<br>
> + return SDValue();<br>
> +}<br>
> +<br>
> +SDValue DAGCombiner::visitMLOAD(SDNode *N) {<br>
> +<br>
> + if (Level >= AfterLegalizeTypes)<br>
> + return SDValue();<br>
> +<br>
> + MaskedLoadSDNode *MLD = dyn_cast<MaskedLoadSDNode>(N);<br>
> + SDValue Mask = MLD->getMask();<br>
> + SDLoc DL(N);<br>
> +<br>
> + // If the MLOAD result requires splitting and the mask is provided by a<br>
> + // SETCC, then split both nodes and its operands before legalization. This<br>
> + // prevents the type legalizer from unrolling SETCC into scalar comparisons<br>
> + // and enables future optimizations (e.g. min/max pattern matching on X86).<br>
> +<br>
> + if (Mask.getOpcode() == ISD::SETCC) {<br>
> + EVT VT = N->getValueType(0);<br>
> +<br>
> + // Check if any splitting is required.<br>
> + if (TLI.getTypeAction(*DAG.getContext(), VT) !=<br>
> + TargetLowering::TypeSplitVector)<br>
> + return SDValue();<br>
> +<br>
> + SDValue MaskLo, MaskHi, Lo, Hi;<br>
> + std::tie(MaskLo, MaskHi) = SplitVSETCC(Mask.getNode(), DAG);<br>
> +<br>
> + SDValue Src0 = MLD->getSrc0();<br>
> + SDValue Src0Lo, Src0Hi;<br>
> + std::tie(Src0Lo, Src0Hi) = DAG.SplitVector(Src0, DL);<br>
> +<br>
> + EVT LoVT, HiVT;<br>
> + std::tie(LoVT, HiVT) = DAG.GetSplitDestVTs(MLD->getValueType(0));<br>
> +<br>
> + SDValue Chain = MLD->getChain();<br>
> + SDValue Ptr = MLD->getBasePtr();<br>
> + EVT MemoryVT = MLD->getMemoryVT();<br>
> + unsigned Alignment = MLD->getOriginalAlignment();<br>
> +<br>
> + // if Alignment is equal to the vector size,<br>
> + // take the half of it for the second part<br>
> + unsigned SecondHalfAlignment =<br>
> + (Alignment == MLD->getValueType(0).getSizeInBits()/8) ?<br>
> + Alignment/2 : Alignment;<br>
> +<br>
> + EVT LoMemVT, HiMemVT;<br>
> + std::tie(LoMemVT, HiMemVT) = DAG.GetSplitDestVTs(MemoryVT);<br>
> +<br>
> + MachineMemOperand *MMO = DAG.getMachineFunction().<br>
> + getMachineMemOperand(MLD->getPointerInfo(),<span class="apple-converted-space"> </span><br>
> + MachineMemOperand::MOLoad, LoMemVT.getStoreSize(),<br>
> + Alignment, MLD->getAAInfo(), MLD->getRanges());<br>
> +<br>
> + Lo = DAG.getMaskedLoad(LoVT, DL, Chain, Ptr, MaskLo, Src0Lo, MMO);<br>
> +<br>
> + unsigned IncrementSize = LoMemVT.getSizeInBits()/8;<br>
> + Ptr = DAG.getNode(ISD::ADD, DL, Ptr.getValueType(), Ptr,<br>
> + DAG.getConstant(IncrementSize, Ptr.getValueType()));<br>
> +<br>
> + MMO = DAG.getMachineFunction().<br>
> + getMachineMemOperand(MLD->getPointerInfo(),<span class="apple-converted-space"> </span><br>
> + MachineMemOperand::MOLoad, HiMemVT.getStoreSize(),<br>
> + SecondHalfAlignment, MLD->getAAInfo(), MLD->getRanges());<br>
> +<br>
> + Hi = DAG.getMaskedLoad(HiVT, DL, Chain, Ptr, MaskHi, Src0Hi, MMO);<br>
> +<br>
> + AddToWorklist(Lo.getNode());<br>
> + AddToWorklist(Hi.getNode());<br>
> +<br>
> + // Build a factor node to remember that this load is independent of the<br>
> + // other one.<br>
> + Chain = DAG.getNode(ISD::TokenFactor, DL, MVT::Other, Lo.getValue(1),<br>
> + Hi.getValue(1));<br>
> +<br>
> + // Legalized the chain result - switch anything that used the old chain to<br>
> + // use the new one.<br>
> + DAG.ReplaceAllUsesOfValueWith(SDValue(MLD, 1), Chain);<br>
> +<br>
> + SDValue LoadRes = DAG.getNode(ISD::CONCAT_VECTORS, DL, VT, Lo, Hi);<br>
> +<br>
> + SDValue RetOps[] = { LoadRes, Chain };<br>
> + return DAG.getMergeValues(RetOps, DL);<br>
> + }<br>
> + return SDValue();<br>
> +}<br>
> +<br>
> SDValue DAGCombiner::visitVSELECT(SDNode *N) {<br>
> SDValue N0 = N->getOperand(0);<br>
> SDValue N1 = N->getOperand(1);<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp (original)<br>
> +++ llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp Thu Dec 4 03:40:44 2014<br>
> @@ -825,6 +825,10 @@ bool DAGTypeLegalizer::PromoteIntegerOpe<br>
> case ISD::SINT_TO_FP: Res = PromoteIntOp_SINT_TO_FP(N); break;<br>
> case ISD::STORE: Res = PromoteIntOp_STORE(cast<StoreSDNode>(N),<br>
> OpNo); break;<br>
> + case ISD::MSTORE: Res = PromoteIntOp_MSTORE(cast<MaskedStoreSDNode>(N),<br>
> + OpNo); break;<br>
> + case ISD::MLOAD: Res = PromoteIntOp_MLOAD(cast<MaskedLoadSDNode>(N),<br>
> + OpNo); break;<br>
> case ISD::TRUNCATE: Res = PromoteIntOp_TRUNCATE(N); break;<br>
> case ISD::FP16_TO_FP:<br>
> case ISD::UINT_TO_FP: Res = PromoteIntOp_UINT_TO_FP(N); break;<br>
> @@ -1091,6 +1095,25 @@ SDValue DAGTypeLegalizer::PromoteIntOp_S<br>
> N->getMemoryVT(), N->getMemOperand());<br>
> }<br>
><span class="apple-converted-space"> </span><br>
> +SDValue DAGTypeLegalizer::PromoteIntOp_MSTORE(MaskedStoreSDNode *N, unsigned OpNo){<br>
> +<br>
> + assert(OpNo == 2 && "Only know how to promote the mask!");<br>
> + EVT DataVT = N->getOperand(3).getValueType();<br>
> + SDValue Mask = PromoteTargetBoolean(N->getOperand(OpNo), DataVT);<br>
> + SmallVector<SDValue, 4> NewOps(N->op_begin(), N->op_end());<br>
> + NewOps[OpNo] = Mask;<br>
> + return SDValue(DAG.UpdateNodeOperands(N, NewOps), 0);<br>
> +}<br>
> +<br>
> +SDValue DAGTypeLegalizer::PromoteIntOp_MLOAD(MaskedLoadSDNode *N, unsigned OpNo){<br>
> + assert(OpNo == 2 && "Only know how to promote the mask!");<br>
> + EVT DataVT = N->getValueType(0);<br>
> + SDValue Mask = PromoteTargetBoolean(N->getOperand(OpNo), DataVT);<br>
> + SmallVector<SDValue, 4> NewOps(N->op_begin(), N->op_end());<br>
> + NewOps[OpNo] = Mask;<br>
> + return SDValue(DAG.UpdateNodeOperands(N, NewOps), 0);<br>
> +}<br>
> +<br>
> SDValue DAGTypeLegalizer::PromoteIntOp_TRUNCATE(SDNode *N) {<br>
> SDValue Op = GetPromotedInteger(N->getOperand(0));<br>
> return DAG.getNode(ISD::TRUNCATE, SDLoc(N), N->getValueType(0), Op);<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeTypes.h<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeTypes.h?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeTypes.h?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeTypes.h (original)<br>
> +++ llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeTypes.h Thu Dec 4 03:40:44 2014<br>
> @@ -285,6 +285,8 @@ private:<br>
> SDValue PromoteIntOp_TRUNCATE(SDNode *N);<br>
> SDValue PromoteIntOp_UINT_TO_FP(SDNode *N);<br>
> SDValue PromoteIntOp_ZERO_EXTEND(SDNode *N);<br>
> + SDValue PromoteIntOp_MSTORE(MaskedStoreSDNode *N, unsigned OpNo);<br>
> + SDValue PromoteIntOp_MLOAD(MaskedLoadSDNode *N, unsigned OpNo);<br>
><span class="apple-converted-space"> </span><br>
> void PromoteSetCCOperands(SDValue &LHS,SDValue &RHS, ISD::CondCode Code);<br>
><span class="apple-converted-space"> </span><br>
> @@ -578,6 +580,7 @@ private:<br>
> void SplitVecRes_FPOWI(SDNode *N, SDValue &Lo, SDValue &Hi);<br>
> void SplitVecRes_INSERT_VECTOR_ELT(SDNode *N, SDValue &Lo, SDValue &Hi);<br>
> void SplitVecRes_LOAD(LoadSDNode *N, SDValue &Lo, SDValue &Hi);<br>
> + void SplitVecRes_MLOAD(MaskedLoadSDNode *N, SDValue &Lo, SDValue &Hi);<br>
> void SplitVecRes_SCALAR_TO_VECTOR(SDNode *N, SDValue &Lo, SDValue &Hi);<br>
> void SplitVecRes_SIGN_EXTEND_INREG(SDNode *N, SDValue &Lo, SDValue &Hi);<br>
> void SplitVecRes_SETCC(SDNode *N, SDValue &Lo, SDValue &Hi);<br>
> @@ -594,6 +597,7 @@ private:<br>
> SDValue SplitVecOp_EXTRACT_SUBVECTOR(SDNode *N);<br>
> SDValue SplitVecOp_EXTRACT_VECTOR_ELT(SDNode *N);<br>
> SDValue SplitVecOp_STORE(StoreSDNode *N, unsigned OpNo);<br>
> + SDValue SplitVecOp_MSTORE(MaskedStoreSDNode *N, unsigned OpNo);<br>
> SDValue SplitVecOp_CONCAT_VECTORS(SDNode *N);<br>
> SDValue SplitVecOp_TRUNCATE(SDNode *N);<br>
> SDValue SplitVecOp_VSETCC(SDNode *N);<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp (original)<br>
> +++ llvm/trunk/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp Thu Dec 4 03:40:44 2014<br>
> @@ -597,6 +597,9 @@ void DAGTypeLegalizer::SplitVectorResult<br>
> case ISD::LOAD:<br>
> SplitVecRes_LOAD(cast<LoadSDNode>(N), Lo, Hi);<br>
> break;<br>
> + case ISD::MLOAD:<br>
> + SplitVecRes_MLOAD(cast<MaskedLoadSDNode>(N), Lo, Hi);<br>
> + break;<br>
> case ISD::SETCC:<br>
> SplitVecRes_SETCC(N, Lo, Hi);<br>
> break;<br>
> @@ -979,6 +982,64 @@ void DAGTypeLegalizer::SplitVecRes_LOAD(<br>
> ReplaceValueWith(SDValue(LD, 1), Ch);<br>
> }<br>
><span class="apple-converted-space"> </span><br>
> +void DAGTypeLegalizer::SplitVecRes_MLOAD(MaskedLoadSDNode *MLD,<br>
> + SDValue &Lo, SDValue &Hi) {<br>
> + EVT LoVT, HiVT;<br>
> + SDLoc dl(MLD);<br>
> + std::tie(LoVT, HiVT) = DAG.GetSplitDestVTs(MLD->getValueType(0));<br>
> +<br>
> + SDValue Ch = MLD->getChain();<br>
> + SDValue Ptr = MLD->getBasePtr();<br>
> + SDValue Mask = MLD->getMask();<br>
> + unsigned Alignment = MLD->getOriginalAlignment();<br>
> +<br>
> + // if Alignment is equal to the vector size,<br>
> + // take the half of it for the second part<br>
> + unsigned SecondHalfAlignment =<br>
> + (Alignment == MLD->getValueType(0).getSizeInBits()/8) ?<br>
> + Alignment/2 : Alignment;<br>
> +<br>
> + SDValue MaskLo, MaskHi;<br>
> + std::tie(MaskLo, MaskHi) = DAG.SplitVector(Mask, dl);<br>
> +<br>
> + EVT MemoryVT = MLD->getMemoryVT();<br>
> + EVT LoMemVT, HiMemVT;<br>
> + std::tie(LoMemVT, HiMemVT) = DAG.GetSplitDestVTs(MemoryVT);<br>
> +<br>
> + SDValue Src0 = MLD->getSrc0();<br>
> + SDValue Src0Lo, Src0Hi;<br>
> + std::tie(Src0Lo, Src0Hi) = DAG.SplitVector(Src0, dl);<br>
> +<br>
> + MachineMemOperand *MMO = DAG.getMachineFunction().<br>
> + getMachineMemOperand(MLD->getPointerInfo(),<span class="apple-converted-space"> </span><br>
> + MachineMemOperand::MOLoad, LoMemVT.getStoreSize(),<br>
> + Alignment, MLD->getAAInfo(), MLD->getRanges());<br>
> +<br>
> + Lo = DAG.getMaskedLoad(LoVT, dl, Ch, Ptr, MaskLo, Src0Lo, MMO);<br>
> +<br>
> + unsigned IncrementSize = LoMemVT.getSizeInBits()/8;<br>
> + Ptr = DAG.getNode(ISD::ADD, dl, Ptr.getValueType(), Ptr,<br>
> + DAG.getConstant(IncrementSize, Ptr.getValueType()));<br>
> +<br>
> + MMO = DAG.getMachineFunction().<br>
> + getMachineMemOperand(MLD->getPointerInfo(),<span class="apple-converted-space"> </span><br>
> + MachineMemOperand::MOLoad, HiMemVT.getStoreSize(),<br>
> + SecondHalfAlignment, MLD->getAAInfo(), MLD->getRanges());<br>
> +<br>
> + Hi = DAG.getMaskedLoad(HiVT, dl, Ch, Ptr, MaskHi, Src0Hi, MMO);<br>
> +<br>
> +<br>
> + // Build a factor node to remember that this load is independent of the<br>
> + // other one.<br>
> + Ch = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, Lo.getValue(1),<br>
> + Hi.getValue(1));<br>
> +<br>
> + // Legalized the chain result - switch anything that used the old chain to<br>
> + // use the new one.<br>
> + ReplaceValueWith(SDValue(MLD, 1), Ch);<br>
> +<br>
> +}<br>
> +<br>
> void DAGTypeLegalizer::SplitVecRes_SETCC(SDNode *N, SDValue &Lo, SDValue &Hi) {<br>
> assert(N->getValueType(0).isVector() &&<br>
> N->getOperand(0).getValueType().isVector() &&<br>
> @@ -1234,6 +1295,9 @@ bool DAGTypeLegalizer::SplitVectorOperan<br>
> case ISD::STORE:<br>
> Res = SplitVecOp_STORE(cast<StoreSDNode>(N), OpNo);<br>
> break;<br>
> + case ISD::MSTORE:<br>
> + Res = SplitVecOp_MSTORE(cast<MaskedStoreSDNode>(N), OpNo);<br>
> + break;<br>
> case ISD::VSELECT:<br>
> Res = SplitVecOp_VSELECT(N, OpNo);<br>
> break;<br>
> @@ -1395,6 +1459,56 @@ SDValue DAGTypeLegalizer::SplitVecOp_EXT<br>
> MachinePointerInfo(), EltVT, false, false, false, 0);<br>
> }<br>
><span class="apple-converted-space"> </span><br>
> +SDValue DAGTypeLegalizer::SplitVecOp_MSTORE(MaskedStoreSDNode *N,<br>
> + unsigned OpNo) {<br>
> + SDValue Ch = N->getChain();<br>
> + SDValue Ptr = N->getBasePtr();<br>
> + SDValue Mask = N->getMask();<br>
> + SDValue Data = N->getData();<br>
> + EVT MemoryVT = N->getMemoryVT();<br>
> + unsigned Alignment = N->getOriginalAlignment();<br>
> + SDLoc DL(N);<br>
> + <span class="apple-converted-space"> </span><br>
> + EVT LoMemVT, HiMemVT;<br>
> + std::tie(LoMemVT, HiMemVT) = DAG.GetSplitDestVTs(MemoryVT);<br>
> +<br>
> + SDValue DataLo, DataHi;<br>
> + GetSplitVector(Data, DataLo, DataHi);<br>
> + SDValue MaskLo, MaskHi;<br>
> + GetSplitVector(Mask, MaskLo, MaskHi);<br>
> +<br>
> + // if Alignment is equal to the vector size,<br>
> + // take the half of it for the second part<br>
> + unsigned SecondHalfAlignment =<br>
> + (Alignment == Data->getValueType(0).getSizeInBits()/8) ?<br>
> + Alignment/2 : Alignment;<br>
> +<br>
> + SDValue Lo, Hi;<br>
> + MachineMemOperand *MMO = DAG.getMachineFunction().<br>
> + getMachineMemOperand(N->getPointerInfo(),<span class="apple-converted-space"> </span><br>
> + MachineMemOperand::MOStore, LoMemVT.getStoreSize(),<br>
> + Alignment, N->getAAInfo(), N->getRanges());<br>
> +<br>
> + Lo = DAG.getMaskedStore(Ch, DL, DataLo, Ptr, MaskLo, MMO);<br>
> +<br>
> + unsigned IncrementSize = LoMemVT.getSizeInBits()/8;<br>
> + Ptr = DAG.getNode(ISD::ADD, DL, Ptr.getValueType(), Ptr,<br>
> + DAG.getConstant(IncrementSize, Ptr.getValueType()));<br>
> +<br>
> + MMO = DAG.getMachineFunction().<br>
> + getMachineMemOperand(N->getPointerInfo(),<span class="apple-converted-space"> </span><br>
> + MachineMemOperand::MOStore, HiMemVT.getStoreSize(),<br>
> + SecondHalfAlignment, N->getAAInfo(), N->getRanges());<br>
> +<br>
> + Hi = DAG.getMaskedStore(Ch, DL, DataHi, Ptr, MaskHi, MMO);<br>
> +<br>
> +<br>
> + // Build a factor node to remember that this store is independent of the<br>
> + // other one.<br>
> + return DAG.getNode(ISD::TokenFactor, DL, MVT::Other, Lo, Hi);<br>
> +<br>
> +}<br>
> +<br>
> SDValue DAGTypeLegalizer::SplitVecOp_STORE(StoreSDNode *N, unsigned OpNo) {<br>
> assert(N->isUnindexed() && "Indexed store of vector?");<br>
> assert(OpNo == 1 && "Can only split the stored value");<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp (original)<br>
> +++ llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAG.cpp Thu Dec 4 03:40:44 2014<br>
> @@ -4917,6 +4917,60 @@ SelectionDAG::getIndexedStore(SDValue Or<br>
> return SDValue(N, 0);<br>
> }<br>
><span class="apple-converted-space"> </span><br>
> +SDValue<br>
> +SelectionDAG::getMaskedLoad(EVT VT, SDLoc dl, SDValue Chain,<br>
> + SDValue Ptr, SDValue Mask, SDValue Src0,<br>
> + MachineMemOperand *MMO) {<br>
> +<br>
> + SDVTList VTs = getVTList(VT, MVT::Other);<br>
> + SDValue Ops[] = { Chain, Ptr, Mask, Src0 };<br>
> + FoldingSetNodeID ID;<br>
> + AddNodeIDNode(ID, ISD::MLOAD, VTs, Ops);<br>
> + ID.AddInteger(VT.getRawBits());<br>
> + ID.AddInteger(encodeMemSDNodeFlags(ISD::NON_EXTLOAD, ISD::UNINDEXED,<br>
> + MMO->isVolatile(),<br>
> + MMO->isNonTemporal(),<br>
> + MMO->isInvariant()));<br>
> + ID.AddInteger(MMO->getPointerInfo().getAddrSpace());<br>
> + void *IP = nullptr;<br>
> + if (SDNode *E = CSEMap.FindNodeOrInsertPos(ID, IP)) {<br>
> + cast<MaskedLoadSDNode>(E)->refineAlignment(MMO);<br>
> + return SDValue(E, 0);<br>
> + }<br>
> + SDNode *N = new (NodeAllocator) MaskedLoadSDNode(dl.getIROrder(),<br>
> + dl.getDebugLoc(), Ops, 4, VTs,<br>
> + VT, MMO);<br>
> + CSEMap.InsertNode(N, IP);<br>
> + InsertNode(N);<br>
> + return SDValue(N, 0);<br>
> +}<br>
> +<br>
> +SDValue SelectionDAG::getMaskedStore(SDValue Chain, SDLoc dl, SDValue Val,<br>
> + SDValue Ptr, SDValue Mask, MachineMemOperand *MMO) {<br>
> + assert(Chain.getValueType() == MVT::Other &&<br>
> + "Invalid chain type");<br>
> + EVT VT = Val.getValueType();<br>
> + SDVTList VTs = getVTList(MVT::Other);<br>
> + SDValue Ops[] = { Chain, Ptr, Mask, Val };<br>
> + FoldingSetNodeID ID;<br>
> + AddNodeIDNode(ID, ISD::MSTORE, VTs, Ops);<br>
> + ID.AddInteger(VT.getRawBits());<br>
> + ID.AddInteger(encodeMemSDNodeFlags(false, ISD::UNINDEXED, MMO->isVolatile(),<br>
> + MMO->isNonTemporal(), MMO->isInvariant()));<br>
> + ID.AddInteger(MMO->getPointerInfo().getAddrSpace());<br>
> + void *IP = nullptr;<br>
> + if (SDNode *E = CSEMap.FindNodeOrInsertPos(ID, IP)) {<br>
> + cast<MaskedStoreSDNode>(E)->refineAlignment(MMO);<br>
> + return SDValue(E, 0);<br>
> + }<br>
> + SDNode *N = new (NodeAllocator) MaskedStoreSDNode(dl.getIROrder(),<br>
> + dl.getDebugLoc(), Ops, 4,<br>
> + VTs, VT, MMO);<br>
> + CSEMap.InsertNode(N, IP);<br>
> + InsertNode(N);<br>
> + return SDValue(N, 0);<br>
> +}<br>
> +<br>
> SDValue SelectionDAG::getVAArg(EVT VT, SDLoc dl,<br>
> SDValue Chain, SDValue Ptr,<br>
> SDValue SV,<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp (original)<br>
> +++ llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp Thu Dec 4 03:40:44 2014<br>
> @@ -3617,6 +3617,70 @@ void SelectionDAGBuilder::visitStore(con<br>
> DAG.setRoot(StoreNode);<br>
> }<br>
><span class="apple-converted-space"> </span><br>
> +void SelectionDAGBuilder::visitMaskedStore(const CallInst &I) {<br>
> + SDLoc sdl = getCurSDLoc();<br>
> +<br>
> + Value *PtrOperand = I.getArgOperand(0);<br>
> + SDValue Ptr = getValue(PtrOperand);<br>
> + SDValue Src0 = getValue(I.getArgOperand(1));<br>
> + SDValue Mask = getValue(I.getArgOperand(3));<br>
> + EVT VT = Src0.getValueType();<br>
> + unsigned Alignment = (cast<ConstantInt>(I.getArgOperand(2)))->getZExtValue();<br>
> + if (!Alignment)<br>
> + Alignment = DAG.getEVTAlignment(VT);<br>
> +<br>
> + AAMDNodes AAInfo;<br>
> + I.getAAMetadata(AAInfo);<br>
> +<br>
> + MachineMemOperand *MMO =<br>
> + DAG.getMachineFunction().<br>
> + getMachineMemOperand(MachinePointerInfo(PtrOperand),<br>
> + MachineMemOperand::MOStore, VT.getStoreSize(),<br>
> + Alignment, AAInfo);<br>
> + SDValue StoreNode = DAG.getMaskedStore(getRoot(), sdl, Src0, Ptr, Mask, MMO);<br>
> + DAG.setRoot(StoreNode);<br>
> + setValue(&I, StoreNode);<br>
> +}<br>
> +<br>
> +void SelectionDAGBuilder::visitMaskedLoad(const CallInst &I) {<br>
> + SDLoc sdl = getCurSDLoc();<br>
> +<br>
> + Value *PtrOperand = I.getArgOperand(0);<br>
> + SDValue Ptr = getValue(PtrOperand);<br>
> + SDValue Src0 = getValue(I.getArgOperand(1));<br>
> + SDValue Mask = getValue(I.getArgOperand(3));<br>
> +<br>
> + const TargetLowering &TLI = DAG.getTargetLoweringInfo();<br>
> + EVT VT = TLI.getValueType(I.getType());<br>
> + unsigned Alignment = (cast<ConstantInt>(I.getArgOperand(2)))->getZExtValue();<br>
> + if (!Alignment)<br>
> + Alignment = DAG.getEVTAlignment(VT);<br>
> +<br>
> + AAMDNodes AAInfo;<br>
> + I.getAAMetadata(AAInfo);<br>
> + const MDNode *Ranges = I.getMetadata(LLVMContext::MD_range);<br>
> +<br>
> + SDValue InChain = DAG.getRoot();<br>
> + if (AA->pointsToConstantMemory(<br>
> + AliasAnalysis::Location(PtrOperand,<br>
> + AA->getTypeStoreSize(I.getType()),<br>
> + AAInfo))) {<br>
> + // Do not serialize (non-volatile) loads of constant memory with anything.<br>
> + InChain = DAG.getEntryNode();<br>
> + }<br>
> +<br>
> + MachineMemOperand *MMO =<br>
> + DAG.getMachineFunction().<br>
> + getMachineMemOperand(MachinePointerInfo(PtrOperand),<br>
> + MachineMemOperand::MOLoad, VT.getStoreSize(),<br>
> + Alignment, AAInfo, Ranges);<br>
> +<br>
> + SDValue Load = DAG.getMaskedLoad(VT, sdl, InChain, Ptr, Mask, Src0, MMO);<br>
> + SDValue OutChain = Load.getValue(1);<br>
> + DAG.setRoot(OutChain);<br>
> + setValue(&I, Load);<br>
> +}<br>
> +<br>
> void SelectionDAGBuilder::visitAtomicCmpXchg(const AtomicCmpXchgInst &I) {<br>
> SDLoc dl = getCurSDLoc();<br>
> AtomicOrdering SuccessOrder = I.getSuccessOrdering();<br>
> @@ -4918,6 +4982,12 @@ SelectionDAGBuilder::visitIntrinsicCall(<br>
> return nullptr;<br>
> }<br>
><span class="apple-converted-space"> </span><br>
> + case Intrinsic::masked_load:<br>
> + visitMaskedLoad(I);<br>
> + return nullptr;<br>
> + case Intrinsic::masked_store:<br>
> + visitMaskedStore(I);<br>
> + return nullptr;<br>
> case Intrinsic::x86_mmx_pslli_w:<br>
> case Intrinsic::x86_mmx_pslli_d:<br>
> case Intrinsic::x86_mmx_pslli_q:<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h (original)<br>
> +++ llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h Thu Dec 4 03:40:44 2014<br>
> @@ -769,6 +769,8 @@ private:<br>
> void visitAlloca(const AllocaInst &I);<br>
> void visitLoad(const LoadInst &I);<br>
> void visitStore(const StoreInst &I);<br>
> + void visitMaskedLoad(const CallInst &I);<br>
> + void visitMaskedStore(const CallInst &I);<br>
> void visitAtomicCmpXchg(const AtomicCmpXchgInst &I);<br>
> void visitAtomicRMW(const AtomicRMWInst &I);<br>
> void visitFence(const FenceInst &I);<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp (original)<br>
> +++ llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp Thu Dec 4 03:40:44 2014<br>
> @@ -269,6 +269,8 @@ std::string SDNode::getOperationName(con<br>
> // Other operators<br>
> case ISD::LOAD: return "load";<br>
> case ISD::STORE: return "store";<br>
> + case ISD::MLOAD: return "masked_load";<br>
> + case ISD::MSTORE: return "masked_store";<br>
> case ISD::VAARG: return "vaarg";<br>
> case ISD::VACOPY: return "vacopy";<br>
> case ISD::VAEND: return "vaend";<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/lib/IR/Function.cpp<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/IR/Function.cpp?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/IR/Function.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/IR/Function.cpp (original)<br>
> +++ llvm/trunk/lib/IR/Function.cpp Thu Dec 4 03:40:44 2014<br>
> @@ -551,7 +551,8 @@ enum IIT_Info {<br>
> IIT_ANYPTR = 26,<br>
> IIT_V1 = 27,<br>
> IIT_VARARG = 28,<br>
> - IIT_HALF_VEC_ARG = 29<br>
> + IIT_HALF_VEC_ARG = 29,<br>
> + IIT_SAME_VEC_WIDTH_ARG = 30<br>
> };<br>
><span class="apple-converted-space"> </span><br>
><span class="apple-converted-space"> </span><br>
> @@ -659,6 +660,12 @@ static void DecodeIITType(unsigned &Next<br>
> ArgInfo));<br>
> return;<br>
> }<br>
> + case IIT_SAME_VEC_WIDTH_ARG: {<br>
> + unsigned ArgInfo = (NextElt == Infos.size() ? 0 : Infos[NextElt++]);<br>
> + OutputTable.push_back(IITDescriptor::get(IITDescriptor::SameVecWidthArgument,<br>
> + ArgInfo));<br>
> + return;<br>
> + }<br>
> case IIT_EMPTYSTRUCT:<br>
> OutputTable.push_back(IITDescriptor::get(IITDescriptor::Struct, 0));<br>
> return;<br>
> @@ -766,7 +773,14 @@ static Type *DecodeFixedType(ArrayRef<In<br>
> case IITDescriptor::HalfVecArgument:<br>
> return VectorType::getHalfElementsVectorType(cast<VectorType>(<br>
> Tys[D.getArgumentNumber()]));<br>
> - }<br>
> + case IITDescriptor::SameVecWidthArgument:<br>
> + Type *EltTy = DecodeFixedType(Infos, Tys, Context);<br>
> + Type *Ty = Tys[D.getArgumentNumber()];<br>
> + if (VectorType *VTy = dyn_cast<VectorType>(Ty)) {<br>
> + return VectorType::get(EltTy, VTy->getNumElements());<br>
> + }<br>
> + llvm_unreachable("unhandled");<br>
> + }<br>
> llvm_unreachable("unhandled");<br>
> }<br>
><span class="apple-converted-space"> </span><br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/lib/IR/IRBuilder.cpp<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/IR/IRBuilder.cpp?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/IR/IRBuilder.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/IR/IRBuilder.cpp (original)<br>
> +++ llvm/trunk/lib/IR/IRBuilder.cpp Thu Dec 4 03:40:44 2014<br>
> @@ -183,3 +183,29 @@ CallInst *IRBuilderBase::CreateAssumptio<br>
> return createCallHelper(FnAssume, Ops, this);<br>
> }<br>
><span class="apple-converted-space"> </span><br>
> +/// Create a call to a Masked Load intrinsic.<br>
> +/// Ops - an array of operands.<br>
> +CallInst *IRBuilderBase::CreateMaskedLoad(ArrayRef<Value *> Ops) {<br>
> + // The only one overloaded type - the type of passthru value in this case<br>
> + Type *DataTy = Ops[1]->getType();<br>
> + return CreateMaskedIntrinsic(Intrinsic::masked_load, Ops, DataTy);<br>
> +}<br>
> +<br>
> +/// Create a call to a Masked Store intrinsic.<br>
> +/// Ops - an array of operands.<br>
> +CallInst *IRBuilderBase::CreateMaskedStore(ArrayRef<Value *> Ops) {<br>
> + // DataTy - type of the data to be stored - the only one overloaded type<br>
> + Type *DataTy = Ops[1]->getType();<br>
> + return CreateMaskedIntrinsic(Intrinsic::masked_store, Ops, DataTy);<br>
> +}<br>
> +<br>
> +/// Create a call to a Masked intrinsic, with given intrinsic Id,<br>
> +/// an array of operands - Ops, and one overloaded type - DataTy<br>
> +CallInst *IRBuilderBase::CreateMaskedIntrinsic(unsigned Id,<br>
> + ArrayRef<Value *> Ops,<br>
> + Type *DataTy) {<br>
> + Module *M = BB->getParent()->getParent();<br>
> + Type *OverloadedTypes[] = { DataTy };<br>
> + Value *TheFn = Intrinsic::getDeclaration(M, (Intrinsic::ID)Id, OverloadedTypes);<br>
> + return createCallHelper(TheFn, Ops, this);<br>
> +}<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/lib/IR/Verifier.cpp<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/IR/Verifier.cpp?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/IR/Verifier.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/IR/Verifier.cpp (original)<br>
> +++ llvm/trunk/lib/IR/Verifier.cpp Thu Dec 4 03:40:44 2014<br>
> @@ -2406,6 +2406,19 @@ bool Verifier::VerifyIntrinsicType(Type<br>
> !isa<VectorType>(ArgTys[D.getArgumentNumber()]) ||<br>
> VectorType::getHalfElementsVectorType(<br>
> cast<VectorType>(ArgTys[D.getArgumentNumber()])) != Ty;<br>
> + case IITDescriptor::SameVecWidthArgument: {<br>
> + if (D.getArgumentNumber() >= ArgTys.size())<br>
> + return true;<br>
> + VectorType * ReferenceType =<br>
> + dyn_cast<VectorType>(ArgTys[D.getArgumentNumber()]);<br>
> + VectorType *ThisArgType = dyn_cast<VectorType>(Ty);<br>
> + if (!ThisArgType || !ReferenceType ||<span class="apple-converted-space"> </span><br>
> + (ReferenceType->getVectorNumElements() !=<br>
> + ThisArgType->getVectorNumElements()))<br>
> + return true;<br>
> + return VerifyIntrinsicType(ThisArgType->getVectorElementType(),<br>
> + Infos, ArgTys);<br>
> + }<br>
> }<br>
> llvm_unreachable("unhandled");<br>
> }<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/lib/Target/X86/X86ISelLowering.cpp<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86ISelLowering.cpp?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86ISelLowering.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/Target/X86/X86ISelLowering.cpp (original)<br>
> +++ llvm/trunk/lib/Target/X86/X86ISelLowering.cpp Thu Dec 4 03:40:44 2014<br>
> @@ -1319,13 +1319,21 @@ void X86TargetLowering::resetOperationAc<br>
><span class="apple-converted-space"> </span><br>
> // Extract subvector is special because the value type<br>
> // (result) is 128-bit but the source is 256-bit wide.<br>
> - if (VT.is128BitVector())<br>
> + if (VT.is128BitVector()) {<br>
> + if (VT.getScalarSizeInBits() >= 32) {<br>
> + setOperationAction(ISD::MLOAD, VT, Custom);<br>
> + setOperationAction(ISD::MSTORE, VT, Custom);<br>
> + }<br>
> setOperationAction(ISD::EXTRACT_SUBVECTOR, VT, Custom);<br>
> -<br>
> + }<br>
> // Do not attempt to custom lower other non-256-bit vectors<br>
> if (!VT.is256BitVector())<br>
> continue;<br>
><span class="apple-converted-space"> </span><br>
> + if (VT.getScalarSizeInBits() >= 32) {<br>
> + setOperationAction(ISD::MLOAD, VT, Legal);<br>
> + setOperationAction(ISD::MSTORE, VT, Legal);<br>
> + }<br>
> setOperationAction(ISD::BUILD_VECTOR, VT, Custom);<br>
> setOperationAction(ISD::VECTOR_SHUFFLE, VT, Custom);<br>
> setOperationAction(ISD::INSERT_VECTOR_ELT, VT, Custom);<br>
> @@ -1492,9 +1500,13 @@ void X86TargetLowering::resetOperationAc<br>
> unsigned EltSize = VT.getVectorElementType().getSizeInBits();<br>
> // Extract subvector is special because the value type<br>
> // (result) is 256/128-bit but the source is 512-bit wide.<br>
> - if (VT.is128BitVector() || VT.is256BitVector())<br>
> + if (VT.is128BitVector() || VT.is256BitVector()) {<br>
> setOperationAction(ISD::EXTRACT_SUBVECTOR, VT, Custom);<br>
> -<br>
> + if ( EltSize >= 32) {<br>
> + setOperationAction(ISD::MLOAD, VT, Legal);<br>
> + setOperationAction(ISD::MSTORE, VT, Legal);<br>
> + }<br>
> + }<br>
> if (VT.getVectorElementType() == MVT::i1)<br>
> setOperationAction(ISD::EXTRACT_SUBVECTOR, VT, Legal);<br>
><span class="apple-converted-space"> </span><br>
> @@ -1510,6 +1522,8 @@ void X86TargetLowering::resetOperationAc<br>
> setOperationAction(ISD::EXTRACT_VECTOR_ELT, VT, Custom);<br>
> setOperationAction(ISD::SCALAR_TO_VECTOR, VT, Custom);<br>
> setOperationAction(ISD::INSERT_SUBVECTOR, VT, Custom);<br>
> + setOperationAction(ISD::MLOAD, VT, Legal);<br>
> + setOperationAction(ISD::MSTORE, VT, Legal);<br>
> }<br>
> }<br>
> for (int i = MVT::v32i8; i != MVT::v8i64; ++i) {<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/lib/Target/X86/X86InstrAVX512.td<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86InstrAVX512.td?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86InstrAVX512.td?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/Target/X86/X86InstrAVX512.td (original)<br>
> +++ llvm/trunk/lib/Target/X86/X86InstrAVX512.td Thu Dec 4 03:40:44 2014<br>
> @@ -2122,6 +2122,41 @@ def: Pat<(int_x86_avx512_mask_storeu_pd_<br>
> (VMOVUPDZmrk addr:$ptr, (v8i1 (COPY_TO_REGCLASS GR8:$mask, VK8WM)),<br>
> VR512:$src)>;<br>
><span class="apple-converted-space"> </span><br>
> +def: Pat<(masked_store addr:$ptr, VK8WM:$mask, (v8f32 VR256:$src)),<br>
> + (VMOVUPSZmrk addr:$ptr,<br>
> + (v16i1 (COPY_TO_REGCLASS VK8WM:$mask, VK16WM)),<br>
> + (INSERT_SUBREG (v16f32 (IMPLICIT_DEF)), VR256:$src, sub_ymm))>;<br>
> +<br>
> +def: Pat<(v8f32 (masked_load addr:$ptr, VK8WM:$mask, undef)),<br>
> + (v8f32 (EXTRACT_SUBREG (v16f32 (VMOVUPSZrmkz<span class="apple-converted-space"> </span><br>
> + (v16i1 (COPY_TO_REGCLASS VK8WM:$mask, VK16WM)), addr:$ptr)), sub_ymm))>;<br>
> +<br>
> +def: Pat<(masked_store addr:$ptr, VK16WM:$mask, (v16f32 VR512:$src)),<br>
> + (VMOVUPSZmrk addr:$ptr, VK16WM:$mask, VR512:$src)>;<br>
> +<br>
> +def: Pat<(masked_store addr:$ptr, VK8WM:$mask, (v8f64 VR512:$src)),<br>
> + (VMOVUPDZmrk addr:$ptr, VK8WM:$mask, VR512:$src)>;<br>
> +<br>
> +def: Pat<(v16f32 (masked_load addr:$ptr, VK16WM:$mask, undef)),<br>
> + (VMOVUPSZrmkz VK16WM:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v16f32 (masked_load addr:$ptr, VK16WM:$mask,<br>
> + (bc_v16f32 (v16i32 immAllZerosV)))),<br>
> + (VMOVUPSZrmkz VK16WM:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v16f32 (masked_load addr:$ptr, VK16WM:$mask, (v16f32 VR512:$src0))),<br>
> + (VMOVUPSZrmk VR512:$src0, VK16WM:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v8f64 (masked_load addr:$ptr, VK8WM:$mask, undef)),<br>
> + (VMOVUPDZrmkz VK8WM:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v8f64 (masked_load addr:$ptr, VK8WM:$mask,<br>
> + (bc_v8f64 (v16i32 immAllZerosV)))),<br>
> + (VMOVUPDZrmkz VK8WM:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v8f64 (masked_load addr:$ptr, VK8WM:$mask, (v8f64 VR512:$src0))),<br>
> + (VMOVUPDZrmk VR512:$src0, VK8WM:$mask, addr:$ptr)>;<br>
> +<br>
> defm VMOVDQA32 : avx512_load_vl<0x6F, "vmovdqa32", "alignedload", "i", "32",<br>
> "16", "8", "4", SSEPackedInt, HasAVX512>,<br>
> avx512_store_vl<0x7F, "vmovdqa32", "alignedstore",<br>
> @@ -2196,6 +2231,46 @@ def : Pat<(v16i32 (vselect VK16WM:$mask,<br>
> (VMOVDQU32Zrrkz (KNOTWrr VK16WM:$mask), VR512:$src)>;<br>
> }<br>
><span class="apple-converted-space"> </span><br>
> +def: Pat<(v16i32 (masked_load addr:$ptr, VK16WM:$mask, (v16i32 immAllZerosV))),<br>
> + (VMOVDQU32Zrmkz VK16WM:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v16i32 (masked_load addr:$ptr, VK16WM:$mask, undef)),<br>
> + (VMOVDQU32Zrmkz VK16WM:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v16i32 (masked_load addr:$ptr, VK16WM:$mask, (v16i32 VR512:$src0))),<br>
> + (VMOVDQU32Zrmk VR512:$src0, VK16WM:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v8i64 (masked_load addr:$ptr, VK8WM:$mask,<br>
> + (bc_v8i64 (v16i32 immAllZerosV)))),<br>
> + (VMOVDQU64Zrmkz VK8WM:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v8i64 (masked_load addr:$ptr, VK8WM:$mask, undef)),<br>
> + (VMOVDQU64Zrmkz VK8WM:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v8i64 (masked_load addr:$ptr, VK8WM:$mask, (v8i64 VR512:$src0))),<br>
> + (VMOVDQU64Zrmk VR512:$src0, VK8WM:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(masked_store addr:$ptr, VK16WM:$mask, (v16i32 VR512:$src)),<br>
> + (VMOVDQU32Zmrk addr:$ptr, VK16WM:$mask, VR512:$src)>;<br>
> +<br>
> +def: Pat<(masked_store addr:$ptr, VK8WM:$mask, (v8i64 VR512:$src)),<br>
> + (VMOVDQU64Zmrk addr:$ptr, VK8WM:$mask, VR512:$src)>;<br>
> +<br>
> +// SKX replacement<br>
> +def: Pat<(masked_store addr:$ptr, VK8WM:$mask, (v8i32 VR256:$src)),<br>
> + (VMOVDQU32Z256mrk addr:$ptr, VK8WM:$mask, VR256:$src)>;<br>
> +<br>
> +// KNL replacement<br>
> +def: Pat<(masked_store addr:$ptr, VK8WM:$mask, (v8i32 VR256:$src)),<br>
> + (VMOVDQU32Zmrk addr:$ptr,<br>
> + (v16i1 (COPY_TO_REGCLASS VK8WM:$mask, VK16WM)),<br>
> + (INSERT_SUBREG (v16i32 (IMPLICIT_DEF)), VR256:$src, sub_ymm))>;<br>
> +<br>
> +def: Pat<(v8i32 (masked_load addr:$ptr, VK8WM:$mask, undef)),<br>
> + (v8i32 (EXTRACT_SUBREG (v16i32 (VMOVDQU32Zrmkz<span class="apple-converted-space"> </span><br>
> + (v16i1 (COPY_TO_REGCLASS VK8WM:$mask, VK16WM)), addr:$ptr)), sub_ymm))>;<br>
> +<br>
> +<br>
> // Move Int Doubleword to Packed Double Int<br>
> //<br>
> def VMOVDI2PDIZrr : AVX512BI<0x6E, MRMSrcReg, (outs VR128X:$dst), (ins GR32:$src),<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/lib/Target/X86/X86InstrSSE.td<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86InstrSSE.td?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86InstrSSE.td?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/Target/X86/X86InstrSSE.td (original)<br>
> +++ llvm/trunk/lib/Target/X86/X86InstrSSE.td Thu Dec 4 03:40:44 2014<br>
> @@ -9260,6 +9260,61 @@ defm VPMASKMOVQ : avx2_pmovmask<"vpmaskm<br>
> int_x86_avx2_maskstore_q,<br>
> int_x86_avx2_maskstore_q_256>, VEX_W;<br>
><span class="apple-converted-space"> </span><br>
> +def: Pat<(masked_store addr:$ptr, (v8i32 VR256:$mask), (v8f32 VR256:$src)),<br>
> + (VPMASKMOVDYmr addr:$ptr, VR256:$mask, VR256:$src)>;<br>
> +<br>
> +def: Pat<(masked_store addr:$ptr, (v8i32 VR256:$mask), (v8i32 VR256:$src)),<br>
> + (VPMASKMOVDYmr addr:$ptr, VR256:$mask, VR256:$src)>;<br>
> +<br>
> +def: Pat<(v8f32 (masked_load addr:$ptr, (v8i32 VR256:$mask), undef)),<br>
> + (VPMASKMOVDYrm VR256:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v8f32 (masked_load addr:$ptr, (v8i32 VR256:$mask),<br>
> + (bc_v8f32 (v8i32 immAllZerosV)))),<br>
> + (VPMASKMOVDYrm VR256:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v8f32 (masked_load addr:$ptr, (v8i32 VR256:$mask), (v8f32 VR256:$src0))),<br>
> + (VBLENDVPSYrr VR256:$src0, (VPMASKMOVDYrm VR256:$mask, addr:$ptr),<br>
> + VR256:$mask)>;<br>
> +<br>
> +def: Pat<(v8i32 (masked_load addr:$ptr, (v8i32 VR256:$mask), undef)),<br>
> + (VPMASKMOVDYrm VR256:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v8i32 (masked_load addr:$ptr, (v8i32 VR256:$mask), (v8i32 immAllZerosV))),<br>
> + (VPMASKMOVDYrm VR256:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v8i32 (masked_load addr:$ptr, (v8i32 VR256:$mask), (v8i32 VR256:$src0))),<br>
> + (VBLENDVPSYrr VR256:$src0, (VPMASKMOVDYrm VR256:$mask, addr:$ptr),<br>
> + VR256:$mask)>;<br>
> +<br>
> +def: Pat<(masked_store addr:$ptr, (v4i64 VR256:$mask), (v4f64 VR256:$src)),<br>
> + (VPMASKMOVQYmr addr:$ptr, VR256:$mask, VR256:$src)>;<br>
> +<br>
> +def: Pat<(masked_store addr:$ptr, (v4i64 VR256:$mask), (v4i64 VR256:$src)),<br>
> + (VPMASKMOVQYmr addr:$ptr, VR256:$mask, VR256:$src)>;<br>
> +<br>
> +def: Pat<(v4f64 (masked_load addr:$ptr, (v4i64 VR256:$mask), undef)),<br>
> + (VPMASKMOVQYrm VR256:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v4f64 (masked_load addr:$ptr, (v4i64 VR256:$mask),<br>
> + (v4f64 immAllZerosV))),<br>
> + (VPMASKMOVQYrm VR256:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v4f64 (masked_load addr:$ptr, (v4i64 VR256:$mask), (v4f64 VR256:$src0))),<br>
> + (VBLENDVPDYrr VR256:$src0, (VPMASKMOVQYrm VR256:$mask, addr:$ptr),<br>
> + VR256:$mask)>;<br>
> +<br>
> +def: Pat<(v4i64 (masked_load addr:$ptr, (v4i64 VR256:$mask), undef)),<br>
> + (VPMASKMOVQYrm VR256:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v4i64 (masked_load addr:$ptr, (v4i64 VR256:$mask),<br>
> + (bc_v4i64 (v8i32 immAllZerosV)))),<br>
> + (VPMASKMOVQYrm VR256:$mask, addr:$ptr)>;<br>
> +<br>
> +def: Pat<(v4i64 (masked_load addr:$ptr, (v4i64 VR256:$mask), (v4i64 VR256:$src0))),<br>
> + (VBLENDVPDYrr VR256:$src0, (VPMASKMOVQYrm VR256:$mask, addr:$ptr),<br>
> + VR256:$mask)>;<br>
> +<br>
><span class="apple-converted-space"> </span><br>
> //===----------------------------------------------------------------------===//<br>
> // Variable Bit Shifts<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp (original)<br>
> +++ llvm/trunk/lib/Target/X86/X86TargetTransformInfo.cpp Thu Dec 4 03:40:44 2014<br>
> @@ -111,6 +111,8 @@ public:<br>
> Type *Ty) const override;<br>
> unsigned getIntImmCost(Intrinsic::ID IID, unsigned Idx, const APInt &Imm,<br>
> Type *Ty) const override;<br>
> + bool isLegalPredicatedLoad (Type *DataType, int Consecutive) const override;<br>
> + bool isLegalPredicatedStore(Type *DataType, int Consecutive) const override;<br>
><span class="apple-converted-space"> </span><br>
> /// @}<br>
> };<br>
> @@ -1156,3 +1158,19 @@ unsigned X86TTI::getIntImmCost(Intrinsic<br>
> }<br>
> return X86TTI::getIntImmCost(Imm, Ty);<br>
> }<br>
> +<br>
> +bool X86TTI::isLegalPredicatedLoad(Type *DataType, int Consecutive) const {<br>
> + int ScalarWidth = DataType->getScalarSizeInBits();<br>
> + <span class="apple-converted-space"> </span><br>
> + // Todo: AVX512 allows gather/scatter, works with strided and random as well<br>
> + if ((ScalarWidth < 32) || (Consecutive == 0))<br>
> + return false;<br>
> + if (ST->hasAVX512() || ST->hasAVX2())<span class="apple-converted-space"> </span><br>
> + return true;<br>
> + return false;<br>
> +}<br>
> +<br>
> +bool X86TTI::isLegalPredicatedStore(Type *DataType, int Consecutive) const {<br>
> + return isLegalPredicatedLoad(DataType, Consecutive);<br>
> +}<br>
> +<br>
><span class="apple-converted-space"> </span><br>
> Added: llvm/trunk/test/CodeGen/X86/masked_memop.ll<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/CodeGen/X86/masked_memop.ll?rev=223348&view=auto"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/CodeGen/X86/masked_memop.ll?rev=223348&view=auto</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/test/CodeGen/X86/masked_memop.ll (added)<br>
> +++ llvm/trunk/test/CodeGen/X86/masked_memop.ll Thu Dec 4 03:40:44 2014<br>
> @@ -0,0 +1,73 @@<br>
> +; RUN: llc -mtriple=x86_64-apple-darwin -mcpu=knl < %s | FileCheck %s -check-prefix=AVX512<br>
> +; RUN: llc -mtriple=x86_64-apple-darwin -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2<br>
> +<br>
> +; AVX512-LABEL: test1<br>
> +; AVX512: vmovdqu32 (%rdi), %zmm0 {%k1} {z}<br>
> +<br>
> +; AVX2-LABEL: test1<br>
> +; AVX2: vpmaskmovd 32(%rdi)<br>
> +; AVX2: vpmaskmovd (%rdi)<br>
> +; AVX2-NOT: blend<br>
> +<br>
> +define <16 x i32> @test1(<16 x i32> %trigger, i8* %addr) {<br>
> + %mask = icmp eq <16 x i32> %trigger, zeroinitializer<br>
> + %res = call <16 x i32> @llvm.masked.load.v16i32(i8* %addr, <16 x i32>undef, i32 4, <16 x i1>%mask)<br>
> + ret <16 x i32> %res<br>
> +}<br>
> +<br>
> +; AVX512-LABEL: test2<br>
> +; AVX512: vmovdqu32 (%rdi), %zmm0 {%k1} {z}<br>
> +<br>
> +; AVX2-LABEL: test2<br>
> +; AVX2: vpmaskmovd {{.*}}(%rdi)<br>
> +; AVX2: vpmaskmovd {{.*}}(%rdi)<br>
> +; AVX2-NOT: blend<br>
> +define <16 x i32> @test2(<16 x i32> %trigger, i8* %addr) {<br>
> + %mask = icmp eq <16 x i32> %trigger, zeroinitializer<br>
> + %res = call <16 x i32> @llvm.masked.load.v16i32(i8* %addr, <16 x i32>zeroinitializer, i32 4, <16 x i1>%mask)<br>
> + ret <16 x i32> %res<br>
> +}<br>
> +<br>
> +; AVX512-LABEL: test3<br>
> +; AVX512: vmovdqu32 %zmm1, (%rdi) {%k1}<br>
> +<br>
> +define void @test3(<16 x i32> %trigger, i8* %addr, <16 x i32> %val) {<br>
> + %mask = icmp eq <16 x i32> %trigger, zeroinitializer<br>
> + call void @llvm.masked.store.v16i32(i8* %addr, <16 x i32>%val, i32 4, <16 x i1>%mask)<br>
> + ret void<br>
> +}<br>
> +<br>
> +; AVX512-LABEL: test4<br>
> +; AVX512: vmovups (%rdi), %zmm{{.*{%k[1-7]}}}<br>
> +<br>
> +; AVX2-LABEL: test4<br>
> +; AVX2: vpmaskmovd {{.*}}(%rdi)<br>
> +; AVX2: vpmaskmovd {{.*}}(%rdi)<br>
> +; AVX2: blend<br>
> +define <16 x float> @test4(<16 x i32> %trigger, i8* %addr, <16 x float> %dst) {<br>
> + %mask = icmp eq <16 x i32> %trigger, zeroinitializer<br>
> + %res = call <16 x float> @llvm.masked.load.v16f32(i8* %addr, <16 x float>%dst, i32 4, <16 x i1>%mask)<br>
> + ret <16 x float> %res<br>
> +}<br>
> +<br>
> +; AVX512-LABEL: test5<br>
> +; AVX512: vmovupd (%rdi), %zmm1 {%k1}<br>
> +<br>
> +; AVX2-LABEL: test5<br>
> +; AVX2: vpmaskmovq<br>
> +; AVX2: vblendvpd<br>
> +; AVX2: vpmaskmovq <span class="apple-converted-space"> </span><br>
> +; AVX2: vblendvpd<br>
> +define <8 x double> @test5(<8 x i32> %trigger, i8* %addr, <8 x double> %dst) {<br>
> + %mask = icmp eq <8 x i32> %trigger, zeroinitializer<br>
> + %res = call <8 x double> @llvm.masked.load.v8f64(i8* %addr, <8 x double>%dst, i32 4, <8 x i1>%mask)<br>
> + ret <8 x double> %res<br>
> +}<br>
> +<br>
> +declare <16 x i32> @llvm.masked.load.v16i32(i8*, <16 x i32>, i32, <16 x i1>)<span class="apple-converted-space"> </span><br>
> +declare void @llvm.masked.store.v16i32(i8*, <16 x i32>, i32, <16 x i1>)<span class="apple-converted-space"> </span><br>
> +declare <16 x float> @llvm.masked.load.v16f32(i8*, <16 x float>, i32, <16 x i1>)<span class="apple-converted-space"> </span><br>
> +declare void @llvm.masked.store.v16f32(i8*, <16 x float>, i32, <16 x i1>)<span class="apple-converted-space"> </span><br>
> +declare <8 x double> @llvm.masked.load.v8f64(i8*, <8 x double>, i32, <8 x i1>)<span class="apple-converted-space"> </span><br>
> +declare void @llvm.masked.store.v8f64(i8*, <8 x double>, i32, <8 x i1>)<span class="apple-converted-space"> </span><br>
> +<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/utils/TableGen/CodeGenTarget.cpp<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/TableGen/CodeGenTarget.cpp?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/TableGen/CodeGenTarget.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/utils/TableGen/CodeGenTarget.cpp (original)<br>
> +++ llvm/trunk/utils/TableGen/CodeGenTarget.cpp Thu Dec 4 03:40:44 2014<br>
> @@ -534,7 +534,8 @@ CodeGenIntrinsic::CodeGenIntrinsic(Recor<br>
> // variants with iAny types; otherwise, if the intrinsic is not<br>
> // overloaded, all the types can be specified directly.<br>
> assert(((!TyEl->isSubClassOf("LLVMExtendedType") &&<br>
> - !TyEl->isSubClassOf("LLVMTruncatedType")) ||<br>
> + !TyEl->isSubClassOf("LLVMTruncatedType") &&<br>
> + !TyEl->isSubClassOf("LLVMVectorSameWidth")) ||<br>
> VT == MVT::iAny || VT == MVT::vAny) &&<br>
> "Expected iAny or vAny type");<br>
> } else<br>
><span class="apple-converted-space"> </span><br>
> Modified: llvm/trunk/utils/TableGen/IntrinsicEmitter.cpp<br>
> URL:<span class="apple-converted-space"> </span><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/TableGen/IntrinsicEmitter.cpp?rev=223348&r1=223347&r2=223348&view=diff"><span style="color:purple">http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/TableGen/IntrinsicEmitter.cpp?rev=223348&r1=223347&r2=223348&view=diff</span></a><br>
> ==============================================================================<br>
> --- llvm/trunk/utils/TableGen/IntrinsicEmitter.cpp (original)<br>
> +++ llvm/trunk/utils/TableGen/IntrinsicEmitter.cpp Thu Dec 4 03:40:44 2014<br>
> @@ -257,7 +257,8 @@ enum IIT_Info {<br>
> IIT_ANYPTR = 26,<br>
> IIT_V1 = 27,<br>
> IIT_VARARG = 28,<br>
> - IIT_HALF_VEC_ARG = 29<br>
> + IIT_HALF_VEC_ARG = 29,<br>
> + IIT_SAME_VEC_WIDTH_ARG = 30<br>
> };<br>
><span class="apple-converted-space"> </span><br>
><span class="apple-converted-space"> </span><br>
> @@ -305,6 +306,13 @@ static void EncodeFixedType(Record *R, s<br>
> Sig.push_back(IIT_TRUNC_ARG);<br>
> else if (R->isSubClassOf("LLVMHalfElementsVectorType"))<br>
> Sig.push_back(IIT_HALF_VEC_ARG);<br>
> + else if (R->isSubClassOf("LLVMVectorSameWidth")) {<br>
> + Sig.push_back(IIT_SAME_VEC_WIDTH_ARG);<br>
> + Sig.push_back((Number << 2) | ArgCodes[Number]);<br>
> + MVT::SimpleValueType VT = getValueType(R->getValueAsDef("ElTy"));<br>
> + EncodeFixedValueType(VT, Sig);<br>
> + return;<br>
> + }<br>
> else<br>
> Sig.push_back(IIT_ARG);<br>
> return Sig.push_back((Number << 2) | ArgCodes[Number]);<br>
><span class="apple-converted-space"> </span><br>
><span class="apple-converted-space"> </span><br>
> _______________________________________________<br>
> llvm-commits mailing list<br>
><span class="apple-converted-space"> </span><a href="mailto:llvm-commits@cs.uiuc.edu"><span style="color:purple">llvm-commits@cs.uiuc.edu</span></a><br>
><span class="apple-converted-space"> </span><a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits"><span style="color:purple">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</span></a></span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">---------------------------------------------------------------------<br>
Intel Israel (74) Limited<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">This e-mail and any attachments may contain confidential material for<br>
the sole use of the intended recipient(s). Any review or distribution<br>
by others is strictly prohibited. If you are not the intended<br>
recipient, please contact the sender and delete all copies.<o:p></o:p></span></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p>---------------------------------------------------------------------<br>
Intel Israel (74) Limited</p>
<p>This e-mail and any attachments may contain confidential material for<br>
the sole use of the intended recipient(s). Any review or distribution<br>
by others is strictly prohibited. If you are not the intended<br>
recipient, please contact the sender and delete all copies.</p></body>
</html>