<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Book Antiqua";
panose-1:2 4 6 2 5 3 5 3 3 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Book Antiqua",serif;
color:#943634;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-IE link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-family:"Book Antiqua",serif;color:#943634;mso-fareast-language:EN-US'>That’s how the pattern is expressed, but it selects a single instruction. Not sure how TableGen does this, but it does match that particular IR pattern to the single instruction match without actually performing the element-by-element extract. It is crude though, and is very particular about specific IR structures. If it was elevated to a true ISD node supported by a C++ expressed IR normalisation, then it would catch a wider number of variants such as the classic loop for you show. Other patterns simply reduce to element extract and scalar ADD, but my pattern catches a common case. Horizontal patterns as first class citizens would catch a broader range of IR forms.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Book Antiqua",serif;color:#943634;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Book Antiqua",serif;color:#943634;mso-fareast-language:EN-US'> MartinO<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Book Antiqua",serif;color:#943634;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'> Rail Shafigulin [mailto:rail@esenciatech.com] <br><b>Sent:</b> 18 May 2016 18:55<br><b>To:</b> Martin J. O'Riordan <martin.oriordan@movidius.com><br><b>Cc:</b> LLVM Developers <llvm-dev@lists.llvm.org><br><b>Subject:</b> Re: [llvm-dev] sum elements in the vector<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>On Wed, May 18, 2016 at 5:56 AM, Martin J. O'Riordan <<a href="mailto:martin.oriordan@movidius.com" target="_blank">martin.oriordan@movidius.com</a>> wrote:<o:p></o:p></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'>Hi Rail,</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'>We used a very simple pattern expansion (actually, not a pattern fragment). For example, for AND, ADD (horizontal sum), OR and XOR of 4 elements we use something like the following TableGen structure:</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt'><span style='font-family:"Courier New";color:black'>class HORIZ_Op4<SDNode opc, RegisterClass regVT, ValueType rt, ValueType vt, string asmstr> :</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt'><span style='font-family:"Courier New";color:black'> SHAVE_Instr<(outs regVT:$dst), (ins VRF128:$src),</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt'><span style='font-family:"Courier New";color:black'> !strconcat(asmstr, " $dst $src"),</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt'><span style='font-family:"Courier New";color:black'> [(set regVT:$dst,</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt'><span style='font-family:"Courier New";color:black'> (opc (rt (vector_extract(vt VRF128:$src), 0 ) ),</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt'><span style='font-family:"Courier New";color:black'> (opc (rt (vector_extract(vt VRF128:$src), 1 ) ),</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt'><span style='font-family:"Courier New";color:black'> (opc (rt (vector_extract(vt VRF128:$src), 2 ) ),</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt'><span style='font-family:"Courier New";color:black'> (rt (vector_extract(vt VRF128:$src), 3 ) )</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt'><span style='font-family:"Courier New";color:black'> )</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt'><span style='font-family:"Courier New";color:black'> )</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt'><span style='font-family:"Courier New";color:black'> )</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt'><span style='font-family:"Courier New";color:black'> )]>;</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'>This is okay for 4 element vectors, and it will get selected if the programmer writes something like:</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt'><span style='font-family:"Courier New";color:black'>vec[0] & vec[1] & vec[2] & vec[3]</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'>but not with a simple variant like:</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt'><span style='font-family:"Courier New";color:black'>vec[0] & vec[2] & vec[1] & vec[3]</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'>If this was properly represented by an ISD node, the other permutations could be more easily handled through normalisation. We “could” write patterns for each of the permutations, but it is verbose, and in practice most people only write it one way anyway.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'>The 8-lane equivalent has TableGen left thinking for quite a long time, and the 16-lane equivalent seems to hang TableGen.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'> MartinO</span><o:p></o:p></p></div></div></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Martin,<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Thanks for the reply. If I read a pattern correctly (and I'm not sure if I do) then you are extracting data from the vector first and then perform an operation. What I'm trying to place data into a vector and perform and operation. Here is what I'm talking about:<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Convert the following:<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>int a[] = {1, 2, 3, 4};<o:p></o:p></p></div><div><p class=MsoNormal>int sum = 0;<o:p></o:p></p></div><div><p class=MsoNormal>for (int i = 0; i < 4; i++)<o:p></o:p></p></div><div><p class=MsoNormal> sum+= a[i];<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>into<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>vector.load vector.register.0, addressOfA<o:p></o:p></p></div><div><p class=MsoNormal>horizontal.add gpr.0, vector.register.0<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Original thought was to match a pattern of adds and then use insert_elt instruction, but this solution doesn't produce a good result, since it uses more instructions than a chain of adds. Do you think there is a simple solution to the problem that I'm suggesting or I have to make some major code changes? As I said before, my experience with all of this is very limited, so any help is greatly appreciated.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>P.S. I wasn't able to find anything related to SHAVE_Instr in the LLVM trunk. Are you guys not committing your work.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'> Rail Shafigulin [mailto:<a href="mailto:rail@esenciatech.com" target="_blank">rail@esenciatech.com</a>] <br><b>Sent:</b> 16 May 2016 23:50<br><b>To:</b> Martin J. O'Riordan<br><b>Cc:</b> LLVM Developers</span><o:p></o:p></p><div><div><p class=MsoNormal><br><b>Subject:</b> Re: [llvm-dev] sum elements in the vector<o:p></o:p></p></div></div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Mon, May 16, 2016 at 3:11 AM, Martin J. O'Riordan via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'>This would be really cool. We have several instructions that perform horizontal vector operations, and have to use built-ins to select them as there is no easy way of expressing them in a TD file. Some like SUM for a ‘</span><span style='font-family:"Courier New";color:black'>v4i32</span><span style='font-family:"Book Antiqua",serif;color:#1F497D'>’ are easy enough to express with a pattern fragment,</span><o:p></o:p></p></div></div></blockquote><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Do you mind sharing how to do it with a pattern fragment? I'm not new to TD files but all the work I've done was very simple. <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'>SUM ‘</span><span style='font-family:"Courier New";color:black'>v8i16</span><span style='font-family:"Book Antiqua",serif;color:#1F497D'>’ takes TableGen a long time to compute, but SUM ‘</span><span style='font-family:"Courier New";color:black'>v16i8</span><span style='font-family:"Book Antiqua",serif;color:#1F497D'>’ resulted in TableGen disappearing into itself for hours trying to reduce the patterns before I gave up and cancelled it.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'>If there were ISD nodes for these, then it would be far simpler to express in TableGen, and also, the pattern fragments only match a very specific form of IR to the desired instruction.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'>The horizontal operations are particularly useful for finalising a vectorised operation - for example I may want to compute the scalar MAX, MIN or SUM of a large number of items. If the number of items is divisible by the vector lanes (e.g. 4, 8, or 16 in our case), then 4, 8 or 16 at a time can be computed using normal vector operation, and then the final scalar value can be computed using a single horizontal operation.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'> MartinO</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Book Antiqua",serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'> llvm-dev [mailto:<a href="mailto:llvm-dev-bounces@lists.llvm.org" target="_blank">llvm-dev-bounces@lists.llvm.org</a>] <b>On Behalf Of </b>Chandler Carruth via llvm-dev<br><b>Sent:</b> 16 May 2016 2:16<br><b>To:</b> Shahid, Asghar-ahmad; Rail Shafigulin; llvm-dev; Hal Finkel</span><o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><br><b>Subject:</b> Re: [llvm-dev] sum elements in the vector<o:p></o:p></p></div></div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I'm starting to think we should directly implement horizontal operations on vector types.<o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>My suspicion is that coming up with a nice model for this would help us a lot with things like:<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>- Idiom recognition of reduction patterns that use horizontal arithmetic<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>- Ability to use horizontal operations in SLPVectorizer<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>- Significantly easier cost modeling of vectorizing loops with reductions in LoopVectorize<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>- Other things I've not thought of?<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><br><span style='font-family:"Helvetica",sans-serif'> Curious what others think?</span><o:p></o:p></p></div></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Helvetica",sans-serif'>-Chandler</span><o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Wed, May 11, 2016 at 10:07 PM Shahid, Asghar-ahmad via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>></span><span lang=EN-US> why in order to add this particular instruction (sum elements in a vector) I need to add an insrinsic?</span><o:p></o:p></p></div></div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Adding intrinsic is not the only way, it is one of the way and user WILL-NOT be required to invoke</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>It specifically.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Currently LLVM does not have any instruction to directly represent “sum of elements in a vector” and</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>generate your particular instruction.However, you can do it without intrinsic by pattern matching the</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>LLVM-IRs representing “sum of elements in vector” to your particular instruction in DAGCombiner.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Regards,</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Shahid</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'> </span><o:p></o:p></p><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma",sans-serif'>From:</span></b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma",sans-serif'> Rail Shafigulin [mailto:<a href="mailto:rail@esenciatech.com" target="_blank">rail@esenciatech.com</a>] <br><b>Sent:</b> Monday, May 09, 2016 11:59 PM<br><b>To:</b> Shahid, Asghar-ahmad; llvm-dev<br><b>Cc:</b> Das, Dibyendu</span><o:p></o:p></p></div></div></div></div></div><div><div><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma",sans-serif'><br><b>Subject:</b> Re: [llvm-dev] sum elements in the vector</span><o:p></o:p></p></div></div></div></div></div><div><div><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>I'm a little confused. Here is why.</span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>I was able to add a vector add instruction to my target without using any intrinsics and without adding any new instructions to LLVM. So here is my question: how come I managed to add a new vector instruction without adding an intrinsic and why in order to add this particular instruction (sum elements in a vector) I need to add an insrinsic?</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Another question that I have is whether compiler will be able to target this new instruction (sum elements in a vector) if it is implemented as an intrinsic or the user will have to specifically invoke an instrinsic.</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Pardon if questions seem dumb, I'm still learning things.</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Any help is appreciated. </span><o:p></o:p></p></div></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>On Fri, May 6, 2016 at 1:51 PM, Rail Shafigulin <<a href="mailto:rail@esenciatech.com" target="_blank">rail@esenciatech.com</a>> wrote:</span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Thanks for the reply. These steps will add an instruction as an intrinsic. Is it possible to add an actual new instruction so that a compiler could target it during an optimization? How hard is it to do it? Is that a realistic objective.</span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='color:#888888'>Rail</span><o:p></o:p></p></div></div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>On Mon, Apr 4, 2016 at 9:02 PM, Shahid, Asghar-ahmad <<a href="mailto:Asghar-ahmad.Shahid@amd.com" target="_blank">Asghar-ahmad.Shahid@amd.com</a>> wrote:</span><o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>Hi Rail,</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>We had done this for generation of X86 PSAD (sum of absolute difference) instruction through</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>Llvm intrinsic. Doing this requires following</span><o:p></o:p></p><p><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>1.</span><span lang=EN-US style='font-size:7.0pt;color:#1F497D'> </span><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>Define an intrinsic, xyz(), for the required instruction and corresponding SDNode</span><o:p></o:p></p><p><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>2.</span><span lang=EN-US style='font-size:7.0pt;color:#1F497D'> </span><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>Generate the “call xyz() “ IR based the matched pattern</span><o:p></o:p></p><p><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>3.</span><span lang=EN-US style='font-size:7.0pt;color:#1F497D'> </span><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>Map “call xyz()” IR to corresponding SDNode in SelectionDagBuilder.cpp</span><o:p></o:p></p><p><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>4.</span><span lang=EN-US style='font-size:7.0pt;color:#1F497D'> </span><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>Provide default expansion of the xyz() intrinsic</span><o:p></o:p></p><p><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>5.</span><span lang=EN-US style='font-size:7.0pt;color:#1F497D'> </span><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>Legalize type and/or operation</span><o:p></o:p></p><p><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>6.</span><span lang=EN-US style='font-size:7.0pt;color:#1F497D'> </span><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>Provide Lowering of intrinsic/SDNode to generate your target instruction</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>You can visit <a href="http://llvm.org/docs/ExtendingLLVM.html" target="_blank">http://llvm.org/docs/ExtendingLLVM.html</a> for details.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>Regards,</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>Shahid </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'> </span><o:p></o:p></p><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma",sans-serif'>From:</span></b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma",sans-serif'> llvm-dev [mailto:<a href="mailto:llvm-dev-bounces@lists.llvm.org" target="_blank">llvm-dev-bounces@lists.llvm.org</a>] <b>On Behalf Of </b>Rail Shafigulin via llvm-dev<br><b>Sent:</b> Monday, April 04, 2016 11:00 PM<br><b>To:</b> Das, Dibyendu<br><b>Cc:</b> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br><b>Subject:</b> Re: [llvm-dev] sum elements in the vector</span><o:p></o:p></p></div></div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Thanks for the pointers. I looked at hadd instructions. They seem to do very similar to what I need. Unfortunately as I said before my LLVM experience is limited. My understanding is that when I create a new type of SDNode I need to specify a pattern for it, so that when LLVM is analyzing the code and is seeing a given pattern it would create this particular node. I'm really struggling to understand how it is done. So here are the problems that I'm having.</span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>1. How do I identify that pattern that should be used?</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>2. How do I specify a given pattern? </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Do you (or someone else) mind helping me out? </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Any help is appreciated.</span><o:p></o:p></p></div></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>On Mon, Apr 4, 2016 at 9:59 AM, Das, Dibyendu <<a href="mailto:Dibyendu.Das@amd.com" target="_blank">Dibyendu.Das@amd.com</a>> wrote:</span><o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>This is roughly along the lines of x86 hadd* instructions though the semantics of hadd* may not exactly match what you are looking for. This is probably more in line with x86/ARM SAD-like instructions but I don’t think llvm generates SAD without intrinsics.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'> llvm-dev [mailto:<a href="mailto:llvm-dev-bounces@lists.llvm.org" target="_blank">llvm-dev-bounces@lists.llvm.org</a>] <b>On Behalf Of </b>Rail Shafigulin via llvm-dev<br><b>Sent:</b> Monday, April 04, 2016 9:34 AM<br><b>To:</b> llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br><b>Subject:</b> [llvm-dev] sum elements in the vector</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>My target has an instruction that adds up all elements in the vector and stores the result in a register. I'm trying to implement it in my compiler but I'm not sure even where to start.</span><o:p></o:p></p><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>I did look at other targets, but they don't seem to have anything like it ( I could be wrong. My experience with LLVM is limited, so if I missed it, I'd appreciate if someone could point it out ). </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>My understanding is that if SDNode for such an instruction doesn't exist I have to define one. Unfortunately, I don't know how to do it. I don't even know where to start looking. Would someone care to point me in the right direction? </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Any help is appreciated.<br clear=all></span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>-- </span><o:p></o:p></p><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Rail Shafigulin</span><o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Software Engineer <br>Esencia Technologies</span><o:p></o:p></p></div></div></div></div></div></div></div></div></div></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US><br><br clear=all></span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>-- </span><o:p></o:p></p><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Rail Shafigulin</span><o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Software Engineer <br>Esencia Technologies</span><o:p></o:p></p></div></div></div></div></div></div></div></div></div></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US><br><br clear=all></span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>-- </span><o:p></o:p></p><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Rail Shafigulin</span><o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Software Engineer <br>Esencia Technologies</span><o:p></o:p></p></div></div></div></div></div></div></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US><br><br clear=all></span><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>-- </span><o:p></o:p></p><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Rail Shafigulin</span><o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Software Engineer <br>Esencia Technologies</span><o:p></o:p></p></div></div></div></div></div></div></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>_______________________________________________<br>LLVM Developers mailing list<br><a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><o:p></o:p></p></blockquote></div></div></div></div></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;margin-bottom:12.0pt'><br>_______________________________________________<br>LLVM Developers mailing list<br><a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><o:p></o:p></p></blockquote></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><br><br clear=all><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>-- <o:p></o:p></p><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Rail Shafigulin<o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Software Engineer <br>Esencia Technologies<o:p></o:p></p></div></div></div></div></div></div></div></div></div></div></blockquote></div><p class=MsoNormal><br><br clear=all><o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal>-- <o:p></o:p></p><div><div><div><div><div><p class=MsoNormal>Rail Shafigulin<o:p></o:p></p></div><p class=MsoNormal>Software Engineer <br>Esencia Technologies<o:p></o:p></p></div></div></div></div></div></div></div></body></html>