<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi all,<div dir="ltr" class=""></div><div class=""><br class=""></div><div class="">After the previous RFC[1], there were multiple discussions on the ML and in person at the DevMtg. I will summarize the options discussed and propose a path forward.</div><div class=""><br class=""></div><div class=""><font face="Menlo" class="">===========================</font></div><div class="">Options</div><div class=""><span style="font-family: Menlo;" class="">===========================</span></div><div class=""><br class=""></div><div class="">A. Extend VectorType to be multidimensional</div><div class=""><br class=""></div><div class="">B. Flatten matrices into the current VectorType. Matrix shape and layout information is passed to matrix intrinsics. All matrix operations including element-wise matrix operations are implemented via intrinsics</div><div class=""><br class=""></div><div class="">C. Same as B but padding is explicitly managed by shufflevector instructions, element-wise operations are implemented via built-in operators (e.g. fadd)</div><div class=""><br class=""></div><div class=""><span style="font-family: Menlo;" class="">===========================</span></div><div class="">tl;dr</div><div class=""><span style="font-family: Menlo;" class="">===========================</span></div><div class=""><br class=""></div><div class="">There was some support for option A to introduce first-class matrices (or multidimensional vectors) but also many concerns. I have sketched out many examples in IR and flattening matrices to vectors does not seem to present any clear show-stoppers. Thus, I am scaling back the proposal to option B which is a more incremental step. </div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class="">Once option B is completed, we can reevaluate. If there is evidence that first-class type support is necessary we can change course (option A). Otherwise, we can use this stage as a performance baseline and start incrementally replacing the intrinsics with a combination of shufflevectors and built-in operators in order to experiment with option C. </span></div></div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class="">Throughout this work, an important goal is to provide a matrix-aware IRBuilder API. E.g.:</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 20.3px;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class=""><br class=""></span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class=""> Value *CreateMatrixAdd(<span style="color: rgb(2, 37, 199);" class="">Value</span> *<span style="color: rgb(202, 48, 199);" class="">Op0</span>, <span style="color: rgb(2, 37, 199);" class="">Value</span> <span style="color: rgb(202, 48, 199);" class="">Op1</span>,</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class=""> <span style="color: rgb(2, 37, 199);" class="">unsigned</span> <span style="color: rgb(202, 48, 199);" class="">Rows</span>, <span style="color: rgb(2, 37, 199);" class="">unsigned</span> <span style="color: rgb(202, 48, 199);" class="">Cols</span>,</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 197, 199);" class=""><span style="background-color: rgba(255, 255, 255, 0);" class=""><span style="color: rgb(0, 0, 0);" class=""> </span><span style="color: rgb(2, 37, 199);" class="">MatrixLayout</span><span style="color: rgb(0, 0, 0);" class=""> </span><span style="color: rgb(202, 48, 199);" class="">ML</span><span style="color: rgb(0, 0, 0);" class=""> </span>/* row/column-major, padding */<span style="color: rgb(0, 0, 0);" class="">);</span></span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 197, 199);" class=""><span style="color: rgb(0, 0, 0); background-color: rgba(255, 255, 255, 0);" class=""><br class=""></span></div></div></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class="">This will allow for simpler front-ends and would allow us to swap out the generated IR if the design needs to change.</span></div></div><div class=""><br class=""></div><div class=""><span style="font-family: Menlo;" class="">===========================</span></div><div class="">Details </div><div class=""><span style="font-family: Menlo;" class="">===========================</span></div><div class=""><br class=""></div><div class=""><span style="font-size: 13px;" class=""><font face="Menlo" class="">-------------------------</font></span></div><div class="">Introduction</div><div class=""><span style="font-family: Menlo; font-size: 13px;" class="">-------------------------</span></div><div class=""><br class=""></div><div class="">Representing multi-dimensional vectors in the IR through types makes the IR more expressive (<b class="">option A</b>). Additionally, if we have a new type we have the freedom to implicitly map it to a layout. E.g. <3 x 3 x float> could imply column-major order and one element of padding between each column. When it’s passed or returned from functions it should be passed in 3 vector registers.</div><div class=""><div style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">This is a sample IR to add two 3x3 matrices followed by a matrix multiply with a 3x2:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""> <span style="color: rgb(202, 48, 199);" class="">%a</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <3 x 3 x float>, <3 x 3 x float>* <span style="color: rgb(202, 48, 199);" class="">%A</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""> <span style="color: rgb(202, 48, 199);" class="">%b</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <3 x 3 x float>, <3 x 3 x float>* <span style="color: rgb(202, 48, 199);" class="">%B</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""> <span style="color: rgb(202, 48, 199);" class="">%c</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <3 x 2 x float>, <3 x 2 x float>* <span style="color: rgb(202, 48, 199);" class="">%C</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""> <span style="color: rgb(202, 48, 199);" class="">%add</span> = <span style="color: rgb(201, 27, 0);" class="">fadd</span> <3 x 3 x float> <span style="color: rgb(202, 48, 199);" class="">%a</span>, <span style="color: rgb(202, 48, 199);" class="">%b</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""> <span style="color: rgb(202, 48, 199);" class="">%mul</span> = <span style="color: rgb(201, 27, 0);" class="">call</span> <3 x 2 x float> @llvm.matrix.multiply(<3 x 3 x float> <span style="color: rgb(202, 48, 199);" class="">%add</span>,</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""> <3 x 2 x float> <span style="color: rgb(202, 48, 199);" class="">%c</span>)</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""> <span style="color: rgb(201, 27, 0);" class="">store</span><span class="Apple-tab-span" style="white-space: pre;"> </span><3 x 2 x float><span class="Apple-tab-span" style="white-space: pre;"> </span><span style="color: rgb(202, 48, 199);" class="">%mul</span>, <3 x 2 x float>* <span style="color: rgb(202, 48, 199);" class="">%MUL</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="color: rgb(202, 48, 199);" class=""><br class=""></span></div></div></div></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">Note that the type always implies a layout here. If we have multiple layouts appear in the same module beyond the default specified by DataLayout, we would have these represented in the type, e.g. <3 x 3 x float column-major pad(1)> specifying column-major layout with a single element of padding after each 3-element column vector.</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">Also note that we’re using built-in fadd operator for element-wise operation and an intrinsic for non-element-wise operations like matrix multiply.</div><div style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">Instead of extending the type system, we can map the matrix instances onto existing types. The vector type is a natural fit as it can be considered a SequenceType with Row*Column elements of the element type. For operations, like matrix multiply we just need to pass the shape information for the extra dimension.</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">One question arises though, how padding should be handled. For one, performing operations like division with the padding can cause spurious faults. But even for non-trapping operations excluding padding should be an option. For example in the case of a <3 x 3 x double>, we may want to lower a single row/column into the combination of a 128B vector register (2 elts) and a scalar rather than two vectors. This should be more beneficial for power. Thus we want to make padding explicit in the IR.</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">One option is to expose the shape to all operations including element-wise operations. This is <b class="">option B</b>. With that, the above sequence looks like this:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><br class=""></div><div style="font-family: Menlo; font-size: 11px; margin: 0px; font-stretch: normal; line-height: normal;" class=""> <span style="color: rgb(202, 48, 199);" class="">%a</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <b class=""><12 x float></b>, <12 x float>* <span style="color: rgb(202, 48, 199);" class="">%A</span>, align 16</div><div style="font-family: Menlo; font-size: 11px; margin: 0px; font-stretch: normal; line-height: normal;" class=""> <span style="color: rgb(202, 48, 199);" class="">%b</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <12 x float>, <12 x float>* <span style="color: rgb(202, 48, 199);" class="">%B</span>, align 16</div><div style="font-family: Menlo; font-size: 11px; margin: 0px; font-stretch: normal; line-height: normal;" class=""> <span style="color: rgb(202, 48, 199);" class="">%c</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <8 x float>, <8 x float>* <span style="color: rgb(202, 48, 199);" class="">%C</span>, align 16</div><div style="font-family: Menlo; font-size: 11px; margin: 0px; font-stretch: normal; line-height: normal; min-height: 13px;" class=""><br class=""></div><div style="font-family: Menlo; font-size: 11px; margin: 0px; font-stretch: normal; line-height: normal;" class=""><b class=""> <span style="color: rgb(202, 48, 199);" class="">%add</span> = <span style="color: rgb(201, 27, 0);" class="">call</span> <12 x float> @llvm.matrix.fadd(<12 x float> <span style="color: rgb(202, 48, 199);" class="">%a</span>, <12 x float> <span style="color: rgb(202, 48, 199);" class="">%b</span>,</b></div><div style="font-family: Menlo; font-size: 11px; margin: 0px; font-stretch: normal; line-height: normal;" class=""><b class=""> <span class="Apple-tab-span" style="white-space: pre;"> </span> <span class="Apple-tab-span" style="white-space: pre;"> </span> <span class="Apple-tab-span" style="white-space: pre;"> </span> <span class="Apple-tab-span" style="white-space: pre;"> </span> <span class="Apple-tab-span" style="white-space: pre;"> </span> <span style="color: rgb(0, 197, 199);" class="">; 3 x 3 column-major:</span></b></div><div style="font-family: Menlo; font-size: 11px; margin: 0px; font-stretch: normal; line-height: normal;" class=""><b class=""> <span style="color: rgb(2, 37, 199);" class="">i32</span> 3, <span style="color: rgb(2, 37, 199);" class="">i32</span> 3, <span style="color: rgb(2, 37, 199);" class="">i1</span> <span style="color: rgb(201, 27, 0);" class="">true</span>)</b></div><div style="font-family: Menlo; font-size: 11px; margin: 0px; font-stretch: normal; line-height: normal; min-height: 13px;" class=""><br class=""></div><div style="font-family: Menlo; font-size: 11px; margin: 0px; font-stretch: normal; line-height: normal;" class=""> <span style="color: rgb(202, 48, 199);" class="">%mul</span> = <span style="color: rgb(201, 27, 0);" class="">call</span> <8 x float> @llvm.matrix.multiply(<12 x float> <span style="color: rgb(202, 48, 199);" class="">%add</span>, <8 x float> <span style="color: rgb(202, 48, 199);" class="">%c</span>,</div><div style="font-family: Menlo; font-size: 11px; margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 197, 199);" class=""><span style="color: rgb(0, 0, 0);" class=""><span class="Apple-tab-span" style="white-space: pre;"> </span> </span>; 3 x 3 3 x 2 column-major:</div><div style="font-family: Menlo; font-size: 11px; margin: 0px; font-stretch: normal; line-height: normal;" class=""> <span style="color: rgb(2, 37, 199);" class="">i32</span> 3, <span style="color: rgb(2, 37, 199);" class="">i32</span> 3, <span style="color: rgb(2, 37, 199);" class="">i32</span> 3, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i1</span> <span style="color: rgb(201, 27, 0);" class="">true</span>)</div><div style="font-family: Menlo; font-size: 11px; margin: 0px; font-stretch: normal; line-height: normal;" class=""> <span style="color: rgb(201, 27, 0);" class="">store</span> <8 x float> <span style="color: rgb(202, 48, 199);" class="">%mul</span>, <8 x float>* <span style="color: rgb(202, 48, 199);" class="">%MUL</span>, align 16</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""> </div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">Each computation takes full shape information. The matrix shape is described with the row and columns dimensions and are passed to the intrinsic as constant parameters. We can also pass layout information like whether the matrices are laid out in row-major or column-major order. We’re using column major order in the example and as such the 3 x 3 x float matrix is flattened into a 12 x float vector with one element padding at the end of each column.</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">The amount of padding does not require any new parameters. We can compute it using the shape information and the size of the flattened matrix (e.g. %c which is a <3 x 2 x float> also has one element of padding: Elts / Columns - Rows = 8 / 2 - 3 = 1).</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">In order to expose the padding elements to element-wise operation (fadd), option B maps those to intrinsics. We can expose the padding bytes in other ways such that we can still use the built-in element-wise operators. One way would be to extend the vector types with specifying the padding, something like <12 x float pad(3, 7, 11)> (John McCall’s idea) or removing the padding with explicit shufflevectors (Chandler’s idea). I explored the lattter under <b class="">option C</b>. With option C, the same sequence of operations look like this:</div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(202, 48, 199);" class="">%a.padded</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <12 x float>, <12 x float>* <span style="color: rgb(202, 48, 199);" class="">%A</span>, align 16</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 197, 199);" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; remove padding</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""><b class=""> <span style="color: rgb(202, 48, 199);" class="">%a</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <12 x float> <span style="color: rgb(202, 48, 199);" class="">%a.padded</span>, <12 x float> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</b></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""><b class=""> <9 x i32> <i32 0, <span style="color: rgb(2, 37, 199);" class="">i32</span> 1, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> 4, <span style="color: rgb(2, 37, 199);" class="">i32</span> 5, <span style="color: rgb(2, 37, 199);" class="">i32</span> 6, <span style="color: rgb(2, 37, 199);" class="">i32</span> 8, <span style="color: rgb(2, 37, 199);" class="">i32</span> 9, <span style="color: rgb(2, 37, 199);" class="">i32</span> 10></b></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(202, 48, 199);" class="">%b.padded</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <12 x float>, <12 x float>* <span style="color: rgb(202, 48, 199);" class="">%B</span>, align 16</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""><b class=""> <span style="color: rgb(202, 48, 199);" class="">%b</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <12 x float> <span style="color: rgb(202, 48, 199);" class="">%b.padded</span>, <12 x float> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</b></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""><b class=""> <9 x i32> <i32 0, <span style="color: rgb(2, 37, 199);" class="">i32</span> 1, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> 4, <span style="color: rgb(2, 37, 199);" class="">i32</span> 5, <span style="color: rgb(2, 37, 199);" class="">i32</span> 6, <span style="color: rgb(2, 37, 199);" class="">i32</span> 8, <span style="color: rgb(2, 37, 199);" class="">i32</span> 9, <span style="color: rgb(2, 37, 199);" class="">i32</span> 10></b></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(202, 48, 199);" class="">%c.padded</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <8 x float>, <8 x float>* <span style="color: rgb(202, 48, 199);" class="">%C</span>, align 16</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""><b class=""> <span style="color: rgb(202, 48, 199);" class="">%c</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <8 x float> <span style="color: rgb(202, 48, 199);" class="">%c.padded</span>, <8 x float> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</b></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""><b class=""> <6 x i32> <i32 0, <span style="color: rgb(2, 37, 199);" class="">i32</span> 1, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> 4, <span style="color: rgb(2, 37, 199);" class="">i32</span> 5, <span style="color: rgb(2, 37, 199);" class="">i32</span> 6></b></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""><b class=""> <span style="color: rgb(202, 48, 199);" class="">%add</span> = <span style="color: rgb(201, 27, 0);" class="">fadd</span> <9 x float> <span style="color: rgb(202, 48, 199);" class="">%a</span>, <span style="color: rgb(202, 48, 199);" class="">%b</span></b></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(202, 48, 199);" class="">%mul</span> = <span style="color: rgb(201, 27, 0);" class="">call</span> <6 x float> @llvm.matrix.multiply(<9 x float> <span style="color: rgb(202, 48, 199);" class="">%add</span>, <6 x float> <span style="color: rgb(202, 48, 199);" class="">%c</span>,</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(0, 197, 199);" class="">; 3 x 3 3 x 2 column-major:</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(2, 37, 199);" class="">i32</span> 3, <span style="color: rgb(2, 37, 199);" class="">i32</span> 3, <span style="color: rgb(2, 37, 199);" class="">i32</span> 3, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i1</span> <span style="color: rgb(201, 27, 0);" class="">true</span>)</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 197, 199);" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; add back padding</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""><b class=""> <span style="color: rgb(202, 48, 199);" class="">%mul.padded</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <6 x float> <span style="color: rgb(202, 48, 199);" class="">%mul</span>, <6 x float> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</b></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""><b class=""> <8 x i32> <i32 0, <span style="color: rgb(2, 37, 199);" class="">i32</span> 1, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span>, <span style="color: rgb(2, 37, 199);" class="">i32</span> 3, <span style="color: rgb(2, 37, 199);" class="">i32</span> 4, <span style="color: rgb(2, 37, 199);" class="">i32</span> 5, <span style="color: rgb(2, 37, 199);" class="">i32</span> undef></b></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(201, 27, 0);" class="">store</span><span class="Apple-tab-span" style="white-space: pre;"> </span><8 x float> <span style="color: rgb(202, 48, 199);" class="">%mul.padded</span>, <8 x float>* <span style="color: rgb(202, 48, 199);" class="">%MUL</span>, align 16</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""><br class=""></div></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">The expectation is that these shufflevectors will be removed as we lower to HW-sized vectors, e.g. using three 4-wide vector adds.</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><br class=""></div></div><div style="font-size: 14px;" class=""><span style="font-family: Menlo; font-size: 13px;" class="">------------------------- </span></div><div class="">Evaluation</div><div style="font-size: 14px;" class=""><span style="font-family: Menlo; font-size: 13px;" class="">------------------------- </span></div><div class=""><br class=""></div><div class="">While we have a prototype for option A which seems to work fine, I wanted to evaluate the other options and see how they measure up. I chose these criteria:</div><div class=""><ul class=""><li class="">Intrusiveness of the change</li><li class="">Matrix Operation Lowering and Fusion</li><li class="">Inlining to enable more fusion</li><li class="">Matrix HW, matrix library</li><li class="">Proper padding for small matrices</li><li class="">Pass and return small matrices in vector registers</li></ul></div><div class=""><div style="font-size: 14px;" class=""><span style="font-family: Menlo; font-size: 13px;" class="">------------------------- </span></div><div class="">Intrusiveness of the change</div><div class=""><div style="font-size: 14px;" class=""><span style="font-family: Menlo; font-size: 13px;" class="">------------------------- </span></div></div><div class=""><span style="font-family: Menlo; font-size: 13px;" class=""><br class=""></span></div><div class="">Admittedly, option A is the most intrusive. All the places where VectorType is handled would have to be audited and potentially extended to multiple dimensions. Operations like extract/insertelement would have to be modified to take an index list.</div><div class=""><br class=""></div><div class="">On the CodeGen side, legalization support for VectorTypes would have to be extended to multiple dimensions as well.</div><div class=""><br class=""></div><div class="">Option B and C are less intrusive. They are also fairly similar. One difference is that Option B replicates some of the built-in operators in intrinsics which may need dedicated support in some optimizations.</div><div class=""><br class=""></div></div><div style="font-size: 14px;" class=""><span style="font-family: Menlo; font-size: 13px;" class="">------------------------- </span></div><div class="">Matrix Operation Lowering and Fusion</div><div class=""><div style="font-size: 14px;" class=""><span style="font-family: Menlo; font-size: 13px;" class="">------------------------- </span></div></div><div class=""><span style="font-family: Menlo; font-size: 13px;" class=""><br class=""></span></div><div class="">Common to all of these options is that we are proposing a new IR pass that pre-legalizes matrix operations by lowering them to operations that are natively supported by the HW. This means decomposing the operations into native SIMD operations.</div><div class=""><br class=""></div><div class="">This pass will be used to also de-interleave a chain of matrix operations to manage register pressure. The idea is to analyze the chain and then work only on subsections of the matrices at a time to avoid the need of keeping everything in registers. </div><div class=""><br class=""></div><div class="">For simple lowering (not fusion), let’s consider a 3x2 addition followed by a matrix-multiply with a 2x4 for option A:</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco;" class=""><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="color: rgb(202, 48, 199);" class="">%a</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <3 x 2 x float>, <3 x 2 x float>* <span style="color: rgb(202, 48, 199);" class="">%A</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="color: rgb(202, 48, 199);" class="">%b</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <3 x 2 x float>, <3 x 2 x float>* <span style="color: rgb(202, 48, 199);" class="">%B</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="color: rgb(202, 48, 199);" class="">%c</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <2 x 4 x float>, <2 x 4 x float>* <span style="color: rgb(202, 48, 199);" class="">%C</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="color: rgb(202, 48, 199);" class="">%add</span> = <span style="color: rgb(201, 27, 0);" class="">fadd</span> <3 x 2 x float> <span style="color: rgb(202, 48, 199);" class="">%a</span>, <span style="color: rgb(202, 48, 199);" class="">%b</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="color: rgb(202, 48, 199);" class="">%mul</span> = <span style="color: rgb(201, 27, 0);" class="">call</span> <3 x 3 x float> @llvm.matrix.multiply(<3 x 2 x float> <span style="color: rgb(202, 48, 199);" class="">%add</span>,</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <2 x 4 x float> <span style="color: rgb(202, 48, 199);" class="">%c</span>)</div><div class=""><br class=""></div></div></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">We can lower this by replacing the matrix instructions with a set of instructions computing each HW-vector-sized chunk of each column (assuming column-major order and padded columns to a 16B alignment). We can do this in reverse-post order starting with the loads:</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 197, 199);" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; %a = load <3 x 2 x float>, <3 x 2 x float>* %A</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="color: rgb(202, 48, 199);" class="">%a0</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <4 x float>, <4 x float>* <span style="color: rgb(202, 48, 199);" class="">%A0</span>, align 16</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="color: rgb(202, 48, 199);" class="">%a1</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <4 x float>, <4 x float>* <span style="color: rgb(202, 48, 199);" class="">%A1</span>, align 16</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 197, 199);" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>;; ... similarly for b and c ...</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 197, 199);" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; %add = fadd <3 x 2 x float> %a, %b</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="color: rgb(202, 48, 199);" class="">%add0</span> = <span style="color: rgb(201, 27, 0);" class="">fadd</span> <4 x float> <span style="color: rgb(202, 48, 199);" class="">%a0</span>, <span style="color: rgb(202, 48, 199);" class="">%b0</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="color: rgb(202, 48, 199);" class="">%add1</span> = <span style="color: rgb(201, 27, 0);" class="">fadd</span> <4 x float> <span style="color: rgb(202, 48, 199);" class="">%a1</span>, <span style="color: rgb(202, 48, 199);" class="">%b1</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 197, 199);" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; %mul = call <3 x 4 x float> @llvm.matrix.multiply(<3 x 2 x float> %add,</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 197, 199);" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; <2 x 4 x float> %c)</div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(0, 0, 187);" class=""> ...</div></div><div class=""><br class=""></div><div class="">We obviously want to arrive to the same lowering for the other options too. The original IR for option B contains flattened matrices and intrinsics for all matrix operations:</div><div class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco;" class=""><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 197, 199);" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; 3 x 2 pad(1)</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(202, 48, 199);" class="">%a</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <8 x float>, <8 x float>* <span style="color: rgb(202, 48, 199);" class="">%A</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; 3 x 2 pad(1)</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(202, 48, 199);" class="">%b</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <8 x float>, <8 x float>* <span style="color: rgb(202, 48, 199);" class="">%B</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; 2 x 4 pad(0)</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(202, 48, 199);" class="">%c</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <8 x float>, <8 x float>* <span style="color: rgb(202, 48, 199);" class="">%C</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; 3 x 2 pad(1)</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(202, 48, 199);" class="">%add</span> = <span style="color: rgb(201, 27, 0);" class="">call</span> <8 x float> @llvm.matrix.fadd(<8 x float> <span style="color: rgb(202, 48, 199);" class="">%a</span>, <8 x float> <span style="color: rgb(202, 48, 199);" class="">%b</span>,</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(0, 197, 199);" class="">; 3 x 2</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(2, 37, 199);" class="">i32</span> 3, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2)</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; 3 x 4 pad(1)</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(202, 48, 199);" class="">%mul</span> = <span style="color: rgb(201, 27, 0);" class="">call</span> <16 x float> @llvm.matrix.multiply(<8 x float> <span style="color: rgb(202, 48, 199);" class="">%add</span>, <8 x float> <span style="color: rgb(202, 48, 199);" class="">%c</span>,</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(0, 197, 199);" class="">; 3 x 2 2 x 4</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(2, 37, 199);" class="">i32</span> 3, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> 4)</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""><br class=""></div></div></div></div><div class="">Note that we only have shape and layout information on computations. We don’t have them on other instructions like: load, store, phi, select, bitcast, memcpy intrinsic etc. Since the shape and layout information is critical to avoid unnecessary shuffles when working on rows/columns we need to recover this by propagating this information to all matrix operations.</div><div class=""><br class=""></div><div class="">This is a forward, backward data-flow problem. It is simplified because we only have two states: “unspecified" or "specified shape and layout information". The IR is malformed if operands don’t have consistent shape or layout (e.g. for addition they have to match; for matmul the inner dimensions need to match). Since we only revisit uses or defs as they transition from unspecified to specified we can only visit each instruction twice at most.</div><div class=""><br class=""></div><div class="">Option C is also has flattened matrices but shape and layout information (padding) is spread across computations and shufflevectors:</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 197, 199);" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; 3 x 2 pad(1)</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(202, 48, 199);" class="">%a.padded</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <8 x float>, <8 x float>* <span style="color: rgb(202, 48, 199);" class="">%A</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; 3 x 2 pad(1)</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(202, 48, 199);" class="">%b.padded</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <8 x float>, <8 x float>* <span style="color: rgb(202, 48, 199);" class="">%B</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; 2 x 4 pad(0)</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(202, 48, 199);" class="">%c</span> = <span style="color: rgb(201, 27, 0);" class="">load</span> <8 x float>, <8 x float>* <span style="color: rgb(202, 48, 199);" class="">%C</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(202, 48, 199);" class="">%a</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <8 x float> <span style="color: rgb(202, 48, 199);" class="">%a.padded</span>, <6 x i32> <i32 0, <span style="color: rgb(2, 37, 199);" class="">i32</span> 1, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> 4, <span style="color: rgb(2, 37, 199);" class="">i32</span> 5, <span style="color: rgb(2, 37, 199);" class="">i32</span> 6></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(202, 48, 199);" class="">%b</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <8 x float> <span style="color: rgb(202, 48, 199);" class="">%b.padded</span>, <6 x i32> <i32 0, <span style="color: rgb(2, 37, 199);" class="">i32</span> 1, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> 4, <span style="color: rgb(2, 37, 199);" class="">i32</span> 5, <span style="color: rgb(2, 37, 199);" class="">i32</span> 6></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; 3 x 2</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(202, 48, 199);" class="">%add</span> = <span style="color: rgb(201, 27, 0);" class="">fadd</span> <6 x float> <span style="color: rgb(202, 48, 199);" class="">%a</span>, <span style="color: rgb(202, 48, 199);" class="">%b</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; 3 x 4</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(202, 48, 199);" class="">%mul</span> = <span style="color: rgb(201, 27, 0);" class="">call</span> <12 x float> @llvm.matrix.multiply(<6 x float> <span style="color: rgb(202, 48, 199);" class="">%add</span>, <8 x float> <span style="color: rgb(202, 48, 199);" class="">%c</span>,</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""> <span style="color: rgb(0, 197, 199);" class="">; 3 x 2 2 x 4</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""><span class="Apple-tab-span" style="white-space: pre;"> </span><span style="color: rgb(2, 37, 199);" class="">i32</span> 3, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> 4)</div><div style="margin: 0px; font-stretch: normal; line-height: normal; color: rgb(0, 0, 0);" class=""><br class=""></div></div></div><div class="">Since the information is more spread around, the data-flow analysis becomes more complicated as we need to track shape and layout separately. Additionally, since optimizations can transform the shufflevectors, we may require more granular tracking of the layout which would make the analysis even more complex.</div><div class=""><br class=""></div><div class="">To summarize we have data flow problems to solve for option B and C in order to recover information that is directly represented in the IR for option A. Solving the data flow problem is also expected to be harder for option C than B.</div><div class=""><br class=""></div><div class="">Fusion opportunistically lowers larger subgraphs rather than individual operations . It needs the same shape information. If we fail to successfully recover this information for the entire subgraph, fusion won’t take place. It’s hard to foresee the potential problems here but I expect that the complexity and the failure rate of fusion would increase from option A to B and then to C.</div><div class=""><br class=""></div><div style="font-size: 14px;" class=""><span style="font-family: Menlo; font-size: 13px;" class="">------------------------- </span></div><div class="">Inlining to enable more fusion</div><div class=""><div style="font-size: 14px;" class=""><span style="font-family: Menlo; font-size: 13px;" class="">------------------------- </span></div></div><div style="font-size: 14px;" class=""><span style="font-family: Menlo; font-size: 13px;" class=""><br class=""></span></div><div class="">The goal is to make the inliner aware of fusion opportunities in order to build larger graphs. Additionally for option C we will have to tweak the profitability estimation logic to recognize and exclude the shufflevector instructions.</div><div style="font-size: 14px;" class=""><span style="font-family: Menlo; font-size: 13px;" class=""><br class=""></span></div><div style="font-size: 14px;" class=""><span style="font-family: Menlo; font-size: 13px;" class="">------------------------- </span></div><div class="">Matrix HW, matrix library</div><div class=""><div style="font-size: 14px;" class=""><span style="font-family: Menlo; font-size: 13px;" class="">------------------------- </span></div></div><div class=""><span style="font-family: Menlo; font-size: 13px;" class=""><br class=""></span></div><div class="">This can potentially pose very diverse requirements but essentially instead of decomposing matrix operations into efficient vector operations, the goal is to decompose them into matrix operation of a certain natively-supported shape. As outlined in the lowering section, since we can already establish the shape and layout information at each matrix operations, we should be able to decompose them into small matrix operations rather than going directly to vectors. Thus, this does not impose any new requirements, at least not in general. I am sure specific ports may have some interesting requirements in which case we may have to adapt the design.</div><div style="font-size: 14px;" class=""><span style="font-family: Menlo; font-size: 13px;" class=""><br class=""></span></div><div style="font-size: 14px;" class=""><span style="font-family: Menlo; font-size: 13px;" class="">------------------------- </span></div><div class="">Proper padding for small matrices</div><div class=""><div style="font-size: 14px;" class=""><span style="font-family: Menlo; font-size: 13px;" class="">------------------------- </span></div></div><div class=""><br class=""></div><div class="">We want the individual rows/columns (for row/column-major) to be naturally aligned by no more than the max vector alignment. E.g. for a 3 x 3 x float with column major order, we should have a single element padding after each column which is a total of 48 bytes. For option A, since it’s a new type we can just define this in the new ABI.</div><div class=""><br class=""></div><div class="">For option B and C, on the other hand, vector alignment and padding is already mandated by the VectorType. This is part of the ABI when people use vector_size or ext_vector_type attributes on the clang side.</div><div class=""><br class=""></div><div class="">Alignment and allocation size (including padding) is the original size of vector rounded up to the next power-of-2. So for example a 3 x 3 x float pad(1) or 12 x float is rounded up to 64 bytes. This is excessive. I also need to support unpadded matrices that are effectively ABI-compatible with arrays.</div><div class=""><br class=""></div><div class="">Front-ends could lower to arrays instead of vectors and then cast them to vectors specifying the proper alignment. This would complicate front-ends and the IR. A more reasonable solution would be allow reducing the alignment and in turn the padding of the vector type itself. We could have an align attribute on the type itself:</div><div class=""><br class=""></div><div class="">For example <12 x float align(16)> for naturally aligned columns or <9 x float align(1)> for the unpadded case.</div><div class=""><br class=""></div><div class="">In summary, option B and C can be made to work with some IR extensions.</div><div class=""><br class=""></div><div style="font-size: 14px;" class=""><span style="font-family: Menlo; font-size: 13px;" class="">------------------------- </span></div><div class="">Pass and return small matrices in vector registers</div><div class=""><div style="font-size: 14px;" class=""><span style="font-family: Menlo; font-size: 13px;" class="">------------------------- </span></div></div><div class=""><span style="font-family: Menlo; font-size: 13px;" class=""><br class=""></span></div><div class="">Typically (e.g. AArch64, x86_64) non-power-of-2-sized vectors are passed and returned in memory. We want small matrices like a 3 x 3 x float to be passed and returned in three vector registers assuming a vector width of 128 bits.</div><div class=""><br class=""></div><div class="">For option A, we can directly specify this in the new ABI. For option B and C however we need a new parameter attribute to enforce this. For example, ‘split_vec(N)’ below would specify that the vector argument is passed split up in chunks of N x <elt_type> in argument register where <elt_type> is the same as the element type of the vector argument.</div><div class=""><br class=""></div><div class="">For example, this is a function taking two and returning a 3x3 matrix (with option B syntax): </div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="color: rgb(201, 27, 0);" class="">define</span> <12 x float> split_vec(4) @f(<12 x float> split_vec(4) <span style="color: rgb(202, 48, 199);" class="">%a</span>, <12 x float> split_vec(4) <span style="color: rgb(202, 48, 199);" class="">%b</span>) {</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><font face="Menlo" class=""><span style="font-size: 11px;" class=""> </span></font><span style="font-family: Menlo; font-size: 11px; color: rgb(202, 48, 199);" class="">%c</span><font face="Menlo" class=""><span style="font-size: 11px;" class=""> = </span></font><span style="font-family: Menlo; font-size: 11px; color: rgb(201, 27, 0);" class="">call</span><font face="Menlo" class=""><span style="font-size: 11px;" class=""> <12 x float> @llvm.matrix.fadd(<12 x float> </span></font><span style="font-family: Menlo; font-size: 11px; color: rgb(202, 48, 199);" class="">%a</span><font face="Menlo" class=""><span style="font-size: 11px;" class="">, <12 x float> </span></font><span style="font-family: Menlo; font-size: 11px; color: rgb(202, 48, 199);" class="">%b</span><span style="font-family: Menlo; font-size: 11px;" class="">, </span><span style="font-family: Menlo; font-size: 11px; color: rgb(2, 37, 199);" class="">i32</span><span style="font-family: Menlo; font-size: 11px;" class=""> 3, </span><span style="font-family: Menlo; font-size: 11px; color: rgb(2, 37, 199);" class="">i32</span><span style="font-family: Menlo; font-size: 11px;" class=""> 3</span><font face="Menlo" class=""><span style="font-size: 11px;" class="">)</span></font></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="color: rgb(201, 27, 0);" class="">ret</span> <12 x float> <span style="color: rgb(202, 48, 199);" class="">%c</span></div></div><div class="">}</div><div class=""><br class=""></div><div class="">Let’s see how this is all lowered. First, the IR lowering pass splits up the flattened matrices in the body, arguments remain un-lowered:</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="color: rgb(201, 27, 0);" class="">define</span> <12 x float> split_vec(4) @f(<12 x float> split_vec(4) <span style="color: rgb(202, 48, 199);" class="">%a</span>, <12 x float> split_vec(4) <span style="color: rgb(202, 48, 199);" class="">%b</span>) {</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 197, 199);" class=""><b class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; Extract columns</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><b class=""> <span style="color: rgb(202, 48, 199);" class="">%a0</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <12 x float> <span style="color: rgb(202, 48, 199);" class="">%a</span>, <12 x float> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><b class=""> <4 x i32> <i32 0, <span style="color: rgb(2, 37, 199);" class="">i32</span> 1, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> undef></b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><b class=""> <span style="color: rgb(202, 48, 199);" class="">%a1</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <12 x float> <span style="color: rgb(202, 48, 199);" class="">%a</span>, <12 x float> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><b class=""> <4 x i32> <i32 4, <span style="color: rgb(2, 37, 199);" class="">i32</span> 5, <span style="color: rgb(2, 37, 199);" class="">i32</span> 6, <span style="color: rgb(2, 37, 199);" class="">i32</span> undef></b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><b class=""> <span style="color: rgb(202, 48, 199);" class="">%a2</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <12 x float> <span style="color: rgb(202, 48, 199);" class="">%a</span>, <12 x float> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><b class=""> <4 x i32> <i32 8, <span style="color: rgb(2, 37, 199);" class="">i32</span> 9, <span style="color: rgb(2, 37, 199);" class="">i32</span> 10, <span style="color: rgb(2, 37, 199);" class="">i32</span> undef></b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 197, 199);" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; same for b</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 197, 199);" class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; ...</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="color: rgb(202, 48, 199);" class="">%c0</span> = <span style="color: rgb(201, 27, 0);" class="">fadd</span> <4 x float> <span style="color: rgb(202, 48, 199);" class="">%a0</span>, <span style="color: rgb(202, 48, 199);" class="">%b0</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="color: rgb(202, 48, 199);" class="">%c1</span> = <span style="color: rgb(201, 27, 0);" class="">fadd</span> <4 x float> <span style="color: rgb(202, 48, 199);" class="">%a1</span>, <span style="color: rgb(202, 48, 199);" class="">%b1</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="color: rgb(202, 48, 199);" class="">%c2</span> = <span style="color: rgb(201, 27, 0);" class="">fadd</span> <4 x float> <span style="color: rgb(202, 48, 199);" class="">%a2</span>, <span style="color: rgb(202, 48, 199);" class="">%b2</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 197, 199);" class=""><b class=""><span style="color: rgb(0, 0, 0);" class=""> </span>; Insert columns</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><b class=""> <span style="color: rgb(202, 48, 199);" class="">%c01</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <4 x float> <span style="color: rgb(202, 48, 199);" class="">%c0</span>, <4 x float> <span style="color: rgb(202, 48, 199);" class="">%c1</span>,</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><b class=""> <8 x i32> <i32 0, <span style="color: rgb(2, 37, 199);" class="">i32</span> 1, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><b class=""> <span style="color: rgb(2, 37, 199);" class="">i32</span> 4, <span style="color: rgb(2, 37, 199);" class="">i32</span> 5, <span style="color: rgb(2, 37, 199);" class="">i32</span> 6, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span> ></b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><b class=""> <span style="color: rgb(202, 48, 199);" class="">%c2.w</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <4 x float> <span style="color: rgb(202, 48, 199);" class="">%c2</span>, <4 x float> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><b class=""> <8 x i32> <i32 0, <span style="color: rgb(2, 37, 199);" class="">i32</span> 1, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><b class=""> <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span>, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span>, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span>, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span> ></b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><b class=""> <span style="color: rgb(202, 48, 199);" class="">%c</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <8 x float> <span style="color: rgb(202, 48, 199);" class="">%c01</span>, <8 x float> <span style="color: rgb(202, 48, 199);" class="">%c2.w</span>,</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><b class=""> <12 x i32> <i32 0, <span style="color: rgb(2, 37, 199);" class="">i32</span> 1, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><b class=""> <span style="color: rgb(2, 37, 199);" class="">i32</span> 4, <span style="color: rgb(2, 37, 199);" class="">i32</span> 5, <span style="color: rgb(2, 37, 199);" class="">i32</span> 6, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><b class=""> <span style="color: rgb(2, 37, 199);" class="">i32</span> 8, <span style="color: rgb(2, 37, 199);" class="">i32</span> 9, <span style="color: rgb(2, 37, 199);" class="">i32</span> 10, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span> ></b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="color: rgb(201, 27, 0);" class="">ret</span> <12 x float> <span style="color: rgb(202, 48, 199);" class="">%c</span></div></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="color: rgb(202, 48, 199);" class="">}</span></div><div class=""><br class=""></div><div class="">Then during SDAG building, the arguments are split to map to ABI argument registers and a CONCAT_VECTOR is used to recreate the original un-split value (SDAG code in comments):</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""><span style="color: rgb(201, 27, 0);" class="">define</span> <12 x float> split_vec(4) @<span style="color: rgb(201, 27, 0);" class="">f</span>(<12 x float> split_vec(4) <span style="color: rgb(202, 48, 199);" class="">%a</span>, <12 x float> split_vec(4) <span style="color: rgb(202, 48, 199);" class="">%b</span>) {</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class="">; Register:v4f32 %0, %1, %2 Register:v4f32 %3, %4, %5</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class="">; t0: ch = EntryToken</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class="">; t4: v4f32,ch = CopyFromReg t0, Register:v4f32 %1</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class="">; t6: v4f32,ch = CopyFromReg t0, Register:v4f32 %2</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class="">; t8: v4f32,ch = CopyFromReg t0, Register:v4f32 %3</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class="">; t10: v4f32,ch = CopyFromReg t0, Register:v4f32 %4</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class="">; t12: v4f32,ch = CopyFromReg t0, Register:v4f32 %5</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class="">; t14: v4f32,ch = CopyFromReg t0, Register:v4f32 %6</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class=""><b class="">; t15: v12f32 = CONCAT_VECTORS t4, t6, t8</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class=""><b class="">; t16: v12f32 = CONCAT_VECTORS t10, t12, t14</b></div></div><div class=""><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class="">;------------</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""> <span style="color: rgb(202, 48, 199);" class="">%a0</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <12 x float> <span style="color: rgb(202, 48, 199);" class="">%a</span>, <12 x float> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""> <4 x i32> <i32 0, <span style="color: rgb(2, 37, 199);" class="">i32</span> 1, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> undef></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""> <span style="color: rgb(202, 48, 199);" class="">%a1</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <12 x float> <span style="color: rgb(202, 48, 199);" class="">%a</span>, <12 x float> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""> <4 x i32> <i32 4, <span style="color: rgb(2, 37, 199);" class="">i32</span> 5, <span style="color: rgb(2, 37, 199);" class="">i32</span> 6, <span style="color: rgb(2, 37, 199);" class="">i32</span> undef></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""> <span style="color: rgb(202, 48, 199);" class="">%a2</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <12 x float> <span style="color: rgb(202, 48, 199);" class="">%a</span>, <12 x float> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""> <4 x i32> <i32 8, <span style="color: rgb(2, 37, 199);" class="">i32</span> 9, <span style="color: rgb(2, 37, 199);" class="">i32</span> 10, <span style="color: rgb(2, 37, 199);" class="">i32</span> undef></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class=""><b class="">; t17: v4f32 = EXTRACT_SUBVECTOR t15, 0</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class=""><b class="">; t18: v4f32 = EXTRACT_SUBVECTOR t15, 4</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class=""><b class="">; t19: v4f32 = EXTRACT_SUBVECTOR t15, 8</b></div></div><div class=""><b class=""><br class=""></b></div><div class="">The CONCAT_VECTORs will be cancelled out by the EXTRACT_SUBVECTOR generated for the first set of shufflevector. So the argument registers will be forwarded directly to the adds.</div><div class=""><br class=""></div><div class="">Similarly on the return path, the CONCAT_VECTOR generated for the second set of shufflevectors will be cancelled out by the EXTRACT_SUBVECTOR that splits the vector across the registers for the return value:</div><div class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""> <span style="color: rgb(202, 48, 199);" class="">%c01</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <4 x float> <span style="color: rgb(202, 48, 199);" class="">%c0</span>, <4 x float> <span style="color: rgb(202, 48, 199);" class="">%c1</span>,</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""> <8 x i32> <i32 0, <span style="color: rgb(2, 37, 199);" class="">i32</span> 1, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""> <span style="color: rgb(2, 37, 199);" class="">i32</span> 4, <span style="color: rgb(2, 37, 199);" class="">i32</span> 5, <span style="color: rgb(2, 37, 199);" class="">i32</span> 6, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span> ></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""> <span style="color: rgb(202, 48, 199);" class="">%c2.w</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <4 x float> <span style="color: rgb(202, 48, 199);" class="">%c2</span>, <4 x float> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""> <8 x i32> <i32 0, <span style="color: rgb(2, 37, 199);" class="">i32</span> 1, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""> <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span>, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span>, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span>, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span> ></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""> <span style="color: rgb(202, 48, 199);" class="">%c</span> = <span style="color: rgb(201, 27, 0);" class="">shufflevector</span> <8 x float> <span style="color: rgb(202, 48, 199);" class="">%c01</span>, <8 x float> <span style="color: rgb(202, 48, 199);" class="">%c2.w</span>,</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""> <12 x i32> <i32 0, <span style="color: rgb(2, 37, 199);" class="">i32</span> 1, <span style="color: rgb(2, 37, 199);" class="">i32</span> 2, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""> <span style="color: rgb(2, 37, 199);" class="">i32</span> 4, <span style="color: rgb(2, 37, 199);" class="">i32</span> 5, <span style="color: rgb(2, 37, 199);" class="">i32</span> 6, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span>,</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""> <span style="color: rgb(2, 37, 199);" class="">i32</span> 8, <span style="color: rgb(2, 37, 199);" class="">i32</span> 9, <span style="color: rgb(2, 37, 199);" class="">i32</span> 10, <span style="color: rgb(2, 37, 199);" class="">i32</span> <span style="color: rgb(201, 27, 0);" class="">undef</span> ></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class=""><b class="">; t26: v12f32 = CONCAT_VECTORS t23, t24, t25</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class=""> <span style="color: rgb(201, 27, 0);" class="">ret</span> <12 x float> <span style="color: rgb(202, 48, 199);" class="">%c</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class="">;-----------</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class=""><b class="">; t27: v4f32 = EXTRACT_SUBVECTOR t26, 0</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class=""><b class="">; t28: v4f32 = EXTRACT_SUBVECTOR t26, 4</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class=""><b class="">; t29: v4f32 = EXTRACT_SUBVECTOR t26, 8</b></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class="">; t42: ch,glue = CopyToReg t0, Register:v4f32 $q0, t27</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class="">; t44: ch,glue = CopyToReg t42, Register:v4f32 $q1, t28, t42:1</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class="">; t46: ch,glue = CopyToReg t44, Register:v4f32 $q2, t29, t44:1</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69); min-height: 13px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(2, 37, 199);" class="">; t49: ch = AArch64ISD::RET_FLAG t48, Register:v4f32 $q0, Register:v4f32 $q1, Register:v4f32 $q2, Register:v4f32 $q3, t48:1</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(69, 69, 69);" class="">}</div><div class=""> </div><div class=""><div class=""><font face="Menlo" class="">===========================</font></div><div class="">Summary</div><div class=""><span style="font-family: Menlo;" class="">===========================</span></div><div class=""><br class=""></div><div class="">Based on the feedback and the evaluation, option B seems like a natural first step. It needs some IR extensions (split_vec(N) and align(N) on the vector type) but it’s still the fastest way to something working so that we can experiment if in fact a full-fledge multi-dimensional vector support is necessary in the IR. The only downside it has that it can’t leverage the existing optimizations for element-wise operations.</div><div class=""><br class=""></div><div class="">Option C could address that but may pose more challenges in recovering the original matrix layout information. The good news is that this can be explored incrementally from a working option B.</div></div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Adam</div><div class=""><br class=""></div><div class="">[1] <a href="http://lists.llvm.org/pipermail/llvm-dev/2018-October/126871.html" class="">http://lists.llvm.org/pipermail/llvm-dev/2018-October/126871.html</a></div></div></body></html>