<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 27 Jul 2020 at 13:28, Hal Finkel <<a href="mailto:hfinkel@anl.gov">hfinkel@anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p><br>
</p>
<div>On 7/20/20 11:13 PM, Baptiste Saleil
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, 22 Jun 2020 at
19:01, Hal Finkel <<a href="mailto:hfinkel@anl.gov" target="_blank">hfinkel@anl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p><br>
</p>
<div>On 6/19/20 9:31 PM, Baptiste Saleil via cfe-dev
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Summary<br>
-------<br>
<br>
New Power ISA v3.1 [0] introduces instructions to
accelerate matrix<br>
multiplication. We want to expose these instructions
through a list of<br>
target-dependent builtins and new Clang types in the
form of a language<br>
extension. This RFC gives more details on the
requirements for these<br>
types and explains how we (IBM) are implementing them
in Clang.<br>
<br>
We present the frontend implementation as an RFC
because we need to add<br>
target-specific checks in Sema and want to get
feedback on our implementation<br>
of these checks. The backend implementation does not
impact the other targets<br>
so it is not part of this RFC. Comments and questions
are welcome.<br>
<br>
Introduction<br>
------------<br>
<br>
The new instructions manipulate matrices that the CPU
represents by new 512-bit<br>
registers called `accumulators`. Copying matrices,
modifying values and<br>
extracting values of matrices may cause the CPU to
copy values from/to the<br>
matrix multiplication unit. To avoid degrading
performance, we thus want to<br>
minimize the number of times these operations are
used. So the user will be able<br>
to modify and extract values of the matrices and
perform computations with them<br>
by using the dedicated builtins only. The instructions
are designed to be used in<br>
computational kernels and we want to enforce that
specific workflow.<br>
<br>
Because of this restriction, we cannot rely on the
target-independent matrix<br>
types [1].</div>
</blockquote>
<p><br>
</p>
<p>If this is part of the documented system ABI, and what
will be supported by GCC, then we should support it too.</p>
<p>That having been said, I'm not convinced that this is a
good idea, and supporting the target-independent matrix
types would be better. I understand that the copying
will be expensive, and is something that should be
avoided, but this is true to some extent for everything:
there are some usages that compile to machine code
efficiently and some that don't. We generally, however,
favor the ability to create abstractions that *can* be
compiled efficiently as part of expected use cases, even
if we cannot guarantee that all uses will produce
efficient code. In his case, you're prohibiting the
creation of abstractions (by semantically restricting to
local variables) because you fear that not all uses will
compile to efficient code. Are there some other
structural reasons why supporting these are regular
values would be problematic?<br>
</p>
</div>
</blockquote>
Supporting these as regular values would be problematic for
several reasons. These new accumulator registers are actually
each associated with 4 of the existing 128-bit VSR vector
registers. A particularity of MMA is that when an accumulator
contains defined data, its 4 associated registers contain
undefined data and cannot be used. When copying an
accumulator, we need to:<br>
1. Copy its value back to its four associated VSRs<br>
2. Copy these 4 VSRs to the VSRs associated with the
destination accumulator<br>
3. Copy these VSRs to the destination accumulator <br>
4. If the copy is not a kill, copy the 4 VSRs associated
with the source back to the source accumulator<br>
<br>
So if these registers were supported as regular values, we
would have really expensive copy (and also expensive function
calls and returns</div>
</div>
</blockquote>
<p><br>
</p>
<p>I don't see why you call four vector moves or memory access
expensive? What's the alternative? The programmer needs to move
the data around somehow anyway if that's the thing that they need
to do.<br></p></div></blockquote>The reason I say it's expensive is because moving an accumulator cannot be done by just moving its 4 associated VSRs. With the new ISA, before being able to use an accumulator, it needs to be primed: we need to generate an instruction (xxmtacc) to copy the 4 VSRs to the accumulator. Another particularity is that when an accumulator is primed, the data in its associated VSRs is undefined, so to get the data back to the VSRs, the accumulator needs to be unprimed: we need to generate an instruction (xxmfacc) to copy the accumulator back to its associated VSRs. That means that to copy an accumulator, we need to generate an xxmfacc, generate the VSR copies, generate an xxmtacc for the destination and if the copy is not a kill, generate an xxmtacc to also reprime the source.<br><br>The alternative would be to try to prevent the user from copying these registers (this is what we want to do with Sema checks).<br>Because these registers are designed to be used for matrix multiplication, so in typed kernels with a very limited set of features, the user shouldn't need to move the data around.<br><div>For a user, the classical way of using these registers for matrix multiplication is to prime the accumulators he wants to use, then compute successive outer products in loops using the accumulators as builtin call arguments and finally unprime the accumulators to get the result of the matrix multiplications. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><p>
</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">) and we would prevent from using 4
vector registers per live accumulator. More importantly
(something I should have mentioned in the RFC), the new
instructions actually implement a single operation that is the
outer product. That means that supporting these as regular
values would imply copying accumulators back to their
associated VSRs and generating non-MMA instructions for any
other operation anyway. Therefore, it is likely that programs
using matrices would actually be less efficient.<br>
</div>
</div>
</blockquote>
<p><br>
</p>
<p>I'm assuming that you'll model the registers explicitly (using
RegisterTuples or similar in TableGen), so you'll end up with a
collection of registers that alias appropriately with their
underlying VSRs, and the general infrastructure will handle the
details of copying, killing, and so on. Is that correct?</p></div></blockquote><div>That's correct, these registers will be defined as any other register class in the PowerPC backend and we let the general infrastructure handle copying, killing, etc... </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<p>If we add patterns for, say, adding, that use subregister
extraction aong with the underlying VSR instructions, then
hopefully the infrastructure will coalesce away any unnecessary
copies and we'll get the right "in place" matrix addition. To say
that the types support only outer product is, based on my
interpretation of your description, technically correct, but on
the other hand, elementwise operations (e.g., add) can be directly
supported using the underlying operations on the VSRs at
reasonably-low cost. Is this correct?<br>
</p>
<p>I'm not sure exactly what our legalization framework does for
matrix types currently, but presumably it should handle expansion
for the other cases.</p></div></blockquote><div>This is where the concept of prime/unprime is an issue. The infrastructure would allow to support these operations at reasonably-low cost (with a generated code that would be similar to what we have now) except that the accumulator would need to be unprimed (xxmfacc) before each operation and primed (xxmtacc) after. And there would be no way to avoid generating these instructions. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<p>Regardless of what the frontend accepts, I would prefer to see,
where possible, types modeled using generic LLVM types and
operations.<br></p></div></blockquote><div>Yes, we're trying to avoid using target-specialized code as much as possible.</div><div><br></div><div>Thanks,</div><div><br></div><div>Baptiste.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><p>
</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote"><br>
<div>However, although we're not planning on supporting the
target-independent matrix types for these reasons, we're not
excluding supporting the target-independent matrix
operations. We are exploring implementing the
target-independent matrix multiplication operation with MMA
kernels. That way, on PowerPC, programs using
target-independent matrix types and operations would
actually benefit from MMA for matrix multiplication with no
additional effort.</div>
</div>
</div>
</blockquote>
<p><br>
</p>
<p>That sounds good to me.</p>
<p>Thanks again,</p>
<p>Hal<br>
</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<div>Baptiste.</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p> </p>
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr"> We need to add a new target-dependent
type and restrict its use.<br>
We give more details on these restrictions below. To
be able to manipulate<br>
these matrices, we want to add the `__vector_quad`
type to Clang. This type<br>
would be a PowerPC-specific builtin type mapped to the
new 512-bit registers.<br>
</div>
</blockquote>
<p><br>
</p>
<p>Okay.</p>
<p> -Hal<br>
</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr"><br>
Similarly, some of these instructions take 256-bit
values that must be stored<br>
in two consecutive VSX registers. To represent these
values and minimize the<br>
number of copies between VSX registers, we also want
to add the PowerPC-specific<br>
builtin type `__vector_pair` that would be mapped to
consecutive VSX registers.<br>
<br>
Value initialization<br>
--------------------<br>
<br>
The only way to initialize a `__vector_pair` is by
calling a builtin taking two<br>
128-bit vectors and assembling them to form a 256-bit
pair. A similar builtin<br>
exists to assemble four 128-bit vectors to form a
512-bit `__vector_quad`:<br>
<br>
vector unsigned char v1 = ...;<br>
vector unsigned char v2 = ...;<br>
vector unsigned char v3 = ...;<br>
vector unsigned char v4 = ...;<br>
__vector_pair vp;<br>
__vector_quad vq;<br>
__builtin_mma_assemble_pair(&vp, v1, v2);<br>
__builtin_mma_assemble_acc(&vq, v1, v2, v3, v4);<br>
<br>
The other way to initialize a `__vector_quad` is to
call a builtin mapped to an<br>
instruction generating a new value of this type:<br>
<br>
__vector_quad vq1;<br>
__builtin_mma_xxsetaccz(&vq1); // zero-initializes
vq1<br>
__vector_quad vq2;<br>
__builtin_mma_xvi4ger8(&vq2, v1, v2); // new value
generated in vq2<br>
<br>
Both `__vector_pair` and `__vector_quad` can also be
loaded from pointers that<br>
can potentially be casted from void or char pointers.<br>
<br>
Value extraction<br>
----------------<br>
<br>
The only way to extract values from a matrix is to
call the builtins<br>
disassembling `__vector_pair` and `__vector_quad`
values back into two<br>
and four 128-bit vectors respectively:<br>
<br>
vector unsigned char* vpr = ...;<br>
vector unsigned char* vqr = ...;<br>
__builtin_mma_disassemble_pair(vpr, &vp);<br>
__builtin_mma_disassemble_acc(vqr, &vq);<br>
<br>
Once the values are disassembled to vectors, the user
can extract values as<br>
usual, for example using the subscript operator on the
vector unsigned char<br>
values. So the typical workflow to efficiently use
these instructions in a<br>
kernel is to first initialize the matrices, then
perform computations and finally<br>
disassemble them to extract the result of the
computations. These three steps<br>
should be done using the provided builtins.<br>
<br>
Semantics<br>
---------<br>
<br>
To enforce using values of these types in kernels,
thus to avoid copies from/to<br>
the matrix multiplication unit, we want to prevent as
many implicit copies<br>
as possible. That means that it should only be
possible to declare values of<br>
these types as local variables. We want to prevent any
other way to declare and<br>
use non-pointer variables of these types (global
variable, function parameter,<br>
function return, etc...).<br>
<br>
The only situations in which these types and values of
these types can be<br>
used are:<br>
* Local variable declaration<br>
* Assignment operator<br>
* Builtin call parameter<br>
* Memory allocation<br>
* Typedef & alias<br>
<br>
Implementation<br>
--------------<br>
<br>
We have implemented the support of these types,
builtins and intrinsics in both<br>
Clang's frontend and the LLVM PowerPC backend. We will
post the backend<br>
implementation later. We implemented and tested this
support out-of-tree in<br>
conjunction with the GCC team to ensure a common API
and ensure source<br>
compatibility. For this RFC, we have 5 patches for the
frontend:<br>
* Add options to control MMA support on PowerPC
targets [2].<br>
* Define the two new types as Clang target-dependent
builtin types.<br>
As the other targets, we decided to define these
types in a separate<br>
`PPCtypes.def` file to improve extensibility in
case we need to add other<br>
PowerPC-specific types in the future [3].<br>
* Add the builtin definitions. These builtins use
the two new types,<br>
so they use custom type descriptors. To avoid
pervasive changes,<br>
we use custom decoding of these descriptors [4].<br>
* Add the Sema checks to restrict the use of the two
types.<br>
We prevent the use of non-pointer values of these
types in any declaration<br>
that is not a local variable declaration. We also
prevent them to<br>
be passed as function arguments and to be returned
from functions [5].<br>
* Implement the minimal required changes to LLVM to
support the builtins.<br>
In this patch, we enable the use of v256i1 for
intrinsic arguments and<br>
define all the MMA intrinsics the builtins are
mapped to [6].<br>
<br>
The backend implementation should not impact other
targets. We do not plan to<br>
add any type to LLVM. `__vector_pair` and
`__vector_quad` are generated as<br>
`v256i1` and `v512i1` respectively (both are currently
unused in the PowerPC<br>
backend). VSX pair registers will be allocated to the
`v256i1` type and the<br>
new accumulator registers will be allocated to the
`v512i1` type.<br>
</div>
</blockquote>
<blockquote type="cite">
<div dir="ltr"><br>
[0] Power ISA v3.1, <a href="https://ibm.ent.box.com/s/hhjfw0x0lrbtyzmiaffnbxh2fuo0fog0" target="_blank">https://ibm.ent.box.com/s/hhjfw0x0lrbtyzmiaffnbxh2fuo0fog0</a><br>
[1] <a href="https://clang.llvm.org/docs/MatrixTypes.html" target="_blank">https://clang.llvm.org/docs/MatrixTypes.html</a><br>
[2] <a href="https://reviews.llvm.org/D81442" target="_blank">https://reviews.llvm.org/D81442</a><br>
[3] <a href="https://reviews.llvm.org/D81508" target="_blank">https://reviews.llvm.org/D81508</a><br>
[4] <a href="https://reviews.llvm.org/D81748" target="_blank">https://reviews.llvm.org/D81748</a><br>
[5] <a href="https://reviews.llvm.org/D82035" target="_blank">https://reviews.llvm.org/D82035</a><br>
[6] <a href="https://reviews.llvm.org/D81744" target="_blank">https://reviews.llvm.org/D81744</a><br>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
cfe-dev mailing list
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a>
</pre>
</blockquote>
<pre cols="72">--
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory</pre>
</div>
</blockquote>
</div>
</div>
</blockquote>
<pre cols="72">--
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory</pre>
</div>
</blockquote></div></div>