<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">**** Proposal for TLX: Tensor LLVM eXtensions<br>
===================================================================================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">Authors: Akash Kothari (UIUC), Abdul Rafae Noor (UIUC), Dounia Khaldi (Intel),
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New";color:black"> Vikram Adve (UIUC), Yuanke Luo(Intel), Sudipta Sengupta (Amazon AWS),<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New";color:black"> Milind Girkar (Intel), Charith Mendis (UIUC)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">------------------------------------------------------------------------------------<o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:10.5pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:10.5pt;font-family:"Courier New";color:black"> </span><span style="font-size:10.5pt;color:black"><o:p></o:p></span></p>
<p style="margin:0in;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:10.5pt;font-family:"Courier New";color:black">***</span><span class="apple-converted-space"><span style="font-size:10.5pt;font-family:"Courier New";color:black"> </span></span><span style="font-size:10.5pt;font-family:"Courier New";color:black">RATIONALE<br>
========================================================================================<br>
Diverse hardware vendors are developing new hardware support for (mostly dense)</span><span style="font-size:10.5pt;color:black"><o:p></o:p></span></p>
<p style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-size:10.5pt;font-family:"Courier New";color:black">tensor computations, which have become increasingly important for machine</span><span style="font-size:10.5pt;color:black"><o:p></o:p></span></p>
<p style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-size:10.5pt;font-family:"Courier New";color:black">learning applications. These include both ISA extensions on CPUs and GPUs (such</span><span style="font-size:10.5pt;color:black"><o:p></o:p></span></p>
<p style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-size:10.5pt;font-family:"Courier New";color:black">as Intel AMX, Power MMA, NVIDIA’s tensor cores, AMD’s matrix cores, and</span><span class="apple-converted-space"><span style="font-size:10.5pt;font-family:"Courier New";color:black"> </span></span><span style="font-size:10.5pt;color:black"><o:p></o:p></span></p>
<p style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-size:10.5pt;font-family:"Courier New";color:black">Qualcomm’s HVX vector ISA) and dedicated accelerators for compute offload (such</span><span style="font-size:10.5pt;color:black"><o:p></o:p></span></p>
<p style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-size:10.5pt;font-family:"Courier New";color:black">as NVIDIA’s NVDLA, Amazon’s Inferentia and Trainium, and numerous ML</span><span class="apple-converted-space"><span style="font-size:10.5pt;font-family:"Courier New";color:black"> </span></span><span style="font-size:10.5pt;color:black"><o:p></o:p></span></p>
<p style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-size:10.5pt;font-family:"Courier New";color:black">accelerators from smaller companies). While ML workloads are the primary</span><span class="apple-converted-space"><span style="font-size:10.5pt;font-family:"Courier New";color:black"> </span></span><span style="font-size:10.5pt;color:black"><o:p></o:p></span></p>
<p style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-size:10.5pt;font-family:"Courier New";color:black">motivation and likely to be the dominant use cases, other tensor-intensive</span><span class="apple-converted-space"><span style="font-size:10.5pt;font-family:"Courier New";color:black"> </span></span><span style="font-size:10.5pt;color:black"><o:p></o:p></span></p>
<p style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-size:10.5pt;font-family:"Courier New";color:black">application domains, such as image processing, scientific computing, quantum</span><span class="apple-converted-space"><span style="font-size:10.5pt;font-family:"Courier New";color:black"> </span></span><span style="font-size:10.5pt;color:black"><o:p></o:p></span></p>
<p style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-size:10.5pt;font-family:"Courier New";color:black">simulations, financial modeling, and others can benefit from this hardware</span><span class="apple-converted-space"><span style="font-size:10.5pt;font-family:"Courier New";color:black"> </span></span><span style="font-size:10.5pt;color:black"><o:p></o:p></span></p>
<p style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-size:10.5pt;font-family:"Courier New";color:black">support as well, via languages like C++, DPC++, Julia, Fortran, Halide, CUDA, OpenCL,</span><span style="font-size:10.5pt;color:black"><o:p></o:p></span></p>
<p style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-size:10.5pt;font-family:"Courier New";color:black">and others.</span><span style="font-size:10.5pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">LLVM can play a crucial role in making it easier for these vendors to create<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">optimizing compiler back-ends for their emerging hardware (if the existing<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">vector and matrix support in LLVM were generalized to support tensor operations).</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">LLVM is already widely-used today by many of the vendors that develop these<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">tensor architectures, e.g., to target CPUs and GPUs. LLVM is highly<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">retargetable, by design. For the CPU targets, LLVM allows an integrated code<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">generation framework for tensor operations with optimized intermixing of scalar,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">1-D vector and 2-D matrix operations in the same code section (e.g., loop<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">body). And LLVM has front-ends for a wide range of high-level languages,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">including essentially all the languages used widely for relevant application<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">domains today.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">No existing infrastructure we know of meets these needs. MLIR is likely<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">the<span class="apple-converted-space"> </span>best option, and we believe it is entirely complementary to LLVM.<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">MLIR provides strong support for high-level tensor operations in<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">TOSA, relevant optimizations in<span class="apple-converted-space"> </span><i>Affine</i><span class="apple-converted-space"> </span>and<span class="apple-converted-space"> </span><i>Linalg</i>, and lowering
paths to<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">accelerators, GPUs and (via the LLVM dialect) CPUs. Crucially, however, MLIR<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">does not have a separate low-level code generation framework that is<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">retargetable to diverse hardware: it relies on LLVM for this purpose. If LLVM<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">could be extended with tensor operations and a corresponding retargetable<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">tensor code generation framework, MLIR could leverage this as well. Moreover,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">there are enough vendors and also languages that rely heavily</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">on LLVM (but do not use MLIR) that it seems worthwhile to have a high-quality<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">tensor code generation<span class="apple-converted-space"> in </span>both LLVM as well as in MLIR. Ideally, both systems<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">would largely<span class="apple-converted-space"> </span>share the same code.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">The broad goal of our project is to add a retargetable tensor code generation<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">framework to LLVM. We are currently working on a prototype implementation<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">with our<span class="apple-converted-space"> </span>collaborators at Amazon AWS, Intel, IBM and Qualcomm. This RFC<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">focuses on the first stage: extending the LLVM IR with tensor operations<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">which we refer to as TLX (Tensor LLVM eXtensions).</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
<br>
*** OVERALL PROJECT OBJECTIVES<br>
===============================================================================<br>
*<span class="apple-converted-space"> </span>A unified retargetable code generation and optimization framework for LLVM<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">to target diverse tensor architectures with a common set of IR extensions,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">instead of using target-specific solutions.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">*<span class="apple-converted-space"> (Subject of this RFC.)</span>A single set of target-agnostic tensor extensions in</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">LLVM that higher-level<span class="apple-converted-space"> </span><span style="background:white">tensor code generation frameworks such as XLA, Halide,<span class="apple-converted-space"> </span></span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black;background:white">TVM, MLIR, etc. can<span class="apple-converted-space"> </span>target instead of lowering to target-specific intrinsics<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black;background:white">in LLVM, while<span class="apple-converted-space"> </span>retaining the optimizations in these high-level frameworks.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">*<span class="apple-converted-space"> </span>A pathway for LLVM-based languages such as C/C++, DPC++, Fortran, Rust,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">Julia, etc.<span class="apple-converted-space"> </span>that do not have frontends for compiler systems like MLIR, TVM, XLA,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">etc. to target<span class="apple-converted-space"> </span>modern tensor architectures by lowering to our tensor extensions<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">in LLVM.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">*<span class="apple-converted-space"> </span>Target-independent optimizations (e.g. peephole and generic SSA-based<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">optimizations) and also flexible code generation capabilities in LLVM that<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">could involve mixing instructions operating on vector and rectangular<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">registers, and involve developing cost models which could help reduce register<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">spills and maximize usage of available hardware resources.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">*<span class="apple-converted-space"> </span>Contribute our tensor extensions (this RFC) and retargetable code generation<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">framework (as a followup) to the LLVM project for the community<span class="apple-converted-space"> </span>to experiment with<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">and provide feedback.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="vertical-align:baseline;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
*** RFC: INTRODUCTION OF TENSOR CONCEPT IN LLVM<br>
======================================================================<br>
To achieve our objectives, we need to introduce the<span class="apple-converted-space"> </span>concept of tensors.<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">To do this, we need to add a tensor type - N-dimensional data </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">type, generalizing 1-D vectors and 2-D matrices. We also need crucial tensor<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">operations which front-ends for high-level<span class="apple-converted-space"> </span>languages can target, and which<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">represent or can be implemented via ISAs of<span class="apple-converted-space"> </span>different tensor architectures.<br>
<br>
<br>
*** IMPLEMENTATION OF TENSOR TYPE IN LLVM<br>
=======================================================================<br>
<br>
** OVERVIEW:<br>
----------------------<br>
The concept of dense tensors can be implemented as a new, first-class<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">n-dimensional vector type in LLVM. However, doing this would be extremely<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">intrusive since it will require changes<span class="apple-converted-space"> </span>to hundreds of files in LLVM. While </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">this may be the correct option in the long term, once the design has been<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">properly evaluated and refined, the effort<span class="apple-converted-space"> </span>required to do so for an initial </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">prototype and evaluation is not justified. So<span class="apple-converted-space"> </span>we propose to implement the tensor<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">concept as an LLVM intrinsic called llvm.tensor.typeinfo while representing tensor<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">data in “flattened” form as ordinary LLVM vector types. The intrinsic takes as operands<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">a<span class="apple-converted-space"> </span>“flattened” LLVM vector, together with shape, layout and padding vectors, and<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">returns a value of LLVM token type. By returning a value of token type, this<span class="apple-converted-space"> </span>intrinsic </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">avoids the risk of being eliminated by optimizations (especially,<span class="apple-converted-space"> </span>dead code elimination)<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">when it has uses. This intrinsic is marked with the<span class="apple-converted-space"> </span>'readnone' and 'speculatable' </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">attributes so that it does not inhibit<span class="apple-converted-space"> </span>optimizations like redundancy elimination, dead<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">code elimination,<span class="apple-converted-space"> </span>code motion, etc.<br>
<br>
token llvm.tensor.typeinfo(<llvm-vector-type> %tensor, <n x i32> %shape,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <n x i32> %layout, <n x i32> %padding)<br>
<br>
** OPERANDS:<br>
-----------------------<br>
==============================================================================|<br>
Operand | Description |<br>
============|=================================================================|<br>
%tensor | n-dimensional tensor value represented as a “flattened” vector<span class="apple-converted-space"> </span> |<br>
------------|-----------------------------------------------------------------|<br>
%shape | Vector of dimension values of a tensor |<br>
------------|-----------------------------------------------------------------|<br>
%layout | Vector of permutation of dimension indices ranging from 0 to n-1|<br>
------------|-----------------------------------------------------------------|<br>
%padding <span class="apple-converted-space"> </span> | Vector of padding values along every dimension of a tensor |<br>
==============================================================================|<br>
<br>
** RESULT:<br>
-----------------------<br>
==============================================================================|<br>
Result | Description |<br>
=================|============================================================|<br>
token value | LLVM value of token type associated with a tensor value<span class="apple-converted-space"> </span> |<br>
==============================================================================|<br>
<br>
** SEMANTICS:<br>
-----------------------<br>
The ‘llvm.tensor.typeinfo’ intrinsic is used to produce a unique token value<span class="apple-converted-space"> </span><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">associated with a tensor value represented as a “flattened” vector. The layout<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">operand of this intrinsic is expressed as a permutation of dimension indices<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">(from 0 to n-1 for an n-dimensional tensor). This represents tensor layouts in<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">LLVM in a generic way. The number of elements in shape, layout and padding<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">vectors must be the same and equal to the number of dimensions of the given<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">tensor.<br>
<br>
Note that this intrinsic is only meant to hold information such as shape,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">layout and padding of a tensor value in LLVM IR. It does not read nor write<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">memory nor perform any computations, and it does not exhibit any kind of<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">undefined behavior.<br>
<br>
<br>
** EXAMPLE:<br>
-----------------------<br>
; The first argument (%tensor) is the tensor that is being modelled as a<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">flattened vector. The second argument is the shape (16 x 5 x 3), the third<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">argument is layout (<0, 1, 2>) and the fourth argument is padding<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">(<3, 2, 1> along the corresponding dimensions) for the given tensor.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%input = call token @llvm.tensor.typeinfo(<240 x float> %tensor,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 16, i32 5, i32 3>, <3 x i32> <i32 0, i32 1, i32 2>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 3, i32 2, i32 1>)<br>
<br>
; The first argument is the input virtual tensor register and the second<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">argument is the new permutation of the layout of the input tensor. This<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">operation produces a tensor of layout <2, 0, 1>. </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%output = call <240 x float> @llvm.tensor.transpose(token %input,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 2, i32 0, i32 1>)<br>
<br>
; The first argument (%output) the output tensor that is being modelled as a<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">flattened vector. The second argument is the new shape (3 x 16 x 5), the third<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">argument is the new layout (<2, 0, 1>) and the fourth argument is the new padding<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">(<1, 3, 2> along the corresponding dimensions) for the output tensor.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%typed_output = call token @llvm.tensor.typeinfo(<240 x float> %output,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 3, i32 16, i32 5>, <3 x i32> <i32 2, i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 1, i32 3, i32 2>)<br>
<br>
<br>
<br>
*** TENSOR OPERATIONS IN LLVM<br>
=================================================================================<br>
<br>
** INTRINSIC: llvm.tensor.load<br>
================================<br>
<br>
** OVERVIEW:<br>
---------------------<br>
This operation loads a tensor or sub-tensor with the given shape, layout and<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">padding from memory into a register. This operation is strided, unlike the<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">existing load instruction in LLVM, to be able to load sub-tensors from memory.<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">This intrinsic is marked with 'speculatable' attribute to prevent it from<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">inhibiting optimizations like redundancy elimination, dead code elimination,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">code motion, etc.<br>
<br>
token llvm.tensor.load(<element_type>* %mem_ptr, <n x i32> %shape,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <n x i32> %layout, <n x i32> %pad, <n xi32> %strides)<br>
<br>
** OPERANDS:<br>
---------------------<br>
<br>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">==============================================================================|<br>
Operand <span class="apple-converted-space"> </span> | Description |<br>
=============|================================================================|<br>
%mem_ptr <span class="apple-converted-space"> </span>| Starting address of a tensor/subtensor in memory |<br>
-------------|----------------------------------------------------------------|<br>
%shape <span class="apple-converted-space"> </span>| Vector of dimension values of the loaded tensor/sub-tensor |<br>
-------------|----------------------------------------------------------------|<br>
%layout | Vector of permutation of dimension indices ranging from 0 to<span class="apple-converted-space"> </span> |</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> |<span class="apple-converted-space"> </span>n-1 |<span class="apple-converted-space"> </span><br>
-------------|----------------------------------------------------------------|<br>
%padding <span class="apple-converted-space"> </span> | Vector of padding values along every dimension of<span class="apple-converted-space"> </span> |</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> |<span class="apple-converted-space"> </span>the loaded tensor/sub-tensor |<br>
-------------|----------------------------------------------------------------|<br>
%strides | Vector of strides in memory along every dimension of the loaded|</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> |<span class="apple-converted-space"> </span>tensor/sub-tensor |</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">==============================================================================|<br>
<br>
** RESULT:<br>
------------------<br>
==============================================================================|<br>
Result | Description |<br>
=============|================================================================|<br>
token value | LLVM value of token type associated with a tensor value |<br>
==============================================================================|<br>
<br>
** SEMANTICS:<br>
-----------------------<br>
The ‘llvm.tensor.load' intrinsic loads a tensor or subtensor with the given<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">shape, layout and padding from memory into a register. This operation is<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">strided based on %strides, unlike the existing load instruction in LLVM, to be<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">able to load subtensors from memory since sub-tensors are not laid out<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">contiguously in memory. This intrinsic reads from memory, but does not write<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">to memory.<br>
<br>
** EXAMPLE:<br>
---------------------<br>
; This loads a sub-tensor from the memory location pointed to by %mem_ptr. The<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">;<span class="apple-converted-space"> </span>sub-tensor has the shape <16 x 6 x 4> (second argument), layout <0, 1, 2> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">;<span class="apple-converted-space"> </span>(third argument) and zero padding (fourth argument). The strides in memory<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">;<span class="apple-converted-space"> </span>along every dimension are <0, 0, 8>, which means that the rows of the loaded<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">;<span class="apple-converted-space"> </span>sub-tensor have a distance of 8 bytes in memory. This produces a unique token<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">;<span class="apple-converted-space"> </span>%tensor.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%tensor = call token @llvm.tensor.load(i8* %mem_ptr,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 16, i32 6, i32 4>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1, i32 2>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 0, i32 0, i32 0>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 0, i32 0, i32 8>)<br>
<br>
<br>
<br>
** INTRINSIC: llvm.tensor.store<br>
=================================<br>
<br>
** OVERVIEW:<br>
---------------------<br>
This operation stores a tensor or subtensor from a register into memory. This<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">operation is strided, unlike the existing store instruction in LLVM, to be<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">able to store sub-tensors into memory. This intrinsic is marked with 'readnone'<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">attribute to prevent it from inhibiting optimizations like redundancy<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">elimination, dead code elimination, code motion, etc.<br>
<br>
void llvm.tensor.store(<element_type>* %mem_ptr,token %tensor,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <n xi32> %strides)<br>
<br>
** OPERANDS:<br>
----------------------<br>
==============================================================================|<br>
Operand <span class="apple-converted-space"> </span> | Description |<br>
=============|================================================================|<br>
%mem_ptr <span class="apple-converted-space"> </span>| Starting address of a tensor/subtensor in memory |<br>
-------------|----------------------------------------------------------------|<br>
%tensor | Stored subtensor/tensor |<br>
-------------|----------------------------------------------------------------|<br>
%strides | Vector of strides in memory along every dimension of the stored|</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> |<span class="apple-converted-space"> </span>tensor/sub-tensor |<br>
==============================================================================|<br>
<br>
** RESULT:<br>
------------------<br>
Intrinsic does not return anything.<br>
<br>
<br>
** SEMANTICS:<br>
-----------------------<br>
The ‘llvm.tensor.store' intrinsic stores a tensor or subtensor from a register<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">into memory. This operation is strided based on %strides, unlike the existing<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">store instruction in LLVM, to be able to store sub-tensors to memory since<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">sub-tensors are not laid out contiguously in memory. This intrinsic writes to<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">memory, but does not read from memory.<br>
<br>
<br>
** EXAMPLE:<br>
---------------------<br>
%tensor = call token @llvm.tensor.typeinfo(<240 x float> %tensor,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 16, i32 6, i32 4>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 0, i32 1, i32 2>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 0, i32 0, i32 0>)<br>
<br>
; This stores a tensor from the memory location pointed to by %mem_ptr and<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">;<span class="apple-converted-space"> </span>the second argument is the stored tensor itself. The strides in memory along<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">;<span class="apple-converted-space"> </span>every dimension are <0, 12, 10> (third argument), which means that the rows<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">;<span class="apple-converted-space"> </span>of %tensor are stored 10*sizeof(float) bytes apart and columns of %tensor<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">;<span class="apple-converted-space"> </span>are 12*sizeof(float) bytes apart in memory.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <br>
call void @llvm.tensor.store(float* %mem_ptr, token %tensor,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 0, i32 12, i32 10>)<br>
<br>
<br>
<br>
** INTRINSIC: llvm.tensor.matmul<br>
=================================<br>
<br>
** OVERVIEW:<br>
---------------------<br>
This intrinsic performs batched matrix multiplication between the inner<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">dimensions of two multidimensional tensors. This intrinsic is marked with<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">the 'readnone' and 'speculatable' attributes to prevent it from inhibiting<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">optimizations like redundancy elimination, dead code elimination,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">code motion, etc.<br>
<br>
<vector_ty> llvm.tensor.matmul(token %input1, token %input2)<br>
<br>
<br>
** OPERANDS:<br>
----------------------<br>
==============================================================================|<br>
Operand <span class="apple-converted-space"> </span> | Description |<br>
=============|================================================================|<br>
%input1 <span class="apple-converted-space"> </span>| Token value representing the first input tensor |<br>
-------------|----------------------------------------------------------------|<br>
%input2 <span class="apple-converted-space"> </span>| Token value representing the second input tensor |<br>
==============================================================================|<br>
<br>
** RESULT:<br>
------------------<br>
==============================================================================|<br>
Result | Description |<br>
=============|================================================================|<br>
token value | LLVM value of token type associated with a tensor value |<br>
==============================================================================|<br>
<br>
** SEMANTICS:<br>
-----------------------<br>
The ‘llvm.tensor.matmul' intrinsic performs batched matrix multiplication<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">between two input tensors. The inner two dimensions of the input tensors must<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">have valid matrix multiplication dimensions, and any further outer dimensions<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">must be of matching batch size. This intrinsic does not read nor write memory,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">nor does it exhibit any kind of undefined behavior.<br>
<br>
<br>
** EXAMPLE:<br>
---------------------<br>
%input1 = call token @llvm.tensor.typeinfo(<12 x float> %tensor1,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black"> <2 x i32> <i32 3, i32 4>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%input2 = call token @llvm.tensor.typeinfo(<12 x float> %tensor2,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 4, i32 3>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)<br>
<br>
%output = call <9 x float> @llvm.tensor.matmul(token %input1,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> token %input2)<br>
<br>
%typed_output = call token @llvm.tensor.typeinfo(<9 x float> %output,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 3, i32 3>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)<br>
<br>
<br>
<br>
** INTRINSIC: llvm.tensor.contract<br>
===================================<br>
<br>
** OVERVIEW:<br>
---------------------<br>
This intrinsic performs tensor contraction on two multidimensional tensors.<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">This intrinsic is marked with the 'readnone' and 'speculatable' attributes<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">to prevent it from inhibiting optimizations like redundancy elimination,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">dead code elimination, code motion, etc.<br>
<br>
<vector_ty> llvm.tensor.contract(token %input1, token %input2,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <m x i32> %in1_contraction_axes,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <m x i32> %in2_contraction_axes)<br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black"><br>
** OPERANDS:<br>
----------------------<br>
<br>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">==============================================================================|<br>
Operand <span class="apple-converted-space"> </span> | Description |<br>
======================|=======================================================|<br>
%input1 <span class="apple-converted-space"> </span>| Token value representing the first input tensor |<br>
----------------------|-------------------------------------------------------|<br>
%input2 | Token value representing the second input tensor |<br>
----------------------|-------------------------------------------------------|<br>
%in1_contraction_axes | Vector for m axes for the first input tensor along<span class="apple-converted-space"> </span> |</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> |<span class="apple-converted-space"> which </span>contraction/reduction is performed | <span class="apple-converted-space"> </span><br>
----------------------|-------------------------------------------------------|<br>
%in2_contraction_axes | Vector for m axes for the second input tensor along<span class="apple-converted-space"> </span>|</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> |<span class="apple-converted-space"> </span>which contraction/reduction is performed |</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">==============================================================================|<span class="apple-converted-space"> </span><br>
<br>
** RESULT:<br>
------------------<br>
<br>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">==============================================================================|<br>
Result | Description |<br>
=====================|========================================================|<br>
token value <span class="apple-converted-space"> </span> | LLVM value of token type associated with a tensor value|</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">==============================================================================|<br>
<br>
** SEMANTICS:<br>
-----------------------<br>
The ‘llvm.tensor.contract' intrinsic multiplies and contracts two input tensors<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">along given axes, %in1_contraction_axes and %in2_contraction_axes. The axes<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">vectors contain a list of dimension indices for the input tensors along which<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">the reduction takes place. This intrinsic does not read nor write memory, nor<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">does it exhibit any kind of undefined behavior.<br>
<br>
<br>
** EXAMPLE:<br>
---------------------<br>
%input1 = call token @llvm.tensor.typeinfo(<8 x float> %tensor1,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 2, i32 2, i32 2>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 0, i32 1, i32 2>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 0, i32 0, i32 0>)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%input2 = call token @llvm.tensor.typeinfo(<8 x float> %tensor2,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 2, i32 2, i32 2>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 0, i32 1, i32 2>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 0, i32 0, i32 0>)<br>
<br>
%output = call <8 x float> @llvm.tensor.contract(token %input1, token %input2,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 2>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>)<br>
<br>
%typed_output = call token @llvm.tensor.typeinfo(<8 x float> %output,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 2, i32 2, i32 2>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 0, i32 1, i32 2>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 0, i32 0, i32 0>)<br>
<br>
<br>
<br>
** INTRINSIC: llvm.tensor.umma<br>
================================<br>
<br>
** OVERVIEW:<br>
---------------------<br>
This intrinsic performs matrix multiplication between two given input tensors<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">and then accumulates the result using the third input tensor. This intrinsic<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">is marked with the 'readnone' and 'speculatable' attributes to prevent it<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">from inhibiting optimizations like redundancy elimination, dead code<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">elimination, code motion, etc.<br>
<br>
<vector_ty> llvm.tensor.umma(token %acc, token %op1, token %op2)<br>
<br>
** OPERANDS:<br>
----------------------<br>
<br>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">==============================================================================|</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">Operand <span class="apple-converted-space"> </span> | Description |<br>
=============|================================================================|<br>
%acc <span class="apple-converted-space"> </span>| Token value representing a tensor which accumulates results |<br>
-------------|----------------------------------------------------------------|<br>
%op1 <span class="apple-converted-space"> </span>| Token value representing the first input tensor |<br>
-------------|----------------------------------------------------------------|<br>
%op2 <span class="apple-converted-space"> </span>| Token value representing the second input tensor |<br>
==============================================================================|<br>
<br>
** RESULT:<br>
------------------<br>
==============================================================================|</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">Result | Description |<br>
=============|================================================================|<br>
token value | LLVM value of token type associated with a tensor value |</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">==============================================================================|<br>
<br>
** SEMANTICS:<br>
-----------------------<br>
The ‘llvm.tensor.umma' intrinsic performs matrix-multiply-accumulation between<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">two unsigned operands and accumulates the result with the given register:<br>
%output = %acc + matmul(%op1, %op2)<br>
<br>
When full product (e.g. i8 * i8 -> i16) is needed, the input tensors must be<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">sign-extended.<br>
Note that this intrinsic does not read nor write memory, nor does it exhibit<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">any kind of undefined behavior.<br>
<br>
<br>
** EXAMPLE:<br>
---------------------<br>
%acc = call token @llvm.tensor.typeinfo(<9 x i32> %acc_tensor,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 3, i32 3>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%op1 = call token @llvm.tensor.typeinfo(<12 x i32> %tensor1,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 3, i32 4>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%op2 = call token @llvm.tensor.typeinfo(<12 x i32> %tensor2,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 4, i32 3>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)<br>
<br>
; The first argument is the accumulator virtual tensor register, and the<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">;<span class="apple-converted-space"> </span>second and third arguments are the input virtual tensor registers. <br>
%output = call <9 x i32> @llvm.tensor.umma(token %acc, token %op1, token %op2)<br>
<br>
<br>
<br>
** INTRINSIC: llvm.tensor.smma<br>
===============================<br>
<br>
** OVERVIEW:<br>
---------------------<br>
This intrinsic performs matrix multiplication between two given input tensors<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">and then accumulates the result using the third input tensor. This intrinsic is<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">marked with the 'readnone' and 'speculatable' attributes to prevent it from<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">inhibiting optimizations like redundancy elimination, dead code elimination,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">code motion, etc.<br>
<br>
<vector_ty> llvm.tensor.smma(token %acc, token %op1, token %op2)<br>
<br>
<br>
** OPERANDS:<br>
----------------------<br>
<br>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">==============================================================================|<br>
Operand <span class="apple-converted-space"> </span> | Description |<br>
=============|================================================================|<br>
%acc <span class="apple-converted-space"> </span>| Token value representing a tensor which accumulates results |<br>
-------------|----------------------------------------------------------------|<br>
%op1 <span class="apple-converted-space"> </span>| Token value representing the first input tensor |<br>
-------------|----------------------------------------------------------------|<br>
%op2 <span class="apple-converted-space"> </span>| Token value representing the second input tensor |<br>
==============================================================================|<br>
<br>
** RESULT:<br>
------------------<br>
==============================================================================|<br>
Result | Description |<br>
=============|================================================================|<br>
token value | LLVM value of token type associated with a tensor value |<br>
==============================================================================|<br>
<br>
** SEMANTICS:<br>
---------------------<br>
The ‘llvm.tensor.smma' intrinsic performs matrix-multiply-accumulation between<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">two signed operands and accumulates the result with the given register:<br>
%output = %acc + matmul(%op1, %op2)<br>
<br>
When full product (e.g. i8 * i8 -> i16) is needed, the input tensors must be<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">sign-extended.<br>
Note that this intrinsic does not read nor write memory, nor does it exhibit<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">any kind of undefined behavior.<br>
<br>
** EXAMPLE:<br>
---------------------<br>
%acc = call token @llvm.tensor.typeinfo(<9 x i32> %acc_tensor,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 3, i32 3>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%op1 = call token @llvm.tensor.typeinfo(<12 x i32> %tensor1,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 3, i32 4>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%op2 = call token @llvm.tensor.typeinfo(<12 x i32> %tensor2,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 4, i32 3>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)<br>
<br>
; The first argument is the accumulator virtual tensor register, </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">;<span class="apple-converted-space"> </span>and the second and third arguments are the input virtual tensor registers. <br>
%output = call <9 x i32> @llvm.tensor.smma(token %acc, token %op1, token %op2)<br>
<br>
<br>
<br>
** INTRINSIC: llvm.tensor.usmma<br>
=================================<br>
<br>
** OVERVIEW:<br>
---------------------<br>
This intrinsic performs matrix multiplication between two given input tensors<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">and then accumulates the result using the third input tensor. This intrinsic<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">is marked with the 'readnone' and 'speculatable' attributes to prevent it from<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">inhibiting optimizations like redundancy elimination, dead code elimination,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">code motion, etc.<br>
<br>
<vector_ty> llvm.tensor.usmma(token %acc, token %op1, token %op2)<br>
<br>
<br>
** OPERANDS:<br>
----------------------<br>
==============================================================================|<br>
Operand <span class="apple-converted-space"> </span> | Description |<br>
=============|================================================================|<br>
%acc <span class="apple-converted-space"> </span>| Token value representing a tensor which accumulates results |<br>
-------------|----------------------------------------------------------------|<br>
%op1 <span class="apple-converted-space"> </span>| Token value representing the first input tensor |<br>
-------------|----------------------------------------------------------------|<br>
%op2 <span class="apple-converted-space"> </span>| Token value representing the second input tensor |<br>
==============================================================================|<br>
<br>
** RESULT:<br>
------------------<br>
<br>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">==============================================================================|<br>
Result | Description |<br>
=============|================================================================|<br>
token value | LLVM value of token type associated with a tensor value |<br>
==============================================================================|<br>
<br>
** SEMANTICS:<br>
-----------------------<br>
The ‘llvm.tensor.usmma' intrinsic performs matrix-multiply-accumulation between<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">unsigned and signed operands and accumulates the result with the given register:<br>
%output = %acc + matmul(%op1, %op2)<br>
<br>
When full product (e.g. i8 * i8 -> i16) is needed, the input tensors must be<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">sign-extended.<br>
Note that this intrinsic does not read nor write memory, nor does it exhibit any<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">kind of undefined behavior.<br>
<br>
** EXAMPLE:<br>
---------------------<br>
%acc = call token @llvm.tensor.typeinfo(<9 x i32> %acc_tensor,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 3, i32 3>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%op1 = call token @llvm.tensor.typeinfo(<12 x i32> %tensor1,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 3, i32 4>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%op2 = call token @llvm.tensor.typeinfo(<12 x i32> %tensor2,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 4, i32 3>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)<br>
<br>
; The first argument is the accumulator virtual tensor register, </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">;<span class="apple-converted-space"> </span>and the second and third arguments are the input virtual tensor<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">;<span class="apple-converted-space"> </span>registers. <br>
%output = call <9 x i32> @llvm.tensor.usmma(token %acc,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> token %op1, token %op2)<br>
<br>
<br>
<br>
** INTRINSIC: llvm.tensor.summa<br>
=================================<br>
<br>
** OVERVIEW:<br>
---------------------<br>
This intrinsic performs matrix multiplication between two given<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">input tensors and then accumulates the result using the third input<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">tensor. This intrinsic is marked with the 'readnone' and 'speculatable'<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">attributes to prevent it from inhibiting optimizations like redundancy </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">elimination, dead code elimination, code motion, etc.<br>
<br>
<vector_ty> llvm.tensor.summa(token %acc, token %op1, token %op2)<br>
<br>
<br>
** OPERANDS:<br>
----------------------<br>
<br>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">==============================================================================|<br>
Operand <span class="apple-converted-space"> </span> | Description |<br>
=============|================================================================|<br>
%acc <span class="apple-converted-space"> </span>| Token value representing a tensor which accumulates results |<br>
-------------|----------------------------------------------------------------|<br>
%op1 <span class="apple-converted-space"> </span>| Token value representing the first input tensor |<br>
-------------|----------------------------------------------------------------|<br>
%op2 <span class="apple-converted-space"> </span>| Token value representing the second input tensor |<br>
==============================================================================|<br>
<br>
** RESULT:<br>
------------------<br>
<br>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">==============================================================================|<br>
Result | Description |<br>
=============|================================================================|<br>
token value | LLVM value of token type associated with a tensor value |<br>
==============================================================================|<br>
<br>
** SEMANTICS:<br>
-----------------------<br>
The ‘llvm.tensor.summa' intrinsic performs matrix-multiply-accumulation between<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">signed and unsigned operands and accumulates the result with the given<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">register:</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%output = %acc + matmul(%op1, %op2)<br>
<br>
When full product (e.g. i8 * i8 -> i16) is needed, the input tensors must be<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">sign-extended.<br>
Note that this intrinsic does not read nor write memory, nor does it exhibit any<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">kind of undefined behavior.<br>
<br>
<br>
** EXAMPLE:<br>
---------------------<br>
%acc = call token @llvm.tensor.typeinfo(<9 x i32> %acc_tensor,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 3, i32 3>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%op1 = call token @llvm.tensor.typeinfo(<12 x i32> %tensor1,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 3, i32 4>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%op2 = call token @llvm.tensor.typeinfo(<12 x i32> %tensor2,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 4, i32 3>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)<br>
<br>
; The first argument is the accumulator virtual tensor register, and the<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">;<span class="apple-converted-space"> </span>second and third arguments are the input virtual tensor registers. <br>
%output = call <9 x i32> @llvm.tensor.summa(token %acc, token %op1,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> token %op2)<br>
<br>
<br>
<br>
** INTRINSIC: llvm.tensor.convolution<br>
======================================<br>
<br>
** OVERVIEW:<br>
---------------------<br>
This intrinsic performs convolution between input and kernel tensors. This<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">intrinsic is marked with the 'readnone' and 'speculatable' attributes to<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">prevent it from inhibiting optimizations like redundancy elimination, dead<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">code elimination, code motion, etc.<br>
<br>
<vector_ty> llvm.tensor.convolution(token %input, token %kernel,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <vector_ty> %strides,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <vector_ty> %input_dilations,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <vector_ty> %kernel_dilations)<br>
<br>
<br>
** OPERANDS:<br>
----------------------<br>
==============================================================================|<br>
Operand | Description |<br>
==================|===========================================================|<br>
%input | Token value representing the input tensor |<br>
------------------|-----------------------------------------------------------|<br>
%kernel | Token value representing the kernel tensor |<br>
------------------|-----------------------------------------------------------|<br>
%strides | Vector containing strides values for the sliding kernel |<br>
------------------|-----------------------------------------------------------|<br>
%input_dilations | Vector containing dilation values for the input |<br>
------------------|-----------------------------------------------------------|<br>
%kernel_dilations | Vector containing dilation values for the kernel |<br>
==============================================================================|<br>
<br>
** RESULT:<br>
------------------</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">==============================================================================|<br>
Result | Description |<br>
==================|===========================================================|<br>
token value | LLVM value of token type associated with a tensor value |<br>
==============================================================================|<br>
<br>
** SEMANTICS:<br>
-----------------------<br>
The ‘llvm.tensor.convolution' intrinsic performs the convolution between input<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">and kernel tensors.<br>
This intrinsic does not read nor write memory, nor does it exhibit any kind of<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">undefined behavior.<br>
<br>
<br>
** EXAMPLE:<br>
---------------------<br>
%input = call token @llvm.tensor.typeinfo(<12 x float> %tensor1,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 3, i32 4>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%kernel = call token @llvm.tensor.typeinfo(<12 x float> %tensor2,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 4, i32 3>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)<br>
<br>
; The third argument is a vector of kernel stride values along every dimension<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">;<span class="apple-converted-space"> </span>and the fourth argument is a vector of dilation values along every dimension.<br>
%output = call <9 x float> @llvm.tensor.convolution(token %input, token %kernel,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 1, i32 2>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)<br>
<br>
<br>
<br>
** INTRINSIC: llvm.tensor.transpose<br>
====================================<br>
<br>
** OVERVIEW:<br>
---------------------<br>
This intrinsic changes the layout of a given tensor by permuting the indices of<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">its dimensions. This intrinsic is marked with the 'readnone' and 'speculatable'<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">attributes to prevent it from inhibiting optimizations like redundancy<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">elimination, dead code elimination, code motion, etc.<br>
<br>
<vector_ty> llvm.tensor.transpose(token %input, <n x i32> %new_layout)<br>
<br>
<br>
** OPERANDS:<br>
----------------------<br>
<br>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">==============================================================================|<br>
Operand <span class="apple-converted-space"> </span> | Description |<br>
=============|================================================================|<br>
%input | Token value representing the input tensor |<br>
-------------|----------------------------------------------------------------|<br>
%new_layout<span class="apple-converted-space"> </span> | This is the new permutation of tensor layout |<br>
==============================================================================|<br>
<br>
** RESULT:<br>
------------------</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
==============================================================================|<br>
Result | Description |<br>
=============|================================================================|<br>
token value | LLVM value of token type associated with a tensor value |<br>
==============================================================================|<br>
<br>
** SEMANTICS:<br>
-----------------------<br>
The ‘llvm.tensor.transpose' intrinsic operates on the given tensor and produces<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">an output tensor with the given layout. This operation changes the physical layout of</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">the input tensor and leads to changes to the shape and padding. Note that the operation</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">does not lead to any change in the number of dimensions.<br>
<br>
Note that this intrinsic does not read nor write memory, nor does it exhibit<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">any kind of undefined behavior.<br>
<br>
<br>
** EXAMPLE:<br>
---------------------<br>
<br>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">%input = call token @llvm.tensor.typeinfo(<240 x float> %tensor,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 16, i32 5, i32 3>, <3 x i32> <i32 0, i32 1, i32 2>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 3, i32 2, i32 1>)<br>
<br>
; The first argument is the input virtual tensor register and the second<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">argument is the new permutation of the layout of the input tensor. This<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">operation produces a tensor of layout <2, 0, 1>. </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%output = call <240 x float> @llvm.tensor.transpose(token %input,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 2, i32 0, i32 1>)<br>
<br>
<br>
%typed_output = call token @llvm.tensor.typeinfo(<240 x float> %output,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 3, i32 16, i32 5>, <3 x i32> <i32 2, i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <3 x i32> <i32 1, i32 3, i32 2>)<br>
<br>
<br>
<br>
*** DESIGN OF TENSOR EXTENSIONS IN LLVM<br>
=========================================================================================<br>
Tensor extensions we have added to LLVM are described in the<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">document here </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">(<a href="https://urldefense.com/v3/__https:/docs.google.com/document/d/1A3xbrtouckRsPz94v2XttjoaTSqQlz1pSzVe80-Jmro/edit?usp=sharing__;!!DZ3fjg!vQEzOO44fdipIUpplKG3F5Q6ZJXDI24nlWm35qyIyzPFGcJa_7d6JONzCoX39As5qHkThMGsxBIZiahRQkU$" title="https://urldefense.com/v3/__https:/docs.google.com/document/d/1A3xbrtouckRsPz94v2XttjoaTSqQlz1pSzVe80-Jmro/edit?usp=sharing__;!!DZ3fjg!vQEzOO44fdipIUpplKG3F5Q6ZJXDI24nlWm35qyIyzPFGcJa_7d6JONzCoX39As5qHkThMGsxBIZiahRQkU$"><span style="color:#0563C1">https://docs.google.com/document/d/1A3xbrtouckRsPz94v2XttjoaTSqQlz1pSzVe80-Jmro/edit?usp=sharing</span></a>).<br>
<br>
<br>
<br>
<br>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">===============================================================================|<br>
LLVM Tensor Intrinsics | Frontend Equivalent | Target<span class="apple-converted-space"> </span>Equivalent |<br>
===========================|=======================|===========================|<br>
llvm.tensor.matmul | XLA dot op | |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.contract | XLA dot general op <span class="apple-converted-space"> </span> | | <br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.umma | | Intel AMX mma instruction,|</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> | |<span class="apple-converted-space"> </span>Power MMA instruction |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.smma | | Intel AMX mma instruction,|</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> | |<span class="apple-converted-space"> </span>Power MMA instruction |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.usmma | | Intel AMX mma instruction,|</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> | |<span class="apple-converted-space"> </span>Power MMA instruction |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.summa | | Intel AMX mma instruction,|</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> | |<span class="apple-converted-space"> </span>Power MMA instruction |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.convolution | XLA convolution op | NVDLA convolution<span class="apple-converted-space"> </span> |</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> | |<span class="apple-converted-space"> </span>instruction |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.tanh | XLA element-wise op <span class="apple-converted-space"> </span>|<span class="apple-converted-space"> </span>NVDLA element-wise<span class="apple-converted-space"> </span> |</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> | |<span class="apple-converted-space"> </span>instruction |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.sigmoid | | NVDLA element-wise |</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> | | <span class="apple-converted-space"> </span>instruction|<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.relu <span class="apple-converted-space"> </span>| | NVDLA element-wise<span class="apple-converted-space"> </span> |</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> | |<span class="apple-converted-space"> </span>instruction |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.broadcast | XLA broadcast op | Intel AMX fill instruction|<br>
---------------------------|---------------------------------------------------|<br>
llvm.tensor.load | | Intel AMX load instruction|<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.store | | Intel AMX store<span class="apple-converted-space"> </span> |</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> | |<span class="apple-converted-space"> </span>instruction| |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.reduce.max | XLA reduce window op | NVDLA pooling instruction |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.reduce.min | XLA reduce window op | NVDLA pooling instruction | <br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.reduce.add <span class="apple-converted-space"> </span>| XLA reduce window op | |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.reduce.mul | XLA reduce window op | |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.reduce.and | XLA reduce window op | |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.reduce.or | XLA reduce window op | | <br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.reduce.xor | XLA reduce window op | |<br>
---------------------------|-----------------------|---------------------------|<br>
lvm.tensor.reshape.block | OneDNN Layouts | |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.reshape.permute| Tensorflow reshape op | | <br>
---------------------------|-----------------------|---------------------------|<br>
lllvm.tensor.transpose | Tensorflow transpose<span class="apple-converted-space"> </span> |<span class="apple-converted-space"> </span>NVDLA reshape instruction | </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> |<span class="apple-converted-space"> </span>op <span class="apple-converted-space"> </span>|<span class="apple-converted-space"> </span> |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.pad |<span class="apple-converted-space"> </span>XLA pad op <span class="apple-converted-space"> </span> | |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.concat | XLA concat op | NVDLA reshape instruction |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.tensor.tovector | | Power unprime instruction |<br>
---------------------------|-----------------------|---------------------------|<br>
llvm.vector.totensor | | Power prime instruction |</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">===============================================================================|<br>
<br>
<br>
<br>
*** LOWERING STRATEGY<br>
================================================================================================<br>
<br>
The lowering strategy is divided into three stages:<br>
* Lower the N-dimensional tensor operations to 2-dimensional tensor operations.<br>
* Lower 2-dimensional tensor operations to target-agnostic 1-D and 2-D tensor<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> intrinsics.<br>
* Legalize the target-agnostic 1-D and 2-D tensor intrinsics to target-specific<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> intrnsics.<br>
<br>
<br>
*** LOWERING EXAMPLE<br>
------------------------------------------------------------<br>
<br>
As an example, we want to lower the following code with matmul intrinsic in<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">LLVM IR:<br>
<br>
%tensor1_token = call token @llvm.tensor.typeinfo(<640000 x i32>* byval(<640000 x i32>) %tensor1,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <4 x i32> <i32 1, i32 10, i32 800, i32 800>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <4 x i32> <i32 0, i32 1, i32 2, i32 3>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <4 x i32> <i32 0, i32 0, i32 0, i32 0>)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
%tensor2_token = call token @llvm.tensor.typeinfo(<640000 x i32>* byval(<640000 x i32>) %tensor2,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <4 x i32> <i32 1, i32 10, i32 800, i32 800>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <4 x i32> <i32 0, i32 1, i32 2, i32 3>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <4 x i32> <i32 0, i32 0, i32 0, i32 0>)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%tensor3 = call <64000 x i32> @llvm.tensor.matmul(token %tensor1_token, token %tensor2_token)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%tensor3_token = call token @llvm.tensor.typeinfo(<640000 x i32>* byval(<640000 x i32>) %tensor3,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <4 x i32> <i32 1, i32 10, i32 800, i32 800>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <4 x i32> <i32 0, i32 1, i32 2, i32 3>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <4 x i32> <i32 0, i32 0, i32 0, i32 0>)<br>
<br>
<br>
The code gets lowered to the following code in the first lowering stage:<br>
<br>
%malloc_ptr1 = call i8* @malloc(…)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%malloc_ptr2 = call i8* @malloc(…)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%malloc_ptr3 = call i8* @malloc(…)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%tensor_ptr1 = bitcast i8*<span class="apple-converted-space"> </span></span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black">%malloc_ptr1 to i32*<br>
%tensor_ptr2 = bitcast i8*<span class="apple-converted-space"> </span></span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black">%malloc_ptr2 to i32*<br>
%tensor_ptr3 = bitcast i8*<span class="apple-converted-space"> </span></span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black">%malloc_ptr3 to i32*</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
for (unsigned I = 0; I < 1; I++)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
for (unsigned J = 0; J < 10; J++)<span class="apple-converted-space"> </span></span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
// Compute the indices into the %tensor1 and %tensor2 and load the feature<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> //<span class="apple-converted-space"> </span>maps</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
….</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%matrix_ptr1 = getelementptr i32* %tensor_ptr1, ….</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%matrix_ptr2 = getelementptr i32* %tensor_ptr2, ….</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%matrix_ptr3 = getelementptr i32* %tensor_ptr3, ….</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%matrix1_token = call token @llvm.tensor.load(i32* %matrix_ptr1,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 800, i32 800>, </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <span class="apple-converted-space"> </span>….)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%matrix2_token = call token @llvm.tensor.load(i32* %matrix_ptr2,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 800, i32 800>, </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <span class="apple-converted-space"> </span>….)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%matrix3 = call <640000 x i32> @llvm.tensor.matmul(token %matrix1_token,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> token %matrix2_token)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%matrix3_token = call token @llvm.tensor.typeinfo(<640000 x i32> %matrix3,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <32 800, i32 800>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 1>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 0, i32 0>)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
call void @llvm.tensor.store(i32* %tensor_ptr3, token %matrix3_token, …)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
….</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
<br>
<br>
<br>
After the second lowering stage, the 2-dimensional matmul intrinsic gets lowered<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">to 2-dimensional target-agnostic intrinsics:<br>
<br>
%malloc_ptr1 = call i8* @malloc(…)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%malloc_ptr2 = call i8* @malloc(…)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%malloc_ptr3 = call i8* @malloc(…)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%tensor_ptr1 = bitcast i8*<span class="apple-converted-space"> </span></span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black">%malloc_ptr1 to i32*<br>
%tensor_ptr2 = bitcast i8*<span class="apple-converted-space"> </span></span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black">%malloc_ptr2 to i32*<br>
%tensor_ptr3 = bitcast i8*<span class="apple-converted-space"> </span></span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black">%malloc_ptr3 to i32*</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
for (unsigned I = 0; I < 1; I++)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
for (unsigned J = 0; J < 10; J++)<span class="apple-converted-space"> </span></span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
for (unsigned M = 0; M < 800; M+=16)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
for (unsigned N = 0; N < 800; N+=16)<span class="apple-converted-space"> </span></span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
for (unsigned K = 0; K < 800; K+=16)<span class="apple-converted-space"> </span></span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"> // Compute the indices into the %tensor1 and %tensor2<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> //<span class="apple-converted-space"> </span>and load the tiles</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
….</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%tile1_token = call token @llvm.tensor.load(i32* %tile_ptr1,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 16, i32 16>, </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <span class="apple-converted-space"> </span>….)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%tile2_token = call token @llvm.tensor.load(i32* %tile_ptr2,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 16, i32 16>, </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> ….)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
<span class="apple-converted-space"> </span>%tile2_token = call token @llvm.tensor.load(i32* %acc_ptr, <span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <i32 16, i32 16>, </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <span class="apple-converted-space"> </span>….)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%acc = call <256 x i32> @llvm.tensor.smma(token %acc_token,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> token %tile1_token,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> token %tile2_token)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%new_acc_token = call token @llvm.tensor.typeinfo(<256 x i32> %acc,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <2 x i32> <32 16, i32 16>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> ...)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"> .…<br>
}<br>
<span class="apple-converted-space"> </span>call void @llvm.tensor.store(i32* %acc_ptr, token %new_acc_token, …)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
….<br>
<br>
<br>
<br>
Last stage of lowering is the legalization stage. In this example, we lower down<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">to Intel AMX intrinsics:<br>
<br>
%malloc_ptr1 = call i8* @malloc(…)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%malloc_ptr2 = call i8* @malloc(…)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%malloc_ptr3 = call i8* @malloc(…)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%tensor_ptr1 = bitcast i8*<span class="apple-converted-space"> </span></span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black">%malloc_ptr1 to i32*<br>
%tensor_ptr2 = bitcast i8*<span class="apple-converted-space"> </span></span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black">%malloc_ptr2 to i32*<br>
%tensor_ptr3 = bitcast i8*<span class="apple-converted-space"> </span></span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black">%malloc_ptr3 to i32*</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
for (unsigned I = 0; I < 1; I++)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
for (unsigned J = 0; J < 10; J++)<span class="apple-converted-space"> </span></span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
for (unsigned M = 0; M < 800; M+=16)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
for (unsigned N = 0; N < 800; N+=16)<span class="apple-converted-space"> </span></span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
for (unsigned K = 0; K < 800; K+=16)<span class="apple-converted-space"> </span></span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"> // Compute the indices into the %tensor1 and %tensor2 and<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> //<span class="apple-converted-space"> </span>load the tiles</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
….</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%cast_tile_ptr1 = bitcast i32* %tile_ptr1 to i8*<br>
%cast_tile_ptr2 = bitcast i32* %tile_ptr2 to i8*<br>
%cast_acc_ptr = bitcast i32* %acc_ptr to i8*<br>
%tile1_amx = call x86_amx @llvm.x86.tileloadd64.internal(i16 16,</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> i16 64, i8* %cast_tile_ptr1, i64 3200)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%tile2_amx = call x86_amx @llvm.x86.tileloadd64.internal(i16 16,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> i16 64, i8* %cast_tile_ptr2, i64 3200)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%acc_amx = call x86_amx @llvm.x86.tileloadd64.internal(i16 16,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> i16 64, i8* %cast_acc_ptr, i64 3200)<br>
%mma_amx = call x86_amx @llvm.x86.tdpbssd.internal(i16 16,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> i16 64, i16 64, x86_amx %acc_amx,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> x86_amx %tile2_amx , x86_amx %tile1_amx)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
.....<br>
}<br>
call void @llvm.x86.tilestored64.internal(i16 16, i16 64, i8* %cast_acc_ptr,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> i64 3200, x86_amx %mma_amx)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
….<br>
<br>
<br>
<br>
*** COMPATIBILITY WITH AND BENEFITS OVER MATRIX EXTENSIONS<br>
==========================================================================================<br>
The existing matrix extensions model vectors as matrices in LLVM can co-exist and can<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">be used with the tensor extensions that we propose. We argue that our tensor extensions<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">provide an extensible and flexible long-term solution that LLVM developers can<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">experiment with and adopt overtime. We believe that our tensor extensions provide the<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">following benefits over the existing matrix extensions:</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><br>
* Our tensor extensions support an arbitrary number of dimensions for tensors. This<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> affords LLVM developers the flexibility to use higher-dimensional tensors as<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> opposed to confining to rigidly supporting two dimensional tensors only. This<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> support for generality also makes the tensor extensions more easy to maintain in<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> the future.<br>
* Currently, information about matrix shapes and layouts is encoded within the<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> matrix intrinsics in LLVM. They do not provide a separation between the matrix<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> properties and matrix operations. This makes the existing matrix extensions<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> rigid and difficult to extend in the future because if developers decide to<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> encode more matrix properties in the IR, they would have to modify all matrix<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> intrinsics and modify several lines of code using these matrix extensions.<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> Our tensor extensions provide a separation between the tensor concept and tensor </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <span class="apple-converted-space"> </span>operations, thereby providing the flexibility of extending the tensor properties<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> represented in the IR without having to modify all the tensor operations that<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> operate on tensors. Note that this flexibility also allows supporting new kinds<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> of tensors (such as sparse and ragged tensors) more easily in the future as well.<br>
* Matrix padding is modelled using vector shuffle instructions. This requires<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> optimizations, analyses and transformations to infer padding information by<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> carefully inspecting all the shuffle instructions and their masks. We encode<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> tensor padding information as a set of tensor properties directly represented<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> and readily available in the IR and we use an intrinsic to represent a padding<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> operation.<br>
<br>
<br>
*** CURRENT STATUS OF THE IMPLEMENTATION<br>
======================================================================================<br>
* Lowering of most high-level tensor operations to LLVM scalar and vector<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> instructions is supported.<br>
* Tensor code generation framework is capable of targeting Intel AMX.<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> Support for targeting NVDLA and NVIDIA tensor cores is in progress.<br>
* Lowering support to target Intel VNNI and Hexagon Vector Extension (HVX) is<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> underway.<br>
* Example of lowering from Julia to the proposed tensor extensions is in the<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> design document </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <span class="apple-converted-space"> </span>(<a href="https://urldefense.com/v3/__https:/docs.google.com/document/d/1A3xbrtouckRsPz94v2XttjoaTSqQlz1pSzVe80-Jmro/edit*heading=h.17j13gwxto8i__;Iw!!DZ3fjg!vQEzOO44fdipIUpplKG3F5Q6ZJXDI24nlWm35qyIyzPFGcJa_7d6JONzCoX39As5qHkThMGsxBIZctWEYsA$" title="https://urldefense.com/v3/__https:/docs.google.com/document/d/1A3xbrtouckRsPz94v2XttjoaTSqQlz1pSzVe80-Jmro/edit*heading=h.17j13gwxto8i__;Iw!!DZ3fjg!vQEzOO44fdipIUpplKG3F5Q6ZJXDI24nlWm35qyIyzPFGcJa_7d6JONzCoX39As5qHkThMGsxBIZctWEYsA$"><span style="color:#0563C1">https://docs.google.com/document/d/1A3xbrtouckRsPz94v2XttjoaTSqQlz1pSzVe80-Jmro/edit#heading=h.17j13gwxto8i</span></a>) .<br>
<br>
<br>
*** CURRENT TESTING SUPPORT<br>
========================================================================================<br>
<br>
Currently, the tests are written in C/C++. The tensor operations are written using<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">“dummy” functions such as tensor_typeinfo, tensor_matmul and so on as shown in the<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black">following example:<br>
<br>
typedef int _tensor_t __attribute__((__vector_size__(25600000)));<br>
typedef int _shape_t __attribute__((__vector_size__(16), __aligned__(4)));</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
typedef int _layout_t __attribute__((__vector_size__(16), __aligned__(4)));</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
typedef int _padding_t __attribute__((__vector_size__(16), __aligned__(4)));</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
typedef int _token_t;</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
void example(_tensor_t tensor1, _tensor_t tensor2) {</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
_shape_t shape = {1, 10, 800, 800};<br>
_layout_t layout = {0, 1, 2, 3};</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
_padding_t padding = {0, 0, 0, 1};</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"> <br>
/* Define type information for the input tensors */<br>
_token_t tensor1_token = tensor_typeinfo(tensor1, shape, layout, padding);</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
_token_t tensor2_token = tensor_typeinfo(tensor2, shape, layout, padding);</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
<br>
/* Perform Matmul */<br>
_tensor_t tensor3 = tensor_matmul(tensor1_token, tensor2_token); <br>
<br>
/* Define type information for the output tensor */<br>
_token_t tensor3_token = tensor_typeinfo(tensor3, shape, layout, padding);</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
}</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
<br>
<br>
The above code gets translated into tensor intrinsics in LLVM IR:<br>
<br>
define void @example(<6400000 x i32>* byval(<6400000 x i32>) %tensor1,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <6400000 x i32>* byval(<6400000 x i32>) %tensor2) {</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%tensor1_token = call token @llvm.tensor.typeinfo(</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <span class="apple-converted-space"> </span><6400000 x i32>* byval(<6400000 x i32>) %tensor1,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <4 x i32> <i32 1, i32 10, i32 800, i32 800>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <4 x i32> <i32 0, i32 1, i32 2, i32 3>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <4 x i32> <i32 0, i32 0, i32 0, i32 0>)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%tensor2_token = call token @llvm.tensor.typeinfo(</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <span class="apple-converted-space"> </span><6400000 x i32>* byval(<6400000 x i32>) %tensor2,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <4 x i32> <i32 1, i32 10, i32 800, i32 800>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <4 x i32> <i32 0, i32 1, i32 2, i32 3>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <4 x i32> <i32 0, i32 0, i32 0, i32 0>)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%tensor3 = call <640000 x i32> @llvm.tensor.matmul(</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> token %tensor1_token,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> token %tensor2_token)</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
</span><span style="font-family:"Cambria Math",serif;color:black"></span><span style="font-family:"Courier New";color:black"><br>
%tensor3_token = call token @llvm.tensor.typeinfo(</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <span class="apple-converted-space"> </span><6400000 x i32>* byval(<6400000 x i32>) %tensor3,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <4 x i32> <i32 1, i32 10, i32 800, i32 800>, </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <span class="apple-converted-space"> </span><4 x i32> <i32 0, i32 1, i32 2, i32 3>,<span class="apple-converted-space"> </span></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> <4 x i32> <i32 0, i32 0, i32 0, i32 0>)<br>
<br>
ret <br>
}<br>
<br>
<br>
Link to the Google doc for this proposal:</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"><a href="https://urldefense.com/v3/__https:/docs.google.com/document/d/1r3bHerTFqloldHH-OcMNF2kuCaYOB5JEeGtb7ftLmNg/edit?usp=sharing__;!!DZ3fjg!vQEzOO44fdipIUpplKG3F5Q6ZJXDI24nlWm35qyIyzPFGcJa_7d6JONzCoX39As5qHkThMGsxBIZ-YbQ2IE$" title="https://urldefense.com/v3/__https:/docs.google.com/document/d/1r3bHerTFqloldHH-OcMNF2kuCaYOB5JEeGtb7ftLmNg/edit?usp=sharing__;!!DZ3fjg!vQEzOO44fdipIUpplKG3F5Q6ZJXDI24nlWm35qyIyzPFGcJa_7d6JONzCoX39As5qHkThMGsxBIZ-YbQ2IE$"><span style="color:#0563C1">https://docs.google.com/document/d/1r3bHerTFqloldHH-OcMNF2kuCaYOB5JEeGtb7ftLmNg/edit?usp=sharing</span></a><br>
<br>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="font-family:"Courier New";color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;caret-color: rgb(0, 0, 0);word-spacing:0px">
<span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>