<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class=""><span class=""><font color="#006d8f" class="">Hi Chris,</font></span></div>
<div class=""><span class=""><font color="#006d8f" class=""><br class="">
</font></span></div>
<div class=""><font color="#006d8f" class=""><span class="">Thank you for your questions and comments. I have reordered your questions/comments to respond to them in a logical progression. </span><span class="" style="font-family: Calibri, sans-serif;">In particular,
you have some questions about the technical design of TLX (extensibility, memory allocation, etc.) and some about *whether a tensor / matrix code-gen framework belongs in LLVM in the first place*. I thought I should address the latter first. </span></font></div>
<div class=""><span class=""><font color="#006d8f" class=""><br class="">
</font></span></div>
<div class=""><span class=""><font color="#006d8f" class="">-Akash</font></span></div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Nov 27, 2021, at 7:57 PM, Chris Lattner <<a href="mailto:clattner@nondot.org" class="">clattner@nondot.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Thank you for the interesting proposal Akash (et al). I have a few other questions:
<div class=""><br class="">
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>
<blockquote type="cite" class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div class="">As per the motivation section, you point out "Crucially, however, MLIR does not have a low-level code generation framework that is retargetable to diverse hardware: it relies on LLVM for this purpose.” I happen to agree with you, but the lack
of this in MLIR isn’t evidence that LLVM IR is the natural place to put matrix lowering support. Why do you think LLVM IR is a better place to put this than a high level IR? Whether it is MLIR, XLA, or something else, it seems that there is a very clear
separation of concerns here, and (as you point out) LLVM is being successfully used as the backend for a wide variety of tensor compilers already.</div>
</div>
</blockquote>
<br class="">
</div>
<div>
<div class=""><font color="#006d8f" class="">I think LLVM is the natural place for put support for tensor lowering for the following reasons:</font></div>
<div class=""><font color="#006d8f" class=""><br class="">
</font></div>
<div class="">
<ul class="MailOutline">
<li class=""><font color="#006d8f" class="">Compilers such as TVM, Halide, XLA,Glow, etc. use LLVM for backend code generation for different hardware architectures, so it makes sense to add tensor lowering support in an abstraction layer shared across multiple
compilers. Today, compilers such as TVM and Halide, for instance, have separate backends to generate target-specific intrinsics for different targets, which is a serious weakness. These compilers can target a common set of target-agnostic intrinsics instead
to target multiple tensor architectures and benefit from community-wide shared improvements and efforts. </font></li><li class=""><font color="#006d8f" class="">Languages such as C/C++, Rust, DPC++, Julia, etc. do not have frontends for compilers like MLIR, XLA, TVM, etc.yet. Developing frontends for these languages for production use requires non-trivial engineering effort.
Extending LLVM with our extensions and getting the existing languages frontends to target our extensions would require relatively less engineering effort and time. </font></li><li class=""><font color="#006d8f" class="">TLX could be added to the LLVM dialect in MLIR. Also, lessons learned from the experience of supporting retargetable code generation in LLVM for modern tensor architectures could be very valuable and could help inspire
ideas for new dialects and abstractions to make MLIR retargetable, too.</font></li></ul>
<div class=""><br class="">
</div>
</div>
<div class=""><br class="">
</div>
</div>
<div>
<blockquote type="cite" class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div class="">Finally, I’m also a bit concerned because the IR extensions are not really the meat of this proposal - this is effectively proposing something akin to the entire LLVM “CodeGen” framework but for tensors. The IR abstractions and framework need
to be co-designed together, and it isn’t clear how general or powerful the framework will turn out to be. We’ve seen a *LOT* of ML compiler frameworks (incl notably Glow, XLA, TVM, etc) that are successful handling important subsets of the ML inference space,
but very few have scaled up to solving the full generality of the problem.</div>
</div>
</blockquote>
</div>
<div><br class="">
</div>
<div><font color="#006d8f" class="">I agree with you that IR extensions and the code generation framework must be co-designed together. We have done exactly just that on our end in collaboration with folks from Intel, Qualcomm, IBM and AWS. <font face="Calibri, sans-serif" class=""><span class="" style="caret-color: rgb(0, 112, 192);">There
are three main parts to this end-to-end tensor support in LLVM: (1) tensor IR (TLX), (2) code generation that includes lowering from N-d to 2-d and the legalization support for target-agnostic to target-specific intrinsics, (3) </span></font><span class="" style="caret-color: rgb(0, 112, 192); font-family: Calibri, sans-serif;">extension</span>s
to <font face="Calibri, sans-serif" class=""><span class="" style="caret-color: rgb(0, 112, 192);">Target Transform Info (TTI) for efficient matrix (2d) code generation</span></font> In the future, we will post two more RFCs about TTI enhancements and lowering
strategies. Our core proposal is here: </font><a href="https://urldefense.com/v3/__https://docs.google.com/document/d/1IW6VIJ4lMYbGRTOle7S5QXP7Sb5UlucZ3gf-L-4Ccfs/edit?usp=sharing__;!!DZ3fjg!uQXYK87Zi2_ZCnuI1ZiCnrs6lz_zYFoJIU1kidxX5f1LWPx8q6p9Qh2JtDemE1QZv338OwHchCwHIF5V4pQ$" class="">https://docs.google.com/document/d/1IW6VIJ4lMYbGRTOle7S5QXP7Sb5UlucZ3gf-L-4Ccfs/edit?usp=sharing</a>.<font color="#0042aa" class="">
</font><font color="#006d8f" class="">People who may be interested in learning more details about the IR extensions and the other parts of this support can take a look at the specification document here: </font><a href="https://urldefense.com/v3/__https://docs.google.com/document/d/1A3xbrtouckRsPz94v2XttjoaTSqQlz1pSzVe80-Jmro/edit?usp=sharing__;!!DZ3fjg!uQXYK87Zi2_ZCnuI1ZiCnrs6lz_zYFoJIU1kidxX5f1LWPx8q6p9Qh2JtDemE1QZv338OwHchCwHkZyqnb0$" class="">https://docs.google.com/document/d/1A3xbrtouckRsPz94v2XttjoaTSqQlz1pSzVe80-Jmro/edit?usp=sharing</a>.</div>
<div><br class="">
</div>
<br class="">
<blockquote type="cite" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Florian pointed out that this is a very large proposal which is introducing a bunch of new concepts which makes it difficult to review. My major concern with it is that it is proposing a single tensor model for LLVM, something that is inappropriate
for a wide variety of frameworks, and doesn’t appear to be very general. For example, it isn’t clear how to model the strided tensor model of pytorch, doesn’t appear to support dynamic shapes, sparse tensors, and it isn’t clear (in a quick reading) what the
op-extensibiliy story is. Further, there are a bunch of design decisions inherent to this approach (e.g. putting the layout information on the ops, instead of in the types) that make certain optimizations (e.g. layout transformations) more complicated.</div>
<div class=""><br class="">
</div>
</div>
</blockquote>
<br class="">
<div class=""><font color="#006d8f" class="">I think there is some misunderstanding about some of the extensions we are proposing. We do not propose that the tensor operations have layout information on them; in fact, we propose an intrinsic called llvm.tensor.typeinfo
should have the layout information (and other tensor type information) and help decouple tensor type information from the tensor operations. The tensor load intrinsic also has layout information embedded. Current matrix extensions have type information for
matrices embedded in intrinsics for matrix operations such as transpose, matrix multiply, etc. We also support strided tensor loads and stores — these are akin to strided tensors in Pytorch.</font></div>
<div class=""><font color="#006d8f" class=""><br class="">
</font></div>
<div class=""><font color="#006d8f" class="">Decoupling tensor type information and intrinsics for tensor operations, allows us to extend the tensor type information, if needed, without having to make any changes to other intrinsics for tensor operations. We
do not propose extensions for sparse tensors, but one can support sparse tensors by introducing a new variant of typeinfo intrinsic to describe sparse tensors and continue using the intrinsics for tensor operations we propose. Supporting sparse tensors would
also require adding new intrinsics for operations such as coalescing, for example. Same applies for ragged tensors. </font></div>
<div class=""><font color="#006d8f" class=""><br class="">
</font></div>
<div class=""><font color="#006d8f" class="">By representing shape information in llvm.tensor.typeinfo as a vector of dimension sizes, dynamic shapes can be represented in this intrinsic using a vector of SSA values of dimension sizes (and not just constant
values). </font></div>
<div><br class="">
</div>
<br class="">
<blockquote type="cite" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">This isn’t to say that this is the _wrong_ design, merely that it is only one of many plausible and important designs. Standardizing "one thing" in LLVM can have a chilling effect on innovation (particularly for such a rapidly evolving field)
which is one of the reasons that MLIR favors an “open extensibility” approach.</div>
<div class=""><br class="">
</div>
</div>
</blockquote>
<br class="">
<div><font color="#006d8f" class=""><span class="" style="caret-color: rgb(255, 0, 0); font-family: Calibri, sans-serif;">While it’s true that this is only one of many plausible designs, that will be true of every LLVM extension that is ever proposed. </span><span class="" style="caret-color: rgb(255, 0, 0); font-family: Calibri, sans-serif;">We
do not think that adding TLX is going to stifle innovation. </span><font face="Calibri, sans-serif" class=""><span class="" style="caret-color: rgb(255, 0, 0);">It is absolutely true that LLVM itself has only limited support for extensibility, whereas MLIR
is inherently more extensible, but that is not a reason not to add new functionality in LLVM within the limitations of what is possible. Our tensor extensions we propose are extensible for the aforementioned reasons. We expect these extensions to be experimented
with and refined further </span></font><span class="" style="caret-color: rgb(255, 0, 0); font-family: Calibri, sans-serif;">by the community</span></font><font face="Calibri, sans-serif" class=""><span class="" style="caret-color: rgb(255, 0, 0);"><font color="#006d8f" class="">.
We are open to any ideas that you and other folks in the LLVM community may have to make these extensions more general and more extensible. Note that we have added a section on the methodology to extend TLX in our proposal document: </font><a href="https://urldefense.com/v3/__https://docs.google.com/document/d/1IW6VIJ4lMYbGRTOle7S5QXP7Sb5UlucZ3gf-L-4Ccfs/edit*heading=h.ltfq7r4wczwl__;Iw!!DZ3fjg!uQXYK87Zi2_ZCnuI1ZiCnrs6lz_zYFoJIU1kidxX5f1LWPx8q6p9Qh2JtDemE1QZv338OwHchCwH7gS2pPk$" class="">https://docs.google.com/document/d/1IW6VIJ4lMYbGRTOle7S5QXP7Sb5UlucZ3gf-L-4Ccfs/edit#heading=h.ltfq7r4wczwl</a>.</span></font></div>
<div><br class="">
</div>
<div><br class="">
</div>
<br class="">
<blockquote type="cite" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">In terms of detailed design, it isn’t clear to me that representing heap allocated things like this as a token type will work out well. There have been a variety of proposals over the years (incl adding F90 style arrays as a first class entity)
that haven’t worked well because of a wide variety of design assumptions in LLVM). The <a href="https://urldefense.com/v3/__https://llvm.org/docs/LangRef.html*token-type__;Iw!!DZ3fjg!tuN5EzICOHuuxFRM6_umsefbjEjqODtXvR1oCW2PszH1lChn4IWVQpt76RTMS0R4A4o$" class="">token
type</a> in particular is not composable with control flow, functional calls and other things, and ML models frequently have loops and other controls flow in them - how do you plan to represent that?</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>
<div class=""><font color="#006d8f" class="">We are *not* proposing to represent heap-allocated objects using token type. Values of token type merely represent SSA tensor values. Tensor loads and stores contain the information about the tensors they read from
or write to memory. We have implemented these intrinsics in LLVM already and have not encountered any problems so far. </font></div>
<div class=""><font color="#006d8f" class=""><br class="">
</font></div>
<div class=""><font color="#006d8f" class="">In order to handle cases where tensor information from block1 and block2 has to be used in block3:</font></div>
<div class=""><font color="#006d8f" class=""><br class="">
</font></div>
<div class=""><i class=""><font color="#006d8f" class="">block1:</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""> ….</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""> %tensor1_info = call token @llvm.tensor.typeinfo(<256 x i8> %tensor1, <3 x i32> %shape1, <3 x i32> %layout1, <3 x i32> %padding1)</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""> br %block3</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""><br class="">
</font></i></div>
<div class="">
<div class=""><i class=""><font color="#006d8f" class="">block2:</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""> .....</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""> %tensor2_info = call token @llvm.tensor.typeinfo(<256 x i8> %tensor2, <3 x i32> %shape2, <3 x i32> %layout2, <3 x i32> %padding2)</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""> br %block3</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""><br class="">
</font></i></div>
<div class=""><i class=""><font color="#006d8f" class="">block3:</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""> ….</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""> %tensor3 = phi < 256 x i8> [%tensor1, %block1], [%tensor2, %block2]</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""> %shape3 = phi <3 x i32> [%shape1, %block1], [%shape2, %block2]</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""> %layout3 = phi <3 x i32> [%layout1, %block1], [%layout2, %block2]</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""> %padding3 = phi <3 x i32> [%padding1, %block1], [%padding2, %block2]</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""> %tensor3_info = call token @llvm.tensor.typeinfo(<256 x i8> %tensor3, <3 x i32> %shape3, <3 x i32> %layout3, <3 x i32> %padding3)</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""> …..</font></i></div>
</div>
<div class=""><font color="#006d8f" class=""><br class="">
</font></div>
<div class=""><font color="#006d8f" class=""><br class="">
</font></div>
<div class=""><font color="#006d8f" class="">We do not discuss how tensor information could be passed across function call boundaries, but we could use<font face="Calibri, sans-serif" class=""> 3 new parameter attributes: tensorshape, tensorlayout, tensorpad.
These attributes indicate what property of a tensor parameter they represent. We could also introduce an attribute named tensorargid to give each set of parameters representing a tensor and its shape, layout and padding a unique ID.</font></font></div>
<div class=""><font face="Calibri, sans-serif" class="" color="#006d8f"><br class="">
</font></div>
<div class=""><i class=""><font color="#006d8f" class="">define void @callee (<256 x i8> <b class="">tensorargid 0 </b>%tensor1, <2 x i32><b class=""> tensorshape tensorargid 0 </b>%shape1, <2 x i32> <b class="">tensorlayout tensorargid 0 </b>%layout1, <2 x
i32> <b class="">tensorpad tensorargid 0 </b>%pad1, <256 x i8> <b class="">tensorargid 1</b> %tensor2, <2 x i32> <b class="">tensorshape tensorargid 1</b> %shape2, <2 x i32> <b class="">tensorlayout tensorargid 1</b> %layout2, <2 x i32> <b class="">tensorpad
tensorargid 1</b> %pad2) {</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""><br class="">
</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""> ; Define typed input tensors</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""> %typed_tensor1 = call <256 x i8> llvm.tensor.typeinfo(<256 x i8> %tensor1, <2 x i32> %shape1, <2 x i32> %layout1, <2 x i32> %pad1) </font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""> %typed_tensor2 = call <256 x i8> llvm.tensor.typeinfo(<256 x i8> %tensor2, <2 x i32> %shape2, <2 x i32> %layout2, <2 x i32> %pad2)</font></i></div>
<div class=""><i class=""><font color="#006d8f" class=""> ….</font></i></div>
<div class=""><i class=""><font color="#006d8f" class="">}</font></i></div>
<div class=""><i class=""><br class="">
</i></div>
</div>
<br class="">
<blockquote type="cite" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
<div class="">In your convolution operation, it doesn’t look like you’re handling the various edge conditions (replicating, mirroring, etc) common in ML frameworks. How do you plan to handle that? Similarly, how do you handle quantization?</div>
</div>
</blockquote>
<br class="">
<div class=""><span class=""><font color="#006d8f" class="">We could extend our convolution intrinsic to include one more operand with number of feature groups along every outer dimension to support depthwise convolutions. I am not sure what you mean by other
edge conditions such as mirroring, replicating, etc. </font></span></div>
<div class=""><i class=""><font color="#006d8f" class=""><br class="">
</font></i></div>
<div class=""><font color="#006d8f" class="">We do not propose any intrinsics for quantization in this proposal, but quantization could be added as a set of <span class="" style="caret-color: rgb(0, 112, 192); font-family: Calibri, sans-serif;">additional intrinsics
that use the same tensor typeinfo intrinsic. We are looking for feedback/ideas from the LLVM community on which specific intrinsics should be absolutely added along with this RFC. </span>As Aditya Atluri pointed out in the comment section of the specification
document, we may have to think more about how to allow vendors to support custom types and how to allow vendors to specify the legal conversions between other custom and existing LLVM types. So this is an open question that merits more discussion. However, <span class="" style="font-family: Calibri, sans-serif;">this
concern is relatively minor and should not impact most of the rest of the design details in the RFC. </span><span class="" style="font-family: Calibri, sans-serif;"> </span></font></div>
<br class="">
<blockquote type="cite" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">-Chris</div>
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Nov 15, 2021, at 10:18 AM, Kothari, Akash via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
For those who may have been having trouble viewing the RFC in plain text format, we have our proposal in a Google doc: <a href="https://urldefense.com/v3/__https://docs.google.com/document/d/1IW6VIJ4lMYbGRTOle7S5QXP7Sb5UlucZ3gf-L-4Ccfs/edit?usp=sharing__;!!DZ3fjg!tuN5EzICOHuuxFRM6_umsefbjEjqODtXvR1oCW2PszH1lChn4IWVQpt76RTMHUZES-M$" class="">https://docs.google.com/document/d/1IW6VIJ4lMYbGRTOle7S5QXP7Sb5UlucZ3gf-L-4Ccfs/edit?usp=sharing</a>.
It would be great if y’all could comment in the google doc or respond via email.
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Akash Kothari</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Nov 12, 2021, at 1:28 PM, Kothari, Akash <<a href="mailto:akashk4@illinois.edu" class="">akashk4@illinois.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">**** Proposal for TLX: Tensor LLVM eXtensions<br class="">
===================================================================================<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">Authors: Akash Kothari (UIUC), Abdul Rafae Noor (UIUC), Dounia Khaldi (Intel),<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> Vikram Adve (UIUC), Yuanke Luo(Intel), Sudipta Sengupta (Amazon AWS),<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> Milind Girkar (Intel), Charith Mendis (UIUC)<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">------------------------------------------------------------------------------------<o:p class=""></o:p></span></div>
<div style="margin: 0in;" class=""><span style="font-size: 10.5pt; font-family: "Courier New";" class=""> <o:p class=""></o:p></span></div>
<div style="margin: 0in;" class=""><span style="font-size: 10.5pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; font-variant-caps: normal; text-align: start; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); word-spacing: 0px;" class="">
<span style="font-family: "Courier New";" class=""></span></div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
_______________________________________________<br class="">
LLVM Developers mailing list<br class="">
<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a><br class="">
<a href="https://urldefense.com/v3/__https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev__;!!DZ3fjg!tuN5EzICOHuuxFRM6_umsefbjEjqODtXvR1oCW2PszH1lChn4IWVQpt76RTMjMO964A$" class="">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>