<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Thank you for the interesting proposal Akash (et al). I have a few other questions:<div class=""><br class=""></div><div class="">Florian pointed out that this is a very large proposal which is introducing a bunch of new concepts which makes it difficult to review. My major concern with it is that it is proposing a single tensor model for LLVM, something that is inappropriate for a wide variety of frameworks, and doesn’t appear to be very general. For example, it isn’t clear how to model the strided tensor model of pytorch, doesn’t appear to support dynamic shapes, sparse tensors, and it isn’t clear (in a quick reading) what the op-extensibiliy story is. Further, there are a bunch of design decisions inherent to this approach (e.g. putting the layout information on the ops, instead of in the types) that make certain optimizations (e.g. layout transformations) more complicated.</div><div class=""><br class=""></div><div class="">This isn’t to say that this is the _wrong_ design, merely that it is only one of many plausible and important designs. Standardizing "one thing" in LLVM can have a chilling effect on innovation (particularly for such a rapidly evolving field) which is one of the reasons that MLIR favors an “open extensibility” approach.</div><div class=""><br class=""></div><div class="">In terms of detailed design, it isn’t clear to me that representing heap allocated things like this as a token type will work out well. There have been a variety of proposals over the years (incl adding F90 style arrays as a first class entity) that haven’t worked well because of a wide variety of design assumptions in LLVM). The <a href="https://llvm.org/docs/LangRef.html#token-type" class="">token type</a> in particular is not composable with control flow, functional calls and other things, and ML models frequently have loops and other controls flow in them - how do you plan to represent that?</div><div class=""><br class=""></div><div class="">In your convolution operation, it doesn’t look like you’re handling the various edge conditions (replicating, mirroring, etc) common in ML frameworks. How do you plan to handle that? Similarly, how do you handle quantization?</div><div class=""><br class=""></div><div class="">As per the motivation section, you point out "Crucially, however, MLIR does not have a low-level code generation framework that is retargetable to diverse hardware: it relies on LLVM for this purpose.” I happen to agree with you, but the lack of this in MLIR isn’t evidence that LLVM IR is the natural place to put matrix lowering support. Why do you think LLVM IR is a better place to put this than a high level IR? Whether it is MLIR, XLA, or something else, it seems that there is a very clear separation of concerns here, and (as you point out) LLVM is being successfully used as the backend for a wide variety of tensor compilers already.</div><div class=""><br class=""></div><div class="">Finally, I’m also a bit concerned because the IR extensions are not really the meat of this proposal - this is effectively proposing something akin to the entire LLVM “CodeGen” framework but for tensors. The IR abstractions and framework need to be co-designed together, and it isn’t clear how general or powerful the framework will turn out to be. We’ve seen a *LOT* of ML compiler frameworks (incl notably Glow, XLA, TVM, etc) that are successful handling important subsets of the ML inference space, but very few have scaled up to solving the full generality of the problem.</div><div class=""><br class=""></div><div class="">-Chris</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Nov 15, 2021, at 10:18 AM, Kothari, Akash via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
For those who may have been having trouble viewing the RFC in plain text format, we have our proposal in a Google doc: <a href="https://docs.google.com/document/d/1IW6VIJ4lMYbGRTOle7S5QXP7Sb5UlucZ3gf-L-4Ccfs/edit?usp=sharing" class="">https://docs.google.com/document/d/1IW6VIJ4lMYbGRTOle7S5QXP7Sb5UlucZ3gf-L-4Ccfs/edit?usp=sharing</a>.
It would be great if y’all could comment in the google doc or respond via email.
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Akash Kothari</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Nov 12, 2021, at 1:28 PM, Kothari, Akash <<a href="mailto:akashk4@illinois.edu" class="">akashk4@illinois.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">**** Proposal for TLX: Tensor LLVM eXtensions<br class="">
===================================================================================<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">Authors: Akash Kothari (UIUC), Abdul Rafae Noor (UIUC), Dounia Khaldi (Intel),<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> Vikram Adve (UIUC), Yuanke Luo(Intel), Sudipta Sengupta (Amazon AWS),<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> Milind Girkar (Intel), Charith Mendis (UIUC)<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">------------------------------------------------------------------------------------<o:p class=""></o:p></span></div>
<div style="margin: 0in;" class=""><span style="font-size: 10.5pt; font-family: "Courier New";" class=""> <o:p class=""></o:p></span></div>
<div style="margin: 0in;" class=""><span style="font-size: 10.5pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; font-variant-caps: normal; text-align: start; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); word-spacing: 0px;" class="">
<span style="font-family: "Courier New";" class=""></span></div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
_______________________________________________<br class="">LLVM Developers mailing list<br class=""><a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a><br class="">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<br class=""></div></blockquote></div><br class=""></div></body></html>