<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi Kiran,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you for the detailed explanation and investigation of the OpenMP framework for lowering and code-generation. It all looks good to me, in particular the introduction of an OpenMP MLIR dialect and the placement of OpenMP IRBuilder
in the overall scheme.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Brian’s proposal for OpenMP symbol tables says that explicit symbol table entries will be created for entities that have different attributes inside of OpenMP regions. E.g. if there’s an X in an outer scope that becomes thread private,
then there’s an additional symbol table entry in the OpenMP region scope for that symbol. That new symbol has the additional OpenMP attributes and a link back to the original entity in the outer scope. We think this organization will simplify outlining.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I don’t recall much discussion about outlining. Do you think OpenMP IRBuilder will drive the decisions about when and what to outline? After outlining, will the compiler be able to run the outlined code back through the MLIR+ transformations
and optimizations? Are there situations where the outlined code presents more opportunities for optimization?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">- Steve<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Kiran Chandramohan <Kiran.Chandramohan@arm.com><br>
<b>Date: </b>Friday, November 22, 2019 at 9:26 AM<br>
<b>To: </b>"flang-dev@lists.llvm.org" <flang-dev@lists.llvm.org><br>
<b>Cc: </b>"Doerfert, Johannes" <jdoerfert@anl.gov>, Roger Ferrer Ibáñez <rofirrim@gmail.com>, Stephen Scalpone <sscalpone@nvidia.com>, Richard Barton <Richard.Barton@arm.com>, Will Lovett <Will.Lovett@arm.com><br>
<b>Subject: </b>RFC: Summary of F18 OpenMP design<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black;background:white">Hi all,</span><span style="font-size:12.0pt;font-family:"Segoe UI";color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black;background:white"><br>
We gave a presentation [1] a few months back regarding the OpenMP design for F18 during the Flang/F18 technical call, also sent a summary mail [2] and a set of walkthroughs [3] to the mailing list. We received some feedback and have incorporated that into the
design. </span><span style="font-size:12.0pt;color:black">A summary of the design and status of the OpenMP implementation in F18 from lowering of the parse tree to LLVM IR generation is presented in this mail. (Note that Semantic and Structural checks are
not covered. Refer to Gary's biweekly mail for the status.)<br>
</span><span style="font-size:12.0pt;font-family:"Segoe UI";color:black"><br>
</span><span style="font-size:12.0pt;color:black">The proposed design can be seen in slide 10 of the presentation [1]. The design summary is as follows.<br>
i) Uses the following two components.<br>
a) MLIR [4]: Necessary since MLIR has already been chosen as the framework for building the Fortran IR (FIR) for Flang. By using MLIR for OpenMP we have a common representation for OpenMP and Fortran constructs in the MLIR framework and thereby take advantage
of optimisations and avoid black boxes.<br>
b) OpenMP IRBuilder [5]: For sharing of OpenMP codegen with Clang.<br>
<br>
ii) Current and Proposed Flow<br>
a) The current sequential code flow in Flang (Slide 5) of the presentation [1] can be summarised as follows,</span><span style="font-size:12.0pt;font-family:"Segoe UI";color:black"><br>
</span><span style="font-size:12.0pt;color:black">[Fortran code] -> Parser -> [AST] -> Lowering -> [FIR MLIR] -> Conversion -> [LLVM MLIR] -> Translation -> [LLVM IR]</span><span style="font-size:12.0pt;font-family:"Segoe UI";color:black"><br>
</span><span style="font-size:12.0pt;color:black">b) The modified flow with OpenMP (Slide 10) will have lowering of the AST to a mix of FIR and OpenMP dialects. These are then optimised and finally converted to mix of OpenMP and LLVM MLIR dialects. The mix
is translated to LLVM IR using the existing translation library for LLVM MLIR and the OpenMP IRBuilder currently under construction.</span><span style="font-size:12.0pt;font-family:"Segoe UI";color:black"><br>
</span><span style="font-size:12.0pt;color:black">[Fortran code] -> Parser -> [AST] -> Lowering -> [FIR + OpenMP MLIR] -> Conversion -> [LLVM + OpenMP MLIR] -> Translation (Use OpenMP IRBuilder) -> [LLVM IR]<br>
c) The MLIR infrastructure provides a lot of optimisation passes for loops. It is desirable that we take advantage of some of these. But the LLVM infrastructure also provides several optimisations. So there exist some questions regarding where the optimisations
should be carried out. We will decide on which framework to choose only after some experimentation. If we decide that the OpenMP construct (for e.g. collapse) can be handled fully in MLIR and that is the best place to do it (based on experiments) then we will
not use the OpenMP IRBuilder for these constructs.<br>
<br>
iii) OpenMP MLIR dialect<br>
The proposed plan involves writing an MLIR dialect for OpenMP. Operations of the dialect will be a mix of fine and coarse-grained. e.g. Coarse: omp.parallel, omp.target, Fine: omp.flush. The operations in MLIR can have regions, hence there is no need for outlining
at the MLIR level. While the detailed design of the dialect is TBD, the next section provides links to walkthrough examples which provides a summary of the full flow as well as use of MLIR operations for OpenMP directives, and attributes for representing clauses
which are constant. The proposed plan involves a) lowering F18 AST with OpenMP directly to a mix of OpenMP and FIR dialects. b) converting this finally to a mix of OpenMP and LLVM dialects. This requires that OpenMP dialect can coexist and operate with other
dialects. The design is also intended to be modular so that other frontends (C/C++) can reuse the OpenMP dialect in the future. <br>
<br>
iv) Examples<br>
A few walkthroughs have been sent before [3]. These walkthroughs illustrate with an example, the flow for a few constructs (parallel, target, collapse, simd). For the parallel and target constructs will use the OpenMP IRBuilder for lowering to LLVM IR. MLIR
offers infrastructure to do loop transformations hence for the collapse clause the transformation is done inside the MLIR framework. While both LLVM and MLIR offers infrastructure for vectorisation the LLVM vectoriser is more mature and hence LLVM is preferred.
For details refer to the links [3].<br>
<br>
v) Progress<br>
i) OpenMP MLIR<br>
-> Sent a few walkthroughs which illustrate the flow from AST to LLVM IR. <br>
-> First patch [6] which registers the OpenMP dialect with MLIR has been submitted.<br>
-> Design of a minimal dialect with a single construct (barrier) is in progress.<br>
ii) OpenMP IRBuilder</span><span style="font-size:12.0pt;font-family:"Segoe UI";color:black"><br>
</span><span style="font-size:12.0pt;color:black">-> <a href="mailto:jdoerfert@anl.gov" target="_blank">@Doerfert, Johannes</a> has a series of patches [7] introducing preliminary support for the OpenMP IRBuilder which are either approved or under review. The initial
set adds support for the parallel and barrier construct.<br>
-> Others (Roger Ferrer, Kiran) have tried it for other constructs like taskwait and flush.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;font-family:"Segoe UI";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black">vi) Next Steps<br>
-> Send the design to the MLIR mailing list to get approval and to enable code review.<br>
-> Implement the accepted plan on a construct by construct basis starting with the parallel construct.</span><span style="font-size:12.0pt;font-family:"Segoe UI";color:#201F1E"><o:p></o:p></span></p>
</div>
</div>
<blockquote style="margin-left:30.0pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black">-> Represent construct in OpenMP MLIR</span><span style="font-size:12.0pt;font-family:"Segoe UI";color:#201F1E"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black">-> Refactor the code for the construct in OpenMP IRBuilder</span><span style="font-size:12.0pt;font-family:"Segoe UI";color:#201F1E"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black">-> Set up the translation library for OpenMP in MLIR to call the OpenMP IRBuilder</span><span style="font-size:12.0pt;font-family:"Segoe UI";color:#201F1E"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black">-> Set up the transformation from the frontend to OpenMP MLIR for this construct<br>
-> Upstream changes</span><span style="font-size:12.0pt;font-family:"Segoe UI";color:#201F1E"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<div>
<pre style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">1) <a href="https://drive.google.com/file/d/1vU6LsblsUYGA35B_3y9PmBvtKOTXj1Fu/view?usp=sharing" target="_blank">https://drive.google.com/file/d/1vU6LsblsUYGA35B_3y9PmBvtKOTXj1Fu/view?usp=sharing</a><o:p></o:p></span></pre>
<pre style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">2) </span><span style="color:#201F1E"><a href="http://lists.flang-compiler.org/pipermail/flang-dev_lists.flang-compiler.org/2019-August/000264.html" target="_blank"><span style="font-family:"Calibri",sans-serif;color:black">http://lists.flang-compiler.org/pipermail/flang-dev_lists.flang-compiler.org/2019-August/000264.html</span></a></span><span style="color:black"><o:p></o:p></span></pre>
<pre style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">3) Links for walkthroughs<o:p></o:p></span></pre>
<pre style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">parallel: </span><span style="color:black"><a href="http://lists.flang-compiler.org/pipermail/flang-dev_lists.flang-compiler.org/2019-August/000269.html" target="_blank"><span style="font-family:"Calibri",sans-serif">http://lists.flang-compiler.org/pipermail/flang-dev_lists.flang-compiler.org/2019-August/000269.html</span></a></span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></pre>
<pre style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">collapse: </span><span style="color:black"><a href="http://lists.flang-compiler.org/pipermail/flang-dev_lists.flang-compiler.org/2019-September/000273.html" target="_blank" title="http://lists.flang-compiler.org/pipermail/flang-dev_lists.flang-compiler.org/2019-September/000273.html"><span style="font-family:"Calibri",sans-serif">http://lists.flang-compiler.org/pipermail/flang-dev_lists.flang-compiler.org/2019-September/000273.html</span></a><o:p></o:p></span></pre>
<div>
<pre style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">collapse, Modification after feedback: </span><span style="color:black"><a href="http://lists.flang-compiler.org/pipermail/flang-dev_lists.flang-compiler.org/2019-September/000277.html" target="_blank" title="http://lists.flang-compiler.org/pipermail/flang-dev_lists.flang-compiler.org/2019-September/000277.html"><span style="font-family:"Calibri",sans-serif">http://lists.flang-compiler.org/pipermail/flang-dev_lists.flang-compiler.org/2019-September/000277.html</span></a><o:p></o:p></span></pre>
</div>
<div>
<pre style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">simd: </span><span style="color:black"><a href="http://lists.flang-compiler.org/pipermail/flang-dev_lists.flang-compiler.org/2019-September/000278.html" target="_blank" title="http://lists.flang-compiler.org/pipermail/flang-dev_lists.flang-compiler.org/2019-September/000278.html"><span style="font-family:"Calibri",sans-serif">http://lists.flang-compiler.org/pipermail/flang-dev_lists.flang-compiler.org/2019-September/000278.html</span></a><o:p></o:p></span></pre>
</div>
<pre style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">target: </span><span style="color:black"><a href="http://lists.flang-compiler.org/pipermail/flang-dev_lists.flang-compiler.org/2019-September/000285.html" target="_blank"><span style="font-family:"Calibri",sans-serif">http://lists.flang-compiler.org/pipermail/flang-dev_lists.flang-compiler.org/2019-September/000285.html</span></a><o:p></o:p></span></pre>
<pre style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">4) </span><span style="color:black"><a href="https://github.com/tensorflow/mlir" target="_blank"><span style="font-family:"Calibri",sans-serif">https://github.com/tensorflow/mlir</span></a><o:p></o:p></span></pre>
<pre style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">5) </span><span style="color:black"><a href="http://lists.flang-compiler.org/pipermail/flang-dev_lists.flang-compiler.org/2019-May/000197.html" target="_blank"><span style="font-family:"Calibri",sans-serif">http://lists.flang-compiler.org/pipermail/flang-dev_lists.flang-compiler.org/2019-May/000197.html</span></a><o:p></o:p></span></pre>
<pre style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">6) </span><span style="color:#201F1E"><a href="https://github.com/tensorflow/mlir/pull/244" target="_blank"><span style="font-family:"Calibri",sans-serif">https://github.com/tensorflow/mlir/pull/244</span></a><o:p></o:p></span></pre>
<pre style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">7) OpenMP IRBuilder (only a few included here)</span><span style="color:#201F1E"><o:p></o:p></span></pre>
<pre style="background:white"><span style="color:#201F1E"><a href="https://reviews.llvm.org/D69785" target="_blank"><span style="font-family:"Calibri",sans-serif">https://reviews.llvm.org/D69785</span></a><o:p></o:p></span></pre>
<pre style="background:white"><span style="color:#201F1E"><a href="https://reviews.llvm.org/D70290" target="_blank"><span style="font-family:"Calibri",sans-serif">https://reviews.llvm.org/D70290</span></a><o:p></o:p></span></pre>
<pre style="background:white"><span style="color:#201F1E"><a href="https://reviews.llvm.org/D69853" target="_blank"><span style="font-family:"Calibri",sans-serif">https://reviews.llvm.org/D69853</span></a><o:p></o:p></span></pre>
<pre style="background:white"><span style="color:#201F1E"><a href="https://reviews.llvm.org/D70290" target="_blank"><span style="font-family:"Calibri",sans-serif">https://reviews.llvm.org/D70290</span></a><o:p></o:p></span></pre>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;color:black">Thanks,</span><span style="font-size:12.0pt;font-family:"Segoe UI";color:#201F1E"><br>
</span><span style="font-size:12.0pt;color:black">Kiran</span><span style="font-size:12.0pt;font-family:"Segoe UI";color:#201F1E"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
</div>
<DIV>
<HR>
</DIV>
<DIV>This email message is for the sole use of the intended recipient(s) and may
contain confidential information. Any unauthorized review, use, disclosure
or distribution is prohibited. If you are not the intended recipient,
please contact the sender by reply email and destroy all copies of the original
message. </DIV>
<DIV>
<HR>
</DIV>
</body>
</html>