<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.pre
{mso-style-name:pre;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
p.msipheader251902e5, li.msipheader251902e5, div.msipheader251902e5
{mso-style-name:msipheader251902e5;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="msipheader251902e5" style="margin:0in"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#317100">[AMD Public Use]</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">When I talk about device arch, I mean the option passed by -mcpu. My understanding is that <sub> is intended for a variation of the <arch> in triple, which covers a set of CPU’s. I don’t think it is proper to treat each of -mcpu option
as a variation of the <arch>.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Sam<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Alexey Bataev <a.bataev@hotmail.com> <br>
<b>Sent:</b> Saturday, December 12, 2020 11:46 AM<br>
<b>To:</b> Liu, Yaxun (Sam) <Yaxun.Liu@amd.com><br>
<b>Cc:</b> cfe-dev (cfe-dev@lists.llvm.org) <cfe-dev@lists.llvm.org>; Lieberman, Ron <Ron.Lieberman@amd.com>; georgakoudis1@llnl.gov; jdoerfert@anl.gov; Artem Belevich <tra@google.com>; Searles, Mark <Mark.Searles@amd.com>; Chan, SiuChi <siuchi.chan@amd.com><br>
<b>Subject:</b> Re: [RFC] Unified offloading option for CUDA/HIP/OpenMP<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[CAUTION: External Email] <o:p></o:p></p>
<div>
<p class="MsoNormal">I still don’t understand why do we need kind but -offload-kind=kind looks much better to me.
<o:p></o:p></p>
<div>
<p class="MsoNormal">As to archs, I think the arch can be included in triple <span style="font-size:18.0pt;font-family:"Arial",sans-serif;color:#333333;background:white"> </span><span class="pre"><span style="font-family:"Courier New";color:#333333;background:#E2E2E2"><arch><sub>-<vendor>-<sys>-<abi></span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">You can specify sub target `amdgcngfx906-amd-amdhsa` but you need to add subarchs for triple class. How about this?<o:p></o:p></p>
</div>
<div id="AppleMailSignature">
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div id="AppleMailSignature">
<p class="MsoNormal"><a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fclang.llvm.org%2Fdocs%2FCrossCompilation.html&data=04%7C01%7CYaxun.Liu%40amd.com%7C31982767621948ab68d808d89ebd6daa%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637433884031446571%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=16lsQnCYmYjpxklqfcokC0UEnMVgdsV%2FYRO84F7mxWk%3D&reserved=0">https://clang.llvm.org/docs/CrossCompilation.html</a><o:p></o:p></p>
</div>
<div id="AppleMailSignature">
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Best regards, <o:p></o:p></p>
<div>
<p class="MsoNormal">Alexey Bataev<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt">12 дек. 2020 г., в 10:51, Liu, Yaxun (Sam) <<a href="mailto:Yaxun.Liu@amd.com">Yaxun.Liu@amd.com</a>> написал(а):<o:p></o:p></p>
</blockquote>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:5.0pt;margin-right:.5in;margin-bottom:12.0pt;margin-left:.5in">
[AMD Public Use]<br>
<br>
The offload kind is intended for situations where offload kind cannot be inferred from input file type, e.g. bundled LLVM bitcode or bundled assembly file. If it is redundant to specify it multiple times, we may consider a separate -offload-kind= option. I
understand OpenMP can use -fopenmp to indicate OpenMP offload kind, but HIP may need an -offload-kind=hip option since it currently does not have one.<br>
<br>
`-offload=target1,target2,... -offload-target1=“-march xxx -opt1” -offload-target2=“-march yyy -opt2 -opt3” does not well work for CUDA/HIP since CUDA/HIP can have multiple device archs for the same target.<br>
<br>
How about -offload=target1,arch1,opt1a,opt1b -offload=target2,arch2,opt2a,opt2b ?<br>
<br>
Thanks.<br>
<br>
Sam<br>
<br>
-----Original Message-----<br>
From: Alexey Bataev <<a href="mailto:a.bataev@hotmail.com">a.bataev@hotmail.com</a>>
<br>
Sent: Saturday, December 12, 2020 9:40 AM<br>
To: Liu, Yaxun (Sam) <<a href="mailto:Yaxun.Liu@amd.com">Yaxun.Liu@amd.com</a>><br>
Cc: cfe-dev (<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>) <<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>>; Lieberman, Ron <<a href="mailto:Ron.Lieberman@amd.com">Ron.Lieberman@amd.com</a>>;
<a href="mailto:georgakoudis1@llnl.gov">georgakoudis1@llnl.gov</a>; <a href="mailto:jdoerfert@anl.gov">
jdoerfert@anl.gov</a>; Artem Belevich <<a href="mailto:tra@google.com">tra@google.com</a>>; Searles, Mark <<a href="mailto:Mark.Searles@amd.com">Mark.Searles@amd.com</a>>; Chan, SiuChi <<a href="mailto:siuchi.chan@amd.com">siuchi.chan@amd.com</a>><br>
Subject: Re: [RFC] Unified offloading option for CUDA/HIP/OpenMP<br>
<br>
[CAUTION: External Email]<br>
<br>
Why do we need “kind” here? We know the kind already, if -fopenmp is used - kind is omp, if .hip is compiled - kind is hip.<br>
Adding arch also does not look good. We may need to pass some extra params. Better to have something like `-offload=target1,target2,... -offload-target1=“-march xxx -opt1” -offload-target2=“-march yyy -opt2 -opt3” ...`.<br>
<br>
Best regards,<br>
Alexey Bataev<br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">12 дек. 2020 г., в 09:11, Liu, Yaxun (Sam) <<a href="mailto:Yaxun.Liu@amd.com">Yaxun.Liu@amd.com</a>> написал(а):<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:5.0pt;margin-right:1.0in;margin-bottom:5.0pt;margin-left:1.0in">
[AMD Public Use]<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Currently CUDA/HIP and OpenMP has different offloading options, e.g.<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">clang++ -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx900 test.cpp<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">clang++ -offload-arch=gfx906 test.hip<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Our users request to have a concise way to specify offloading options for OpenMP. Ideally, one option to convey offloading kind, offloading triple, and offloading device arch.<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">On the other hand, there are some limitations of the current offloading option for CUDA/HIP:<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">1. It does not specify offloading kind whereas relies on file type to infer offloading kind. If input file is not CUDA/HIP source code (e.g. bundled LLVM bit code), there needs a way to specify offloading kind.<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">2. It does not specify offloading target triple whereas relies on device arch to infer target triple. As HIP is ported to different targets, there needs a way to specify offloading target triple.<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">In summary, a unified offloading option is preferred, which conveys offloading kind, offloading target triple and offloading device arch.<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">I would like to propose to either have a new option or extend the existing -offload-arch option for that, in the format kind-triple-arch, e.g.<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">-offload=omp-amd-gfx900<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">-offload=hip-amd-gfx906<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Whereas kind and triple can be abbreviations for conciseness, e.g. omp expands to openmp, amd expands to amdgcn-amd-amdhsa. Arch can be omitted, in which case clang will use the default arch for the triple.<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Your feedbacks are welcome.<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Thanks.<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Sam<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>