<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
p.msipheader251902e5, li.msipheader251902e5, div.msipheader251902e5
{mso-style-name:msipheader251902e5;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="msipheader251902e5" style="margin:0in"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#317100">[AMD Public Use]</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The amdgpu xnack and sramecc need to be part of GPU arch name the same way as for --offload-arch, e.g.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">--offload=amdgcn-gfx906:xnack+,amdgcn-gfx906:xnack-<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">They behave like GPU arch.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Sam <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Artem Belevich <tra@google.com> <br>
<b>Sent:</b> Monday, March 8, 2021 2:01 PM<br>
<b>To:</b> Liu, Yaxun (Sam) <Yaxun.Liu@amd.com><br>
<b>Cc:</b> Doerfert, Johannes <jdoerfert@anl.gov>; Ben Boeckel <ben.boeckel@kitware.com>; Lieberman, Ron <Ron.Lieberman@amd.com>; a.bataev@hotmail.com; Chan, SiuChi <siuchi.chan@amd.com>; Searles, Mark <Mark.Searles@amd.com>; cfe-dev (cfe-dev@lists.llvm.org)
<cfe-dev@lists.llvm.org>; jeffrey.sandoval@hpe.com; Jon Chesterfield <jonathanchesterfield@gmail.com>; Rodgers, Gregory <Gregory.Rodgers@amd.com><br>
<b>Subject:</b> Re: [cfe-dev] [RFC] Unified offloading option for CUDA/HIP/OpenMP<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[CAUTION: External Email] <o:p></o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Verdana",sans-serif"><o:p> </o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Sat, Mar 6, 2021 at 7:13 AM Liu, Yaxun (Sam) <<a href="mailto:Yaxun.Liu@amd.com">Yaxun.Liu@amd.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal">[AMD Public Use]<br>
<br>
We need to different target triples since it may not always be possible to infer target triple by cpu name. So I guess it would be like:<br>
<br>
"--offload=amdgcn-gfx906,amdgcn-gfx1010"<br>
"--Xoffload=amdgcn-gfx* options common to all AMD GPUs"<br>
"--Xoffload=amdgcn-gfx906 -mcpu=gfx906 --fsomething-specific-to-gfx906"<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Verdana",sans-serif">SGTM.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Verdana",sans-serif">Do you expect the AMDGPU's features (+xnack, -ecc, etc) to be part of the offload target ? Or would they be specified via -Xoffload arguments?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"> </span><span style="font-family:"Verdana",sans-serif"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">--Artem</span><span style="font-family:"Verdana",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Verdana",sans-serif"><o:p> </o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal"><br>
Sam<br>
<br>
-----Original Message-----<br>
From: Doerfert, Johannes <<a href="mailto:jdoerfert@anl.gov" target="_blank">jdoerfert@anl.gov</a>>
<br>
Sent: Friday, March 5, 2021 1:25 PM<br>
To: Artem Belevich <<a href="mailto:tra@google.com" target="_blank">tra@google.com</a>>; Liu, Yaxun (Sam) <<a href="mailto:Yaxun.Liu@amd.com" target="_blank">Yaxun.Liu@amd.com</a>><br>
Cc: Ben Boeckel <<a href="mailto:ben.boeckel@kitware.com" target="_blank">ben.boeckel@kitware.com</a>>; Lieberman, Ron <<a href="mailto:Ron.Lieberman@amd.com" target="_blank">Ron.Lieberman@amd.com</a>>;
<a href="mailto:a.bataev@hotmail.com" target="_blank">a.bataev@hotmail.com</a>; Chan, SiuChi <<a href="mailto:siuchi.chan@amd.com" target="_blank">siuchi.chan@amd.com</a>>; Searles, Mark <<a href="mailto:Mark.Searles@amd.com" target="_blank">Mark.Searles@amd.com</a>>;
cfe-dev (<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>) <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>>;
<a href="mailto:jeffrey.sandoval@hpe.com" target="_blank">jeffrey.sandoval@hpe.com</a>; Jon Chesterfield <<a href="mailto:jonathanchesterfield@gmail.com" target="_blank">jonathanchesterfield@gmail.com</a>>; Rodgers, Gregory <<a href="mailto:Gregory.Rodgers@amd.com" target="_blank">Gregory.Rodgers@amd.com</a>><br>
Subject: Re: [cfe-dev] [RFC] Unified offloading option for CUDA/HIP/OpenMP<br>
<br>
[CAUTION: External Email]<br>
<br>
On 3/4/21 3:05 PM, Artem Belevich wrote:<br>
> On Thu, Mar 4, 2021 at 10:34 AM Liu, Yaxun (Sam) <<a href="mailto:Yaxun.Liu@amd.com" target="_blank">Yaxun.Liu@amd.com</a>> wrote:<br>
><br>
>> [AMD Public Use]<br>
>><br>
>> There is another aspect we need to consider: how to modify the <br>
>> -target option by additional options?<br>
>><br>
>> For the existing --offload-arch option, we could use -Xarch_ to add <br>
>> specific options for it.<br>
>><br>
> `-Xarch_xxx` as implemented right now is a rather limiter hack. IIRC <br>
> it only accepts options w/o arguments which limits its usability.<br>
><br>
><br>
>> Assuming we have an -offload="amdgcn -mcpu=gfx906" option, then we <br>
>> want to add some options specific to it by an additional option, what <br>
>> should we do?<br>
>><br>
> I think we've been conflating telling the driver what to compile for <br>
> and customizing individual sub-compilations.<br>
><br>
> We could explicitly separate the two tasks. E.g.:<br>
> `--[no-]offload=target1,target2,target3...`<br>
> `--Xoffload=target_pattern target_options...`<br>
><br>
> This way your example would be handled with:<br>
> "--offload=gfx906,gfx1010"<br>
> "--Xoffload=gfx* options common to all AMD GPUs"<br>
> "--Xoffload=gfx906 -mcpu=gfx906 --fsomething-specific-to-gfx906"<br>
><br>
> In the end `-Xarch_xxx` would become an alias for '-Xoffload=xxx'.<br>
<br>
+1<br>
<br>
<br>
> --Artem<br>
><br>
><br>
><br>
><br>
>> Thanks.<br>
>><br>
>> Sam<br>
>><br>
>> -----Original Message-----<br>
>> From: Doerfert, Johannes <<a href="mailto:jdoerfert@anl.gov" target="_blank">jdoerfert@anl.gov</a>><br>
>> Sent: Thursday, February 11, 2021 12:59 PM<br>
>> To: Artem Belevich <<a href="mailto:tra@google.com" target="_blank">tra@google.com</a>>; Liu, Yaxun (Sam)
<br>
>> <<a href="mailto:Yaxun.Liu@amd.com" target="_blank">Yaxun.Liu@amd.com</a>><br>
>> Cc: Ben Boeckel <<a href="mailto:ben.boeckel@kitware.com" target="_blank">ben.boeckel@kitware.com</a>>; Lieberman, Ron <
<br>
>> <a href="mailto:Ron.Lieberman@amd.com" target="_blank">Ron.Lieberman@amd.com</a>>;
<a href="mailto:a.bataev@hotmail.com" target="_blank">a.bataev@hotmail.com</a>; Chan, SiuChi <
<br>
>> <a href="mailto:siuchi.chan@amd.com" target="_blank">siuchi.chan@amd.com</a>>; Searles, Mark <<a href="mailto:Mark.Searles@amd.com" target="_blank">Mark.Searles@amd.com</a>>; cfe-dev (<br>
>> <a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>) <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>>;
<br>
>> <a href="mailto:jeffrey.sandoval@hpe.com" target="_blank">jeffrey.sandoval@hpe.com</a>; Jon Chesterfield
<br>
>> <<a href="mailto:jonathanchesterfield@gmail.com" target="_blank">jonathanchesterfield@gmail.com</a>><br>
>> Subject: Re: [cfe-dev] [RFC] Unified offloading option for <br>
>> CUDA/HIP/OpenMP<br>
>><br>
>> [CAUTION: External Email]<br>
>><br>
>> I'm OK with either.<br>
>><br>
>> On 2/11/21 11:42 AM, Artem Belevich wrote:<br>
>>> On Thu, Feb 11, 2021 at 8:30 AM Liu, Yaxun (Sam) <<a href="mailto:Yaxun.Liu@amd.com" target="_blank">Yaxun.Liu@amd.com</a>><br>
>> wrote:<br>
>>>> [AMD Public Use]<br>
>>>><br>
>>>><br>
>>>><br>
>>>> Sorry for the delay.<br>
>>>><br>
>>>><br>
>>>><br>
>>>> Both Johannes’ and Artem’s proposals should satisfy the needs of users:<br>
>>>><br>
>>>><br>
>>>><br>
>>>> Option 1:<br>
>>>><br>
>>>><br>
>>>><br>
>>>> `-offload=<offload-pattern> optA optB optC`.<br>
>>>><br>
>>>><br>
>>>><br>
>>>> Option 2:<br>
>>>><br>
>>>><br>
>>>><br>
>>>> `-offload=<offload-pattern>,optA,optB,optC`.<br>
>>>><br>
>>> I'm fine with #2. We're using something similar with our build tools <br>
>>> and it works reasonably well.<br>
>>> However, it does have one annoying corner case. There's no easy way <br>
>>> to pass an option which has a comma in it. E.g. if I want to pass <br>
>>> `-Wl,something,something`. Perhaps we could use sed-like approach <br>
>>> and allow changing the separator. E.g. `s/a/b/` == `s@a@b@`.<br>
>>><br>
>>> --Artem<br>
>>><br>
>>><br>
>>><br>
>>>> Compared to the old options, they are more concise and more readable.<br>
>>>><br>
>>>><br>
>>>><br>
>>>> The main difference is the delimiter. To me option 2 is more <br>
>>>> attractive since it does not need quotations for most cases.<br>
>>>><br>
>>>><br>
>>>><br>
>>>> Can we reach an agreement on option 2?<br>
>>>><br>
>>>><br>
>>>><br>
>>>> Thanks.<br>
>>>><br>
>>>><br>
>>>><br>
>>>> Sam<br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>> *From:* Artem Belevich <<a href="mailto:tra@google.com" target="_blank">tra@google.com</a>><br>
>>>> *Sent:* Tuesday, December 15, 2020 2:13 PM<br>
>>>> *To:* Ben Boeckel <<a href="mailto:ben.boeckel@kitware.com" target="_blank">ben.boeckel@kitware.com</a>><br>
>>>> *Cc:* Doerfert, Johannes <<a href="mailto:jdoerfert@anl.gov" target="_blank">jdoerfert@anl.gov</a>>; Liu, Yaxun (Sam) <
<br>
>>>> <a href="mailto:Yaxun.Liu@amd.com" target="_blank">Yaxun.Liu@amd.com</a>>; Lieberman, Ron <<a href="mailto:Ron.Lieberman@amd.com" target="_blank">Ron.Lieberman@amd.com</a>>;
<br>
>>>> <a href="mailto:a.bataev@hotmail.com" target="_blank">a.bataev@hotmail.com</a>; Chan, SiuChi <<a href="mailto:siuchi.chan@amd.com" target="_blank">siuchi.chan@amd.com</a>>; Searles,
<br>
>>>> Mark < <a href="mailto:Mark.Searles@amd.com" target="_blank">Mark.Searles@amd.com</a>>; cfe-dev (<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>) <
<br>
>>>> <a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>><br>
>>>> *Subject:* Re: [cfe-dev] [RFC] Unified offloading option for <br>
>>>> CUDA/HIP/OpenMP<br>
>>>><br>
>>>><br>
>>>><br>
>>>> [CAUTION: External Email]<br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>> On Tue, Dec 15, 2020 at 10:23 AM Ben Boeckel <br>
>>>> <<a href="mailto:ben.boeckel@kitware.com" target="_blank">ben.boeckel@kitware.com</a>><br>
>>>> wrote:<br>
>>>><br>
>>>> On Mon, Dec 14, 2020 at 14:04:43 -0800, Artem Belevich via cfe-dev<br>
>> wrote:<br>
>>>>> It all may be an utter overkill, too. WDYT?<br>
>>>> Note that tools such as ccache and sccache generally need to be <br>
>>>> able to understand what's going on (I believe distcc and other <br>
>>>> distributed compilation tools also generally need to know too), so <br>
>>>> making it sensible enough for interpretation based on just the <br>
>>>> flags to be possible should be considered.<br>
>>>><br>
>>>><br>
>>>><br>
>>>> I think this is somewhat orthogonal to how we specify per-target<br>
>> options.<br>
>>>> Such a tool almost never knows about all possible compiler options <br>
>>>> and has to pass through the unknown options as-is. However, any <br>
>>>> form<br>
>> of 'nested'<br>
>>>> options specified on the command line will have a chance to confuse <br>
>>>> such tool. E.g. if I want to pass '-E' to some sub-tool for a <br>
>>>> particular offload-target, ccache, not being aware that it's not a <br>
>>>> top-level compilation option, may interpret it as an attempt to<br>
>> preprocess the TU.<br>
>>>><br>
>>>><br>
>>>> I wonder if it would make sense to just move all this per-target <br>
>>>> option complexity into an external response file. As far as <br>
>>>> existing tools are concerned, it would look like <br>
>>>> `--offload-options=target-opts.file` without affecting tool's <br>
>>>> general idea what this compilation is about to do, and the external <br>
>>>> file would allow us to be as flexible as we need to be to specify <br>
>>>> per-target<br>
>> options. It could be just a flat list of pairs `-Xarch_...<br>
>>>> optA`. Or we could use YAML.<br>
>>>><br>
>>>><br>
>>>><br>
>>>> That approach, however, has its own issues and would still need to <br>
>>>> be optional. If it's the only way to specify offload options, that <br>
>>>> will complicate other use cases as now they would have to deal with <br>
>>>> temporary files.<br>
>>>><br>
>>>><br>
>>>><br>
>>>> Maybe a slightly modified variant of jdoefert@'s idea would work<br>
>> better:<br>
>>>>>>> -offload="amd -march=gfx906 -fno-vectorize" -fopenmp<br>
>>>><br>
>>>> Implement it in a way similar to -Wl,optA,optB,optC and extend it <br>
>>>> to match an offload scope glob/regex.<br>
>>>><br>
>>>> E.g. `-offload=<offload-pattern>,optA,optB,optC`.<br>
>>>><br>
>>>> As far as the external tools are concerned, it's just one option to <br>
>>>> pass though. At the same time it should be flexible enough to apply <br>
>>>> the options to subset of offload targets in a human-manageable way.<br>
>>>><br>
>>>><br>
>>>><br>
>>>> --<br>
>>>><br>
>>>> --Artem Belevich<br>
>>>><br>
><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">--Artem Belevich<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>