<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
span.hoenzb
{mso-style-name:hoenzb;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoPlainText">> 4. Store the device code in special sections of the host object file. This seems the most build-system friendly, although perhaps the most complicated to implementation on our end. Also, as has been pointed out, is also the technique
nvcc uses.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Making the device code stored in special (non-loadable) sections of the host object (Hal’s option 4) is what Intel’s compiler has implemented and would like to support for the following reasons:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">- We have single source files supporting offloading to devices. That should produce single fat objects that include the device objects as well.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">- Invocation of linker will result in a single dynamic library or executable with device binaries embedded.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">- It will make the use of device offload transparent to users, support separate compilations, and existing Makefiles.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">- It ensures the host and target object dependencies are easily maintained.
<o:p></o:p></p>
<p class="MsoPlainText"> Makefiles create object files and may move them during build process. These Makefiles will have to be changed to support separate device objects. Naming conventions could also be an issue for separate device objects.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">- Static libraries can be made up of fat objects as well. When the driver invokes the target linker it knows to look for device objects in the static libraries as well.<o:p></o:p></p>
<p class="MsoPlainText"> Static libraries provided is still only one library even with device code embedded.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">- With support for fat executables/dynamic libraries it should be fairly straight forward to make fat objects as well.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">- Customers we have worked with have provided the feedback to generate fat objects for ease of use.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">- This does leave an open issue of how to assembly files and intermediate IT files are handled and naming conventions for these.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Thanks,<o:p></o:p></p>
<p class="MsoPlainText">Knud<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Andrey Bokhanko [<a href="mailto:andreybokhanko@gmail.com">mailto:andreybokhanko@gmail.com</a>]
<br>
<b>Sent:</b> Monday, March 07, 2016 1:43 PM<br>
<b>To:</b> Kirkegaard, Knud J <<a href="mailto:knud.j.kirkegaard@intel.com">knud.j.kirkegaard@intel.com</a>><br>
<b>Subject:</b> Fwd: [cfe-dev] [RFC][OpenMP][CUDA] Unified Offloading Support in Clang Driver<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">---------- Forwarded message ----------<br>
From: <b>Hal Finkel via cfe-dev</b> <<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>><br>
Date: Mon, Mar 7, 2016 at 7:56 AM<br>
Subject: Re: [cfe-dev] [RFC][OpenMP][CUDA] Unified Offloading Support in Clang Driver<br>
To: Justin Lebar <<a href="mailto:jlebar@google.com">jlebar@google.com</a>><br>
Cc: Alexey Bataev <<a href="mailto:a.bataev@hotmail.com">a.bataev@hotmail.com</a>>, C Bergström via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>>, John McCall <<a href="mailto:rjmccall@gmail.com">rjmccall@gmail.com</a>>, Samuel
F Antao <<a href="mailto:sfantao@us.ibm.com">sfantao@us.ibm.com</a>><br>
<br>
<br>
----- Original Message -----<br>
> From: "Justin Lebar via cfe-dev" <<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>><br>
> To: "Samuel F Antao" <<a href="mailto:sfantao@us.ibm.com">sfantao@us.ibm.com</a>><br>
> Cc: "Alexey Bataev" <<a href="mailto:a.bataev@hotmail.com">a.bataev@hotmail.com</a>>, "C Bergström via cfe-dev" <<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>>, "John McCall"<br>
> <<a href="mailto:rjmccall@gmail.com">rjmccall@gmail.com</a>><br>
> Sent: Saturday, March 5, 2016 11:18:54 AM<br>
> Subject: Re: [cfe-dev] [RFC][OpenMP][CUDA] Unified Offloading Support in Clang Driver<br>
><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">> > Ok, you could link each component, but then you couldn't do<br>
> > anything, because the device side only works if you have that<br>
> > specific host code, allocating the data and invoking the kernel.<br>
><br>
> Sure, you'd have to do something after linking to group everything<br>
> together. Like, more reasonably, you could link together all the<br>
> device object files, and then link together the host object files<br>
> plus<br>
> the one device blob using a tool which understands this blob.<br>
><br>
> Or you could just pass all of the object files to a tool which<br>
> understands the difference between host and device object files and<br>
> will DTRT.<br>
><br>
> > B: nvcc -rdc=true a.o b.o -o a.out<br>
> > Wouldn't be desirable to have clang supporting case B as well?<br>
><br>
> Sure, yes. It's maybe worth elaborating on how we support case A<br>
> today. We compile the .cu file for device once for each device<br>
> architecture, generating N .s files and N corresponding .o files.<br>
> (The .s files are assembled by a black-box tool from nvidia.) We<br>
> then<br>
> feed both the .s and .o files to another tool from nvidia, which<br>
> makes<br>
> one "fat binary". We finally incorporate the fatbin into the host<br>
> object file while compiling.<br>
><br>
> Which sounds a lot like what I was telling you I didn't want to do, I<br>
> know. :) But the reason I think it's different is that there exists<br>
> a widely-adopted one-object-file format for cuda/nvptx. So if you do<br>
> the above in the right way, which we do, all of nvidia's binary tools<br>
> (objdump, etc) just work. Moreover, there are no real alternative<br>
> tools to break by this scheme -- the ISA is proprietary, and nobody<br>
> has bothered to write such a tool, to my knowledge. If they did, I<br>
> suspect they'd make it compatible with nvidia's (and thus our)<br>
> format.<br>
><br>
> Since we already have this format and it's well-supported by tools<br>
> etc, we'd probably want to support in clang unbundling the CUDA code<br>
> at linktime, just like nvcc.<br>
><br>
> Anyway, back to your question, where we're dealing with an ISA which<br>
> does not have a well-established bundling format. In this case, I<br>
> don't think it would be unreasonable to support<br>
><br>
> clang a-host.o a-device.o b-host.o b-device.o -o a.out<br>
><br>
> clang could presumably figure out the architecture of each file<br>
> either<br>
> from its name, from some sort of -x params, or by inspecting the file<br>
> -- all three would have good precedent.<br>
><br>
> The only issue is whether or not this should instead look like<br>
><br>
> clang a.tar b.tar -o a.out<br>
><br>
> The functionality is exactly the same.<br>
><br>
> If we use tar or invent a new format, we don't necessarily have to<br>
> change build systems. But we've either opened a new can of worms by<br>
> adding a rather more expressive than we want file format into clang<br>
> (tar is the obvious choice, but it's not a great fit; no random<br>
> access, no custom metadata, lots of edge cases to handle as errors,<br>
> etc), or we've made up a new file format with all the problems we've<br>
> discussed.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal">Many of the projects that will use this feature are very large, with highly non-trivial build systems. Requiring significant changes (beyond the normal changes to compiler paths and flags) in order to use OpenMP (including with accelerator
support) should be avoided where ever possible. This is much more important than ensuring tools compatibility with other compilers (although accomplishing both goals simultaneously seems even better still). Based on my reading of this thread, it seems like
we have several options (these come to mind):<br>
<br>
1. Use multiple object files. Many build-system changes can be avoided by using some scheme for guessing the name of the device object files from that of the host object file. This often won't work, however, because many build systems copy object files around,
add them to static archives, etc. and the device object files would be missed in this operations.<br>
<br>
2. Use some kind of bundling format. tar, zip, ar, etc. seem like workable options. Any user who runs 'file' on them will easily guess how to extract the data. objdump, etc., however, won't know how to handle these directly (which can also have build-system
implications, although more rare than for (1)).<br>
<br>
3. Treat the input/output object file name as a directory, and store in that directory the host and device object files. This might be effectively transparent, but also suffers from potential build-system problems (rm -f won't work, for example).<br>
<br>
4. Store the device code in special sections of the host object file. This seems the most build-system friendly, although perhaps the most complicated to implementation on our end. Also, as has been pointed out, is also the technique nvcc uses.<br>
<br>
All things considered, I think that I'd prefer (4). If we're picking an option to minimize build-system changes, which I fully support, picking the option with the smallest chance of incompatibilities seems optimal. There is also other (prior) art here, and
we should find out how GCC is handling this in GCC 6 for OpenACC and/or OpenMP 4 (<a href="https://gcc.gnu.org/wiki/OpenACC" target="_blank">https://gcc.gnu.org/wiki/OpenACC</a>). Also, we can check on PGI and/or Pathscale (for OpenACC, OpenHMPP, etc.), in
addition to any relevant details of what nvcc does here.<br>
<br>
Thanks again,<br>
Hal<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>