<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Courier;
panose-1:2 7 4 9 2 2 5 2 4 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="RU" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hi Doru,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">> By "offload link step" do you mean the linking phase that happens on the device toolchain? I assume so.</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">> Will the device link step just work with "-L. -labc" even if libabc.a contains fat objects?</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">> From the above I understand that what you want is to created your very own version of an NVLINK-like tool which uses the clang-offload-bundler format instead of
the CUDA fatbin format.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">No, I do not want to change linking for offloading devices, these steps will be done by device toolchains on device objects
exactly the same way as it is done today. I suggest to change the initial step of what I called “offload linking” (i.e. linking when offload is enabled) – add partial linking step which is performed by the host toolchain linker on fat objects and libraries,
followed by the unbundling of the partial linking result which is done by the existing clang-offload-bundler tool. By “offload linking” I just meant the full set of actions which are needed to link the final binary; it includes link steps for all offloading
devices as well as for the host.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">This diagram shows the action graph for the following command<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:Courier">clang++ -fopenmp -fopenmp-targets=dev1,dev2 test.cpp -L. -labc -o test<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:Courier"> [host compilation] -> h.o host.o --------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:Courier"> / \ / \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:Courier">test.cpp – [dev1 compilation] -> d1.o – [bundle] -> test.o -> [ld –r test.o –L. -labc] -> [unbundle] – [dev1 link] -> d1.out – [linker script] - [host link] -> test<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:Courier"> \ / \ /<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:Courier"> [dev2 compilation] -> d2.o [dev2 link] -> d2.out<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:Courier"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Here
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">[bundle] is [clang-offload-bundler --bundle]<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">[unbundle] is [clang-offload-bundler --unbundle]<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Bundle operation produces fat object (it is an object that can be passed to the host toolchain). Partial linking (ld -r)
is done on fat objects and libraries by the linker from the host toolchain. Unbundle operation extracts host and device parts from the fat object, it will create one host object and one or more objects for each offloading target. Device linking is done by
the device toolchain linkers on the device objects extracted from the partially linked object by the unbundle operation.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">BTW, I omitted dependencies between host compilation and device compilations to simply the diagram (in reality host bit
code is passed to the device compilations as input. These details are not significant to the topic which we are discussing.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I hope this diagram explains my suggestion. Do you see any potential problems which can be caused by this change?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Serguei<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><a name="_____replyseparator"></a><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Gheorghe-Teod Bercea [mailto:Gheorghe-Teod.Bercea@ibm.com]
<br>
<b>Sent:</b> Friday, August 17, 2018 3:26 PM<br>
<b>To:</b> Dmitriev, Serguei N <serguei.n.dmitriev@intel.com><br>
<b>Cc:</b> 'cfe-dev@lists.llvm.org' <cfe-dev@lists.llvm.org>; Jonas Hahnfeld <hahnjo@hahnjo.de><br>
<b>Subject:</b> RE: [cfe-dev] [OpenMP] offload support for static libraries<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">Hi Serguei,</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">Thanks a lot for the explanation on the partial linking step.</span><span lang="EN-US"><br>
<br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">> "<span style="color:#004080">the intent is to change offload link step to always operate with fat objects and libs</span>"</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">By "offload link step" do you mean the linking phase that happens on the device toolchain? I assume so.</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">Will the device link step just work with "-L. -labc" even if libabc.a contains fat objects?</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">From the above I understand that what you want is to created your very own version of an NVLINK-like tool which uses the clang-offload-bundler format instead of the CUDA fatbin
format. This to me means that the "unbundling" step could be included in the device linker step after the resolution of the "-L. -labc" options. The Clang Driver should only invoke the unbundler in the device link step (or have the device linker invoke it
internally whatever works in your case). There is never a need to invoke it for the host part. The Clang Driver will be calling the "bundler" to put together the host and device objects.</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">If you do all this then, at least for your host and device toolchains, you don't need to call "ld -r" at all. I talk below about where to use "ld -r" though.</span><span lang="EN-US"><br>
<br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">> "<span style="color:#004080">Clang-offload-bundler would need to extract each device object individually while doing unbundling operation on the partially linked object.</span>"</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">I think that as soon as you start unbundling you need to do a full linking step of all device parts to avoid any mess. This supports the idea that you should do the unbundling
inside the device linking step where you know exactly which parts you need to link in and where to find them. You then output a single device-only object which you can then bundle with the host object file.</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">I think this will be equivalent to doing the following:</span><span lang="EN-US"><br>
<br>
</span><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">A. When creating object files "clang++ -fopenmp -fopenmp-targets=intel-device-triple test.cpp -c -o test.o" I need to call the clang-offload-bundler</span></b><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:Courier">DEVICE TC: ---[device compilation]---> dev-test.o
</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:Courier"> \-------[clang-offload-bundler --bundle]---> test.o</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:Courier"> HOST TC: ---[host compilation]---> host-test.o /</span><span lang="EN-US"><br>
<br>
<br>
</span><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">B. when doing: "clang++ -fopenmp -fopenmp-targets=intel-device-triple test.cpp -L. -labc -o test"</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">there
will be no explicit clang-offload-bundler invocations by the clang driver. The device toolchain will look like this:</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:Courier">DEVICE TC: ---[device compilation]---> dev-test.o ---[MyDeviceLinker -L. -labc]--> device --> ....</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">The important part is that the device link stage will be able to seamlessly link the static lib, which would involve the following steps:</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">1. find the library using the device linker's capabilities;</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">2. unpack the library into object files;</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">3. unbundle each object file to obtain the device object;</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">4. create a static library with the device objects;</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">5. link dev-test.o against the device static library you just created.</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">[I have implemented steps 2->4 before in the clang-ykt compiler so it's definitely feasible once you know the path to your static library, use the linker to resolve that for you]</span><span lang="EN-US"><br>
<br>
<br>
</span><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">C. when only linking is needed: "clang++ -fopenmp -fopenmp-targets=intel-device-triple test1.o test2.o -L. -labc -o test"</span></b><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">Exact same happens as above only you need to also unbundle the individual objects and then invoke your usual linker. The clang-offload-bundler will be called by the device link
step to "unbundle" the objects on the device.</span><span lang="EN-US"><br>
<br>
<br>
<br>
<br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">NVLINK cannot be made to work with the clang-offload-bundler in the way that your device linker can so there will always have to be something special being done for NVPTX targets
no matter what solution you choose.</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">HOWEVER, there is an upside!!</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">I suspect that the fat object produced by the clang-offload-bundler and the object produced by the OpenMP NVPTX toolchain (with my patch applied) can be successfully partially
linked together by "ld -r".</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">Assuming we have something like this:</span><span lang="EN-US"><br>
<br>
</span><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">clang++ -fopenmp -fopenmp-targets=intel-device-triple,nvptx64-nvidia-cuda test.cpp -c -o test.o</span></b><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">then we can do this:</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif"> </span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:Courier">NVPTX TC: ---[nvptx compilation]---------------------> nvptx-test.o ------------------------\
</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:Courier">INTEL TC: ---[intel compilation]---> intel-test.o \</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:Courier"> \ ----[clang-offload-bundler]----> tmp.o ---[ld -r]---> test.o</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:Courier">HOST TC: ---[host compilation]---> host-test.o /</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">Essentially we first resolve the clang-offload-bundler bundling to obtain tmp.o for intel and other toolchains. Then we can partially link tmp.o against the nvptx-test.o object
file.</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">Sanity checks:</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">The nvptx-test.o file will be treated by "ld -r" just like any other host file that doesn't contain a "bundled" device section (it was not bundled with the clang-offload-bundler
but with the CUDA specific format). So if partially linking together a host only file and a bundled file works then this should work too.</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">test.o can be processed by the NVPTX toolchain? Yes, NVLINK will know where the device side of the object is, no unbundling required to get to the device part.</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">test.o can be processed by the INTEL toolchain? Yes, the object will be passed to the Device Linker where it will be unbundled and device side extracted.</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">static linking works for both toolchains? Yes, previous patch enables it for NVPTX, this patch would enable it for INTEL.</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">static linking works for both toolchains when used together? Yes, because the CUDA specific fatbinary format will not be obstructed by the clang-offload-bundling format.</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">Thanks,</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">--Doru</span><span lang="EN-US"><br>
<br>
<br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
</span><span lang="EN-US"><br>
<br>
<br>
<br>
</span><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">From: </span><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">"Dmitriev, Serguei N" <</span><a href="mailto:serguei.n.dmitriev@intel.com"><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">serguei.n.dmitriev@intel.com</span></a><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">></span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">To: </span><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">Gheorghe-Teod Bercea <</span><a href="mailto:Gheorghe-Teod.Bercea@ibm.com"><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">Gheorghe-Teod.Bercea@ibm.com</span></a><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">></span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Cc: </span><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">"'cfe-dev@lists.llvm.org'" <</span><a href="mailto:cfe-dev@lists.llvm.org"><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">cfe-dev@lists.llvm.org</span></a><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">>,
Jonas Hahnfeld <</span><a href="mailto:hahnjo@hahnjo.de"><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">hahnjo@hahnjo.de</span></a><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">></span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Date: </span><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">08/16/2018 07:26 PM</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Subject: </span><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">RE: [cfe-dev] [OpenMP] offload support for static libraries</span><span lang="EN-US"><o:p></o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" noshade="" style="color:#A0A0A0" align="center">
</div>
<p class="MsoNormal"><span lang="EN-US"><br>
<br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#004080">Hi Doru,</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#004080">Thank you for the detailed description of your changes.</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#004080"> </span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">> Regarding your proposal, from your slides I understand that you perform a partial linking step as the first action for all object files and/or static libraries given as input.
So this clang invocation:<br>
> clang++ -L. -labc test.cpp -o test<br>
> would result in the same compilation steps as the current Clang version performs because the initial stage of partial linking would have no work to do (since there are no object files present to be partially linked).</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#004080">No, the intent is to change offload link step to always operate with fat objects and libs, so the compilation part of the action graph for that command should produce
temporary fat object which is then passed to the partial linking. I have not described that in slides, but I agree that it is probably not obvious and should have been mentioned.
</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#004080"> </span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">> Another question I have is regarding the "ld -r" box in your slides.<br>
> How does ld -r work with "bundled" objects? Your diagram seems to imply that ld -r does the concatenation of all device images out of the box. Is this accurate?</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#004080">Actually the technique that is currently used by the clang-offload-bundler tool for creating bundled (or fat) objects is very similar to what you are going to do
for bundling NVPTX and host objects. The difference is in the initial “wrapper” that is created for the device code – you are using C++ structure, while clang-offload-bundler uses LLVM bitcode file. The bundler tool creates a temporary LLVM IR which contains
a global initialized array holding the device object, and this array is allocated in a section with predefined name (the name includes offload target triple). This “wrapper” bitcode file is then compiled for the host and then partially linked against the host
object.</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#004080"> </span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#004080">So technically fat/bundled object is just a host ELF object which has one or more additional ELF section containing device object as data (one extra section per
each offloading target). Linker concatenates sections with the same name while linking multiple objects files, so the result of partial linking will have the same named ELF sections holding device code concatenated from all input (fat) objects.</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#004080"> </span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#004080">Clang-offload-bundler would need to extract each device object individually while doing unbundling operation on the partially linked object. A possible way to enable
this would be creating one more section holding the device object size in addition to existing section with device object in fat/bundled object. That would allow clang bundler to get sizes of device objects that were concatenated by partial linking.</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#004080"> </span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#004080">Thanks,</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#004080">Serguei</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#004080"> </span><span lang="EN-US"><br>
</span><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">Gheorghe-Teod Bercea [</span><a href="mailto:Gheorghe-Teod.Bercea@ibm.com"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">mailto:Gheorghe-Teod.Bercea@ibm.com</span></a><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">]
<b><br>
Sent:</b> Thursday, August 16, 2018 6:15 AM<b><br>
To:</b> Dmitriev, Serguei N <</span><a href="mailto:serguei.n.dmitriev@intel.com"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">serguei.n.dmitriev@intel.com</span></a><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">><b><br>
Cc:</b> 'cfe-dev@lists.llvm.org' <</span><a href="mailto:cfe-dev@lists.llvm.org"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">cfe-dev@lists.llvm.org</span></a><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">>;
Jonas Hahnfeld <</span><a href="mailto:hahnjo@hahnjo.de"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">hahnjo@hahnjo.de</span></a><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">><b><br>
Subject:</b> RE: [cfe-dev] [OpenMP] offload support for static libraries</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">Hi Serguei,</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
Thanks a lot for the proposal.</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
My proposal reworks a little bit the way the OpenMP-NVPTX toolchain creates device object files: the device specific part of the object is "wrapped" in an NVLINK-friendly C++ structure that is then compiled for the host. The result is a host object file with
a device part which NVLINK can detect (D.o). The D.o object file is then partially linked against the host object file H.o and thus we obtain HD.o. This is required because compilation is required to produce a single output object file (when doing "-c -o"
for example). HD.o can now be passed to NVLINK directly or put in a static library and then passed to NVLINK. Either way, NVLINK will be able to detect the device part (due to the special wrapping that we did previously) without the need to "unbundle" the
object file (prior to passing it to NVLINK).</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
The reason why the clang-offload-bundler is not involved in this is because we are using the standard object format for the object file that the OpenMP-NVPTX toolchain outputs so there's no need for a custom format in this case. The partial linking step is
required to put together the host and device object files and to ensure that only one object file is produced even if we actually invoked two toolchains (one for host and one for the device).</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
Regarding your proposal, from your slides I understand that you perform a partial linking step as the first action for all object files and/or static libraries given as input. So this clang invocation:<br>
clang++ -L. -labc test.cpp -o test<br>
would result in the same compilation steps as the current Clang version performs because the initial stage of partial linking would have no work to do (since there are no object files present to be partially linked).</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
Another question I have is regarding the "ld -r" box in your slides.<br>
How does ld -r work with "bundled" objects? Your diagram seems to imply that ld -r does the concatenation of all device images out of the box. Is this accurate?</span><span lang="EN-US"><br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
Thanks a lot,</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
--Doru</span><span lang="EN-US"><br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
</span><span lang="EN-US"><br>
<br>
<br>
<br>
</span><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F"><br>
From: </span><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">"Dmitriev, Serguei N" <</span><a href="mailto:serguei.n.dmitriev@intel.com"><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">serguei.n.dmitriev@intel.com</span></a><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">><span style="color:#5F5F5F"><br>
To: </span>Jonas Hahnfeld <</span><a href="mailto:hahnjo@hahnjo.de"><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">hahnjo@hahnjo.de</span></a><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">><span style="color:#5F5F5F"><br>
Cc: </span>"'cfe-dev@lists.llvm.org'" <</span><a href="mailto:cfe-dev@lists.llvm.org"><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">cfe-dev@lists.llvm.org</span></a><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">>,
Doru Bercea <</span><a href="mailto:gheorghe-teod.bercea@ibm.com"><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">gheorghe-teod.bercea@ibm.com</span></a><span lang="EN-US" style="font-size:7.5pt;font-family:"Arial",sans-serif">><span style="color:#5F5F5F"><br>
Date: </span>08/15/2018 04:27 PM<span style="color:#5F5F5F"><br>
Subject: </span>RE: [cfe-dev] [OpenMP] offload support for static libraries</span><span lang="EN-US"><o:p></o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" noshade="" style="color:#A0A0A0" align="center">
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><br>
<br>
<br>
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New""><br>
Hi Jonas,<br>
<br>
I guess this patch implements the proposal which Doru presented on the "OpenMP / HPC in Clang / LLVM Multi-company" meeting. As I remember he suggested to eliminate use of clang-offload-bundler tool when offload target is NTVPTX by replacing bundling operation
with partial linking of host and device objects, and then relying of the NVPTX linker to perform the unbundling operation at link phase. Based on Doru's explanations NVPTX linker "knows" how to extract device parts from such objects, so the explicit unbundling
operation in not required. Doru, please correct me if my understanding is not fully accurate. Doru's proposal definitely achieves the same goal for NVPTX offloading target (i.e. enables offload in static libraries), but it is NVPTX specific and cannot be extended
to other offloading targets (at least that is how it looked like when Doru described it).<br>
<br>
I propose slightly different solution which I think should work for any generic OpenMP offload target (it was also discussed on the OpenMP multi-company meeting). In general case we have to use clang-offload-bundler because we cannot assume that device object(s)
can be bundled with the host object by performing partial linking of host and device objects. So bundling and unbundling operation will still be done by the clang-offload-bundler tool. The main part of my suggestion is adding partial linking of fat objects
(created by offload bundler tool) and static libraries (which are composed of fat objects) and only after that do the unbundling operation on the partially linked object (followed by the appropriate link actions for all offloading devices and then for the
host). This would guarantee that device parts of fat objects from static libraries will participate in the device link actions, and thus would enable offloading for static libraries.<br>
<br>
Thanks,<br>
Serguei<br>
<br>
-----Original Message-----<br>
From: Jonas Hahnfeld [</span><a href="mailto:hahnjo@hahnjo.de"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">mailto:hahnjo@hahnjo.de</span></a><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">]
<br>
Sent: Tuesday, August 14, 2018 1:52 PM<br>
To: Dmitriev, Serguei N <</span><a href="mailto:serguei.n.dmitriev@intel.com"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">serguei.n.dmitriev@intel.com</span></a><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">><br>
Cc: 'cfe-dev@lists.llvm.org' <</span><a href="mailto:cfe-dev@lists.llvm.org"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">cfe-dev@lists.llvm.org</span></a><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">>; Doru
Bercea <</span><a href="mailto:gheorghe-teod.bercea@ibm.com"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">gheorghe-teod.bercea@ibm.com</span></a><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">><br>
Subject: Re: [cfe-dev] [OpenMP] offload support for static libraries<br>
<br>
This proposal has already been proposed for NVPTX in </span><a href="https://reviews.llvm.org/D47394"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">https://reviews.llvm.org/D47394</span></a><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">,
adding Doru.<br>
<br>
Cheers,<br>
Jonas<br>
<br>
On 2018-08-14 18:43, Dmitriev, Serguei N via cfe-dev wrote:<br>
> PROBLEM OVERVIEW<br>
> <br>
> OpenMP offload functionality is currently not supported in static <br>
> libraries. Because of that an attempt to use offloading in static <br>
> libraries ends up with a fallback execution of target regions on the <br>
> host. This limitation clearly has significant impact on OpenMP offload <br>
> usability.<br>
> <br>
> An output object file that is created by the compiler for offload <br>
> compilation is a fat object. Such object files besides the code for <br>
> the host architecture also contains code for the offloading targets <br>
> which is stored as data in ELF sections with predefined names. Thus, a <br>
> static library that is created from object files produced by offload <br>
> compilation would be an archive of fat objects.<br>
> <br>
> Clang driver currently never passes fat objects directly to any <br>
> toolchain. Instead it performs an unbundling operation for each fat <br>
> object which extract host and device parts from the object. These <br>
> parts are then independently processed by the corresponding target <br>
> toolchains. However, current implementation does not assume that <br>
> static archives may also be composed from fat objects. No unbundling <br>
> is done for static archives (they are passed to linker as is) and thus <br>
> device parts of objects from such archives get ignored.<br>
> <br>
> SUGGESTED SOLUTION<br>
> <br>
> It seems feasible to resolve this problem by changing the offload link <br>
> process - adding an extra step to the link flow which will do a <br>
> partial linking (ld -r) of fat objects and static libraries as shown <br>
> on this diagram<br>
> <br>
> [Fat objects] \ / [Target1 link] \<br>
> <br>
> [Partial linking] - [Unbundling] - [TargetN link] - <br>
> [Host link]<br>
> <br>
> [Static libs] / \--- Host part --/<br>
> <br>
> (You can also look at the .pdf file on this link <br>
> </span><a href="https://drive.google.com/file/d/1ZTNoB-Ghin1BTaiZ312FMSRS6rISDtlr/view"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">https://drive.google.com/file/d/1ZTNoB-Ghin1BTaiZ312FMSRS6rISDtlr/view</span></a><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New""><br>
> ?usp=sharing [1] for illustrations for the suggested change)<br>
> <br>
> Linker will pull in all necessary dependencies from static libraries <br>
> while performing partial linking, so the result of partial linking <br>
> would be a fat object with concatenated device parts from input fat <br>
> objects and required dependencies from static libraries. These <br>
> concatenated device objects will be stored in the corresponding ELF <br>
> sections of the partially linked object.<br>
> <br>
> Unbundling operation on the partially linked object will create one or <br>
> more device objects for each offloading target, and these objects will <br>
> be linked by corresponding target toolchains the same way as it is <br>
> done now. Offload bundler tool would require enhancements to support <br>
> unbundling of multiple concatenated device objects for each offloading <br>
> target.<br>
> <br>
> Host link action can be changed to use host part of the partially <br>
> linked object while linking the final image.<br>
> <br>
> Do you see any potential problems in the proposed change?<br>
> <br>
> Links:<br>
> ------<br>
> [1]<br>
> </span><a href="https://drive.google.com/file/d/1ZTNoB-Ghin1BTaiZ312FMSRS6rISDtlr/view"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">https://drive.google.com/file/d/1ZTNoB-Ghin1BTaiZ312FMSRS6rISDtlr/view</span></a><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New""><br>
> ?usp=sharing _______________________________________________<br>
> cfe-dev mailing list<br>
> </span><a href="mailto:cfe-dev@lists.llvm.org"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">cfe-dev@lists.llvm.org</span></a><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New""><br>
> </span><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</span></a><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New""><br>
</span><span lang="EN-US"><br>
<br>
<br>
<br>
<o:p></o:p></span></p>
</div>
</body>
</html>