<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi Serguei,</p>
<p>I don't see a lot of problems with this example. As I can see
there is only one problem: clang-offload-bundler generates an
empty object file that cannot be recognized by the linker. If we
teach clang-offload-bundler to generate correct empty object files
(what we need to do anyway, because currently it may produce wrong
utout), your example will work without any changes. <br>
</p>
<pre class="moz-signature" cols="72">-------------
Best regards,
Alexey Bataev</pre>
<div class="moz-cite-prefix">31.07.2018 16:50, Dmitriev, Serguei N
пишет:<br>
</div>
<blockquote type="cite"
cite="mid:1112AE43C04F2E428633A4D42126DB32ACB88A02@ORSMSX111.amr.corp.intel.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle21
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"
lang="EN-US">Hi Alexey,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"
lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"
lang="EN-US">Such change would fix the link issue, but I
believe it would be a short term solution that will still be
revised in future.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"
lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"
lang="EN-US">Let me explain my concerns. Current
implementation has one more limitation which I assume would
also be addressed in future – target binaries are expected
to have entries for all OpenMP target regions in the
program, though it seems to be too restrictive. I assume
there would be use cases when you would want to tune target
regions in your program for particular targets and
offloading to other targets would not make much sense for
those regions (or probably won’t even be possible due to
limitations of a particular target). It seems reasonable to
compile those region only for the targets they were tuned
for, thus I assume compiler will support the following usage
model in future<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"
lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"
lang="EN-US">clang -fopenmp
-fopenmp-targets=nvptx64-nvidia-cuda -c a.c<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"
lang="EN-US">clang -fopenmp
-fopenmp-targets=x86_64-pc-linux-gnu -c b.c<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"
lang="EN-US">clang -fopenmp
-fopenmp-targets=nvptx64-nvidia-cuda,x86_64-pc-linux-gnu a.o
b.o<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"
lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"
lang="EN-US">And such usage model would anyway require
redesigning the way how offload initialization code is
generated. It has to be delayed till link time because the
final set of offload targets that need to be registered in
runtime would be known only at link step and thus compiler
cannot create correct target binary descriptor object (which
is a part of the offload initialization code) at compile
time as it is done now.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"
lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"
lang="EN-US">Does that sound reasonable?<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"
lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"
lang="EN-US">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"
lang="EN-US">Serguei<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"
lang="EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><a name="_____replyseparator"
moz-do-not-send="true"></a><b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext"
lang="EN-US">From:</span></b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext"
lang="EN-US"> Alexey Bataev
[<a class="moz-txt-link-freetext" href="mailto:a.bataev@outlook.com">mailto:a.bataev@outlook.com</a>] <br>
<b>Sent:</b> Tuesday, July 31, 2018 9:55 AM<br>
<b>To:</b> Dmitriev, Serguei N
<a class="moz-txt-link-rfc2396E" href="mailto:serguei.n.dmitriev@intel.com"><serguei.n.dmitriev@intel.com></a>;
<a class="moz-txt-link-abbreviated" href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a><br>
<b>Subject:</b> Re: [cfe-dev] [RFC][OpenMP] Usability
improvement, allow dropping offload targets<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p><span lang="EN-US">Hi Serguei,<o:p></o:p></span></p>
<p><span lang="EN-US">Actually your problem can be fixed easily
with a simple patch that changes the linkage of the
<o:p></o:p></span></p>
<pre><span lang="EN-US">`.omp_offloading.img_[start|end]` symbols from external to external weak. After this change your example compiles and works perfectly without any additional changes. I'm going to commit this patch in few minutes.<o:p></o:p></span></pre>
<pre><span lang="EN-US">-------------<o:p></o:p></span></pre>
<pre><span lang="EN-US">Best regards,<o:p></o:p></span></pre>
<pre><span lang="EN-US">Alexey Bataev<o:p></o:p></span></pre>
<div>
<p class="MsoNormal"><span lang="EN-US">30.07.2018 19:50,
Dmitriev, Serguei N via cfe-dev
</span>пишет<span lang="EN-US">:<o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre><span lang="EN-US">Motivation<o:p></o:p></span></pre>
<pre><span lang="EN-US"><o:p> </o:p></span></pre>
<pre><span lang="EN-US">The existing OpenMP offloading implementation in clang does not allow dropping<o:p></o:p></span></pre>
<pre><span lang="EN-US">offload targets at link time. That is, if an object file is created with one set<o:p></o:p></span></pre>
<pre><span lang="EN-US">of offload targets you must use exactly the same set of offload targets at the<o:p></o:p></span></pre>
<pre><span lang="EN-US">link stage. Otherwise, linking will fail<o:p></o:p></span></pre>
<pre><span lang="EN-US"><o:p> </o:p></span></pre>
<pre><span lang="EN-US">$ clang -fopenmp -fopenmp-targets=x86_64-pc-linux-gnu,nvptx64-nvidia-cuda foo.c -c<o:p></o:p></span></pre>
<pre><span lang="EN-US">$ clang -fopenmp -fopenmp-targets=x86_64-pc-linux-gnu foo.o<o:p></o:p></span></pre>
<pre><span lang="EN-US">/tmp/foo-dd79f7.o:(.rodata..omp_offloading.device_images[.omp_offloading.descriptor_reg]+0x20): undefined reference to `.omp_offloading.img_start.nvptx64-nvidia-cuda'<o:p></o:p></span></pre>
<pre><span lang="EN-US">/tmp/foo-dd79f7.o:(.rodata..omp_offloading.device_images[.omp_offloading.descriptor_reg]+0x28): undefined reference to `.omp_offloading.img_end.nvptx64-nvidia-cuda'<o:p></o:p></span></pre>
<pre><span lang="EN-US">clang-7: error: linker command failed with exit code 1 (use -v to see invocation)<o:p></o:p></span></pre>
<pre><span lang="EN-US">$ <o:p></o:p></span></pre>
<pre><span lang="EN-US"><o:p> </o:p></span></pre>
<pre><span lang="EN-US">This limits OpenMP offload usability. So far, this has not been a high priority<o:p></o:p></span></pre>
<pre><span lang="EN-US">issue but the importance of this problem will grow once clang offload starts<o:p></o:p></span></pre>
<pre><span lang="EN-US">supporting static libraries with offload functionality. For instance, this<o:p></o:p></span></pre>
<pre><span lang="EN-US">limitation won't allow creating general purpose static libraries targeting<o:p></o:p></span></pre>
<pre><span lang="EN-US">multiple types of offload devices and later linking them into a program that<o:p></o:p></span></pre>
<pre><span lang="EN-US">uses only one offload target.<o:p></o:p></span></pre>
<pre><span lang="EN-US"><o:p> </o:p></span></pre>
<pre><span lang="EN-US">Problem description<o:p></o:p></span></pre>
<pre><span lang="EN-US"><o:p> </o:p></span></pre>
<pre><span lang="EN-US">Offload targets cannot be dropped at the link phase because object files<o:p></o:p></span></pre>
<pre><span lang="EN-US">produced by the compiler for the host have dependencies on the offload targets<o:p></o:p></span></pre>
<pre><span lang="EN-US">specified during compilation. These dependencies arise from the offload<o:p></o:p></span></pre>
<pre><span lang="EN-US">initialization code.<o:p></o:p></span></pre>
<pre><span lang="EN-US"><o:p> </o:p></span></pre>
<pre><span lang="EN-US">The clang front-end adds offload initialization code to each host object in<o:p></o:p></span></pre>
<pre><span lang="EN-US">addition to all necessary processing of OpenMP constructs. This initialization<o:p></o:p></span></pre>
<pre><span lang="EN-US">code is intended to register target binaries for all offload targets in the<o:p></o:p></span></pre>
<pre><span lang="EN-US">runtime library at program startup. This code consists of two compiler-generated<o:p></o:p></span></pre>
<pre><span lang="EN-US">routines. One of these routines is added to the list of global constructors and<o:p></o:p></span></pre>
<pre><span lang="EN-US">the other to the global destructors. The constructor routine calls a<o:p></o:p></span></pre>
<pre><span lang="EN-US">libomptarget API which registers the target binaries and the destructor<o:p></o:p></span></pre>
<pre><span lang="EN-US">correspondingly calls a similar API for unregistering target binaries.<o:p></o:p></span></pre>
<pre><span lang="EN-US"><o:p> </o:p></span></pre>
<pre><span lang="EN-US">Both these APIs accept a pointer to the target binary descriptor object which<o:p></o:p></span></pre>
<pre><span lang="EN-US">specifies the number of offload target binaries to register and the start/end<o:p></o:p></span></pre>
<pre><span lang="EN-US">addresses of target binary images. Since the start/end addresses of target<o:p></o:p></span></pre>
<pre><span lang="EN-US">binaries are not available at compile time, the target binary descriptors are<o:p></o:p></span></pre>
<pre><span lang="EN-US">initialized using link-time constants which reference (undefined) symbols<o:p></o:p></span></pre>
<pre><span lang="EN-US">containing the start/end addresses of all target images. These symbols are<o:p></o:p></span></pre>
<pre><span lang="EN-US">created by the dynamically-generated linker script which the clang driver<o:p></o:p></span></pre>
<pre><span lang="EN-US">creates for the host link action.<o:p></o:p></span></pre>
<pre><span lang="EN-US"><o:p> </o:p></span></pre>
<pre><span lang="EN-US">References to the target specific symbols from host objects make them dependent<o:p></o:p></span></pre>
<pre><span lang="EN-US">on particular offload targets and prevents dropping offload targets at the link<o:p></o:p></span></pre>
<pre><span lang="EN-US">step. Therefore, the OpenMP offload initialization needs to be redesigned to<o:p></o:p></span></pre>
<pre><span lang="EN-US">make offload targets discardable.<o:p></o:p></span></pre>
<pre><span lang="EN-US"><o:p> </o:p></span></pre>
<pre><span lang="EN-US">Proposed change<o:p></o:p></span></pre>
<pre><span lang="EN-US"><o:p> </o:p></span></pre>
<pre><span lang="EN-US">Host objects should be independent of offload targets in order to allow dropping<o:p></o:p></span></pre>
<pre><span lang="EN-US">code for offload targets. That can be achieved by removing offload<o:p></o:p></span></pre>
<pre><span lang="EN-US">initialization code from host objects. The compiler should not inject this code<o:p></o:p></span></pre>
<pre><span lang="EN-US">into host objects.<o:p></o:p></span></pre>
<pre><span lang="EN-US"><o:p> </o:p></span></pre>
<pre><span lang="EN-US">However, offload initialization should still be done, so it is proposed to move<o:p></o:p></span></pre>
<pre><span lang="EN-US">the initialization code into a special dynamically generated object file<o:p></o:p></span></pre>
<pre><span lang="EN-US">(referred to as 'wrapper object' here onwards), which, besides the<o:p></o:p></span></pre>
<pre><span lang="EN-US">initialization code, will also contain embedded images for offload targets.<o:p></o:p></span></pre>
<pre><span lang="EN-US"><o:p> </o:p></span></pre>
<pre><span lang="EN-US">The wrapper object file will be generated by the clang driver with the help of<o:p></o:p></span></pre>
<pre><span lang="EN-US">a new tool: clang-offload-wrapper. This tool will take offload target binaries<o:p></o:p></span></pre>
<pre><span lang="EN-US">as input and produce bitcode files containing offload initialization code and<o:p></o:p></span></pre>
<pre><span lang="EN-US">embedded target images. The output bitcode is then passed to the backend and<o:p></o:p></span></pre>
<pre><span lang="EN-US">assembler tools from the host toolchain to produce the wrapper object which is<o:p></o:p></span></pre>
<pre><span lang="EN-US">then added as an input to the linker for host linking.<o:p></o:p></span></pre>
<pre><span lang="EN-US"><o:p> </o:p></span></pre>
<pre><span lang="EN-US">The offload action builder in the clang driver needs to be changed to use this<o:p></o:p></span></pre>
<pre><span lang="EN-US">tool while building the actions graph for OpenMP offload compilations.<o:p></o:p></span></pre>
<pre><span lang="EN-US"><o:p> </o:p></span></pre>
<pre><span lang="EN-US">Current status<o:p></o:p></span></pre>
<pre><span lang="EN-US"><o:p> </o:p></span></pre>
<pre><span lang="EN-US">A patch with initial implementation of the proposed changes has been uploaded to<o:p></o:p></span></pre>
<pre><span lang="EN-US">phabricator for review - </span><a href="https://reviews.llvm.org/D49510" moz-do-not-send="true"><span lang="EN-US">https://reviews.llvm.org/D49510</span></a><span lang="EN-US">.<o:p></o:p></span></pre>
<pre><span lang="EN-US"><o:p> </o:p></span></pre>
<pre><span lang="EN-US">Looking for a feedback for this proposal.<o:p></o:p></span></pre>
<pre><span lang="EN-US"><o:p> </o:p></span></pre>
<pre>Thanks,<o:p></o:p></pre>
<pre>Sergey<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
</blockquote>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</blockquote>
<br>
</body>
</html>