<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi Jonas,</p>
<p>thanks for answer, the command indeed works, I was just missing
the vendor-part of the triple "-unknown" in some places. <br>
</p>
<p>The clang-offload-bundler and the clang linking step seem to be
very sensitive to vendor-part of the target triple. <br>
</p>
<p>The following series of commands produces a working executable:</p>
<p>
<blockquote type="cite">clang -fopenmp
-fopenmp-targets=x86_64-unknown-linux-gnu -S -emit-llvm -O3 -o
bundled.ll code.c<br>
clang-offload-bundler --inputs=bundled.ll
--outputs=host.in.ll,device.in.ll
--targets=host-x86_64-unknown-linux-gnu,openmp-x86_64-unknown-linux-gnu
--type=ll --unbundle<br>
opt -o host.opt.bc -O3 host.in.ll<br>
llc -o host.o -filetype=obj host.opt.bc<br>
opt -o device.opt.bc -O3 --relocation-model=pic device.in.ll<br>
llc -o device.o -filetype=obj --relocation-model=pic
device.opt.bc<br>
clang-offload-bundler --inputs=host.o,device.o --outputs=app.o
--targets=host-x86_64-unknown-linux-gnu,openmp-x86_64-unknown-linux-gnu
--type=o<br>
clang -fopenmp -fopenmp-targets=x86_64-unknown-linux-gnu -L
../../llvm-bin/lib app.o</blockquote>
<br>
</p>
<p>The following series (watch the missing "-unknown") produces a
linker error, probably because the device-part of the app.o bundle
is empty:</p>
<p>
<blockquote type="cite">clang -fopenmp
-fopenmp-targets=x86_64-linux-gnu -S -emit-llvm -O3 -o
bundled.ll code.c<br>
clang-offload-bundler --inputs=bundled.ll
--outputs=host.in.ll,device.in.ll
--targets=host-x86_64-unknown-linux-gnu,openmp-x86_64-linux-gnu
--type=ll --unbundle<br>
opt -o host.opt.ll -O3 host.in.ll<br>
llc -o host.o -filetype=obj host.opt.ll<br>
opt -o device.opt.ll -O3 --relocation-model=pic device.in.ll<br>
llc -o device.o -filetype=obj --relocation-model=pic
device.opt.ll<br>
clang-offload-bundler --inputs=host.o,device.o --outputs=app.o
--targets=host-x86_64-unknown-linux-gnu,openmp-x86_64-linux-gnu
--type=o<br>
clang -L ../../llvm-bin/lib -fopenmp
-fopenmp-targets=x86_64-linux-gnu -o app.exe app.o</blockquote>
<br>
</p>
<p>The error is:</p>
<p>
<blockquote type="cite">/usr/bin/ld: /tmp/app-ba8c99.o: file not
recognized: file truncated<br>
collect2: error: ld returned 1 exit status<br>
/usr/bin/ld: cannot find /tmp/app-d4cba4.out inside /<br>
/usr/bin/ld: cannot find /tmp/app-d4cba4.out inside /<br>
clang-9: error: linker (via gcc) command failed with exit code 1
(use -v to see invocation)<br>
clang-9: error: linker command failed with exit code 1 (use -v
to see invocation)<br>
</blockquote>
<br>
I tested two other command sequences (with more "-unknown" in
different places), both run without error, but the resulting
executable is not working (same error as yesterday in the
libomptarget-rtl):</p>
<p>
<blockquote type="cite">clang -fopenmp
-fopenmp-targets=x86_64-linux-gnu -S -emit-llvm -O3 -o
bundled.ll code.c<br>
clang-offload-bundler --inputs=bundled.ll
--outputs=host.in.ll,device.in.ll
--targets=host-x86_64-unknown-linux-gnu,openmp-x86_64-linux-gnu
--type=ll --unbundle<br>
opt -o host.opt.ll -O3 host.in.ll<br>
llc -o host.o -filetype=obj host.opt.ll<br>
opt -o device.opt.ll -O3 --relocation-model=pic device.in.ll<br>
llc -o device.o -filetype=obj --relocation-model=pic
device.opt.ll<br>
clang-offload-bundler --inputs=host.o,device.o --outputs=app.o
--targets=host-x86_64-unknown-linux-gnu,openmp-x86_64-unknown-linux-gnu
--type=o<br>
clang -L ../../llvm-bin/lib -fopenmp
-fopenmp-targets=x86_64-unknown-linux-gnu -o app.exe app.o</blockquote>
</p>
<p><br>
<blockquote type="cite">clang -fopenmp
-fopenmp-targets=x86_64-linux-gnu -S -emit-llvm -O3 -o
bundled.ll code.c<br>
clang-offload-bundler --inputs=bundled.ll
--outputs=host.in.ll,device.in.ll
--targets=host-x86_64-unknown-linux-gnu,openmp-x86_64-unknown-linux-gnu
--type=ll --unbundle<br>
opt -o host.opt.ll -O3 host.in.ll<br>
llc -o host.o -filetype=obj host.opt.ll<br>
opt -o device.opt.ll -O3 --relocation-model=pic
-mtriple=x86_64-unknown-linux-gnu device.in.ll<br>
llc -o device.o -mtriple=x86_64-unknown-linux-gnu -filetype=obj
--relocation-model=pic device.opt.ll<br>
clang-offload-bundler --inputs=host.o,device.o --outputs=app.o
--targets=host-x86_64-unknown-linux-gnu,openmp-x86_64-unknown-linux-gnu
--type=o<br>
clang -L ../../llvm-bin/lib -fopenmp
-fopenmp-targets=x86_64-unknown-linux-gnu -o app.exe app.o</blockquote>
</p>
<p><br>
</p>
<p>I'm wondering if clang-offload-bundler should be so sensitive to
the vendor-string in the target triple. In particular, if you run
clang with "-fopenmp-targets=x86_64-linux-gnu", you still have to
run clang-offload-bundler with the "unknown" vendor-string,
otherwise the device-part (device.in.ll) will be empty. <br>
</p>
<p>Is this the intended behavior?</p>
<p>Thanks,</p>
<p>Lukas<br>
</p>
<pre class="moz-signature" cols="72">Lukas Sommer, M.Sc.
TU Darmstadt
Embedded Systems and Applications Group (ESA)
Hochschulstr. 10, 64289 Darmstadt, Germany
Phone: +49 6151 1622429
<a class="moz-txt-link-abbreviated" href="http://www.esa.informatik.tu-darmstadt.de">www.esa.informatik.tu-darmstadt.de</a></pre>
<div class="moz-cite-prefix">On 11/5/19 10:46 PM, Jonas Hahnfeld
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:918382016c12a777b223e440a45ab4677367875f.camel@hahnjo.de">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div>So it seems to work for me:</div>
<pre>$ LIBOMPTARGET_DEBUG=1 LD_LIBRARY_PATH=libomptarget/libomptarget-debug/:$LD_LIBRARY_PATH ./a.out</pre>
<pre>Libomptarget --> Loading RTLs...</pre>
<pre>[...]</pre>
<pre>Libomptarget --> Loading library 'libomptarget.rtl.x86_64.so'...</pre>
<pre>Libomptarget --> Successfully loaded library 'libomptarget.rtl.x86_64.so'!</pre>
<pre>Libomptarget --> Registering RTL libomptarget.rtl.x86_64.so supporting 4 devices!</pre>
<pre>[...]</pre>
<pre>Libomptarget --> RTLs loaded!</pre>
<pre>Libomptarget --> Image 0x0000000000602050 is compatible with RTL libomptarget.rtl.x86_64.so!</pre>
<pre>Libomptarget --> RTL 0x000000000061b7c0 has index 0!</pre>
<pre>Libomptarget --> Registering image 0x0000000000602050 with RTL libomptarget.rtl.x86_64.so!</pre>
<pre>Libomptarget --> Done registering entries!</pre>
<pre>Libomptarget --> Call to omp_get_num_devices returning 4</pre>
<pre>Libomptarget --> Default TARGET OFFLOAD policy is now mandatory (devices were found)</pre>
<pre>Libomptarget --> Entering target region with entry point 0x00000000004008c0 and device Id -1</pre>
<pre>Libomptarget --> Checking whether device 0 is ready.</pre>
<pre>Libomptarget --> Is the device 0 (local ID 0) initialized? 0</pre>
<pre>Libomptarget --> Device 0 is ready to use.</pre>
<pre>Target x86_64 RTL --> Dev 0: load binary from 0x0000000000602050 image</pre>
<pre>Target x86_64 RTL --> Expecting to have 1 entries defined.</pre>
<pre>Target x86_64 RTL --> Offset of entries section is (0x0000000000201020).</pre>
<pre>Target x86_64 RTL --> Pointer to first entry to be loaded is (0x00002ad4f4632020).</pre>
<pre>Target x86_64 RTL --> Entries table range is (0x00002ad4f4632020)->(0x00002ad4f4632040)</pre>
<pre>Libomptarget --> Launching target execution __omp_offloading_48_54503d96_main_l8 with pointer 0x00002ad4f44316b0 (index=0).</pre>
<pre>Target x86_64 RTL --> Running entry point at 0x00002ad4f44316b0...</pre>
<pre>omp_is_initial_device: 0</pre>
<pre>Libomptarget --> Unloading target library!</pre>
<pre>Libomptarget --> Image 0x0000000000602050 is compatible with RTL 0x000000000061b7c0!</pre>
<pre>Libomptarget --> Unregistered image 0x0000000000602050 from RTL 0x000000000061b7c0!</pre>
<pre>Libomptarget --> Done unregistering images!</pre>
<pre>Libomptarget --> Removing translation table for descriptor 0x00000000006040e0</pre>
<pre>Libomptarget --> Done unregistering library!</pre>
<div><br>
</div>
<div>I mostly used the steps from your initial post, but passed
--relocation-model=pic for the device code (both opt and llc)
and used clang -fopenmp
-fopenmp-targets=x86_64-unknown-linux-gnu app.o for the final
linking.</div>
<div><br>
</div>
<div>It's possible that my code is just too simple (target region
with a printf of omp_is_initial_device) to trigger the problem
you're seeing. Could you share your exact code?</div>
<div>For the error message "invalid operand", I think libomptarget
uses libelf incorrectly. Maybe you can try replacing
"elf_errmsg(-1)" by "elf_errmsg(elf_errno())"? At least that's
how I think it should be according to the the man page.</div>
<div><br>
</div>
<div>Regards,</div>
<div>Jonas</div>
<div><br>
</div>
<div>Am Dienstag, den 05.11.2019, 11:25 +0100 schrieb Lukas
Sommer:</div>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px
#729fcf solid;padding-left:1ex">
<p>Hi Jonas,</p>
<p>thanks for the hint, with "--relocation-model=pic" added to
opt and llc, I can run the bundled object file through clang
without any errors.</p>
<p>However, the linking in clang does not seem to work right, at
runtime the libomptarget-plugin (rtl.cpp) throws an error
because it is unable to handle the ELF:</p>
<p> </p>
<blockquote type="cite" style="margin:0 0 0 .8ex;
border-left:2px #729fcf solid;padding-left:1ex">Libomptarget
--> Loading RTLs...<br>
Libomptarget --> Loading library
'libomptarget.rtl.ppc64.so'...<br>
Libomptarget --> Unable to load library
'libomptarget.rtl.ppc64.so': libomptarget.rtl.ppc64.so: cannot
open shared object file: No such file or directory!<br>
Libomptarget --> Loading library
'libomptarget.rtl.x86_64.so'...<br>
Libomptarget --> Successfully loaded library
'libomptarget.rtl.x86_64.so'!<br>
Libomptarget --> Registering RTL libomptarget.rtl.x86_64.so
supporting 4 devices!<br>
Libomptarget --> Loading library
'libomptarget.rtl.cuda.so'...<br>
Libomptarget --> Unable to load library
'libomptarget.rtl.cuda.so': libomptarget.rtl.cuda.so: cannot
open shared object file: No such file or directory!<br>
Libomptarget --> Loading library
'libomptarget.rtl.aarch64.so'...<br>
Libomptarget --> Unable to load library
'libomptarget.rtl.aarch64.so': libomptarget.rtl.aarch64.so:
cannot open shared object file: No such file or directory!<br>
Libomptarget --> RTLs loaded!<br>
Target x86_64 RTL --> Unable to get ELF handle: invalid
operand!<br>
Libomptarget --> Image 0x0000000000000000 is NOT compatible
with RTL libomptarget.rtl.x86_64.so!<br>
Libomptarget --> No RTL found for image 0x0000000000000000!<br>
Libomptarget --> Done registering entries!<br>
Libomptarget --> Call to omp_get_num_devices returning 0<br>
Libomptarget --> Default TARGET OFFLOAD policy is now
disabled (no devices were found)<br>
</blockquote>
Do I need to specify additional flags to clang?
<p>Thanks,</p>
<p>Lukas<br>
</p>
<pre>Lukas Sommer, M.Sc.</pre>
<pre>TU Darmstadt</pre>
<pre>Embedded Systems and Applications Group (ESA)</pre>
<pre>Hochschulstr. 10, 64289 Darmstadt, Germany</pre>
<pre>Phone: +49 6151 1622429</pre>
<a class="moz-txt-link-abbreviated"
href="http://www.esa.informatik.tu-darmstadt.de"
moz-do-not-send="true">
<pre>www.esa.informatik.tu-darmstadt.de</pre>
</a>
<div class="moz-cite-prefix">On 11/4/19 5:43 PM, Jonas Hahnfeld
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:f5cefdcc925868b5eac76f8d641bd1ecfa572de2.camel@hahnjo.de"
style="margin:0 0 0 .8ex; border-left:2px #729fcf
solid;padding-left:1ex">
<div>Am Montag, den 04.11.2019, 17:28 +0100 schrieb Lukas
Sommer:</div>
<blockquote type="cite" style="margin:0 0 0 .8ex;
border-left:2px #729fcf solid;padding-left:1ex">
<p>Hi Jonas,</p>
<p>thanks for your help!</p>
<p>I tried the suggested command, unfortunately it still
fails:</p>
<p> </p>
<blockquote type="cite" style="margin:0 0 0 .8ex;
border-left:2px #729fcf solid;padding-left:1ex">clang
-fopenmp -fopenmp-targets=x86_64-unknown-linux-gnu app.o</blockquote>
yields:
<p> </p>
<blockquote type="cite" style="margin:0 0 0 .8ex;
border-left:2px #729fcf solid;padding-left:1ex">/usr/bin/ld:
/tmp/app-c48732.o: relocation R_X86_64_32 against `.data'
can not be used when making a shared object; recompile
with -fPIC<br>
/usr/bin/ld: final link failed: nonrepresentable section
on output<br>
/usr/bin/ld: cannot find /tmp/app-23d932.out inside /<br>
/usr/bin/ld: cannot find /tmp/app-23d932.out inside /<br>
clang-9: error: linker command failed with exit code 1
(use -v to see invocation)<br>
clang-9: error: linker command failed with exit code 1
(use -v to see invocation)</blockquote>
I also tried to generate and bundle assembly (.asm) and run
that through clang (with openmp-flags as above), but that
gave the same error.<br>
</blockquote>
<div><br>
</div>
<div>Hmm, the target image is built as a shared library. To
make this work, Clang adds -shared -fPIC in
Generic_GCC::TranslateArgs.</div>
<div>You're replacing this step with a manual invocation of
`opt', so we need to pass some extra flags. I'm currently
looking into what's needed, opt --relocation-model=pic might
be a good start but I'm not sure that's all.</div>
<div><br>
</div>
<div>Jonas</div>
<div><br>
</div>
<blockquote type="cite" style="margin:0 0 0 .8ex;
border-left:2px #729fcf solid;padding-left:1ex">
<p>Lukas<br>
</p>
<pre>Lukas Sommer, M.Sc.</pre>
<pre>TU Darmstadt</pre>
<pre>Embedded Systems and Applications Group (ESA)</pre>
<pre>Hochschulstr. 10, 64289 Darmstadt, Germany</pre>
<pre>Phone: +49 6151 1622429</pre>
<a class="moz-txt-link-abbreviated"
href="http://www.esa.informatik.tu-darmstadt.de"
moz-do-not-send="true">
<pre>www.esa.informatik.tu-darmstadt.de</pre>
</a>
<div class="moz-cite-prefix">On 11/4/19 5:01 PM, Jonas
Hahnfeld wrote:<br>
</div>
<blockquote type="cite"
cite="mid:79be0a771d7cf25e2a3444808c5ebcef59856ded.camel@hahnjo.de"
style="margin:0 0 0 .8ex; border-left:2px #729fcf
solid;padding-left:1ex">
<div>Hi Lukas,</div>
<div><br>
</div>
<div>Am Montag, den 28.10.2019, 13:42 +0100 schrieb Lukas
Sommer via Openmp-dev:</div>
<blockquote type="cite" style="margin:0 0 0 .8ex;
border-left:2px #729fcf solid;padding-left:1ex">
<pre>Hello,</pre>
<pre>I'm trying to use the clang-offload-bundler to merge the results of the</pre>
<pre>compilation for device and host for an application using OpenMP target</pre>
<pre>pragmas.</pre>
<pre>My goal is to run a custom pass on the device code and then bundle an</pre>
<pre>application from the modified device code and the host code. For testing</pre>
<pre>purposes, I use a very simple vector addition with target offloading</pre>
<pre>pragmas and the "offloading" to x86_64 (LLVM/Clang release 9.0).</pre>
<pre>I use clang to generate LLVM IR from the application code. Then I use</pre>
<pre>the clang-offload-bundler to unbundle the LLVM IR for device and host. I</pre>
<pre>run opt and llc separately on the LLVM IR for host and device.</pre>
<pre>To this end, on my application code (code.c), I run the following steps:</pre>
<blockquote type="cite" style="margin:0 0 0 .8ex;
border-left:2px #729fcf solid;padding-left:1ex">
<pre>clang -fopenmp -fopenmp-targets=x86_64-unknown-linux-gnu -S -emit-llvm</pre>
<pre>-O3 -o bundled.ll code.c</pre>
<pre>clang-offload-bundler --inputs=bundled.ll</pre>
<pre>--outputs=host.in.ll,device.in.ll</pre>
<pre>--targets=host-x86_64-unknown-linux-gnu,openmp-x86_64-unknown-linux-gnu</pre>
<pre>--type=ll --unbundle</pre>
<pre>opt -o host.opt.ll -O3 host.in.ll</pre>
<pre>llc -o host.o -filetype=obj host.opt.ll</pre>
<pre>opt -o device.opt.ll -O3 device.in.ll (custom pass should be added</pre>
<pre>here later on)</pre>
<pre>llc -o device.o -filetype=obj device.opt.ll</pre>
</blockquote>
<pre>Afterwards, I tried to bundle the object files for host & device:</pre>
<blockquote type="cite" style="margin:0 0 0 .8ex;
border-left:2px #729fcf solid;padding-left:1ex">
<pre>clang-offload-bundler --inputs=host.o,device.o --outputs=app.o</pre>
<pre>--targets=host-x86_64-unknown-linux-gnu,openmp-x86_64-unknown-linux-gnu</pre>
<pre>--type=o</pre>
</blockquote>
<pre>This command completes without any warning or error. </pre>
<pre>I tried to invoke gcc on the output to generate the executable from the</pre>
<pre>object file (app.o):</pre>
<blockquote type="cite" style="margin:0 0 0 .8ex;
border-left:2px #729fcf solid;padding-left:1ex">
<pre>gcc app.o -lomp -lomptarget</pre>
</blockquote>
<pre>However, I get the following error:</pre>
<blockquote type="cite" style="margin:0 0 0 .8ex;
border-left:2px #729fcf solid;padding-left:1ex">
<pre>/usr/bin/ld:</pre>
<pre>app.o:(.rodata..omp_offloading.device_images[.omp_offloading.descriptor_reg.x86_64-unknown-linux-gnu]+0x10):</pre>
<pre>undefined reference to `.omp_offloading.entries_begin'</pre>
<pre>/usr/bin/ld:</pre>
<pre>app.o:(.rodata..omp_offloading.device_images[.omp_offloading.descriptor_reg.x86_64-unknown-linux-gnu]+0x18):</pre>
<pre>undefined reference to `.omp_offloading.entries_end'</pre>
<pre>/usr/bin/ld:</pre>
<pre>app.o:(.rodata..omp_offloading.descriptor[.omp_offloading.descriptor_reg.x86_64-unknown-linux-gnu]+0x10):</pre>
<pre>undefined reference to `.omp_offloading.entries_begin'</pre>
<pre>/usr/bin/ld:</pre>
<pre>app.o:(.rodata..omp_offloading.descriptor[.omp_offloading.descriptor_reg.x86_64-unknown-linux-gnu]+0x18):</pre>
<pre>undefined reference to `.omp_offloading.entries_end'</pre>
</blockquote>
<pre>Do I need to run any additional steps before going through gcc or pass</pre>
<pre>additional flags? Is there another way of accomplishing the desired</pre>
<pre>behavior?</pre>
</blockquote>
<div><br>
</div>
<div>I think you should use clang with the full -fopenmp
-fopenmp-targets=x86_64-unknown-linux-gnu to build the
final executable, ie</div>
<div> $ clang -fopenmp
-fopenmp-targets=x86_64-unknown-linux-gnu app.o</div>
<div>If I remember correctly, the above symbols are
provided by a linker script in Clang 9.0, which gcc has
no clue about.</div>
<div><br>
</div>
<div>Please note that there have been recent changes in
that area and trunk now relies on a new tool
clang-offload-wrapper AFAICS (see <a
href="https://github.com/llvm/llvm-project/commit/5836c356fa6e17d0e10a2f9e0e111b7236dc15fb"
moz-do-not-send="true">https://github.com/llvm/llvm-project/commit/5836c356fa6e17d0e10a2f9e0e111b7236dc15fb</a>).</div>
<div>As before this should be called transparently by
clang, as long as it's correctly invoked with a
consistent value for -fopenmp-targets.</div>
<div><br>
</div>
<div>I haven't tested above command, but I hope it works.
Please let me know if it doesn't and I'll take a closer
look!</div>
<div>Jonas</div>
<div><br>
</div>
<blockquote type="cite" style="margin:0 0 0 .8ex;
border-left:2px #729fcf solid;padding-left:1ex">
<pre>Thanks a lot in advance,</pre>
<pre>Best regards</pre>
<pre>Lukas</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</body>
</html>