[Openmp-dev] Using clang-offload-bundler to bundle application/executable

Tue Nov 5 02:25:39 PST 2019

Hi Jonas,

thanks for the hint, with "--relocation-model=pic" added to opt and llc,
I can run the bundled object file through clang without any errors.

However, the linking in clang does not seem to work right, at runtime
the libomptarget-plugin (rtl.cpp) throws an error because it is unable
to handle the ELF:

> Libomptarget --> Loading RTLs...
> Libomptarget --> Loading library 'libomptarget.rtl.ppc64.so'...
> Libomptarget --> Unable to load library 'libomptarget.rtl.ppc64.so':
> libomptarget.rtl.ppc64.so: cannot open shared object file: No such
> file or directory!
> Libomptarget --> Loading library 'libomptarget.rtl.x86_64.so'...
> Libomptarget --> Successfully loaded library 'libomptarget.rtl.x86_64.so'!
> Libomptarget --> Registering RTL libomptarget.rtl.x86_64.so supporting
> 4 devices!
> Libomptarget --> Loading library 'libomptarget.rtl.cuda.so'...
> Libomptarget --> Unable to load library 'libomptarget.rtl.cuda.so':
> libomptarget.rtl.cuda.so: cannot open shared object file: No such file
> or directory!
> Libomptarget --> Loading library 'libomptarget.rtl.aarch64.so'...
> Libomptarget --> Unable to load library 'libomptarget.rtl.aarch64.so':
> libomptarget.rtl.aarch64.so: cannot open shared object file: No such
> file or directory!
> Libomptarget --> RTLs loaded!
> Target x86_64 RTL --> Unable to get ELF handle: invalid operand!
> Libomptarget --> Image 0x0000000000000000 is NOT compatible with RTL
> libomptarget.rtl.x86_64.so!
> Libomptarget --> No RTL found for image 0x0000000000000000!
> Libomptarget --> Done registering entries!
> Libomptarget --> Call to omp_get_num_devices returning 0
> Libomptarget --> Default TARGET OFFLOAD policy is now disabled (no
> devices were found)
Do I need to specify additional flags to clang?

Thanks,

Lukas

Lukas Sommer, M.Sc.
TU Darmstadt
Embedded Systems and Applications Group (ESA)
Hochschulstr. 10, 64289 Darmstadt, Germany
Phone: +49 6151 1622429
www.esa.informatik.tu-darmstadt.de

On 11/4/19 5:43 PM, Jonas Hahnfeld wrote:
> Am Montag, den 04.11.2019, 17:28 +0100 schrieb Lukas Sommer:
>>
>> Hi Jonas,
>>
>> thanks for your help!
>>
>> I tried the suggested command, unfortunately it still fails:
>>
>>> clang -fopenmp -fopenmp-targets=x86_64-unknown-linux-gnu app.o
>> yields:
>>
>>> /usr/bin/ld: /tmp/app-c48732.o: relocation R_X86_64_32 against
>>> `.data' can not be used when making a shared object; recompile with
>>> -fPIC
>>> /usr/bin/ld: final link failed: nonrepresentable section on output
>>> /usr/bin/ld: cannot find /tmp/app-23d932.out inside /
>>> /usr/bin/ld: cannot find /tmp/app-23d932.out inside /
>>> clang-9: error: linker command failed with exit code 1 (use -v to
>>> see invocation)
>>> clang-9: error: linker command failed with exit code 1 (use -v to
>>> see invocation)
>> I also tried to generate and bundle assembly (.asm) and run that
>> through clang (with openmp-flags as above), but that gave the same error.
>
> Hmm, the target image is built as a shared library. To make this work,
> Clang adds -shared -fPIC in Generic_GCC::TranslateArgs.
> You're replacing this step with a manual invocation of `opt', so we
> need to pass some extra flags. I'm currently looking into what's
> needed, opt --relocation-model=pic might be a good start but I'm not
> sure that's all.
>
> Jonas
>
>> Lukas
>>
>> Lukas Sommer, M.Sc.
>> TU Darmstadt
>> Embedded Systems and Applications Group (ESA)
>> Hochschulstr. 10, 64289 Darmstadt, Germany
>> Phone: +49 6151 1622429
>> www.esa.informatik.tu-darmstadt.de
>> On 11/4/19 5:01 PM, Jonas Hahnfeld wrote:
>>> Hi Lukas,
>>>
>>> Am Montag, den 28.10.2019, 13:42 +0100 schrieb Lukas Sommer via
>>> Openmp-dev:
>>>> Hello,
>>>> I'm trying to use the clang-offload-bundler to merge the results of the
>>>> compilation for device and host for an application using OpenMP target
>>>> pragmas.
>>>> My goal is to run a custom pass on the device code and then bundle an
>>>> application from the modified device code and the host code. For testing
>>>> purposes, I use a very simple vector addition with target offloading
>>>> pragmas and the "offloading" to x86_64 (LLVM/Clang release 9.0).
>>>> I use clang to generate LLVM IR from the application code. Then I use
>>>> the clang-offload-bundler to unbundle the LLVM IR for device and host. I
>>>> run opt and llc separately on the LLVM IR for host and device.
>>>> To this end, on my application code (code.c), I run the following steps:
>>>>> clang -fopenmp -fopenmp-targets=x86_64-unknown-linux-gnu -S -emit-llvm
>>>>> -O3 -o bundled.ll code.c
>>>>> clang-offload-bundler --inputs=bundled.ll
>>>>> --outputs=host.in.ll,device.in.ll
>>>>> --targets=host-x86_64-unknown-linux-gnu,openmp-x86_64-unknown-linux-gnu
>>>>> --type=ll --unbundle
>>>>> opt -o host.opt.ll -O3 host.in.ll
>>>>> llc -o host.o -filetype=obj host.opt.ll
>>>>> opt -o device.opt.ll -O3 device.in.ll (custom pass should be added
>>>>> here later on)
>>>>> llc -o device.o -filetype=obj device.opt.ll
>>>> Afterwards, I tried to bundle the object files for host & device:
>>>>> clang-offload-bundler --inputs=host.o,device.o --outputs=app.o
>>>>> --targets=host-x86_64-unknown-linux-gnu,openmp-x86_64-unknown-linux-gnu
>>>>> --type=o
>>>> This command completes without any warning or error. 
>>>> I tried to invoke gcc on the output to generate the executable from the
>>>> object file (app.o):
>>>>> gcc app.o -lomp -lomptarget
>>>> However, I get the following error:
>>>>> /usr/bin/ld:
>>>>> app.o:(.rodata..omp_offloading.device_images[.omp_offloading.descriptor_reg.x86_64-unknown-linux-gnu]+0x10):
>>>>> undefined reference to `.omp_offloading.entries_begin'
>>>>> /usr/bin/ld:
>>>>> app.o:(.rodata..omp_offloading.device_images[.omp_offloading.descriptor_reg.x86_64-unknown-linux-gnu]+0x18):
>>>>> undefined reference to `.omp_offloading.entries_end'
>>>>> /usr/bin/ld:
>>>>> app.o:(.rodata..omp_offloading.descriptor[.omp_offloading.descriptor_reg.x86_64-unknown-linux-gnu]+0x10):
>>>>> undefined reference to `.omp_offloading.entries_begin'
>>>>> /usr/bin/ld:
>>>>> app.o:(.rodata..omp_offloading.descriptor[.omp_offloading.descriptor_reg.x86_64-unknown-linux-gnu]+0x18):
>>>>> undefined reference to `.omp_offloading.entries_end'
>>>> Do I need to run any additional steps before going through gcc or pass
>>>> additional flags? Is there another way of accomplishing the desired
>>>> behavior?
>>>
>>> I think you should use clang with the full -fopenmp
>>> -fopenmp-targets=x86_64-unknown-linux-gnu to build the final
>>> executable, ie
>>> $ clang -fopenmp -fopenmp-targets=x86_64-unknown-linux-gnu app.o
>>> If I remember correctly, the above symbols are provided by a linker
>>> script in Clang 9.0, which gcc has no clue about.
>>>
>>> Please note that there have been recent changes in that area and
>>> trunk now relies on a new tool clang-offload-wrapper AFAICS
>>> (see https://github.com/llvm/llvm-project/commit/5836c356fa6e17d0e10a2f9e0e111b7236dc15fb).
>>> As before this should be called transparently by clang, as long as
>>> it's correctly invoked with a consistent value for -fopenmp-targets.
>>>
>>> I haven't tested above command, but I hope it works. Please let me
>>> know if it doesn't and I'll take a closer look!
>>> Jonas
>>>
>>>> Thanks a lot in advance,
>>>> Best regards
>>>> Lukas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/openmp-dev/attachments/20191105/b251a2be/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.llvm.org/pipermail/openmp-dev/attachments/20191105/b251a2be/attachment-0001.sig>