[Openmp-dev] Using clang-offload-bundler to bundle application/executable
Jonas Hahnfeld via Openmp-dev
openmp-dev at lists.llvm.org
Tue Nov 5 13:46:05 PST 2019
So it seems to work for me:$ LIBOMPTARGET_DEBUG=1 LD_LIBRARY_PATH=libomptarget/libomptarget-debug/:$LD_LIBRARY_PATH ./a.outLibomptarget --> Loading RTLs...[...]Libomptarget --> Loading library 'libomptarget.rtl.x86_64.so'...Libomptarget --> Successfully loaded library 'libomptarget.rtl.x86_64.so'!Libomptarget --> Registering RTL libomptarget.rtl.x86_64.so supporting 4 devices![...]Libomptarget --> RTLs loaded!Libomptarget --> Image 0x0000000000602050 is compatible with RTL libomptarget.rtl.x86_64.so!Libomptarget --> RTL 0x000000000061b7c0 has index 0!Libomptarget --> Registering image 0x0000000000602050 with RTL libomptarget.rtl.x86_64.so!Libomptarget --> Done registering entries!Libomptarget --> Call to omp_get_num_devices returning 4Libomptarget --> Default TARGET OFFLOAD policy is now mandatory (devices were found)Libomptarget --> Entering target region with entry point 0x00000000004008c0 and device Id -1Libomptarget --> Checking whether device 0 is ready.Libomptarget --> Is the device 0 (local ID 0) initialized? 0Libomptarget --> Device 0 is ready to use.Target x86_64 RTL --> Dev 0: load binary from 0x0000000000602050 imageTarget x86_64 RTL --> Expecting to have 1 entries defined.Target x86_64 RTL --> Offset of entries section is (0x0000000000201020).Target x86_64 RTL --> Pointer to first entry to be loaded is (0x00002ad4f4632020).Target x86_64 RTL --> Entries table range is (0x00002ad4f4632020)->(0x00002ad4f4632040)Libomptarget --> Launching target execution __omp_offloading_48_54503d96_main_l8 with pointer 0x00002ad4f44316b0 (index=0).Target x86_64 RTL --> Running entry point at 0x00002ad4f44316b0...omp_is_initial_device: 0Libomptarget --> Unloading target library!Libomptarget --> Image 0x0000000000602050 is compatible with RTL 0x000000000061b7c0!Libomptarget --> Unregistered image 0x0000000000602050 from RTL 0x000000000061b7c0!Libomptarget --> Done unregistering images!Libomptarget --> Removing translation table for descriptor 0x00000000006040e0Libomptarget --> Done unregistering library!
I mostly used the steps from your initial post, but passed --relocation-model=pic for the device code (both opt and llc) and used clang -fopenmp -fopenmp-targets=x86_64-unknown-linux-gnu app.o for the final linking.
It's possible that my code is just too simple (target region with a printf of omp_is_initial_device) to trigger the problem you're seeing. Could you share your exact code?For the error message "invalid operand", I think libomptarget uses libelf incorrectly. Maybe you can try replacing "elf_errmsg(-1)" by "elf_errmsg(elf_errno())"? At least that's how I think it should be according to the the man page.
Regards,Jonas
Am Dienstag, den 05.11.2019, 11:25 +0100 schrieb Lukas Sommer:
> Hi Jonas,
> thanks for the hint, with "--relocation-model=pic" added to opt
> and llc, I can run the bundled object file through clang without
> any errors.
> However, the linking in clang does not seem to work right, at
> runtime the libomptarget-plugin (rtl.cpp) throws an error because
> it is unable to handle the ELF:
>
>
> > Libomptarget --> Loading RTLs...
> >
> > Libomptarget --> Loading library
> > 'libomptarget.rtl.ppc64.so'...
> >
> > Libomptarget --> Unable to load library
> > 'libomptarget.rtl.ppc64.so': libomptarget.rtl.ppc64.so: cannot
> > open shared object file: No such file or directory!
> >
> > Libomptarget --> Loading library
> > 'libomptarget.rtl.x86_64.so'...
> >
> > Libomptarget --> Successfully loaded library
> > 'libomptarget.rtl.x86_64.so'!
> >
> > Libomptarget --> Registering RTL libomptarget.rtl.x86_64.so
> > supporting 4 devices!
> >
> > Libomptarget --> Loading library
> > 'libomptarget.rtl.cuda.so'...
> >
> > Libomptarget --> Unable to load library
> > 'libomptarget.rtl.cuda.so': libomptarget.rtl.cuda.so: cannot
> > open shared object file: No such file or directory!
> >
> > Libomptarget --> Loading library
> > 'libomptarget.rtl.aarch64.so'...
> >
> > Libomptarget --> Unable to load library
> > 'libomptarget.rtl.aarch64.so': libomptarget.rtl.aarch64.so:
> > cannot open shared object file: No such file or directory!
> >
> > Libomptarget --> RTLs loaded!
> >
> > Target x86_64 RTL --> Unable to get ELF handle: invalid
> > operand!
> >
> > Libomptarget --> Image 0x0000000000000000 is NOT compatible
> > with RTL libomptarget.rtl.x86_64.so!
> >
> > Libomptarget --> No RTL found for image 0x0000000000000000!
> >
> > Libomptarget --> Done registering entries!
> >
> > Libomptarget --> Call to omp_get_num_devices returning 0
> >
> > Libomptarget --> Default TARGET OFFLOAD policy is now
> > disabled (no devices were found)
> >
> >
>
> Do I need to specify additional flags to clang?
> Thanks,
> Lukas
>
>
> Lukas Sommer, M.Sc.TU DarmstadtEmbedded Systems and Applications Group (ESA)Hochschulstr. 10, 64289 Darmstadt, GermanyPhone: +49 6151 1622429www.esa.informatik.tu-darmstadt.de
> On 11/4/19 5:43 PM, Jonas Hahnfeld
> wrote:
>
>
>
> >
> > Am Montag, den 04.11.2019, 17:28 +0100 schrieb Lukas Sommer:
> >
> > > Hi Jonas,
> > > thanks for your help!
> > > I tried the suggested command, unfortunately it still fails:
> > >
> > >
> > > > clang -fopenmp
> > > > -fopenmp-targets=x86_64-unknown-linux-gnu app.o
> > >
> > > yields:
> > >
> > >
> > > > /usr/bin/ld:
> > > > /tmp/app-c48732.o: relocation R_X86_64_32 against `.data' can
> > > > not be used when making a shared object; recompile with -fPIC
> > > >
> > > > /usr/bin/ld: final link failed: nonrepresentable section on
> > > > output
> > > >
> > > > /usr/bin/ld: cannot find /tmp/app-23d932.out inside /
> > > >
> > > > /usr/bin/ld: cannot find /tmp/app-23d932.out inside /
> > > >
> > > > clang-9: error: linker command failed with exit code 1 (use -v
> > > > to see invocation)
> > > >
> > > > clang-9: error: linker command failed with exit code 1 (use -v
> > > > to see invocation)
> > >
> > > I also tried to generate and bundle assembly (.asm) and run that
> > > through clang (with openmp-flags as above), but that gave the
> > > same error.
> > >
> > >
> >
> >
> >
> >
> > Hmm, the target image is built as a shared library. To make
> > this work, Clang adds -shared -fPIC in
> > Generic_GCC::TranslateArgs.
> > You're replacing this step with a manual invocation of `opt',
> > so we need to pass some extra flags. I'm currently looking into
> > what's needed, opt --relocation-model=pic might be a good start
> > but I'm not sure that's all.
> >
> >
> >
> > Jonas
> >
> >
> >
> >
> > > Lukas
> > >
> > >
> > > Lukas Sommer, M.Sc.
> > > TU Darmstadt
> > > Embedded Systems and Applications Group (ESA)
> > > Hochschulstr. 10, 64289 Darmstadt, Germany
> > > Phone: +49 6151 1622429
> > >
> > > www.esa.informatik.tu-darmstadt.de
> > >
> > > On 11/4/19 5:01 PM, Jonas Hahnfeld
> > > wrote:
> > >
> > >
> > >
> > > > Hi Lukas,
> > > >
> > > >
> > > >
> > > > Am Montag, den 28.10.2019, 13:42 +0100 schrieb Lukas
> > > > Sommer via Openmp-dev:
> > > >
> > > > > Hello,
> > > > > I'm trying to use the clang-offload-bundler to merge the results of the
> > > > > compilation for device and host for an application using OpenMP target
> > > > > pragmas.
> > > > > My goal is to run a custom pass on the device code and then bundle an
> > > > > application from the modified device code and the host code. For testing
> > > > > purposes, I use a very simple vector addition with target offloading
> > > > > pragmas and the "offloading" to x86_64 (LLVM/Clang release 9.0).
> > > > > I use clang to generate LLVM IR from the application code. Then I use
> > > > > the clang-offload-bundler to unbundle the LLVM IR for device and host. I
> > > > > run opt and llc separately on the LLVM IR for host and device.
> > > > > To this end, on my application code (code.c), I run the following steps:
> > > > >
> > > > > > clang -fopenmp -fopenmp-targets=x86_64-unknown-linux-gnu -S -emit-llvm
> > > > > > -O3 -o bundled.ll code.c
> > > > > > clang-offload-bundler --inputs=bundled.ll
> > > > > > --outputs=host.in.ll,device.in.ll
> > > > > > --targets=host-x86_64-unknown-linux-gnu,openmp-x86_64-unknown-linux-gnu
> > > > > > --type=ll --unbundle
> > > > > > opt -o host.opt.ll -O3 host.in.ll
> > > > > > llc -o host.o -filetype=obj host.opt.ll
> > > > > > opt -o device.opt.ll -O3 device.in.ll (custom pass should be added
> > > > > > here later on)
> > > > > > llc -o device.o -filetype=obj device.opt.ll
> > > > > >
> > > > >
> > > > > Afterwards, I tried to bundle the object files for host & device:
> > > > >
> > > > > > clang-offload-bundler --inputs=host.o,device.o --outputs=app.o
> > > > > > --targets=host-x86_64-unknown-linux-gnu,openmp-x86_64-unknown-linux-gnu
> > > > > > --type=o
> > > > > >
> > > > >
> > > > > This command completes without any warning or error.
> > > > > I tried to invoke gcc on the output to generate the executable from the
> > > > > object file (app.o):
> > > > >
> > > > > > gcc app.o -lomp -lomptarget
> > > > > >
> > > > >
> > > > > However, I get the following error:
> > > > >
> > > > > > /usr/bin/ld:
> > > > > > app.o:(.rodata..omp_offloading.device_images[.omp_offloading.descriptor_reg.x86_64-unknown-linux-gnu]+0x10):
> > > > > > undefined reference to `.omp_offloading.entries_begin'
> > > > > > /usr/bin/ld:
> > > > > > app.o:(.rodata..omp_offloading.device_images[.omp_offloading.descriptor_reg.x86_64-unknown-linux-gnu]+0x18):
> > > > > > undefined reference to `.omp_offloading.entries_end'
> > > > > > /usr/bin/ld:
> > > > > > app.o:(.rodata..omp_offloading.descriptor[.omp_offloading.descriptor_reg.x86_64-unknown-linux-gnu]+0x10):
> > > > > > undefined reference to `.omp_offloading.entries_begin'
> > > > > > /usr/bin/ld:
> > > > > > app.o:(.rodata..omp_offloading.descriptor[.omp_offloading.descriptor_reg.x86_64-unknown-linux-gnu]+0x18):
> > > > > > undefined reference to `.omp_offloading.entries_end'
> > > > > >
> > > > >
> > > > > Do I need to run any additional steps before going through gcc or pass
> > > > > additional flags? Is there another way of accomplishing the desired
> > > > > behavior?
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > > I think you should use clang with the full -fopenmp
> > > > -fopenmp-targets=x86_64-unknown-linux-gnu to build the final
> > > > executable, ie
> > > > $ clang -fopenmp
> > > > -fopenmp-targets=x86_64-unknown-linux-gnu app.o
> > > > If I remember correctly, the above symbols are provided
> > > > by a linker script in Clang 9.0, which gcc has no clue
> > > > about.
> > > >
> > > >
> > > >
> > > > Please note that there have been recent changes in that
> > > > area and trunk now relies on a new tool
> > > > clang-offload-wrapper AFAICS (see https://github.com/llvm/llvm-project/commit/5836c356fa6e17d0e10a2f9e0e111b7236dc15fb).
> > > > As before this should be called transparently by clang,
> > > > as long as it's correctly invoked with a consistent value
> > > > for -fopenmp-targets.
> > > >
> > > >
> > > >
> > > > I haven't tested above command, but I hope it works.
> > > > Please let me know if it doesn't and I'll take a closer
> > > > look!
> > > > Jonas
> > > >
> > > >
> > > >
> > > >
> > > > > Thanks a lot in advance,
> > > > > Best regards
> > > > > Lukas
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/openmp-dev/attachments/20191105/fe23394f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <http://lists.llvm.org/pipermail/openmp-dev/attachments/20191105/fe23394f/attachment-0001.sig>
More information about the Openmp-dev
mailing list