[Openmp-dev] Declare target functions and libomptarget image registration order

Tue Nov 3 09:44:14 PST 2020

I would recommend you attending the meeting tomorrow where we can discuss this.
Info for the meeting is here https://docs.google.com/document/d/1Tz8WFN13n7yJ-SCE0Qjqf9LmjGUw0dWO9Ts1ss4YOdg/edit#heading=h.4txf37fnh7lf

-----Original Message-----
From: Manoel Römmer <roemmer at itc.rwth-aachen.de> 
Sent: Thursday, October 29, 2020 10:21 AM
To: Narayanaswamy, Ravi <ravi.narayanaswamy at intel.com>; Johannes Doerfert <johannesdoerfert at gmail.com>; openmp-dev at lists.llvm.org
Cc: Eachempati, Deepak <deepak.eachempati at hpe.com>; Kelvin Li <kli at ca.ibm.com>
Subject: Re: [Openmp-dev] Declare target functions and libomptarget image registration order

Our plugin uses a libdl like interface on the device, so if the images are loaded in the 'correct' order, symbols are resolved correctly.
The 'correct' order should be the order that the images are registered with libomptarget.
We need to load every image manually (as opposed to having the device interface we use resolve library dependencies) because target images are, by default, not linked against other images at compile time

On 10/27/20 00:42, Narayanaswamy, Ravi wrote:
> The different scenarios of offloading are
> 1)   offloading from the main executable.
>     a)    the kernel is self contained.  Ie  has no calls to routines outside the main image.
>     b)   has calls to routine in know device libraries like device rtl and math
>     c)  has calls to user libraries.  I am considering only .so here since .a would have been linked in during compile time.
>
> 2)  offload from dynamic library .so
>     a)  the kernel is self contained.  Ie  has no calls to routines outside the .so image.
>     b)  has calls to routine in know device libraries like device rtl and math
>     c) has calls to user libraries
>
> You are requesting support for 1c and 2c.  All we need it to register the dynamic libraries with the plugin since they are registered 1st to libomptarget.
> The plugin keeps a table of all registered images and when any binary is compiled it also looks up all the registered image to resolve any required symbols.
> So your plugin needs to support some sort of dynamic linking..  Does your plugin for NEC SX-Aurora  have this support?
>
>
> -----Original Message-----
> From: Johannes Doerfert <johannesdoerfert at gmail.com>
> Sent: Monday, October 26, 2020 3:59 PM
> To: Manoel Römmer <roemmer at itc.rwth-aachen.de>; 
> openmp-dev at lists.llvm.org
> Cc: Narayanaswamy, Ravi <ravi.narayanaswamy at intel.com>; Eachempati, 
> Deepak <deepak.eachempati at hpe.com>; Kelvin Li <kli at ca.ibm.com>
> Subject: Re: [Openmp-dev] Declare target functions and libomptarget 
> image registration order
>
> This looks like a problem.
>
> For me to understand this right, do you explicitly call any target library functions, if so, which and in which order?
>
>
> @Ravi, @Deepak, @Kelvin, please take a look as well
>
>
> On 10/26/20 3:23 PM, Manoel Römmer wrote:
>> Hi Johannes,
>>
>> so the debug ouptut is:
>>
>> Libomptarget --> Loading RTLs...
>> Libomptarget --> Loading library 'libomptarget.rtl.ve.so'...
>> Target ve RTL --> Found 8 VE devices
>> Libomptarget --> Successfully loaded library 'libomptarget.rtl.ve.so'!
>> Libomptarget --> Registering RTL libomptarget.rtl.ve.so supporting 8 
>> devices!
>> Libomptarget --> Loading library 'libomptarget.rtl.ppc64.so'...
>> Libomptarget --> Unable to load library 'libomptarget.rtl.ppc64.so':
>> libomptarget.rtl.ppc64.so: cannot open shared object file: No such 
>> file or directory!
>> Libomptarget --> Loading library 'libomptarget.rtl.x86_64.so'...
>> Libomptarget --> Successfully loaded library 
>> 'libomptarget.rtl.x86_64.so'!
>> Libomptarget --> Registering RTL libomptarget.rtl.x86_64.so 
>> supporting
>> 4 devices!
>> Libomptarget --> Loading library 'libomptarget.rtl.cuda.so'...
>> Libomptarget --> Unable to load library 'libomptarget.rtl.cuda.so':
>> libomptarget.rtl.cuda.so: cannot open shared object file: No such 
>> file or directory!
>> Libomptarget --> Loading library 'libomptarget.rtl.aarch64.so'...
>> Libomptarget --> Unable to load library 'libomptarget.rtl.aarch64.so':
>> libomptarget.rtl.aarch64.so: cannot open shared object file: No such 
>> file or directory!
>> Libomptarget --> RTLs loaded!
>> Libomptarget --> Image 0x00002b57e6062a20 is compatible with RTL 
>> libomptarget.rtl.ve.so!
>> Libomptarget --> RTL 0x0000000000617580 has index 0!
>> Libomptarget --> Registering image 0x00002b57e6062a20 with RTL 
>> libomptarget.rtl.ve.so!
>> Libomptarget --> Done registering entries!
>> Libomptarget --> New requires flags 1 compatible with existing 1!
>> Libomptarget --> Image is compatible with RTL libomptarget.rtl.ve.so!
>> Libomptarget --> Registering image 0x0000000000400bc0 with RTL 
>> libomptarget.rtl.ve.so!
>> Libomptarget --> Done registering entries!
>> Libomptarget --> Call to omp_get_num_devices returning 8 Libomptarget 
>> --> Default TARGET OFFLOAD policy is now mandatory (devices were 
>> found) Libomptarget --> Entering target region with entry point
>> 0x0000000000400b40 and device Id -1
>> Libomptarget --> Checking whether device 0 is ready.
>> Libomptarget --> Is the device 0 (local ID 0) initialized? 0 Target 
>> ve RTL --> Available VEO version: 9 Libomptarget --> Device 0 is 
>> ready to use.
>> Target ve RTL --> Dev 0: load binary from 0x0000000000400bc0 image 
>> Target ve RTL --> Expecting to have 1 entries defined.
>> Target ve RTL --> Wrote target image to /tmp/tmpfile_XwAVwB.
>> ImageSize=5648
>> Target ve RTL --> ELF Type: 3
>> Target ve RTL --> Aurora device successfully initialized with loaded
>> binary: proc_handle=0x621d10, ctx=0x623720 [VE] ERROR: 
>> loadlib_handler() dlerror: /tmp/tmpfile_XwAVwB: undefined
>> symbol: target_func_in_lib
>> Target ve RTL --> veo_load_library() failed: LibHandle=0 
>> Name=/tmp/tmpfile_XwAVwB. Set env VEORUN_BIN for static linked target 
>> code.
>> Libomptarget --> Unable to generate entries table for device id 0.
>> Libomptarget --> Failed to init globals on device 0 Libomptarget --> 
>> Failed to get device 0 ready Libomptarget fatal error 1: failure of 
>> target construct while offloading is mandatory Libomptarget --> 
>> Unloading target library!
>> Libomptarget --> Image 0x0000000000400bc0 is compatible with RTL 
>> 0x0000000000617580!
>> Libomptarget --> Unregistered image 0x0000000000400bc0 from RTL 
>> 0x0000000000617580!
>> Libomptarget --> Done unregistering images!
>> Libomptarget --> Removing translation table for descriptor
>> 0x0000000000402210
>> Libomptarget --> Done unregistering library!
>> Libomptarget --> Unloading target library!
>> Libomptarget --> Image 0x00002b57e6062a20 is compatible with RTL 
>> 0x0000000000617580!
>> Libomptarget --> Unregistered image 0x00002b57e6062a20 from RTL 
>> 0x0000000000617580!
>> Libomptarget --> Done unregistering images!
>> Libomptarget --> Removing translation table for descriptor
>> 0x00002b57e6064400
>> Libomptarget --> Done unregistering library!
>> Libomptarget --> Deinit target library!
>>
>>
>> As you can see, we libomptarget finds two images 0x00002b57e6062a20 
>> (the shared library) and 0x0000000000400bc0 (the main program) in the 
>> correct order (the main program requires a symbol in the library so 
>> the library gets loaded first):
>>
>>> Libomptarget --> Registering image 0x00002b57e6062a20 with RTL
>> libomptarget.rtl.ve.so!
>>> ...
>>> Libomptarget --> Registering image 0x0000000000400bc0 with RTL
>> libomptarget.rtl.ve.so!
>>
>>
>> But then libomptarget calls __tgt_rtl_load_binary() with the image
>> 0x0000000000400bc0 (the main program) without first calling
>> __tgt_rtl_load_binary() with the image for the library:
>>
>>> Target ve RTL --> Dev 0: load binary from 0x0000000000400bc0 image
>> Which the leads to our plugin not being able to actually load the 
>> image due to unresolved symbols.
>>
>>
>> On 10/21/20 5:18 AM, Johannes Doerfert wrote:
>>> Hi Manoel,
>>>
>>> we briefly discussed it today in our meeting 
>>> https://docs.google.com/document/d/1Tz8WFN13n7yJ-SCE0Qjqf9LmjGUw0dWO
>>> 9Ts1ss4YOdg/edit?usp=sharing If we don't solve this over the list, 
>>> feel free to join next week.
>>> In the meantime, I'm unsure I grasp the situation.
>>> Could you use a debug enabled libomptarget and use the 
>>> LIBOMPTARGET_DEBUG environment variable to get the sequence of events?
>>>
>>> ~ Johannes
>>>
>>>
>>> On 10/20/20 5:48 AM, Römmer, Manoel via Openmp-dev wrote:
>>>> Hi,
>>>>
>>>> We have the following problem: We have a shared library containing 
>>>> a function which is declared with '#pragma omp declare target', and 
>>>> a main executable with a target region in which this function is 
>>>> called. Now the target image in the shared library is registered 
>>>> with libomptarget (__tgt_register_lib()) before the target image of 
>>>> the main executable.
>>>> However, libomptarget then passes the target image of the main 
>>>> executable to our RTL plugin (with __tgt_rtl_load_binary()) before 
>>>> the target image of the shared library.
>>>> This is a problem for us because our plugin then tries to load the 
>>>> main executable's image first and fails due to unresolved symbols.
>>>>
>>>> So, it seems to me, that libomptarget calls __tgt_rtl_load_binary() 
>>>> with images not in the order which they were registered but in the 
>>>> order they are placed in memory.
>>>>
>>>>
>>>> Is this intended behaviour?
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Manoel
>>>>
>>>>
>>>> _______________________________________________
>>>> Openmp-dev mailing list
>>>> Openmp-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev