[Openmp-dev] Declare target functions and libomptarget image registration order

Mon Nov 9 08:30:53 PST 2020

Our meeting is weekly (with a few exceptions) and we can discuss this 
whenever you make it to one.

~ Johannes

On 11/9/20 9:12 AM, Manoel Römmer via Openmp-dev wrote:
> Hi
>
> I'm sorry that I couldn't make it. Is there any chance I can join this 
> week or next week to discuss this?
>
> Manoel
>
> Am 03.11.20 um 18:44 schrieb Narayanaswamy, Ravi:
>> I would recommend you attending the meeting tomorrow where we can 
>> discuss this.
>> Info for the meeting is here 
>> https://docs.google.com/document/d/1Tz8WFN13n7yJ-SCE0Qjqf9LmjGUw0dWO9Ts1ss4YOdg/edit#heading=h.4txf37fnh7lf
>>
>>
>> -----Original Message-----
>> From: Manoel Römmer <roemmer at itc.rwth-aachen.de>
>> Sent: Thursday, October 29, 2020 10:21 AM
>> To: Narayanaswamy, Ravi <ravi.narayanaswamy at intel.com>; Johannes 
>> Doerfert <johannesdoerfert at gmail.com>; openmp-dev at lists.llvm.org
>> Cc: Eachempati, Deepak <deepak.eachempati at hpe.com>; Kelvin Li 
>> <kli at ca.ibm.com>
>> Subject: Re: [Openmp-dev] Declare target functions and libomptarget 
>> image registration order
>>
>> Our plugin uses a libdl like interface on the device, so if the 
>> images are loaded in the 'correct' order, symbols are resolved 
>> correctly.
>> The 'correct' order should be the order that the images are 
>> registered with libomptarget.
>> We need to load every image manually (as opposed to having the device 
>> interface we use resolve library dependencies) because target images 
>> are, by default, not linked against other images at compile time
>>
>>
>> On 10/27/20 00:42, Narayanaswamy, Ravi wrote:
>>> The different scenarios of offloading are
>>> 1)   offloading from the main executable.
>>>      a)    the kernel is self contained.  Ie  has no calls to 
>>> routines outside the main image.
>>>      b)   has calls to routine in know device libraries like device 
>>> rtl and math
>>>      c)  has calls to user libraries.  I am considering only .so 
>>> here since .a would have been linked in during compile time.
>>>
>>> 2)  offload from dynamic library .so
>>>      a)  the kernel is self contained.  Ie  has no calls to routines 
>>> outside the .so image.
>>>      b)  has calls to routine in know device libraries like device 
>>> rtl and math
>>>      c) has calls to user libraries
>>>
>>> You are requesting support for 1c and 2c.  All we need it to 
>>> register the dynamic libraries with the plugin since they are 
>>> registered 1st to libomptarget.
>>> The plugin keeps a table of all registered images and when any 
>>> binary is compiled it also looks up all the registered image to 
>>> resolve any required symbols.
>>> So your plugin needs to support some sort of dynamic linking..  Does 
>>> your plugin for NEC SX-Aurora  have this support?
>>>
>>>
>>> -----Original Message-----
>>> From: Johannes Doerfert <johannesdoerfert at gmail.com>
>>> Sent: Monday, October 26, 2020 3:59 PM
>>> To: Manoel Römmer <roemmer at itc.rwth-aachen.de>;
>>> openmp-dev at lists.llvm.org
>>> Cc: Narayanaswamy, Ravi <ravi.narayanaswamy at intel.com>; Eachempati,
>>> Deepak <deepak.eachempati at hpe.com>; Kelvin Li <kli at ca.ibm.com>
>>> Subject: Re: [Openmp-dev] Declare target functions and libomptarget
>>> image registration order
>>>
>>> This looks like a problem.
>>>
>>> For me to understand this right, do you explicitly call any target 
>>> library functions, if so, which and in which order?
>>>
>>>
>>> @Ravi, @Deepak, @Kelvin, please take a look as well
>>>
>>>
>>> On 10/26/20 3:23 PM, Manoel Römmer wrote:
>>>> Hi Johannes,
>>>>
>>>> so the debug ouptut is:
>>>>
>>>> Libomptarget --> Loading RTLs...
>>>> Libomptarget --> Loading library 'libomptarget.rtl.ve.so'...
>>>> Target ve RTL --> Found 8 VE devices
>>>> Libomptarget --> Successfully loaded library 'libomptarget.rtl.ve.so'!
>>>> Libomptarget --> Registering RTL libomptarget.rtl.ve.so supporting 8
>>>> devices!
>>>> Libomptarget --> Loading library 'libomptarget.rtl.ppc64.so'...
>>>> Libomptarget --> Unable to load library 'libomptarget.rtl.ppc64.so':
>>>> libomptarget.rtl.ppc64.so: cannot open shared object file: No such
>>>> file or directory!
>>>> Libomptarget --> Loading library 'libomptarget.rtl.x86_64.so'...
>>>> Libomptarget --> Successfully loaded library
>>>> 'libomptarget.rtl.x86_64.so'!
>>>> Libomptarget --> Registering RTL libomptarget.rtl.x86_64.so
>>>> supporting
>>>> 4 devices!
>>>> Libomptarget --> Loading library 'libomptarget.rtl.cuda.so'...
>>>> Libomptarget --> Unable to load library 'libomptarget.rtl.cuda.so':
>>>> libomptarget.rtl.cuda.so: cannot open shared object file: No such
>>>> file or directory!
>>>> Libomptarget --> Loading library 'libomptarget.rtl.aarch64.so'...
>>>> Libomptarget --> Unable to load library 'libomptarget.rtl.aarch64.so':
>>>> libomptarget.rtl.aarch64.so: cannot open shared object file: No such
>>>> file or directory!
>>>> Libomptarget --> RTLs loaded!
>>>> Libomptarget --> Image 0x00002b57e6062a20 is compatible with RTL
>>>> libomptarget.rtl.ve.so!
>>>> Libomptarget --> RTL 0x0000000000617580 has index 0!
>>>> Libomptarget --> Registering image 0x00002b57e6062a20 with RTL
>>>> libomptarget.rtl.ve.so!
>>>> Libomptarget --> Done registering entries!
>>>> Libomptarget --> New requires flags 1 compatible with existing 1!
>>>> Libomptarget --> Image is compatible with RTL libomptarget.rtl.ve.so!
>>>> Libomptarget --> Registering image 0x0000000000400bc0 with RTL
>>>> libomptarget.rtl.ve.so!
>>>> Libomptarget --> Done registering entries!
>>>> Libomptarget --> Call to omp_get_num_devices returning 8 Libomptarget
>>>> --> Default TARGET OFFLOAD policy is now mandatory (devices were
>>>> found) Libomptarget --> Entering target region with entry point
>>>> 0x0000000000400b40 and device Id -1
>>>> Libomptarget --> Checking whether device 0 is ready.
>>>> Libomptarget --> Is the device 0 (local ID 0) initialized? 0 Target
>>>> ve RTL --> Available VEO version: 9 Libomptarget --> Device 0 is
>>>> ready to use.
>>>> Target ve RTL --> Dev 0: load binary from 0x0000000000400bc0 image
>>>> Target ve RTL --> Expecting to have 1 entries defined.
>>>> Target ve RTL --> Wrote target image to /tmp/tmpfile_XwAVwB.
>>>> ImageSize=5648
>>>> Target ve RTL --> ELF Type: 3
>>>> Target ve RTL --> Aurora device successfully initialized with loaded
>>>> binary: proc_handle=0x621d10, ctx=0x623720 [VE] ERROR:
>>>> loadlib_handler() dlerror: /tmp/tmpfile_XwAVwB: undefined
>>>> symbol: target_func_in_lib
>>>> Target ve RTL --> veo_load_library() failed: LibHandle=0
>>>> Name=/tmp/tmpfile_XwAVwB. Set env VEORUN_BIN for static linked target
>>>> code.
>>>> Libomptarget --> Unable to generate entries table for device id 0.
>>>> Libomptarget --> Failed to init globals on device 0 Libomptarget -->
>>>> Failed to get device 0 ready Libomptarget fatal error 1: failure of
>>>> target construct while offloading is mandatory Libomptarget -->
>>>> Unloading target library!
>>>> Libomptarget --> Image 0x0000000000400bc0 is compatible with RTL
>>>> 0x0000000000617580!
>>>> Libomptarget --> Unregistered image 0x0000000000400bc0 from RTL
>>>> 0x0000000000617580!
>>>> Libomptarget --> Done unregistering images!
>>>> Libomptarget --> Removing translation table for descriptor
>>>> 0x0000000000402210
>>>> Libomptarget --> Done unregistering library!
>>>> Libomptarget --> Unloading target library!
>>>> Libomptarget --> Image 0x00002b57e6062a20 is compatible with RTL
>>>> 0x0000000000617580!
>>>> Libomptarget --> Unregistered image 0x00002b57e6062a20 from RTL
>>>> 0x0000000000617580!
>>>> Libomptarget --> Done unregistering images!
>>>> Libomptarget --> Removing translation table for descriptor
>>>> 0x00002b57e6064400
>>>> Libomptarget --> Done unregistering library!
>>>> Libomptarget --> Deinit target library!
>>>>
>>>>
>>>> As you can see, we libomptarget finds two images 0x00002b57e6062a20
>>>> (the shared library) and 0x0000000000400bc0 (the main program) in the
>>>> correct order (the main program requires a symbol in the library so
>>>> the library gets loaded first):
>>>>
>>>>> Libomptarget --> Registering image 0x00002b57e6062a20 with RTL
>>>> libomptarget.rtl.ve.so!
>>>>> ...
>>>>> Libomptarget --> Registering image 0x0000000000400bc0 with RTL
>>>> libomptarget.rtl.ve.so!
>>>>
>>>>
>>>> But then libomptarget calls __tgt_rtl_load_binary() with the image
>>>> 0x0000000000400bc0 (the main program) without first calling
>>>> __tgt_rtl_load_binary() with the image for the library:
>>>>
>>>>> Target ve RTL --> Dev 0: load binary from 0x0000000000400bc0 image
>>>> Which the leads to our plugin not being able to actually load the
>>>> image due to unresolved symbols.
>>>>
>>>>
>>>> On 10/21/20 5:18 AM, Johannes Doerfert wrote:
>>>>> Hi Manoel,
>>>>>
>>>>> we briefly discussed it today in our meeting
>>>>> https://docs.google.com/document/d/1Tz8WFN13n7yJ-SCE0Qjqf9LmjGUw0dWO
>>>>> 9Ts1ss4YOdg/edit?usp=sharing If we don't solve this over the list,
>>>>> feel free to join next week.
>>>>> In the meantime, I'm unsure I grasp the situation.
>>>>> Could you use a debug enabled libomptarget and use the
>>>>> LIBOMPTARGET_DEBUG environment variable to get the sequence of 
>>>>> events?
>>>>>
>>>>> ~ Johannes
>>>>>
>>>>>
>>>>> On 10/20/20 5:48 AM, Römmer, Manoel via Openmp-dev wrote:
>>>>>> Hi,
>>>>>>
>>>>>> We have the following problem: We have a shared library containing
>>>>>> a function which is declared with '#pragma omp declare target', and
>>>>>> a main executable with a target region in which this function is
>>>>>> called. Now the target image in the shared library is registered
>>>>>> with libomptarget (__tgt_register_lib()) before the target image of
>>>>>> the main executable.
>>>>>> However, libomptarget then passes the target image of the main
>>>>>> executable to our RTL plugin (with __tgt_rtl_load_binary()) before
>>>>>> the target image of the shared library.
>>>>>> This is a problem for us because our plugin then tries to load the
>>>>>> main executable's image first and fails due to unresolved symbols.
>>>>>>
>>>>>> So, it seems to me, that libomptarget calls __tgt_rtl_load_binary()
>>>>>> with images not in the order which they were registered but in the
>>>>>> order they are placed in memory.
>>>>>>
>>>>>>
>>>>>> Is this intended behaviour?
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Manoel
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Openmp-dev mailing list
>>>>>> Openmp-dev at lists.llvm.org
>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev
>
> _______________________________________________
> Openmp-dev mailing list
> Openmp-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev