<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><blockquote type="cite" class="">Message: 1<br class="">Date: Mon, 26 Oct 2020 15:18:45 -0500<br class="">From: Kelvin Li via Openmp-dev <<a href="mailto:openmp-dev@lists.llvm.org" class="">openmp-dev@lists.llvm.org</a>><br class="">To: <a href="mailto:openmp-dev@lists.llvm.org" class="">openmp-dev@lists.llvm.org</a><br class="">Subject: [Openmp-dev] undefined symbol: ompt_start_tool<br class="">Message-ID:<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span><<a href="mailto:OFF5259549.0EC65D66-ON8525860D.006EC181-8525860D.006F94A6@notes.na.collabserv.com" class="">OFF5259549.0EC65D66-ON8525860D.006EC181-8525860D.006F94A6@notes.na.collabserv.com</a>><br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span><br class="">Content-Type: text/plain; charset="utf-8"<br class=""><br class="">Has anyone encounter the following error? I am wondering if it is <br class="">something to do with how I build libomp.so.<br class=""><br class="">$ LD_LIBRARY_PATH=/home/kli/clang-install/lib mpirun -np 1 ./a.out<br class="">a.out: symbol lookup error: /home/kli/clang-install/lib/libomp.so: <br class="">undefined symbol: ompt_start_tool<br class="">--------------------------------------------------------------------------<br class="">Primary job terminated normally, but 1 process returned<br class="">a non-zero exit code. Per user-direction, the job has been aborted.<br class="">--------------------------------------------------------------------------<br class="">--------------------------------------------------------------------------<br class="">mpirun detected that one or more processes exited with non-zero status, <br class="">thus causing<br class="">the job to be terminated. The first process to do so was:<br class=""><br class=""> Process name: [[14546,1],0]<br class=""> Exit code: 127<br class="">--------------------------------------------------------------------------<br class=""><br class="">But it works without mpirun.<br class=""><br class="">$ LD_LIBRARY_PATH=/home/kli/clang-install/lib ./a.out<br class="">0<br class="">1<br class="">2<br class="">3<br class=""><br class=""><br class="">Kelvin<br class=""></blockquote>Are you confident that <font face="Courier New" class="">/homie/kli/clang-install/lib</font> is the same on all of the nodes used by the MPI program?<div class="">And that it contains the same version of<font face="Courier New" class=""> libomp.so </font>everywhere?</div><div class=""><br class=""></div><div class="">Perhaps you should also set an envirable to have the OpenMP runtime print its version, something like this </div><div class=""><span style="font-style: normal;" class=""><font face="Courier New" class="">$ KMP_VERSION=1 ./a.out</font></span></div><div class=""><div class=""><span style="font-style: normal;" class=""><font face="Courier New" class="">LLVM OMP version: 5.0.20140926</font></span></div><div class=""><span style="font-style: normal;" class=""><font face="Courier New" class="">LLVM OMP library type: performance</font></span></div><div class=""><span style="font-style: normal;" class=""><font face="Courier New" class="">LLVM OMP link type: dynamic</font></span></div><div class=""><span style="font-style: normal;" class=""><font face="Courier New" class="">LLVM OMP build time: no_timestamp</font></span></div><div class=""><span style="font-style: normal;" class=""><font face="Courier New" class="">LLVM OMP build compiler: Clang 12.0</font></span></div><div class=""><span style="font-style: normal;" class=""><font face="Courier New" class="">LLVM OMP alternative compiler support: yes</font></span></div><div class=""><span style="font-style: normal;" class=""><font face="Courier New" class="">LLVM OMP API version: 5.0 (201611)</font></span></div><div class=""><span style="font-style: normal;" class=""><font face="Courier New" class="">LLVM OMP dynamic error checking: no</font></span></div><div class=""><span style="font-style: normal;" class=""><font face="Courier New" class="">LLVM OMP thread affinity support: no</font></span></div><div class=""><br class=""></div><div><blockquote type="cite" class=""><div class="">On 26 Oct 2020, at 23:44, via Openmp-dev <<a href="mailto:openmp-dev@lists.llvm.org" class="">openmp-dev@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Send Openmp-dev mailing list submissions to<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span><a href="mailto:openmp-dev@lists.llvm.org" class="">openmp-dev@lists.llvm.org</a><br class=""><br class="">To subscribe or unsubscribe via the World Wide Web, visit<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev<br class="">or, via email, send a message with subject or body 'help' to<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>openmp-dev-request@lists.llvm.org<br class=""><br class="">You can reach the person managing the list at<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>openmp-dev-owner@lists.llvm.org<br class=""><br class="">When replying, please edit your Subject line so it is more specific<br class="">than "Re: Contents of Openmp-dev digest..."<br class=""><br class=""><br class="">Today's Topics:<br class=""><br class=""> 1. undefined symbol: ompt_start_tool (Kelvin Li via Openmp-dev)<br class=""> 2. Re: Declare target functions and libomptarget image<br class=""> registration order (Manoel Römmer via Openmp-dev)<br class=""> 3. Re: Declare target functions and libomptarget image<br class=""> registration order (Johannes Doerfert via Openmp-dev)<br class=""> 4. Re: Declare target functions and libomptarget image<br class=""> registration order (Narayanaswamy, Ravi via Openmp-dev)<br class=""><br class=""><br class="">----------------------------------------------------------------------<br class=""><br class="">Message: 1<br class="">Date: Mon, 26 Oct 2020 15:18:45 -0500<br class="">From: Kelvin Li via Openmp-dev <openmp-dev@lists.llvm.org><br class="">To: openmp-dev@lists.llvm.org<br class="">Subject: [Openmp-dev] undefined symbol: ompt_start_tool<br class="">Message-ID:<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span><OFF5259549.0EC65D66-ON8525860D.006EC181-8525860D.006F94A6@notes.na.collabserv.com><br class=""><span class="Apple-tab-span" style="white-space:pre"> </span><br class="">Content-Type: text/plain; charset="utf-8"<br class=""><br class="">Has anyone encounter the following error? I am wondering if it is <br class="">something to do with how I build libomp.so.<br class=""><br class="">$ LD_LIBRARY_PATH=/home/kli/clang-install/lib mpirun -np 1 ./a.out<br class="">a.out: symbol lookup error: /home/kli/clang-install/lib/libomp.so: <br class="">undefined symbol: ompt_start_tool<br class="">--------------------------------------------------------------------------<br class="">Primary job terminated normally, but 1 process returned<br class="">a non-zero exit code. Per user-direction, the job has been aborted.<br class="">--------------------------------------------------------------------------<br class="">--------------------------------------------------------------------------<br class="">mpirun detected that one or more processes exited with non-zero status, <br class="">thus causing<br class="">the job to be terminated. The first process to do so was:<br class=""><br class=""> Process name: [[14546,1],0]<br class=""> Exit code: 127<br class="">--------------------------------------------------------------------------<br class=""><br class="">But it works without mpirun.<br class=""><br class="">$ LD_LIBRARY_PATH=/home/kli/clang-install/lib ./a.out<br class="">0<br class="">1<br class="">2<br class="">3<br class=""><br class=""><br class="">Kelvin<br class=""><br class=""><br class="">-------------- next part --------------<br class="">An HTML attachment was scrubbed...<br class="">URL: <http://lists.llvm.org/pipermail/openmp-dev/attachments/20201026/8ae7fa34/attachment-0001.html><br class=""><br class="">------------------------------<br class=""><br class="">Message: 2<br class="">Date: Mon, 26 Oct 2020 21:23:28 +0100<br class="">From: Manoel Römmer via Openmp-dev <openmp-dev@lists.llvm.org><br class="">To: Johannes Doerfert <johannesdoerfert@gmail.com>,<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>"openmp-dev@lists.llvm.org" <openmp-dev@lists.llvm.org><br class="">Subject: Re: [Openmp-dev] Declare target functions and libomptarget<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>image registration order<br class="">Message-ID: <5e5949e5-732c-81a8-5fa6-a274d5ac5e31@itc.rwth-aachen.de><br class="">Content-Type: text/plain; charset="utf-8"; format=flowed<br class=""><br class="">Hi Johannes,<br class=""><br class="">so the debug ouptut is:<br class=""><br class="">Libomptarget --> Loading RTLs...<br class="">Libomptarget --> Loading library 'libomptarget.rtl.ve.so'...<br class="">Target ve RTL --> Found 8 VE devices<br class="">Libomptarget --> Successfully loaded library 'libomptarget.rtl.ve.so'!<br class="">Libomptarget --> Registering RTL libomptarget.rtl.ve.so supporting 8 <br class="">devices!<br class="">Libomptarget --> Loading library 'libomptarget.rtl.ppc64.so'...<br class="">Libomptarget --> Unable to load library 'libomptarget.rtl.ppc64.so': <br class="">libomptarget.rtl.ppc64.so: cannot open shared object file: No such file <br class="">or directory!<br class="">Libomptarget --> Loading library 'libomptarget.rtl.x86_64.so'...<br class="">Libomptarget --> Successfully loaded library 'libomptarget.rtl.x86_64.so'!<br class="">Libomptarget --> Registering RTL libomptarget.rtl.x86_64.so supporting 4 <br class="">devices!<br class="">Libomptarget --> Loading library 'libomptarget.rtl.cuda.so'...<br class="">Libomptarget --> Unable to load library 'libomptarget.rtl.cuda.so': <br class="">libomptarget.rtl.cuda.so: cannot open shared object file: No such file <br class="">or directory!<br class="">Libomptarget --> Loading library 'libomptarget.rtl.aarch64.so'...<br class="">Libomptarget --> Unable to load library 'libomptarget.rtl.aarch64.so': <br class="">libomptarget.rtl.aarch64.so: cannot open shared object file: No such <br class="">file or directory!<br class="">Libomptarget --> RTLs loaded!<br class="">Libomptarget --> Image 0x00002b57e6062a20 is compatible with RTL <br class="">libomptarget.rtl.ve.so!<br class="">Libomptarget --> RTL 0x0000000000617580 has index 0!<br class="">Libomptarget --> Registering image 0x00002b57e6062a20 with RTL <br class="">libomptarget.rtl.ve.so!<br class="">Libomptarget --> Done registering entries!<br class="">Libomptarget --> New requires flags 1 compatible with existing 1!<br class="">Libomptarget --> Image is compatible with RTL libomptarget.rtl.ve.so!<br class="">Libomptarget --> Registering image 0x0000000000400bc0 with RTL <br class="">libomptarget.rtl.ve.so!<br class="">Libomptarget --> Done registering entries!<br class="">Libomptarget --> Call to omp_get_num_devices returning 8<br class="">Libomptarget --> Default TARGET OFFLOAD policy is now mandatory (devices <br class="">were found)<br class="">Libomptarget --> Entering target region with entry point <br class="">0x0000000000400b40 and device Id -1<br class="">Libomptarget --> Checking whether device 0 is ready.<br class="">Libomptarget --> Is the device 0 (local ID 0) initialized? 0<br class="">Target ve RTL --> Available VEO version: 9<br class="">Libomptarget --> Device 0 is ready to use.<br class="">Target ve RTL --> Dev 0: load binary from 0x0000000000400bc0 image<br class="">Target ve RTL --> Expecting to have 1 entries defined.<br class="">Target ve RTL --> Wrote target image to /tmp/tmpfile_XwAVwB. ImageSize=5648<br class="">Target ve RTL --> ELF Type: 3<br class="">Target ve RTL --> Aurora device successfully initialized with loaded <br class="">binary: proc_handle=0x621d10, ctx=0x623720<br class="">[VE] ERROR: loadlib_handler() dlerror: /tmp/tmpfile_XwAVwB: undefined <br class="">symbol: target_func_in_lib<br class="">Target ve RTL --> veo_load_library() failed: LibHandle=0 <br class="">Name=/tmp/tmpfile_XwAVwB. Set env VEORUN_BIN for static linked target code.<br class="">Libomptarget --> Unable to generate entries table for device id 0.<br class="">Libomptarget --> Failed to init globals on device 0<br class="">Libomptarget --> Failed to get device 0 ready<br class="">Libomptarget fatal error 1: failure of target construct while offloading <br class="">is mandatory<br class="">Libomptarget --> Unloading target library!<br class="">Libomptarget --> Image 0x0000000000400bc0 is compatible with RTL <br class="">0x0000000000617580!<br class="">Libomptarget --> Unregistered image 0x0000000000400bc0 from RTL <br class="">0x0000000000617580!<br class="">Libomptarget --> Done unregistering images!<br class="">Libomptarget --> Removing translation table for descriptor <br class="">0x0000000000402210<br class="">Libomptarget --> Done unregistering library!<br class="">Libomptarget --> Unloading target library!<br class="">Libomptarget --> Image 0x00002b57e6062a20 is compatible with RTL <br class="">0x0000000000617580!<br class="">Libomptarget --> Unregistered image 0x00002b57e6062a20 from RTL <br class="">0x0000000000617580!<br class="">Libomptarget --> Done unregistering images!<br class="">Libomptarget --> Removing translation table for descriptor <br class="">0x00002b57e6064400<br class="">Libomptarget --> Done unregistering library!<br class="">Libomptarget --> Deinit target library!<br class=""><br class=""><br class="">As you can see, we libomptarget finds two images 0x00002b57e6062a20 (the <br class="">shared library) and 0x0000000000400bc0 (the main program) in the correct <br class="">order (the main program requires a symbol in the library so the library <br class="">gets loaded first):<br class=""><br class=""><blockquote type="cite" class="">Libomptarget --> Registering image 0x00002b57e6062a20 with RTL <br class=""></blockquote>libomptarget.rtl.ve.so!<br class=""><blockquote type="cite" class="">...<br class="">Libomptarget --> Registering image 0x0000000000400bc0 with RTL <br class=""></blockquote>libomptarget.rtl.ve.so!<br class=""><br class=""><br class="">But then libomptarget calls __tgt_rtl_load_binary() with the image <br class="">0x0000000000400bc0 (the main program) without first calling <br class="">__tgt_rtl_load_binary() with the image for the library:<br class=""><br class=""><blockquote type="cite" class="">Target ve RTL --> Dev 0: load binary from 0x0000000000400bc0 image<br class=""></blockquote><br class="">Which the leads to our plugin not being able to actually load the image <br class="">due to unresolved symbols.<br class=""><br class=""><br class="">On 10/21/20 5:18 AM, Johannes Doerfert wrote:<br class=""><blockquote type="cite" class="">Hi Manoel,<br class=""><br class="">we briefly discussed it today in our meeting <br class="">https://docs.google.com/document/d/1Tz8WFN13n7yJ-SCE0Qjqf9LmjGUw0dWO9Ts1ss4YOdg/edit?usp=sharing<br class="">If we don't solve this over the list, feel free to join next week.<br class="">In the meantime, I'm unsure I grasp the situation.<br class="">Could you use a debug enabled libomptarget and use the <br class="">LIBOMPTARGET_DEBUG environment variable to get the sequence of events?<br class=""><br class="">~ Johannes<br class=""><br class=""><br class="">On 10/20/20 5:48 AM, Römmer, Manoel via Openmp-dev wrote:<br class=""><blockquote type="cite" class="">Hi,<br class=""><br class="">We have the following problem: We have a shared library containing a <br class="">function which is declared with '#pragma omp declare target', and a <br class="">main executable with a target region in which this function is <br class="">called. Now the target image in the shared library is registered with <br class="">libomptarget (__tgt_register_lib()) before the target image of the <br class="">main executable.<br class="">However, libomptarget then passes the target image of the main <br class="">executable to our RTL plugin (with __tgt_rtl_load_binary()) before <br class="">the target image of the shared library.<br class="">This is a problem for us because our plugin then tries to load the <br class="">main executable's image first and fails due to unresolved symbols.<br class=""><br class="">So, it seems to me, that libomptarget calls __tgt_rtl_load_binary() <br class="">with images not in the order which they were registered but in the <br class="">order they are placed in memory.<br class=""><br class=""><br class="">Is this intended behaviour?<br class=""><br class=""><br class="">Thanks,<br class=""><br class="">Manoel<br class=""><br class=""><br class="">_______________________________________________<br class="">Openmp-dev mailing list<br class="">Openmp-dev@lists.llvm.org<br class="">https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev<br class=""></blockquote></blockquote><br class=""><br class=""><br class="">------------------------------<br class=""><br class="">Message: 3<br class="">Date: Mon, 26 Oct 2020 17:58:37 -0500<br class="">From: Johannes Doerfert via Openmp-dev <openmp-dev@lists.llvm.org><br class="">To: Manoel Römmer <roemmer@itc.rwth-aachen.de>,<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>"openmp-dev@lists.llvm.org" <openmp-dev@lists.llvm.org><br class="">Cc: "Eachempati, Deepak" <deepak.eachempati@hpe.com><br class="">Subject: Re: [Openmp-dev] Declare target functions and libomptarget<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>image registration order<br class="">Message-ID: <18160c18-e591-71f8-e5a7-208e729a1fda@gmail.com><br class="">Content-Type: text/plain; charset=utf-8; format=flowed<br class=""><br class="">This looks like a problem.<br class=""><br class="">For me to understand this right, do you explicitly call any target <br class="">library functions, if so, which and in which order?<br class=""><br class=""><br class="">@Ravi, @Deepak, @Kelvin, please take a look as well<br class=""><br class=""><br class="">On 10/26/20 3:23 PM, Manoel Römmer wrote:<br class=""><blockquote type="cite" class="">Hi Johannes,<br class=""><br class="">so the debug ouptut is:<br class=""><br class="">Libomptarget --> Loading RTLs...<br class="">Libomptarget --> Loading library 'libomptarget.rtl.ve.so'...<br class="">Target ve RTL --> Found 8 VE devices<br class="">Libomptarget --> Successfully loaded library 'libomptarget.rtl.ve.so'!<br class="">Libomptarget --> Registering RTL libomptarget.rtl.ve.so supporting 8 <br class="">devices!<br class="">Libomptarget --> Loading library 'libomptarget.rtl.ppc64.so'...<br class="">Libomptarget --> Unable to load library 'libomptarget.rtl.ppc64.so': <br class="">libomptarget.rtl.ppc64.so: cannot open shared object file: No such <br class="">file or directory!<br class="">Libomptarget --> Loading library 'libomptarget.rtl.x86_64.so'...<br class="">Libomptarget --> Successfully loaded library <br class="">'libomptarget.rtl.x86_64.so'!<br class="">Libomptarget --> Registering RTL libomptarget.rtl.x86_64.so supporting <br class="">4 devices!<br class="">Libomptarget --> Loading library 'libomptarget.rtl.cuda.so'...<br class="">Libomptarget --> Unable to load library 'libomptarget.rtl.cuda.so': <br class="">libomptarget.rtl.cuda.so: cannot open shared object file: No such file <br class="">or directory!<br class="">Libomptarget --> Loading library 'libomptarget.rtl.aarch64.so'...<br class="">Libomptarget --> Unable to load library 'libomptarget.rtl.aarch64.so': <br class="">libomptarget.rtl.aarch64.so: cannot open shared object file: No such <br class="">file or directory!<br class="">Libomptarget --> RTLs loaded!<br class="">Libomptarget --> Image 0x00002b57e6062a20 is compatible with RTL <br class="">libomptarget.rtl.ve.so!<br class="">Libomptarget --> RTL 0x0000000000617580 has index 0!<br class="">Libomptarget --> Registering image 0x00002b57e6062a20 with RTL <br class="">libomptarget.rtl.ve.so!<br class="">Libomptarget --> Done registering entries!<br class="">Libomptarget --> New requires flags 1 compatible with existing 1!<br class="">Libomptarget --> Image is compatible with RTL libomptarget.rtl.ve.so!<br class="">Libomptarget --> Registering image 0x0000000000400bc0 with RTL <br class="">libomptarget.rtl.ve.so!<br class="">Libomptarget --> Done registering entries!<br class="">Libomptarget --> Call to omp_get_num_devices returning 8<br class="">Libomptarget --> Default TARGET OFFLOAD policy is now mandatory <br class="">(devices were found)<br class="">Libomptarget --> Entering target region with entry point <br class="">0x0000000000400b40 and device Id -1<br class="">Libomptarget --> Checking whether device 0 is ready.<br class="">Libomptarget --> Is the device 0 (local ID 0) initialized? 0<br class="">Target ve RTL --> Available VEO version: 9<br class="">Libomptarget --> Device 0 is ready to use.<br class="">Target ve RTL --> Dev 0: load binary from 0x0000000000400bc0 image<br class="">Target ve RTL --> Expecting to have 1 entries defined.<br class="">Target ve RTL --> Wrote target image to /tmp/tmpfile_XwAVwB. <br class="">ImageSize=5648<br class="">Target ve RTL --> ELF Type: 3<br class="">Target ve RTL --> Aurora device successfully initialized with loaded <br class="">binary: proc_handle=0x621d10, ctx=0x623720<br class="">[VE] ERROR: loadlib_handler() dlerror: /tmp/tmpfile_XwAVwB: undefined <br class="">symbol: target_func_in_lib<br class="">Target ve RTL --> veo_load_library() failed: LibHandle=0 <br class="">Name=/tmp/tmpfile_XwAVwB. Set env VEORUN_BIN for static linked target <br class="">code.<br class="">Libomptarget --> Unable to generate entries table for device id 0.<br class="">Libomptarget --> Failed to init globals on device 0<br class="">Libomptarget --> Failed to get device 0 ready<br class="">Libomptarget fatal error 1: failure of target construct while <br class="">offloading is mandatory<br class="">Libomptarget --> Unloading target library!<br class="">Libomptarget --> Image 0x0000000000400bc0 is compatible with RTL <br class="">0x0000000000617580!<br class="">Libomptarget --> Unregistered image 0x0000000000400bc0 from RTL <br class="">0x0000000000617580!<br class="">Libomptarget --> Done unregistering images!<br class="">Libomptarget --> Removing translation table for descriptor <br class="">0x0000000000402210<br class="">Libomptarget --> Done unregistering library!<br class="">Libomptarget --> Unloading target library!<br class="">Libomptarget --> Image 0x00002b57e6062a20 is compatible with RTL <br class="">0x0000000000617580!<br class="">Libomptarget --> Unregistered image 0x00002b57e6062a20 from RTL <br class="">0x0000000000617580!<br class="">Libomptarget --> Done unregistering images!<br class="">Libomptarget --> Removing translation table for descriptor <br class="">0x00002b57e6064400<br class="">Libomptarget --> Done unregistering library!<br class="">Libomptarget --> Deinit target library!<br class=""><br class=""><br class="">As you can see, we libomptarget finds two images 0x00002b57e6062a20 <br class="">(the shared library) and 0x0000000000400bc0 (the main program) in the <br class="">correct order (the main program requires a symbol in the library so <br class="">the library gets loaded first):<br class=""><br class=""><blockquote type="cite" class="">Libomptarget --> Registering image 0x00002b57e6062a20 with RTL <br class=""></blockquote>libomptarget.rtl.ve.so!<br class=""><blockquote type="cite" class="">...<br class="">Libomptarget --> Registering image 0x0000000000400bc0 with RTL <br class=""></blockquote>libomptarget.rtl.ve.so!<br class=""><br class=""><br class="">But then libomptarget calls __tgt_rtl_load_binary() with the image <br class="">0x0000000000400bc0 (the main program) without first calling <br class="">__tgt_rtl_load_binary() with the image for the library:<br class=""><br class=""><blockquote type="cite" class="">Target ve RTL --> Dev 0: load binary from 0x0000000000400bc0 image<br class=""></blockquote><br class="">Which the leads to our plugin not being able to actually load the <br class="">image due to unresolved symbols.<br class=""><br class=""><br class="">On 10/21/20 5:18 AM, Johannes Doerfert wrote:<br class=""><blockquote type="cite" class="">Hi Manoel,<br class=""><br class="">we briefly discussed it today in our meeting <br class="">https://docs.google.com/document/d/1Tz8WFN13n7yJ-SCE0Qjqf9LmjGUw0dWO9Ts1ss4YOdg/edit?usp=sharing<br class="">If we don't solve this over the list, feel free to join next week.<br class="">In the meantime, I'm unsure I grasp the situation.<br class="">Could you use a debug enabled libomptarget and use the <br class="">LIBOMPTARGET_DEBUG environment variable to get the sequence of events?<br class=""><br class="">~ Johannes<br class=""><br class=""><br class="">On 10/20/20 5:48 AM, Römmer, Manoel via Openmp-dev wrote:<br class=""><blockquote type="cite" class="">Hi,<br class=""><br class="">We have the following problem: We have a shared library containing a <br class="">function which is declared with '#pragma omp declare target', and a <br class="">main executable with a target region in which this function is <br class="">called. Now the target image in the shared library is registered <br class="">with libomptarget (__tgt_register_lib()) before the target image of <br class="">the main executable.<br class="">However, libomptarget then passes the target image of the main <br class="">executable to our RTL plugin (with __tgt_rtl_load_binary()) before <br class="">the target image of the shared library.<br class="">This is a problem for us because our plugin then tries to load the <br class="">main executable's image first and fails due to unresolved symbols.<br class=""><br class="">So, it seems to me, that libomptarget calls __tgt_rtl_load_binary() <br class="">with images not in the order which they were registered but in the <br class="">order they are placed in memory.<br class=""><br class=""><br class="">Is this intended behaviour?<br class=""><br class=""><br class="">Thanks,<br class=""><br class="">Manoel<br class=""><br class=""><br class="">_______________________________________________<br class="">Openmp-dev mailing list<br class="">Openmp-dev@lists.llvm.org<br class="">https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev<br class=""></blockquote></blockquote><br class=""></blockquote><br class=""><br class="">------------------------------<br class=""><br class="">Message: 4<br class="">Date: Mon, 26 Oct 2020 23:42:14 +0000<br class="">From: "Narayanaswamy, Ravi via Openmp-dev" <openmp-dev@lists.llvm.org><br class="">To: Johannes Doerfert <johannesdoerfert@gmail.com>, Manoel Römmer<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span><roemmer@itc.rwth-aachen.de>, "openmp-dev@lists.llvm.org"<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span><openmp-dev@lists.llvm.org><br class="">Cc: "Eachempati, Deepak" <deepak.eachempati@hpe.com><br class="">Subject: Re: [Openmp-dev] Declare target functions and libomptarget<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>image registration order<br class="">Message-ID:<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span><BY5PR11MB4210ED46396813590081925897190@BY5PR11MB4210.namprd11.prod.outlook.com><br class=""><span class="Apple-tab-span" style="white-space:pre"> </span><br class="">Content-Type: text/plain; charset="utf-8"<br class=""><br class="">The different scenarios of offloading are<br class="">1) offloading from the main executable.<br class=""> a) the kernel is self contained. Ie has no calls to routines outside the main image.<br class=""> b) has calls to routine in know device libraries like device rtl and math<br class=""> c) has calls to user libraries. I am considering only .so here since .a would have been linked in during compile time.<br class=""><br class="">2) offload from dynamic library .so<br class=""> a) the kernel is self contained. Ie has no calls to routines outside the .so image.<br class=""> b) has calls to routine in know device libraries like device rtl and math<br class=""> c) has calls to user libraries<br class=""><br class="">You are requesting support for 1c and 2c. All we need it to register the dynamic libraries with the plugin since they are registered 1st to libomptarget. <br class="">The plugin keeps a table of all registered images and when any binary is compiled it also looks up all the registered image to resolve any required symbols.<br class="">So your plugin needs to support some sort of dynamic linking.. Does your plugin for NEC SX-Aurora have this support?<br class=""><br class=""><br class="">-----Original Message-----<br class="">From: Johannes Doerfert <johannesdoerfert@gmail.com> <br class="">Sent: Monday, October 26, 2020 3:59 PM<br class="">To: Manoel Römmer <roemmer@itc.rwth-aachen.de>; openmp-dev@lists.llvm.org<br class="">Cc: Narayanaswamy, Ravi <ravi.narayanaswamy@intel.com>; Eachempati, Deepak <deepak.eachempati@hpe.com>; Kelvin Li <kli@ca.ibm.com><br class="">Subject: Re: [Openmp-dev] Declare target functions and libomptarget image registration order<br class=""><br class="">This looks like a problem.<br class=""><br class="">For me to understand this right, do you explicitly call any target library functions, if so, which and in which order?<br class=""><br class=""><br class="">@Ravi, @Deepak, @Kelvin, please take a look as well<br class=""><br class=""><br class="">On 10/26/20 3:23 PM, Manoel Römmer wrote:<br class=""><blockquote type="cite" class="">Hi Johannes,<br class=""><br class="">so the debug ouptut is:<br class=""><br class="">Libomptarget --> Loading RTLs...<br class="">Libomptarget --> Loading library 'libomptarget.rtl.ve.so'...<br class="">Target ve RTL --> Found 8 VE devices<br class="">Libomptarget --> Successfully loaded library 'libomptarget.rtl.ve.so'!<br class="">Libomptarget --> Registering RTL libomptarget.rtl.ve.so supporting 8 <br class="">devices!<br class="">Libomptarget --> Loading library 'libomptarget.rtl.ppc64.so'...<br class="">Libomptarget --> Unable to load library 'libomptarget.rtl.ppc64.so': <br class="">libomptarget.rtl.ppc64.so: cannot open shared object file: No such <br class="">file or directory!<br class="">Libomptarget --> Loading library 'libomptarget.rtl.x86_64.so'...<br class="">Libomptarget --> Successfully loaded library <br class="">'libomptarget.rtl.x86_64.so'!<br class="">Libomptarget --> Registering RTL libomptarget.rtl.x86_64.so supporting <br class="">4 devices!<br class="">Libomptarget --> Loading library 'libomptarget.rtl.cuda.so'...<br class="">Libomptarget --> Unable to load library 'libomptarget.rtl.cuda.so': <br class="">libomptarget.rtl.cuda.so: cannot open shared object file: No such file <br class="">or directory!<br class="">Libomptarget --> Loading library 'libomptarget.rtl.aarch64.so'...<br class="">Libomptarget --> Unable to load library 'libomptarget.rtl.aarch64.so': <br class="">libomptarget.rtl.aarch64.so: cannot open shared object file: No such <br class="">file or directory!<br class="">Libomptarget --> RTLs loaded!<br class="">Libomptarget --> Image 0x00002b57e6062a20 is compatible with RTL <br class="">libomptarget.rtl.ve.so!<br class="">Libomptarget --> RTL 0x0000000000617580 has index 0!<br class="">Libomptarget --> Registering image 0x00002b57e6062a20 with RTL <br class="">libomptarget.rtl.ve.so!<br class="">Libomptarget --> Done registering entries!<br class="">Libomptarget --> New requires flags 1 compatible with existing 1!<br class="">Libomptarget --> Image is compatible with RTL libomptarget.rtl.ve.so!<br class="">Libomptarget --> Registering image 0x0000000000400bc0 with RTL <br class="">libomptarget.rtl.ve.so!<br class="">Libomptarget --> Done registering entries!<br class="">Libomptarget --> Call to omp_get_num_devices returning 8<br class="">Libomptarget --> Default TARGET OFFLOAD policy is now mandatory <br class="">(devices were found)<br class="">Libomptarget --> Entering target region with entry point <br class="">0x0000000000400b40 and device Id -1<br class="">Libomptarget --> Checking whether device 0 is ready.<br class="">Libomptarget --> Is the device 0 (local ID 0) initialized? 0<br class="">Target ve RTL --> Available VEO version: 9<br class="">Libomptarget --> Device 0 is ready to use.<br class="">Target ve RTL --> Dev 0: load binary from 0x0000000000400bc0 image<br class="">Target ve RTL --> Expecting to have 1 entries defined.<br class="">Target ve RTL --> Wrote target image to /tmp/tmpfile_XwAVwB. <br class="">ImageSize=5648<br class="">Target ve RTL --> ELF Type: 3<br class="">Target ve RTL --> Aurora device successfully initialized with loaded <br class="">binary: proc_handle=0x621d10, ctx=0x623720<br class="">[VE] ERROR: loadlib_handler() dlerror: /tmp/tmpfile_XwAVwB: undefined <br class="">symbol: target_func_in_lib<br class="">Target ve RTL --> veo_load_library() failed: LibHandle=0 <br class="">Name=/tmp/tmpfile_XwAVwB. Set env VEORUN_BIN for static linked target <br class="">code.<br class="">Libomptarget --> Unable to generate entries table for device id 0.<br class="">Libomptarget --> Failed to init globals on device 0<br class="">Libomptarget --> Failed to get device 0 ready<br class="">Libomptarget fatal error 1: failure of target construct while <br class="">offloading is mandatory<br class="">Libomptarget --> Unloading target library!<br class="">Libomptarget --> Image 0x0000000000400bc0 is compatible with RTL <br class="">0x0000000000617580!<br class="">Libomptarget --> Unregistered image 0x0000000000400bc0 from RTL <br class="">0x0000000000617580!<br class="">Libomptarget --> Done unregistering images!<br class="">Libomptarget --> Removing translation table for descriptor <br class="">0x0000000000402210<br class="">Libomptarget --> Done unregistering library!<br class="">Libomptarget --> Unloading target library!<br class="">Libomptarget --> Image 0x00002b57e6062a20 is compatible with RTL <br class="">0x0000000000617580!<br class="">Libomptarget --> Unregistered image 0x00002b57e6062a20 from RTL <br class="">0x0000000000617580!<br class="">Libomptarget --> Done unregistering images!<br class="">Libomptarget --> Removing translation table for descriptor <br class="">0x00002b57e6064400<br class="">Libomptarget --> Done unregistering library!<br class="">Libomptarget --> Deinit target library!<br class=""><br class=""><br class="">As you can see, we libomptarget finds two images 0x00002b57e6062a20 <br class="">(the shared library) and 0x0000000000400bc0 (the main program) in the <br class="">correct order (the main program requires a symbol in the library so <br class="">the library gets loaded first):<br class=""><br class=""><blockquote type="cite" class="">Libomptarget --> Registering image 0x00002b57e6062a20 with RTL <br class=""></blockquote>libomptarget.rtl.ve.so!<br class=""><blockquote type="cite" class="">...<br class="">Libomptarget --> Registering image 0x0000000000400bc0 with RTL <br class=""></blockquote>libomptarget.rtl.ve.so!<br class=""><br class=""><br class="">But then libomptarget calls __tgt_rtl_load_binary() with the image <br class="">0x0000000000400bc0 (the main program) without first calling <br class="">__tgt_rtl_load_binary() with the image for the library:<br class=""><br class=""><blockquote type="cite" class="">Target ve RTL --> Dev 0: load binary from 0x0000000000400bc0 image<br class=""></blockquote><br class="">Which the leads to our plugin not being able to actually load the <br class="">image due to unresolved symbols.<br class=""><br class=""><br class="">On 10/21/20 5:18 AM, Johannes Doerfert wrote:<br class=""><blockquote type="cite" class="">Hi Manoel,<br class=""><br class="">we briefly discussed it today in our meeting <br class="">https://docs.google.com/document/d/1Tz8WFN13n7yJ-SCE0Qjqf9LmjGUw0dWO9Ts1ss4YOdg/edit?usp=sharing<br class="">If we don't solve this over the list, feel free to join next week.<br class="">In the meantime, I'm unsure I grasp the situation.<br class="">Could you use a debug enabled libomptarget and use the <br class="">LIBOMPTARGET_DEBUG environment variable to get the sequence of events?<br class=""><br class="">~ Johannes<br class=""><br class=""><br class="">On 10/20/20 5:48 AM, Römmer, Manoel via Openmp-dev wrote:<br class=""><blockquote type="cite" class="">Hi,<br class=""><br class="">We have the following problem: We have a shared library containing a <br class="">function which is declared with '#pragma omp declare target', and a <br class="">main executable with a target region in which this function is <br class="">called. Now the target image in the shared library is registered <br class="">with libomptarget (__tgt_register_lib()) before the target image of <br class="">the main executable.<br class="">However, libomptarget then passes the target image of the main <br class="">executable to our RTL plugin (with __tgt_rtl_load_binary()) before <br class="">the target image of the shared library.<br class="">This is a problem for us because our plugin then tries to load the <br class="">main executable's image first and fails due to unresolved symbols.<br class=""><br class="">So, it seems to me, that libomptarget calls __tgt_rtl_load_binary() <br class="">with images not in the order which they were registered but in the <br class="">order they are placed in memory.<br class=""><br class=""><br class="">Is this intended behaviour?<br class=""><br class=""><br class="">Thanks,<br class=""><br class="">Manoel<br class=""><br class=""><br class="">_______________________________________________<br class="">Openmp-dev mailing list<br class="">Openmp-dev@lists.llvm.org<br class="">https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev<br class=""></blockquote></blockquote><br class=""></blockquote><br class="">------------------------------<br class=""><br class="">Subject: Digest Footer<br class=""><br class="">_______________________________________________<br class="">Openmp-dev mailing list<br class="">Openmp-dev@lists.llvm.org<br class="">https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev<br class=""><br class=""><br class="">------------------------------<br class=""><br class="">End of Openmp-dev Digest, Vol 82, Issue 10<br class="">******************************************<br class=""></div></div></blockquote></div><br class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">-- Jim<br class="">James Cownie <<a href="mailto:jcownie@gmail.com" class="">jcownie@gmail.com</a>><br class="">Mob: +44 780 637 7146<br class=""><br class=""><br class=""><br class=""></div></div>
</div>
<br class=""></div></body></html>