<span style=" font-size:10pt;font-family:sans-serif">Hi Joachim,</span><br><br><span style=" font-size:10pt;font-family:sans-serif">Thanks.  I

still think that both "./a.out" and "mpirun -np 1 ./a.out"

use the same library.  Here are the ldd output.</span><br><br><span style=" font-size:10pt;font-family:sans-serif">$ ldd a.out</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    linux-vdso64.so.1 (0x00007fffa4810000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    libomp.so => /home/kli/clang-install/lib/libomp.so

(0x00007fffa46b0000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    libpthread.so.0 => /lib64/power9/libpthread.so.0

(0x00007fffa4650000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    libc.so.6 => /lib64/power9/libc.so.6 (0x00007fffa4440000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    libdl.so.2 => /lib64/libdl.so.2 (0x00007fffa4410000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    /lib64/ld64.so.2 (0x00007fffa4830000)</span><br><br><span style=" font-size:10pt;font-family:sans-serif">$ LD_LIBRARY_PATH=/home/kli/clang-install/lib

ldd a.out</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    linux-vdso64.so.1 (0x00007fffb54d0000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    libomp.so => /home/kli/clang-install/lib/libomp.so

(0x00007fffb5370000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    libpthread.so.0 => /lib64/power9/libpthread.so.0

(0x00007fffb5310000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    libc.so.6 => /lib64/power9/libc.so.6 (0x00007fffb5100000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    libdl.so.2 => /lib64/libdl.so.2 (0x00007fffb50d0000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    /lib64/ld64.so.2 (0x00007fffb54f0000)</span><br><br><span style=" font-size:10pt;font-family:sans-serif">$ LD_LIBRARY_PATH=/home/kli/clang-install/lib

mpirun -np 1 ldd a.out</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    linux-vdso64.so.1 (0x0000200000050000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    .../spectrum_mpi/latest/container/../lib/libpami_cudahook.so

(0x0000200000070000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    libomp.so => /home/kli/clang-install/lib/libomp.so

(0x00002000000a0000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    libpthread.so.0 => /lib64/power9/libpthread.so.0

(0x0000200000210000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    libc.so.6 => /lib64/power9/libc.so.6 (0x0000200000260000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    libdl.so.2 => /lib64/libdl.so.2 (0x0000200000470000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00002000004a0000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    libm.so.6 => /lib64/power9/libm.so.6 (0x00002000006d0000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000200000800000)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

    /lib64/ld64.so.2 (0x0000200000000000)</span><br><span style=" font-size:10pt;font-family:sans-serif"><br></span><br><span style=" font-size:10pt;font-family:sans-serif">Preloading the

libomp.so does not seem to help.</span><br><br><span style=" font-size:10pt;font-family:sans-serif">$ LD_PRELOAD=/home/kli/clang-install/lib/libomp.so

mpirun -np 1 ./a.out</span><br><span style=" font-size:10pt;font-family:sans-serif">./a.out: symbol

lookup error: /home/kli/clang-install/lib/libomp.so: undefined symbol:

ompt_start_tool</span><br><span style=" font-size:10pt;font-family:sans-serif">--------------------------------------------------------------------------</span><br><span style=" font-size:10pt;font-family:sans-serif">Primary job  terminated

normally, but 1 process returned</span><br><span style=" font-size:10pt;font-family:sans-serif">a non-zero exit

code. Per user-direction, the job has been aborted.</span><br><span style=" font-size:10pt;font-family:sans-serif">--------------------------------------------------------------------------</span><br><span style=" font-size:10pt;font-family:sans-serif">--------------------------------------------------------------------------</span><br><span style=" font-size:10pt;font-family:sans-serif">mpirun detected

that one or more processes exited with non-zero status, thus causing</span><br><span style=" font-size:10pt;font-family:sans-serif">the job to be

terminated. The first process to do so was:</span><br><br><span style=" font-size:10pt;font-family:sans-serif">  Process

name: [[53352,1],0]</span><br><span style=" font-size:10pt;font-family:sans-serif">  Exit code:

   127</span><br><span style=" font-size:10pt;font-family:sans-serif">--------------------------------------------------------------------------</span><br><br><span style=" font-size:10pt;font-family:sans-serif">I tried both commands

with LD_DEBUG.  It seems that somehow the libarcher.so cannot be found

to resolve ompt_start_tool.</span><br><br><span style=" font-size:10pt;font-family:sans-serif">$ LD_DEBUG=bindings

LD_LIBRARY_PATH=/home/kli/clang-install/lib ./a.out 2>&1| grep ompt_start_tool</span><br><span style=" font-size:10pt;font-family:sans-serif">   

  9105:        binding file /home/kli/clang-install/lib/libomp.so

[0] to /home/kli/clang-install/lib/libomp.so [0]: normal symbol `ompt_start_tool'

[VERSION]</span><br><span style=" font-size:10pt;font-family:sans-serif">   

  9105:        /home/kli/clang-install/lib/libomp.so:

error: symbol lookup error: undefined symbol: ompt_start_tool (fatal)</span><br><span style=" font-size:10pt;font-family:sans-serif">   

  9105:        binding file /home/kli/clang-install/lib/libarcher.so

[0] to /home/kli/clang-install/lib/libarcher.so [0]: normal symbol `ompt_start_tool'</span><br><br><span style=" font-size:10pt;font-family:sans-serif">$ LD_DEBUG=bindings

LD_LIBRARY_PATH=/home/kli/clang-install/lib mpirun -np 1 ./a.out 2>&1|

grep ompt_start_tool</span><br><span style=" font-size:10pt;font-family:sans-serif">   

 27652:        binding file /home/kli/clang-install/lib/libomp.so

[0] to /home/kli/clang-install/lib/libomp.so [0]: normal symbol `ompt_start_tool'

[VERSION]</span><br><span style=" font-size:10pt;font-family:sans-serif">   

 27652:        /home/kli/clang-install/lib/libomp.so:

error: symbol lookup error: undefined symbol: ompt_start_tool (fatal)</span><br><span style=" font-size:10pt;font-family:sans-serif">./a.out: symbol

lookup error: /home/kli/clang-install/lib/libomp.so: undefined symbol:

ompt_start_tool</span><br><br><br><span style=" font-size:10pt;font-family:sans-serif">Thanks,<br>Kelvin<br></span><br><br><br><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">From:

       </span><span style=" font-size:9pt;font-family:sans-serif">Joachim

Protze <protze.joachim@gmail.com> <span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">To:

       </span><span style=" font-size:9pt;font-family:sans-serif">Kelvin

Li <kli@ca.ibm.com>, openmp-dev@lists.llvm.org <span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Cc:

       </span><span style=" font-size:9pt;font-family:sans-serif">Jim

Cownie <jcownie@gmail.com> <span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Date:

       </span><span style=" font-size:9pt;font-family:sans-serif">2020/10/28

03:57 AM</span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Subject:

       </span><span style=" font-size:9pt;font-family:sans-serif">[EXTERNAL]

Re: [Openmp-dev] : undefined symbol: ompt_start_tool</span><br><hr noshade><br><br><br><tt><span style=" font-size:10pt">Hi Kelvin,<br><br>while this LD_PRELOAD is a workaround for the symptom (as it happen to<br>also implement ompt_start_tool), it does not explain your issue.<br><br>For my local installation I get:<br><br>$ readelf --syms libomp.so | grep ompt_start_tool<br>   658: 0000000000098050    54 FUNC    WEAK  

DEFAULT   12<br>ompt_start_tool@@VERSION<br>   776: 00000000000c8f40     8 OBJECT  LOCAL  DEFAULT

  26<br>_ZL22ompt_start_tool_resu<br>  2821: 0000000000098050    54 FUNC    WEAK  

DEFAULT   12 ompt_start_tool<br><br>So, the runtime has a (weak) implementation of this function.<br><br><br>My suspicion is that mpirun adds some path to LD_LIBRARY_PATH, so that

a different libomp is loaded.  You might compare $ ldd a.out and $ LD_LIBRARY_PATH=/home/kli/clang-install/lib ldd a.out and $ LD_LIBRARY_PATH=/home/kli/clang-install/lib mpirun -np 1 ldd a.out Instead of preloading libarcher, you can also preload a specific OpenMP runtime to be used for execution with: $ LD_PRELOAD=/home/kli/clang-install/lib/libomp.so Is this the same runtime used, when you execute without mpirun? Do you get the same error, when preloading this runtime without mpirun? Best Joachim Am 27.10.20 um 21:22 schrieb Kelvin Li via Openmp-dev: > I figure out how to make it work.  I need to preload libarcher.so.

 I <br>> don't understand why it cannot be done automatically in the "mpirun

... <br>> ./a.out" case.<br>> <br>> $ LD_PRELOAD=/home/kli/clang-install/lib/libarcher.so <br>> LD_LIBRARY_PATH=/home/kli/clang-install/lib KMP_VERSION=1 mpirun -np

1 <br>> ./a.out<br>> LLVM OMP version: 5.0.20140926<br>> LLVM OMP library type: performance<br>> LLVM OMP link type: dynamic<br>> LLVM OMP build time: no_timestamp<br>> LLVM OMP build compiler: Clang 11.0<br>> LLVM OMP alternative compiler support: yes<br>> LLVM OMP API version: 5.0 (201611)<br>> LLVM OMP dynamic error checking: no<br>> LLVM OMP plain barrier branch bits: gather=2, release=2<br>> LLVM OMP forkjoin barrier branch bits: gather=2, release=2<br>> LLVM OMP reduction barrier branch bits: gather=1, release=1<br>> LLVM OMP plain barrier pattern: gather=hyper, release=hyper<br>> LLVM OMP forkjoin barrier pattern: gather=hyper, release=hyper<br>> LLVM OMP reduction barrier pattern: gather=hyper, release=hyper<br>> LLVM OMP lock type: run time selectable<br>> LLVM OMP thread affinity support: not used<br>> 0<br>> 1<br>> 2<br>> 3<br>> <br>> Kelvin<br>> <br>> <br>> <br>> <br>> From:   Kelvin Li via Openmp-dev <openmp-dev@lists.llvm.org><br>> To:     Jim Cownie <jcownie@gmail.com><br>> Cc:     via Openmp-dev <openmp-dev@lists.llvm.org><br>> Date:   2020/10/27 01:24 PM<br>> Subject:        [EXTERNAL] Re: [Openmp-dev] :

 undefined symbol: <br>> ompt_start_tool<br>> Sent by:        "Openmp-dev" <openmp-dev-bounces@lists.llvm.org><br>> <br>> <br>> <br>> Hi Jim,<br>> <br>> Here is what I get with KMP_VERSION=1.<br>> <br>> $ LD_LIBRARY_PATH=$HOME/clang-install/lib KMP_VERSION=1 ./a.out<br>> LLVM OMP version: 5.0.20140926<br>> LLVM OMP library type: performance<br>> LLVM OMP link type: dynamic<br>> LLVM OMP build time: no_timestamp<br>> LLVM OMP build compiler: Clang 11.0<br>> LLVM OMP alternative compiler support: yes<br>> LLVM OMP API version: 5.0 (201611)<br>> LLVM OMP dynamic error checking: no<br>> LLVM OMP plain barrier branch bits: gather=2, release=2<br>> LLVM OMP forkjoin barrier branch bits: gather=2, release=2<br>> LLVM OMP reduction barrier branch bits: gather=1, release=1<br>> LLVM OMP plain barrier pattern: gather=hyper, release=hyper<br>> LLVM OMP forkjoin barrier pattern: gather=hyper, release=hyper<br>> LLVM OMP reduction barrier pattern: gather=hyper, release=hyper<br>> LLVM OMP lock type: run time selectable<br>> LLVM OMP thread affinity support: not used<br>> 0<br>> 1<br>> 3<br>> 2<br>> <br>> For the mpirun case, <br>> <br>> $ KMP_VERSION=1 mpirun -np 1 ./a.out<br>> ./a.out: symbol lookup error: /home/kli/clang-install/lib/libomp.so:

<br>> undefined symbol: ompt_start_tool<br>> --------------------------------------------------------------------------<br>> Primary job  terminated normally, but 1 process returned<br>> a non-zero exit code. Per user-direction, the job has been aborted.<br>> --------------------------------------------------------------------------<br>> --------------------------------------------------------------------------<br>> mpirun detected that one or more processes exited with non-zero status,

<br>> thus causing<br>> the job to be terminated. The first process to do so was:<br>> <br>>   Process name: [[52030,1],0]<br>>   Exit code:    127<br>> --------------------------------------------------------------------------<br>> <br>> <br>> Kelvin<br>> <br>> <br>> <br>> <br>> From:        Jim Cownie <jcownie@gmail.com><br>> To:        Kelvin Li <kli@ca.ibm.com><br>> Cc:        via Openmp-dev <openmp-dev@lists.llvm.org><br>> Date:        2020/10/27 11:09 AM<br>> Subject:        [EXTERNAL] Re: [Openmp-dev] :

 undefined symbol: <br>> ompt_start_tool<br>> <br>> <br>> <br>> On 27 Oct 2020, at 15:00, Kelvin Li wrote: I don't think that is the...

   <br>>    <br>> <br>> <br>> <br>> This Message Is From an External Sender<br>> This message came from outside your organization.<br>> <br>> <br>> <br>> <br>> <br>> On 27 Oct 2020, at 15:00, Kelvin Li <kli@ca.ibm.com> wrote:<br>> <br>> I don't think that is the case.  There is only one task "-np

1" on one <br>> node.  Both './a.out' and 'mpirun -np 1 ./a.out' are issued on

the same <br>> node which has the same library in /home/kli/clang-install/lib.  That

is <br>> puzzling me!<br>> <br>> It really looks as if you’re getting two different versions of the

<br>> runtime, though, so having the runtime tell you its properties is

still <br>> likely useful.<br>> If nothing else, it may show up that you’re not propagating envirables

as <br>> you might have hoped (if the MPI version doesn’t print anything !)<br>> <br>> -- Jim<br>> James Cownie <jcownie@gmail.com><br>> Mob: +44 780 637 7146<br>> Kelvin<br>> <br>> <br>> <br>> <br>> From:        Jim Cownie via Openmp-dev <openmp-dev@lists.llvm.org><br>> To:        via Openmp-dev <openmp-dev@lists.llvm.org>,

<br>> openmp-dev-request@lists.llvm.org<br>> Date:        2020/10/27 04:46 AM<br>> Subject:        [EXTERNAL] Re: [Openmp-dev] :

 undefined symbol: <br>> ompt_start_tool<br>> Sent by:        "Openmp-dev" <openmp-dev-bounces@lists.llvm.org><br>> <br>> <br>> <br>> Message: 1 Date: Mon, 26 Oct 2020 15:18:45 -0500 From: Kelvin Li via

<br>> Openmp-dev...   <br>> <br>> <br>> <br>> This Message Is From an External Sender<br>> This message came from outside your organization.<br>> <br>> <br>> <br>> <br>> <br>> Message: 1<br>> Date: Mon, 26 Oct 2020 15:18:45 -0500<br>> From: Kelvin Li via Openmp-dev <openmp-dev@lists.llvm.org><br>> To: openmp-dev@lists.llvm.org<br>> Subject: [Openmp-dev] undefined symbol: ompt_start_tool<br>> Message-ID:<br>> <<br>> OFF5259549.0EC65D66-ON8525860D.006EC181-8525860D.006F94A6@notes.na.collabserv.com<br>>><br>> <br>> Content-Type: text/plain; charset="utf-8"<br>> <br>> Has anyone encounter the following error?  I am wondering if

it is <br>> something to do with how I build libomp.so.<br>> <br>> $ LD_LIBRARY_PATH=/home/kli/clang-install/lib mpirun -np 1 ./a.out<br>> a.out: symbol lookup error: /home/kli/clang-install/lib/libomp.so:

<br>> undefined symbol: ompt_start_tool<br>> --------------------------------------------------------------------------<br>> Primary job  terminated normally, but 1 process returned<br>> a non-zero exit code. Per user-direction, the job has been aborted.<br>> --------------------------------------------------------------------------<br>> --------------------------------------------------------------------------<br>> mpirun detected that one or more processes exited with non-zero status,

<br>> thus causing<br>> the job to be terminated. The first process to do so was:<br>> <br>> Process name: [[14546,1],0]<br>> Exit code:    127<br>> --------------------------------------------------------------------------<br>> <br>> But it works without mpirun.<br>> <br>> $ LD_LIBRARY_PATH=/home/kli/clang-install/lib ./a.out<br>> 0<br>> 1<br>> 2<br>> 3<br>> <br>> <br>> Kelvin<br>> Are you confident that /homie/kli/clang-install/libis the same on

all of <br>> the nodes used by the MPI program?<br>> And that it contains the same version oflibomp.so everywhere?<br>> <br>> Perhaps you should also set an envirable to have the OpenMP runtime

print <br>> its version, something like this <br>> $ KMP_VERSION=1 ./a.out<br>> LLVM OMP version: 5.0.20140926<br>> LLVM OMP library type: performance<br>> LLVM OMP link type: dynamic<br>> LLVM OMP build time: no_timestamp<br>> LLVM OMP build compiler: Clang 12.0<br>> LLVM OMP alternative compiler support: yes<br>> LLVM OMP API version: 5.0 (201611)<br>> LLVM OMP dynamic error checking: no<br>> LLVM OMP thread affinity support: no<br>> <br>> On 26 Oct 2020, at 23:44, via Openmp-dev <openmp-dev@lists.llvm.org>

<br>> wrote:<br>> <br>> Send Openmp-dev mailing list submissions to<br>> openmp-dev@lists.llvm.org<br>> <br>> To subscribe or unsubscribe via the World Wide Web, visit<br>> </span></tt><a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev"><tt><span style=" font-size:10pt">https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev</span></tt></a><tt><span style=" font-size:10pt"><br>> or, via email, send a message with subject or body 'help' to<br>> openmp-dev-request@lists.llvm.org<br>> <br>> You can reach the person managing the list at<br>> openmp-dev-owner@lists.llvm.org<br>> <br>> When replying, please edit your Subject line so it is more specific<br>> than "Re: Contents of Openmp-dev digest..."<br>> <br>> <br>> Today's Topics:<br>> <br>> 1. undefined symbol: ompt_start_tool (Kelvin Li via Openmp-dev)<br>> 2. Re: Declare target functions and libomptarget image<br>>    registration order (Manoel Römmer via Openmp-dev)<br>> 3. Re: Declare target functions and libomptarget image<br>>    registration order (Johannes Doerfert via Openmp-dev)<br>> 4. Re: Declare target functions and libomptarget image<br>>    registration order (Narayanaswamy, Ravi via Openmp-dev)<br>> <br>> <br>> ----------------------------------------------------------------------<br>> <br>> Message: 1<br>> Date: Mon, 26 Oct 2020 15:18:45 -0500<br>> From: Kelvin Li via Openmp-dev <openmp-dev@lists.llvm.org><br>> To: openmp-dev@lists.llvm.org<br>> Subject: [Openmp-dev] undefined symbol: ompt_start_tool<br>> Message-ID:<br>> <OFF5259549.0EC65D66-ON8525860D.006EC181-8525860D.006F94A6@notes.na.collabserv.com><br>> <br>> Content-Type: text/plain; charset="utf-8"<br>> <br>> Has anyone encounter the following error?  I am wondering if

it is <br>> something to do with how I build libomp.so.<br>> <br>> $ LD_LIBRARY_PATH=/home/kli/clang-install/lib mpirun -np 1 ./a.out<br>> a.out: symbol lookup error: /home/kli/clang-install/lib/libomp.so:

<br>> undefined symbol: ompt_start_tool<br>> --------------------------------------------------------------------------<br>> Primary job  terminated normally, but 1 process returned<br>> a non-zero exit code. Per user-direction, the job has been aborted.<br>> --------------------------------------------------------------------------<br>> --------------------------------------------------------------------------<br>> mpirun detected that one or more processes exited with non-zero status,

<br>> thus causing<br>> the job to be terminated. The first process to do so was:<br>> <br>> Process name: [[14546,1],0]<br>> Exit code:    127<br>> --------------------------------------------------------------------------<br>> <br>> But it works without mpirun.<br>> <br>> $ LD_LIBRARY_PATH=/home/kli/clang-install/lib ./a.out<br>> 0<br>> 1<br>> 2<br>> 3<br>> <br>> <br>> Kelvin<br>> <br>> <br>> <br>> <br>> <br>> <br>> <br>> <br>> _______________________________________________<br>> Openmp-dev mailing list<br>> Openmp-dev@lists.llvm.org<br>> </span></tt><a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev"><tt><span style=" font-size:10pt">https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev</span></tt></a><tt><span style=" font-size:10pt"> <br>> <br>> <br>> <br>> <br>> <br>> <br>> _______________________________________________<br>> Openmp-dev mailing list<br>> Openmp-dev@lists.llvm.org<br>> </span></tt><a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev"><tt><span style=" font-size:10pt">https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev</span></tt></a><tt><span style=" font-size:10pt"><br>> <br><br></span></tt><br><br><BR>