[cfe-users] Fwd: Offload to GPU fails due to undefined references
Cristobal Ortega via cfe-users
cfe-users at lists.llvm.org
Tue Oct 9 00:41:20 PDT 2018
Hello,
I'm trying to compile an HPC application (LULESH) with offloading to a
CUDA GPU.
This is the application:
https://github.com/LLNL/LULESH/tree/2.0.2-dev/omp_4.0
I compiled it with success with GCC and XLC.
I can compile and run codes that offload to the GPU with clang for
simple applications.
So, going into the details of the issue.
I try to compile with the following line:
mpic++ -v -o lulesh2.0 lulesh.cc lulesh-comm.cc lulesh-viz.cc
lulesh-util.cc lulesh-init.cc -O0 -fopenmp=libomp
-fopenmp-targets=nvptx64-nvidia-cuda -DUSE_MPI=1 -lstdc++
When compiling with clang I get the following error:
"clang-7: error: nvlink command failed with exit code 255 (use -v to see
invocation)
make: *** [lulesh2.0] Error 255"
This error is caused because 'nvlink' gets several Undefined references
to different functions (GCC and XLC don't encounter this problem)
The output generated when compiling with clang (due to its size, only the relevant text is attached, let me know if you need the full output)
clang information (with MPI wrapper):
$ mpic++ -v
Version 6
clang version 7.0.0 (tags/RELEASE_700/final)
Target: powerpc64le-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/user/pkg/clang/7.0.0/bin
Found candidate GCC installation:
/apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0
Selected GCC installation:
/apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0
Candidate multilib: .;@m64
Selected multilib: .;@m64
Found CUDA installation: /usr/local/cuda-9.2, version 9.2
Also, using openmpi 3.0.0
Hopefully somebody has an idea on what's going on here.
If you need any more information to find the issue, let me know.
Thank you,
-Cristobal
http://bsc.es/disclaimer
-------------- next part --------------
"/home/user/gc/pkg/clang/7.0.0/bin/clang-7" -cc1 -triple nvptx64-nvidia-cuda -aux-triple powerpc64le-unknown-linux-gnu -S -disable-free -main-file-name lulesh-init.cc -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -no-integrated-as -fuse-init-array -mlink-cuda-bitcode /usr/local/cuda-9.2/nvvm/libdevice/libdevice.10.bc -target-feature +ptx61 -mlink-cuda-bitcode /home/user/pkg/llvm/7.0.0/lib/libomptarget-nvptx-sm_70.bc -target-cpu sm_70 -dwarf-column-info -debugger-tuning=gdb -v -resource-dir /home/user/pkg/clang/7.0.0/lib/clang/7.0.0 -D USE_MPI=1 -I /apps/OPENMPI/3.0.0/GCC/include/openmpi -I /apps/OPENMPI/3.0.0/GCC/include -c-isystem /apps/OPENMPI/3.0.0/GCC/include -c-isystem /usr/local/cuda-9.2/extras/CUPTI/include -c-isystem /usr/local/cuda-9.2/nvvm/include -c-isystem /usr/local/cuda-9.2/include -c-isystem /home/user/pkg/gcc/8.2.0/include -c-isystem /home/user/pkg/clang/7.0.0/include -c-isystem /home/user/pkg/llvm/7.0.0/include -cxx-isystem /apps/OPENMPI/3.0.0/GCC/include -cxx-isystem /usr/local/cuda-9.2/extras/CUPTI/include -cxx-isystem /usr/local/cuda-9.2/nvvm/include -cxx-isystem /usr/local/cuda-9.2/include -cxx-isystem /home/user/pkg/gcc/8.2.0/include -cxx-isystem /home/user/pkg/clang/7.0.0/include -cxx-isystem /home/user/pkg/llvm/7.0.0/include -internal-isystem /apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0/../../../../include/c++/7.3.0 -internal-isystem /apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0/../../../../include/c++/7.3.0/powerpc64le-unknown-linux-gnu -internal-isystem /apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0/../../../../include/c++/7.3.0/backward -internal-isystem /apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0/../../../../include/c++/7.3.0 -internal-isystem /apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0/../../../../include/c++/7.3.0/powerpc64le-unknown-linux-gnu -internal-isystem /apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0/../../../../include/c++/7.3.0/backward -internal-isystem /usr/local/include -internal-isystem /home/user/pkg/clang/7.0.0/lib/clang/7.0.0/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /home/user/pkg/clang/7.0.0/lib/clang/7.0.0/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -O0 -fdeprecated-macro -fno-dwarf-directory-asm -fdebug-compilation-dir /home/user/LULESH/omp_4.0 -ferror-limit 19 -fmessage-length 0 -fopenmp -pthread -fobjc-runtime=gcc -fcxx-exceptions -fexceptions -fdiagnostics-show-option -o /scratch/tmp/39058/lulesh-init-b05765.s -x c++ lulesh-init.cc -fopenmp-is-device -fopenmp-host-ir-file-path /scratch/tmp/39058/lulesh-init-39deab.bc
clang -cc1 version 7.0.0 based upon LLVM 7.0.0 default target powerpc64le-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/apps/OPENMPI/3.0.0/GCC/include"
as it is a non-system directory that duplicates a system directory
ignoring duplicate directory "/apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0/../../../../include/c++/7.3.0"
ignoring duplicate directory "/apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0/../../../../include/c++/7.3.0/powerpc64le-unknown-linux-gnu"
ignoring duplicate directory "/apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0/../../../../include/c++/7.3.0/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/user/gc/pkg/clang/7.0.0/lib/clang/7.0.0/include"
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
/apps/OPENMPI/3.0.0/GCC/include/openmpi
/apps/OPENMPI/3.0.0/GCC/include
/usr/local/cuda-9.2/extras/CUPTI/include
/usr/local/cuda-9.2/nvvm/include
/usr/local/cuda-9.2/include
/home/user/gc/pkg/gcc/8.2.0/include
/home/user/gc/pkg/clang/7.0.0/include
/home/user/gc/pkg/llvm/7.0.0/include
/apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0/../../../../include/c++/7.3.0
/apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0/../../../../include/c++/7.3.0/powerpc64le-unknown-linux-gnu
/apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0/../../../../include/c++/7.3.0/backward
/usr/local/include
/home/user/gc/pkg/clang/7.0.0/lib/clang/7.0.0/include
/usr/include
End of search list.
"/usr/local/cuda-9.2/bin/ptxas" -m64 -O0 -v --gpu-name sm_70 --output-file /scratch/tmp/39058/lulesh-init-2d7672.cubin /scratch/tmp/39058/lulesh-init-b05765.s -c
ptxas info : 4737 bytes gmem
ptxas info : Function properties for _ZN3MPI9IntercommC2EP19ompi_communicator_t
32 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for _ZN3MPI9Comm_NullC2EP19ompi_communicator_t
16 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZN3MPI4CommC2EP19ompi_communicator_t
32 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for _ZN3MPI8DatatypeC2EP15ompi_datatype_t
16 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZN3MPI8CartcommD1Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI6StatusD2Ev
8 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZN3MPI8Grequest8CompleteEv
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZL17getMasterThreadIdv
8 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI10ErrhandlerD2Ev
8 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZN3MPI3WinD0Ev
24 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for _ZNK3MPI9Intercomm5CloneEv
48 bytes stack frame, 32 bytes spill stores, 32 bytes spill loads
ptxas info : Function properties for _ZN3MPI7RequestD1Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN42omptarget_nvptx_SimpleThreadPrivateContext8AllocateEm
24 bytes stack frame, 20 bytes spill stores, 20 bytes spill loads
ptxas info : Function properties for _ZN3MPI8CartcommD2Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI10Errhandler4FreeEv
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI2OpD1Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI8PrequestD0Ev
24 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for _Z10SafeMallocmPKc
16 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for _ZN3MPI9GraphcommD0Ev
24 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for _ZNK3MPI8Datatype17Create_contiguousEi
56 bytes stack frame, 36 bytes spill stores, 36 bytes spill loads
ptxas info : Function properties for _ZN3MPI9Comm_NullD0Ev
24 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for _ZN3MPI8GrequestD2Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI4CommD1Ev
8 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _Z8SafeFreePvPKc
8 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI9IntercommD2Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI9IntracommD0Ev
24 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for _ZN3MPI8DatatypeC1EP15ompi_datatype_t
24 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI8GrequestD1Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZL13__assert_failPKcS0_jS0_
8 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI3WinD2Ev
8 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZNK3MPI4Comm4SendEPKviRKNS_8DatatypeEii
80 bytes stack frame, 36 bytes spill stores, 36 bytes spill loads
ptxas info : Function properties for _ZN3MPI9IntercommD1Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI6StatusD0Ev
24 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for __kmpc_data_sharing_push_stack
64 bytes stack frame, 64 bytes spill stores, 64 bytes spill loads
ptxas info : Function properties for _ZN3MPI4CommD2Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI10ErrhandlerD1Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI4InfoD0Ev
24 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for _ZN3MPI7Request4WaitERNS_6StatusE
24 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZL11getThreadIdv
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZNK3MPI8Cartcomm5CloneEv
48 bytes stack frame, 32 bytes spill stores, 32 bytes spill loads
ptxas info : Function properties for _ZN3MPI8DatatypeD1Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI7RequestD2Ev
8 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZN3MPI5GroupD0Ev
24 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for __kmpc_data_sharing_pop_stack
24 bytes stack frame, 20 bytes spill stores, 20 bytes spill loads
ptxas info : Function properties for _ZN3MPI8CartcommD0Ev
24 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for _ZNK3MPI8DatatypecvP15ompi_datatype_tEv
8 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZN3MPI5GroupD1Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZL13getNumThreadsv
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZN3MPI9IntracommD2Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZNK3MPI3Win14Set_errhandlerERKNS_10ErrhandlerE
32 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for _ZN3MPI4InfoD1Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI7RequestD0Ev
24 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for _ZL9getWarpIdv
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZN3MPI9GraphcommD2Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI9IntracommD1Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI8DatatypeD2Ev
8 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZN3MPI8Prequest5StartEv
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZL9getLaneIdv
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZNK3MPI9Graphcomm5CloneEv
48 bytes stack frame, 32 bytes spill stores, 32 bytes spill loads
ptxas info : Function properties for _ZN3MPI6StatusD1Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI4InfoD2Ev
8 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZN3MPI5GroupD2Ev
8 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZN3MPI4CommD0Ev
8 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZN3MPI9IntercommC1EP19ompi_communicator_t
24 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZNK3MPI9Intracomm4ScanEPKvPviRKNS_8DatatypeERKNS_2OpE
96 bytes stack frame, 44 bytes spill stores, 44 bytes spill loads
ptxas info : Function properties for _ZN3MPI8GrequestD0Ev
24 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for _Z10isSPMDModev
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZNK3MPI10ErrhandlercvP17ompi_errhandler_tEv
8 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZN3MPI3WinD1Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI9IntercommD0Ev
24 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for _ZNK3MPI6Status9Get_countERKNS_8DatatypeE
56 bytes stack frame, 36 bytes spill stores, 36 bytes spill loads
ptxas info : Function properties for _Z22isRuntimeUninitializedv
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZN3MPI9Comm_NullD2Ev
8 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _ZN3MPI10ErrhandlerD0Ev
24 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for _ZN3MPI4Info6DeleteEPKc
24 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI8PrequestD1Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZL14IsMasterThreadv
16 bytes stack frame, 12 bytes spill stores, 12 bytes spill loads
ptxas info : Function properties for _ZN3MPI9GraphcommD1Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI8DatatypeD0Ev
24 bytes stack frame, 16 bytes spill stores, 16 bytes spill loads
ptxas info : Function properties for _ZN3MPI9Comm_NullD1Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZN3MPI8PrequestD2Ev
16 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
ptxas info : Function properties for _ZNK3MPI5Group8Get_sizeEv
32 bytes stack frame, 24 bytes spill stores, 24 bytes spill loads
ptxas info : Function properties for _ZN42omptarget_nvptx_SimpleThreadPrivateContext10DeallocateEPv
8 bytes stack frame, 8 bytes spill stores, 8 bytes spill loads
"/usr/local/cuda-9.2/bin/nvlink" -o /scratch/tmp/39058/lulesh-56a1e7.out -v -arch sm_70 -L/apps/OPENMPI/3.0.0/GCC/lib -L/usr/local/cuda-9.2/extras/CUPTI/lib64 -L/usr/local/cuda-9.2/nvvm/lib64 -L/usr/local/cuda-9.2/lib64 -L/usr/lib64/nvidia/xorg -L/usr/lib64/nvidia -L/home/user/gc/pkg/llvm/7.0.0/lib -L/gpfs/apps/POWER9/LLVM/7.0.0/GCC/lib -L/home/user/pkg/clang/7.0.0/lib -lomptarget-nvptx /scratch/tmp/39058/lulesh-ab1515.cubin /scratch/tmp/39058/lulesh-comm-cd949d.cubin /scratch/tmp/39058/lulesh-viz-6ac5fb.cubin /scratch/tmp/39058/lulesh-util-526295.cubin /scratch/tmp/39058/lulesh-init-2d7672.cubin
nvlink error : Undefined reference to '_ZNK3MPI8Datatype13Create_vectorEiii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype14Create_indexedEiPKiS2_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype15Create_hindexedEiPKiPKl' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype14Create_hvectorEiil' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype20Create_indexed_blockEiiPKi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype14Create_resizedEll' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype8Get_sizeEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype10Get_extentERlS1_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype15Get_true_extentERlS1_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI8Datatype6CommitEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI8Datatype4FreeEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype4PackEPKviPviRiRKNS_4CommE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype6UnpackEPKviPviRiRKNS_4CommE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype9Pack_sizeEiRKNS_4CommE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype13Pack_externalEPKcPKviPvlRl' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype18Pack_external_sizeEPKci' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype15Unpack_externalEPKcPKvlRlPvi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype15Create_subarrayEiPKiS2_S2_i' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype13Create_darrayEiiiPKiS2_S2_S2_i' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype3DupEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI8Datatype11Delete_attrEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype8Get_attrEiPv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype12Get_contentsEiiiPiPlPS0_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype12Get_envelopeERiS1_S1_S1_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Datatype8Get_nameEPcRi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI8Datatype8Set_attrEiPKv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI8Datatype8Set_nameEPKc' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI6Status12Is_cancelledEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI6Status12Get_elementsERKNS_8DatatypeE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI6Status10Get_sourceEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI6Status10Set_sourceEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI6Status7Get_tagEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI6Status7Set_tagEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI6Status9Get_errorEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI6Status9Set_errorEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI6Status12Set_elementsERKNS_8DatatypeEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI6Status13Set_cancelledEb' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI7Request4WaitEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI7Request4TestERNS_6StatusE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI7Request4TestEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI7Request4FreeEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI7Request6CancelEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI7Request10Get_statusERNS_6StatusE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI7Request10Get_statusEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI5Group8Get_rankEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI5Group4InclEiPKi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI5Group4ExclEiPKi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI5Group10Range_inclEiPA3_Ki' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI5Group10Range_exclEiPA3_Ki' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI5Group4FreeEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm4RecvEPviRKNS_8DatatypeEiiRNS_6StatusE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm4RecvEPviRKNS_8DatatypeEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm5BsendEPKviRKNS_8DatatypeEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm5SsendEPKviRKNS_8DatatypeEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm5RsendEPKviRKNS_8DatatypeEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm5IsendEPKviRKNS_8DatatypeEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm6IbsendEPKviRKNS_8DatatypeEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm6IssendEPKviRKNS_8DatatypeEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm6IrsendEPKviRKNS_8DatatypeEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm5IrecvEPviRKNS_8DatatypeEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm6IprobeEiiRNS_6StatusE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm6IprobeEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm5ProbeEiiRNS_6StatusE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm5ProbeEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm9Send_initEPKviRKNS_8DatatypeEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm10Bsend_initEPKviRKNS_8DatatypeEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm10Ssend_initEPKviRKNS_8DatatypeEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm10Rsend_initEPKviRKNS_8DatatypeEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm9Recv_initEPviRKNS_8DatatypeEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm8SendrecvEPKviRKNS_8DatatypeEiiPviS5_iiRNS_6StatusE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm8SendrecvEPKviRKNS_8DatatypeEiiPviS5_ii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm16Sendrecv_replaceEPviRKNS_8DatatypeEiiiiRNS_6StatusE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm16Sendrecv_replaceEPviRKNS_8DatatypeEiiii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm9Get_groupEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm8Get_sizeEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm8Get_rankEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '__cxa_pure_virtual' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI4Comm4FreeEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm8Is_interEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm7BarrierEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm5BcastEPviRKNS_8DatatypeEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm6GatherEPKviRKNS_8DatatypeEPviS5_i' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm7GathervEPKviRKNS_8DatatypeEPvPKiS8_S5_i' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm7ScatterEPKviRKNS_8DatatypeEPviS5_i' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm8ScattervEPKvPKiS4_RKNS_8DatatypeEPviS7_i' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm9AllgatherEPKviRKNS_8DatatypeEPviS5_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm10AllgathervEPKviRKNS_8DatatypeEPvPKiS8_S5_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm8AlltoallEPKviRKNS_8DatatypeEPviS5_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm9AlltoallvEPKvPKiS4_RKNS_8DatatypeEPvS4_S4_S7_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm9AlltoallwEPKvPKiS4_PKNS_8DatatypeEPvS4_S4_S7_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm6ReduceEPKvPviRKNS_8DatatypeERKNS_2OpEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm9AllreduceEPKvPviRKNS_8DatatypeERKNS_2OpE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm14Reduce_scatterEPKvPvPiRKNS_8DatatypeERKNS_2OpE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI4Comm10DisconnectEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm8Get_nameEPcRi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI4Comm8Set_nameEPKc' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm12Get_topologyEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI4Comm5AbortEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI4Comm14Set_errhandlerERKNS_10ErrhandlerE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm14Get_errhandlerEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm8Set_attrEiPKv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Comm8Get_attrEiPv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI4Comm11Delete_attrEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI3Win14Get_errhandlerEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI3Win10AccumulateEPKviRKNS_8DatatypeEiliS5_RKNS_2OpE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI3Win8CompleteEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI3Win5FenceEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI3Win4FreeEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI3Win3GetEPKviRKNS_8DatatypeEiliS5_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI3Win9Get_groupEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI3Win4LockEiii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI3Win4PostERKNS_5GroupEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI3Win3PutEPKviRKNS_8DatatypeEiliS5_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI3Win5StartERKNS_5GroupEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI3Win4TestEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI3Win6UnlockEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI3Win4WaitEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI3Win15Call_errhandlerEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI3Win11Delete_attrEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI3Win8Get_nameEPcRi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI3Win8Set_attrEiPKv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI3Win8Set_nameEPKc' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Intracomm5CloneEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Intracomm6ExscanEPKvPviRKNS_8DatatypeERKNS_2OpE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Intracomm6CreateERKNS_5GroupE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Intracomm5SplitEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Intracomm16Create_intercommEiRKNS_4CommEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Intracomm11Create_cartEiPKiPKbb' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Intracomm12Create_graphEiPKiS2_b' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Intracomm6AcceptEPKcRKNS_4InfoEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Intracomm7ConnectEPKcRKNS_4InfoEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Intracomm5SpawnEPKcPS2_iRKNS_4InfoEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Intracomm5SpawnEPKcPS2_iRKNS_4InfoEiPi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI9Intracomm14Spawn_multipleEiPPKcPS3_PKiPKNS_4InfoEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI9Intracomm14Spawn_multipleEiPPKcPS3_PKiPKNS_4InfoEiPi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Cartcomm7Get_dimEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Cartcomm8Get_topoEiPiPbS1_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Cartcomm13Get_cart_rankEPKi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Cartcomm10Get_coordsEiiPi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Cartcomm5ShiftEiiRiS1_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Cartcomm3SubEPKb' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI8Cartcomm3MapEiPKiPKb' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Graphcomm8Get_dimsEPiS1_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Graphcomm8Get_topoEiiPiS1_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Graphcomm19Get_neighbors_countEi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Graphcomm13Get_neighborsEiiPi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Graphcomm3MapEiPKiS2_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Intercomm15Get_remote_sizeEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Intercomm16Get_remote_groupEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Intercomm5MergeEb' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Intercomm6CreateERKNS_5GroupE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI9Intercomm5SplitEii' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Info3DupEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI4Info4FreeEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Info3GetEPKciPc' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Info9Get_nkeysEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Info10Get_nthkeyEiPc' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI4Info12Get_valuelenEPKcRi' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI4Info3SetEPKcS2_' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI2OpD0Ev' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI2Op4FreeEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI2Op12Reduce_localEPKvPviRKNS_8DatatypeE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI2Op14Is_commutativeEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to 'MPI_Info_delete' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZdlPv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to 'MPI_Group_size' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to 'MPI_Get_count' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to 'MPI_Scan' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZNK3MPI2OpcvP9ompi_op_tEv' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to 'MPI_Comm_dup' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_Znwm' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI9GraphcommC1ERKP19ompi_communicator_t' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to 'MPI_Start' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to 'cbrt' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to 'MPI_Win_set_errhandler' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI2OpD2Ev' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to 'MPI_Errhandler_free' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to 'MPI_Type_contiguous' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to 'MPI_Grequest_complete' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to 'MPI_Send' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to 'MPI_Wait' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZN3MPI8CartcommC1ERKP19ompi_communicator_t' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZTVN10__cxxabiv117__class_type_infoE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink error : Undefined reference to '_ZTVN10__cxxabiv120__si_class_type_infoE' in '/scratch/tmp/39058/lulesh-ab1515.cubin'
nvlink info : 1039035300 bytes gmem
nvlink info : Function properties for '__omp_offloading_2f_6a44eea__Z22CalcKinematicsForElemsR6DomainPddi_l1897':
nvlink info : used 108 registers, 2280 stack, 1060 bytes smem, 504 bytes cmem[0], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_2f_6a44eea__ZL28CalcFBHourglassForceForElemsR6DomainPdS1_S1_S1_S1_S1_S1_dii_l1135':
nvlink info : used 56 registers, 768 stack, 1060 bytes smem, 440 bytes cmem[0], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_2f_6a44eea__ZL43ApplyAccelerationBoundaryConditionsForNodesR6Domain_l1471':
nvlink info : used 58 registers, 784 stack, 1060 bytes smem, 448 bytes cmem[0], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_2f_6a44eea__ZL15EvalEOSForElemsR6DomainPd_l2631':
nvlink info : used 98 registers, 1512 stack, 1060 bytes smem, 528 bytes cmem[0], 24 bytes cmem[2], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_2f_6a44eea__ZL23InitStressTermsForElemsR6DomainPdS1_S1_i_l334':
nvlink info : used 44 registers, 600 stack, 1060 bytes smem, 416 bytes cmem[0], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_2f_6a44eea__ZL20CalcPositionForNodesR6Domaindi_l1544':
nvlink info : used 50 registers, 688 stack, 1060 bytes smem, 432 bytes cmem[0], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_2f_6a44eea__ZL23IntegrateStressForElemsR6DomainPdS1_S1_S1_ii_l686':
nvlink info : used 56 registers, 768 stack, 1060 bytes smem, 440 bytes cmem[0], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_2f_6a44eea__ZL31CalcMonotonicQGradientsForElemsR6DomainPd_l2108':
nvlink info : used 82 registers, 1584 stack, 1060 bytes smem, 496 bytes cmem[0], 8 bytes cmem[2], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_2f_6a44eea__ZL17CalcForceForNodesR6Domain_l1387':
nvlink info : used 44 registers, 520 stack, 1060 bytes smem, 400 bytes cmem[0], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_2f_6a44eea__ZL28CalcFBHourglassForceForElemsR6DomainPdS1_S1_S1_S1_S1_S1_dii_l907':
nvlink info : used 116 registers, 2400 stack, 1060 bytes smem, 544 bytes cmem[0], 8 bytes cmem[2], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_2f_6a44eea__ZL24CalcAccelerationForNodesR6Domaini_l1435':
nvlink info : used 50 registers, 696 stack, 1060 bytes smem, 432 bytes cmem[0], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_2f_6a44eea__ZL28CalcMonotonicQRegionForElemsR6DomainPdd_l2303':
nvlink info : used 126 registers, 1520 stack, 1060 bytes smem, 568 bytes cmem[0], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_2f_6a44eea__ZL20CalcVelocityForNodesR6Domainddi_l1503':
nvlink info : used 56 registers, 760 stack, 1060 bytes smem, 440 bytes cmem[0], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_2f_6a44eea__ZL23IntegrateStressForElemsR6DomainPdS1_S1_S1_ii_l600':
nvlink info : used 108 registers, 1944 stack, 1060 bytes smem, 520 bytes cmem[0], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_2f_6a44eea__ZL20CalcLagrangeElementsR6DomainPd_l2025':
nvlink info : used 48 registers, 672 stack, 1060 bytes smem, 424 bytes cmem[0], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_2f_6a44eea__ZL28CalcHourglassControlForElemsR6DomainPdd_l1201':
nvlink info : used 94 registers, 1608 stack, 1060 bytes smem, 488 bytes cmem[0], 8 bytes cmem[2], 0 bytes lmem
"/home/user/gc/pkg/clang/7.0.0/bin/clang-7" -cc1 -triple powerpc64le-unknown-linux-gnu -emit-obj -mrelax-all -disable-free -main-file-name lulesh.cc -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -fuse-init-array -target-cpu ppc64le -mfloat-abi hard -target-abi elfv2 -dwarf-column-info -debugger-tuning=gdb -v -resource-dir /home/user/pkg/clang/7.0.0/lib/clang/7.0.0 -O0 -fdebug-compilation-dir /home/user/LULESH/omp_4.0 -ferror-limit 19 -fmessage-length 0 -fopenmp -pthread -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-option -o /scratch/tmp/39058/lulesh-749e62.o -x ir /scratch/tmp/39058/lulesh-248c31.bc -fopenmp-targets=nvptx64-nvidia-cuda -faddrsig
clang -cc1 version 7.0.0 based upon LLVM 7.0.0 default target powerpc64le-unknown-linux-gnu
"/home/user/gc/pkg/clang/7.0.0/bin/clang-7" -cc1 -triple powerpc64le-unknown-linux-gnu -emit-obj -mrelax-all -disable-free -main-file-name lulesh-comm.cc -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -fuse-init-array -target-cpu ppc64le -mfloat-abi hard -target-abi elfv2 -dwarf-column-info -debugger-tuning=gdb -v -resource-dir /home/user/pkg/clang/7.0.0/lib/clang/7.0.0 -O0 -fdebug-compilation-dir /home/user/LULESH/omp_4.0 -ferror-limit 19 -fmessage-length 0 -fopenmp -pthread -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-option -o /scratch/tmp/39058/lulesh-comm-39b085.o -x ir /scratch/tmp/39058/lulesh-comm-dff5f1.bc -fopenmp-targets=nvptx64-nvidia-cuda -faddrsig
clang -cc1 version 7.0.0 based upon LLVM 7.0.0 default target powerpc64le-unknown-linux-gnu
"/home/user/gc/pkg/clang/7.0.0/bin/clang-7" -cc1 -triple powerpc64le-unknown-linux-gnu -emit-obj -mrelax-all -disable-free -main-file-name lulesh-viz.cc -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -fuse-init-array -target-cpu ppc64le -mfloat-abi hard -target-abi elfv2 -dwarf-column-info -debugger-tuning=gdb -v -resource-dir /home/user/pkg/clang/7.0.0/lib/clang/7.0.0 -O0 -fdebug-compilation-dir /home/user/LULESH/omp_4.0 -ferror-limit 19 -fmessage-length 0 -fopenmp -pthread -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-option -o /scratch/tmp/39058/lulesh-viz-b361bc.o -x ir /scratch/tmp/39058/lulesh-viz-60d6e0.bc -fopenmp-targets=nvptx64-nvidia-cuda -faddrsig
clang -cc1 version 7.0.0 based upon LLVM 7.0.0 default target powerpc64le-unknown-linux-gnu
"/home/user/gc/pkg/clang/7.0.0/bin/clang-7" -cc1 -triple powerpc64le-unknown-linux-gnu -emit-obj -mrelax-all -disable-free -main-file-name lulesh-util.cc -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -fuse-init-array -target-cpu ppc64le -mfloat-abi hard -target-abi elfv2 -dwarf-column-info -debugger-tuning=gdb -v -resource-dir /home/user/pkg/clang/7.0.0/lib/clang/7.0.0 -O0 -fdebug-compilation-dir /home/user/LULESH/omp_4.0 -ferror-limit 19 -fmessage-length 0 -fopenmp -pthread -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-option -o /scratch/tmp/39058/lulesh-util-6d9d30.o -x ir /scratch/tmp/39058/lulesh-util-723abf.bc -fopenmp-targets=nvptx64-nvidia-cuda -faddrsig
clang -cc1 version 7.0.0 based upon LLVM 7.0.0 default target powerpc64le-unknown-linux-gnu
"/home/user/gc/pkg/clang/7.0.0/bin/clang-7" -cc1 -triple powerpc64le-unknown-linux-gnu -emit-obj -mrelax-all -disable-free -main-file-name lulesh-init.cc -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -fuse-init-array -target-cpu ppc64le -mfloat-abi hard -target-abi elfv2 -dwarf-column-info -debugger-tuning=gdb -v -resource-dir /home/user/pkg/clang/7.0.0/lib/clang/7.0.0 -O0 -fdebug-compilation-dir /home/user/LULESH/omp_4.0 -ferror-limit 19 -fmessage-length 0 -fopenmp -pthread -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-option -o /scratch/tmp/39058/lulesh-init-f86aa4.o -x ir /scratch/tmp/39058/lulesh-init-39deab.bc -fopenmp-targets=nvptx64-nvidia-cuda -faddrsig
clang -cc1 version 7.0.0 based upon LLVM 7.0.0 default target powerpc64le-unknown-linux-gnu
"/usr/bin/ld" --hash-style=gnu --no-add-needed --eh-frame-hdr -m elf64lppc -dynamic-linker /lib64/ld64.so.2 -o lulesh2.0 /lib/../lib64/crt1.o /lib/../lib64/crti.o /apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0/crtbegin.o -L/apps/OPENMPI/3.0.0/GCC/lib -L/apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0 -L/apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0/../../.. -L/home/user/gc/pkg/clang/7.0.0/bin/../lib -L/lib -L/usr/lib /scratch/tmp/39058/lulesh-749e62.o /scratch/tmp/39058/lulesh-comm-39b085.o /scratch/tmp/39058/lulesh-viz-b361bc.o /scratch/tmp/39058/lulesh-util-6d9d30.o /scratch/tmp/39058/lulesh-init-f86aa4.o -lstdc++ -rpath /apps/OPENMPI/3.0.0/GCC/lib --enable-new-dtags -lmpi_cxx -lmpi -L/apps/OPENMPI/3.0.0/GCC/lib -L/usr/local/cuda-9.2/extras/CUPTI/lib64 -L/usr/local/cuda-9.2/nvvm/lib64 -L/usr/local/cuda-9.2/lib64 -L/usr/lib64/nvidia/xorg -L/usr/lib64/nvidia -L/home/user/pkg/llvm/7.0.0/lib -L/gpfs/apps/POWER9/LLVM/7.0.0/GCC/lib -lstdc++ -lm -lomp -lomptarget -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc /apps/GCC/7.3.0-offload/lib/gcc/powerpc64le-unknown-linux-gnu/7.3.0/crtend.o /lib/../lib64/crtn.o -T /scratch/tmp/39058/lulesh2-118aff.lk
clang-7: error: nvlink command failed with exit code 255 (use -v to see invocation)
More information about the cfe-users
mailing list