<div dir="ltr"><div class="gmail_default" style="font-family:courier new,monospace">Thanks for the explanation Alexey.</div><div class="gmail_default" style="font-family:courier new,monospace"><br></div><div class="gmail_default" style="font-family:courier new,monospace">Chris<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 6, 2018 at 2:47 PM, Alexey Bataev <span dir="ltr"><<a href="mailto:a.bataev@hotmail.com" target="_blank">a.bataev@hotmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
Hi Christopher, we aware of this problem. LLVM performs some aggressive optimizations at O3, which are not compatible with NVPTX target. Clang-ykt is based on llvm 4.0, which does not have these optimizations. <br>
<br>
<div id="m_5393275610584685607AppleMailSignature">Best regards,
<div>Alexey Bataev</div>
</div>
<div><br>
6 авг. 2018 г., в 17:28, Christopher Daley via Openmp-dev <<a href="mailto:openmp-dev@lists.llvm.org" target="_blank">openmp-dev@lists.llvm.org</a>> написал(а):<br>
<br>
</div><div><div class="h5">
<blockquote type="cite">
<div>
<div dir="ltr">
<div class="gmail_default" style="font-family:courier new,monospace">Hello all,<br>
<br>
I encounter a deadlock when using -O3 and -Ofast optimization in OpenMP target offload regions. -O2 optimization does not deadlock. I am using the 7.0.0 release branch, specifically commit e7966c0 from Aug 2.<br>
<br>
The last 5 lines of the OpenMP debug stamps are<br>
"""<br>
Libomptarget --> Launching target execution __omp_offloading_34_3b44cbf_<wbr>main_l67 with pointer 0x000000003d6a11a0 (index=0).<br>
Target CUDA RTL --> Setting CUDA threads per block to default 128<br>
Target CUDA RTL --> Using default number of teams 128<br>
Target CUDA RTL --> Launch kernel with 128 blocks and 128 threads<br>
Target CUDA RTL --> Launch of entry point at 0x000000003d6a11a0 successful!<br>
"""<br>
<br>
The stack trace shows that a thread is waiting in cuCtxSynchronize. The gdb stack is<br>
"""<br>
(gdb) thread apply all bt<br>
<br>
Thread 3 (Thread 0x200007b0f180 (LWP 31472)):<br>
#0 0x000020000039b31c in accept4 () from /lib64/libc.so.6<br>
#1 0x0000200000aa469c in ?? () from /usr/lib64/nvidia/libcuda.so.1<br>
#2 0x0000200000a8f13c in ?? () from /usr/lib64/nvidia/libcuda.so.1<br>
#3 0x0000200000aa5d7c in ?? () from /usr/lib64/nvidia/libcuda.so.1<br>
#4 0x0000200000238af4 in start_thread () from /lib64/libpthread.so.0<br>
#5 0x0000000000000000 in ?? ()<br>
<br>
Thread 2 (Thread 0x20000821f180 (LWP 31473)):<br>
#0 0x0000200000387ad8 in poll () from /lib64/libc.so.6<br>
#1 0x0000200000aa2f30 in ?? () from /usr/lib64/nvidia/libcuda.so.1<br>
#2 0x0000200000b299f4 in ?? () from /usr/lib64/nvidia/libcuda.so.1<br>
#3 0x0000200000aa5d7c in ?? () from /usr/lib64/nvidia/libcuda.so.1<br>
#4 0x0000200000238af4 in start_thread () from /lib64/libpthread.so.0<br>
Backtrace stopped: previous frame inner to this frame (corrupt stack?)<br>
<br>
Thread 1 (Thread 0x2000000463f0 (LWP 31459)):<br>
#0 0x00002000002419d4 in do_futex_wait () from /lib64/libpthread.so.0<br>
#1 0x0000200000241aec in __new_sem_wait_slow.constprop.<wbr>0 () from /lib64/libpthread.so.0<br>
#2 0x0000200000aa39a0 in ?? () from /usr/lib64/nvidia/libcuda.so.1<br>
#3 0x000020000097cba0 in ?? () from /usr/lib64/nvidia/libcuda.so.1<br>
#4 0x0000200000b638b4 in ?? () from /usr/lib64/nvidia/libcuda.so.1<br>
#5 0x0000200000a38ffc in ?? () from /usr/lib64/nvidia/libcuda.so.1<br>
#6 0x000020000094de18 in ?? () from /usr/lib64/nvidia/libcuda.so.1<br>
#7 0x0000200000adccec in cuCtxSynchronize () from /usr/lib64/nvidia/libcuda.so.1<br>
#8 0x0000200000835a74 in __tgt_rtl_run_target_team_<wbr>region (device_id=1031115376, tgt_entry_ptr=0x0, tgt_args=0x3cd8d688, tgt_offsets=0x3cd8d3f0, arg_num=0, team_num=-358104080,
<br>
thread_limit=32767, loop_tripcount=1140852866)<br>
at /autofs/nccs-svm1_home1/<wbr>cdaley/Repos/software_<wbr>packages/llvm-openmp/llvm-<wbr>openmp-build.htzoek/openmp/<wbr>libomptarget/plugins/cuda/src/<wbr>rtl.cpp:735<br>
Backtrace stopped: previous frame inner to this frame (corrupt stack?)<br>
"""<br>
<br>
I can reproduce the error on two platforms: one with P9+V100 another with Intel-Haswell+V100. Also, it does not deadlock when using Clang-ykt.<br>
<br>
The deadlock can be reproduced in the test kernel at <a href="https://bitbucket.org/crpl_cisc/sollve_vv/src/master/tests/application_kernels/mmm_target_parallel_for_simd.c" target="_blank">
https://bitbucket.org/crpl_<wbr>cisc/sollve_vv/src/master/<wbr>tests/application_kernels/mmm_<wbr>target_parallel_for_simd.c</a> when using the following compiler flags: clang -O3 -fopenmp -fopenmp-targets=nvptx64-<wbr>nvidia-cuda mmm_target_parallel_for_simd.c -o mmm_target_parallel_for_simd.<br>
<br>
Am I doing anything obviously wrong? Can you also reproduce the deadlock?<br>
<br>
Thanks for any help,<br>
Chris<br>
<br>
</div>
</div>
</div>
</blockquote>
</div></div><blockquote type="cite">
<div><span>______________________________<wbr>_________________</span><br>
<span>Openmp-dev mailing list</span><br>
<span><a href="mailto:Openmp-dev@lists.llvm.org" target="_blank">Openmp-dev@lists.llvm.org</a></span><br>
<span><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/openmp-dev</a></span><br>
</div>
</blockquote>
</div>
</blockquote></div><br></div>