<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<div>
<div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" dir="ltr">
Probably in addition to printing things out, cuda-gdb would be the best tool here.</div>
</div>
<div id="ms-outlook-mobile-signature">
<div><br>
</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
Regards,</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
Shilei</div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Itaru Kitayama <itaru.kitayama@gmail.com><br>
<b>Sent:</b> Saturday, December 11, 2021 6:52:02 PM<br>
<b>To:</b> Shilei Tian <tianshilei1992@gmail.com><br>
<b>Cc:</b> openmp-dev <openmp-dev@lists.llvm.org><br>
<b>Subject:</b> Re: [Openmp-dev] warning: Cuda API error detected: cuModuleGetGlobal_v2 returned (0x1f4)</font>
<div> </div>
</div>
<div>
<div dir="auto">Shilei,</div>
<div dir="auto"><br>
</div>
<div dir="auto">I tried to debug the hang with CUDA-gdb, though not sure if I was using it correctly. If there are other general approaches, would you let us know?</div>
<div><br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Sun, Dec 12, 2021 at 8:38 Shilei Tian <<a href="mailto:tianshilei1992@gmail.com">tianshilei1992@gmail.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div lang="EN-US" style="word-wrap:break-word">
<div class="x_m_7553383898060825560WordSection1">
<p class="x_MsoNormal">If the function call is to get the device environment variable from the image, that could return CUDA_ERROR_NOT_FOUND (500). However, it doesn't affect the execution of the program. The hang could be something else.<u></u><u></u></p>
<p class="x_MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="x_MsoNormal"><span style="color:black">-- <u></u><u></u></span></p>
<p class="x_MsoNormal"><span style="color:black">Regards,<u></u><u></u></span></p>
</div>
</div>
<p class="x_MsoNormal"><span style="color:black">Shilei Tian</span><u></u><u></u></p>
</div>
</div>
<div lang="EN-US" style="word-wrap:break-word">
<div class="x_m_7553383898060825560WordSection1">
<p class="x_MsoNormal"><u></u> <u></u></p>
<div style="border:none; border-top:solid #b5c4df 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt; color:black">From:
</span></b><span style="font-size:12.0pt; color:black">Openmp-dev <<a href="mailto:openmp-dev-bounces@lists.llvm.org" target="_blank">openmp-dev-bounces@lists.llvm.org</a>> on behalf of Itaru Kitayama via Openmp-dev <<a href="mailto:openmp-dev@lists.llvm.org" target="_blank">openmp-dev@lists.llvm.org</a>><br>
<b>Date: </b>Sunday, December 5, 2021 at 5:03 AM<br>
<b>To: </b>openmp-dev <<a href="mailto:openmp-dev@lists.llvm.org" target="_blank">openmp-dev@lists.llvm.org</a>><br>
<b>Subject: </b>[Openmp-dev] warning: Cuda API error detected: cuModuleGetGlobal_v2 returned (0x1f4)<u></u><u></u></span></p>
</div>
</div>
</div>
<div lang="EN-US" style="word-wrap:break-word">
<div class="x_m_7553383898060825560WordSection1">
<div>
<p class="x_MsoNormal"></p>
</div>
</div>
</div>
<div lang="EN-US" style="word-wrap:break-word">
<div class="x_m_7553383898060825560WordSection1">
<div>
<p class="x_MsoNormal">Hi,<br>
My offloading app runs correctly on host, but gets stuck (no progress)<br>
on the device.<br>
<br>
In a cuda gdb session I see:<br>
<br>
(cuda-gdb) r<br>
<br>
The program being debugged has been started already.<br>
<br>
Start it from the beginning? (y or n) y<br>
<br>
Starting program:<br>
/p/project/cjinb33/kitayama1/opt/nest-clang-offload/bin/nest<br>
hpc_benchmark.sli<br>
<br>
warning: Cannot parse .gnu_debugdata section; LZMA support was<br>
disabled at compile time<br>
<br>
[Thread debugging using libthread_db enabled]<br>
<br>
Using host libthread_db library "/usr/lib64/libthread_db.so.1".<br>
<br>
warning: File "/p/software/jurecadc/stages/2020/software/GCCcore/10.3.0/lib64/libstdc++.<a href="http://so.6.0.28-gdb.py" target="_blank">so.6.0.28-gdb.py</a>"<br>
auto-loading has been declined by your `auto-load safe-path' set to<br>
"$debugdir:$datadir/auto-load".<br>
<br>
warning: Cannot parse .gnu_debugdata section; LZMA support was<br>
disabled at compile time<br>
<br>
warning: Cannot parse .gnu_debugdata section; LZMA support was<br>
disabled at compile time<br>
<br>
OMP: Warning #249: OMP_TARGET_OFFLOAD: syntax error, using DEFAULT.<br>
<br>
warning: Cannot parse .gnu_debugdata section; LZMA support was<br>
disabled at compile time<br>
<br>
[Detaching after fork from child process 27044]<br>
<br>
[New Thread 0x15554a92d700 (LWP 27050)]<br>
<br>
[New Thread 0x15554a12c700 (LWP 27051)]<br>
<br>
virtual void nest::SimulationManager::initialize() Map this pointer at<br>
0x15552e1021a8<br>
<br>
warning: Cuda API error detected: cuModuleGetGlobal_v2 returned (0x1f4)<br>
<br>
<br>
warning: Cuda API error detected: cuModuleGetGlobal_v2 returned (0x1f4)<br>
<br>
<br>
warning: Cuda API error detected: cuModuleGetGlobal_v2 returned (0x1f4)<br>
<br>
<br>
warning: Cuda API error detected: cuModuleGetGlobal_v2 returned (0x1f4)<br>
<br>
<br>
warning: Cuda API error detected: cuModuleGetGlobal_v2 returned (0x1f4)<br>
<br>
<br>
warning: Cuda API error detected: cuModuleGetGlobal_v2 returned (0x1f4)<br>
<br>
<br>
warning: Cuda API error detected: cuModuleGetGlobal_v2 returned (0x1f4)<br>
<br>
Should I be concerned about the API error?<br>
</p>
</div>
</div>
</div>
<div lang="EN-US" style="word-wrap:break-word">
<div class="x_m_7553383898060825560WordSection1">
<div>
<p class="x_MsoNormal">_______________________________________________<br>
Openmp-dev mailing list<br>
<a href="mailto:Openmp-dev@lists.llvm.org" target="_blank">Openmp-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev</a><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>