[Openmp-dev] Failed to read local memory on device

Sun Feb 2 19:41:09 PST 2020

@Alexey

Could you please reply to the list as well. It is way more difficult to
follow the conversation this way.

On 02/03, Itaru Kitayama via Openmp-dev wrote:
> Alexey,
> Am still seeing the issue on POWER8 at JSC:
> [New Thread 0x10008b9ff180 (LWP 79809)]
> [Switching focus to CUDA kernel 0, grid 2, block (0,0,0), thread (0,0,0),
> device 0, sm 0, warp 1, lane 0]
> 0x00000000145f0868 in
> __omp_offloading_3b_8077e6c5__ZN4nest20EventDeliveryManager15deliver_events_INS_9SpikeDataEEEbiRKSt6vectorIT_SaIS4_EE_l666<<<(1,1,1),(128,1,1)>>>
> (
>     nranks=<error reading variable: Error: Failed to read local memory at
> address 0xc8f0308 on device 0 sm 0 warp 1 lane 0,
> error=CUDBG_ERROR_INVALID_MEMORY_ACCESS(0x8).
> >,
>     vla=<error reading variable: Error: Failed to read local memory at
> address 0xc8f0310 on device 0 sm 0 warp 1 lane 0,
> error=CUDBG_ERROR_INVALID_MEMORY_ACCESS(0x8).
> >,
>     r_buf=<error reading variable: Error: Failed to read local memory at
> address 0xc8f0318 on device 0 sm 0 warp 1 lane 0,
> error=CUDBG_ERROR_INVALID_MEMORY_ACCESS(0x8).
> >,
>     send_recv_count_spike_data_per_rank=<error reading variable: Error:
> Failed to read local memory at address 0xc8f0320 on device 0 sm 0 warp 1
> lane 0, error=CUDBG_ERROR_INVALID_MEMORY_ACCESS(0x8).
> >,
>     are_others_completed=<error reading variable: Error: Failed to read
> local memory at address 0xc8f0328 on device 0 sm 0 warp 1 lane 0,
> error=CUDBG_ERROR_INVALID_MEMORY_ACCESS(0x8).
> >,
>     spike_data=<error reading variable: Error: Failed to read local memory
> at address 0xc8f0330 on device 0 sm 0 warp 1 lane 0,
> error=CUDBG_ERROR_INVALID_MEMORY_ACCESS(0x8).
> >,
>     tid=<error reading variable: Error: Failed to read local memory at
> address 0xc8f0338 on device 0 sm 0 warp 1 lane 0,
> error=CUDBG_ERROR_INVALID_MEMORY_ACCESS(0x8).
> >,
>     se=<error reading variable: Error: Failed to read local memory at
> address 0xc8f0340 on device 0 sm 0 warp 1 lane 0,
> error=CUDBG_ERROR_INVALID_MEMORY_ACCESS(0x8).
> >,
>     vla=<error reading variable: Error: Failed to read local memory at
> address 0xc8f0310 on device 0 sm 0 warp 1 lane 0,
> error=CUDBG_ERROR_INVALID_MEMORY_ACCESS(0x8).
> >,
>     prepared_timestamps=<error reading variable: Error: Failed to read
> local memory at address 0xc8f0350 on device 0 sm 0 warp 1 lane 0,
> error=CUDBG_ERROR_INVALID_MEMORY_ACCESS(0x8).
> >,
> 
> How do I work-around this?
> Thanks,
> Itaru.
> 
> On Fri, Jan 17, 2020 at 12:46 PM Alexey Bataev <a.bataev at outlook.com> wrote:
> 
> > There is a small issue with the debig info, will try to fix it soon. You
> > can make a single step and after that you should be in the context of the
> > function with correct debug info. (such functions has suffix _debug__).
> >
> > Best regards,
> > Alexey Bataev
> >
> > 16 янв. 2020 г., в 15:38, Itaru Kitayama <itaru.kitayama at gmail.com>
> > написал(а):
> >
> > 
> > Sure, if I code it in a sample code that works (offloaded properly), as
> > expected.
> >
> > On Fri, Jan 17, 2020 at 10:37 AM Itaru Kitayama <itaru.kitayama at gmail.com>
> > wrote:
> >
> >> In CUDA-GDB.
> >>
> >> On Fri, Jan 17, 2020 at 10:34 AM Alexey Bataev <a.bataev at outlook.com>
> >> wrote:
> >>
> >>> This is what you see in cuda-gdb? Or at the runtime when you run the
> >>> program directly?
> >>>
> >>> -------------
> >>> Best regards,
> >>> Alexey Bataev
> >>>
> >>> 16.01.2020 3:22 PM, Itaru Kitayama via Openmp-dev пишет:
> >>>
> >>> Hi,
> >>> Adding this small loop to my app:
> >>>
> >>> #pragma omp target parallel for map(to: nranks)
> >>>         for (int i=0;i<0;i++) {
> >>>         printf("%d\n", nranks);
> >>>         }
> >>>
> >>> leads to a run-time error on device. See below:
> >>>
> >>> [New Thread 0x10008b9ff180 (LWP 103236)]
> >>> [Switching focus to CUDA kernel 0, grid 2, block (0,0,0), thread
> >>> (0,0,0), device 0, sm 0, warp 0, lane 0]
> >>> 0x0000000014588068 in
> >>> __omp_offloading_3b_801278cf__ZN4nest20EventDeliveryManager15deliver_events_INS_9SpikeDataEEEbiRKSt6vectorIT_SaIS4_EE_l666<<<(1,1,1),(128,1,1)>>>
> >>> (
> >>>     nranks=<error reading variable: Error: Failed to read local memory
> >>> at address 0xc8ee158 on device 0 sm 0 warp 0 lane 0,
> >>> error=CUDBG_ERROR_INVALID_MEMORY_ACCESS(0x8).
> >>> >) at
> >>> /p/project/cpcp0/pcp0151/projects/nest-simulator/nestkernel/event_delivery_manager.cpp:667
> >>> 667 for (int i=0;i<0;i++) {
> >>>
> >>> I am using the latest trunk Clang for this.
> >>>
> >>> _______________________________________________
> >>> Openmp-dev mailing listOpenmp-dev at lists.llvm.orghttps://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev
> >>>
> >>>

> _______________________________________________
> Openmp-dev mailing list
> Openmp-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev

-- 

Johannes Doerfert
Researcher

Argonne National Laboratory
Lemont, IL 60439, USA

jdoerfert at anl.gov
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 228 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/openmp-dev/attachments/20200202/dddccbef/attachment.sig>