[llvm-bugs] [Bug 49816] New: target nowait doesn't respect taskwait when helper threads are used

[via llvm-bugs]

https://bugs.llvm.org/show_bug.cgi?id=49816

            Bug ID: 49816
           Summary: target nowait doesn't respect taskwait when helper
                    threads are used
           Product: OpenMP
           Version: unspecified
          Hardware: PC
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P
         Component: Runtime Library
          Assignee: unassignedbugs at nondot.org
          Reporter: xw111luoye at gmail.com
                CC: llvm-bugs at lists.llvm.org

Use https://github.com/ye-luo/qmcpack/tree/debug-task-race

```
cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DQMC_MPI=0 -D
ENABLE_CUDA=1 -D CUDA_ARCH=sm_70 -D CUDA_HOST_COMPILER=`which gcc` -D
ENABLE_OFFLOAD=ON -D USE_OBJECT_TARGET=ON -DOFFLOAD_ARCH=sm_70 .. ; make -j32
qmcpack
LIBOMPTARGET_DEBUG=1 ctest -R deterministic-diamondC_1x1x1_pp-param-grad
grep "stride 16 add\|Moving 512 bytes" Testing/Temporary/LastTest.log
```

The run should have two host to device transfer and then two device to host
transfer. However with helper threads, the H2D transfer happens when closing
the parallel region.
```
stride 16 addr 0x33c5040 size(bytes) 512
stride 16 addr 0x3416d40 size(bytes) 512

Libomptarget --> Moving 512 bytes (tgt:0x00007f53d5005880) ->
(hst:0x00000000033c5040)
Libomptarget --> Moving 512 bytes (tgt:0x00007f53d5006e80) ->
(hst:0x0000000003416d40)
Libomptarget --> Moving 512 bytes (hst:0x0000000003416d40) ->
(tgt:0x00007f53d5006e80)
Libomptarget --> Moving 512 bytes (hst:0x00000000033c5040) ->
(tgt:0x00007f53d5005880)
```

Then turn off helper threads, the behavior is expected.
```
LIBOMPTARGET_DEBUG=1 LIBOMP_USE_HIDDEN_HELPER_TASK=OFF ctest -R
deterministic-diamondC_1x1x1_pp-param-grad
grep "stride 16 add\|Moving 512 bytes" Testing/Temporary/LastTest.log
stride 16 addr 0x3baf080 size(bytes) 512
Libomptarget --> Moving 512 bytes (hst:0x0000000003baf080) ->
(tgt:0x00007f15b5005880)
stride 16 addr 0x3b74e00 size(bytes) 512
Libomptarget --> Moving 512 bytes (hst:0x0000000003b74e00) ->
(tgt:0x00007f15b5006e80)

Libomptarget --> Moving 512 bytes (tgt:0x00007f15b5005880) ->
(hst:0x0000000003baf080)
Libomptarget --> Moving 512 bytes (tgt:0x00007f15b5006e80) ->
(hst:0x0000000003b74e00)
```

H2D is launched here
https://github.com/ye-luo/qmcpack/blob/09ea73cef59b99b1cfb440b9e453536f300274d9/src/QMCWaveFunctions/Fermion/DiracDeterminantBatched.cpp#L911
Multiple taskwait for example:
https://github.com/ye-luo/qmcpack/blob/09ea73cef59b99b1cfb440b9e453536f300274d9/src/QMCWaveFunctions/Fermion/DiracDeterminantBatched.cpp#L916

Note: ignore the ctest reported failure because I modified the test input just
to make the reproducer run easier.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20210402/b228feed/attachment.html>