<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/64624>64624</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            [OpenMP][libomptarget][DeviceRTL] Executable hangs when offloading to GPU
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
            new issue
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          tavakkoliamirmohammad
      </td>
    </tr>
</table>

<pre>
    I have a problem with using OpenMP target offloading on NVIDIA A100.

Here is my build step from the latest cloned LLVM.
```
mkdir build
cd build
cmake -G Ninja ../llvm -DLLVM_ENABLE_PROJECTS="clang;lld" -DLLVM_ENABLE_RUNTIMES="openmp" -DLLVM_TARGETS_TO_BUILD="Native;NVPTX;AMDGPU" -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DLLVM_ENABLE_LLD=ON -DLLVM_CCACHE_BUILD=ON
cmake --build . -j 128
```
The build finishes with the follwoing message

`[4047/4047] Completed 'runtimes'
`

Now using the example provided in the [Clang documentation](https://clang.llvm.org/docs/OffloadingDesign.html#offloading-example)
```
#include <complex>

using complex = std::complex<double>;

void zaxpy(complex *X, complex *Y, complex D, std::size_t N) {
#pragma omp target teams distribute parallel for
  for (std::size_t i = 0; i < N; ++i)
    Y[i] = D * X[i] + Y[i];
}

int main() {
  const std::size_t N = 1024;
  complex X[N], Y[N], D;
#pragma omp target data map(to:X[0 : N]) map(tofrom:Y[0 : N])
  zaxpy(X, Y, D, N);
}
```
Saved as `zaxpy.cpp`

Now compile command 

```
./bin/clang++ --libomptarget-nvptx-bc-path=./runtimes/runtimes-bins/openmp/libomptarget/DeviceRTL/ -fopenmp -fopenmp-targets=nvptx64 -O3 zaxpy.cpp -c
```
Also checking the .o file.
```
$  llvm-readelf -WS zaxpy.o
There are 24 section headers, starting at offset 0x90c8:

Section Headers:
  [Nr] Name Type            Address          Off    Size   ES Flg Lk Inf Al
 [ 0]                   NULL            0000000000000000 000000 000000 00 0   0  0
  [ 1] .strtab           STRTAB          0000000000000000 008d0c 0003b9 00      0   0  1
  [ 2] .text             PROGBITS 0000000000000000 000040 00057a 00  AX  0   0 16
  [ 3] .rela.text RELA            0000000000000000 0088d0 000318 18   I 23   2  8
  [ 4] .rodata.cst16     PROGBITS        0000000000000000 0005c0 000030 10  AM  0 0 16
  [ 5] .text.startup     PROGBITS        0000000000000000 0005f0 00000a 00  AX  0   0 16
  [ 6] .rela.text.startup RELA 0000000000000000 008be8 000018 18   I 23   5  8
  [ 7] .rodata.str1.1 PROGBITS        0000000000000000 0005fa 000017 01 AMS  0   0  1
  [ 8] .data.rel.ro      PROGBITS        0000000000000000 000618 000048 00  WA  0 0  8
  [ 9] .rela.data.rel.ro RELA            0000000000000000 008c00 000048 18   I 23   8  8
  [10] .rodata           PROGBITS 0000000000000000 000660 000001 00   A  0   0  1
  [11] .rodata.cst32 PROGBITS        0000000000000000 000670 000040 20  AM  0   0 16
  [12] .rodata.str1.16   PROGBITS        0000000000000000 0006b0 00003e 01 AMS  0 0 16
  [13] omp_offloading_entries PROGBITS   0000000000000000 0006ee 000020 00  WA  0   0  1
  [14] .relaomp_offloading_entries RELA 0000000000000000 008c48 000030 18   I 23  13  8
  [15] .init_array.0 INIT_ARRAY      0000000000000000 000710 000008 00  WA  0   0  8
  [16] .rela.init_array.0 RELA           0000000000000000 008c78 000018 18   I 23 15  8
  [17] .llvm.offloading  LLVM_OFFLOADING 0000000000000000 000718 007d18 00   E  0   0  8
  [18] .comment          PROGBITS 0000000000000000 008430 00006a 01  MS  0   0  1
  [19] .note.GNU-stack PROGBITS        0000000000000000 00849a 000000 00      0   0  1
  [20] .eh_frame         X86_64_UNWIND   0000000000000000 0084a0 000178 00   A  0 0  8
  [21] .rela.eh_frame    RELA            0000000000000000 008c90 000078 18   I 23  20  8
  [22] .llvm_addrsig     LLVM_ADDRSIG 0000000000000000 008d08 000004 00   E 23   0  1
  [23] .symtab SYMTAB          0000000000000000 008618 0002b8 18      1  15  8
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  R (retain), l (large), p (processor specific)
```
Now compiling with command 
```
./bin/clang++ --libomptarget-nvptx-bc-path=./runtimes/runtimes-bins/openmp/libomptarget/DeviceRTL/ -fopenmp -fopenmp-targets=nvptx64 -O3 zaxpy.cpp -o zaxpy
export LD_LIBRARY_PATH=/uufs/chpc.utah.edu/common/home/u1419116/projects/openmp/llvm-project/build/lib:$LD_LIBRARY_PATH
./zaxpy
```
The binary hangs indefinitely!

Also, the tests are failing using the command
```
cmake --build . --target check-openmp
```
</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJzUWN93or4S_2vSlzl6IKDiQx9QtOv9Wu1Ru9s-eSIEzRYIN4n9sX_9PUkQkdr99vV6dgtMks_MfGYmMCFSsn1B6S3qjVAvuiFHdeDiVpFX8vLCM0ZyJnJ-IHlOkpsdTz5uZ3AgrxQIlILvMprDG1MHOEpW7GFZ0uL-ARQRe6qAp2nGSaIHeAGLn7NoFkLoOk4XORFyQvv3BxUUmIT8A3ZHliUgFS0hFTwHdaCQEUWlgjjjBU1gPv95f1rdd6p_5jF_SZiwCFYQJxdPOXmh0LmDBSt-E-h2EZ5m2WsOnUhjbieLcDSfbB9Wy_9Mxps18iKEcZyRYo-8UZYlCOPW1NXjYjO7n1RTeUmLvGzM2oSru8lmvd0st6PH2Tyy0xZEsVeKvNHi58PmCXmj8D66e3i068b34T8TO3u7eX6YIC9a0YwSSVuqw_V6strMlgutfLmol4634-X9w2w-WSEvMsafh56ePg0iPEJ41MKeG1MNqBGPx-H4x6T2Ybm44LNjQ9aFzm9wcXA1MpsDrSKbsoLJA5U2ZXR0U55lb1xnSE6lJHvazAyN0Bv5jj9AeGouvQjGPC8zqmgCCA_EsVAspxLhwXlJA2HB36rM1MroO9FrdeK-soQmwAojR73R2JCV8PiY00IRxXiBehHCwUGpUiIvRHiK8NTQ1tV50-Vij_A04bFEeLqsMz2iup66B5VnCHvnCuhUyhEeXiUJYY8VcXZMKCBvHBsn35E3aXpjPanGAHkRSJVo27ywXjBO-HGXUb3SGzUXv3KWwB_yXn4gHNQYOHxCeAyN5-fmc6QfaiWS_aFbBQuEh4AGo9rwUpB9ToDn5anyFSW5hIRJJdjuqCiURJAsoxmkXNh1oG8B4aANz4xnDvJG5nYMC31rc5XV9AEAPKPeiOmc0AsibTw81SI8qsfPTAyiJiWsUJATViAcXLgEEPNCqs-OG0Wug_0aEWqqtOaFyZmxUXy6b2i_xlRCFIGclAgHiiMv1DAOIC8ECzCsB_WOiLzwuT3hZMgptk_WAqMaj020rhBwmX1r8koTIBJQ3zE43bgsr9SSdpZlVF9zUiTQrtYGpN5gd5rb6cVe08nYjuel9b5TvJbqvbOLOyVRB-RFetG5puvbzo4V-vm0yU6bIAhPI_rKYrrazBGeQie10-qbjp0nkRcZfX0fOksPaj-hE1_1IMwkh_hA45fTBtLlkLKMXn8BIewD6K2hIyhJaJZC59e60sLrnVBQIIIC9kHSWG8zcNCzhbSVRoTSyoh5eUqqwHkfOnGg07BB9bpa-qNaehoFvZUthM7_BckpbD5KCo1fmCSCSnkWLNNUX9bsj543WcM028P8BWZFCmFWgaLeCByN-fm3eJzPm89O6wftCzh6FoDTMBhcDd6VSiiya4CtN6tNOPoreJA4sZZ7u6EGt9Psf7epARsNir6rC_MfVsu70Wyzvm63by69ATHQ4dMJ2u03oT0DLWhGLP5qMg__hZIgSAy05wbgBgAwA-wBAAYImtC-heZ6j-jGUrn9S6v_Qnovtj54Drja-HtteNv0Xs1K12Tesfw-floF9e_k9C_JqdUYkq4xs6OBkbeY6bWYGTSZkUq4XfebZhMLPwDHhfB-fT1dAgNvwAXNuoLDt4npu9YBPzDE_Aot8ZfmD8-0NJV8J3XiU3JeEhRcanCdBkHfTfh-v4qpa2spvEqO67ay0sPfI2ZQVxWuM_JTxrj4c2T736V-V-U8bQS3jW_Klefl9vxptqWFEozKppKr-JQaOXYaof3Mj18H9ws1XyZ_7Ad10Z5j63qt2NqyZQVTWyIE-eg6MFvMNttwtQqfv6Zn4FbhDdrmX6A3SvZCRSs5r9o_uFK8bqt2XVu79hv63CCazm67nE7nyzCaLe6-cEErGCSudQEm112w5au_UWihvpf8ge9ZdvpEJw98sTO4tnILrmj3bvHYkYrEL99JzsAfksZb8KsXFbZ1Sw_bVOgX-On3FPS3fX_7uPg1W0RfaSDGA3cQNMq3RQ12z9FtKvnWzjO0DA0udh7c1oDr8G5JkgjJ9gbRhDeMotV6djW2QeLY5HH8U2zNxtYmyL5u5UeuvxXWz_f_-oVQbch4V5kNAC40s_If-gGKwzQj--aX1C_dm7wJpkzLhscQagHJMh5XgictoO80PtZz7rUop2J_EqxthyNYsZeVaKZFrEi5fT7pm2txxooX4CKhopq8tDqUILBc68Y1ptJ0gYL-98gETap5d3reXvBjWQk2WrCZry-VjKFq_vRnYL1Y933BsXgp-FtRibgWLdcgSxqzlJ1cnlhzTJ96ibzSI4Iq006ZuZlxiJy5KLWgcoGLC-hr39PnfkP7a04MWm3H_2_DwatuzRhO30suFMyj7Xw2WoWr5-1DuPlhzoumx2Oq7YgPZdw9KnLo0uSon3mec-3qgedUT3N9d-i6fYSnpeC_aawurdddSTWgWTLHYsYpc7Lht1XXfDasvHaowwoiPuBAir0EViQ01W8Mmn0g7DbbFd1I6QTQLZSiUknTAqXERvZ8PlOF96q-T-dNFcm2QetUrl6uvEluvWToDckNvXX7Q3cwcPrD_s3hNnV2OEl3vWTQ77m7HnZSL3ATL6b-MPXSeHDDbrGDPSdwXbfvDhy_O3Sxk6Y-CZKdk3jEQ75Dc8Ky-izohkl5pLd9v4_9m4zsaCbNkSrGBX0DM4jM3ngjbk04dse9RL6TMankGUUxlZmzWHuSqjv83ugi-YzknH-9CCZmCyK7jFaheDvQonn4qjjcPTzeHEV2e3mctWfqcNzpN2WVJJ9zxViuk8l49r8AAAD__4il7uw">