<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/65811>65811</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            [libomptarget] segfault in amdgpu plugin
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
            new issue
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          ye-luo
      </td>
    </tr>
</table>

<pre>
    If assertion is not on
```
Core was generated by `/lustre/orion/mat151/scratch/yeluo/opt/qmcpack_frontier/build_frontier_aomp2023'.
Program terminated with signal SIGSEGV, Segmentation fault.

warning: Section `.reg-xstate/22288' in core file too small.
#0  0x00007fff6d3ad259 in llvm::omp::target::plugin::AMDGPUStreamTy::asyncActionCallback(long, void*) () from /ccs/proj/mat151/opt_frontier/aomp/20230907/aomp/lib/libomptarget.rtl.amdgpu.so
```
if assertion is on
```
qmcpack: /lustre/orion/csc372/scratch/cabarcasf/ums012/llvm17_git/llvm-project/openmp/libomptarget/plugins-nextgen/amdgpu/src/rtl.cpp:1038: static bool llvm::omp::target::plugin::AMDGPUStreamTy::asyncActionCallback(hsa_signal_value_t, void *): Assertion `Slot->Signal && "Invalid signal"' failed.
```

it is not a simple reproducer.
Need to build qmcpack using
```
cmake -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx \
-DCMAKE_C_FLAGS=--gcc-toolchain=/opt/cray/pe/gcc/12.2.0/snos -DCMAKE_CXX_FLAGS="--gcc-toolchain=/opt/cray/pe/gcc/12.2.0/snos -g" \
-DQMC_MIXED_PRECISION=ON -DENABLE_OFFLOAD=ON -DENABLE_CUDA=ON -DQMC_CUDA2HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx90a ..
``

Unpack [reproducer.zip](https://github.com/llvm/llvm-project/files/12563788/reproducer.zip)
```
curl -L -O -J https://anl.box.com/shared/static/g5ceycyjhb2b6segk7ibxup2hxnd77ih.h5
```
put the file `NiO-fcc-supertwist111-supershift000-S2.h5` at the top level of reproducer folder.

Then run all the `qmcpack.sub` job in `dmc-a*` folders.

The runs may pass, hang, segfault or fail at lapack Xgetrf.
Folders with `r1-t7` in its name can be used to run 1 MPI rank 7 thread each.
Others are 8 rank 7 threads runs for nodes with 4 MI250X GPUs.
</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJy0Vktzq7AV_jXy5gyMEAHshRe-tsl1m8RunHSy8wghQDdCopJI7P76joA8my46nc54sB7oO4_vfEdQa0WtOF-i5BdKNjPau0ab5YUHstezQpeX5a4Cai03TmgFwoLSDrRCeIPwCqV4-g3TtTYcXqmFmituqOMlFBfw2ySXvXWGI5JrI7RCJG-pi5IIkdwyQx1rEMkvXPbav9I5RPJ_tKyj7PlUGa2c4AaRvOiFLN8XTlS3HcEkRiQLRw8ORteGtuC4aYUaPHgVrgEfJZVw3F0ft9d_R2QNR163XDk6hFXRXroJYny-UqOEqlG8giNnw0soxaHhdXC2jjofCSFkPkckA6GA-dArITk4rcG2VMo3PBJjAHzGGOOsqqq0jGlJkoU_JeVLi-IVile67caBo6bmbhx3sq-FGser28314fHoDKftw2Vco_ai2Grwbk2lLCh7RmQutap9hC9alIisEFkAInP_VxndAiI5YxaRvDP6z2cedOc-59on1weJSYwXOPtYkaIYn7rtRm9D42RI27Lu-tDqH0tDfKui_1BBE-c-7T_VDLMszsiXmmG0oIZRWyGS963Fkd_2eY2yUy3cNAl8sJy5IUyu3uJ4j8CnY0i2DRQ_u5p7a2NE3pphiOQ-SNZ5miIcz72LvhAEg0Jr-f-hsrH0NJbu6YXKnp_cG7EwMuu9WL3nFaX4KLULULw9jgWPSIpICoiQnXqhUpSTEhAhvnArKiQvwx-pmHhzb5qnYEXbSQ6Gd0aXPeNmOnjHeQlOw6BOmCiE3nr9_ITMWvrMIdisb1d_3Z7Wp_X-9rC72d6jeNN2grGPraen75vnM6BkPeJ8IOQ3q-sjijdBUDMWOK0la6hP9-a9mzBDL55lX04183xGJCShb01WafvF5hscIuR_QawRIZ-9_dvt-nS7e9puTof77Xp33O3vULzZ30Gw2d6tft1sT_s8v9mvNt9W14-b1duSx_Bz8nt3eFsbHf-9O5xW9-vfu4ft-uHxfusDqKvzAlMIvzL8id5HNXCFkl-fWP2n6FCy8eXnXGd9fZLcxyhc0xch0-0kqn_Xlm-AdkhEksaZb475N1yy-LkkeiMhuIFgD8Ff4KtdqmRY6PNk2DbU8NIPBvF5vxLGL-zypylIkVpeP2eiOPcdac6qzDLRhE3yo82ud-CaqWujFN-JfVAxFti-48a9CuuiKBpnthGVwxgHR-LRUgx0POt0B5K_cAm6-iQMqLQs3_UxPh8arsD0CqiUw1mU4kkroe0LD_pHF_5WQCkuWxZQL_EUT1j2O5jHstDSC3TUWt8XGjo2fsvr4UIDbQaFe2clHYh-qrkz1QSVj8DjDYlSbKLAZd6iUCCcBUVbDowqKDj0dtS4DyCC28MODFXPkIFrDKclcMqaCXbvGo9KDYf517fs6HOlDShd8snyFdzuSIKf4PrwaMNZuYzLRbygM76M0sUVwXMSR7NmyeZXySJdYD5PCsYpmdMyztg8LTJO53HFZ2I53VVzEpE4icJiwbOKp0VW8DJhVYKuMG-pkKGv2lCbeias7fkyTeZRNJO04NIOH0KEKP4Kw6ZvlMlmZpZDpRd9bdEVlsI6-4HihJPDF9SXGyXZfBAhFIyXCYz3wKw3cvlfq2twyMtrcPhfAQAA___dOATB">