<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/60119>60119</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            [libomptarget][AMDGPU]: Deadlock during initialization with multiple shared libraries
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
            new issue
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          Maetveis
      </td>
    </tr>
</table>

<pre>
    # Short description

When an executable links to multiple shared libraries using OpenMP offloading to AMDGPU, it will hang during application startup.

# Steps to reproduce
## Environment

<details>
<summary>LLVM version used</summary>

```console
❯ clang --version
clang version 16.0.0 (git@github.com:llvm/llvm-project.git ebfb1ddbe18457723ea62f4ecc94667a44ca4569)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/local/bin
```

</details>

<details>
<summary>ROCM version used</summary>

Installed from ROCM 5.4.1 from the rocm repositories.

```console
❯ dpkg -s hsa-rocr | grep Version
Version: 1.7.0.50400-72~20.04
```

</details>

## Reproducer

```console
touch empty.c
echo "int main(){}" > main.c
clang -fPIC -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx90a -shared empty.c -o liba.so 
clang -fPIC -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx90a -shared empty.c -o libb.so 
clang -L . -l a -l b main.c
LD_LIBRARY_PATH=. ./a.out
```

# Observed behavior
The application hangs indefinitely.

# Expected behavior
The application does not hang.

# Details

Seems to be a regression as [AOMP release 15.0-3](https://github.com/ROCm-Developer-Tools/aomp/releases/tag/rel_15.0-3)
(which is based on llvm 15) does not reproduce.

I captured the following backtrace:
<details>
<summary>gdb backtrace</summary>

```plaintext
Thread 1 "a.out" received signal SIGINT, Interrupt.
futex_wait (private=0, expected=1, futex_word=0x55555556af10) at ../sysdeps/nptl/futex-internal.h:141
141     ../sysdeps/nptl/futex-internal.h: No such file or directory.
(gdb) bt
#0  futex_wait (private=0, expected=1, futex_word=0x55555556af10) at ../sysdeps/nptl/futex-internal.h:141
#1  futex_wait_simple (private=0, expected=1, futex_word=0x55555556af10) at ../sysdeps/nptl/futex-internal.h:172
#2  __pthread_once_slow (once_control=0x55555556af10, init_routine=0x7ffff7ac5c20 <__once_proxy>) at pthread_once.c:105
#3  0x00007ffff7c4fd57 in __tgt_register_lib () from /usr/local/lib/libomptarget.so.16git
#4  0x00007ffff7fc206d in omp_offloading.descriptor_reg () from ./liba.so
#5  0x00007ffff7fe0b9a in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7fffffffd9e8, env=env@entry=0x7fffffffd9f8) at dl-init.c:72
#6  0x00007ffff7fe0ca1 in call_init (env=0x7fffffffd9f8, argv=0x7fffffffd9e8, argc=1, l=<optimized out>) at dl-init.c:30
#7  _dl_init (main_map=0x7ffff7fc7000, argc=1, argv=0x7fffffffd9e8, env=0x7fffffffd9f8) at dl-init.c:119
#8  0x00007ffff7f14985 in __GI__dl_catch_exception (exception=<optimized out>, operate=<optimized out>, args=<optimized out>) at dl-error-skeleton.c:182
#9  0x00007ffff7fe50cf in dl_open_worker (a=a@entry=0x7fffffffc3d0) at dl-open.c:758
#10 0x00007ffff7f14928 in __GI__dl_catch_exception (exception=<optimized out>, operate=<optimized out>, args=<optimized out>) at dl-error-skeleton.c:208
#11 0x00007ffff7fe460a in _dl_open (file=0x7fffffffc660 "./liba.so", mode=-2147483647, caller_dlopen=<optimized out>, nsid=-2, argc=1, argv=0x7fffffffd9e8, env=0x7fffffffd9f8) at dl-open.c:837
#12 0x00007ffff7c0334c in dlopen_doit (a=a@entry=0x7fffffffc5f0) at dlopen.c:66
#13 0x00007ffff7f14928 in __GI__dl_catch_exception (exception=exception@entry=0x7fffffffc590, operate=<optimized out>, args=<optimized out>) at dl-error-skeleton.c:208
#14 0x00007ffff7f149f3 in __GI__dl_catch_error (objname=0x55555556b0e0, errstring=0x55555556b0e8, mallocedp=0x55555556b0d8, operate=<optimized out>, args=<optimized out>) at dl-error-skeleton.c:227
#15 0x00007ffff7c03b59 in _dlerror_run (operate=operate@entry=0x7ffff7c032f0 <dlopen_doit>, args=args@entry=0x7fffffffc5f0) at dlerror.c:170
#16 0x00007ffff7c033da in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#17 0x00007ffff74304fc in rocr::os::LoadLib (filename=...) at /workspaces/deps/ROCR-Runtime-rocm-5.4.1/src/core/util/lnx/os_linux.cpp:179
#18 rocr::os::GetLoadedLibs () at /workspaces/deps/ROCR-Runtime-rocm-5.4.1/src/core/util/lnx/os_linux.cpp:224
#19 0x00007ffff7473b63 in rocr::core::Runtime::LoadTools (this=this@entry=0x5555555bbac0) at /workspaces/deps/ROCR-Runtime-rocm-5.4.1/src/core/runtime/runtime.cpp:1532
#20 0x00007ffff746df91 in rocr::core::Runtime::Load (this=0x5555555bbac0) at /workspaces/deps/ROCR-Runtime-rocm-5.4.1/src/core/runtime/runtime.cpp:1386
#21 0x00007ffff746de70 in rocr::core::Runtime::Acquire () at /workspaces/deps/ROCR-Runtime-rocm-5.4.1/src/core/runtime/runtime.cpp:112
#22 0x00007ffff7454b9b in rocr::HSA::hsa_init () at /workspaces/deps/ROCR-Runtime-rocm-5.4.1/src/core/runtime/hsa.cpp:206
#23 0x00007ffff767d560 in RTLDeviceInfoTy::RTLDeviceInfoTy() () from /usr/local/lib/libomptarget.rtl.amdgpu.so
#24 0x00007ffff76574b0 in _GLOBAL__sub_I_rtl.cpp () from /usr/local/lib/libomptarget.rtl.amdgpu.so
#25 0x00007ffff7fe0b9a in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7fffffffd9e8, env=env@entry=0x7fffffffd9f8) at dl-init.c:72
#26 0x00007ffff7fe0ca1 in call_init (env=0x7fffffffd9f8, argv=0x7fffffffd9e8, argc=1, l=<optimized out>) at dl-init.c:30
#27 _dl_init (main_map=0x5555555b00e0, argc=1, argv=0x7fffffffd9e8, env=0x7fffffffd9f8) at dl-init.c:119
#28 0x00007ffff7f14985 in __GI__dl_catch_exception (exception=<optimized out>, operate=<optimized out>, args=<optimized out>) at dl-error-skeleton.c:182
#29 0x00007ffff7fe543d in dl_open_worker (a=a@entry=0x7fffffffd4c0) at dl-open.c:758
#30 0x00007ffff7f14928 in __GI__dl_catch_exception (exception=<optimized out>, operate=<optimized out>, args=<optimized out>) at dl-error-skeleton.c:208
#31 0x00007ffff7fe460a in _dl_open (file=0x5555555ab950 "libomptarget.rtl.amdgpu.so", mode=-2147483391, caller_dlopen=<optimized out>, nsid=-2, argc=1, argv=0x7fffffffd9e8, env=0x7fffffffd9f8) at dl-open.c:837
#32 0x00007ffff7c0334c in dlopen_doit (a=a@entry=0x7fffffffd6e0) at dlopen.c:66
#33 0x00007ffff7f14928 in __GI__dl_catch_exception (exception=exception@entry=0x7fffffffd680, operate=<optimized out>, args=<optimized out>) at dl-error-skeleton.c:208
#34 0x00007ffff7f149f3 in __GI__dl_catch_error (objname=0x55555556b0e0, errstring=0x55555556b0e8, mallocedp=0x55555556b0d8, operate=<optimized out>, args=<optimized out>) at dl-error-skeleton.c:227
#35 0x00007ffff7c03b59 in _dlerror_run (operate=operate@entry=0x7ffff7c032f0 <dlopen_doit>, args=args@entry=0x7fffffffd6e0) at dlerror.c:170
#36 0x00007ffff7c033da in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#37 0x00007ffff7c6f29e in llvm::sys::DynamicLibrary::getPermanentLibrary(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) () from /usr/local/lib/libomptarget.so.16git
#38 0x00007ffff7c5c415 in RTLsTy::attemptLoadRTL(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RTLInfoTy&) [clone .localalias] () from /usr/local/lib/libomptarget.so.16git
#39 0x00007ffff7c5cb63 in RTLsTy::loadRTLs() () from /usr/local/lib/libomptarget.so.16git
#40 0x00007ffff7c254df in __pthread_once_slow (once_control=0x55555556af10, init_routine=0x7ffff7ac5c20 <__once_proxy>) at pthread_once.c:116
#41 0x00007ffff7c4fd57 in __tgt_register_lib () from /usr/local/lib/libomptarget.so.16git
#42 0x00007ffff7fbc06d in omp_offloading.descriptor_reg () from ./libb.so
#43 0x00007ffff7fe0b9a in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7fffffffd9e8, env=env@entry=0x7fffffffd9f8) at dl-init.c:72
#44 0x00007ffff7fe0ca1 in call_init (env=0x7fffffffd9f8, argv=0x7fffffffd9e8, argc=1, l=<optimized out>) at dl-init.c:30
#45 _dl_init (main_map=0x7ffff7ffe190, argc=1, argv=0x7fffffffd9e8, env=0x7fffffffd9f8) at dl-init.c:119
#46 0x00007ffff7fd013a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#47 0x0000000000000001 in ?? ()
#48 0x00007fffffffde46 in ?? ()
#49 0x0000000000000000 in ?? ()
```

</details>

It seems that the amdgpu target plugin `libomptarget.rtl.amdgpu.so` initializes the rocr runtime (`rocr::HSA::hsa_init`) which will in turn `dlopen`s the other shared library finally ending up in `__tgt_register_lib` recursively. At this point the call to `std::call_once` in [interface.cpp:38](https://github.com/llvm/llvm-project/blob/94c772dc923a63abc744c011db51cecfe80cf093/openmp/libomptarget/src/interface.cpp#L38) deadlocks.

# Proposed solution

During plugin initialization any extra calls to `__tgt_register_lib` should be remembered and delayed until the outermost call, similarly `__tgt_unregister_lib` should just remove items from this delayed list if it happens during plugin initialization.
</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJzkWklz27i2_jX0BiUWCHAQF14oVpR2ldNJOX793luxQPBQQgckeAHQlntxf_stgNRAyXaGTtJJXVciicRwvnPw4QwkmDFi3QJcBsmrIFlesN5ulL58y8DegzAXpaoeLwNC0YeN0hZVYLgWnRWqDfAywIvh83830CLWItgC7y0rJSAp2o8GWYWaXlrRSUBmwzRUSIpSMy3AoN6Ido3eddC-fY9UXUvFKnfHKrR4u3zz_n8CcoWERQ9CSrRh7RpVvXYdWNdJwZlDgYxl2vZdeAzH47XQefkaOq2qnsO-zTW_bu-FVm0DrZ2MpFcVWCakCejr_S3TNw3TjwF9fXPzx1t0D9o40b2BKqBXAVkdOkwmS_Hwj6vWKLkD8JoE-TJYrBCXTqfZbJxvaB5u7kREaYhDjAIyXwsbxHgt7KYvQ66agC6kvG8CsnJfs06rP4HbcC0sgrIuo6oqIZrHSZYRCiwldQyc53GaZiyOOYuTNA9IPsi8Y3oNNqALtJ2nRRrP-vZjqx7amRRtv52t237st9HAKtSoCqTr3SkjtkPTdWsskxKqpdCuKSCr3miHTnEmA7IqRXtilRO7B2R1ZvrPWpPbd1efvyZ7nKjWqkF-bBLGYTRc2w0grXjjWKOMsMoRNfySRa26j2s0M2hj2EwrrlGQXaG1hg79cbzMuwu6QFGYhThMcIzxLCP_JjjE8deaaiD37Y7y-tPIrer5BkHT2ceQD7eAbxQKCBGtRQ0TbUDmjirZqyBbBoSggL7293f9Rx7X76-v0KxWHbRNt_8xs55cJqBL1lRr3s5YU7n_G8PQ7P8mnZ7u0zDNNwFdruttjhmajX5khIxmyrkUFhqFfh445RmcGxSimUTMfZQT890si5vrV7eL2_8v3i_ufgvoMkRhQFYsVL19iQiEonelAX0PFSphw-6F0ruNChMn6ZynQaKtoBatsCAfz_zl620H3H5ypkqBQa2yfsqzSZYjK49ufwBovB8uATGkYa3B-J3KDAqSV4t3b98jDRKYARQlIZ7RIFkGZL6xtjMBXQRkFZDVkd8jq9t3V81sCfcgVQd6dqeUNM5equkCshonc3csWw83inHmncMLyPxhI_gGCYNKZqBCqkXOj6IoCUh-0HIfPCaqXiPOOtu7ZXcOo1ZSqgcXmErGP1rNODjgn-W81lV5POozgkknmWgtbO3EJ0duww6MIQRp4CAcLVxwZxJ9uH5z_fudC6fXrQWt-86OCtW9hW3xwIR1QabT4p5ZCOgSu84wciKgy8hdj52VdnfwNhn-UlZH2FmNWRQ64ppHU0HnFqDtrPP9ftzModYtk-EmoIsojgYAURwh9_fZI9HvChnns2ohASmNKqGBW6X3nCbzdVU6QKXdUxMj9I-rGhAaHcMojGhcYvRj0WRkj4YgVBSd9QwqVMuhMFI9ODz-gqvWaiWfEHmFnBsptOqtaD3qbVbXdZ0xnnCCUUCvimHCTqutp_IA81hYyB0cnOzhUITwFmOMh7l4XFdJhkSLisKubaFhLYwFXUhRoiEmDUH7LNeQohw-VdMNrjw0KoxSl0PthMVTYTUnOK2cMNV0xSEZDXcZr9IOwERuOAhxoWc_bXIyLeAyZ25azqQsnNXcFM6mAb1SnRWN-Mu5n956I10hptfcBR73FWNorfMEAwuYXt8PTfdHTaPp67qucph79rSum_t8ulc9H1ejkjOHyC_EES3SMx04i850GKSczrtHeQ5r1Myr8qwBznBRvMeVIVRUBwAuhhYN647oV_MMY3wq7FlET6twBiGK9mGDzk9sE8X5PBk4-ua6cPA4s3xTwJaDL5O8qXYXzy-7i2SDA3ieF-ZTRgOtlZ6ZjyDBqnbAPj-sa366rgnmtcNeycIlPs7BfATtEDPHsyfpw2mFDxLdsIE-yfzg5PCZjcj8J7YRwUfYoxMbxSn2-7cYjeSwusgzNUqauiKNHHsEQhwkVysFdDkjUZzFc5rGmbvrNhLoonIJzAsKt0ZUfvA3YvR-teY0O2hMpl4XUxrzgRWeFJUattsLlEjqAyX2MtL0IIL-TUIcfj8tP8c_jB_xmSo1fUoVN42PpOWfLWtgEkRLDEOY19pYLdr1aatfy4ZJqThU3UlrNf-uypIjaiSn1CiTfNwMfmihe79WBzC7X2fr5EaT2qcGR8SaYvVfn2SYlzw4t-wQHKL0jMbVsHHHbXa0cZ-x2LhXX7LYYQcdWSmbSI4pjmu_gVz57woBulBm-L5RrLoZshcHZSRG6JI3LyAgK-eDTce4L2DGbO723dXt7LZvrWhgphVvZv6Rhcv4NA_IiisNLgmywmc_7TYgK2UK__Qm5F3nTXWIYdH8CWhvwDp04PCZXZrzXSEREh8g5VMbZrRM6dSGfkb_axR7MKmvAB1muxGOR_7rmEfj5ilLxvE30UuPEPa_dmZO6FF6PY2DcVrVefTZSh3p8yMVoPOD3ybRqQKQ4c9SYMH_1QsN34xHz8KNjsw9DWRxEpd5OUX724fF8GNj2D6P_MYAN4btCI6PTDkNgWlWJak35e3dzRLuBYfrtlZ3j6MtT24OIL-05tFWhqyp1l1_XKGQaQRLkywuPZLizc27V4ubojB9WVwXbjTvum8lNfn16yKS_px1Ecmer4t2jgOPKcf3qovI_Neti0h-WhfFtPrSuqiK-afqIvoL10X0i-qikXWszBNfF73gIJ4sk2ge_WxlEv3bZVKVwstlEv2-ZVKVzn9YmUT_m8ok-rOUSROGPV0m0X-sTKLTMomnNcnBSfavsX3aYx7HYmT52LJG8Bt_VGDMidZg34NuWAut3TWQOd8wjbhqjQ3IwqEzthr6FwXfbqNouCiZEbzYsejKjZp0djcKq5mwZtc8KLvv4VnGrNKHduT7LL4mNTt7HE2n8ZMnPI6SMT80u7SQWQtN5-u027ubgMx_sLI7Q6eu8-3dzS47Tb0JkldcqhZQ6PVmUjATJMtvYJr81DRjbXhsGjnYxHxlqnz-emAaqzlJ4qoe9sk__MIkOgSMOPqRL0ymAbAu-de-MCmPC4OY_vqFQRz_nIVBnHzyhUkNUf5dC4P4pGiqcET3iaM_vFX0Zkiuj_kpRZnG7rsaDyFt5-ksjR0tj-y-CylHf97uAV0FdDXOeeg-8bEOPsTpC93z89nxc92_9KTOtUVmOBmxYdafIhhSYjRsP9TJfu1EpfiF3Nk_QRBWMCn-ArM7vKTR-DDCI0zxC48_HGCSo-EshD9mJ1pke-0Fj2l3ioeZld2Anh7ie0S1aJmUjwhaf3Kv79AA-twBObAaeK-NuAf5GKKFU1sY1CnRDhZwuwVZ5cYfYpXbQc75Dcq6KONfZ9eM754B0fknT4w8cVIuIKtSKuf28phnGal4TihLKSt5FsccR1FVJhEHXsMc8xrnNCCr4XjQiavcPwqaAiP0hnpKV8AqqfhHc3Zc5r1WnTJQIaNkf3qmcjmcdRyJsF_n4RgOax8RbK1m3mhmtNrTVjcb1csKlYA0NNCU4BaQtRWqQLJHqJBjixzWuLegG2Wsn9anBKIRkmn5eJi_b5-R8GdvrJOh7gEJ68g9nqkTZi9LCmORqJGwaMO6DlqzO9T5pKLhRXVJq5zm7AIuozSLMU3zLL7YXOa4xjGkES0hrms8j3kVR5zjxL8vyNILcUkwoTiK5hHGKY5DnlKKE1pnQOs0p75Ia5iQoaNFqPT6QhjTw2WKoyi_kKwEafyxWEJaeEC-0ZWryfJCX3oqlb1PxJ1O5jCLFVb687QTkiTLIHk1HmtNlgFdoOXIi50BTpb4QdjN84dnL3otL7-Y8V4HE5CV1_E_AQAA__9OEpJ8">