<table border="1" cellspacing="0" cellpadding="8">
<tr>
<th>Issue</th>
<td>
<a href=https://github.com/llvm/llvm-project/issues/114701>114701</a>
</td>
</tr>
<tr>
<th>Summary</th>
<td>
LLDB: `DYLDRendezvous::GetAction()` found two `eAdd` states in a row
</td>
</tr>
<tr>
<th>Labels</th>
<td>
new issue
</td>
</tr>
<tr>
<th>Assignees</th>
<td>
</td>
</tr>
<tr>
<th>Reporter</th>
<td>
georglauterbach
</td>
</tr>
</table>
<pre>
> [!NOTE]
>
> I am by no means an LLDB (or LLVM, for that matter) expert. Please assume that I may be missing something obvious...
### Prerequisites
I am debugging a program with LLDB that loads many shared objects, more than 100. While the overwhelming majority works flawlessly, I am stuck at one point, and I can't seem to make any progress for two weeks now. I have tried every configuration that seemed to have made sense to me, and I am now here because there may be the possibility that there is something wrong with LLDB itself - possibly.
The code that I am working on closed-source (which is a shame, I know - I cannot do anything about that). But I will share as many details as I can so we can find the problem.
While collecting the information for this report, I uncovered information that I wasn't aware of either; maybe some of the log and information down below are actually the problem's solution?
### Environment information
I am debugging inside a container that runs Linux, more specifically: Yocto Linux (Kirkstone). This Linux has `glibc` 2.45 (no `musl`) and debug symbols should be available. The `glibc` source code is not available under `/usr/src/debug/glibc/2.35-r0/git`, though.
```console
$ /lib/libc.so.6
GNU C Library (GNU libc) stable release version 2.35.
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 11.3.0.
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<https://www.gnu.org/software/libc/bugs.html>.
```
The program is written in C++ and compiled with `g++`:
```console
$ g++ --version
g++ (GCC) 11.3.0
Copyright (C) 2021 Free Software Foundation, Inc.
```
The C++ standard used is mostly C++17, but for some libraries it could still be C++14 or C++11. (I am not a C++ expert; maybe this is bad?)
I am using the command line `lldb` that ships with [`vadimcn/codelldb`](https://github.com/vadimcn/codelldb):
```console
$ ./lldb --version
lldb version 19.1.0-custom
```
I have checked the target's environment, the debugger's environment, but changing the environment does not seem to make a difference here.
ASLR does not seem to make a difference either.
### Examining the Problem
The application uses `dlopen` to load shared objects. The call to `dlopen` that does not work has a backtrace that looks like this:
```txt
(lldb) thread backtrace
error: <ANOTHER SHARED LIBRARY LOADED EARLIER 1>.so 0x027572fd: DW_TAG_member '_M_local_buf' refers to type 0x0000000002866a2e which extends beyond the bounds of 0x027572f3
error: <YET ANOTHER SHARED LIBRARY LOADED EARLIER 2>.so.so 0x003d9ca2: DW_TAG_member '_M_local_buf' refers to type 0x00000000004713cf which extends beyond the bounds of 0x003d9c98
* thread #42, name = 'libref_adv_ld_d', stop reason = signal SIGSTOP
* frame #0: 0x00007ffff7fca110 ld-linux-x86-64.so.2`__GI__dl_debug_state at dl-debug.c:116:1
frame #1: 0x00007ffff7fcfa6c ld-linux-x86-64.so.2`_dl_map_object_from_fd(name=<unavailable>, origname=<unavailable>, fd=-1, fbp=<unavailable>, realname=<unavailable>, loader=<unavailable>, l_type=<unavailable>, mode=<unavailable>, stack_endp=<unavailable>, nsid=<unavailable>) at dl-load.c:1501:7
frame #2: 0x00007ffff7fd0488 ld-linux-x86-64.so.2`_dl_map_object(loader=0x0000000000000000, name="<BUILD DIR>/<PATH TO LIBRARY THAT DOES NOT WORK>/<LIBRARY THAT DOES NOT WOTRK>.so", type=2, trace_mode=0, mode=<unavailable>, nsid=<unavailable>) at dl-load.c:2309:10
frame #3: 0x00007ffff7fd4299 ld-linux-x86-64.so.2`dl_open_worker_begin(a=0x00007fff815fe740) at dl-open.c:534:21
frame #4: 0x00007ffff7062631 libc.so.6`__GI__dl_catch_exception(exception=<unavailable>, operate=<unavailable>, args=<unavailable>) at dl-error-skeleton.c:208:8
frame #5: 0x00007ffff7fd3a3a ld-linux-x86-64.so.2`dl_open_worker(a=0x00007fff815fe740) at dl-open.c:782:15
frame #6: 0x00007ffff7062631 libc.so.6`__GI__dl_catch_exception(exception=<unavailable>, operate=<unavailable>, args=<unavailable>) at dl-error-skeleton.c:208:8
frame #7: 0x00007ffff7fd3e1f ld-linux-x86-64.so.2`_dl_open(file=<unavailable>, mode=-2147483646, caller_dlopen=0x00007fffa4e1f3a1, nsid=-2, argc=5, argv=0x00007fffffffd5e8, env=0x00007fffffffd618) at dl-open.c:886:17
frame #8: 0x00007ffff6f9a5ac libc.so.6`dlopen_doit(a=0x00007fff815fe9b0) at dlopen.c:56:15
frame #9: 0x00007ffff7062631 libc.so.6`__GI__dl_catch_exception(exception=0x00007fff815fe910, operate=<unavailable>, args=<unavailable>) at dl-error-skeleton.c:208:8
frame #10: 0x00007ffff70626e3 libc.so.6`__GI__dl_catch_error(objname=0x00007fff815fe970, errstring=0x00007fff815fe978, mallocedp=0x00007fff815fe96f, operate=<unavailable>, args=<unavailable>) at dl-error-skeleton.c:227:19
frame #11: 0x00007ffff6f9a0ee libc.so.6`_dlerror_run(operate=<unavailable>, args=<unavailable>) at dlerror.c:138:17
frame #12: 0x00007ffff6f9a641 libc.so.6`___dlopen [inlined] dlopen_implementation(dl_caller=<unavailable>, mode=<unavailable>, file=<unavailable>) at dlopen.c:71:10
frame #13: 0x00007ffff6f9a620 libc.so.6`___dlopen(file=<unavailable>, mode=<unavailable>) at dlopen.c:81:12
frame #14: 0x00007fffa4e1f3a1 libsystem_router_d.so`RsCMW::Rpm::CSystemRouter::load_dll(this=0x0000555555e4c030, iDllName="<BUILD DIR>/<PATH TO LIBRARY THAT DOES NOT WORK>/<LIBRARY THAT DOES NOT WOTRK>.so") at system_router.cpp:545:21
```
When enabling additional logs via `log enable lldb dyld`, I see this:
```txt
<LOG FROM APPLICATION>
<LOADING A BUNCH OF SHARED LIBRARIES (more than 100) CORRECTLY>
intern-state DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit called for pid 50587
intern-state DYLDRendezvous::Resolve address size: 8, padding 4
intern-state DYLDRendezvous::Resolve cursor = 0x7ffff7ffe0d8
intern-state m_previous Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 1 (eAdd), ldbase = 0x00007ffff7fc7000
intern-state m_current Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 0 (eConsistent), ldbase = 0x00007ffff7fc7000
intern-state bool DYLDRendezvous::UpdateSOEntriesFromRemote() action = eAddModules
intern-state DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit pid 50587 stop_when_images_change=false
<LOG FROM THE APPLICATION>
Process 50587 stopped and restarted: thread 1 received signal: SIGCHLD
<LOG FROM THE APPLICATION>
Process 50587 stopped and restarted: thread 1 received signal: SIGCHLD
<LOG FROM THE APPLICATION>
intern-state DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit called for pid 50587
intern-state DYLDRendezvous::Resolve address size: 8, padding 4
intern-state DYLDRendezvous::Resolve cursor = 0x7ffff7ffe0d8
intern-state m_previous Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 0 (eConsistent), ldbase = 0x00007ffff7fc7000
intern-state m_current Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 1 (eAdd), ldbase = 0x00007ffff7fc7000
intern-state bool DYLDRendezvous::UpdateSOEntriesFromRemote() action = eNoAction
intern-state bool DYLDRendezvous::UpdateSOEntries() action = eNoAction
intern-state DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit pid 50587 stop_when_images_change=false
<AND THEN REPEAT FOREVER:>
intern-state DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit called for pid 50587
intern-state DYLDRendezvous::Resolve address size: 8, padding 4
intern-state DYLDRendezvous::Resolve cursor = 0x7ffff7ffe0d8
intern-state m_previous Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 1 (eAdd), ldbase = 0x00007ffff7fc7000
intern-state m_current Rendezvous: version = 1, map_addr = 0x00007ffff7ffe2a0, brk = 0x00007ffff7fca110, state = 1 (eAdd), ldbase = 0x00007ffff7fc7000
intern-state DYLDRendezvous::GetAction() found two eAdd states in a row, check process for multiple "_r_debug" symbols. Returning eAddModules to ensure shared libraries get loaded correctly
intern-state bool DYLDRendezvous::UpdateSOEntriesFromRemote() action = eAddModules
intern-state DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit pid 50587 stop_when_images_change=false
```
---
I am really out of ideas at this point. Any help is greatly appreciated!
</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJzseltz4rqy_6dxXrqgbJnrQx4IlxlqMyF_Qvb858kl223Qimz5SHIY9qc_1bKBhEBW1uWsc9auTU0xxGq1-vLriyRzY8SmQLz1unded3LDK7tV-naDSm8kryzqmCfbm1il-1svnAKRseB-uZ563YnnTzx_5IXT4w-YA88h3kOhIEdeGOAFLBaTO_DYQGlYLP75zWNjyJQGu-UWcm4tao8NAX-WqG0bHiRyg8CNqXKsieaQ8z3ECLkwRhQbMCpHu6VfKn4RqjLtdruRof5mYf0PHjRq_K9KGGHRvCZxgqYYV5sN8eFQarXRPIedsNtaZLe2VDw1kPNiD2bLNaag4l8wsYbUyJV2IhYQ-H4bvm-FpL8R1Avq3RZlTrxz_ovSwu5hp_SzgUzynURj5J5YODmMrZJn4BZUgVAqUVga4kUKc0h44bG-BYOYg1WQ82cEEscJjMbUxtwp2CE-GyjUrg1z2PIXBKsFpoAvqPeQqCITm0pzK1RR60YsMSWmjjrnKYLBwqBbB08y8JzYwhY1QowJr4xTU-PBMaRzqYwRsZCkqWNfUwjzyl07rej7aGJhDcoMWs1kuX_jxvUWIVHpEQXkHaWfnd8LSKQymLaMqnSChK_dViRbWo-Tq2r55_BMkrdqQxbKQqrIerU4PFaVddw9NmzDXUWr7ISUta-BN55P0XIhDf3t-IAha7tfmSjSWn2tYon5GwVqQCRKSkwsLUiEosiUzms31HEgDGgslba1xFWREH4wfUPa2GDHTY0HviMJVQYoyNBeeEfOiNFZm57TWlJtnAtfM0rVroAYpdqB0zGxFZdy_1oJj_XJa7KiCV44uxxb0-JFaFXkWNjXC3wQZaIwIkXgBEbLRYFNFtBVYWAhiurnMapMiYnIREKieeEIfqjEqpqGfP0PoZ-NVQU6x63JhPXYlhvwev5Gijjxej6wdqdLEwpFj_PKSK_nU74hszjJwOzzWEkDZqsqmRKc-QsXkscSiTW-4dfAzcFSULTZEzVURYoa3AKzymiPzYxOPDZz63hsVnNhM9YOuy1NVBthHfkY7FZVm-3bNNbz63-JKoySeHBABzw2kyKuv5O2Ue0e1INf7p9gDAsRa673pDc9qFcdgrFOSo11in1BbQgQJE2z7liVey02W0tTxzSH-YzBTCPCo8qsw9xMVUVau5rwWiTtQ7gKQzbJiNo01IRLg3WGaGxHoE9UuSdEJKpIBbEyRyZN0rhfwo5rzQu7Jx5kaHzBOmS-TVfjr6P79ehuvpivf4DSMJuv76ePjzBbrqAx38NotZ6PnxajFTw8rR6Wj9OjlnkpJKZUqJzBxkdbBEE7bPsNHdkNRndzQwB8up__v6cpzGdP92MY3T0uF0_rpvDNlAYCUh3EDdCtrhKnGBmprC1uEL3w4NxwvLW2JN4em3lsttvt2puiaitNUDnar_axx2ZxtTHtrc2lFx4UOQLkPGseipkwsNPCWixAFDD22J3H7hz0k4MNXDImhNeDxOwo4q-gsJkCrVZjvnro8JjQN3Ygqq36AcSCT0LsqsIH1YzlRcp1CpWh_GkgV8bK_WE86BO_uLIORi5TShcrAg0IC4nLAMZSCYiPXIMOIezwR9AmyZuiaIEf1647mFMitk08xDylHMqG7zJjZQ5FIVF5Tm6RonD5Rso0pnRT1-mtKE3jqC556IWnIk8Kj80oDzW01JCxwVtMbYTdVnE7UbnHZhdmseGnnd0mJMo0Pve2e3YMn2E7aPutpDJW5R-4rOlPki0mz1jXT8v1Bq2rPHgqLHVmxKaGUKf4fpzcmWx5sTkY89U4pArrLP22f4JUZBlqLBJ0bc2bvDt6XKw-M7GuvFc6z-lPnoviINJDU1jPYMvLUoqkrsuVQVe6UqlKLJzzlWs_zxrPuiZRYSSCtxMILUfBqVdy9ZBDzJNnq3mCh6ZWPRuQ4rkG6TUM2J_2oNWggQvYrUaenhi-nohaK0250gvHo_vl-ut0BY9fR6vpBBbzu9Vo9QMWy9FkOoHpaLWYT1cQUC4zCvyfPut3-yxLafrke7QefYlyzGMqp6wffYukSriM4irzWB80ZqgNqW_3JdLsw4cNej3OEOpeEH9aLFIDMe5V06bFlFkM9UfHNcP30v-YruFzGrBag0YJP0yHCWd_UAm_0w_CJPukEm7N4eAtCkcHR3ks7DCKkoLnCF44IVEo52EW8fQlkmmUesylRWNVCRq5UYUjpJ0hl_A4__K4Xj7UnAGId6YdMxZSrWgk72dZlvWzhAeBDzJtSWrHWj8HvVavQwZiXs-Poi_zKEpl5MI5MpZbpG1PKlvuSTvxwlEQ9Oj7sB6cVgver5bxXnJ1tVRGOS-jOm6iTKs8ylLqBWlzEE68cFwVx-aNtrFsDEqT1tfHCaGTVuB-xuU1Ko1cfsSFwpp69iujESHi2miu0qtjxvLkOcIivSoZdeCXx4aNJ0i22hFdnyzebzxxdAN754bU7wwGn3MDpZKD8q8hX38OQCUJGfPC8d3TfDGByXzlRJx54fhhtP4K6-UxHtdfR2uYLKePcL9cw_fl6h9H0mska0fTNorWoAJTW7v-TUktamzs_5q9f4M1WehTsQ38C8AO31u0w4bDaxZNZUQZP6IEjzqKcSMKjw340aTEZBB0M-x3_JMoNMeJ0g07JNClEOucS-L3WC8M4LjPeB3FCbfJNsKfCZZ1qzY4_b4WXiVqbq_ak-uN-dieLku3zDNKtKpWh_kDLxwNzmHafW_UkIf8c0b9LebsD5iLlgvm7P37mLN_wZwYZB9FvWtL2CAT8teSWYsFnX5nEPY6PXpKzQ3qqGlsXruBdzDIQh68Cr4Wa1RNvHDSbX6_vJlFn7SLAxrE4sJYLxhccOxg4CpR_4JjB2fW6GVD3uXJG8fW4kepEvYynIbxCU6n4OxdA9PwzwPTuSCB_xejKXjXOZAyGH6ojGvQ2EDFvzQ14lyNvlMDtTZWi2JzicJhIOdSqgRdlTyn6GX_Y6ZgFEPB8J0tzvsaApOP-NYWqXQsI12RM_-ofI5XXeUdli-CPDiv9CRYr3OOuCZOaW8qCtrApl530kA6EnkpkXZizW5-4NxJ4f17-pvrqeQ8jPrBtWIbnFdbpxXzL2v1ufz1CYEGTiB2SaCzonvIcSSQ2RuLeaRVZSkhUs_S81dm_O07bdrC0arM6x_jR0e5coT1I2o9olRKjw3qXV6D9a77YCfxQxcvYiLl_V_cdDnrvFGunZQlpb9O91VvcvHw4PsWC8CCx9Kd4af1ASKXINXGwIvg7gBFbWoaBHdCke5l2py0zpsDyc9sfMPxYvkFZqvlNxg9PCzm49F6vrw_XXmdqEaT-f0XGMHd0_34Kyxnb_eN8-kjeGzw5sKIrDBerlbT8Xrx44ylKCzqolXvkCb7guciWbi--WH5OP__kx-LSQMALFL814uqzJ1G_uwuj74KW5fQ1B1zlSKFrt8d9C-wpg8xO7E5sDVKviBZ190xGfEvJJS69FmSzYsNdH4Hw6TSRmm3v_R_No1Ehn46uMIrj0qN7o4P3rA8HjkRp6BO6mVE4ja8XzUqGTLuPB_r5_ejbsfabJ5svUUOyFc4SlOPDd2GLI25wQtT-7RnuSZ4UmmNhYW_UHDfCT5WhRHGuuOx3y1_rJS86MmnMuUWH5fTwmqBZqZVvsJcWfRY3US5Y2-3GJnwm0orebh7_cOwPmLZnVZEu60rMXyDJnJHgJTEMi4NvovPJorXX6cfRPKDVgnB_bREiak7L9doLNcW3QFVc7YSgMYExQumzVkJjT3Ov4y_LiZ_z_X_k3b-N9POnxi9p-zz98qaf0rWuVej5HQT_bvX-O2c_8qMNrqfUDDfw2r6MB2tYbZcTf85XbnF_hPT_2di-m_bSvxRwS957gvaJn7q0MpUVaTu3SVaqF7cgCiAg1Y7dxK0xeQZyqYsEgTzSlpRSto5sUhHzasV7PAWRxtWaCvtLt5etR9gFWBhKo2H27TTve8G69e8MIVEaY2Jlft_66bo0qaq1Wq9u57W6F4MUpUFlYFIkRtw73UJU7-l1oZRsYctyhKEgY1GbuUeeFlqTASnZoUFN-ltmA7DIb_B26Af-j3W63e7N9tbxochwzTAOOnEwwB5P8Mwzga0Q_V5v38jbpnPOkHgh_4wHAbddjfo8Czp8EGP9-JOB72OjzkXsi3lS95WenMjjKnwNgg6fT-4kTxGadybjYwVuAM3StvP7uRG39KkVlxtjNfxpTDWnNhYYSXeLhaTO3cf2PM_AWav57_Cs9fzXez0_HNU31Ra3n5wS09CNP-1Sq3qO5OZE914bNbo9nLL_jsAAP__i43piQ">