<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/118019>118019</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            __llvm_profile_counter_bias is undefined Error in Large CUDA-related .so with PGO Continuation Mode
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
            new issue
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          xuesu
      </td>
    </tr>
</table>

<pre>
    Here’s a revised version of your text with improved grammar and structure. It maintains your original details while improving clarity for proposing an issue:

---

**Title**: `__llvm_profile_counter_bias is undefined` Error in Large CUDA-related .so with PGO Continuation Mode

Hi,

I've been struggling with an issue that has consumed me for nearly a year. I am building some large shared libraries (`.so`) related to CUDA using PGO continuation mode, but I consistently encounter the following error:

```c
if (UseBiasVar && BiasAddr == BiasDefaultAddr) {
    PROF_ERR("%s\n", "__llvm_profile_counter_bias is undefined");
    return;
}
```

Here:
- `BiasAddr = &__llvm_profile_counter_bias`
- `BiasDefaultAddr = &__llvm_profile_counter_bias_default`

>From my understanding, `__llvm_profile_counter_bias` should be determined later, after passing this check, inside `mmapForContinuousMode`. From the code of `Value *InstrProfiling::getCounterAddress(InstrProfInstBase *I)`, I assume that when counters are inserted, the compiler should insert `__llvm_profile_counter_bias`. 

Since the counter size exceeds 100k, I assume this insertion has already happened, but the symbol might be lost because it is a global weak symbol. This seems similar to a case like [[this Rust issue](https://github.com/rust-lang/rust/issues/120842)](https://github.com/rust-lang/rust/issues/120842). However, the build system is too large and complex, and I cannot pinpoint where this symbol might be lost.

### Problem Details
The `__llvm_profile_counter_bias` symbol is consistently reported as undefined, and I cannot determine why this happens. The error always occurs, and due to CUDA's limitations, I can only use LLVM-15. While I reviewed LLVM-19's source code, the logic seems quite similar, so I suspect the root cause persists across these versions.

I am building the PaddlePaddle framework. After compiling with the following flags:

- `-fprofile-instr-generate`
- `-fcoverage-mapping`
- `-g`
- `-mllvm`
- `-runtime-counter-relocation`

It produces four instrumented shared libraries (`.so`), which `dlopen` each other. The first three shared libraries behave as expected, but the largest one (`libpaddle.so`) reports this error:

```plaintext
dll = ctypes.CDLL("/Paddle/build/python/paddle/libs/libphi.so")
LLVM Profile Error: LALALA: initializeProfileForContinuousMode
LLVM Profile Error: LALALA: CountersSize 32480
LLVM Profile Error: LALALA: BiasAddr 133895675533552
LLVM Profile Error: LALALA: BiasDefaultAddr 133896653571776
LALALA: mmapForContinuousMode 2
LLVM Profile Error: LALALA: initializeProfileForContinuousMode
LLVM Profile Error: LALALA: CountersSize 92448
LLVM Profile Error: LALALA: BiasAddr 133896026068704
LLVM Profile Error: LALALA: BiasDefaultAddr 133896653571776
LALALA: mmapForContinuousMode 2
LLVM Profile Error: LALALA: initializeProfileForContinuousMode
LLVM Profile Error: LALALA: CountersSize 7777352
LLVM Profile Error: LALALA: BiasAddr 133896653571776
LLVM Profile Error: LALALA: BiasDefaultAddr 133896653571776
LLVM Profile Error: __llvm_profile_counter_bias is undefined
```

I inspected the compiled `.so` files with `nm` and found the following symbol table entries:

#### `libcommon.so`:
```plaintext
00000000001298f0 B __llvm_profile_counter_bias_default
00000000001299d8 V __llvm_profile_filename
...
```

#### `libpaddle.so`:
```plaintext
00000000268d9c08 B __llvm_profile_counter_bias_default
                 U __llvm_profile_filename
...
```

#### `libphi.so`:
```plaintext
000000002566c6c0 B __llvm_profile_counter_bias_default
                 U __llvm_profile_filename
...
```

#### `libphi_kernel_gpu.so`:
```plaintext
0000000014e496e0 B __llvm_profile_counter_bias_default
                 U __llvm_profile_filename
...
```

As seen, the weak symbol `__llvm_profile_counter_bias_default` exists in all shared libraries, but the actual `__llvm_profile_counter_bias` symbol is missing.

---

### Questions

1. Can I safely ignore this check?
   ```c
   if (UseBiasVar && BiasAddr == BiasDefaultAddr) {
       PROF_ERR("%s\n", "__llvm_profile_counter_bias is undefined");
       return;
   }
   ```
   Since `mmapForContinuousMode` should eventually initialize the `__llvm_profile_counter_bias` symbol, would ignoring this check resolve the issue?

2. What scenarios might cause `__llvm_profile_counter_bias` to be undefined?
   - From my understanding, if counters are being inserted (over 100k in this case), the compiler should have inserted `__llvm_profile_counter_bias`. Could this issue arise due to its nature as a weak global symbol and loss during linking?

---

### Key Observations

- The issue consistently occurs with `libpaddle.so`, the largest `.so`, which also reports the largest `CountersSize`.
- The `__llvm_profile_counter_bias_default` symbol exists, but `__llvm_profile_counter_bias` is not present in any of the libraries.
- CUDA limitations restrict me to LLVM-15, but the logic appears similar in LLVM-19.

Any insights or hints on debugging this issue further would be greatly appreciated. Thank you!
</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJzcWF9z4jgS_zTKSxeUkcHAAw-ELLepy9bOze7MPaZku7F1kSWfJEPYT3_Vkk0gyUy42tl7uBkq2Nhq_frfr7slnJOVRlyx2S2b3d2IztfGrp47dN1Nbsrj6me0yH7ibJGw5dKBAIt76bCEPVonjQazg6PpLHh89nCQvgbZtNbssYTKiqYRFoQuwXnbFb6zOIZ7D42Q2gupXVxrrKykFgpK9EIqB4daKuwFSV1BoYSV_gg7Y6G1pjWOfhUapHMdsnTNEvqMRqN4wTh9fpdeYbxk6RpYljw-KrVvHltrdlLhY2E67dE-5lI4kA46XeJOaixZlsBP1hoLUsODsBXC5svdemRRCY8ljJ2Jun7626-wMdpL3QlP5vjFlBgx_CwZ38TLe8bne4QcUQdDVJUi_EHCoAT4WniohYPCaNc1WEKDQWGNwqojCDiisGO4B9FA3klVkgxnGgQVELpaWCxBydwKK9EB4wuWJWNnWJYwvoQBvTdBG-iCFUmF4lyFhlTgG8g7D_cBjXQetVdHQN2bDHxN4JQyB5KBZKuTG2i78ClYspY7wvHF4a0U7quwwHjGeAZ0uy5LCyy9Y-lduL_DneiUp58JL5vfsmQNAPDp86_bx58-fyaNOGd85thso8PlBhjnV_uVVixZOsi16Dur4z2b351j751I4R_0GlH8nIMmRb6zbxRxWnWm2hWLH8v4-gnH1poGmmPQxDovNDk_KP_doKY4drXpVAk5UnahbcgOQIFgab3YkTdbIgJdga-lg6LG4omeSe1kibRF04h2a2wf6aZzIcyzZAwBGAVDYUokMmBZ8lWoDoHx9b123n4KuAhuSmlYod9EhGQLdI7xxek9urgVLi4mT1HgbijkHaVEzJFDjRp6LR0IiwQUrSf3bnosTSsV2kH3-PxjW40hWvs3qQvsJcVwd_IPBHwuEEsHkyR5egVLun4TSiBKYqEsivIItWhb1BEaJRQJdccmNwoaWdWe_KKMo-9CdA5BeopYAZUyuVBwQPHULxjD77SPQ2wcONlIJSzlsoCCTKbkE0Lg8duA53PnfM-PszvGF7X3rSMf8C3j20r6usvHhWkY39rO-ZESFFHhmvFtWOgY3054spiGrPkhUsbwszngPgYfGSMwGbij89iQ5t6Yns-obJAnFT6HSNUl0ZHQ2nhopW6N1CEYbO-A98w6HupBGj_wyZpcYQN3sdKwZP17jddkURQu3SUhWmwNRR6IC5Z5hfaUeHCojxFsjAtHPsVInyDUQRwdmKLorBtklFQZIl8zPnegZCN94GkXQ7AQGoxWR6DgeXj4-stoMhvDP0MBvQ_VGg9Y9k-WQYYznS1ixg5eUKaSRR9a_-6kxyHA6AVn4B5c51osYgBbYzzEcG2pDXDegSiscY4eOxyaA9eb_7JkkYRPoiwVxr-ws6LBg7FPY1gHOor5eyqRl7Vmp0TlXko--W606902ksQkowo1WuHxjINHu8Ls0YoKR41oW6Kjs4cXNw0FwvkPttNeNjjqY4K6AFMEH5z4-d5TX1J2BTrYUUsTgHQNagqOjyozGflQy6Km7UplWiTJgKKowfgabQyTnbSOHGDxnWKfYy32SHGIz-SoS8oJGeU8GI395krmbTD_eX9AwexigH6jpLeKWjd89ixZl0qFWlb4Y4tuvLl7eBgK9Da6lvFtcDvj2_boa6PpYniiZO7iV1tLQhGLc7KmWIVYNjC2YdS-PazpP11JLb0USv6B_UtvS9PHQvoi5H4jZk_5dJFcsehU_ydpuljOsvlslqazGb9y6XkTECRk2SydzSfzeUYSTu--W27hml1-vGmWfDpd_NemyRKeJdlinkz_j00zn8_n6dXO_5Zqf94470q4uiV-0_PeE3lFCjnvpko4URaQSBfJmWWJJrYM1WpnOl2-4uu-cnqRKwTUntjqhVeGwky1ObJSYZrG6J6V0m-ST3L6N-HLxS6B2--pfGqoXy1clgv4-noh_dGioXgYj8dvDfQO6Asq_Rg0zxblskgWV4OG1_--_AjQkXavQzzLsiIrrjfzX4X48QmtRvVYtd3V4CdTnC4z_F-CX4dGXQ_91Vkb_0GzeTb6AT6H1kpqEEq9Kfnn9V0UvhMfib7sYxsZZr7xm6OTweT_6NDFRjM8mIxhIzR1gmKH6giy0mZovePImG6j6S7mfwD4IUcAf-EpwKuDANIgnAWcqxJv42D4nYF4mDhxj5o8QnY61Z3gqWtdFHrCOL2SoS-nc7DojNpHif0Z2Da6iVPzLzy4ArWw0rh-IIrt-se7e0Oz04utBqeO4JtnEHJ3OY7nSGiHoZx8T613GJsplqMawmHf-b43sYde9kXCx4P7JiyLg3g4ThNWOhzGJ-kdaOE7GxpkEdOxn7D7jKDypWiCKbtgbCX1Uzi02H4zQf6OR_g1d2j34ixPRqFXjyAuZsU42J2q5psGfHPRqZ_NB8NwIJQzZ036xbvnjQktPQG5nm16Q0TSGcjl44CRDsJEbtGh9oGs9BHMLiIcyCoCCseOZyMsxbG3svDQBD_18-vF5BImU5qWhX059pB6mGh7_lrrYzivqmrvwFiopaYLDSXmXVWd0if6ZddZGqv6BMsRKouCfCTa1mIhhceSZi6hn-BoOsYnN-UqLZfpUtzgajJP-WKS8Onkpl4t8qxIZ8skzSZ8khQo8gXPF9N0mSbTZMrFjVxxenXCF5NskkwnYyym6WS3nE93eTKflBmbJtgIqcZk5rGx1U0AuZpMFslkeaNEjsqFE3rONR76fOecze5u7IoWjfKucmyaKPLcixgvvcLVtaz4Jw68bzqrVt85GQoDdfwatdb8C4vLw6Go6H7F_xMAAP__9bgDMQ">