<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/54066>54066</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
             BOLT-optimized shared object crashes
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
            new issue
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          KenPatrickLehrmann
      </td>
    </tr>
</table>

<pre>
    Hi,

We ran into an issue using BOLT on one of our (huge) shared library.
BOLT-revision:e0f1dd018e0f94a7d694bc615975c3a7d26d9e50

We currently do not have access to perf counters, we therefore rely on instrumentation. We used the options provided in the README (`-reorder-blocks=cache+ -reorder-functions=hfsort -split-functions=2 -split-all-cold -split-eh -dyno-stats`).

We could not use `--update-debug-sections` because it triggers an assertion, but that's not the issue here. (I'll open a separate issue for this)

The optimized shared library crashes in a part of our code that seems relatively benign. Specifically, it crashes in `DecisionBox_SetContextString` (which is simply calling `PrSr_DecisionBoxSetString` inside a for loop). My experience with assembly is limited but I do find the changes made by BOLT suspicious.


Original binary:
```gdb
Dump of assembler code for function DecisionBox_SetContextString(char const*, char const*):
../src/SearchPersonalization/ConfigManagementSteward/src/decisionboxmanagement.cpp:
155     void DecisionBox_SetContextString(const char *sName, const char *Value) {
156       for (unsigned contextIdx = 0; contextIdx < ssp::EnumTraits<DecisionBoxContext>::kSize; ++contextIdx) {                                                                                                  
   0x0000000001158050 <+0>:     41 54   push   %r12
   0x0000000001158052 <+2>:     49 89 fc        mov    %rdi,%r12                                                                                                                                                  
   0x0000000001158055 <+5>:     55      push   %rbp
   0x0000000001158056 <+6>:     48 89 f5        mov    %rsi,%rbp
   0x0000000001158059 <+9>:     53      push   %rbx
   0x000000000115805a <+10>:    31 db   xor    %ebx,%ebx
   0x000000000115805c <+12>:    0f 1f 40 00     nopl   0x0(%rax)

157         DecisionBoxContext context = ssp::Enum::FromIntegralUnchecked<DecisionBoxContext>(contextIdx);
158         PrSr_DecisionBoxSetString(context, sName, Value);
   0x0000000001158060 <+16>:    89 df   mov    %ebx,%edi
   0x0000000001158062 <+18>:    ff c3   inc    %ebx
   0x0000000001158064 <+20>:    48 89 ea        mov    %rbp,%rdx
   0x0000000001158067 <+23>:    4c 89 e6        mov    %r12,%rsi
   0x000000000115806a <+26>:    67 e8 90 39 ee ff       addr32 callq 0x103ba00 <PrSr_DecisionBoxSetString(DecisionBoxContext, char const*, char const*)>

156       for (unsigned contextIdx = 0; contextIdx < ssp::EnumTraits<DecisionBoxContext>::kSize; ++contextIdx) {
   0x0000000001158070 <+32>:    83 fb 0c        cmp    $0xc,%ebx
   0x0000000001158073 <+35>:    75 eb   jne    0x1158060 <DecisionBox_SetContextString(char const*, char const*)+16>
   0x0000000001158075 <+37>:    5b      pop    %rbx
   0x0000000001158076 <+38>:    5d      pop    %rbp
   0x0000000001158077 <+39>:    41 5c   pop    %r12
   0x0000000001158079 <+41>:    c3      retq
End of assembler dump.

```
BOLT-generated binary:
```gdb
Dump of assembler code for function _Z28DecisionBox_SetContextStringPKcS0_:
   0x00000000011587e0 <+0>:     41 54   push   %r12
   0x00000000011587e2 <+2>:     49 89 fc        mov    %rdi,%r12
   0x00000000011587e5 <+5>:     55      push   %rbp
   0x00000000011587e6 <+6>:     48 89 f5        mov    %rsi,%rbp
   0x00000000011587e9 <+9>:     53      push   %rbx
   0x00000000011587ea <+10>:    31 db   xor    %ebx,%ebx
   0x00000000011587ec <+12>:    0f 1f 40 00     nopl   0x0(%rax)
   0x00000000011587f0 <+16>:    89 df   mov    %ebx,%edi
   0x00000000011587f2 <+18>:    ff c3   inc    %ebx
   0x00000000011587f4 <+20>:    48 89 ea        mov    %rbp,%rdx
   0x00000000011587f7 <+23>:    4c 89 e6        mov    %r12,%rsi
   0x00000000011587fa <+26>:    67 0f af bb 02 ff 83 fb imul   -0x47c00fe(%ebx),%edi
   0x0000000001158802 <+34>:    0c 75   or     $0x75,%al
   0x0000000001158804 <+36>:    eb 5b   jmp    0x1158861 <_Z26DecisionBox_SetContextBoolPKcb+33>
   0x0000000001158806 <+38>:    5d      pop    %rbp
   0x0000000001158807 <+39>:    41 5c   pop    %r12
   0x0000000001158809 <+41>:    c3      retq
End of assembler dump.
```

We believe there's an unexpected change caused by a `callq` instruction, resulting into erroneous code. We tried many options, without any luck.

Removing information about `PrSr_DecisionBoxSetString` from the profile data, or telling BOLT not to process `PrSr_DecisionBoxSetString` makes the problem go away (but then, there are similar issues in other calls).

We're unsure as to whether this is a bug or if we are misusing BOLT. We'd gladly share more information if need be.
</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJzNWFtv47YS_jXOC2GDuln2gx-STRZn0e7potlzCvRlQZEjiRtJVEkpsfvrO0NJjryJE_QkBY7g-CKKH2e-uWYyow67f-lF-GHBrxf8cnj_DZgVDdNNZxh9OtcD651uCnb1y89fmWnwBczkzPSWLcJN2RewCLfMlcKCYpXOrLCH1YBGW5YW7rXTpllEl8DzQCkebPDLNhapWm_jTK6DZJsmMsLf4VptIeE_SCR7a6HpqgNThjWmY6W4ByakBOcYCtqCzZk0fdOBdagQewDWlWAhNxb1AdxoSCfX2b5GINGhOCv2G2mGMuOjzLR0z7HWmnut8KZu_P1fby6vP9-Qoos1R1WMVWCXWWXknVtE11LIEtW_YselvG-kh8LVMnfGdmzp2kp3JyvhdFNU1VKaSk2_oWRLdWjM0qGUDs9Eblc_0mF63EA8oPiM5Fr2rRIdLBVkfbF0MJ6z5iwDKegp3bHO6qJAgsiuwjmw9BCxlfW4WIpuEabOw5Lig-WJxBVp_wkXqwppAtzMHLTC4oHjU0gz7tFI_XYu6teR11r_iYSeOgiTVrgSHPEsGKJ1k09Jo8CLg6dA7ch8aK97MmIGjS7QcLctSJ1rieQdSAFUbgaHal_jOrncldl_u4Xug0HP2He3yEBTECuo0EOpZYnyM6frFrEJjLwcl7_YW_ttBoEIj1vRi9A_UGZSujKmJQOxzwcGe3RDDY0E9qC70lNcZ4iMZ1TIQYfKE9OfyIlz3Qx-J0vRFCh3LRA0OwxB5nrXaqlN705MP7z_gmbUjahYhu_2gFE1rqKv-FehsuHOdV-3xOooCYzckuCTL7IXmQo3KB7twshZhJdE9Q83tsfjV6tF-NFZie-3IKwsv6CrGZRT_ykGR_uI4LkuPotGFEBheNvBg7DquE-NsmRmXx8fWsm2PR4SJAmj695o9aroJOQgL4rq_i1q8Aqc3P6vqHqfvhbp1XTEmg0X8YQ4PRq8aNB4cjjgk9ozjGDGF9HV6b0PzDkva3R50_T1Vys0hnD0YSboKOQiuhmeu7vF2CAgzCH4eoQbZWL_-DVojV_4nk9XECQbnnBSCYXig7T-6ThgSYyfbe9K2hwmNgjPQ4QjRDiH2LLNluVyEqA292yEUr4aecx_XvG_e53XMhm1TGZajo46Jyprz0OsR4j1nKiNJyqZBJgR5SaiXsLcjpjbuVjRU7H25yHECBHMfCAKmMrwc4_hMUAAQnhx4CUsOWHNnIHnLMhZzBnnXqzGtNWwlwouCif2P9SUIEmPFnkaV1M8-gA9Ccbh20dr6k_4RGFF9Z8GS7e8A3U2Qn0OeYxHDNNJhs1RhvO14ribss4x_UwJ5wj2DFXrKfKCmTugK6j8xAseace4OYs1hWCwecTKsVsiR9CNnGGdhYinKJ55weCcIJ5xTvTJwTnVC5jphBnNMKXHXD-DiU4zYLoXVJ28NZzRhgfBhm05ixAZSPPhEkrZKPRV_w9ECniUCe55f8miz_jJ05r4XJG8OfXh_6cSc5bPdHLDaBaxm4jlGePH7C2xxfA2ivlevp4F0mjCnCXLNGFAGeU7_mPh98yi4M39yRRFZyWasneUPkqUZGOaNO3Rq19Qasre0SzGEvUU4nyyTqd4iGbZmoqtPIV4odimU76Pg0cIOeZ7C90fw84bbDxPekKFXeJpmzk1krP_4QpogBp-9T5d57ffT2LpiWW__CRv-bfjIU-1TeHN3UkK_3t3ch7zzb1ACu_fC6Tw5l4ghffrBVJ4ay_wDGb-fkUzzd9cNNP8_Ytmmr9_0UzzM0UTrSFylmGqD0nzIe3ruieTLPk-TiXnOQyW8ZxuX6V1wydao3hmdkn5n7HBg4ZCkiYDmKjOY038RjO5sYr4zP19KEpDIdmsA3oUU876-ZRzZUyFCScjsOiFUoEl6a15HrP0W_P8hr9Dnj_N8MfBUgaVhvtxduanQaJhfUOTDUm5fxhWMD9RUjStEDQu8T3UOBrpbC-nsZIF11cdzVT8NBGsNQ2Y3vl64KdvmOsRB__XP0zzNz-8011p-o7R7aqXdyfF6VdABx8wsaLUfrTAREbPvza5ybH59zOX1ppcV8CU6AQdSNMrGKY_fvziR2CGHvMDxtdwa3EHbgJGqmtWGCYexIFau2G2Bp4QTysT-Od0rStsVPz4zA-tDC36dtQ9nfihKXATdok97fcjz4cS_A4au9GESbCsL0gTndP0kw6ptXsc3BLfCKNYUQlVHYZpHCYLfJszibsbINvC6kLtIrWNtuKi010FOw-zfDLPM9l3dI5p_nbR22pXdl3rqHiHH_FVoDn7bCVNjT-q6n76WCJbtBV_DjTglyTm6_VFuVtLAQpA5Fkcb9M0FUEeQyQzHvFM8Si_qAT6qtstEmxpwwYeBibx-yK5vtC7kIf0ikMeJWGwiuI4UKEEnkK6jnm4iDnUQlcrkmNlbHFhd14k5NDhYqVd5x4XMYaoPQd_HOKLHh3U7n6C5otAL5B3P0OJBDbNhZdi57X4C3y2b1s">