<table border="1" cellspacing="0" cellpadding="8">
<tr>
<th>Issue</th>
<td>
<a href=https://github.com/llvm/llvm-project/issues/155451>155451</a>
</td>
</tr>
<tr>
<th>Summary</th>
<td>
[BOLT][AArch64] Bolted Binary Crashes in TCMalloc
</td>
</tr>
<tr>
<th>Labels</th>
<td>
BOLT
</td>
</tr>
<tr>
<th>Assignees</th>
<td>
</td>
</tr>
<tr>
<th>Reporter</th>
<td>
Bradshawz
</td>
</tr>
</table>
<pre>
Hello! We have been experimenting using Bolt on Mongodb aarch64 and have run into it crashing on launch on a subset of our host machines. The crash happens inside tcmalloc, and specifically i have noticed the crashes only happen when either of these two functions end up in the fdata:
```
1 TcmallocSlab_Internal_Pop_trampoline_12/1 0 2
1 TcmallocSlab_Internal_Push_trampoline_13/1 0 1
```
Crash is consistent on the machines it does crash on during startup, but only happens on a subset of OS versions for some reason. The data collected was on a Bolt version ~2 months old, but i tried with the newest head as of this morning and it still repros.
Command we run to Bolt:
```
llvm-bolt ./mongod -o ./mongod.bolt -data ./mongod.fdata -reorder-blocks=ext-tsp -reorder-functions=cdsort -split-functions -split-all-cold -split-eh -dyno-stats --lite -skip-funcs=_ZN8tcmalloc17tcmalloc_internal6subtle6percpu12TcmallocSlab4GrowEimmN4absl12lts_2023080211FunctionRefIFmhEEE,_ZN8tcmalloc17tcmalloc_internal18cpu_cache_internal8CpuCacheINS1_15StaticForwarderEE21DeallocateSlowNoHooksEPvm,_ZN8tcmalloc17tcmalloc_internal18cpu_cache_internal8CpuCacheINS1_15StaticForwarderEE19AllocateSlowNoHooksEm,calloc*,malloc*,_Znwm*,_ZnwmSt11align_val_t*,_ZdaPv*,realloc*,_ZdlPvmSt11align_val_t*,_ZdlPvm*,_ZN8tcmalloc17tcmalloc_internal6subtle6percpu12TcmallocSlab16CacheCpuSlabSlowEv,TcmallocSlab_Internal_PushBatch,TcmallocSlab_Internal_PopBatch
```
Here is a stack trace of the crash:
```
#0 0x0000aaaab3a2e1a4 in tcmalloc::tcmalloc_internal::subtle::percpu::TcmallocSlab::Pop (size_class=3, this=<optimized out>) at src/third_party/tcmalloc/dist/tcmalloc/internal/percpu_tcmalloc.h:877
#1 tcmalloc::tcmalloc_internal::cpu_cache_internal::CpuCache<tcmalloc::tcmalloc_internal::cpu_cache_internal::StaticForwarder>::AllocateFast (size_class=3, this=<optimized out>) at src/third_party/tcmalloc/dist/tcmalloc/cpu_cache.h:698
#2 tcmalloc::tcmalloc_internal::fast_alloc<tcmalloc::tcmalloc_internal::TCMallocPolicy<tcmalloc::tcmalloc_internal::CppOomPolicy, tcmalloc::tcmalloc_internal::DefaultAlignPolicy, tcmalloc::tcmalloc_internal::AllocationAccessHotPolicy, tcmalloc::tcmalloc_internal::InvokeHooksPolicy, tcmalloc::tcmalloc_internal::NonSizeReturningPolicy, tcmalloc::tcmalloc_internal::LocalNumaPartitionPolicy>, void*> (size=32, policy=...) at src/third_party/tcmalloc/dist/tcmalloc/tcmalloc.cc:1011
#3 TCMallocInternalNew (size=32) at src/third_party/tcmalloc/dist/tcmalloc/tcmalloc.cc:1102
#4 0x0000aaaab2524a0c in google::protobuf::internal::ShutdownData::get () at src/third_party/protobuf/dist/src/google/protobuf/message_lite.cc:679
#5 google::protobuf::internal::OnShutdownRun (f=0xaaaab24de878 <google::protobuf::internal::DestroyString(void const*)>, arg=0xaaaab3f6d5f0 <google::protobuf::internal::fixed_address_empty_string[abi:cxx11]>)
at src/third_party/protobuf/dist/src/google/protobuf/message_lite.cc:697
#6 0x0000aaaab24de8e4 in google::protobuf::internal::OnShutdownDestroyString (ptr=0x3) at src/third_party/protobuf/dist/src/google/protobuf/generated_message_util.h:180
```
The instruction it crashes on:
```
0x0000aaaab3a2e198 <+88>: adrp x15, 0xaaaab3e8a000 <_ZN4absl12lts_2025051212_GLOBAL__N_119submit_profile_dataE>
0x0000aaaab3a2e19c <+92>: add x15, x15, #0x380
0x0000aaaab3a2e1a0 <+96>: str x15, [x11]
=> 0x0000aaaab3a2e1a4 <+100>: ldr x13, [x12]
0x0000aaaab3a2e1a8 <+104>: tbz x13, #63, 0xaaaab3a2e1f8 <TCMallocInternalNew(size_t)+184>
```
Memory at that instruction when it crashes:
```
(gdb) x/10gx $x12
0xfffff7fb789c: 0x0000000600000000 0xb3e8a38000000006
0xfffff7fb78ac: 0x000000000000aaaa 0x0000000000000000
0xfffff7fb78bc: 0xb3a9b440ffffffff 0x000000000000aaaa
0xfffff7fb78cc: 0xac49f12200000000 0xac49f1220000aaaa
0xfffff7fb78dc: 0x007070610000aaaa 0x0000000000000000
```
Memory at that instruction on a machine where it doesn't crash:
```
(gdb) x/10gx $x12
0xfffff7fb389c: 0x0000000000000000 0xb3e8a380fffffffe
0xfffff7fb38ac: 0x000000000000aaaa 0x0000000000000000
0xfffff7fb38bc: 0xb3a9b440ffffffff 0x000000000000aaaa
0xfffff7fb38cc: 0xac49f12200000000 0xac49f1220000aaaa
0xfffff7fb38dc: 0x007070610000aaaa 0x0000000000000000
```
Let me know if there's any other data you want me to collect.
As an aside, is there a way to avoid this in aarch64? I've seen some comments in older bugs mentioning --skip-funcs= isn't supported on aarch64, is that still true? I've noticed manually removing
```
TcmallocSlab_Internal_Pop_trampoline_12
TcmallocSlab_Internal_Push_trampoline_13
```
from the fdata will cause it not to reproduce, but i'm unsure of how good an idea that is (if it actually prevents optimization on that function or if its possible some other function has the ability to bring in those functions to be optimized anyways).
</pre>
<img width="1" height="1" alt="" src="http://email.email.llvm.org/o/eJy8WEtv4zjy_zTKpWCDovyQDzk4fkw30JMOJg38gbkIlFiy-G-KFEjKjz7sZ1-QkhzbSXrSuzMbGJFE8ldvFovFrBU7hXgfTR-i6fqOta7S5v7BMG4rdvhxl2t-uv-EUuqIxvB_CBXbI-SICvDYoBE1KifUDlrr_z9o6UAr-F2rneY5MGaKajYBpniHNK0CoZwG4aAwzFYepRVI1qqi8m8MbJtbdKBL0K2BSlsHNSsqodCO4VuFHRAq1jSoLAhlBUdwRc2k1EVEV4GdbbAQpSiYlCcQHXelnSiQgxuIoAWt5KmnBYfK6yVchcazdxVaBHfQULaqcEIrC6g4tA0IFYiUnDkWJcuIhN-M9D-yjOFbL9CzZHn2WTk0isnsSTeZM6xutBQKs5hGdBsDAfozTGurK1DSg-IbpqtgGGGh0MoK61AFb3hJBwt6w3ONtjeiVsBb451gHTOubbz58tZdmsXeuuXrM-zR2GCPUhuwukYwyKxWnYO8VaDQUmLhkMOB9SRCePRQ-BeFWitXWdCSD2wFOCM8RLgqyK3wgNZBhYyDJ-O9IizU2igvtfe0cGCdkBIMNkbbceeMla5rP3vogs7pwL331YXNpNzXo9wLNo7otg6BCyN98TUOs6Og1MVo8D2MDGrD0YxyqYvvNkrWeHQjZ5uXmXPwRMm64FYbByPbSOFeZoYBJuWo0JIP31jBiJ-UHlnHnIXRSAqHMLLfRRPAnmT252M6BH88H94y0UfPzLa5kzhr0BRNG9PLEJv8ZvRhI-r6ccJyK2Mqnc0ooQlJCY3jbS_eH1h-3tbVZrOJ6Oov2MVp0bRZwYoKz2PpqmlXfuTz43OcxdNnx5wottocmDfQZkPjNQYyzOGz1IdH_Unr73bztK__IY7xYvkGP8-t6LPIMqKr-uI9-1Md6ovXZxfHTIqdyvZMZm6Y4exp370bvEJz-bR_FySDpsu_VvZn3oxnQeVV0_ovr9dmH9HV-znlgbmien-FbroF1_ul-31Cgz7RMJ83iu_gDCuwz5ldanm90yKaEAByJIQQxhjLE0YxZpOQTIfsnSyjZPlK7W6407177wzQvV_K34086QYimlrxA7NCMuv3SeJzjE8eUbKOkpVunKjFD-SgWxclm4gugDmwpojo1lXC8Kxhxp381_lo2XJh3fXIWUS67WTKhsmxt0E6n3eqxx_T8XUsd-NDQEfJ6r-ic7MTvOJhfNgOW2bd_8p0ZxmDpWaLtLMU_ZilSmZd1i_7mE2-rX4Pw09aiuL0UdSqab7qusd4O3wEtMaStdIt_V7_RWjvCaHVsijQ2k_a_SKFz2qvv2NIab-IfNTqWfzAP9C14Xz9RfgXXTD52NbsiRknvAqDqf3JAXstuM9yyWaIMB9b1E81_br1eDz-T8PpvO8KL2VM4riLpwQGxw-57REPNxL8LRxjQjuOk8ssR6d0wkjhs9xO6905gRntdN6W3dfNJq1ax_VBrfvqMlnuMOzKnwh6pncWtFvV87xaUKO1bIeZryY62WfzRSf69MNCflWDmH-0ygtXRsmaHDudJxzTeQpRsvoouTVaZ_Tp2flqNKKpD5ZQx3an5KKPIWZ2L2yScsanJfkVNqU4Is8Y5watzbBu3CmzHc_pA8uFT53HYxxH03WX2SKyBIB_yOqL_nCYXQWMNx5OfiVgXnxxZUbvlcaZYLDkb4qdHSo0zCHPBn1aJ2TI4HFKXhcL_j4glHWmDbXk-eIXrl6viwR4XSEsQhxF9CFNu-Mq-IObxj-P8dSHxRAQmDJCQkBkf94UtVMyjWlMs9--fH1YfsmyxyyOF7bNa-GyxuhSSMx8Sb_xTMjytRhFL8aCXorB4UKM_uErnWMSrPG63iEDmdkLGevMJZlo-tCHIFmGI3fzVtnUkYkJOdORfCCTnMnQjswbdmXpmcTkTMLlP65I0GSWXBrYA8sAfCOpDnWD8_uGPsTppLPlTUj8jrU2Jx-MrmLuKjrCDfwlRN4qItMdz30sH_0lmOyOENGJV5MsybH0f_Myn6cLv796lQkhs_5JgnJAjiFUknQYnt3A2TWcDJYb4OTm7wae9_A8YYt8MiFl__cW3JO9gRc9nBWTRRlTeiv85fgbcH4Wfk7mZBZ_QPiPeyhc5PtugneXvwd0LQUV0bl7v_j_iN-Szm8RWbwl5eLFbb098QbN3kR3JnqT5BU6P6Nv_fYOySt0cUbfuu3V4BtofiH5tdfek_zSul_QQY3wXekDiHAJMxjRuQWmTqBDPyt0K066hQNTYbHTQ4um75gs_XJgVnD0O17Yjg4wOLCTX87CmRz6L0INvb0o2cLniM73CBZRdc2gQtc1KhfWacnRQN7uLIRmoQ6Nm9F1EwNEHz62bRptnL9bvLAYpGFDp8eZFi8YD329mqk29PsM1nrvz_QLM4E_jD7Yk3t35atOHNx4ojS6fmkNwsFLW7DWhl2itPN2DH0q3hZ47npFdF5Dq2xrwiW60gd_9HPvD8GR9bvQ-vNclJ4SK1ynaWNwHyzdX8rYsEkDZGgwgTYQgBYaba3IJXaO6mLjvKxiwenAciGFC07PQyER-p3a4kUn1M8hvNwFmTod2MlGdDG-4_cJXyQLdof38Xw6o3RB0_lddY-zMk4TSmk6n8xnpMQyx5jyZJESnKb54k7ch5M6pbN4TlMyH6eM8bwsZ4zxuJjP02hCsGZCjqXc12NtdnfC2hbv4-l0Mo3vJMtR2tDOpvTh65dvEfUH4J257_p87c5GEyKFdfaFghNOhhZ4AEzX0fRhuewib7oObUPk8CAUMydY9bWLUOdbxV1r5H3lXBOOK7r1RZJwVZuPC11HdOv59I9RY_T_Y-ErrCC29bmwk3x_T_8dAAD__7_kktk">