<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/153123>153123</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            [BOLT] Instrumented aarch64 clang-21 crashes with illegal instruction or hangs
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
            BOLT
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          nathanchance
      </td>
    </tr>
</table>

<pre>
    When building LLVM 21.1.0-rc2 for kernel.org, I have noticed an infinite loop or `Illegal instruction` crash when running a `clang` binary instrumented by BOLT on some AArch64 machines (but not others).

```
$ llvm-bolt --instrument --instrumentation-file=/tmp/clang.fdata --instrumentation-file-append-pid -o clang.inst clang-21
BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: aarch64
BOLT-INFO: BOLT version: d8e9216c27b82b4292e83437d58aebf594adb111
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x6c00000, offset 0x6c00000
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-WARNING: 1 collisions detected while hashing binary objects. Use -v=1 to see the list.
BOLT-INFO: number of removed linker-inserted veneers: 0
BOLT-INFO: 0 out of 129351 functions in the binary (0.0%) have non-empty execution profile
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 47228
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 127221
BOLT-INSTRUMENTER: Number of function descriptors: 127221
BOLT-INSTRUMENTER: Number of branch counters: 1381499
BOLT-INSTRUMENTER: Number of ST leaf node counters: 678042
BOLT-INSTRUMENTER: Number of direct call counters: 0
BOLT-INSTRUMENTER: Total number of counters: 2059541
BOLT-INSTRUMENTER: Total size of counters: 16476328 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 14682724 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 144601856 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file /tmp/clang.fdata
BOLT-INFO: Starting stub-insertion pass
BOLT-INFO: Inserted 3810 stubs in the hot area and 0 stubs in the cold area. Shared 64123 times, iterated 4 times.
BOLT-INFO: padding code to 0x10400000 to accommodate hot text
BOLT-INFO: output linked against instrumentation runtime library, lib entry point is 0x121639fc
BOLT-INFO: clear procedure is 0x121600f0
BOLT-INFO: setting __bolt_runtime_start to 0x121639fc
BOLT-INFO: setting __bolt_runtime_fini to 0x12163a8c
BOLT-INFO: setting __hot_start to 0x6e00000
BOLT-INFO: setting __hot_end to 0x10318b94
```

Using this same exact binary...

On my [Honeycomb LX2](https://www.solid-run.com/arm-servers-networking-platforms/honeycomb-lx2/), which has an SOC with Cortex-A72 cores:

```
$ ./clang-21 --version
ClangBuiltLinux clang version 21.1.0-rc2 (https://github.com/llvm/llvm-project.git d8e9216c27b82b4292e83437d58aebf594adb111)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /tmp

$ ./clang.inst --version
fish: Job 1, './clang.inst --version' terminated by signal SIGILL (Illegal instruction)
```

On an Ampere Altra system, which has Neoverse-N1 cores:

```
$ ./clang-21 --version
ClangBuiltLinux clang version 21.1.0-rc2 (https://github.com/llvm/llvm-project.git d8e9216c27b82b4292e83437d58aebf594adb111)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /tmp

$ ./clang.inst --version
ClangBuiltLinux clang version 21.1.0-rc2 (https://github.com/llvm/llvm-project.git d8e9216c27b82b4292e83437d58aebf594adb111)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /tmp
```

In a virtual machine on my M1 Max Mac Studio:

```
$ ./clang-21 --version
ClangBuiltLinux clang version 21.1.0-rc2 (https://github.com/llvm/llvm-project.git d8e9216c27b82b4292e83437d58aebf594adb111)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/nathan

$ ./clang.inst --version
<hangs indefinitely>
```

I have uploaded the original and instrumented binaries above [here](https://github.com/nathanchance/bug-files/tree/bc8dcc862cf01f1da5ea3020f587b62c9c29850a/bolt-clang-aarch64-issues), compressed with `zstd -19` to stay under GitHub's single file size limit.
</pre>
<img width="1" height="1" alt="" src="http://email.email.llvm.org/o/eJzsWF9z4jgS_zTKS5cpWTbGPPDAJMturjKZq0nm9t6mZKnB2pElSpID7Ke_ahsyhCS7s3ePtylSgOl_6r-_lozRbBzigk0_sOnNlexT68PCydRKp-gfrxqvD4tfW3TQ9MZq4zZwd_evjyDyST7hWVAC1j7ANwwO7cSHDRPXcAutfEJwPhmFGqQD49bGmYRgvd-CD8AqfmstbqQF42IKvUrGO1ZxUEHGFnakMvTOkUZJ5MpKtyGCxjgZDke2Dl1CDc0BPny6ewTvIPoOYbkMqq1K6KRqjcMITNRNn8gk8KnFEJmYTxhf0qvixxdfMlGCtU9d1nibIMu-K3nxRZKx2dpYZMUNE6vUbZlYDRZO1lom-Q51JrdbdDrbGg2Zh5GBCMePmcgZX9JBstv71SdWLCG2MqAG3_yGKpHjtj6aQZ5xGkkY2YZ7VH2SjUXQmFAl1JeCHmXYYAIZVGuIog9Ij6UcHHVJPTjzCUOkoBRL0DXORV4pMWtq0ZRiLrAuymKmp7XEZj2dl1I3ef7K_LUJMYG01iuQWgeMEUwEvueXlCqgTBRshzvYBr8JsoMWpcYA48lkehbB95Xi9EfZ5tfriOns2YVkdLKxJDmg9WoIBnRe4ytbfVBElv3Wd9ts0BlZcdP5Jxxy_CKgJ_Zfl5_vb-9_Jgk5KG-tIafF50DArjUWoZWxJenH7B0DGifwJSJkT6y4ySF5iIiQWgRrYppcGuj6rsEAfg0BySoN1rhvGCjVMJCqJ3RIuV0s4ZUbOPg-EXMu5sU0h3XvhpqLYNyg82gZEzWfcCamTMxPdewy7LbpcMwzcuA2-CH9n5U8PH7-8vGn-8efPpOy-2dTjdMmUO4qaS1E6gEaowpmm_xoaTkTov6rgtKYzheicjET5zX0nqzT2f9b_iZQgwTle5eO_s6LOi_n8z_nfXgEi3INzmt8IaGa1bwUfy7g3A3n_Pw91kefpD3LnnMmwafzafnukUfWaH7HS8a8KmdVIWpoDmnsr5HqQh2LvcPOhwMT7zrkpeSYAhXHWOjYmUQdjLSUVS1mojxqMQ7-KOteyryMbVlWPK-n1Y_J-ueY4LAz1kKDECUVXPIDE7zV8y8L7iHJMHS0mPrmWKND6cgYL2lvTxVc1DkfGJ7LsvXUtVGCdBouflPe6uHHCTyMg6Iqc1FAMh1G6o0mYZAktxyfveopW6mHoa4oG5MHvs95ObRR-iaV8l3ntUyjHQn36VKC79O2T2Mr0iA3cphnF82SRjkZANY0QVJiXNNHQJfCAbbeuDTOhVzkVTFfq1fjwaIM1HUU6j7gd2LO169aXcQ0OP7rVxrjX4-6v0YKyPGQ76h5h5PAyxmjrP-AsfXpXFOFbw6ll_To9Mn5RV438_ISlvDllzgUSGsiRNkh4F6qdOzZk8kRynxy0B2ATT_84h0elO8auPu3YNMbJuo2pS0VAhMrJla73W4SvTU6C72bKN8xsZKhyyIGGvuZw7Tz4Ztxm2xrZVr70EUmVu1Jbmb3YpA0p1DuWqNaGnGE9R4-XcPOpBaufUi4z5YzAcoHHJS_ibgmp0LKRA5ZdsIdfHlNDz_0xqY74_r9iJJOuOQcg74638aktm-OByNEd3zLtsHT5J1sTPpxVDO0sRFBnYGmrHffnN-5zJJx2cb1RNUGlHoAGHYoMB_NnvHlrYtJWov6xgR6fuwgoz_OfTDiwXMvrE1sieUfvoGc3M3E7F1yMYOEoTNOHnExYXxp4eH259u7O3LUW7h7OOBFxn1yFM1lt8WAsLQpSIiHmLB7GfF79KQbs_v87zD_L2H-v_DCyxS7dSDhyYTUS3va1WiH6w7wMYePcg8fpYKH1Gvj_06rlw5tfYdMrMZt_YcSjBXXrXQbQg8ax33cHljx0xtxGYF_v7VeaoI9LYIPZmOokxAOebl80xAyGEE2tCux6YcWA741dV749fyegYlV02-GHZnGTAo4PFK1VqquhFrzfJ1rOUVZcMHX03rWVELNlZjXUy6J1NuUjVlw8q-JsSdpw4RSvtvS6kj7GM0mVvHfY9KQ5XNWDVgnJnmA3tG--bNJv_QNE7MINHUtjqBvAJbWdCZNrvSi0PNiLq9wkc-m5bwqeVFetYuGlzlXUheSK1UWeYlYi1qVZdPUjWjWV2YhuJjyOhdccC5mk3Jd8WkzK1Xd1JLrhpUcO2nshBJu4sPmajjIIp8WuSiurGzQxuHKRgiCFEzQgL8Ki_Heot9EVnLaH-N3CckkO1zzDAzTG8Kb3-N3dNjzHcR4BYNx9JR5PS7ABxgy6aoPdvHXSoeJ1Skwq-ORnhbiPwEAAP__a6nxWQ">