<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/54628>54628</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            Codegen regression for AArch64 Android Kernel
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
            new issue
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          Gelbpunkt
      </td>
    </tr>
</table>

<pre>
    ## Preamble

Hi there,

first off I would like to mention that I have no clue what info might be useful to the LLVM developers to assist in fixing this issue, I tried my best to track it down for you as much as possible, but I know nothing about LLVM internals, kernel internals or Xiaomi's bootloader.

## The Problem

We have a modified Android kernel (repository: https://github.com/DavinciCodeOS/msm-4.14) for the Xiaomi Mi 9T. For building it, we use a LLVM-only toolchain (aka clang, lld, llvm-*).

For building it we initially used Google's "14.0.1" clang prebuilt (https://android.googlesource.com/platform/prebuilts/clang/host/linux-x86/+/refs/tags/ndk-r25-beta2/clang-r437112b/clang_source_info.md), based on LLVM 79d58b4d3017d159bf09a77398c9a116128de193. **This one builds the kernel, and it boots fine.**

Switching to a newer toolchain, Google's "14.0.2" clang prebuilt (https://android.googlesource.com/platform/prebuilts/clang/host/linux-x86/+/refs/tags/ndk-r25-beta2/clang-r445002/clang_source_info.md), based on LLVM 18308e171b5b1dd99627a4d88c7d6c5ff21b8c96, results in **kernel building, but not booting**.

Since the bootloader is proprietary and we have no insight in its logs, we don't know what was wrong. kmesg in recovery is also empty, indicating that a kernel was never booted, instead the bootloader straight up rejected the kernel, even though all we did was use a newer compiler.

## Tracking it down

I was certain that this is a bug in LLVM, and decided to manually bisect it down to a single commit by building specific revisions of LLVM (using ClangBuiltLinux' tc-build with a Dockerfile for reproducible builds: https://github.com/DavinciCodeOS/tc-build) and checking whether a kernel built by this revision of LLVM booted. Here's the results:

- 79d58b4d3017d159bf09a77398c9a116128de193 (Sep 3, 2021): GOOD
- 1f0bc617bdd11a4f3b0d820603b0089f81aca79b (Sep 30, 2021): GOOD
- a944f801cacdaa40b3869986844a6ffa08b87c19 (Oct 3, 2021): GOOD
- f38cbd785985be73be0d0129781d929e55abffb1 (Oct 4, 2021): GOOD
- c4dd0fe4b3709d4603df34f89b25cc03c1c5060d (Oct 4, 2021): GOOD
- 8b3d944a97cc944a24ed1296c801da5654896092 (Oct 4, 2021): GOOD
- 993555beb8ff33aa35773af01cb3821ae45e6e39 (Oct 4, 2021): GOOD
- 0f0e31cf511def3e92244e615b2646c1fd0df0cd (Oct 4, 2021): BAD (Only 3 commits in between, this is the only code change, other ones are doc & test)
- 6f80339b18da61be986e75b4824ff5b308412ead (Oct 4, 2021): BAD
- 7f28b4d5b7a5f3581dbdba0ee8936b519034ae65 (Oct 4, 2021): BAD
- c800119c46fb266b7fc75409fd9cbbb1a6d8f72a (Oct 5, 2021): BAD
- bbf4b3dbbe3bfaeecdff2a29ed49da172895d82c (Oct 9, 2021): BAD
- 239b4d62b6c07f27b1763a090f82c9f92ce06b8c (Oct 18, 2021): BAD
- 18308e171b5b1dd99627a4d88c7d6c5ff21b8c96 (Nov 30, 2021): BAD

I was able to narrow it down between two commits: a working one and a broken one, so the ones that might cause the breakage are t hese: https://github.com/llvm/llvm-project/compare/993555beb8ff33aa35773af01cb3821ae45e6e39...0f0e31cf511def3e92244e615b2646c1fd0df0cd. One of these two is a typo fix, the other one does not boot, indicating that  0f0e31cf511def3e92244e615b2646c1fd0df0cd is the commit that started the regression.

Latest clang 15 dev versions still have this issue.

## Figuring what is wrong

As I said, I don't know much C nor do I know the kernel or LLVM internally. From what I see, this commit changed `gnu_inline`, which only appears in the kernel tree once in a relevant place:

```
[adrian@lillia msm-4.14]$ grep -Rn "gnu_inline" .
./Documentation/trace/tracepoint-analysis.txt:307:         :      extern __inline void __attribute__((__gnu_inline__, __always_inline__, _
./include/linux/compiler-gcc.h:69: * Feature detection for gnu_inline (gnu89 extern inline semantics). Either
./include/linux/compiler-gcc.h:75:# define __gnu_inline    __attribute__((gnu_inline))
./include/linux/compiler-gcc.h:77:# define __gnu_inline
./include/linux/compiler-gcc.h:81: * Prefer gnu_inline, so that extern inline functions do not emit an
./include/linux/compiler-gcc.h:87:#define inline inline        __attribute__((unused)) notrace __gnu_inline
./include/linux/string.h:235:#define __FORTIFY_INLINE extern __always_inline __attribute__((gnu_inline))
```

The line in string.h has not been touched in a rather long time and still exists in 5.17+ upstream kernels, so from my perspective the issue should not be related to our changes, but I am unsure about that.

I wanted to try and compare the binary diff of the two compiler's build artifacts, but did not get anywhere with that.

I have all 13 toolchains that I used and compiled on my disk, so if those are needed for comparison, I can xz them and upload them somewhere.
</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJzNWMly20gS_RrqUkEG9uWgg2y33I5x2462Y5aTopYsskYgioFFFPvr52UBECmP5ZFvo2AIBIjK5VXmy8xS3pyuV0mKj_jSkdyrhlbRu1V0M_3_3YlhRx2tkreXj63r-kF4a8UHcfRjY0Tj7kkMXuypHZxvsUoO-HEnH0i0XuhmJHHkZ661eMttd4NQJMae7NjwQqgRHz_-_Q9h6IEaf6Cu58ey713Pq4R1j67d4j3XC9f3I9sEDUPnyIj9CdLwHgvqpL4XbhDGH7HKd-LkR8gR-1Hv-HrwkMl-Yr0a2cr71h9hJURDgVQeD4Mprh2oa2XT86v3-ErN-ZmA5H866fdulZS9UN4PjZeGus0lUjO23-Ddl85D6_7y13_QhJAUe2-cZU9uWtN5ZxZ1q6TqCBa7wXenVXojdsNw6PFlldzis3XDblQb7fe4eScfXKvdW2_o81fc7_v9OtvE2SqpAw4M8WSx-MOJ-ttG3OKpGl1j2HE3sJvHsCmwiBFY-7Y5AVTf6J3EHsAaeS-xm7Ld8stNY6bLw369SmBT_cz578SzbNe6wckGUqHFiPfeb3knAOAqSeJsE21ifJk0iENHvHxgvc_9lhNKm21Y3_ux0zSjcGjkAG_D13k99u92tvl253v4edu4dnxcP1ZFkPcG_zuy_N4gt3xpzf26S_K1okEmy_J1l6VlHCdqeXA3ab7joN7sgUUdgkqya0iCEERlbfJKZSaN4tLEea1sVMuyTOtK1zKOizipDMV1uhEBwZtvHOC-pQm5PuzaFAwsG44zkhxtPVKipc206hL2r0c36BDLnECipSN1501kKT-APfm_hj3Loyj5FdTjKo0qistY5So2pq6LpJSZqSpdmkLn1iaxwgYUvLKjfoS1IgQ4gznn3hK6C1GAIQLw4RG_9yzYvyL3KOzWmQpAVEDTH8BRg-xOYfeO9MSKru0DEUKxg_7Gb_s5BY3HPpXDxEyBN49grmPn2-1G3O-p3_KijrR_IMiFGjCSF7Q_DCcW4VrjtBwmwsRqufAJi2nBsF2wksz0cj-QNN_b3oNJg3njAZr-TRqvfxeNEMRU78ctmLVpguXgLlYykcgUe4iQg2teYEam65kfmLEvX_kQJGnqBiaf4MhM_xCtxoAB7_aSGIa0M2wkKoxsx0AzyvWw_KkehIzooa8hNmvPuXQ6k1R_gAjrNBx-cD0KGVLRThGFZBh5oXjLMfiGA_wjRzP2SQx6HUQIZB6QEO-8BkgWPgfiBYN33oyaq86c1r9I5YsCpnL2VO9oQu24Iy7Q5w2eMhc-BaQWN568mHZ9I34PRb2c6GVOADblAv31q6mLsflKB5HyRiRREnNKwsH3nz-_W2TFNlK6iEtlTBzLzKYqMlUSFRG-RFVtq1hqWdbqSVb0E2GyzjJbRbGW2kiZRSqtirquiirLZGGtjCpVlTquWdhn7P7PDLNppZUpq7yuckVlqigyUZzUZRWbOqkpz6WyVsWLrOwnsnRmTGQpU2kZ1SaDc8amsLRWSa51lOpY53DZvEZWpVIDN2Vdas3XJCMDswoNv43Mizyr6iKqk9fIqus0z-GdqqxNUynTHLsoLQAEckksKcupoLR-jazIRpTG2uZxbMimVCdJllER5yopskLH1kTGRvpFH9_cvAs_cWeRzjkYuBecfyQK9WnJcg7O0IJoJAKCHokX2jYfYh5FEkTQMVtqiCzEQFxk6sXSAiGSprWKKyOLWBEChMpcZVWSWZsrFIgsTpj4Xrb0KRNswomQq1LmNs0RGsooGRFVdVqoPK6jNJNU5K8Qhf2L4rjWWWGBWKFKq8s8i2praq2UimVhKlsmchGVvyxKKYtQM0pRqqwk0gZ1TSJmTVYbGZdJVedIMr2Iql8WlQCnzBSJKnQEZ0sVl0UqozqyWF_bOtEUFaiYi6i4elnWa2svy_rkH_471Z9kXZYByeQJ-m5l16EoLoQ-R40Yjn4JJhYhMZd0gSG5lWLKRMXo_D3exAPW1_s5vKifSss0lGjJlSuUQsxD93JLIcQGsaOe_gdrcx88X9ZgfC6Z3LOg-klm29vXpuFms3ltlm3EZ_gHdh_YvoBCqI7D6eB5ZJqyic4ZA9Tg8NLK_KhXeH2Gzxk619GwuEefs_QJHW1RWbj8PCv8HyXn6dxrxjkPfAINyVRt-8GhjwgN0nnS-1HfcOu2YzdVQJ4q5-bo8sWbHqNdL52ZBsVnLVWYBd8Chg7Plwnw3NvwbPdsAmxOGJY6v5-0QSzRE0_N7k_sBDIpom07ojlFr0u4CR3dzkFfYDJ5OJDsAuFd6Bs64lDUPCBh9zpq6EG2g0BXrem7uswyp890m7-RpnOyXWVRA_Achsll8MvfrZJMYBcOYv0nd7fJhWno92dgN9xxeD3y8C55fOeOo2PN8_XgAcRaAogTxvHN8DjApjQqOR-Wv6fv9MiYibtZj3jgcfbuTg4Y1dFG090d8h6fu7uzMfzsLb_UHOWpf_7wbCNaomY0tMwRc25xZ7near3ZwYQiEAh6c3FLchi5ONCAPOQeiFuxs0pmH9xV9WLw_Lgn9I6D0z1Ps-I3x4nzSyaUeWCHFJp5QBOXfjI-P4Diclfqpwr2Wn3ly_p-SVAVL9h9wVRGl2A9ESai_zlcdmwDuj0nEtMKcTLI9tc0zy7MHsyiz5C9ANvY8iHCBBnr5lB9tfsYb0AfQX2S5s_0393dfv7z24fbf919-PTxw6ffziH9LD5fv5Pfp2z4z2dCs59iMQbMN5NzqGkeNAVKmThBBgpvPPO0209FbaJLenT91EPlm7jEXI2RDRJJ7md-6ef9s0xh-5Pg07UDZ8XDVOoCy4p-F47yJvXMQXKYhimM2zO99edDMwgf254zbDox49jYfF-221nCMA-_cy2c6qtreSY2ztq5gC1FPATHdKoWpiqUFGelHp6085DJZm6JQ-105CPKafb6gRXTCRtgitPzKUi_nFCGY6jFNOgNRwh7Nqu_n1FzbJzvp0agJeIRk7lkcsb1vp0qjJatePyLHdkHieOBJ-npvvd7CmZursw1Gvu0lleDGxq65hlvS-1FtQzCb246vSuyp_PAv4WNvBq75vqXO5CwvXy8kmdFUl3truOirFKpCD1nntpaKb4nzHcyytBMVlcNfmz6a9QWlAkM8cuJa4KScuWu0aslUZqgMYirPN4YDHYql3FiSNoiN6hEIFHXbNiOje-2V911MAkzex_KFML1_COf8W6Ba1AH-XIE3N31e2rUYWzvh6ug_DoY_x_j7CYP">