<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/99848>99848</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            [BOLT][AArch64] Runtime crash with `-split-functions -lite=1`
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
            BOLT
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          paschalis-mpeis
      </td>
    </tr>
</table>

<pre>
    ## Bug summary:
This issue provides a reproducer for a bug that is discussed at:
- #96609

When `-split-functions -lite=1` is used on AArch64 and the below MongoDB setup, it crashes at runtime. Doing a tentative code-layout size estimation hides this issue probably in most cases (the patch provided by #96609). However, the bug should not have occurred without that improvement.
See **point (2)** [here](https://github.com/llvm/llvm-project/pull/96609#issuecomment-2188867516) for more details.

---

# Reproducing Instructions:

### Instruction summary:
- To **reproduce the crash directly**, follow instructions from step (3). You must supply the input binary ([mongod.tar.gz](https://gist.github.com/paschalis-mpeis/9eb878f73e18fb9d3f996ae7c59d4792/raw/e9de7728a517e153d4980225ecac92d2e0923498/mongod.tar.gz)) and the bolt profile ([perf.boltdata](https://gist.githubusercontent.com/paschalis-mpeis/9eb878f73e18fb9d3f996ae7c59d4792/raw/e9de7728a517e153d4980225ecac92d2e0923498/perf.boltdata)) to bolt-optimize the binary and finally use it to run a workload.

- To generate the stage-0 binary, follow instructions from step (1).
- To generate the sampled profile follow instructions from step (2). You must supply the input binary ([mongod.tar.gz](https://gist.github.com/paschalis-mpeis/9eb878f73e18fb9d3f996ae7c59d4792/raw/e9de7728a517e153d4980225ecac92d2e0923498/mongod.tar.gz)).

NOTE: Find all attachments at the end of this post. The mongod binary (in mongod.tar.gz) has stripped debug data due to upload size requirements. If those are needed please follow all steps to get a binary with debug info.


## 1. Generate stage-0 binary from sources:

Install required software and compile mongodb from sources.  

The patch [mongod.patch](https://gist.githubusercontent.com/paschalis-mpeis/9eb878f73e18fb9d3f996ae7c59d4792/raw/e9de7728a517e153d4980225ecac92d2e0923498/mongo.patch) needs to be applied. It deals with a compilation error on a library dependency.

```bash
# 1. pre-requisites
sudo apt-get install build-essential
  
# if ran into errrors, you may also need:
sudo apt install \
    libcurl4-openssl-dev \
    liblzma-dev \
 libssl-dev \
    python3-pip \
    python3-venv \
 openjdk-21-jdk
  
# 2. get mongo sources  
git clone --branch r7.0.5 https://github.com/mongodb/mongo.git
  
# 3. setup python environment
python3 -m venv mongo-db-venv --prompt mongo
source mongo-db-venv/bin/activate
  
# 4. install mongod dependencies and apply needed patch  
cd mongo
pip install --upgrade pip
pip install -r etc/pip/compile-requirements.txt
git am mongo.patch # apply patch
 
# 5. compile stage-0 binary
python3 buildscripts/scons.py install-mongod \
  CC=clang-18 CXX=clang++-18 \
 CCFLAGS="-fno-omit-frame-pointer -Wno-deprecated-non-prototype -Wno-enum-constexpr-conversion" \
  LINKFLAGS="-Wl,-q"
  
cd .. # back to parent dir
```
  
## 2. Generate profile and optimize with BOLT

### 2.a. Get YCSB harness

Needed to load the database with some dummy data, and then run a simple workload that will demonstrate the crash.  

```bash
# 1. get maven
wget https://dlcdn.apache.org/maven/maven-3/3.9.6/binaries/apache-maven-3.9.6-bin.tar.gz
 
mkdir -p maven
tar -xf apache-maven-*-bin.tar.gz -C maven
  
# 2. Get and build YSCB from sources:  
git clone --branch production https://github.com/mongodb-labs/YCSB.git
orig_dir=$(pwd)
cd YCSB/ycsb-mongodb  
$orig_dir/maven/apache-maven-3.9.6/bin/mvn clean package
```
  

### 2.b. Build a dataset
```bash

# 1. start the database service (in one tab)
mkdir mongo/dataset # empty dataset
./mongo/build/install/bin/mongod --dbpath=./mongo/datase 
 
# 2. load dataset (in a second tab)  
cd YCSB/ycsb-mongodb
bin/ycsb \
    load mongodb -P workloads/workloada \
    -s \
    -p recordcount=20000 \
    -p operationcount=10000
  
# 3. kill the database service (control-d)
```
  

### 2.c. Record a perf profile
```bash
# 1. start the database service (in one tab)
mkdir mongo/dataset # empty dataset
./mongo/build/install/bin/mongod --dbpath=./mongo/dataset
  
# 2. find it's process id (VERIFY with ps): 
# (sample command)  
PID=$(ps uax | grep /build/install/bin/mongod | grep -v grep | awk '{print $2 }')
  
# 3. start recording samples (in another tab)  
perf record -e cycles:uP -p $PID
  
# 4. start benchmarking (in another tab)  
bin/ycsb run mongodb -P workloads/workloada \
                -s \
                -p recordcount=20000 \
                -p operationcount=10000
  
# 5. stop perf record once YCSB is done, and  
#    also kill the database service.
```



## 3. Generate BOLT'ed binary

### 3.a. Compile LLVM sources:

Checkout latest main or if checking out the pull request [#96609](https://github.com/llvm/llvm-project/pull/96609) ensure you don't include the fixes.  
  
```bash
# assume you cloned llvm/llvm-project at:  
llvm=PATH_TO_LLVM_SOURCES
  
# 1. checkout latest main. otherwise, checkout PR96609, and go  
# back 2 commits so you don't include the fix.  
gh pr checkout 96609
git checkout HEAD~2
 
# 2. compile BOLT from sources (adjust commands):  
cmake -G Ninja \
        ... \
        -DLLVM_TARGETS_TO_BUILD=AArch64 \
 -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_ASSERTIONS=ON \
 -DLLVM_ENABLE_PROJECTS='clang;bolt;lld' \
 -DCMAKE_EXPORT_COMPILE_COMMANDS=1 \
        -DLLVM_USE_LINKER=lld
 
ninja ...
```
  

### 3.b. Generate BOLT'ed binary
```bash
# 1. convert profile:  
$llvm/build/bin/perf2bolt -p perf.data -o perf.boltdata --nl mongo/build/install/bin/mongod
  
# 2. compile bolted binary:  
$llvm/build/bin/llvm-bolt mongo/build/install/bin/mongod -o mongo/build/install/bin/mongod.bolt --data perf.boltdata -lite=1 -split-functions
```


### 3.c. Observe runtime crash..
```bash
# 1. start the database service (in one tab) 
./mongo/build/install/bin/mongod.bolt --dbpath=./mongo/dataset  
 
# 2. start benchmarking  
cd YCSB/ycsb-mongodb
./bin/ycsb run mongodb -P workloads/workloada \
                -s -threads 1 \
 -p recordcount=20000 \
                -p operationcount=10000
```


---


# Binaries and aux files:
All files are hosted on this Github Gist: [9eb878f73e18fb9d3f996ae7c59d4792](https://gist.github.com/paschalis-mpeis/9eb878f73e18fb9d3f996ae7c59d4792).


| Binary | Description                               | Details |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- | ---------------------------------------------- |
| [mongod.tar.gz](https://gist.github.com/paschalis-mpeis/9eb878f73e18fb9d3f996ae7c59d4792/raw/e9de7728a517e153d4980225ecac92d2e0923498/mongod.tar.gz) | Stage-0 binary of mongod service.         | Extract with `tar -xvf mongod.tar.gz`          |
| [perf.boltdata](https://gist.githubusercontent.com/paschalis-mpeis/9eb878f73e18fb9d3f996ae7c59d4792/raw/e9de7728a517e153d4980225ecac92d2e0923498/perf.boltdata) | Converted fdata BOLT profile.             | Needed for optimizing mongod binary with BOLT. |
| [mongod.patch](https://gist.githubusercontent.com/paschalis-mpeis/9eb878f73e18fb9d3f996ae7c59d4792/raw/e9de7728a517e153d4980225ecac92d2e0923498/mongo.patch) | Patch for mongodb sources.                | Needed for compiling mongodb with clang-18.    |
| [llvm-bolt.txt](https://gist.githubusercontent.com/paschalis-mpeis/9eb878f73e18fb9d3f996ae7c59d4792/raw/e9de7728a517e153d4980225ecac92d2e0923498/llvm-bolt.txt) | Output of `llvm-bolt`.                    | Not needed for compilation/optimization.       |
| [perf2bolt.txt](https://gist.githubusercontent.com/paschalis-mpeis/9eb878f73e18fb9d3f996ae7c59d4792/raw/e9de7728a517e153d4980225ecac92d2e0923498/perf2bolt.txt) | Output of `perf2bolt` profile conversion. | Not needed for compilation/optimization.       |

| Binary  | Checksum | 
| ------- | -------------------------------- | 
| mongodb | f379208ed066b85dfb30d317e1758e9e |


</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJzcWk1z2ziT_jXwpYssCdTnwQdLsjLZN4ldtuedySkFEi0JMUnwBUA7msP-9q0GSIqSZSepmd2aWVcqEkmw0R9PP90AJKxV2xLxko0XbLy6ELXbaXNZCZvtRK5sVFSo7EWq5f6S8YTxBBb1FmxdFMLsWXLFBis2uHrYKQvK2hqhMvpJSbQgwGBltKwzNLDRBgSk9RbcTjhQFqSyWW0tShCukxMB48l8MhnMw3X4_7cdlsAmg8hWuXLRpi4zp3RpIcqVQ5ashmwyIJk1idMlXF2ZbDcZgSgluB1Cirl-ho-63OrVAiy6umJ8CcpBZoTdkbIOTF06VWAMK63KLQhwWDrh1BNCpiVGudjr2oFVfyCgdaoQpATsvLHuyAGpSPM9qBIKbR1kwqIFxmekSiVctmudJCHdHyzm8xh-0c_4hIa084qTr3e6ziWU2sFOPCHoLKuNQQnPyu1Io-DRgmRigaWLg9fuEYHxK8avKq1KRwpwxufhFrDxYocG2XjF-GznXGUpCHzN-Hqr3K5O40wXjK_z_Kn9iCqjv2LmGF9XdZ4zvm70TrzhmS5o9ogPZ7PZZDoeThif-8AX2iBIdELlNu4HNoqi_iWh667BDIXgfWmdqUOsO4h0Qxs49ga9gGUED7rxQYdF71cfdpDKYObyfRhBPt_onJCiehPDxugCrMOKPJj4KH3WNRS1dWDrqsr3XqQqq9pBqkphKKYzNl4UBDgZO2Hi7R_nPW1dfOTuk8QjH2M6m8420wSHs006l8lmPp8InGbjuRxN55zxtRHPjK9xLnE65TMxHk5xOE7kaD4bcD7GTGRzLjkO5jwZzWeMr48VI0zMD7mic0f43KgcGzsqNJuY7kvhxHfsqC2aTJeUO_-nJh3rGExy2lsT6cqpgtLW2xciROZuVCnyfE-0QVzgNHEACHjW5jHXQh5jlbC0xRKNcEGSdWKL0aCR-EPwGRJ8XpUniipH2Tn_u9L4_1cwHjn-083DNUuuYK1KCSLPQTgnsh1xjedtshdLCXoTWLjS1sXwsEMIkntu8Ix8PBvshAXrjKoqlCCRGJcgBLJGQkRdERIC6xv8T62M51gbw3uaT1sEYRBKRKLzKkdhu8iRshQrS4K26KgCBl2IupvJVLnRR_b2CQ6GMbxrMXKMtwYKujYZnrIjcSJN3igsweqNeyY9CfaZLirCV3BFeiQoBugLeugq1gFB_vpvygJexUZDPvdR8c5PEURV5QplDO8dSBS5DUEQjTtCMUdjtKEOQkCuUkNullhhKbHM9sdRmgzCv1TY3aF8DWOoDEbe8VY5tOGRraUGUbmIUKCa6KS1ymWE1mLplMjDyM7_PAG1ASNKUKXTpJnRxhLL7CnhxR5EbrU3sYt-O003BRsvW7FAFmW1yUeRrrC0No8kPr0Ykf9RiJMHuUrPjq72bqfLJKpUdf7BE5b9V2jar_Ix4sPoq3x8aS6PfZL4GLZ4bJ9vqVfLdYkQRakRZbYDM40H8Rje6F0agHe42Cr3ctYkDg1hozVg-aSMLinJw6DGGogK8AZ5WZFMg3kRdUVF1ajdhMHrfjyQ8XWqSsbXInPqSTh8qcko7uLWMFeHPUUtKtGfZ_iWbHxiNgIy2deAQtLKiqK62hohESpVnXlsAF1GGaoqxtcNOURHXOe-uUMYRAG9NKP2tVErpF0w62DVOO4I56RgHnnXJ4PNjKockYPNdGnjat-qGTUe6eFsuWTJKstFuY2GM1j-_nt7yfiC8QXdPIxeLtcfrt7ds2TFOI82pY50QUsJIwqMfH-MBqLfSh1JrAxmwqGMSl1SdJ12-wrDUyzrIiLdHH6rDH17QmOVLhnnfeU-vP_0r_6Mv-WML6P_MM6P455JiGPvw1Rkj0RVlTBYOupNT4jmBWCanOnqQ9s3EFC6lseT3OLmw8P57pnHgiQ4-Ly8X8BOmBKtPaq-AWtOg6-DVGypPKZU5rxsqwsEWRfFHkLrtWxbybJpp6yivqbrqsJ65VnlOUgsyJVdB-Tb8pMS9DrRerIQT1iGu890fcwGMs9kGYtKZDuMtdkSE_gXms8oYXydxPN4EtJTGIWEvvBG1Iyh51GqyrZp6CO8eJTKQFT1FXHCQPRtA0dSGL_qyYBo2X_jhAUpHuRDnxPw-X65OC32b9BiWOaEpel3qTHKRUoGU_QP9KiN2n4hABJ2R4zPqmdJjVmLWBrN-Hqf2TRqm4jOglH39sHZL_3Z0WHxVEKWoyihEtmj2OLbqH8B3zSGhXeT8PCz6F6HTR881gnjjuFs0TypDJtGkZzqRNrZHSIdOJavm7l85mJRuf3x7HHrYrKT1GN83TDZwfLAaFEk00q4HUtW_beCOHhBpzwOiXhQgJQVYDHTlHZe4x67nIlVeBaUoPsnPQBJb6Ma3XZpSzBpv4vjVyJ7cl2BwUwbmem6dCxZ8cFgMHgxRlfEW0qX7bAhDTtboR-JLV4LFjWaRufRAaM_jJ4shjuvKQigFWTLod-jnn8Ges50OzymVa8E5RifWjI3Q2tBSdL739d379efA61XlnRP-tBjfBaWqFTQC1HKHtJu368ObGGhFt-ATZewNX6l-j0ruqHRU_PKdAni-REYn7LpojJh_2rEgU1XdK9162kj58MSsKfKbbOitm2SlNrt0JzkiA97eAUihGyf5Z5i61vCKOMjMu1crxYmS7HMdoUwjzTfW_P08o3K4s9lWP_vNNuOnv1A5p2M_9EsHJPFuoK-v3SZYegclAWpS2yrf-89gLBQeTWF47M5--qKOOl1PL6v4VOURy3laZon1OMsmx70w4d_f3xl1bzcYfaoawe5cGipt6BENrQMy-gRRTjstyJUdbO8poFsvOg2cf-K_dQ5YGlrg36lJ6m3nNKSLstrGfqkjfrWW6p33j5PV8LaugiyfKsg4YwOYRO-leQHJKvbq4dfvjzcfCGffbm_-fVueX3_EhjDOLjnxHMx-Cx4Vtajohtye9dYGZCy1X1Zvgnmnl-Us2D1Wy7oHLClrucwQ-_8wPdH7f1frq9W_83PVdR2gUJ4Omq1KKOF_Fpb13JeS4tthS3EI0L0Dj6p8uuZpI2puz-9Ga28Rx-u7t5dP9yThxe_vv9ABNoeXBxeiVbLj1f_ug4jvjx8vr1myeoOwz7Ta5KvP10tPlx_ubq_v757eH_ziVYhN5-OpPbH3d7d_Nf18iGsVaZhDZUsUp07lixyIu7pGY2uf7-9uXv4srz5ePv-wzV9frz6tCIhw1cV-_X--gstja7vWLIiyf1glN6FcXyeEF4r4gm1gN9jhFcLeVjAdRvevcgyPmrypC1egcGJ_7jfJI8CGcZ-uzDScLQFDVFU5vBjVfxsmW4xSQIP5nxfP5_VXr8fbSH0D46Mg9WRN-_E2PYgDk7P6L5P7r1IZjHcpFQWsD2Na5aEp5D4Mw0Z_GyT1dn9RqcFZzv1My3CD3TmcTf_X9MtRG5nUEgL_bz8izuFNwJ8esjXOWjRLLjDzlb9DSgDD0X5Ks_DHb_BvtPWhcNdv8n_ztdUeKesr1tsvPjudvL_2pHH_Nz2_XQZ7Nv7XnaFYWuLVuVv_4XR_qiUvh-kRX_vP_gpHX9udPfKwRv_pAMtb-z98eGN3rR7vG0XfISA62_OiMyFpRibDMKW0tPm5PhqMjgCzpF7_omHtt72ZSjJKGHja4tvypr6HL_IlWZ_cqNNu-dJLHt88tftgcavgegfdKZFit_6nffw04pQGA7Hdy_ppOei0FQcPJQG37S76PE5IHUNhT8G-Hv66FjHxks3tatqR6nGJoNuBJsMXnip85R27eHKwVu-5DG-buDlL-M3ko7_zX11rOM5X3UjiGDaU4XDUUf8Z311WiFD2tMyzdaFv3hR9n6oYJy82kKcvm-S6ZwPZigHk0k6G8tNmgxkQr6ajmc4x1PtLuRlIufJXFzg5XDKh5NkOp2ML3aX6XCUjuRgkIw2E0ySdDjP5ulkNBxJnG42m9GFuuQDPhpMOR_MRnw0jpMM54PpdJaM0ozPU8lGAyyEymOCZKzN9sL_fOpyPp-NZhe5SDG3_ud4nIfVDLUuF-YyILjeWjYa5Mo6exDglMv9T_j8C-MVGy-alSQbr-Cu30p3JeXNH9Nd1Ca__OldDG-Hh6035emS_08AAAD__8SfGlU">