<table border="1" cellspacing="0" cellpadding="8">
<tr>
<th>Issue</th>
<td>
<a href=https://github.com/llvm/llvm-project/issues/56209>56209</a>
</td>
</tr>
<tr>
<th>Summary</th>
<td>
bolt: --instrument merges files not correctly and gets overwritten from previous profile
</td>
</tr>
<tr>
<th>Labels</th>
<td>
</td>
</tr>
<tr>
<th>Assignees</th>
<td>
</td>
</tr>
<tr>
<th>Reporter</th>
<td>
ptr1337
</td>
</tr>
</table>
<pre>
Hey,
As already mentioned here some time ago, https://github.com/ClangBuiltLinux/tc-build/issues/155#issuecomment-1106831385 the bug seems still present in the main branch.
If instrumenting a binary with:
```
llvm-bolt \
--instrument \
--instrumentation-file=/tmp/bolt.fdata \
${BINARY} \
-o ${BINARY}.inst
```
The output of the profile does not get correctly merged automatically, only the latest process seems to profiled correctly.
For example if running a workload with the instrumented binary will result into a profile which has very less branches.
example:
```
BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: f595b51f502b2f3c97d30e826784159438bde9c4
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: enabling relocation mode
BOLT-INFO: enabling lite mode
BOLT-WARNING: Failed to analyze 2424 relocations
BOLT-INFO: pre-processing profile using branch profile reader
BOLT-INFO: forcing -jump-tables=move as PIC jump table was detected in function _ZN4llvm3sys4path11is_absoluteERKNS_5TwineENS1_5StyleE
BOLT-INFO: 1315 out of 96706 functions in the binary (1.4%) have non-empty execution profile
BOLT-INFO: 3 functions with profile could not be optimized
BOLT-INFO: profile for 1 objects was ignored
BOLT-INFO: the input contains 176 (dynamic count : 5974) opportunities for macro-fusion optimization. Will fix instances on a hot path.
BOLT-INFO: 13166 instructions were shortened
BOLT-INFO: removed 11 empty blocks
BOLT-INFO: ICF folded 316 out of 96983 functions in 4 passes. 0 functions had jump tables.
BOLT-INFO: Removing all identical functions will save 94.04 KB of code space. Folded functions were called 3316 times based on profile.
BOLT-INFO: basic block reordering modified layout of 330 (0.34%) functions
BOLT-INFO: UCE removed 1 blocks and 7 bytes of code.
BOLT-INFO: splitting separates 560689 hot bytes from 346146 cold bytes (61.83% of split functions is hot).
BOLT-INFO: 3 Functions were reordered by LoopInversionPass
BOLT-INFO: hfsort+ reduced the number of chains from 1002 to 552
BOLT-INFO: program-wide dynostats after all optimizations before SCTC and FOP:
553848 : executed forward branches
64210 : taken forward branches
191942 : executed backward branches
100756 : taken backward branches
59984 : executed unconditional branches
251734 : all function calls
103854 : indirect calls
96734 : PLT calls
5738078 : executed instructions
1228106 : executed load instructions
632336 : executed store instructions
988 : taken jump table branches
0 : taken unknown indirect branches
805774 : total branches
224950 : taken branches
580824 : non-taken conditional branches
164966 : taken conditional branches
745790 : all conditional branches
525170 : executed forward branches (-5.2%)
25848 : taken forward branches (-59.7%)
220620 : executed backward branches (+14.9%)
95176 : taken backward branches (-5.5%)
30714 : executed unconditional branches (-48.8%)
251734 : all function calls (=)
103854 : indirect calls (=)
96734 : PLT calls (=)
5676670 : executed instructions (-1.1%)
1228106 : executed load instructions (=)
632336 : executed store instructions (=)
988 : taken jump table branches (=)
0 : taken unknown indirect branches (=)
776504 : total branches (-3.6%)
151738 : taken branches (-32.5%)
624766 : non-taken conditional branches (+7.6%)
121024 : taken conditional branches (-26.6%)
745790 : all conditional branches (=)
BOLT-INFO: SCTC: patched 4 tail calls (4 forward) tail calls (0 backward) from a total of 4 while removing 0 double jumps and removing 4 basic blocks totalling 20 bytes of code. CTCs total execution count is 36 and the number of times CTCs are taken is 8.
BOLT-INFO: padding code to 0x5000000 to accommodate hot text
BOLT-INFO: setting __hot_start to 0x4e00000
BOLT-INFO: setting __hot_end to 0x4e956f4
```
running the same workload with following options:
```
llvm-bolt \
--instrument \
--instrumentation-file-append-pid \
--instrumentation-file=~/data/bolt.fdata \
${BINARY} \
-o ${BINARY}.inst
```
Will result into a massive amount of profiles which uses alot of space. After compiling clang with the instrumented binary results into a over 100GB of fdata files.
After merging them with `merge-fdata` and using the profile with llvm-bolt shows following stats:
```
BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: f595b51f502b2f3c97d30e826784159438bde9c4
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: enabling relocation mode
BOLT-INFO: enabling lite mode
BOLT-WARNING: Failed to analyze 2424 relocations
BOLT-INFO: pre-processing profile using branch profile reader
BOLT-INFO: forcing -jump-tables=move as PIC jump table was detected in function _ZN5clang4Decl19castFromDeclContextEPKNS_11DeclContextE
BOLT-WARNING: 7 collisions detected while hashing binary objects. Use -v=1 to see the list.
BOLT-INFO: 16889 out of 96706 functions in the binary (17.5%) have non-empty execution profile
BOLT-INFO: 495 functions with profile could not be optimized
BOLT-WARNING: 9 (0.1% of all profiled) functions have invalid (possibly stale) profile. Use -report-stale to see the list.
BOLT-WARNING: 695817 out of 2165380220666 samples in the binary (0.0%) belong to functions with invalid (possibly stale) profile.
BOLT-INFO: profile for 1 objects was ignored
BOLT-INFO: the input contains 9616 (dynamic count : 19121657995) opportunities for macro-fusion optimization. Will fix instances on a hot path.
BOLT-INFO: 782089 instructions were shortened
BOLT-INFO: removed 1458 empty blocks
BOLT-INFO: ICF folded 1687 out of 96983 functions in 5 passes. 2 functions had jump tables.
BOLT-INFO: Removing all identical functions will save 192.87 KB of code space. Folded functions were called 6708861534 times based on profile.
BOLT-INFO: basic block reordering modified layout of 9503 (9.97%) functions
BOLT-INFO: UCE removed 1 blocks and 7 bytes of code.
BOLT-INFO: splitting separates 11629561 hot bytes from 11335899 cold bytes (50.64% of split functions is hot).
BOLT-INFO: 223 Functions were reordered by LoopInversionPass
BOLT-INFO: hfsort+ reduced the number of chains from 15697 to 7487
BOLT-INFO: program-wide dynostats after all optimizations before SCTC and FOP:
1345852442580 : executed forward branches
177695950568 : taken forward branches
279784019546 : executed backward branches
165552345783 : taken backward branches
80707139554 : executed unconditional branches
111711292644 : all function calls
38436135059 : indirect calls
20540549655 : PLT calls
10555075375899 : executed instructions
2627819906958 : executed load instructions
1279424679765 : executed store instructions
16846108018 : taken jump table branches
0 : taken unknown indirect branches
1706343601680 : total branches
423955435905 : taken branches
1282388165775 : non-taken conditional branches
343248296351 : taken conditional branches
1625636462126 : all conditional branches
1280317632631 : executed forward branches (-4.9%)
78974541817 : taken forward branches (-55.6%)
345318829495 : executed backward branches (+23.4%)
155608698163 : taken backward branches (-6.0%)
60656090585 : executed unconditional branches (-24.8%)
111711292644 : all function calls (=)
38436135059 : indirect calls (=)
20540549655 : PLT calls (=)
10505269253372 : executed instructions (-0.5%)
2627819906958 : executed load instructions (=)
1279424679765 : executed store instructions (=)
16846108018 : taken jump table branches (=)
0 : taken unknown indirect branches (=)
1686292552711 : total branches (-1.2%)
295239330565 : taken branches (-30.4%)
1391053222146 : non-taken conditional branches (+8.5%)
234583239980 : taken conditional branches (-31.7%)
1625636462126 : all conditional branches (=)
BOLT-INFO: SCTC: patched 78 tail calls (70 forward) tail calls (8 backward) from a total of 78 while removing 12 double jumps and removing 65 basic blocks totalling 325 bytes of code. CTCs total execution count is 54542385 and the number of times CTCs are taken is 51777916.
BOLT-INFO: setting __hot_start to 0x4e00000
```
Think the discussion at https://github.com/ClangBuiltLinux/tc-build/issues/155 could be also helpful since this bug is there a bit discussed.
The last tests ive run was with commit https://github.com/llvm/llvm-project/commit/96f6ec5090c2f7a1e4804693cbb84c29c574b3de and also their seems everything equal:
Regards.
</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJztWktz20YS_jXUZYqoeWAeOOhgyVZWlZTjsp1K7V5UA2AoIgYBLgawpPz6dA9AimRAkHStvZewVBRBTPf0e7o_Iq3zl-t_uZcZv53RtzP6pn9_44ktG2fzF7JyVVvUlcvJ0jWO-HrlSFvAm32sgYos23btZ-LNjN_B32PRLrs0yuoVXNyWtnq86Yqy_aWoumf4ps3mKVzn8LHwvnMePjApZ1yESyDD7eaMUWUEE0aSdulI2j0S79zKE98WZUnWjfOwjBRVuL2y8CFtbJUto10l7hewwrdNF1SoHoklaVHZ5oU8gZQo8s7imaLDX7gsy6-reVqXLZnJwTJkeM3nr1yn71o03HxRlG4m3qL2qzW8I9dokdvW_p16xuOZvrm5f__m479n-u0I-_pwTYT7jerQv38GC9Vdu-5aUi-CvdZNjTKRvHaeVHVLHl1LsrppXNaW6PDmEbxtu7ZegQaZLUsMD1JXcBPpS9s63yKbzHk_eKatN3zzV14R6WW4qxvinu1qDbsWC9J0VdX746luvpS1zYNHAvNX6wGjrbvA6eDzrkSfw052q8PTssiWZGk9-epgZYkC9aHg_F4wDNtPe_3m118-z-_f3_0Ky4hf2gZkqNM_QBcCGqxrXwSPFlXu1g7eIADcs8u61qZoT9fCSpeP8PpsGzSybbJlgYu6BiUhz0Y9qHhkPV6gRh62w-uFTGQq2UJSnvKFyBKdC-oMV9rETCaxMGnukmyM1aJowFngxDojNs8btFDhCX0e09lVoAm6pnGwPsQvWdW5m1pagkaHi35_8_H9_fufcN2dDTGBXqts-fKnIzzm8c4GfoQ5ZPh8iC_cYuPtLlz1_t1-iWXKNWOa102G6-d_dKv1PPgICtXbVf0VipcnH-5vCd4hvfee4KuNB7GyLLoqC_o__Od9jOVA-Bcfr227ZKzwDzb1ddm17t3Hn99_epCfn4rKvXv_iT3IT-1L6d6NyMMEk5iKmIeJ0lRt9_CbUjYE_IwbFsUzDoUxgeAGeSsIO7daty9DwKFggwFGdhI7nENmbWyV1V2Zh5xPoSqsoY4Xf44G7IYAbEjYkAM-2Kh4rOpmlKZPXyw0WQ3FDzKZMK1Qmfylsqsiw-2xZsJamegYlavX67ppuwoSC4oR7rayWVPPFx2G_kbEECcR-R3rwKJ4DkUCYgAoYI0lS9AHHRONG12poapsLBLOsSXs66pRRRqHMZITxkhv8xRi9ctYnN7f3oHQZQ6LYZ9X5yZG7Ds3BgG9h5pE6M6NJVS-1xD0Y_J_RFlCsQTdCyw5WJH3_As3PMZIEkc0Jj_foAgZ5CPxa5u5iNz1Au6QoAGwrqPYKDee51A3rcd6tw2sMXFgDfgx2APMVDeQeSgcpH-xKIC6tC-DEYSg6HoaiU0gbwUY4fvb7btXsw_2hoKRE03SlxYd3es0JpNfQwkKB7x3a9vg6USkgg4iCZHR0y-aekVErFisgBEkQf81SKhYZASIiFsEVrue88gChB_bV5C7fZMOBsGD64X8Utfr-2qo4R_A-SMclgsPUTjjN0CadxmWScihqlulrgkaL0MWBdkZpRyrqJR8PF8fG7uaP0GIEEi3GhIEEtYuWuCEobObSeBqB6nmyKfbz7fByne_fjg4GcneS0phYhMyt68_GE9182SbfHvcjtERFXNGA11rv7jqLCKWsCTm-5ulNvtympBSLdXObmdRQS1KTLy_Hfi1rvJw2kOyTVFzybToqdHM21MD0-uIkNDW9gTQRxTYKB1fjOfEwP0DdATH1kktDNUH_tmteSM0jMMhQ9U-TejFThASJbgQB4S-xXg6RYkKGbPjoJ3zd9JF-NoNo676UtVP1asFp6gNlVr3Rmzr9pQ_eZzI3a2mFktDDe854wHdE5wbOUzFidqN1nMJdSx1Qrchd5xsVGQMWDqdyVgV5zLifd0edQiXm3owntc9iyTSx3lwThWnJ9Ic-UB9ZHGUTEiTyNBoHM_7QSM5wUNQzc6qAoFXbCIzodpUUQgq4Tw4Snq0PEyTjRWKSQqptFKHgbDXJqGaLGJHtTyvgEyLfV4pOaH6GUXlJIczy8s0H62VpGOFJthSROp4xDCMGDNSdnpSPhW6isd6KCTTVWhIJj0pCJzXQ0mbZjTnaorPGUXq0JgjbQ02KKG9sS1Q5NBGw1xRvoZ3vCk82F_u36LbKhB6T2yh7OAW6KxiRA7C8Di015TkdYchg9HTt57be_Fu2-t7JmH0hfK1350SkHdYsDOo9WMP9JIQ6sh4v8nrm-9AaCHwe6vDYjPWc65hhMedQ38P3SB9ljS8woCdIYBW59ABh863dc_tWL_s-m754QEWPUCj2LQ9q9gFVidJXJVvCBKpFvEE-rSBelBlb1fuAPGB2amsn3AB9qfYNfxIVG5u1wjjzNdFfgmQp9_N-B3idz8SzPv97xjYCoaKArGMVYgvCKVhcPMDLNbBwAnZV7f9aBOGwTdhIoA4WRchhDPEaKfht35Xv9kWZrQGW-2fwqDZ6x52HeK13wFRxMHxq549qBKgxXkggauQCz2kswtLhsWvnoZB_cnvBEoYbP6B8f6B8S6H8WQI9vity0qWZNa3d3Aq4NVtXWGtfPcBsTzGdr8at4ZGDKEsfOhOttv1R8rS-mXQr8-eATuLyG_ekflXkJyhFb1zPZhe-HYUt1LGJOejhXrTIHwLXAgTzzcDhjtGSXq8hw1Yig2_1PQ_CezBP72IRfXVllh4uYGM9EVavmBug4CwdoNA9UZrHIKE83B3ynY7sqhEglE29uNMSZiQceKARsmHXwJGrEgjOhgxhbDGslQf2uUsqb8fnpoodgRQZQlDLXWSyB-Cq2rDKQToNwKrsTQXQKuQC3oCW5VbbJV_N2yVJTwCIS4EVyFtjVFMwmT2PSDWRFKB0ZBEif4_YKyMKQ79HztEWRkTQpokOcBZJY1U_C04K-c_GGmVKtGY-zo2-gdirUxAWkgex1yaEwjNfh_JYPBMoLWgUk1hMvtEXCfQdVCWyFidAGEOdlNSSg6yakjE87FWQzXVTCRSfgPWyhjTjPGEq_hMrFWYWCgmwCbJaayVUxnDXwKaHcdaGQW9qZZCh-A-D2vlimvDkoTigXQJ1srAP9BgKfCSkhdirVAvY8WooezHYa0MOhQBJqewOT2JtcY8BIKQCZWnsFbGDRfG4OGm5cVYK8jEY8MTJSS7BGuF2iaVULHijKtLsVYQmQqmleBKsDOw1mPopjaJjmXMsJU5hbXKcTQG8lQwAwbAJu8srJWLzU_PBzkopaIGjl-mJvM-iKM2rdSBRooq4AJeN_KcKtADTfE41npGUTiK102Xh6NkRwvFOAWUDCq5SrgUQvPjJSOoSUcxvksKyBGxLyolR1U_u6h8Z6wV5IC2g8MRBM4_hrWy8R8voF2BwiMEHJRjZafHWulo-MPJBd4UnHMWX4C1mnHgFo9PI0CWxNATVakXio3-knJ-kfoGrFWbA0RV0-Noq5lEW4HVAdzK-ATeCs45ArgKaB8vQlwllE-OzxGej7tKaKd0wtRoG3wWdHrkObyi-hJEyAufdT7MYbb9nzw_OYzpMKLb0tdk6cr1ooPJpYBZDrYEpfARSvjXhmc48TnIdiOGy6N9MfEBP4_gsUfMD2afpqvCpBoGYESYi0mpEbob_iH4g5MuXPZ08CFRC-UyCWdAxhfaMhcbGqtEZGlq4ownmdRxKqCzRo8FdUDoohkeM3T4mF8b8BX3386WB530R_cIIeijq_xa5IlI7FVbtKW7RiQR_beHEgc00vfYZYA4Xp-BxL0fHeiPYOdTA6MPnrsY1evGfS3qzm8n-quuKa8vNsfWe1Jxmlwtr2Ml0lzbRaa5zE3sMpOm2qVMOC5yQ81VaVNX-uuZvJnJt1fFNQ9wBocDRhiqIpUaZ_CQiEW60CmfxdStIEUj3Diqm8er5roHVbtHDzcRO_GvNxFKfqyc2_C3XQtz_PW6bWCY01dB3Osg61_uxkLm">