[llvm] [BOLT] Add an option to remove pseudo probe sections (PR #218662)

[Jinjie Huang via llvm-commits]

Jinjie-Huang wrote:

> From what I remember, llvm-profdata tool will read this section, but if you're shipping a binary with no pseudoprobes, you can't match the samples anymore.

@rafaelauler Yes, CSSPGO uses these two sections along with debug_info to generate the profile.

Behind this, there may be an interesting question worth discussing: for a CSSPGO + BOLT optimization continues profiling pipeline, iteratively collecting CSSPGO profiles using a post-BOLT binary actually means we need to re-deploy the CSSPGO-optimized version to the production environment to update the BOLT profile. However, due to some performance regressions in this intermediate version (even with BOLT profile inference), we often cannot directly deploy it to production on a large scale for BOLT data collection. In other words, for a CSSPGO + BOLT optimized binary, we often have to make an either-or choice between updating the CSSPGO profile (with `.pseudo_probe` and `.pseudo_probe_desc`) and updating the BOLT profile (with BAT).

To address this issue, we developed a profiling pipeline as a supplement: automatically build from scratch every time. This also ensures that the compiler flags for the target profiling binary meet our requirements (since we want the final binary to be as small as possible, the final CSSPGO+BOLT output of some services might not contain pseudo_probe and debug_info). The overall workflow is roughly: build pre-CSSPGO -> small-scale distributed deployment + collection -> build CSSPGO (pre-BOLT) -> another round of distributed deployment + collection -> build CSSPGO + BOLT.

For this pipeline, the `.pseudo_probe` and `.pseudo_probe_desc` sections in the final CSSPGO + BOLT binary are relatively less important (debug_info still matters for debugging purposes). Currently, we just use objcopy to strip these sections from the pre-BOLT binary (which usually takes a few tens of seconds) to avoid the update overhead in BOLT.

Overall, I'm fairly neutral on this patch itself, as the objcopy approach already works reasonably well for the case.

https://github.com/llvm/llvm-project/pull/218662