[llvm] [ICP] Introduce a hot function cutoff threshold for ICP (PR #208060)

[Congzhe Cao via llvm-commits]

CongzheUalberta wrote:

> > The gain from ICP comes from the following two folds but not relevant to executed counts: 1) By promoting virtual dispatch to direct call, it gets rids of two loads that very often have cache misses which significantly stalls the pipeline since CPU does not know what instructions to fetch in terms of the target function. 2) It further enables more optimizations like I mentioned above.
> 
> IIRC we still load all the way to the function pointer when doing ICP. There is now an option to compare directly against the vtable ptr (#81442) but that's default off and needs `-fwhole-program-vtables`. I've found that the branch itself is also non-trivial (https://discourse.llvm.org/t/rfc-safer-whole-program-class-hierarchy-analysis/65144/11 showed ~0.5% from effectively removing these branches). So that leads me to believe most of the gains come from (2).
> 

Yes we still load all the way to the function pointer. My apologies I should have been clearer: for (1) since virtual dispatches are converted to branches followed by direct calls, the CPU branch predictor works much better with prediction, and instructions can be fetched directly from the target addresses without pipeline stalls. Whereas with virtual dispatch it does not know where to fetch the next instructions until the load actually completes and hence the pipeline has to stall or the branch predictor likely makes false predictions and CPU has to flush the pipeline. With different specific hardware what is happening might be a bit different under the hood but that is roughly the reasoning. So yes some gains did come from (2) but (1) also provides gains.

We measured with vtable type profiling too and we saw that vtable type profiling could provide ~0.5% extra gain for the workload.

https://github.com/llvm/llvm-project/pull/208060