[all-commits] [llvm/llvm-project] 3d89b3: [CSSPGO] Introducing distribution factor for pseud...

[Hongtao Yu via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 3d89b3cbec230633e8228787819b15116c1a1730
      https://github.com/llvm/llvm-project/commit/3d89b3cbec230633e8228787819b15116c1a1730
  Author: Hongtao Yu <hoy at fb.com>
  Date:   2021-02-02 (Tue, 02 Feb 2021)

  Changed paths:
    M clang/test/CodeGen/pseudo-probe-emit.c
    M llvm/include/llvm/IR/IntrinsicInst.h
    M llvm/include/llvm/IR/Intrinsics.td
    M llvm/include/llvm/IR/PseudoProbe.h
    M llvm/include/llvm/Passes/StandardInstrumentations.h
    M llvm/include/llvm/ProfileData/SampleProf.h
    M llvm/include/llvm/Transforms/IPO/SampleProfileProbe.h
    M llvm/lib/IR/PseudoProbe.cpp
    M llvm/lib/Passes/PassBuilder.cpp
    M llvm/lib/Passes/PassRegistry.def
    M llvm/lib/Passes/StandardInstrumentations.cpp
    M llvm/lib/Transforms/IPO/SampleProfile.cpp
    M llvm/lib/Transforms/IPO/SampleProfileProbe.cpp
    A llvm/test/Transforms/SampleProfile/Inputs/pseudo-probe-update.prof
    M llvm/test/Transforms/SampleProfile/pseudo-probe-emit-inline.ll
    M llvm/test/Transforms/SampleProfile/pseudo-probe-emit.ll
    M llvm/test/Transforms/SampleProfile/pseudo-probe-inline.ll
    M llvm/test/Transforms/SampleProfile/pseudo-probe-profile.ll
    A llvm/test/Transforms/SampleProfile/pseudo-probe-update.ll
    A llvm/test/Transforms/SampleProfile/pseudo-probe-verify.ll

  Log Message:
  -----------
  [CSSPGO] Introducing distribution factor for pseudo probe.

Sample re-annotation is required in LTO time to achieve a reasonable post-inline profile quality. However, we have seen that such LTO-time re-annotation degrades profile quality. This is mainly caused by preLTO code duplication that is done by passes such as loop unrolling, jump threading, indirect call promotion etc, where samples corresponding to a source location are aggregated multiple times due to the duplicates. In this change we are introducing a concept of distribution factor for pseudo probes so that samples can be distributed for duplicated probes scaled by a factor. We hope that optimizations duplicating code well-maintain the branch frequency information (BFI) based on which probe distribution factors are calculated. Distribution factors are updated at the end of preLTO pipeline to reflect an estimated portion of the real execution count.

This change also introduces a pseudo probe verifier that can be run after each IR passes to detect duplicated pseudo probes.

A saturated distribution factor stands for 1.0. A pesudo probe will carry a factor with the value ranged from 0.0 to 1.0. A 64-bit integral distribution factor field that represents [0.0, 1.0] is associated to each block probe. Unfortunately this cannot be done for callsite probes due to the size limitation of a 32-bit Dwarf discriminator. A 7-bit distribution factor is used instead.

Changes are also needed to the sample profile inliner to deal with prorated callsite counts. Call sites duplicated by PreLTO passes, when later on inlined in LTO time, should have the callees’s probe prorated based on the Prelink-computed distribution factors. The distribution factors should also be taken into account when computing hotness for inline candidates. Also, Indirect call promotion results in multiple callisites. The original samples should be distributed across them. This is fixed by adjusting the callisites' distribution factors.

Reviewed By: wmi

Differential Revision: https://reviews.llvm.org/D93264