[llvm-branch-commits] [llvm] [llvm-profgen] Account for indirect-call promotion limits in the CS pre-inliner (PR #243631)
via llvm-branch-commits
llvm-branch-commits at lists.llvm.org
Tue Nov 17 13:58:18 PST 2026
llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-pgo
Author: Kunal Pathak (kunalspathak)
<details>
<summary>Changes</summary>
llvm-profgen can predict that an indirect-call target will be inlined even though the compiler later rejects it for promotion. The function then stays out of line, but its samples remain under the caller, leaving its standalone profile nearly empty.
This patch checks indirect targets against hotness, call-site percentage, and promotion-count limits before predicting an inline. Rejected contexts follow the existing path for merging samples into the target's standalone profile.
Nesting a target's profile replaces its call-target count with its entry-count estimate. These counts can differ, so the check updates the site's total before checking whether the target, and the ones that are already accepted, whether they meet the percentage requirement or not.
The check applies only to sites identified by indirect-call pseudo probes. Multiple recorded targets alone do not prove that a call was indirect.
[Godbolt example](https://godbolt.org/z/Mz6MKxq67): with the original profile, `xform1` tail-calls `impl1`. Restoring the rejected context to `xform1`'s standalone profile lets the same compiler inline `impl1`.
The example uses synthetic profiles to demonstrate profile repair, without measuring runtime savings.
Extracted from https://github.com/llvm/llvm-project/pull/238576
Assisted-by: GPT-6 Astra
---
Patch is 32.28 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/243631.diff
13 Files Affected:
- (modified) llvm/include/llvm/MC/MCPseudoProbe.h (+11-3)
- (modified) llvm/include/llvm/Transforms/IPO/SampleProfile.h (+7)
- (modified) llvm/lib/MC/MCPseudoProbe.cpp (+11-5)
- (modified) llvm/lib/Transforms/IPO/SampleProfile.cpp (+16)
- (added) llvm/test/tools/llvm-profgen/X86/Inputs/cs-preinline-icp-map-head.prof (+48)
- (added) llvm/test/tools/llvm-profgen/X86/Inputs/cs-preinline-icp-probe-type.prof (+53)
- (added) llvm/test/tools/llvm-profgen/X86/Inputs/cs-preinline-icp.prof (+48)
- (added) llvm/test/tools/llvm-profgen/X86/cs-preinline-icp.test (+100)
- (modified) llvm/tools/llvm-profgen/CSPreInliner.cpp (+137-2)
- (modified) llvm/tools/llvm-profgen/CSPreInliner.h (+10)
- (modified) llvm/tools/llvm-profgen/ProfiledBinary.cpp (+11-1)
- (modified) llvm/tools/llvm-profgen/ProfiledBinary.h (+15)
- (modified) llvm/unittests/MC/MCPseudoProbeTest.cpp (+33)
``````````diff
diff --git a/llvm/include/llvm/MC/MCPseudoProbe.h b/llvm/include/llvm/MC/MCPseudoProbe.h
index fe63137a3bf166..41e810043615dc 100644
--- a/llvm/include/llvm/MC/MCPseudoProbe.h
+++ b/llvm/include/llvm/MC/MCPseudoProbe.h
@@ -57,6 +57,7 @@
#include "llvm/ADT/ArrayRef.h"
#include "llvm/ADT/DenseMap.h"
#include "llvm/ADT/DenseSet.h"
+#include "llvm/ADT/STLFunctionalExtras.h"
#include "llvm/ADT/SmallVector.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/iterator.h"
@@ -436,6 +437,10 @@ class MCPseudoProbeDecoder {
using Uint64Set = DenseSet<uint64_t>;
using Uint64Map = DenseMap<uint64_t, uint64_t>;
+ // Called with the GUID of the function that owns a probe, the probe index and
+ // the probe type.
+ using ProbeCallback =
+ function_ref<void(uint64_t Guid, uint32_t Index, PseudoProbeType Type)>;
// Decode pseudo_probe_desc section to build GUID to PseudoProbeFuncDesc map.
// If pseudo_probe_desc section is mapped to memory and \p IsMMapped is true,
@@ -451,10 +456,12 @@ class MCPseudoProbeDecoder {
const Uint64Set &GuidFilter);
// Decode pseudo_probe section to build address to probes map for specifed
- // functions only.
+ // functions only. If \p Callback is set, also call it for every probe in the
+ // section, including the probes of functions that the GUID filter skips.
LLVM_ABI bool buildAddress2ProbeMap(const uint8_t *Start, std::size_t Size,
const Uint64Set &GuildFilter,
- const Uint64Map &FuncStartAddrs);
+ const Uint64Map &FuncStartAddrs,
+ ProbeCallback Callback = nullptr);
// Print pseudo_probe_desc section info
LLVM_ABI void printGUID2FuncDescMap(raw_ostream &OS);
@@ -530,7 +537,8 @@ class MCPseudoProbeDecoder {
bool buildAddress2ProbeMap(MCDecodedPseudoProbeInlineTree *Cur,
uint64_t &LastAddr, const Uint64Set &GuildFilter,
const Uint64Map &FuncStartAddrs,
- const uint32_t CurChildIndex);
+ const uint32_t CurChildIndex,
+ ProbeCallback Callback);
};
} // end namespace llvm
diff --git a/llvm/include/llvm/Transforms/IPO/SampleProfile.h b/llvm/include/llvm/Transforms/IPO/SampleProfile.h
index 085b50f18c4686..3c5a9fefd1a3cb 100644
--- a/llvm/include/llvm/Transforms/IPO/SampleProfile.h
+++ b/llvm/include/llvm/Transforms/IPO/SampleProfile.h
@@ -33,6 +33,13 @@ LLVM_ABI extern cl::opt<int> ProfileInlineGrowthLimit;
LLVM_ABI extern cl::opt<int> ProfileInlineLimitMin;
LLVM_ABI extern cl::opt<int> ProfileInlineLimitMax;
LLVM_ABI extern cl::opt<bool> SortProfiledSCC;
+/// The promotion limits used by the sample loader and llvm-profgen.
+LLVM_ABI unsigned
+getSampleProfileICPRelativeHotness(const cl::LibraryOptionsContext &Opts);
+LLVM_ABI unsigned
+getSampleProfileICPRelativeHotnessSkip(const cl::LibraryOptionsContext &Opts);
+LLVM_ABI unsigned
+getSampleProfileICPMaxProm(const cl::LibraryOptionsContext &Opts);
/// The sample profiler data loader pass.
class SampleProfileLoaderPass
diff --git a/llvm/lib/MC/MCPseudoProbe.cpp b/llvm/lib/MC/MCPseudoProbe.cpp
index 85bfde80722eb7..bb020a6b9e305a 100644
--- a/llvm/lib/MC/MCPseudoProbe.cpp
+++ b/llvm/lib/MC/MCPseudoProbe.cpp
@@ -464,7 +464,7 @@ template <bool IsTopLevelFunc>
bool MCPseudoProbeDecoder::buildAddress2ProbeMap(
MCDecodedPseudoProbeInlineTree *Cur, uint64_t &LastAddr,
const Uint64Set &GuidFilter, const Uint64Map &FuncStartAddrs,
- const uint32_t CurChildIndex) {
+ const uint32_t CurChildIndex, ProbeCallback Callback) {
// The pseudo_probe section encodes an inline forest and each tree has a
// format defined in MCPseudoProbe.h
@@ -544,6 +544,10 @@ bool MCPseudoProbeDecoder::buildAddress2ProbeMap(
Discriminator, Cur);
++CurrentProbeCount;
}
+ // The probes of skipped functions are read anyway to find the next record,
+ // so the callback can see them at no extra decoding cost.
+ if (Callback && !isSentinelProbe(Attr))
+ Callback(Guid, Index, PseudoProbeType(Kind));
LastAddr = Addr;
}
@@ -555,7 +559,8 @@ bool MCPseudoProbeDecoder::buildAddress2ProbeMap(
MutableArrayRef(InlineTreeVec).take_back(ChildrenToProcess);
}
for (uint32_t I = 0; I < ChildrenToProcess; I++) {
- buildAddress2ProbeMap<false>(Cur, LastAddr, GuidFilter, FuncStartAddrs, I);
+ buildAddress2ProbeMap<false>(Cur, LastAddr, GuidFilter, FuncStartAddrs, I,
+ Callback);
}
return Cur;
}
@@ -641,7 +646,7 @@ bool MCPseudoProbeDecoder::countRecords(bool &Discard, uint32_t &ProbeCount,
bool MCPseudoProbeDecoder::buildAddress2ProbeMap(
const uint8_t *Start, std::size_t Size, const Uint64Set &GuidFilter,
- const Uint64Map &FuncStartAddrs) {
+ const Uint64Map &FuncStartAddrs, ProbeCallback Callback) {
// For function records in the order of their appearance in the encoded data
// (DFS), count the number of contained probes and inlined function records.
uint32_t ProbeCount = 0;
@@ -668,8 +673,9 @@ bool MCPseudoProbeDecoder::buildAddress2ProbeMap(
uint64_t LastAddr = 0;
uint32_t CurChildIndex = 0;
while (Data < End)
- CurChildIndex += buildAddress2ProbeMap<true>(
- &DummyInlineRoot, LastAddr, GuidFilter, FuncStartAddrs, CurChildIndex);
+ CurChildIndex +=
+ buildAddress2ProbeMap<true>(&DummyInlineRoot, LastAddr, GuidFilter,
+ FuncStartAddrs, CurChildIndex, Callback);
assert(Data == End && "Have unprocessed data in pseudo_probe section");
assert(PseudoProbeVec.size() == ProbeCount &&
"Mismatching probe count pre- and post-parsing");
diff --git a/llvm/lib/Transforms/IPO/SampleProfile.cpp b/llvm/lib/Transforms/IPO/SampleProfile.cpp
index efd858c19ced54..62ebead633455e 100644
--- a/llvm/lib/Transforms/IPO/SampleProfile.cpp
+++ b/llvm/lib/Transforms/IPO/SampleProfile.cpp
@@ -66,6 +66,7 @@
#include "llvm/Support/CommandLine.h"
#include "llvm/Support/Debug.h"
#include "llvm/Support/ErrorOr.h"
+#include "llvm/Support/LibraryOptionsContext.h"
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Support/raw_ostream.h"
#include "llvm/Transforms/IPO.h"
@@ -146,6 +147,21 @@ cl::opt<int> SampleColdCallSiteThreshold(
cl::desc("Threshold for inlining cold callsites"));
} // namespace llvm
+unsigned llvm::getSampleProfileICPRelativeHotness(
+ const cl::LibraryOptionsContext &Opts) {
+ return Opts.get<IPOOptions>().sample_profile_icp_relative_hotness;
+}
+
+unsigned llvm::getSampleProfileICPRelativeHotnessSkip(
+ const cl::LibraryOptionsContext &Opts) {
+ return Opts.get<IPOOptions>().sample_profile_icp_relative_hotness_skip;
+}
+
+unsigned
+llvm::getSampleProfileICPMaxProm(const cl::LibraryOptionsContext &Opts) {
+ return Opts.get<IPOOptions>().sample_profile_icp_max_prom;
+}
+
namespace llvm {
extern cl::opt<bool> EnableExtTspBlockPlacement;
}
diff --git a/llvm/test/tools/llvm-profgen/X86/Inputs/cs-preinline-icp-map-head.prof b/llvm/test/tools/llvm-profgen/X86/Inputs/cs-preinline-icp-map-head.prof
new file mode 100644
index 00000000000000..33e8dbdb6fa2fc
--- /dev/null
+++ b/llvm/test/tools/llvm-profgen/X86/Inputs/cs-preinline-icp-map-head.prof
@@ -0,0 +1,48 @@
+[quick_sort]:2000:1000
+ 1: 1000
+ 4: 1000 t1:600 t2:200 t3:200
+ !CFGChecksum: 1
+[quick_sort:4 @ t1]:6000:600
+ 1: 600
+ 2: 5400
+ !CFGChecksum: 2
+[quick_sort:4 @ t2]:2000:260
+ 1: 260
+ 2: 1740
+ !CFGChecksum: 3
+[quick_sort:4 @ t3]:2000:200
+ 1: 200
+ 2: 1800
+ !CFGChecksum: 4
+[cold]:60:1
+ 1: 2
+ 2: 2
+ 3: 2
+ 4: 2
+ 5: 2
+ 6: 2
+ 7: 2
+ 8: 2
+ 9: 2
+ 10: 2
+ 11: 2
+ 12: 2
+ 13: 2
+ 14: 2
+ 15: 2
+ 16: 2
+ 17: 2
+ 18: 2
+ 19: 2
+ 20: 2
+ 21: 2
+ 22: 2
+ 23: 2
+ 24: 2
+ 25: 2
+ 26: 2
+ 27: 2
+ 28: 2
+ 29: 2
+ 30: 2
+ !CFGChecksum: 5
diff --git a/llvm/test/tools/llvm-profgen/X86/Inputs/cs-preinline-icp-probe-type.prof b/llvm/test/tools/llvm-profgen/X86/Inputs/cs-preinline-icp-probe-type.prof
new file mode 100644
index 00000000000000..5fa771f3cdc18e
--- /dev/null
+++ b/llvm/test/tools/llvm-profgen/X86/Inputs/cs-preinline-icp-probe-type.prof
@@ -0,0 +1,53 @@
+[quick_sort]:3100:2000
+ 1: 2000
+ 4: 100 t1:100
+ 5: 1000 t2:600 t3:200 t4:200
+ !CFGChecksum: 1
+[quick_sort:4 @ t1]:19100:100
+ 1: 100
+ 2: 19000
+ !CFGChecksum: 2
+[quick_sort:5 @ t2]:6000:600
+ 1: 600
+ 2: 5400
+ !CFGChecksum: 3
+[quick_sort:5 @ t3]:2000:200
+ 1: 200
+ 2: 1800
+ !CFGChecksum: 4
+[quick_sort:5 @ t4]:2000:200
+ 1: 200
+ 2: 1800
+ !CFGChecksum: 5
+[cold]:60:1
+ 1: 2
+ 2: 2
+ 3: 2
+ 4: 2
+ 5: 2
+ 6: 2
+ 7: 2
+ 8: 2
+ 9: 2
+ 10: 2
+ 11: 2
+ 12: 2
+ 13: 2
+ 14: 2
+ 15: 2
+ 16: 2
+ 17: 2
+ 18: 2
+ 19: 2
+ 20: 2
+ 21: 2
+ 22: 2
+ 23: 2
+ 24: 2
+ 25: 2
+ 26: 2
+ 27: 2
+ 28: 2
+ 29: 2
+ 30: 2
+ !CFGChecksum: 6
diff --git a/llvm/test/tools/llvm-profgen/X86/Inputs/cs-preinline-icp.prof b/llvm/test/tools/llvm-profgen/X86/Inputs/cs-preinline-icp.prof
new file mode 100644
index 00000000000000..ad69bb20d77b2c
--- /dev/null
+++ b/llvm/test/tools/llvm-profgen/X86/Inputs/cs-preinline-icp.prof
@@ -0,0 +1,48 @@
+[quick_sort]:2000:1000
+ 1: 1000
+ 4: 1000 t1:600 t2:200 t3:200
+ !CFGChecksum: 1
+[quick_sort:4 @ t1]:6000:600
+ 1: 600
+ 2: 5400
+ !CFGChecksum: 2
+[quick_sort:4 @ t2]:2000:200
+ 1: 200
+ 2: 1800
+ !CFGChecksum: 3
+[quick_sort:4 @ t3]:2000:200
+ 1: 200
+ 2: 1800
+ !CFGChecksum: 4
+[cold]:60:1
+ 1: 2
+ 2: 2
+ 3: 2
+ 4: 2
+ 5: 2
+ 6: 2
+ 7: 2
+ 8: 2
+ 9: 2
+ 10: 2
+ 11: 2
+ 12: 2
+ 13: 2
+ 14: 2
+ 15: 2
+ 16: 2
+ 17: 2
+ 18: 2
+ 19: 2
+ 20: 2
+ 21: 2
+ 22: 2
+ 23: 2
+ 24: 2
+ 25: 2
+ 26: 2
+ 27: 2
+ 28: 2
+ 29: 2
+ 30: 2
+ !CFGChecksum: 5
diff --git a/llvm/test/tools/llvm-profgen/X86/cs-preinline-icp.test b/llvm/test/tools/llvm-profgen/X86/cs-preinline-icp.test
new file mode 100644
index 00000000000000..e7b359538d5620
--- /dev/null
+++ b/llvm/test/tools/llvm-profgen/X86/cs-preinline-icp.test
@@ -0,0 +1,100 @@
+; The preinliner must not plan inlining of an indirect call target that the
+; sample profile loader will not promote (top target, then >= 25% of the site,
+; at most 3 targets by default). Such contexts are merged into the callee's base
+; profile instead of being stranded at the call site. In
+; fs-discriminator-probe.perfbin, probe 4 of quick_sort is an indirect call.
+
+; RUN: llvm-profgen --format=text --llvm-sample-profile=%S/Inputs/cs-preinline-icp.prof --binary=%S/Inputs/fs-discriminator-probe.perfbin --csspgo-preinliner=1 --gen-cs-nested-profile=0 --use-context-cost-for-preinliner=0 --sample-profile-hot-inline-threshold=3000 --output=%t1
+; RUN: FileCheck %s --input-file %t1 --check-prefix=CHECK-ICP
+
+; RUN: llvm-profgen --format=text --llvm-sample-profile=%S/Inputs/cs-preinline-icp.prof --binary=%S/Inputs/fs-discriminator-probe.perfbin --csspgo-preinliner=1 --gen-cs-nested-profile=0 --use-context-cost-for-preinliner=0 --sample-profile-hot-inline-threshold=3000 --csspgo-preinliner-honor-icp-limits=0 --output=%t2
+; RUN: FileCheck %s --input-file %t2 --check-prefix=CHECK-ALL
+
+; A lower ICP threshold lets the 20% targets be promoted, so they stay inlined.
+; RUN: llvm-profgen --format=text --llvm-sample-profile=%S/Inputs/cs-preinline-icp.prof --binary=%S/Inputs/fs-discriminator-probe.perfbin --csspgo-preinliner=1 --gen-cs-nested-profile=0 --use-context-cost-for-preinliner=0 --sample-profile-hot-inline-threshold=3000 --sample-profile-icp-relative-hotness=20 --output=%t3
+; RUN: FileCheck %s --input-file %t3 --check-prefix=CHECK-ALL
+
+; Exempting all three targets from the percentage check also keeps them inlined.
+; RUN: llvm-profgen --format=text --llvm-sample-profile=%S/Inputs/cs-preinline-icp.prof --binary=%S/Inputs/fs-discriminator-probe.perfbin --csspgo-preinliner=1 --gen-cs-nested-profile=0 --use-context-cost-for-preinliner=0 --sample-profile-hot-inline-threshold=3000 --sample-profile-icp-relative-hotness-skip=3 --output=%t-skip
+; RUN: FileCheck %s --input-file %t-skip --check-prefix=CHECK-ALL
+
+; The loader also promotes at most sample-profile-icp-max-prom targets per call.
+; With the percentage check at 20%, all three targets pass it, so a limit of 2
+; rejects t3, and a limit of 0 rejects all three.
+; RUN: llvm-profgen --format=text --llvm-sample-profile=%S/Inputs/cs-preinline-icp.prof --binary=%S/Inputs/fs-discriminator-probe.perfbin --csspgo-preinliner=1 --gen-cs-nested-profile=0 --use-context-cost-for-preinliner=0 --sample-profile-hot-inline-threshold=3000 --sample-profile-icp-relative-hotness=20 --sample-profile-icp-max-prom=2 --output=%t-maxprom
+; RUN: FileCheck %s --input-file %t-maxprom --check-prefix=CHECK-MAXPROM --implicit-check-not="@ t3]"
+; RUN: llvm-profgen --format=text --llvm-sample-profile=%S/Inputs/cs-preinline-icp.prof --binary=%S/Inputs/fs-discriminator-probe.perfbin --csspgo-preinliner=1 --gen-cs-nested-profile=0 --use-context-cost-for-preinliner=0 --sample-profile-hot-inline-threshold=3000 --sample-profile-icp-relative-hotness=20 --sample-profile-icp-max-prom=0 --output=%t-noprom
+; RUN: FileCheck %s --input-file %t-noprom --check-prefix=CHECK-NOPROM --implicit-check-not="[quick_sort:4 @"
+
+; CHECK-MAXPROM-DAG: [quick_sort:4 @ t1]:6000:600
+; CHECK-MAXPROM-DAG: [quick_sort:4 @ t2]:2000:200
+; CHECK-MAXPROM-DAG: [t3]:2000:200
+
+; CHECK-NOPROM-DAG: [t1]:6000:600
+; CHECK-NOPROM-DAG: [t2]:2000:200
+; CHECK-NOPROM-DAG: [t3]:2000:200
+
+; A site without an indirect call probe is a direct call, even with several
+; targets: a direct call site can show several functions when the binary lost
+; the inline frames between them. inline-cs-pseudoprobe.perfbin has no function
+; quick_sort, so quick_sort:4 has no call probe there and all three targets are
+; pre-inlined.
+; RUN: llvm-profgen --format=text --llvm-sample-profile=%S/Inputs/cs-preinline-icp.prof --binary=%S/Inputs/inline-cs-pseudoprobe.perfbin --csspgo-preinliner=1 --gen-cs-nested-profile=0 --use-context-cost-for-preinliner=0 --sample-profile-hot-inline-threshold=3000 --output=%t-noprobe
+; RUN: FileCheck %s --input-file %t-noprobe --check-prefix=CHECK-ALL
+
+; The nested profile consumed by the compiler keeps the full samples of the
+; rejected targets in their own profiles.
+; RUN: llvm-profgen --format=text --llvm-sample-profile=%S/Inputs/cs-preinline-icp.prof --binary=%S/Inputs/fs-discriminator-probe.perfbin --csspgo-preinliner=1 --gen-cs-nested-profile=1 --use-context-cost-for-preinliner=0 --sample-profile-hot-inline-threshold=3000 --output=%t4
+; RUN: FileCheck %s --input-file %t4 --check-prefix=CHECK-NESTED
+
+; CHECK-ICP: [quick_sort:4 @ t1]:6000:600
+; CHECK-ICP: !Attributes: 2
+; CHECK-ICP: [quick_sort]:2000:1000
+; CHECK-ICP-NEXT: 1: 1000
+; CHECK-ICP-NEXT: 4: 1000 t1:600 t2:200 t3:200
+; CHECK-ICP-NOT: [quick_sort:4 @ t2]
+; CHECK-ICP-NOT: [quick_sort:4 @ t3]
+; CHECK-ICP: [t2]:2000:200
+; CHECK-ICP: [t3]:2000:200
+
+; CHECK-ALL-DAG: [quick_sort:4 @ t1]:6000:600
+; CHECK-ALL-DAG: [quick_sort:4 @ t2]:2000:200
+; CHECK-ALL-DAG: [quick_sort:4 @ t3]:2000:200
+; CHECK-ALL-NOT: [t2]:
+; CHECK-ALL-NOT: [t3]:
+
+; CHECK-NESTED: quick_sort:7400:1000
+; CHECK-NESTED-NEXT: 1: 1000
+; CHECK-NESTED-NEXT: 4: 400 t2:200 t3:200
+; CHECK-NESTED-NEXT: 4: t1:6000
+; CHECK-NESTED: t2:2000:200
+; CHECK-NESTED: t3:2000:200
+
+; The loader's site total uses the nested head estimate of each inlined target
+; and the map count of each outlined one. Here t2's map count is 200 but its
+; head estimate is 260: inlining it would make the total 1060, and 260 is
+; below 25% of that, so t2 must not be pre-inlined either.
+; RUN: llvm-profgen --format=text --llvm-sample-profile=%S/Inputs/cs-preinline-icp-map-head.prof --binary=%S/Inputs/fs-discriminator-probe.perfbin --csspgo-preinliner=1 --gen-cs-nested-profile=1 --use-context-cost-for-preinliner=0 --sample-profile-hot-inline-threshold=3000 --output=%t5
+; RUN: FileCheck %s --input-file %t5 --check-prefix=CHECK-MAPHEAD
+
+; CHECK-MAPHEAD: quick_sort:7400:1000
+; CHECK-MAPHEAD-NEXT: 1: 1000
+; CHECK-MAPHEAD-NEXT: 4: 400 t2:200 t3:200
+; CHECK-MAPHEAD-NEXT: 4: t1:6000
+; CHECK-MAPHEAD-NOT: 4: t2:
+; CHECK-MAPHEAD: t2:2000:260
+; CHECK-MAPHEAD: t3:2000:200
+
+; Probe 5 of quick_sort is a direct call. The single target at the indirect
+; call is not hot enough for ICP, so it is not pre-inlined. The direct call
+; site is not subject to ICP, so all its targets are pre-inlined.
+; RUN: llvm-profgen --format=text --llvm-sample-profile=%S/Inputs/cs-preinline-icp-probe-type.prof --binary=%S/Inputs/fs-discriminator-probe.perfbin --csspgo-preinliner=1 --gen-cs-nested-profile=0 --use-context-cost-for-preinliner=0 --sample-profile-hot-inline-threshold=3000 --output=%t7
+; RUN: FileCheck %s --input-file %t7 --check-prefix=CHECK-PROBETYPE
+
+; CHECK-PROBETYPE-DAG: [t1]:19100:100
+; CHECK-PROBETYPE-DAG: [quick_sort:5 @ t2]:6000:600
+; CHECK-PROBETYPE-DAG: [quick_sort:5 @ t3]:2000:200
+; CHECK-PROBETYPE-DAG: [quick_sort:5 @ t4]:2000:200
+; CHECK-PROBETYPE-NOT: [quick_sort:4 @ t1]
+; CHECK-PROBETYPE-NOT: [t3]:
+; CHECK-PROBETYPE-NOT: [t4]:
diff --git a/llvm/tools/llvm-profgen/CSPreInliner.cpp b/llvm/tools/llvm-profgen/CSPreInliner.cpp
index 421552f9c6dbb9..feb553b6f235f2 100644
--- a/llvm/tools/llvm-profgen/CSPreInliner.cpp
+++ b/llvm/tools/llvm-profgen/CSPreInliner.cpp
@@ -12,6 +12,7 @@
#include "llvm/ADT/Statistic.h"
#include "llvm/Transforms/IPO/SampleProfile.h"
#include <cstdint>
+#include <map>
#define DEBUG_TYPE "cs-preinliner"
@@ -22,6 +23,9 @@ STATISTIC(PreInlNumCSInlined,
"Number of functions inlined with context sensitive profile");
STATISTIC(PreInlNumCSNotInlined,
"Number of functions not inlined with context sensitive profile");
+STATISTIC(PreInlNumCSICPRejected,
+ "Number of indirect call target contexts not inlined because the "
+ "sample profile loader would not promote them");
STATISTIC(PreInlNumCSInlinedHitMinLimit,
"Number of functions with FDO inline stopped due to min size limit");
STATISTIC(PreInlNumCSInlinedHitMaxLimit,
@@ -49,6 +53,12 @@ static cl::opt<bool> SamplePreInlineReplay(
cl::desc(
"Replay previous inlining and adjust context profile accordingly"));
+static cl::opt<bool> CSPreinlHonorICPLimits(
+ "csspgo-preinliner-honor-icp-limits", cl::Hidden, cl::init(true),
+ cl::desc("Do not inline indirect call target contexts that the sample "
+ "profile loader will not promote (see "
+ "-sample-profile-icp-relative-hotness)."));
+
static cl::opt<int> CSPreinlMultiplierForPrevInl(
"csspgo-preinliner-multiplier-for-previous-inlining", cl::Hidden,
cl::init(100),
@@ -70,6 +80,12 @@ CSPreInliner::CSPreInliner(SampleContextTracker &Tracker,
SampleColdCallSiteThreshold = 0;
if (!ProfileInlineLimitMax.getNumOccurrences())
ProfileInlineLimitMax = 50000;
+
+ ProfileICPRelativeHotness =
+ getSampleProfileICPRelativeHotness(Binary.getLibraryOptions());
+ ProfileICPRelativeHotnessSkip =
+ getSampleProfileICPRelativeHotnessSkip(Binary.getLibraryOptions());
+ ProfileICPMaxProm = getSampleProfileICPMaxProm(Binary.getLibraryOptions());
}
std::vector<FunctionId> CSPreInliner::buildTopDownOrder() {
@@ -102,6 +118,116 @@ std::vector<FunctionId> CSPreInliner::buildTopDownOrder() {
return Order;
}
+// Find the indirect call targets under CallerNode that the sample loader won't
+// promote, so we don't plan to inline them. The loader (see
+// inlineHotFunctionsWithPriority) walks a call's targets from hottest to
+// coldest and promotes one only if it's hot, is at least
+// ProfileICPRelativeHotness percent of the site total (once the first
+// ProfileICPRelativeHotnessSkip targets are promoted), and fewer than
+// ProfileICPMaxProm targets have been promoted so far. Pre-inlining a target
+// that fails these checks just burns the caller's size budget and leaves its
+// samples nested at the call site instead of in its own profile.
+//
+// One wrinkle: nesting a target under the caller makes convertCSProfiles drop
+// its call-target entry, so the loader counts its head samples instead and the
+// site total moves. Before accepting a target, we re-check it and every target
+// accepted so far against the new total; if that fails, it and all colder
+// targets are rejected.
+De...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/243631
More information about the llvm-branch-commits
mailing list