[llvm] [llvm-profgen] Infer callsite samples for probeless inlined functions (PR #191572)
Wei Wang via llvm-commits
llvm-commits at lists.llvm.org
Tue Apr 14 20:27:33 PDT 2026
https://github.com/apolloww updated https://github.com/llvm/llvm-project/pull/191572
>From 815df5a3bd62befd4cadc501e45fbbab319c4d03 Mon Sep 17 00:00:00 2001
From: Wei Wang <apollo.mobility at gmail.com>
Date: Fri, 10 Apr 2026 13:25:42 -0700
Subject: [PATCH] [CSProfileGenerator] Infer callsite samples for functions
without FunctionSamples
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
In probe-based CSSPGO, some context trie nodes lack FunctionSamples.
This can happen in two cases:
1. Probeless functions: inlined functions (thin wrappers like
destructors, copy constructors, forwarding functions) fully
optimized away after inlining, leaving no probes in the binary.
Without probes, they get zero body samples and no FunctionSamples,
making them invisible to the pre-inliner.
2. Cold functions with probes: functions that have probes in the
binary but no LBR samples landed on them. These also end up
with no FunctionSamples.
The two cases are not distinguished by inferCallsiteSamples() — both
result in a missing FunctionSamples. The pre-inliner separates them via
size: probeless functions get size 0 from trackInlineesOptimizedAway(),
while cold functions keep their real (non-zero) size, so the
pre-inliner's size budget naturally limits their inlining.
This patch adds three coordinated fixes:
1. inferCallsiteSamples() in ProfileGenerator.cpp: walks the context
trie bottom-up and creates FunctionSamples for nodes that lack them
by inferring callsite body samples and called targets from callee
entry (head) samples. Also fixes under-counted callsites where LBR
sparsity caused callsite counts to be lower than callee entry
counts. Guarded by --infer-callsite-samples (default: off).
2. trackInlineesOptimizedAway() in ProfiledBinary.cpp: extend to handle
inline tree nodes that have no probes but still have children in the
inline tree. This marks probeless inlined wrappers as size=0 so the
pre-inliner's size budget accounts for them correctly.
3. shouldInline() in CSPreInliner.cpp: change threshold check from
`SizeCost < SampleThreshold` to `SizeCost <= SampleThreshold`. The
probeless functions from (2) have SizeCost=0, and cold callsites
can have SampleThreshold=0, so the strict `<` incorrectly rejects
them (0 < 0 is false). The `<=` fix allows these zero-cost inlinees
through (0 <= 0 is true).
---
.../llvm-profgen/infer-callsite-samples.test | 21 ++++
llvm/tools/llvm-profgen/CSPreInliner.cpp | 2 +-
llvm/tools/llvm-profgen/ProfileGenerator.cpp | 105 ++++++++++++++++++
llvm/tools/llvm-profgen/ProfileGenerator.h | 2 +
llvm/tools/llvm-profgen/ProfiledBinary.cpp | 9 +-
5 files changed, 135 insertions(+), 4 deletions(-)
create mode 100644 llvm/test/tools/llvm-profgen/infer-callsite-samples.test
diff --git a/llvm/test/tools/llvm-profgen/infer-callsite-samples.test b/llvm/test/tools/llvm-profgen/infer-callsite-samples.test
new file mode 100644
index 0000000000000..afcdcd0a03485
--- /dev/null
+++ b/llvm/test/tools/llvm-profgen/infer-callsite-samples.test
@@ -0,0 +1,21 @@
+; Test inferCallsiteSamples with --infer-callsite-samples
+
+; Baseline: no inference
+; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/inline-cs-pseudoprobe.perfscript --binary=%S/Inputs/inline-cs-pseudoprobe.perfbin --output=%t1 --profile-summary-cold-count=0 --csspgo-preinliner=0 --gen-cs-nested-profile=0 --infer-callsite-samples=0
+; RUN: FileCheck %s --input-file %t1 --check-prefix=CHECK-BASELINE
+
+; With probeless inference: creates FunctionSamples for probeless main
+; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/inline-cs-pseudoprobe.perfscript --binary=%S/Inputs/inline-cs-pseudoprobe.perfbin --output=%t2 --profile-summary-cold-count=0 --csspgo-preinliner=0 --gen-cs-nested-profile=0 --infer-callsite-samples=1
+; RUN: FileCheck %s --input-file %t2 --check-prefix=CHECK-INFER
+
+; Baseline: main does not appear, foo total is 74
+; CHECK-BASELINE: [main:2 @ foo]:74:0
+; CHECK-BASELINE: [main:2 @ foo:8 @ bar]:28:14
+; CHECK-BASELINE-NOT: [main]:
+
+; Infer: main gets FunctionSamples inferred from foo's head samples.
+; CHECK-INFER: [main:2 @ foo]:74:0
+; CHECK-INFER: [main:2 @ foo:8 @ bar]:28:14
+; CHECK-INFER: [main]:1:0
+; CHECK-INFER-NEXT: 2: 1 foo:1
+; CHECK-INFER-NEXT: !CFGChecksum: 281479271677951
diff --git a/llvm/tools/llvm-profgen/CSPreInliner.cpp b/llvm/tools/llvm-profgen/CSPreInliner.cpp
index 87df6996aa435..e614f12785e74 100644
--- a/llvm/tools/llvm-profgen/CSPreInliner.cpp
+++ b/llvm/tools/llvm-profgen/CSPreInliner.cpp
@@ -192,7 +192,7 @@ bool CSPreInliner::shouldInline(ProfiledInlineCandidate &Candidate) {
SampleThreshold *= CSPreinlMultiplierForPrevInl;
}
- return (Candidate.SizeCost < SampleThreshold);
+ return (Candidate.SizeCost <= SampleThreshold);
}
void CSPreInliner::processFunction(const FunctionId Name) {
diff --git a/llvm/tools/llvm-profgen/ProfileGenerator.cpp b/llvm/tools/llvm-profgen/ProfileGenerator.cpp
index c3f489040007e..fa98f72494376 100644
--- a/llvm/tools/llvm-profgen/ProfileGenerator.cpp
+++ b/llvm/tools/llvm-profgen/ProfileGenerator.cpp
@@ -119,6 +119,13 @@ cl::opt<bool> InferMissingFrames(
"Infer missing call frames due to compiler tail call elimination."),
cl::Optional, cl::cat(ProfGenCategory));
+cl::opt<bool> InferCallsiteSamples(
+ "infer-callsite-samples", llvm::cl::init(false),
+ llvm::cl::desc(
+ "Infer callsite body samples from callee entry samples for context "
+ "trie nodes that lack samples."),
+ llvm::cl::Optional);
+
namespace sampleprof {
// Initialize the MaxCompressionSize to -1 which means no size limit
@@ -936,6 +943,8 @@ void CSProfileGenerator::generateProfile() {
if (SampleCounters) {
if (Binary->usePseudoProbes()) {
generateProbeBasedProfile();
+ if (InferCallsiteSamples)
+ inferCallsiteSamples();
} else {
generateLineNumBasedProfile();
}
@@ -1373,6 +1382,102 @@ void CSProfileGenerator::populateBoundarySamplesWithProbes(
}
}
+void CSProfileGenerator::inferCallsiteSamples() {
+ // Walk the context trie bottom-up and for each node, ensure that callsite
+ // body samples are at least as large as callee entry (head) samples. This
+ // handles three cases:
+ //
+ // 1. Probeless functions: inlined functions fully optimized away (e.g., thin
+ // wrappers) that have no probes in the binary.
+ //
+ // 2. Cold functions: functions that have probes in the binary but no samples
+ // landed on them.
+ //
+ // 3. Under-counted callsites: nodes that already have samples but callsite
+ // counts is lower than callee entry counts.
+ //
+ // Note: cases 1 and 2 are not distinguished here — both result in a missing
+ // FunctionSamples from the trie node, but the pre-inliner separates them via
+ // size: probeless functions get size 0 from size tracker, while cold
+ // functions keep their real (non-zero) size.
+ SmallVector<ContextTrieNode *, 256> AllNodes;
+ for (auto *Node : ContextTracker)
+ AllNodes.push_back(Node);
+
+ for (auto It = AllNodes.rbegin(), End = AllNodes.rend(); It != End; ++It) {
+ ContextTrieNode *Node = *It;
+
+ if (Node == &getRootContext())
+ continue;
+
+ bool HasChildWithSamples = false;
+ for (auto &Child : Node->getAllChildContext()) {
+ if (Child.second.getFunctionSamples()) {
+ HasChildWithSamples = true;
+ break;
+ }
+ }
+ if (!HasChildWithSamples)
+ continue;
+
+ // Create FunctionSamples for nodes without them (probeless or cold).
+ FunctionSamples *FSamples = Node->getFunctionSamples();
+ if (!FSamples) {
+ FSamples = getOrCreateFunctionSamples(Node);
+ const auto *FuncDesc =
+ Binary->getFuncDescForGUID(Node->getFuncName().getHashCode());
+ FSamples->setFunctionHash(FuncDesc->FuncHash);
+ }
+
+ // Aggregate callee head samples per callsite, since multiple callees
+ // (e.g., indirect calls) can share the same callsite location. Also add
+ // each callee as a called target.
+ std::map<LineLocation, uint64_t> CallsiteHeadSum;
+ for (auto &Child : Node->getAllChildContext()) {
+ FunctionSamples *ChildSamples = Child.second.getFunctionSamples();
+ if (!ChildSamples)
+ continue;
+ uint64_t ChildHeadSamples = ChildSamples->getHeadSamplesEstimate();
+ if (ChildHeadSamples == 0)
+ continue;
+ LineLocation ChildCallsite = Child.second.getCallSiteLoc();
+ // Only add the delta to avoid double-counting if there are already
+ // called target samples from normal profile generation.
+ uint64_t ExistingTargetSamples = 0;
+ auto ExistingTargets = FSamples->findCallTargetMapAt(
+ ChildCallsite.LineOffset, ChildCallsite.Discriminator);
+ if (ExistingTargets) {
+ auto TargetIt = ExistingTargets->find(Child.second.getFuncName());
+ if (TargetIt != ExistingTargets->end())
+ ExistingTargetSamples = TargetIt->second;
+ }
+ // Use the max of head samples and existing target samples to compute
+ // the expected contribution of this callee to the callsite total.
+ CallsiteHeadSum[ChildCallsite] +=
+ std::max(ChildHeadSamples, ExistingTargetSamples);
+ if (ChildHeadSamples > ExistingTargetSamples)
+ FSamples->addCalledTargetSamples(
+ ChildCallsite.LineOffset, ChildCallsite.Discriminator,
+ Child.second.getFuncName(),
+ ChildHeadSamples - ExistingTargetSamples);
+ }
+
+ // Ensure callsite body samples >= sum of callee head samples.
+ for (auto &[Loc, HeadSum] : CallsiteHeadSum) {
+ uint64_t ExistingSamples = 0;
+ auto Existing =
+ FSamples->findSamplesAt(Loc.LineOffset, Loc.Discriminator);
+ if (Existing)
+ ExistingSamples = *Existing;
+ if (HeadSum > ExistingSamples) {
+ uint64_t Delta = HeadSum - ExistingSamples;
+ FSamples->addBodySamples(Loc.LineOffset, Loc.Discriminator, Delta);
+ FSamples->addTotalSamples(Delta);
+ }
+ }
+ }
+}
+
ContextTrieNode *CSProfileGenerator::getContextNodeForLeafProbe(
const AddrBasedCtxKey *CtxKey, const MCDecodedPseudoProbe *LeafProbe) {
diff --git a/llvm/tools/llvm-profgen/ProfileGenerator.h b/llvm/tools/llvm-profgen/ProfileGenerator.h
index 2c512779319d4..087c8cc05cbca 100644
--- a/llvm/tools/llvm-profgen/ProfileGenerator.h
+++ b/llvm/tools/llvm-profgen/ProfileGenerator.h
@@ -359,6 +359,8 @@ class CSProfileGenerator : public ProfileGeneratorBase {
getFunctionProfileForLeafProbe(const AddrBasedCtxKey *CtxKey,
const MCDecodedPseudoProbe *LeafProbe);
+ void inferCallsiteSamples();
+
void convertToProfileMap(ContextTrieNode &Node,
SampleContextFrameVector &Context);
diff --git a/llvm/tools/llvm-profgen/ProfiledBinary.cpp b/llvm/tools/llvm-profgen/ProfiledBinary.cpp
index 88832ba65efea..3a32d837e70cc 100644
--- a/llvm/tools/llvm-profgen/ProfiledBinary.cpp
+++ b/llvm/tools/llvm-profgen/ProfiledBinary.cpp
@@ -160,9 +160,12 @@ void BinarySizeContextTracker::trackInlineesOptimizedAway(
ProbeDecoder.getFuncDescForGUID(ProbeNode.Guid)->FuncName;
ProbeContext.emplace_back(FuncName, 0);
- // This ProbeContext has a probe, so it has code before inlining and
- // optimization. Make sure we mark its size as known.
- if (!ProbeNode.getProbes().empty()) {
+ // Mark the size of this inlinee as known (zero) if it was optimized away.
+ // A function was inlined and then optimized if it either:
+ // 1. Still has probes but no instructions, or
+ // 2. Has no probes left but still has children in the inline tree,
+ // meaning it existed as a real function before optimization.
+ if (!ProbeNode.getProbes().empty() || !ProbeNode.getChildren().empty()) {
ContextTrieNode *SizeContext = &RootContext;
for (auto &ProbeFrame : reverse(ProbeContext)) {
StringRef CallerName = ProbeFrame.first;
More information about the llvm-commits
mailing list