[llvm-branch-commits] [llvm] [llvm-profgen] Account for indirect-call promotion limits in the CS pre-inliner (PR #243631)
Wei Wang via llvm-branch-commits
llvm-branch-commits at lists.llvm.org
Wed Nov 18 23:12:30 PST 2026
================
@@ -102,6 +132,105 @@ std::vector<FunctionId> CSPreInliner::buildTopDownOrder() {
return Order;
}
+// Find indirect call targets to exclude from the pre-inline plan. Apply the
+// loader's promotion limits, using the contextless hot threshold as described
+// in the constructor. The loader (see
+// inlineHotFunctionsWithPriority) walks a call's targets from hottest to
+// coldest and promotes one only if it's hot, is at least
+// ProfileICPRelativeHotness percent of the site total (once the first
+// ProfileICPRelativeHotnessSkip targets are promoted), and fewer than
+// ProfileICPMaxProm targets have been promoted so far. Keep rejected targets
+// out of the caller's inline size budget and leave their contexts for merging
+// into their base profiles.
+//
+// Inlining a target in the profile replaces its call-target count with its
+// head-sample estimate. This can change the total used by the percentage check.
+// Accept the target only if it and all previously accepted targets still pass.
+// Otherwise, reject this target and the remaining, colder targets.
+DenseSet<const ContextTrieNode *>
+CSPreInliner::getICPRejectedCallees(ContextTrieNode *CallerNode,
+ const FunctionSamples *CallerSamples) {
+ DenseSet<const ContextTrieNode *> Rejected;
+ // Only a pseudo probe identifies a single call instruction. With line-based
+ // profiles, distinct direct calls on one source line share a call site, so
+ // several targets there do not imply an indirect call.
+ if (!CSPreinlHonorICPLimits || SamplePreInlineReplay ||
+ !FunctionSamples::ProfileIsProbeBased)
+ return Rejected;
+
+ struct Target {
+ ContextTrieNode *Node;
+ uint64_t HeadCount;
+ uint64_t CallTargetCount = 0;
+ };
+ std::map<LineLocation, SmallVector<Target, 4>> CallSites;
+ for (auto &Child : CallerNode->getAllChildContext()) {
+ ContextTrieNode *CalleeNode = &Child.second;
+ if (FunctionSamples *CalleeSamples = CalleeNode->getFunctionSamples())
+ CallSites[CalleeNode->getCallSiteLoc()].push_back(
+ {CalleeNode, CalleeSamples->getHeadSamplesEstimate()});
+ }
+
+ for (auto &[CallSite, Callees] : CallSites) {
+ // Apply ICP checks only when a decoded probe identifies an indirect call.
+ // Multiple targets alone are not enough: a direct call can record several
+ // functions when inline frames are lost. Without a matching probe, leave
+ // the inline decision to the existing size and hotness checks.
+ if (!Binary.isIndirectCallProbe(CallerNode->getFuncName().getHashCode(),
+ CallSite.LineOffset))
+ continue;
+
+ // Site total with every target outlined, i.e. the raw call target map.
+ uint64_t SiteTotal = 0;
+ if (auto CallTargets = CallerSamples->findCallTargetMapAt(CallSite)) {
+ for (const auto &[Name, Count] : *CallTargets)
+ SiteTotal += Count;
+ for (Target &T : Callees) {
+ auto It = CallTargets->find(T.Node->getFuncName());
+ if (It != CallTargets->end())
+ T.CallTargetCount = It->second;
+ }
+ }
+
+ llvm::sort(Callees, [](const Target &L, const Target &R) {
+ if (L.HeadCount != R.HeadCount)
+ return L.HeadCount > R.HeadCount;
+ return L.Node->getFunctionSamples()->getGUID() <
+ R.Node->getFunctionSamples()->getGUID();
+ });
+ LLVM_DEBUG(dbgs() << " Indirect call site " << CallSite.LineOffset << "."
+ << CallSite.Discriminator << ": " << Callees.size()
+ << " profiled targets, total count " << SiteTotal
+ << "\n");
+ size_t MaxAccepted = std::min<size_t>(Callees.size(), ProfileICPMaxProm);
+ size_t NumAccepted = 0;
+ uint64_t MinScaledHeadCount = std::numeric_limits<uint64_t>::max();
----------------
apolloww wrote:
You can drop `MinScaledHeadCount` here, because `Callees` are already sorted
https://github.com/llvm/llvm-project/pull/243631
More information about the llvm-branch-commits
mailing list