[llvm-branch-commits] [llvm] [llvm-profgen] Account for indirect-call promotion limits in the CS pre-inliner (PR #243631)

Wei Wang via llvm-branch-commits llvm-branch-commits at lists.llvm.org
Wed Nov 18 23:12:30 PST 2026


================
@@ -102,6 +132,105 @@ std::vector<FunctionId> CSPreInliner::buildTopDownOrder() {
   return Order;
 }
 
+// Find indirect call targets to exclude from the pre-inline plan. Apply the
+// loader's promotion limits, using the contextless hot threshold as described
+// in the constructor. The loader (see
+// inlineHotFunctionsWithPriority) walks a call's targets from hottest to
+// coldest and promotes one only if it's hot, is at least
+// ProfileICPRelativeHotness percent of the site total (once the first
+// ProfileICPRelativeHotnessSkip targets are promoted), and fewer than
+// ProfileICPMaxProm targets have been promoted so far. Keep rejected targets
+// out of the caller's inline size budget and leave their contexts for merging
+// into their base profiles.
+//
+// Inlining a target in the profile replaces its call-target count with its
+// head-sample estimate. This can change the total used by the percentage check.
+// Accept the target only if it and all previously accepted targets still pass.
+// Otherwise, reject this target and the remaining, colder targets.
+DenseSet<const ContextTrieNode *>
+CSPreInliner::getICPRejectedCallees(ContextTrieNode *CallerNode,
+                                    const FunctionSamples *CallerSamples) {
+  DenseSet<const ContextTrieNode *> Rejected;
+  // Only a pseudo probe identifies a single call instruction. With line-based
+  // profiles, distinct direct calls on one source line share a call site, so
+  // several targets there do not imply an indirect call.
+  if (!CSPreinlHonorICPLimits || SamplePreInlineReplay ||
+      !FunctionSamples::ProfileIsProbeBased)
+    return Rejected;
+
+  struct Target {
+    ContextTrieNode *Node;
+    uint64_t HeadCount;
+    uint64_t CallTargetCount = 0;
+  };
+  std::map<LineLocation, SmallVector<Target, 4>> CallSites;
+  for (auto &Child : CallerNode->getAllChildContext()) {
+    ContextTrieNode *CalleeNode = &Child.second;
+    if (FunctionSamples *CalleeSamples = CalleeNode->getFunctionSamples())
+      CallSites[CalleeNode->getCallSiteLoc()].push_back(
+          {CalleeNode, CalleeSamples->getHeadSamplesEstimate()});
+  }
+
+  for (auto &[CallSite, Callees] : CallSites) {
+    // Apply ICP checks only when a decoded probe identifies an indirect call.
+    // Multiple targets alone are not enough: a direct call can record several
+    // functions when inline frames are lost. Without a matching probe, leave
+    // the inline decision to the existing size and hotness checks.
+    if (!Binary.isIndirectCallProbe(CallerNode->getFuncName().getHashCode(),
+                                    CallSite.LineOffset))
+      continue;
+
+    // Site total with every target outlined, i.e. the raw call target map.
+    uint64_t SiteTotal = 0;
+    if (auto CallTargets = CallerSamples->findCallTargetMapAt(CallSite)) {
+      for (const auto &[Name, Count] : *CallTargets)
+        SiteTotal += Count;
+      for (Target &T : Callees) {
+        auto It = CallTargets->find(T.Node->getFuncName());
+        if (It != CallTargets->end())
+          T.CallTargetCount = It->second;
+      }
+    }
+
+    llvm::sort(Callees, [](const Target &L, const Target &R) {
+      if (L.HeadCount != R.HeadCount)
+        return L.HeadCount > R.HeadCount;
+      return L.Node->getFunctionSamples()->getGUID() <
+             R.Node->getFunctionSamples()->getGUID();
+    });
+    LLVM_DEBUG(dbgs() << "  Indirect call site " << CallSite.LineOffset << "."
+                      << CallSite.Discriminator << ": " << Callees.size()
+                      << " profiled targets, total count " << SiteTotal
+                      << "\n");
+    size_t MaxAccepted = std::min<size_t>(Callees.size(), ProfileICPMaxProm);
+    size_t NumAccepted = 0;
+    uint64_t MinScaledHeadCount = std::numeric_limits<uint64_t>::max();
----------------
apolloww wrote:

You can drop `MinScaledHeadCount` here, because `Callees` are already sorted

https://github.com/llvm/llvm-project/pull/243631


More information about the llvm-branch-commits mailing list