[llvm] [PGO] Do not promote an indirect call over a known set of targets (PR #222518)
Nazar Mokrynskyi via llvm-commits
llvm-commits at lists.llvm.org
Wed Sep 9 22:49:31 PDT 2026
https://github.com/nazar-pc created https://github.com/llvm/llvm-project/pull/222518
This is the second part of the solution to https://github.com/llvm/llvm-project/issues/222292 (the first and orthogonal part is in https://github.com/llvm/llvm-project/pull/222514).
Despite LLM wrote most of it under my guidance, I edited and audited the changes to the best of my ability.
---
### Problem
A `switch` that selects a handler out of a set of functions and calls it (the dispatch step of a tail-call-threaded interpreter) reaches indirect call promotion as a phi of function addresses feeding an indirect call:
```llvm
dispatch:
%handler = phi ptr [ @op_add, %sw.bb0 ], [ @op_xor, %sw.bb1 ], ...
%r = musttail call i64 %handler(ptr %pc, ptr %regs), !prof !0
```
Without a profile this collapses: SimplifyCFG folds the phi into a lookup table, and each handler ends in a single load and one indirect jump, about 18 bytes of dispatch.
With a profile, ICP guards the hot target with `icmp eq ptr %handler, @op_hot`. Jump-threading then folds that compare back through the phi, giving the hot case an edge of its own. The phi's incoming edges no longer share a destination, so the lookup table is never formed, and every case is left materializing a function address of its own. A threaded interpreter is one such switch per handler, so the whole dispatch table gets replicated into every handler.
The promotion buys nothing here in exchange. The callee is a phi of function constants, so the set of possible targets is already statically known. A profile-driven guess adds no devirtualization the compiler could not reach on its own, and specializing per path is jump threading's and SimplifyCFG's job, not something a value guard is needed for.
### Fix
Skip a call site whose callee operand is a phi all of whose incoming values are functions. A phi with even one unknown incoming value is promoted as before, as is any other callee expression (a load from a vtable or a dispatch array, for instance) - those are not statically known target sets and promotion still does real work there.
### Effect
On a C threaded interpreter with 64 opcodes, `.text`:
| | bytes | dispatch per handler |
|---|---|---|
| no profile | 9672 | ~18 B |
| `-fprofile-use` | 63356 | ~830 B |
| `-fprofile-use` with this patch | 9612 | ~18 B |
The patched size matches `-fprofile-use -mllvm -disable-icp` exactly, i.e. the dispatch goes back to the shape it has without a profile while the rest of PGO is unaffected.
`llvm/test/Transforms/PGOProfile/icp_known_target_set.ll` covers the three cases: a phi of functions is not promoted, a loaded callee still is, and a phi with a non-constant incoming value still is.
>From 7f5d739fd54bb83a68b668f8fa08e99b81d1909a Mon Sep 17 00:00:00 2001
From: Nazar Mokrynskyi <nazar at mokrynskyi.com>
Date: Thu, 10 Sep 2026 05:42:34 +0000
Subject: [PATCH] [PGO] Do not promote an indirect call over a known set of
targets
A call whose callee operand is a phi of functions already dispatches over a
statically known set of targets. Promoting one of them adds no
devirtualization the compiler cannot reach on its own, and it costs the
shape needed to fold the phi into a lookup table: jump threading folds the
promotion guard back into the switch that feeds the phi, the phi's incoming
edges stop sharing a destination, and SimplifyCFG can no longer form the
table. Every case then materializes a function address of its own.
A tail call threaded interpreter dispatches exactly like that, once per
handler, so the whole dispatch table ends up replicated into every handler.
On a C interpreter with 64 opcodes this grows .text from 9672 to 63356
bytes, and a handler from 18 to roughly 830 bytes of dispatch. Skipping the
promotion keeps .text at 9612 bytes, the size it has without a profile.
---
.../Instrumentation/IndirectCallPromotion.cpp | 29 ++++++
.../PGOProfile/icp_known_target_set.ll | 90 +++++++++++++++++++
2 files changed, 119 insertions(+)
create mode 100644 llvm/test/Transforms/PGOProfile/icp_known_target_set.ll
diff --git a/llvm/lib/Transforms/Instrumentation/IndirectCallPromotion.cpp b/llvm/lib/Transforms/Instrumentation/IndirectCallPromotion.cpp
index f18287ba9de27..104394c66d067 100644
--- a/llvm/lib/Transforms/Instrumentation/IndirectCallPromotion.cpp
+++ b/llvm/lib/Transforms/Instrumentation/IndirectCallPromotion.cpp
@@ -871,12 +871,41 @@ bool IndirectCallPromoter::tryToPromoteWithVTableCmp(
return true;
}
+// Returns true if the call already dispatches over a statically known set of
+// targets, that is, its callee operand is a phi node whose incoming values are
+// all functions.
+//
+// A switch that selects a handler out of a table of functions and calls it
+// reaches indirect call promotion in this shape, before SimplifyCFG folds the
+// phi into a lookup table. Guarding one of the targets there buys no
+// devirtualization that the compiler could not do on its own, and it costs the
+// lookup table: the guard is threaded back into the switch, the phi's incoming
+// edges no longer share a destination, and every case ends up materializing a
+// function address of its own. A threaded interpreter, which is one such
+// switch per handler, then carries a copy of the whole dispatch table in every
+// handler.
+static bool hasKnownTargetSet(const CallBase &CB) {
+ const auto *Phi = dyn_cast<PHINode>(CB.getCalledOperand());
+ if (!Phi)
+ return false;
+
+ return all_of(Phi->incoming_values(), [](const Value *V) {
+ return isa<Function>(V->stripPointerCasts());
+ });
+}
+
// Traverse all the indirect-call callsite and get the value profile
// annotation to perform indirect-call promotion.
bool IndirectCallPromoter::processFunction(ProfileSummaryInfo *PSI) {
bool Changed = false;
ICallPromotionAnalysis ICallAnalysis;
for (auto *CB : findIndirectCalls(F)) {
+ if (hasKnownTargetSet(*CB)) {
+ LLVM_DEBUG(dbgs() << "Don't promote a call over a known set of targets: "
+ << *CB << "\n");
+ continue;
+ }
+
uint32_t NumCandidates;
uint64_t TotalCount;
auto ICallProfDataRef = ICallAnalysis.getPromotionCandidatesForInstruction(
diff --git a/llvm/test/Transforms/PGOProfile/icp_known_target_set.ll b/llvm/test/Transforms/PGOProfile/icp_known_target_set.ll
new file mode 100644
index 0000000000000..16210bf714df4
--- /dev/null
+++ b/llvm/test/Transforms/PGOProfile/icp_known_target_set.ll
@@ -0,0 +1,90 @@
+; A call whose callee is a phi of functions already dispatches over a known set
+; of targets, so it is not promoted. Promoting it there would not devirtualize
+; anything the compiler cannot see on its own, and it would keep the phi from
+; being folded into a lookup table.
+
+; RUN: opt < %s -passes=pgo-icall-prom -S | FileCheck %s
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+ at target = external global ptr
+
+define i64 @handler_a(ptr %arg) {
+ ret i64 0
+}
+
+define i64 @handler_b(ptr %arg) {
+ ret i64 1
+}
+
+define i64 @handler_c(ptr %arg) {
+ ret i64 2
+}
+
+define i64 @loaded_target(ptr %arg) {
+ ret i64 3
+}
+
+define i64 @dispatch_over_phi(i8 %op, ptr %arg) {
+; CHECK-LABEL: define i64 @dispatch_over_phi(
+; CHECK-NOT: icmp eq ptr
+; CHECK: %call = call i64 %handler(ptr %arg)
+entry:
+ switch i8 %op, label %c [
+ i8 0, label %a
+ i8 1, label %b
+ ]
+
+a:
+ br label %dispatch
+
+b:
+ br label %dispatch
+
+c:
+ br label %dispatch
+
+dispatch:
+ %handler = phi ptr [ @handler_a, %a ], [ @handler_b, %b ], [ @handler_c, %c ]
+ %call = call i64 %handler(ptr %arg), !prof !0
+ ret i64 %call
+}
+
+; A callee that is not a phi of functions is promoted as before.
+
+define i64 @dispatch_over_load(ptr %arg) {
+; CHECK-LABEL: define i64 @dispatch_over_load(
+; CHECK: icmp eq ptr %handler, @loaded_target
+; CHECK: call i64 @loaded_target(ptr %arg)
+entry:
+ %handler = load ptr, ptr @target, align 8
+ %call = call i64 %handler(ptr %arg), !prof !1
+ ret i64 %call
+}
+
+; A phi with an unknown incoming value is promoted as before.
+
+define i64 @dispatch_over_mixed_phi(i1 %cond, ptr %arg) {
+; CHECK-LABEL: define i64 @dispatch_over_mixed_phi(
+; CHECK: icmp eq ptr %handler, @handler_a
+; CHECK: call i64 @handler_a(ptr %arg)
+entry:
+ %loaded = load ptr, ptr @target, align 8
+ br i1 %cond, label %a, label %other
+
+a:
+ br label %dispatch
+
+other:
+ br label %dispatch
+
+dispatch:
+ %handler = phi ptr [ @handler_a, %a ], [ %loaded, %other ]
+ %call = call i64 %handler(ptr %arg), !prof !2
+ ret i64 %call
+}
+
+!0 = !{!"VP", i32 0, i64 10000, i64 9552685272606748865, i64 9000}
+!1 = !{!"VP", i32 0, i64 10000, i64 1327993795398320997, i64 9000}
+!2 = !{!"VP", i32 0, i64 10000, i64 9552685272606748865, i64 9000}
More information about the llvm-commits
mailing list