[llvm] b0d4ae3 - [Inliner] Don't apply the cold-callsite threshold in non-callable functions (#211255)
via llvm-commits
llvm-commits at lists.llvm.org
Wed Aug 12 06:44:40 PDT 2026
Author: Spencer Bryngelson
Date: 2026-08-12T15:44:35+02:00
New Revision: b0d4ae3b753fd619bef3d14d9e1bf8db12263094
URL: https://github.com/llvm/llvm-project/commit/b0d4ae3b753fd619bef3d14d9e1bf8db12263094
DIFF: https://github.com/llvm/llvm-project/commit/b0d4ae3b753fd619bef3d14d9e1bf8db12263094.diff
LOG: [Inliner] Don't apply the cold-callsite threshold in non-callable functions (#211255)
Partially addresses #211132.
`InlineCostCallAnalyzer::updateThreshold` clamps the threshold to
`-inline-cold-callsite-threshold` (45) when BFI says the callsite is
cold. The reasoning behind the clamp, that cold code does not need to be
fast so it is not worth the code size, does not hold when the caller is
a hardware entry point rather than something callable. Register
allocation there is whole-function and occupancy is set by the worst
case over it, so a call left out of line costs the hot path too, however
cold the call itself is.
So guard the clamp with `isCallableCC(Caller->getCallingConv())`. This
is not AMDGPU-specific: it picks up `SPIR_KERNEL` and `PTX_Kernel` on
the same reasoning. An earlier revision of this PR added a
`TTI::applyColdCallSiteThreshold` hook and returned false from AMDGPU
for entry functions; that was dropped in review in favour of the calling
convention, which states the actual property directly and needs no new
interface.
Where it shows up in practice: DeviceRTL's `__kmpc_parallel_60` is
`always_inline`, so its `OMP_UNLIKELY` branch weights land in every
OpenMP kernel. The microtask call left on the serialized path is then
cold at roughly 1/4000 of entry frequency, stays out of line, and the
kernel stops being a leaf. gfx90a, flang,
`-Rpass-analysis=kernel-resource-usage`, on the reproducer attached to
#211132:
| | VGPRs | scratch | occupancy |
|---|---|---|---|
| before | 212 | 48 B | 2 |
| after | 94 | 0 | 5 |
Scope, since #211132 asks for more than this: the fix only reaches
regions whose inline cost is already under the ordinary threshold. A
larger target region in the same shape costs 3270 against a threshold of
2750 and is still rejected, on cost rather than on coldness, so that
half of #211132 is untouched here. #211136 proposed `alwaysinline` to
cover it; that conflicts with ROCm/llvm-project#3485 and is not the
right answer.
Tested: `llvm/test/Transforms` (11676) and `llvm/test/CodeGen/AMDGPU`
(4920), 16596 tests, 14668 passed with 39 expected failures and no
regression. The inliner tests are `llvm/test/Transforms/Inline`, covered
by the first. The one failure,
`Transforms/ThinLTOBitcodeWriter/no-type-md.ll`, is pre-existing and
fails identically without the patch. The added test fails without the
change and passes with it. End-to-end on gfx942 (MI325X), a WENO5 + HLLC
finite-volume kernel at three sizes, 0.99x / 0.98x / 1.01x, i.e. no
regression where the fix does not apply.
This affects performance-critical applications on large AMD GPU
supercomputers, including [MFC](https://github.com/MFlowCode/MFC).
All numbers above come from the validated reproducers included with this
report and are independently reproducible; they stand on their own.
This was found and root-caused with the assistance of AI tools.
---
Parts of this change were written or audited with Claude Code. I have
reviewed all of it and take
full responsibility for the contribution. See
`llvm/docs/AIToolPolicy.md`.
Added:
llvm/test/Transforms/Inline/AMDGPU/cold-callsite-in-kernel.ll
Modified:
llvm/lib/Analysis/InlineCost.cpp
Removed:
################################################################################
diff --git a/llvm/lib/Analysis/InlineCost.cpp b/llvm/lib/Analysis/InlineCost.cpp
index 6cd84376d8a98..94b1fc6ff7c07 100644
--- a/llvm/lib/Analysis/InlineCost.cpp
+++ b/llvm/lib/Analysis/InlineCost.cpp
@@ -2151,7 +2151,13 @@ void InlineCostCallAnalyzer::updateThreshold(CallBase &Call, Function &Callee) {
// behavior to prevent inlining of hot callsites during ThinLTO
// compile phase.
Threshold = *HotCallSiteThreshold;
- } else if (isColdCallSite(Call, CallerBFI)) {
+ } else if (isCallableCC(Caller->getCallingConv()) &&
+ isColdCallSite(Call, CallerBFI)) {
+ // In a function that is a hardware entry point rather than something
+ // callable, e.g. a GPU kernel, register allocation is whole-function and
+ // occupancy is set by the worst case over it. A call left out of line
+ // there costs the hot path too, however cold the call itself is, so the
+ // reduced threshold does not apply.
LLVM_DEBUG(dbgs() << "Cold callsite.\n");
// Do not apply bonuses for a cold callsite including the
// LastCallToStatic bonus. While this bonus might result in code size
diff --git a/llvm/test/Transforms/Inline/AMDGPU/cold-callsite-in-kernel.ll b/llvm/test/Transforms/Inline/AMDGPU/cold-callsite-in-kernel.ll
new file mode 100644
index 0000000000000..3f4ba6271058d
--- /dev/null
+++ b/llvm/test/Transforms/Inline/AMDGPU/cold-callsite-in-kernel.ll
@@ -0,0 +1,59 @@
+; RUN: opt -passes=inline -mtriple=amdgpu-amd-amdhsa -inline-instr-cost=50 \
+; RUN: -pass-remarks=inline -pass-remarks-missed=inline < %s 2>&1 | FileCheck %s
+
+; A call left out of line in a kernel is register allocated against the worst
+; case for the whole kernel, so it costs the hot path too and the reduced
+; cold-callsite threshold does not apply. It still applies in a callable
+; caller. Both callsites are equally cold and the callee has two uses, so the
+; only
diff erence is the caller's calling convention.
+
+; CHECK-DAG: 'callee' inlined into 'kernel'
+; CHECK-DAG: 'callee' not inlined into 'func' because too costly to inline
+
+define internal void @callee(ptr addrspace(1) %p, i32 %x) {
+entry:
+ %v0 = mul i32 %x, 3
+ %v1 = mul i32 %v0, 4
+ %v2 = mul i32 %v1, 5
+ %v3 = mul i32 %v2, 6
+ %v4 = mul i32 %v3, 7
+ %v5 = mul i32 %v4, 8
+ %v6 = mul i32 %v5, 9
+ %v7 = mul i32 %v6, 10
+ %v8 = mul i32 %v7, 11
+ %v9 = mul i32 %v8, 12
+ %v10 = mul i32 %v9, 13
+ %v11 = mul i32 %v10, 14
+ %v12 = mul i32 %v11, 15
+ %v13 = mul i32 %v12, 16
+ %v14 = mul i32 %v13, 17
+ %v15 = mul i32 %v14, 18
+ store i32 %v15, ptr addrspace(1) %p, align 4
+ ret void
+}
+
+define amdgpu_kernel void @kernel(ptr addrspace(1) %p, i32 %x, i1 %c) {
+entry:
+ br i1 %c, label %cold, label %exit, !prof !0
+
+cold:
+ call void @callee(ptr addrspace(1) %p, i32 %x)
+ br label %exit
+
+exit:
+ ret void
+}
+
+define void @func(ptr addrspace(1) %p, i32 %x, i1 %c) {
+entry:
+ br i1 %c, label %cold, label %exit, !prof !0
+
+cold:
+ call void @callee(ptr addrspace(1) %p, i32 %x)
+ br label %exit
+
+exit:
+ ret void
+}
+
+!0 = !{!"branch_weights", i32 1, i32 4000}
More information about the llvm-commits
mailing list