[clang] [llvm] [clang][Inliner] Add opt-in inliner hints for lambdas and main-file functions (PR #228074)
Kuba Mracek via llvm-commits
llvm-commits at lists.llvm.org
Thu Oct 1 06:28:51 PDT 2026
https://github.com/kubamracek created https://github.com/llvm/llvm-project/pull/228074
Today, inline functions, template instantiations and lambdas (inside those) are emitted as linkonce_odr:
- we expect another TU may use the out-of-line copy, we want one optimized version to be shared across TUs
- the inliner doesn't treat a call as the last one to linkonce_odr functions, even if we observe that in the TU
This patch lets Clang pass down information to the inliner about such callees, allowing lambdas, main-file functions, and template/inline functions from main files, to be more likely to inline and optimize further. This patch just introduces the mechanism, the thresholds, but keeps all inlining changes off by default.
Note that placing templates/inlines/lambdas into an anonymous namespace eliminates the linkonce_odr linkage, inlines well already, and achieves the goal already. This PR is improving the situations where the source code doesn't explicitly use anonymous namespaces or other mechanisms to make the linkage internal, which is quite common in C++ code in the wild.
Motivating cases:
- Lambdas used in STL algorithms can be just over the inline threshold (body not large, just not trivial). Inlining has large outsized benefits e.g. on captures (they can stay as local variables at the caller) or on optimizations beyond one loop iteration. Example:
```c++
template <class V>
long foo(const V& v, int k0, int k1) {
return std::count_if(v.begin(), v.end(), [&](int e) {
int x = ...; // non-trivial, with some branches
return x > e;
});
}
```
Godbolt link: https://godbolt.org/z/6WWjPrPWT (GCC does inline in this case)
- An immediately invoked lambda has exactly one call site, inlining it can't duplicate code. Example:
```c++
template <class T>
T bar(T a, T k) {
const T r = [&] {
T x = ...; // non-trivial, with some branches
return x;
}();
return r + a;
}
```
Godbolt link: https://godbolt.org/z/dz6EYnG1e (GCC does inline in this case too)
Implementation approach is:
- Characterize certain (usually linkonce_odr) functions in Clang and pass that information as function attributes, namely:
- `"clang-lambda"` for lambdas
- `"clang-main-file"` when the function is defined in a main source file (and not a header file)
- `"clang-main-file"="inline-or-template"` meaning it's also a function that is inline or a template instantiation)
- Under `-mllvm -inline-use-clang-hints` (off by default), InlineCost raises the threshold for such callees.
- The bonuses are tiered: lambdas defined in the main file get the largest boost, other lambdas get a smaller bonus, and main-file inline/template functions get yet a smaller bonus.
- Main-file functions that are neither inline nor templates (plain functions) get no bonus.
- Additionally, a linkonce_odr lambda defined in the main file is treated as a local function for the purposes of the sole-call (last-call-to-static) bonus. Ignoring unusual setups like `#include`-ing a .cpp file, no other TU can reuse such a lambda, so inlining its only call site is expected to leave the original dead. In the unusual setup case (e.g. "Unity builds"), such lambdas still get the pre-existing behavior of no inlining.
>From 8c5319cf5ccae6701f4da1163494281249744638 Mon Sep 17 00:00:00 2001
From: Kuba Mracek <mracek at apple.com>
Date: Thu, 1 Oct 2026 11:57:38 +0100
Subject: [PATCH] [clang][Inliner] Add opt-in inliner hints for lambdas and
main-file functions
Clang now marks function definitions with "clang-lambda" (the function is
a lambda) and "clang-main-file" (the function is defined in the main source
file, as opposed to in a header-file) with the value of "inline-or-template" if
it is declared inline or is a template instantiation.
With -mllvm -inline-use-clang-hints (off by default), the inliner raises the
thresholds for such callees, and makes lambdas, template/inline code defined in
main source, and especially the combination of both. Also, allows sole-call
bonus for the lambda-in-a-main-file situation (we can reasonably expect the
original lambda to be completely unused after inlining, even across TUs).
This makes this specific subset of typically linkonce_odr functions more likely
to get inlined and optimized.
No bonus applies to minsize callers or to cold call sites or callees.
---
clang/lib/CodeGen/CodeGenModule.cpp | 16 +++++
clang/test/CodeGenCXX/inline-origin-hints.cpp | 29 +++++++++
llvm/lib/Analysis/InlineCost.cpp | 55 +++++++++++++++-
llvm/test/Transforms/Inline/clang-hints.ll | 63 +++++++++++++++++++
4 files changed, 161 insertions(+), 2 deletions(-)
create mode 100644 clang/test/CodeGenCXX/inline-origin-hints.cpp
create mode 100644 llvm/test/Transforms/Inline/clang-hints.ll
diff --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index d19d64e2e1c60..a2756988464a1 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -3296,6 +3296,22 @@ void CodeGenModule::SetLLVMFunctionAttributesForDefinition(const Decl *D,
if (CodeGenOpts.DisableOutlining || D->hasAttr<NoOutlineAttr>())
B.addAttribute(llvm::Attribute::NoOutline);
+ // Hints for the optimizer (see -mllvm -inline-use-clang-hints) helping with
+ // linkonce_odr C++ functions: is this a lambda, is it defined in the main
+ // source file, is it declared inline or a template instantiation.
+ if (isLambdaCallOperator(dyn_cast<DeclContext>(D)))
+ B.addAttribute("clang-lambda");
+ // D->getLocation(), for a template instantiation, is the location of the
+ // template's definition, not instantiation.
+ const SourceManager &SM = getContext().getSourceManager();
+ if (SM.isInMainFile(D->getLocation())) {
+ const auto *FD = dyn_cast<FunctionDecl>(D);
+ bool InlineOrTemplate =
+ FD && (FD->isInlineSpecified() || FD->isTemplateInstantiation());
+ B.addAttribute("clang-main-file",
+ InlineOrTemplate ? "inline-or-template" : "");
+ }
+
F->addFnAttrs(B);
llvm::MaybeAlign ExplicitAlignment;
diff --git a/clang/test/CodeGenCXX/inline-origin-hints.cpp b/clang/test/CodeGenCXX/inline-origin-hints.cpp
new file mode 100644
index 0000000000000..f73708a8309c0
--- /dev/null
+++ b/clang/test/CodeGenCXX/inline-origin-hints.cpp
@@ -0,0 +1,29 @@
+// RUN: rm -rf %t
+// RUN: split-file %s %t
+// RUN: %clang_cc1 -triple arm64-apple-macosx14.0.0 -std=c++20 -emit-llvm -o - %t/main.cpp | FileCheck %s
+
+//--- hdr.h
+inline int hdr_inline(int x) { return x * 3; }
+template <class F> int hdr_call(F f) { return f(); }
+
+//--- main.cpp
+#include "hdr.h"
+
+int main_plain(int x) { return x + 1; }
+inline int main_inline(int x) { return x * 7; }
+template <class T> int main_tmpl(T x) { return hdr_call([&] { return x * 2; }); }
+
+int use(int x) { return main_plain(x) + main_inline(x) + main_tmpl(x) + hdr_inline(x); }
+
+// CHECK: define{{.*}} i32 @_Z10main_plaini({{.*}}) #[[PLAIN:[0-9]+]]
+// CHECK: define{{.*}} i32 @_Z3usei({{.*}}) #[[PLAIN]]
+// CHECK: define{{.*}} i32 @_Z11main_inlinei({{.*}}) #[[INL:[0-9]+]]
+// CHECK: define{{.*}} i32 @_Z9main_tmplIiEiT_({{.*}}) #[[INL]]
+// CHECK: define{{.*}} i32 @_Z10hdr_inlinei({{.*}}) #[[HDR:[0-9]+]]
+// CHECK: define{{.*}} i32 @_ZZ9main_tmplIiEiT_ENKUlvE_clEv({{.*}}) #[[LAMBDA:[0-9]+]]
+
+// CHECK: attributes #[[PLAIN]] = { {{.*}}"clang-main-file" {{.*}} }
+// CHECK: attributes #[[INL]] = { {{.*}}"clang-main-file"="inline-or-template"{{.*}} }
+// CHECK: attributes #[[HDR]] = {
+// CHECK-NOT: clang-
+// CHECK: attributes #[[LAMBDA]] = { {{.*}}"clang-lambda" "clang-main-file"="inline-or-template"{{.*}} }
diff --git a/llvm/lib/Analysis/InlineCost.cpp b/llvm/lib/Analysis/InlineCost.cpp
index 0a540c1b8fb67..53bcde650c65c 100644
--- a/llvm/lib/Analysis/InlineCost.cpp
+++ b/llvm/lib/Analysis/InlineCost.cpp
@@ -183,6 +183,27 @@ static cl::opt<bool> InlineAllViableCalls(
"inline-all-viable-calls", cl::Hidden, cl::init(false),
cl::desc("Inline all viable calls, even if they exceed the inlining "
"threshold"));
+
+static cl::opt<bool> UseClangHints(
+ "inline-use-clang-hints", cl::Hidden, cl::init(false),
+ cl::desc("Use clang's lambda/main-file hints to raise thresholds"));
+
+static cl::opt<int> ClangHintsLambdaBonus(
+ "clang-hints-lambda-bonus", cl::Hidden, cl::init(500),
+ cl::desc("Threshold bonus for lambdas"));
+
+static cl::opt<int> ClangHintsMainFileTemplateBonus(
+ "clang-hints-main-file-template-bonus", cl::Hidden, cl::init(150),
+ cl::desc("Threshold bonus for main-file inline/template functions"));
+
+static cl::opt<int> ClangHintsLambdaMainFileBonus(
+ "clang-hints-lambda-main-file-bonus", cl::Hidden, cl::init(3000),
+ cl::desc("Threshold bonus for lambdas in the main file"));
+
+static cl::opt<bool> ClangHintsLocalLambdas(
+ "clang-hints-local-lambdas", cl::Hidden, cl::init(true),
+ cl::desc("Treat main-file linkonce_odr lambdas as local functions"));
+
namespace llvm {
std::optional<int> getStringFnAttrAsInt(const Attribute &Attr) {
if (Attr.isValid()) {
@@ -1247,10 +1268,20 @@ class InlineCostCallAnalyzer final : public CallAnalyzer {
};
// Return true if CB is the sole call to local function Callee.
+//
+// With UseClangHints and ClangHintsLocalLambdas, a linkonce_odr function
+// that's (1) a lambda, and (2) defined in the main source file, we count as
+// local too. Ignoring very unusual setups (#include-ing a .cpp file), no other
+// translation unit can reuse such lambda, so we expect inlining it turns the
+// original dead.
static bool isSoleCallToLocalFunction(const CallBase &CB,
const Function &Callee) {
- return Callee.hasLocalLinkage() && Callee.hasOneLiveUse() &&
- &Callee == CB.getCalledFunction();
+ bool Local = Callee.hasLocalLinkage() ||
+ (UseClangHints && ClangHintsLocalLambdas &&
+ Callee.hasLinkOnceODRLinkage() &&
+ Callee.hasFnAttribute("clang-lambda") &&
+ Callee.hasFnAttribute("clang-main-file"));
+ return Local && Callee.hasOneLiveUse() && &Callee == CB.getCalledFunction();
}
class InlineCostFeaturesAnalyzer final : public CallAnalyzer {
@@ -2104,12 +2135,14 @@ void InlineCostCallAnalyzer::updateThreshold(CallBase &Call, Function &Callee) {
int SingleBBBonusPercent = 50;
int VectorBonusPercent = TTI.getInlinerVectorBonusPercent();
int LastCallToStaticBonus = TTI.getInliningLastCallToStaticBonus();
+ bool ClangHintBonusAllowed = true;
// Lambda to set all the above bonus and bonus percentages to 0.
auto DisallowAllBonuses = [&]() {
SingleBBBonusPercent = 0;
VectorBonusPercent = 0;
LastCallToStaticBonus = 0;
+ ClangHintBonusAllowed = false;
};
// Use the OptMinSizeThreshold or OptSizeThreshold knob if they are available
@@ -2122,6 +2155,7 @@ void InlineCostCallAnalyzer::updateThreshold(CallBase &Call, Function &Callee) {
// call/return instructions.
SingleBBBonusPercent = 0;
VectorBonusPercent = 0;
+ ClangHintBonusAllowed = false;
} else if (Caller->hasOptSize())
Threshold = MinIfValid(Threshold, Params.OptSizeThreshold);
@@ -2194,6 +2228,23 @@ void InlineCostCallAnalyzer::updateThreshold(CallBase &Call, Function &Callee) {
SingleBBBonus = Threshold * SingleBBBonusPercent / 100;
VectorBonus = Threshold * VectorBonusPercent / 100;
+ // A lambda is usually specific to its call sites, very often there's only
+ // one call site, and inlining is especially benefitial: Captured variables
+ // turn back into the caller's locals. A function defined in the main source
+ // file (very typically) isn't used in other translation unit, inlining it
+ // cannot multiply copies across them (that's what linkonce_odr tries to
+ // prevent). Boost those situations.
+ if (UseClangHints && ClangHintBonusAllowed) {
+ bool IsLambda = Callee.hasFnAttribute("clang-lambda");
+ Attribute MainFile = Callee.getFnAttribute("clang-main-file");
+ if (IsLambda && MainFile.isValid())
+ Threshold += ClangHintsLambdaMainFileBonus;
+ else if (IsLambda)
+ Threshold += ClangHintsLambdaBonus;
+ else if (MainFile.getValueAsString() == "inline-or-template")
+ Threshold += ClangHintsMainFileTemplateBonus;
+ }
+
// If there is only one call of the function, and it has internal linkage,
// the cost of inlining it drops dramatically. It may seem odd to update
// Cost in updateThreshold, but the bonus depends on the logic in this method.
diff --git a/llvm/test/Transforms/Inline/clang-hints.ll b/llvm/test/Transforms/Inline/clang-hints.ll
new file mode 100644
index 0000000000000..01b3719ce31f8
--- /dev/null
+++ b/llvm/test/Transforms/Inline/clang-hints.ll
@@ -0,0 +1,63 @@
+; RUN: opt -passes=inline -inline-threshold=0 -S < %s | FileCheck %s --check-prefix=OFF
+; RUN: opt -passes=inline -inline-threshold=0 -inline-use-clang-hints -S < %s | FileCheck %s --check-prefix=HINTS
+; RUN: opt -passes=inline -inline-threshold=0 -inline-use-clang-hints -clang-hints-lambda-main-file-bonus=0 -S < %s | FileCheck %s --check-prefix=LOCAL
+; RUN: opt -passes=inline -inline-threshold=0 -inline-use-clang-hints -clang-hints-lambda-main-file-bonus=0 -clang-hints-local-lambdas=false -S < %s | FileCheck %s --check-prefix=NOLOCAL
+
+; OFF-LABEL: define i32 @caller(
+; OFF-COUNT-2: call i32 @lambda(
+; OFF-COUNT-2: call i32 @main_template(
+; OFF-COUNT-2: call i32 @main_plain(
+; OFF: call i32 @sole_main_inline(
+
+; HINTS-LABEL: define i32 @caller(
+; HINTS-NOT: call i32 @lambda(
+; HINTS-COUNT-2: call i32 @main_template(
+; HINTS-COUNT-2: call i32 @main_plain(
+; HINTS-NOT: call i32 @sole_main_inline(
+; HINTS: ret i32
+
+; LOCAL-LABEL: define i32 @caller(
+; LOCAL-NOT: call i32 @sole_main_inline(
+; LOCAL: ret i32
+
+; NOLOCAL-LABEL: define i32 @caller(
+; NOLOCAL: call i32 @sole_main_inline(
+define i32 @caller(i32 %x) {
+ %a1 = call i32 @lambda(i32 %x)
+ %a2 = call i32 @lambda(i32 %a1)
+ %b1 = call i32 @main_template(i32 %a2)
+ %b2 = call i32 @main_template(i32 %b1)
+ %c1 = call i32 @main_plain(i32 %b2)
+ %c2 = call i32 @main_plain(i32 %c1)
+ %d = call i32 @sole_main_inline(i32 %c2)
+ ret i32 %d
+}
+
+define linkonce_odr i32 @lambda(i32 %x) "clang-lambda" "function-inline-cost"="300" {
+ %r = add i32 %x, 1
+ ret i32 %r
+}
+
+define linkonce_odr i32 @main_template(i32 %x) "clang-main-file"="inline-or-template" "function-inline-cost"="300" {
+ %r = add i32 %x, 2
+ ret i32 %r
+}
+
+define linkonce_odr i32 @main_plain(i32 %x) "clang-main-file" "function-inline-cost"="300" {
+ %r = add i32 %x, 3
+ ret i32 %r
+}
+
+define linkonce_odr i32 @sole_main_inline(i32 %x) "clang-lambda" "clang-main-file" {
+ %v1 = mul i32 %x, 3
+ %v2 = mul i32 %v1, 5
+ %v3 = mul i32 %v2, 7
+ %v4 = mul i32 %v3, 11
+ %v5 = mul i32 %v4, 13
+ %v6 = mul i32 %v5, 17
+ %v7 = mul i32 %v6, 19
+ %v8 = mul i32 %v7, 23
+ %v9 = mul i32 %v8, 29
+ %v10 = mul i32 %v9, 31
+ ret i32 %v10
+}
More information about the llvm-commits
mailing list