[llvm-branch-commits] [llvm] [CSPreInliner] Use always_inline/noinline attributes in pre-inliner (PR #234094)
Wei Wang via llvm-branch-commits
llvm-branch-commits at lists.llvm.org
Tue Oct 20 23:13:16 PDT 2026
https://github.com/apolloww created https://github.com/llvm/llvm-project/pull/234094
Teach the CSSPGO pre-inliner to respect the always_inline and noinline
attributes encoded in pseudo probe descriptors
(-preinliner-use-inline-attributes, default off). The attributes are stamped
onto the context profiles of each function. shouldInline() never inlines a
noinline callee and always inlines an always_inline one, since the compiler
enforces both whatever the plan says.
An always_inline callee is not charged to the caller's size budget, and it is
admitted even when that budget is exhausted: the budget loop used to stop
before shouldInline() was reached, so the plan recorded the callee as not
inlined while the compiler inlines it anyway.
-preinliner-charge-always-inline-size (default off) charges an always_inline
callee's size to the budget instead, so an always_inline subtree can crowd out
ordinary candidates; the callee itself is still always inlined.
>From 31f3cd668015abdc030e39acd58bee478a7c16d9 Mon Sep 17 00:00:00 2001
From: Wei Wang <apollo.mobility at gmail.com>
Date: Sun, 18 Oct 2026 21:37:07 -0700
Subject: [PATCH] [CSPreInliner] Use always_inline/noinline attributes in
pre-inliner
Teach the CSSPGO pre-inliner to respect the always_inline and noinline
attributes encoded in pseudo probe descriptors
(-preinliner-use-inline-attributes, default off). The attributes are stamped
onto the context profiles of each function. shouldInline() never inlines a
noinline callee and always inlines an always_inline one, since the compiler
enforces both whatever the plan says.
An always_inline callee is not charged to the caller's size budget, and it is
admitted even when that budget is exhausted: the budget loop used to stop
before shouldInline() was reached, so the plan recorded the callee as not
inlined while the compiler inlines it anyway.
-preinliner-charge-always-inline-size (default off) charges an always_inline
callee's size to the budget instead, so an always_inline subtree can crowd out
ordinary candidates; the callee itself is still always inlined.
---
llvm/include/llvm/ProfileData/SampleProf.h | 2 +
.../cs-preinline-inline-attributes.perfscript | 109 ++++++++++++++++++
.../X86/cs-preinline-inline-attributes.test | 109 ++++++++++++++++++
llvm/tools/llvm-profgen/CSPreInliner.cpp | 60 +++++++++-
llvm/tools/llvm-profgen/ProfiledBinary.h | 3 +
5 files changed, 281 insertions(+), 2 deletions(-)
create mode 100644 llvm/test/tools/llvm-profgen/X86/Inputs/cs-preinline-inline-attributes.perfscript
create mode 100644 llvm/test/tools/llvm-profgen/X86/cs-preinline-inline-attributes.test
diff --git a/llvm/include/llvm/ProfileData/SampleProf.h b/llvm/include/llvm/ProfileData/SampleProf.h
index b2d42ed61c473d..cd4a974fbd9be5 100644
--- a/llvm/include/llvm/ProfileData/SampleProf.h
+++ b/llvm/include/llvm/ProfileData/SampleProf.h
@@ -563,6 +563,8 @@ enum ContextAttributeMask {
ContextShouldBeInlined = 0x2, // Leaf of context should be inlined
ContextDuplicatedIntoBase =
0x4, // Leaf of context is duplicated into the base profile
+ ContextAlwaysInline = 0x8, // Function has always_inline attribute
+ ContextNoInline = 0x10, // Function has noinline attribute
};
// Represents a context frame with profile function and line location
diff --git a/llvm/test/tools/llvm-profgen/X86/Inputs/cs-preinline-inline-attributes.perfscript b/llvm/test/tools/llvm-profgen/X86/Inputs/cs-preinline-inline-attributes.perfscript
new file mode 100644
index 00000000000000..0e78e8bd70a41f
--- /dev/null
+++ b/llvm/test/tools/llvm-profgen/X86/Inputs/cs-preinline-inline-attributes.perfscript
@@ -0,0 +1,109 @@
+PERF_RECORD_MMAP2 2790937/2790937: [0x401000(0x1000) @ 0x1000 00:2a 781039272 758300]: r-xp /home/cs-preinline-inline-attributes.perfbin
+
+ 401194
+ 7fa23062a610
+ 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 40115a
+ 7fa23062a610
+ 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 4011a9
+ 7fa23062a610
+ 0x4011db/0x4011d0/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011e6/0x401140/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 401190
+ 7fa23062a610
+ 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401222/0x401149/P/-/-/0 0x401143/0x401210/P/-/-/0 0x4011e6/0x401140/P/-/-/0 0x4011c7/0x4011dd/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 4011a5
+ 7fa23062a610
+ 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 4011d0
+ 7fa23062a610
+ 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011e6/0x401140/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 401222
+ 7fa23062a610
+ 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 401199
+ 7fa23062a610
+ 0x40111a/0x4011f3/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011e6/0x401140/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 40119d
+ 7fa23062a610
+ 0x4011e6/0x401140/P/-/-/0 0x4011e6/0x401140/P/-/-/0 0x4011c7/0x4011dd/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011db/0x4011d0/P/-/-/0
+
+ 40119d
+ 7fa23062a610
+ 0x40111a/0x4011f3/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011e6/0x401140/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 40115a
+ 7fa23062a610
+ 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 4011e3
+ 7fa23062a610
+ 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011e6/0x401140/P/-/-/0
+
+ 40115e
+ 7fa23062a610
+ 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 401199
+ 7fa23062a610
+ 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 401194
+ 7fa23062a610
+ 0x4011e6/0x401140/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 40118b
+ 7fa23062a610
+ 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 4011a5
+ 7fa23062a610
+ 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401222/0x401149/P/-/-/0 0x401143/0x401210/P/-/-/0 0x4011e6/0x401140/P/-/-/0 0x4011c7/0x4011dd/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 4011bc
+ 7fa23062a610
+ 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401222/0x401149/P/-/-/0 0x401143/0x401210/P/-/-/0 0x4011e6/0x401140/P/-/-/0
+
+ 4011a9
+ 7fa23062a610
+ 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401222/0x401149/P/-/-/0 0x401143/0x401210/P/-/-/0 0x4011e6/0x401140/P/-/-/0 0x4011c7/0x4011dd/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 401163
+ 7fa23062a610
+ 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 4011d2
+ 7fa23062a610
+ 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011e6/0x401140/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0
+
+ 4011c0
+ 7fa23062a610
+ 0x4011ac/0x401190/P/-/-/0 0x401222/0x401149/P/-/-/0 0x401143/0x401210/P/-/-/0 0x4011e6/0x401140/P/-/-/0 0x4011c7/0x4011dd/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011db/0x4011d0/P/-/-/0
+
+ 40119d
+ 7fa23062a610
+ 0x401209/0x401140/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011e6/0x401140/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 4011a5
+ 7fa23062a610
+ 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011e6/0x401140/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 401190
+ 7fa23062a610
+ 0x4011db/0x4011d0/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011e6/0x401140/P/-/-/0 0x4011db/0x4011d0/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0
+
+ 40115e
+ 7fa23062a610
+ 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401222/0x401149/P/-/-/0
+
+ 4011a5
+ 7fa23062a610
+ 0x401222/0x401149/P/-/-/0 0x401143/0x401210/P/-/-/0 0x4011e6/0x401140/P/-/-/0 0x4011c7/0x4011dd/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x4011ac/0x401190/P/-/-/0 0x401209/0x401140/P/-/-/0 0x40111a/0x4011f3/P/-/-/0 0x4011ee/0x401110/P/-/-/0 0x4011db/0x4011d0/P/-/-/0
diff --git a/llvm/test/tools/llvm-profgen/X86/cs-preinline-inline-attributes.test b/llvm/test/tools/llvm-profgen/X86/cs-preinline-inline-attributes.test
new file mode 100644
index 00000000000000..f28c3ff7a1b2a1
--- /dev/null
+++ b/llvm/test/tools/llvm-profgen/X86/cs-preinline-inline-attributes.test
@@ -0,0 +1,109 @@
+; Test that the CS pre-inliner honors the always_inline and noinline attributes
+; encoded in the pseudo probe descriptors (-preinliner-use-inline-attributes).
+;
+; main inlines ai (always_inline) and ord, and calls ni (noinline). ai is the
+; hottest candidate, then ni, then ord; main is 178 bytes and ai 78 (source at
+; the end).
+;
+; Without the option all three callees are inlined.
+; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/cs-preinline-inline-attributes.perfscript --binary=%S/Inputs/cs-preinline-inline-attributes.perfbin --output=%t --profile-summary-cold-count=0
+; RUN: FileCheck %s --input-file %t --check-prefix=DEFAULT
+
+; DEFAULT: main:4219:0
+; DEFAULT: 5: _ZL2aii.__uniq.{{[0-9]+}}:3996
+; DEFAULT: !Attributes: 3
+; DEFAULT-NEXT: 7: ni:19
+; DEFAULT: !Attributes: 2
+; DEFAULT-NEXT: 10: _ZL3ordi.__uniq.{{[0-9]+}}:6
+; DEFAULT: !Attributes: 3
+
+; With the option ni stays a call and its profile is marked noinline (16), and
+; ai is marked always_inline (8 + inlined 3).
+; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/cs-preinline-inline-attributes.perfscript --binary=%S/Inputs/cs-preinline-inline-attributes.perfbin --output=%t --profile-summary-cold-count=0 --preinliner-use-inline-attributes
+; RUN: FileCheck %s --input-file %t --check-prefix=ATTR
+
+; ATTR: main:4222:0
+; ATTR: 7: 22 ni:22
+; ATTR: 5: _ZL2aii.__uniq.{{[0-9]+}}:3996
+; ATTR: !Attributes: 11
+; ATTR-NEXT: 10: _ZL3ordi.__uniq.{{[0-9]+}}:6
+; ATTR: !Attributes: 3
+; ATTR: ni:22:22
+; ATTR-NEXT: 1: 22
+; ATTR-NEXT: !CFGChecksum:
+; ATTR-NEXT: !Attributes: 16
+
+; With no size budget and a size threshold of 1 byte, ai is still inlined, since
+; the compiler inlines it anyway: neither the exhausted budget nor shouldInline()
+; may stop it. ord is not inlined. Without the option nothing is inlined.
+; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/cs-preinline-inline-attributes.perfscript --binary=%S/Inputs/cs-preinline-inline-attributes.perfbin --output=%t --profile-summary-cold-count=0 --preinliner-use-inline-attributes --sample-profile-inline-limit-min=0 --sample-profile-inline-limit-max=0 --sample-profile-hot-inline-threshold=0 --csspgo-preinliner-multiplier-for-previous-inlining=1
+; RUN: FileCheck %s --input-file %t --check-prefix=NOBUDGET-ATTR
+; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/cs-preinline-inline-attributes.perfscript --binary=%S/Inputs/cs-preinline-inline-attributes.perfbin --output=%t --profile-summary-cold-count=0 --sample-profile-inline-limit-min=0 --sample-profile-inline-limit-max=0 --sample-profile-hot-inline-threshold=0 --csspgo-preinliner-multiplier-for-previous-inlining=1
+; RUN: FileCheck %s --input-file %t --check-prefix=NOBUDGET
+
+; NOBUDGET-ATTR: main:4222:0
+; NOBUDGET-ATTR: 10: 6 _ZL3ordi.__uniq.{{[0-9]+}}:6
+; NOBUDGET-ATTR: 5: _ZL2aii.__uniq.{{[0-9]+}}:3996
+; NOBUDGET-ATTR: !Attributes: 11
+; NOBUDGET-ATTR: _ZL3ordi.__uniq.{{[0-9]+}}:6:6
+
+; NOBUDGET: _ZL2aii.__uniq.{{[0-9]+}}:3996:34
+; NOBUDGET: main:260:0
+; NOBUDGET: 5: 34 _ZL2aii.__uniq.{{[0-9]+}}:34
+; NOBUDGET: 7: 22 ni:22
+; NOBUDGET: 10: 6 _ZL3ordi.__uniq.{{[0-9]+}}:6
+; NOBUDGET-NOT: !Attributes: {{[23]}}{{$}}
+
+; A 250-byte budget against main's 178 bytes: ai is free by default, so ord
+; still fits. -preinliner-charge-always-inline-size charges ai's 78 bytes, which
+; uses the budget up, so ord is no longer inlined.
+; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/cs-preinline-inline-attributes.perfscript --binary=%S/Inputs/cs-preinline-inline-attributes.perfbin --output=%t --profile-summary-cold-count=0 --preinliner-use-inline-attributes --sample-profile-inline-limit-min=250 --sample-profile-inline-limit-max=250
+; RUN: FileCheck %s --input-file %t --check-prefix=FREE
+; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/cs-preinline-inline-attributes.perfscript --binary=%S/Inputs/cs-preinline-inline-attributes.perfbin --output=%t --profile-summary-cold-count=0 --preinliner-use-inline-attributes --sample-profile-inline-limit-min=250 --sample-profile-inline-limit-max=250 --preinliner-charge-always-inline-size
+; RUN: FileCheck %s --input-file %t --check-prefix=CHARGED
+
+; FREE: main:4222:0
+; FREE: 5: _ZL2aii.__uniq.{{[0-9]+}}:3996
+; FREE: 10: _ZL3ordi.__uniq.{{[0-9]+}}:6
+; FREE-NEXT: 1: 6
+
+; CHARGED: main:4222:0
+; CHARGED: 10: 6 _ZL3ordi.__uniq.{{[0-9]+}}:6
+; CHARGED: 5: _ZL2aii.__uniq.{{[0-9]+}}:3996
+; CHARGED: _ZL3ordi.__uniq.{{[0-9]+}}:6:6
+
+; Inputs/cs-preinline-inline-attributes.perfbin and .perfscript were made from
+; the source below (the compiler must encode the inline attributes in the
+; pseudo probe descriptors) with
+; clang -O2 -fno-pie -no-pie -fpseudo-probe-for-profiling -fno-omit-frame-pointer
+; -mno-omit-leaf-frame-pointer -fno-unroll-loops -g test.c
+; perf record -e cycles:u -j any,u -c 100003 --call-graph fp
+; perf script -F ip,brstack --show-mmap-events --no-inline
+; keeping 27 samples and the 16 newest LBR entries of each.
+;
+; volatile int sink;
+;
+; // Hot and tiny, but never inlined.
+; __attribute__((noinline)) int ni(int x) { return x * 3 + 1; }
+;
+; // Always inlined; its loop gives the inlined copy samples and a size.
+; __attribute__((always_inline)) static inline int ai(int n) {
+; int s = 0;
+; for (int i = 0; i < n; i++)
+; s += (i * 7) ^ n;
+; return s;
+; }
+;
+; // An ordinary callee, inlined by the compiler.
+; static int ord(int x) { return (x * x) ^ (x >> 3); }
+;
+; int main() {
+; for (int k = 0; k < 4000000; k++) {
+; sink += ai(32 + (k & 7));
+; if (k & 1)
+; sink += ni(k);
+; if ((k & 3) == 0)
+; sink += ord(k);
+; }
+; return 0;
+; }
diff --git a/llvm/tools/llvm-profgen/CSPreInliner.cpp b/llvm/tools/llvm-profgen/CSPreInliner.cpp
index 87df6996aa435a..9e176821a8e0bb 100644
--- a/llvm/tools/llvm-profgen/CSPreInliner.cpp
+++ b/llvm/tools/llvm-profgen/CSPreInliner.cpp
@@ -31,6 +31,10 @@ STATISTIC(PreInlNumCSInlinedHitMaxLimit,
STATISTIC(
PreInlNumCSInlinedHitGrowthLimit,
"Number of functions with FDO inline stopped due to growth size limit");
+STATISTIC(PreInlNumCSInlinedAlwaysInline,
+ "Number of functions inlined due to always_inline attribute");
+STATISTIC(PreInlNumCSNotInlinedNoInline,
+ "Number of functions not inlined due to noinline attribute");
// The switches specify inline thresholds used in SampleProfileLoader inlining.
// TODO: the actual threshold to be tuned here because the size here is based
@@ -57,6 +61,16 @@ static cl::opt<int> CSPreinlMultiplierForPrevInl(
cl::desc(
"Multiplier to bump up callsite threshold for previous inlining."));
+static cl::opt<bool> CSPreinlUseInlineAttributes(
+ "preinliner-use-inline-attributes", cl::Hidden, cl::init(false),
+ cl::desc("Honor always_inline/noinline attributes from pseudo probe "
+ "descriptors in the CS pre-inliner."));
+
+static cl::opt<bool> ChargeAlwaysInlineSize(
+ "preinliner-charge-always-inline-size", cl::Hidden, cl::init(false),
+ cl::desc("Charge the size of always_inline callees to the caller's "
+ "per-function size budget."));
+
CSPreInliner::CSPreInliner(SampleContextTracker &Tracker,
ProfiledBinary &Binary, ProfileSummary *Summary)
: UseContextCost(UseContextCostForPreInliner),
@@ -159,6 +173,19 @@ bool CSPreInliner::shouldInline(ProfiledInlineCandidate &Candidate) {
if (SamplePreInlineReplay)
return WasInlined;
+ if (Candidate.CalleeSamples->getContext().hasAttribute(ContextNoInline)) {
+ ++PreInlNumCSNotInlinedNoInline;
+ LLVM_DEBUG(dbgs() << " Noinline attribute for: "
+ << Candidate.CalleeSamples->getFunction() << "\n");
+ return false;
+ }
+ if (Candidate.CalleeSamples->getContext().hasAttribute(ContextAlwaysInline)) {
+ ++PreInlNumCSInlinedAlwaysInline;
+ LLVM_DEBUG(dbgs() << " AlwaysInline attribute for: "
+ << Candidate.CalleeSamples->getFunction() << "\n");
+ return true;
+ }
+
unsigned int SampleThreshold = SampleColdCallSiteThreshold;
uint64_t ColdCountThreshold = ProfileSummaryBuilder::getColdCountThreshold(
(Summary->getDetailedSummary()));
@@ -214,7 +241,12 @@ void CSPreInliner::processFunction(const FunctionId Name) {
ProfiledCandidateQueue CQueue;
getInlineCandidates(CQueue, FSamples);
- while (!CQueue.empty() && FuncFinalSize < SizeLimit) {
+ while (!CQueue.empty()) {
+ // Inlines always_inline callees regardless of the budget.
+ if (FuncFinalSize >= SizeLimit &&
+ !CQueue.top().CalleeSamples->getContext().hasAttribute(
+ ContextAlwaysInline))
+ break;
ProfiledInlineCandidate Candidate = CQueue.top();
CQueue.pop();
bool ShouldInline = false;
@@ -225,7 +257,10 @@ void CSPreInliner::processFunction(const FunctionId Name) {
ContextTracker.markContextSamplesInlined(Candidate.CalleeSamples);
Candidate.CalleeSamples->getContext().setAttribute(
ContextShouldBeInlined);
- FuncFinalSize += Candidate.SizeCost;
+ bool IsAlwaysInline = Candidate.CalleeSamples->getContext().hasAttribute(
+ ContextAlwaysInline);
+ if (ChargeAlwaysInlineSize || !IsAlwaysInline)
+ FuncFinalSize += Candidate.SizeCost;
getInlineCandidates(CQueue, Candidate.CalleeSamples);
} else {
++PreInlNumCSNotInlined;
@@ -289,6 +324,27 @@ void CSPreInliner::run() {
LLVM_DEBUG(printProfileNames(ContextTracker, true));
+ // Stamp always_inline/noinline attributes from pseudo probe descriptors
+ // onto the corresponding FunctionSamples' contexts. A binary without
+ // pseudo probes has no descriptors, so nothing is stamped.
+ if (CSPreinlUseInlineAttributes) {
+ auto &FuncProfiles = ContextTracker.getFuncToCtxtProfiles();
+ for (const MCPseudoProbeFuncDesc &FuncDesc : Binary.getGUID2FuncDescMap()) {
+ if (!FuncDesc.Attributes)
+ continue;
+ uint32_t CtxAttr = 0;
+ if (FuncDesc.Attributes &
+ (uint8_t)PseudoProbeDescAttributes::AlwaysInline)
+ CtxAttr |= ContextAlwaysInline;
+ if (FuncDesc.Attributes & (uint8_t)PseudoProbeDescAttributes::NoInline)
+ CtxAttr |= ContextNoInline;
+ FunctionId FuncId(FunctionSamples::getCanonicalFnName(FuncDesc.FuncName));
+ auto It = FuncProfiles.find(FuncId);
+ if (It != FuncProfiles.end())
+ for (auto *FSamples : It->second)
+ FSamples->getContext().setAttribute((ContextAttributeMask)CtxAttr);
+ }
+ }
// Execute global pre-inliner to estimate a global top-down inline
// decision and merge profiles accordingly. This helps with profile
// merge for ThinLTO otherwise we won't be able to merge profiles back
diff --git a/llvm/tools/llvm-profgen/ProfiledBinary.h b/llvm/tools/llvm-profgen/ProfiledBinary.h
index 516e0f7cdf5c1a..df07350fcc72fb 100644
--- a/llvm/tools/llvm-profgen/ProfiledBinary.h
+++ b/llvm/tools/llvm-profgen/ProfiledBinary.h
@@ -700,6 +700,9 @@ class ProfiledBinary {
const MCPseudoProbeFuncDesc *getFuncDescForGUID(uint64_t GUID) {
return ProbeDecoder.getFuncDescForGUID(GUID);
}
+ const GUIDProbeFunctionMap &getGUID2FuncDescMap() const {
+ return ProbeDecoder.getGUID2FuncDescMap();
+ }
const MCPseudoProbeFuncDesc *
getInlinerDescForProbe(const MCDecodedPseudoProbe *Probe) {
More information about the llvm-branch-commits
mailing list