[llvm] [SampleProfileMatcher] Fix direct basename matching for suffixed function names (PR #227184)
Farid Zakaria via llvm-commits
llvm-commits at lists.llvm.org
Mon Sep 28 20:46:34 PDT 2026
https://github.com/fzakaria created https://github.com/llvm/llvm-project/pull/227184
Direct basename matching silently skips any function whose name carries a suffix, for two reasons:
1. `getDemangledBaseName` demangles the raw name. For names such as `_ZL3fool.__uniq.123`, `_Z3fool.llvm.7`, `.part.N` or `.cfi` the Itanium demangler's root node is a DotSuffix, for which `getFunctionBaseName()` returns null, so the function is never a candidate on either the IR or the profile side.
2. `UpdateWithSalvagedProfiles` keys `FuncNameToProfNameMap` by the raw IR name, but `SampleProfileReader::getSamplesFor(const Function &)` looks it up by `getCanonicalFnName`. A salvaged `.llvm.N` (ThinLTO-promoted), `.part.N` or `.cfi` function therefore still gets no profile.
### __uniq.N
Canonicalize the name before demangling (additionally dropping `.__uniq.N`, which `getCanonicalFnName` keeps when the profile has uniq names), and key the map by the canonical name. Other suffixes such as coroutine `.resume` are kept so those clones do not make a basename ambiguous.
The `.__uniq.N` case matters most in practice. The suffix hashes the source path (`-funique-internal-linkage-names`), so moving or renaming a file renames every internal-linkage function in it, and these are exactly the functions basename matching is meant to recover.
### Performance Analysis
In CSSPGO profiles from two large production services at Meta, `.__uniq.` functions are 17.4% and 8.6% of top-level profiles but carry 39.9% and 29.2% of the samples (i.e. internal functions are few, but they are small hot helpers), so their sample density is 2.3x-3.4x that of the average function. Today all of them are excluded from basename matching which this PR fixes.
Assisted-by: Claude
>From 15faa0abc25e192a6488e7a6f8e2194a7c236550 Mon Sep 17 00:00:00 2001
From: Farid Zakaria <fmzakari at fb.com>
Date: Mon, 28 Sep 2026 20:39:12 -0700
Subject: [PATCH] [SampleProfileMatcher] Fix direct basename matching for
suffixed function names
Direct basename matching (#184409) silently skips any function whose name
carries a suffix, for two reasons:
1. `getDemangledBaseName` demangles the raw name. For names such as
`_ZL3fool.__uniq.123`, `_Z3fool.llvm.7`, `.part.N` or `.cfi` the Itanium
demangler's root node is a DotSuffix, for which `getFunctionBaseName()`
returns null, so the function is never a candidate on either the IR or
the profile side.
2. `UpdateWithSalvagedProfiles` keys `FuncNameToProfNameMap` by the raw IR
name, but `SampleProfileReader::getSamplesFor(const Function &)` looks it
up by `getCanonicalFnName`. A salvaged `.llvm.N` (ThinLTO-promoted),
`.part.N` or `.cfi` function therefore still gets no profile.
Canonicalize the name before demangling (additionally dropping `.__uniq.N`,
which `getCanonicalFnName` keeps when the profile has uniq names), and key
the map by the canonical name. Other suffixes such as coroutine `.resume`
are kept so those clones do not make a basename ambiguous.
The `.__uniq.N` case matters most in practice. The suffix hashes the source
path (`-funique-internal-linkage-names`), so moving or renaming a file
renames every internal-linkage function in it, and these are exactly the
functions basename matching is meant to recover. In CSSPGO profiles from two
large production services, `.__uniq.` functions are 17.4% and 8.6% of
top-level profiles but carry 39.9% and 29.2% of the samples: internal
functions are few, but they are small hot helpers, so their sample density
is 2.3x-3.4x that of the average function. Today all of them are excluded
from basename matching.
Assisted-by: Claude
---
.../Transforms/IPO/SampleProfileMatcher.cpp | 14 ++-
...-stale-profile-direct-basename-suffix.prof | 12 +++
...be-stale-profile-direct-basename-uniq.prof | 6 ++
...be-stale-profile-direct-basename-suffix.ll | 101 ++++++++++++++++++
...robe-stale-profile-direct-basename-uniq.ll | 68 ++++++++++++
5 files changed, 199 insertions(+), 2 deletions(-)
create mode 100644 llvm/test/Transforms/SampleProfile/Inputs/pseudo-probe-stale-profile-direct-basename-suffix.prof
create mode 100644 llvm/test/Transforms/SampleProfile/Inputs/pseudo-probe-stale-profile-direct-basename-uniq.prof
create mode 100644 llvm/test/Transforms/SampleProfile/pseudo-probe-stale-profile-direct-basename-suffix.ll
create mode 100644 llvm/test/Transforms/SampleProfile/pseudo-probe-stale-profile-direct-basename-uniq.ll
diff --git a/llvm/lib/Transforms/IPO/SampleProfileMatcher.cpp b/llvm/lib/Transforms/IPO/SampleProfileMatcher.cpp
index 2a6c9ee61efd4..3b44b84b051b3 100644
--- a/llvm/lib/Transforms/IPO/SampleProfileMatcher.cpp
+++ b/llvm/lib/Transforms/IPO/SampleProfileMatcher.cpp
@@ -836,7 +836,14 @@ void SampleProfileMatcher::findFunctionsWithoutProfile() {
// templates, and parameter types). Returns an empty string on failure.
static std::string getDemangledBaseName(ItaniumPartialDemangler &Demangler,
StringRef FName) {
- auto FunctionName = FName.str();
+ // The demangler parses a clone suffix such as ".llvm.N" as a DotSuffix root
+ // node, for which getFunctionBaseName() returns null. Strip the known ones,
+ // including ".__uniq.N" which getCanonicalFnName keeps when the profile has
+ // it. Other suffixes (e.g. coroutine ".resume") are kept so such clones do
+ // not collide with the original function's basename.
+ StringRef Canon = FunctionSamples::getCanonicalFnName(FName);
+ auto FunctionName =
+ Canon.take_front(Canon.find(FunctionSamples::UniqSuffix)).str();
if (Demangler.partialDemangle(FunctionName.c_str()))
return std::string();
size_t BaseNameSize = 0;
@@ -1080,7 +1087,10 @@ void SampleProfileMatcher::UpdateWithSalvagedProfiles() {
assert(I.first && "New function is null");
FunctionId FuncName(I.first->getName());
ProfileSalvagedFuncs.insert(I.second.stringRef());
- FuncNameToProfNameMap->emplace(FuncName, I.second);
+ // SampleProfileReader::getSamplesFor(const Function &) looks this map up by
+ // the canonical name.
+ FuncNameToProfNameMap->emplace(
+ FunctionId(FunctionSamples::getCanonicalFnName(*I.first)), I.second);
// We need to remove the old entry to avoid duplicating the function
// processing.
diff --git a/llvm/test/Transforms/SampleProfile/Inputs/pseudo-probe-stale-profile-direct-basename-suffix.prof b/llvm/test/Transforms/SampleProfile/Inputs/pseudo-probe-stale-profile-direct-basename-suffix.prof
new file mode 100644
index 0000000000000..ab85378efa28f
--- /dev/null
+++ b/llvm/test/Transforms/SampleProfile/Inputs/pseudo-probe-stale-profile-direct-basename-suffix.prof
@@ -0,0 +1,12 @@
+_Z3fooi:52:52
+ 1: 52
+ !CFGChecksum: 4294967295
+_Z3bari:53:53
+ 1: 53
+ !CFGChecksum: 4294967295
+_Z3bazi:54:54
+ 1: 54
+ !CFGChecksum: 4294967295
+_Z3quxi:55:55
+ 1: 55
+ !CFGChecksum: 4294967295
diff --git a/llvm/test/Transforms/SampleProfile/Inputs/pseudo-probe-stale-profile-direct-basename-uniq.prof b/llvm/test/Transforms/SampleProfile/Inputs/pseudo-probe-stale-profile-direct-basename-uniq.prof
new file mode 100644
index 0000000000000..bca39b8d69859
--- /dev/null
+++ b/llvm/test/Transforms/SampleProfile/Inputs/pseudo-probe-stale-profile-direct-basename-uniq.prof
@@ -0,0 +1,6 @@
+_ZL3bazl.__uniq.111:54:54
+ 1: 54
+ !CFGChecksum: 4294967295
+_ZL3fool.__uniq.111:52:52
+ 1: 52
+ !CFGChecksum: 4294967295
diff --git a/llvm/test/Transforms/SampleProfile/pseudo-probe-stale-profile-direct-basename-suffix.ll b/llvm/test/Transforms/SampleProfile/pseudo-probe-stale-profile-direct-basename-suffix.ll
new file mode 100644
index 0000000000000..a1e59f8582b37
--- /dev/null
+++ b/llvm/test/Transforms/SampleProfile/pseudo-probe-stale-profile-direct-basename-suffix.ll
@@ -0,0 +1,101 @@
+; Test that direct basename matching handles IR names carrying clone suffixes
+; (e.g. ".llvm.N" added by ThinLTO promotion), which the Itanium demangler
+; does not treat as function encodings. The pseudo probe GUIDs are those of
+; the canonical names, as in a ThinLTO backend. The coroutine clone
+; _Z3fool.resume must not make the basename "foo" ambiguous.
+
+; REQUIRES: x86_64-linux
+; RUN: llvm-profdata merge --sample --extbinary %S/Inputs/pseudo-probe-stale-profile-direct-basename-suffix.prof -o %t.prof
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%t.prof --salvage-stale-profile --salvage-unused-profile -S | FileCheck %s
+
+; CHECK: define dso_local void @_Z3fool.llvm.7(i64 %y) {{.*}} !prof ![[#FOO:]]
+; CHECK: define dso_local void @_Z3barl.part.0(i64 %y) {{.*}} !prof ![[#BAR:]]
+; CHECK: define dso_local void @_Z3bazl.__uniq.123(i64 %y) {{.*}} !prof ![[#BAZ:]]
+; CHECK: define dso_local void @_Z3quxl.cfi(i64 %y) {{.*}} !prof ![[#QUX:]]
+; CHECK-DAG: ![[#FOO]] = !{!"function_entry_count", i64 52}
+; CHECK-DAG: ![[#BAR]] = !{!"function_entry_count", i64 53}
+; CHECK-DAG: ![[#BAZ]] = !{!"function_entry_count", i64 54}
+; CHECK-DAG: ![[#QUX]] = !{!"function_entry_count", i64 55}
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define dso_local void @_Z3fool.llvm.7(i64 %y) #0 !dbg !9 {
+entry:
+ call void @llvm.pseudoprobe(i64 5326982120444056491, i64 1, i32 0, i64 -1), !dbg !12
+ ret void, !dbg !12
+}
+
+define dso_local void @_Z3barl.part.0(i64 %y) #0 !dbg !13 {
+entry:
+ call void @llvm.pseudoprobe(i64 -9164787269840974918, i64 1, i32 0, i64 -1), !dbg !14
+ ret void, !dbg !14
+}
+
+define dso_local void @_Z3bazl.__uniq.123(i64 %y) #0 !dbg !15 {
+entry:
+ call void @llvm.pseudoprobe(i64 -2504982094396869733, i64 1, i32 0, i64 -1), !dbg !16
+ ret void, !dbg !16
+}
+
+define dso_local void @_Z3quxl.cfi(i64 %y) #0 !dbg !17 {
+entry:
+ call void @llvm.pseudoprobe(i64 2523411590769414898, i64 1, i32 0, i64 -1), !dbg !18
+ ret void, !dbg !18
+}
+
+define dso_local void @_Z3fool.resume(i64 %y) #0 {
+entry:
+ ret void
+}
+
+define dso_local void @caller() #0 !dbg !19 {
+entry:
+ call void @llvm.pseudoprobe(i64 -1768971689307247648, i64 1, i32 0, i64 -1), !dbg !20
+ call void @_Z3fool.llvm.7(i64 0), !dbg !21
+ call void @_Z3barl.part.0(i64 0), !dbg !23
+ call void @_Z3bazl.__uniq.123(i64 0), !dbg !25
+ call void @_Z3quxl.cfi(i64 0), !dbg !27
+ ret void, !dbg !29
+}
+
+; Function Attrs: nocallback nofree nosync nounwind willreturn memory(inaccessiblemem: readwrite)
+declare void @llvm.pseudoprobe(i64 immarg, i64 immarg, i32 immarg, i64 immarg) #1
+
+attributes #0 = { "use-sample-profile" }
+attributes #1 = { nocallback nofree nosync nounwind willreturn memory(inaccessiblemem: readwrite) }
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!2, !3}
+!llvm.pseudo_probe_desc = !{!4, !5, !6, !7, !8}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !1, isOptimized: false, runtimeVersion: 0, emissionKind: NoDebug, nameTableKind: None)
+!1 = !DIFile(filename: "test.cpp", directory: "/tmp")
+!2 = !{i32 2, !"Debug Info Version", i32 3}
+!3 = !{i32 7, !"uwtable", i32 2}
+!4 = !{i64 5326982120444056491, i64 4294967295, !"_Z3fool"}
+!5 = !{i64 -9164787269840974918, i64 4294967295, !"_Z3barl"}
+!6 = !{i64 -2504982094396869733, i64 4294967295, !"_Z3bazl"}
+!7 = !{i64 2523411590769414898, i64 4294967295, !"_Z3quxl"}
+!8 = !{i64 -1768971689307247648, i64 1125904201809919, !"caller"}
+!9 = distinct !DISubprogram(name: "foo", linkageName: "_Z3fool", scope: !1, file: !1, line: 3, type: !10, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !0)
+!10 = !DISubroutineType(types: !11)
+!11 = !{}
+!12 = !DILocation(line: 4, column: 1, scope: !9)
+!13 = distinct !DISubprogram(name: "bar", linkageName: "_Z3barl", scope: !1, file: !1, line: 13, type: !10, scopeLine: 13, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !0)
+!14 = !DILocation(line: 14, column: 1, scope: !13)
+!15 = distinct !DISubprogram(name: "baz", linkageName: "_Z3bazl", scope: !1, file: !1, line: 23, type: !10, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !0)
+!16 = !DILocation(line: 24, column: 1, scope: !15)
+!17 = distinct !DISubprogram(name: "qux", linkageName: "_Z3quxl", scope: !1, file: !1, line: 33, type: !10, scopeLine: 33, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !0)
+!18 = !DILocation(line: 34, column: 1, scope: !17)
+!19 = distinct !DISubprogram(name: "caller", linkageName: "caller", scope: !1, file: !1, line: 7, type: !10, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !0)
+!20 = !DILocation(line: 9, column: 1, scope: !19)
+!21 = !DILocation(line: 9, column: 1, scope: !22)
+!22 = !DILexicalBlockFile(scope: !19, file: !1, discriminator: 455082007)
+!23 = !DILocation(line: 9, column: 1, scope: !24)
+!24 = !DILexicalBlockFile(scope: !19, file: !1, discriminator: 455082015)
+!25 = !DILocation(line: 9, column: 1, scope: !26)
+!26 = !DILexicalBlockFile(scope: !19, file: !1, discriminator: 455082023)
+!27 = !DILocation(line: 9, column: 1, scope: !28)
+!28 = !DILexicalBlockFile(scope: !19, file: !1, discriminator: 455082031)
+!29 = !DILocation(line: 10, column: 1, scope: !19)
diff --git a/llvm/test/Transforms/SampleProfile/pseudo-probe-stale-profile-direct-basename-uniq.ll b/llvm/test/Transforms/SampleProfile/pseudo-probe-stale-profile-direct-basename-uniq.ll
new file mode 100644
index 0000000000000..16c17c58cb6ed
--- /dev/null
+++ b/llvm/test/Transforms/SampleProfile/pseudo-probe-stale-profile-direct-basename-uniq.ll
@@ -0,0 +1,68 @@
+; Test direct basename matching against a profile whose names carry
+; ".__uniq.N" suffixes (-funique-internal-linkage-names). The suffix hashes the
+; source path, so moving a file renames every internal function in it; the
+; profile below was collected before such a move. Because the profile has
+; uniq names, getCanonicalFnName keeps ".__uniq.N", and _ZL3fool.__uniq.222.llvm.7
+; is looked up as _ZL3fool.__uniq.222.
+
+; REQUIRES: x86_64-linux
+; RUN: llvm-profdata merge --sample --extbinary %S/Inputs/pseudo-probe-stale-profile-direct-basename-uniq.prof -o %t.prof
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%t.prof --salvage-stale-profile --salvage-unused-profile -S | FileCheck %s
+
+; CHECK: define internal void @_ZL3bazl.__uniq.222(i64 %y) {{.*}} !prof ![[#BAZ:]]
+; CHECK: define void @_ZL3fool.__uniq.222.llvm.7(i64 %y) {{.*}} !prof ![[#FOO:]]
+; CHECK-DAG: ![[#BAZ]] = !{!"function_entry_count", i64 54}
+; CHECK-DAG: ![[#FOO]] = !{!"function_entry_count", i64 52}
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define internal void @_ZL3bazl.__uniq.222(i64 %y) #0 !dbg !7 {
+entry:
+ call void @llvm.pseudoprobe(i64 -8017183693835869934, i64 1, i32 0, i64 -1), !dbg !10
+ ret void, !dbg !10
+}
+
+define void @_ZL3fool.__uniq.222.llvm.7(i64 %y) #0 !dbg !11 {
+entry:
+ call void @llvm.pseudoprobe(i64 7381965757683757353, i64 1, i32 0, i64 -1), !dbg !12
+ ret void, !dbg !12
+}
+
+define dso_local void @caller() #0 !dbg !13 {
+entry:
+ call void @llvm.pseudoprobe(i64 -1768971689307247648, i64 1, i32 0, i64 -1), !dbg !14
+ call void @_ZL3bazl.__uniq.222(i64 0), !dbg !15
+ call void @_ZL3fool.__uniq.222.llvm.7(i64 0), !dbg !17
+ ret void, !dbg !19
+}
+
+declare void @llvm.pseudoprobe(i64 immarg, i64 immarg, i32 immarg, i64 immarg) #1
+
+attributes #0 = { "use-sample-profile" }
+attributes #1 = { nocallback nofree nosync nounwind willreturn memory(inaccessiblemem: readwrite) }
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!2, !3}
+!llvm.pseudo_probe_desc = !{!4, !5, !6}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !1, isOptimized: false, runtimeVersion: 0, emissionKind: NoDebug, nameTableKind: None)
+!1 = !DIFile(filename: "test.cpp", directory: "/tmp")
+!2 = !{i32 2, !"Debug Info Version", i32 3}
+!3 = !{i32 7, !"uwtable", i32 2}
+!4 = !{i64 -8017183693835869934, i64 4294967295, !"_ZL3bazl.__uniq.222"}
+!5 = !{i64 7381965757683757353, i64 4294967295, !"_ZL3fool.__uniq.222"}
+!6 = !{i64 -1768971689307247648, i64 562954248388607, !"caller"}
+!7 = distinct !DISubprogram(name: "baz", linkageName: "_ZL3bazl.__uniq.222", scope: !1, file: !1, line: 23, type: !8, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !0)
+!8 = !DISubroutineType(types: !9)
+!9 = !{}
+!10 = !DILocation(line: 24, column: 1, scope: !7)
+!11 = distinct !DISubprogram(name: "foo", linkageName: "_ZL3fool.__uniq.222", scope: !1, file: !1, line: 3, type: !8, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !0)
+!12 = !DILocation(line: 4, column: 1, scope: !11)
+!13 = distinct !DISubprogram(name: "caller", linkageName: "caller", scope: !1, file: !1, line: 7, type: !8, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !0)
+!14 = !DILocation(line: 9, column: 1, scope: !13)
+!15 = !DILocation(line: 9, column: 1, scope: !16)
+!16 = !DILexicalBlockFile(scope: !13, file: !1, discriminator: 455082007)
+!17 = !DILocation(line: 10, column: 1, scope: !18)
+!18 = !DILexicalBlockFile(scope: !13, file: !1, discriminator: 455082015)
+!19 = !DILocation(line: 11, column: 1, scope: !13)
More information about the llvm-commits
mailing list