[lld] [lld][MachO] Avoid quadratic iteration over already-folded symbols during ICF (PR #213339)
Peter Rong via llvm-commits
llvm-commits at lists.llvm.org
Sat Aug 1 10:01:27 PDT 2026
https://github.com/DataCorrupted updated https://github.com/llvm/llvm-project/pull/213339
>From bf5193818c70c66bb1e6554c5ac786bcf04f6e11 Mon Sep 17 00:00:00 2001
From: Peter Rong <PeterRong at meta.com>
Date: Fri, 31 Jul 2026 11:49:27 -0700
Subject: [PATCH 1/7] [lld][MachO] Make ICF folding linear in the size of an
equivalence class
ConcatInputSection::foldIdentical() re-walked the survivor's entire `symbols`
vector on every call in order to clear originalUnwindEntry. ICF folds a class
of N members with N-1 calls into the same survivor, so folding a single class
was quadratic in its size.
Everything past index 0 was already cleared by an earlier fold, so only the
tail this call appends needs visiting. Record the pre-insert size and start
there. std::max(numSymbols, 1) keeps the survivor's first symbol intact in the
case where it had no symbols of its own -- which is how applySafeThunksToRange()
calls in, since the thunk it folds into is freshly synthesized by
makeSyntheticInputSection() and carries no symbols.
This is invisible on ordinary inputs, where equivalence classes are small. It
becomes the dominant cost of a link once a single class reaches ~1e5 members,
which happens when instrumentation, global function merging and CGData
cross-module outlining combine to emit the same small relocation-free body into
most translation units. On a large instrumented iOS app that produced one class
with 1,861,243 members out of 64.1M ICF input sections -- roughly 1.7e12
pointer-chasing stores on a single thread, since a class is never split across
forEachClass shards. Replaying that link with and without this change:
re-walk 1:14:42
incremental 3:55
with byte-identical output apart from LC_UUID, which lld varies run to run.
The new test scales the axis icf-scale.s does not: icf-scale.s scales the
*number* of equivalence classes (4 Ki functions in 4 classes of 1 Ki), while
fold cost depends on the size of a single class. icf-scale-same-class.s puts
512 Ki functions in one class and checks that they collapse to a single 6-byte
body with the first and last members of the class sharing an address. Link time
for that test goes from 386.46 s to 0.65 s.
---
lld/MachO/InputSection.cpp | 7 +-
lld/test/MachO/icf-scale-same-class.s | 106 ++++++++++++++++++++++++++
2 files changed, 112 insertions(+), 1 deletion(-)
create mode 100644 lld/test/MachO/icf-scale-same-class.s
diff --git a/lld/MachO/InputSection.cpp b/lld/MachO/InputSection.cpp
index 4c4f644889d5f..015d201c70eb0 100644
--- a/lld/MachO/InputSection.cpp
+++ b/lld/MachO/InputSection.cpp
@@ -204,13 +204,18 @@ void ConcatInputSection::foldIdentical(ConcatInputSection *copy,
for (auto ©Sym : copy->symbols)
copySym->identicalCodeFoldingKind = foldKind;
+ const size_t numSymbols = symbols.size();
symbols.insert(symbols.end(), copy->symbols.begin(), copy->symbols.end());
copy->symbols.clear();
// Remove duplicate compact unwind info for symbols at the same address.
if (symbols.empty())
return;
- for (auto it = symbols.begin() + 1; it != symbols.end(); ++it) {
+ // Only the symbols just appended still need clearing -- everything before
+ // numSymbols was cleared by an earlier fold. std::max() keeps the first
+ // symbol's entry when numSymbols == 0.
+ for (auto it = symbols.begin() + std::max<size_t>(numSymbols, 1);
+ it != symbols.end(); ++it) {
assert((*it)->value == 0);
(*it)->originalUnwindEntry = nullptr;
}
diff --git a/lld/test/MachO/icf-scale-same-class.s b/lld/test/MachO/icf-scale-same-class.s
new file mode 100644
index 0000000000000..e8dd8e1051e6a
--- /dev/null
+++ b/lld/test/MachO/icf-scale-same-class.s
@@ -0,0 +1,106 @@
+# REQUIRES: x86
+# RUN: rm -rf %t*
+
+# RUN: llvm-mc -filetype=obj -triple=x86_64-apple-darwin %s -o %t.o
+# RUN: %lld -lSystem --icf=all -o %t %t.o
+# RUN: llvm-objdump --macho --section-headers %t | FileCheck %s --check-prefix=SECT
+# RUN: llvm-objdump --macho --syms %t | FileCheck %s --check-prefix=SYMS
+
+## Wall clock on single class, --icf=all:
+## N re-walk incremental
+## 64 Ki 3.5 s 0.09 s
+## 128 Ki 14.5 s 0.16 s
+## 256 Ki 66.8 s 0.32 s
+## 512 Ki 408.6 s 0.68 s
+
+## Every body folds into one, so __text holds a single 6-byte body plus _main
+## rather than the 3 MiB (0x30000d) the unfolded copies occupy.
+# SECT: Idx Name Size
+# SECT-NEXT: 0 __text 00000009
+
+## The sentinels bracket the group, so checking that the first and last members
+## of the class share an address covers the whole run of folds.
+# SYMS-DAG: [[#%.16x,F:]] g F __TEXT,__text _f_first
+# SYMS-DAG: [[#F]] g F __TEXT,__text _f_last
+
+.subsections_via_symbols
+.text
+.p2align 2
+
+## Unlike icf-scale.s, every generated body here is identical.
+.macro gen_4 c
+ .globl _f0\c, _f1\c, _f2\c, _f3\c
+ _f0\c:; movl $7, %eax; ret
+ _f1\c:; movl $7, %eax; ret
+ _f2\c:; movl $7, %eax; ret
+ _f3\c:; movl $7, %eax; ret
+.endm
+
+.macro gen_16 c
+ gen_4 0\c
+ gen_4 1\c
+ gen_4 2\c
+ gen_4 3\c
+.endm
+
+.macro gen_64 c
+ gen_16 0\c
+ gen_16 1\c
+ gen_16 2\c
+ gen_16 3\c
+.endm
+
+.macro gen_256 c
+ gen_64 0\c
+ gen_64 1\c
+ gen_64 2\c
+ gen_64 3\c
+.endm
+
+.macro gen_1024 c
+ gen_256 0\c
+ gen_256 1\c
+ gen_256 2\c
+ gen_256 3\c
+.endm
+
+.macro gen_4096 c
+ gen_1024 0\c
+ gen_1024 1\c
+ gen_1024 2\c
+ gen_1024 3\c
+.endm
+
+.macro gen_16384 c
+ gen_4096 0\c
+ gen_4096 1\c
+ gen_4096 2\c
+ gen_4096 3\c
+.endm
+
+.macro gen_65536 c
+ gen_16384 0\c
+ gen_16384 1\c
+ gen_16384 2\c
+ gen_16384 3\c
+.endm
+
+.macro gen_262144 c
+ gen_65536 0\c
+ gen_65536 1\c
+ gen_65536 2\c
+ gen_65536 3\c
+.endm
+
+.globl _f_first
+_f_first:; movl $7, %eax; ret
+
+gen_262144 a
+gen_262144 b
+
+.globl _f_last
+_f_last:; movl $7, %eax; ret
+
+.globl _main
+_main:
+ ret
>From 6fdf158236651cec61270236c6268f854f80e2ec Mon Sep 17 00:00:00 2001
From: Peter Rong <PeterRong at meta.com>
Date: Fri, 31 Jul 2026 15:23:39 -0700
Subject: [PATCH 2/7] address reviewer
---
lld/MachO/InputSection.cpp | 16 +++++-----------
lld/test/MachO/icf-scale-same-class.s | 7 -------
2 files changed, 5 insertions(+), 18 deletions(-)
diff --git a/lld/MachO/InputSection.cpp b/lld/MachO/InputSection.cpp
index 015d201c70eb0..e5545b2653c74 100644
--- a/lld/MachO/InputSection.cpp
+++ b/lld/MachO/InputSection.cpp
@@ -204,21 +204,15 @@ void ConcatInputSection::foldIdentical(ConcatInputSection *copy,
for (auto ©Sym : copy->symbols)
copySym->identicalCodeFoldingKind = foldKind;
- const size_t numSymbols = symbols.size();
- symbols.insert(symbols.end(), copy->symbols.begin(), copy->symbols.end());
- copy->symbols.clear();
-
- // Remove duplicate compact unwind info for symbols at the same address.
- if (symbols.empty())
+ if (copy->symbols.empty())
return;
- // Only the symbols just appended still need clearing -- everything before
- // numSymbols was cleared by an earlier fold. std::max() keeps the first
- // symbol's entry when numSymbols == 0.
- for (auto it = symbols.begin() + std::max<size_t>(numSymbols, 1);
- it != symbols.end(); ++it) {
+ for (auto *it = copy->symbols.begin() + (symbols.empty() ? 1 : 0);
+ it != copy->symbols.end(); ++it) {
assert((*it)->value == 0);
(*it)->originalUnwindEntry = nullptr;
}
+ symbols.insert(symbols.end(), copy->symbols.begin(), copy->symbols.end());
+ copy->symbols.clear();
}
void ConcatInputSection::writeTo(uint8_t *buf) {
diff --git a/lld/test/MachO/icf-scale-same-class.s b/lld/test/MachO/icf-scale-same-class.s
index e8dd8e1051e6a..c0b07b9bb6f86 100644
--- a/lld/test/MachO/icf-scale-same-class.s
+++ b/lld/test/MachO/icf-scale-same-class.s
@@ -6,13 +6,6 @@
# RUN: llvm-objdump --macho --section-headers %t | FileCheck %s --check-prefix=SECT
# RUN: llvm-objdump --macho --syms %t | FileCheck %s --check-prefix=SYMS
-## Wall clock on single class, --icf=all:
-## N re-walk incremental
-## 64 Ki 3.5 s 0.09 s
-## 128 Ki 14.5 s 0.16 s
-## 256 Ki 66.8 s 0.32 s
-## 512 Ki 408.6 s 0.68 s
-
## Every body folds into one, so __text holds a single 6-byte body plus _main
## rather than the 3 MiB (0x30000d) the unfolded copies occupy.
# SECT: Idx Name Size
>From 849567ba6259daee5629f21c0103c11002000534 Mon Sep 17 00:00:00 2001
From: Peter Rong <PeterRong at meta.com>
Date: Fri, 31 Jul 2026 17:25:07 -0700
Subject: [PATCH 3/7] address Daniel
---
lld/MachO/InputSection.cpp | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/lld/MachO/InputSection.cpp b/lld/MachO/InputSection.cpp
index e5545b2653c74..db56cf49ef8da 100644
--- a/lld/MachO/InputSection.cpp
+++ b/lld/MachO/InputSection.cpp
@@ -206,8 +206,12 @@ void ConcatInputSection::foldIdentical(ConcatInputSection *copy,
if (copy->symbols.empty())
return;
- for (auto *it = copy->symbols.begin() + (symbols.empty() ? 1 : 0);
- it != copy->symbols.end(); ++it) {
+ auto *ItBegin = copy->symbols.begin();
+ // symbol.front() needs to keep the entry.
+ // In the case of `symbols.empty()`, we skip the first element.
+ if (symbols.empty())
+ ++ItBegin;
+ for (auto *it = ItBegin; it != copy->symbols.end(); ++it) {
assert((*it)->value == 0);
(*it)->originalUnwindEntry = nullptr;
}
>From 7179bd7a7390263be1beae8dcc50c13e66df8083 Mon Sep 17 00:00:00 2001
From: Peter Rong <PeterRong at meta.com>
Date: Fri, 31 Jul 2026 17:34:30 -0700
Subject: [PATCH 4/7] remove redundent
---
lld/MachO/InputSection.cpp | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/lld/MachO/InputSection.cpp b/lld/MachO/InputSection.cpp
index db56cf49ef8da..82a9ec97fe243 100644
--- a/lld/MachO/InputSection.cpp
+++ b/lld/MachO/InputSection.cpp
@@ -206,12 +206,12 @@ void ConcatInputSection::foldIdentical(ConcatInputSection *copy,
if (copy->symbols.empty())
return;
- auto *ItBegin = copy->symbols.begin();
+ auto *it = copy->symbols.begin();
// symbol.front() needs to keep the entry.
- // In the case of `symbols.empty()`, we skip the first element.
+ // In the case of `symbols.empty()`, skip the first element.
if (symbols.empty())
- ++ItBegin;
- for (auto *it = ItBegin; it != copy->symbols.end(); ++it) {
+ ++it;
+ for (; it != copy->symbols.end(); ++it) {
assert((*it)->value == 0);
(*it)->originalUnwindEntry = nullptr;
}
>From 9e0bf91a9dd24b9bc747290fac7811b0afbe47b9 Mon Sep 17 00:00:00 2001
From: Peter Rong <PeterRong at meta.com>
Date: Fri, 31 Jul 2026 21:30:09 -0700
Subject: [PATCH 5/7] remove macro stack
---
lld/test/MachO/icf-scale-same-class.s | 76 ++++-----------------------
1 file changed, 9 insertions(+), 67 deletions(-)
diff --git a/lld/test/MachO/icf-scale-same-class.s b/lld/test/MachO/icf-scale-same-class.s
index c0b07b9bb6f86..5adea5f2243e2 100644
--- a/lld/test/MachO/icf-scale-same-class.s
+++ b/lld/test/MachO/icf-scale-same-class.s
@@ -20,76 +20,18 @@
.text
.p2align 2
-## Unlike icf-scale.s, every generated body here is identical.
-.macro gen_4 c
- .globl _f0\c, _f1\c, _f2\c, _f3\c
- _f0\c:; movl $7, %eax; ret
- _f1\c:; movl $7, %eax; ret
- _f2\c:; movl $7, %eax; ret
- _f3\c:; movl $7, %eax; ret
-.endm
-
-.macro gen_16 c
- gen_4 0\c
- gen_4 1\c
- gen_4 2\c
- gen_4 3\c
-.endm
-
-.macro gen_64 c
- gen_16 0\c
- gen_16 1\c
- gen_16 2\c
- gen_16 3\c
-.endm
-
-.macro gen_256 c
- gen_64 0\c
- gen_64 1\c
- gen_64 2\c
- gen_64 3\c
-.endm
-
-.macro gen_1024 c
- gen_256 0\c
- gen_256 1\c
- gen_256 2\c
- gen_256 3\c
-.endm
-
-.macro gen_4096 c
- gen_1024 0\c
- gen_1024 1\c
- gen_1024 2\c
- gen_1024 3\c
-.endm
-
-.macro gen_16384 c
- gen_4096 0\c
- gen_4096 1\c
- gen_4096 2\c
- gen_4096 3\c
-.endm
-
-.macro gen_65536 c
- gen_16384 0\c
- gen_16384 1\c
- gen_16384 2\c
- gen_16384 3\c
-.endm
-
-.macro gen_262144 c
- gen_65536 0\c
- gen_65536 1\c
- gen_65536 2\c
- gen_65536 3\c
-.endm
-
.globl _f_first
_f_first:; movl $7, %eax; ret
-gen_262144 a
-gen_262144 b
+## Unlike icf-scale.s, every body generated here is identical, so all 512 Ki
+## land in one equivalence class. \+ iterates from 0 to n-1. $$7 is an escape
+## for a literal $7: .rept expands its body as a macro with no parameters, and
+## in one of those the Darwin assembler reads $7 as positional argument 7 and
+## drops it.
+.rept 524288
+ .globl _f\+
+ _f\+:; movl $$7, %eax; ret
+.endr
.globl _f_last
_f_last:; movl $7, %eax; ret
>From 337ee035271ccd3708ab1b1df2f621473fc5cc24 Mon Sep 17 00:00:00 2001
From: Peter Rong <PeterRong at meta.com>
Date: Fri, 31 Jul 2026 22:49:45 -0700
Subject: [PATCH 6/7] comment
---
lld/MachO/InputSection.cpp | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lld/MachO/InputSection.cpp b/lld/MachO/InputSection.cpp
index 82a9ec97fe243..d977830161a8e 100644
--- a/lld/MachO/InputSection.cpp
+++ b/lld/MachO/InputSection.cpp
@@ -207,8 +207,9 @@ void ConcatInputSection::foldIdentical(ConcatInputSection *copy,
if (copy->symbols.empty())
return;
auto *it = copy->symbols.begin();
- // symbol.front() needs to keep the entry.
- // In the case of `symbols.empty()`, skip the first element.
+ // The first symbol in the merged section (symbols.front()) must keep its
+ // unwind entry. If this section is empty, the copy's first symbol becomes the
+ // new front, so we skip clearing it.
if (symbols.empty())
++it;
for (; it != copy->symbols.end(); ++it) {
>From 30f5b6f069517f4cc2745fb4f23f000f9ec4b617 Mon Sep 17 00:00:00 2001
From: Peter Rong <PeterRong at meta.com>
Date: Sat, 1 Aug 2026 10:01:05 -0700
Subject: [PATCH 7/7] less test
---
lld/test/MachO/icf-scale-same-class.s | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)
diff --git a/lld/test/MachO/icf-scale-same-class.s b/lld/test/MachO/icf-scale-same-class.s
index 5adea5f2243e2..7dc19bae6dd60 100644
--- a/lld/test/MachO/icf-scale-same-class.s
+++ b/lld/test/MachO/icf-scale-same-class.s
@@ -23,12 +23,8 @@
.globl _f_first
_f_first:; movl $7, %eax; ret
-## Unlike icf-scale.s, every body generated here is identical, so all 512 Ki
-## land in one equivalence class. \+ iterates from 0 to n-1. $$7 is an escape
-## for a literal $7: .rept expands its body as a macro with no parameters, and
-## in one of those the Darwin assembler reads $7 as positional argument 7 and
-## drops it.
-.rept 524288
+## 64 Ki identical function landing in one equivalence class.
+.rept 65536
.globl _f\+
_f\+:; movl $$7, %eax; ret
.endr
More information about the llvm-commits
mailing list