[lld] [lld-macho] Fix branch extension thunk estimation logic (PR #120529)

[via llvm-commits]

llvmbot wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-lld

Author: None (alx32)

<details>
<summary>Changes</summary>

This patch improves the linker’s ability to estimate stub reachability in the `TextOutputSection::estimateStubsInRangeVA` function. It does so by including thunks that have already been placed ahead of the current call site address when calculating the threshold for direct stub calls.

Before this fix, the estimation process overlooked existing forward thunks. This could result in some thunks not being inserted where needed. In rare situations, particularly with large and specially arranged codebases, this might lead to branch instructions being out of range, causing linking errors.

Although this patch successfully addresses the problem, it is not feasible to create a test for this issue. The specific layout and order of thunk creation required to reproduce the corner case are too complex, making test creation impractical.

---
Full diff: https://github.com/llvm/llvm-project/pull/120529.diff


1 Files Affected:

- (modified) lld/MachO/ConcatOutputSection.cpp (+24-2) 


``````````diff
diff --git a/lld/MachO/ConcatOutputSection.cpp b/lld/MachO/ConcatOutputSection.cpp
index 6a9301f84a03ef..9610a80e309bac 100644
--- a/lld/MachO/ConcatOutputSection.cpp
+++ b/lld/MachO/ConcatOutputSection.cpp
@@ -184,6 +184,15 @@ uint64_t TextOutputSection::estimateStubsInRangeVA(size_t callIdx) const {
     InputSection *isec = inputs[i];
     isecEnd = alignToPowerOf2(isecEnd, isec->align) + isec->getSize();
   }
+
+  // Tally up any thunks that have already been placed that have address higher
+  // than the equivalent callIdx. We first find the index of the first thunk
+  // that is beyond the current inputs[callIdx].
+  auto itPostcallIdxThunks = std::partition_point(
+      thunks.begin(), thunks.end(),
+      [isecVA](const ConcatInputSection *t) { return t->getVA() <= isecVA; });
+  uint32_t existingForwardThunks = thunks.end() - itPostcallIdxThunks;
+
   // Estimate the address after which call sites can safely call stubs
   // directly rather than through intermediary thunks.
   uint64_t forwardBranchRange = target->forwardBranchRange;
@@ -191,8 +200,21 @@ uint64_t TextOutputSection::estimateStubsInRangeVA(size_t callIdx) const {
          "should not run thunk insertion if all code fits in jump range");
   assert(isecEnd - isecVA <= forwardBranchRange &&
          "should only finalize sections in jump range");
-  uint64_t stubsInRangeVA = isecEnd + maxPotentialThunks * target->thunkSize +
-                            in.stubs->getSize() - forwardBranchRange;
+
+  // Estimate the maximum size of the code, right before the stubs section
+  uint32_t maxTextSize = 0;
+  // Add the size of all the inputs, including the unprocessed ones.
+  maxTextSize += isecEnd;
+
+  // Add the size of the thunks that may be created in the future
+  maxTextSize += maxPotentialThunks * target->thunkSize;
+
+  // Add the size of the thunks that have already been created that are ahead
+  maxTextSize += existingForwardThunks * target->thunkSize;
+
+  uint64_t stubsInRangeVA =
+      maxTextSize + in.stubs->getSize() - forwardBranchRange;
+
   log("thunks = " + std::to_string(thunkMap.size()) +
       ", potential = " + std::to_string(maxPotentialThunks) +
       ", stubs = " + std::to_string(in.stubs->getSize()) + ", isecVA = " +

``````````

</details>


https://github.com/llvm/llvm-project/pull/120529