[llvm] [ADT] Out-of-line the trivially-copyable SmallVector::push_back grow path (PR #206213)

[Fangrui Song via llvm-commits]

https://github.com/MaskRay created https://github.com/llvm/llvm-project/pull/206213

In the approximately trivially-copyable specialization, push_back's grow
path does not early return. Both Clang and GCC likely keep `this` and
`Elt` live across the out-of-line `grow_pod` call, forcing a stack frame
onto the hot path.

Move the grow-and-store into a noinline `growAndPushBack` helper and
early-return to it. The fast path no longer needs stack frame.

`noinline` is required: otherwise both Clang and GCC may inline the
helper and the frame returns.

`T Tmp = Elt` copy preserves the internal-reference-during-grow case and
is elided for by-value element types.


>From aa61e3a1d9d3b1499ca7a5cd396a4c009f5a08c8 Mon Sep 17 00:00:00 2001
From: Fangrui Song <i at maskray.me>
Date: Fri, 26 Jun 2026 10:30:16 -0700
Subject: [PATCH] [ADT] Out-of-line the trivially-copyable
 SmallVector::push_back grow path

In the approximately trivially-copyable specialization, push_back's grow
path does not early return. Both Clang and GCC likely keep `this` and
`Elt` live across the out-of-line `grow_pod` call, forcing a stack frame
onto the hot path.

Move the grow-and-store into a noinline `growAndPushBack` helper and
early-return to it. The fast path no longer needs stack frame.

`noinline` is required: otherwise both Clang and GCC may inline the
helper and the frame returns.

`T Tmp = Elt` copy preserves the internal-reference-during-grow case and
is elided for by-value element types.
---
 llvm/include/llvm/ADT/SmallVector.h | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/llvm/include/llvm/ADT/SmallVector.h b/llvm/include/llvm/ADT/SmallVector.h
index ae369c2508113..134c74bfd48d0 100644
--- a/llvm/include/llvm/ADT/SmallVector.h
+++ b/llvm/include/llvm/ADT/SmallVector.h
@@ -565,10 +565,21 @@ class SmallVectorTemplateBase<T, true> : public SmallVectorTemplateCommon<T> {
     return this->back();
   }
 
+  // Out-of-line slow path so the inline push_back needs no callee-saved
+  // registers or stack frame on its hot path.
+  LLVM_ATTRIBUTE_NOINLINE void growAndPushBack(ValueParamT Elt) {
+    // Copy in case Elt is an internal reference invalidated by grow.
+    T Tmp = Elt;
+    this->grow(this->size() + 1);
+    std::memcpy(reinterpret_cast<void *>(this->end()), &Tmp, sizeof(T));
+    this->set_size(this->size() + 1);
+  }
+
 public:
   void push_back(ValueParamT Elt) {
-    const T *EltPtr = reserveForParamAndGetAddress(Elt);
-    std::memcpy(reinterpret_cast<void *>(this->end()), EltPtr, sizeof(T));
+    if (LLVM_UNLIKELY(this->size() >= this->capacity()))
+      return growAndPushBack(Elt);
+    std::memcpy(reinterpret_cast<void *>(this->end()), &Elt, sizeof(T));
     this->set_size(this->size() + 1);
   }