[llvm] fe1fc78 - [SmallVector] Out-of-line the trivially-copyable push_back grow path (#206213)

[via llvm-commits]

Author: Fangrui Song
Date: 2026-06-27T09:43:45-07:00
New Revision: fe1fc78c3e49eea31e4ba7647727680665923dd7

URL: https://github.com/llvm/llvm-project/commit/fe1fc78c3e49eea31e4ba7647727680665923dd7
DIFF: https://github.com/llvm/llvm-project/commit/fe1fc78c3e49eea31e4ba7647727680665923dd7.diff

LOG: [SmallVector] Out-of-line the trivially-copyable push_back grow path (#206213)

In the approximately trivially-copyable specialization, push_back's grow
path does not early return. Both Clang and GCC likely keep `this` and
`Elt` live across the out-of-line `grow_pod` call, saving and restoring
them in the prologue/epilogue. Shrink wrapping can't sink it (the saved
values are used in the store block the fast path also reaches).

Move the grow-and-store into a noinline `growAndPushBack` helper and
tail call it. The fast path needs no callee-saved registers.
`push_back(int)` drops from 14 to 7 instructions on x86-64.

```
// void vec_pb_int(llvm::SmallVectorImpl<int>&v, int x){ v.push_back(x); }

	mov	eax, dword ptr [rdi + 8]
	cmp	eax, dword ptr [rdi + 12]
	jae	_ZN4llvm23SmallVectorTemplateBaseIiLb1EE15growAndPushBackEi # TAILCALL
	mov	rcx, qword ptr [rdi]
	mov	dword ptr [rcx + 4*rax], esi
	inc	dword ptr [rdi + 8]
	ret
```

`noinline` keeps the fast path frame-free and the cold grow path out of
every call site (with a single COMDAT copy). It is load-bearing, as GCC
and Clang otherwise inline it back for some code. However, the
out-of-line call makes the element's address escape, which defeats
construct-in-place for large element types.

`T Tmp = Elt` copy preserves the internal-reference-during-grow case and
is elided for by-value element types.

Added: 
    

Modified: 
    llvm/include/llvm/ADT/SmallVector.h

Removed: 
    


################################################################################
diff  --git a/llvm/include/llvm/ADT/SmallVector.h b/llvm/include/llvm/ADT/SmallVector.h
index ae369c2508113..134c74bfd48d0 100644
--- a/llvm/include/llvm/ADT/SmallVector.h
+++ b/llvm/include/llvm/ADT/SmallVector.h
@@ -565,10 +565,21 @@ class SmallVectorTemplateBase<T, true> : public SmallVectorTemplateCommon<T> {
     return this->back();
   }
 
+  // Out-of-line slow path so the inline push_back needs no callee-saved
+  // registers or stack frame on its hot path.
+  LLVM_ATTRIBUTE_NOINLINE void growAndPushBack(ValueParamT Elt) {
+    // Copy in case Elt is an internal reference invalidated by grow.
+    T Tmp = Elt;
+    this->grow(this->size() + 1);
+    std::memcpy(reinterpret_cast<void *>(this->end()), &Tmp, sizeof(T));
+    this->set_size(this->size() + 1);
+  }
+
 public:
   void push_back(ValueParamT Elt) {
-    const T *EltPtr = reserveForParamAndGetAddress(Elt);
-    std::memcpy(reinterpret_cast<void *>(this->end()), EltPtr, sizeof(T));
+    if (LLVM_UNLIKELY(this->size() >= this->capacity()))
+      return growAndPushBack(Elt);
+    std::memcpy(reinterpret_cast<void *>(this->end()), &Elt, sizeof(T));
     this->set_size(this->size() + 1);
   }