[llvm] cd46c2c - Tweak BumpPtrAllocator to benefit the hot path (#90571)

[via llvm-commits]

Author: Owen Anderson
Date: 2024-05-01T01:19:46-06:00
New Revision: cd46c2c1ba0481e2194231f0f2c2ceeb0810bb79

URL: https://github.com/llvm/llvm-project/commit/cd46c2c1ba0481e2194231f0f2c2ceeb0810bb79
DIFF: https://github.com/llvm/llvm-project/commit/cd46c2c1ba0481e2194231f0f2c2ceeb0810bb79.diff

LOG: Tweak BumpPtrAllocator to benefit the hot path (#90571)

This takes the form of three consecutive but related changes:
- Mark the fast path of BumpPtrAllocator as likely-taken.
- Move the slow path of BumpPtrAllocator to a separate function.
- Mark the slow path of BumpPtrAllocator as noinline.

Overall, this saves geomean 0.4% userspace instructions on CTMark -O3,
and 0.98% on CTMark -O0 -g.


http://llvm-compile-time-tracker.com/compare.php?from=e1622e189e8c0ef457bfac528f90a7a930d9aad2&to=9eb53a4ed3af4a55e769ae1dd22d034b63d046e3&stat=instructions%3Au

Added: 
    

Modified: 
    llvm/include/llvm/Support/Allocator.h

Removed: 
    


################################################################################
diff  --git a/llvm/include/llvm/Support/Allocator.h b/llvm/include/llvm/Support/Allocator.h
index c1e5c6d2853bd5..eb6c4d36685500 100644
--- a/llvm/include/llvm/Support/Allocator.h
+++ b/llvm/include/llvm/Support/Allocator.h
@@ -159,9 +159,9 @@ class BumpPtrAllocatorImpl
 #endif
 
     // Check if we have enough space.
-    if (Adjustment + SizeToAllocate <= size_t(End - CurPtr)
-        // We can't return nullptr even for a zero-sized allocation!
-        && CurPtr != nullptr) {
+    if (LLVM_LIKELY(Adjustment + SizeToAllocate <= size_t(End - CurPtr)
+                    // We can't return nullptr even for a zero-sized allocation!
+                    && CurPtr != nullptr)) {
       char *AlignedPtr = CurPtr + Adjustment;
       CurPtr = AlignedPtr + SizeToAllocate;
       // Update the allocation point of this memory block in MemorySanitizer.
@@ -173,6 +173,11 @@ class BumpPtrAllocatorImpl
       return AlignedPtr;
     }
 
+    return AllocateSlow(Size, SizeToAllocate, Alignment);
+  }
+
+  LLVM_ATTRIBUTE_RETURNS_NONNULL LLVM_ATTRIBUTE_NOINLINE void *
+  AllocateSlow(size_t Size, size_t SizeToAllocate, Align Alignment) {
     // If Size is really big, allocate a separate slab for it.
     size_t PaddedSize = SizeToAllocate + Alignment.value() - 1;
     if (PaddedSize > SizeThreshold) {