[libc-commits] [libc] [libc][string] Optimize inline_memmove routine for Aarch64 (PR #232494)

[Paul Kirth via libc-commits]

https://github.com/ilovepi created https://github.com/llvm/llvm-project/pull/232494

When switching to llvm-libc, Fuchsia developers noticed a performance regression from their previous implementation of memmove. The existing implementation made several assumptions that fail when not using floating point hardware, e.g. through -mgeneral-regs-only.

This patch implements the same pattern used on x86_64 to make better choices when dispatching on smaller sizes. It also avoids a mandatory stack realignment when not using the vector registers, a mandatory check of the stack canary for the large stack adjustment, and a number of register spills.

Now, under -mgeneral-regs-only, memmove becomes a leaf function and shrinks from 332 to 173 instructions. For Cortex-A73 / Cortex-A53, llvm-mca estimated cycles per call drop by 37%–50% for 8-byte disjoint copies, 60%–79% for 8–64 byte overlapping copies, and 29%–40% for 96–256 byte overlapping copies. With NEON enabled, deduplicating the small-size dispatch reduces instruction count by ~35% and avoids the is_disjoint check for copies <= 128 bytes.

Assisted-by: Gemini

>From 778055fd02e8f8c2a07abcf96dc1026e49064b41 Mon Sep 17 00:00:00 2001
From: Paul Kirth <paulkirth at google.com>
Date: Wed, 14 Oct 2026 16:07:25 -0700
Subject: [PATCH] [libc][string] Optimize inline_memmove routines
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

When switching to llvm-libc, Fuchisa developers noticed a performance
regression from their previous implementation of memmove. The existing
implementation made several assumptions that fail when not using
floating point hardware, e.g. through -mgeneral-regs-only.

This patch implements the same pattern used on x86_64 to make better
choices when dispatching on smaller sizes. It also avoids a mandatory
stack realignment when not using the vector registers, a mandatory
check of the stack canary for the large stack adjustment, and a number
of register spills.

Under -mgeneral-regs-only, memmove becoems a leaf function and shrinks
from 332 to 173 instructions. In llvm-mca (Cortex-A73 / Cortex-A53),
estimated cycles per call drop by 37%–50% for 8-byte disjoint copies,
60%–79% for 8–64 byte overlapping copies, and 29%–40% for 96–256 byte
overlapping copies. With NEON enabled, deduplicating the small-size
dispatch reduces static instruction count by ~35% and avoids the
is_disjoint check for copies <= 128 bytes.

Assisted-by: Gemini
---
 .../memory_utils/aarch64/inline_memmove.h     | 71 +++++++++++++------
 libc/src/string/memory_utils/inline_memmove.h |  5 +-
 2 files changed, 54 insertions(+), 22 deletions(-)

diff --git a/libc/src/string/memory_utils/aarch64/inline_memmove.h b/libc/src/string/memory_utils/aarch64/inline_memmove.h
index 6f5f53029985a7..3ec74f1f8870e3 100644
--- a/libc/src/string/memory_utils/aarch64/inline_memmove.h
+++ b/libc/src/string/memory_utils/aarch64/inline_memmove.h
@@ -17,33 +17,64 @@
 
 namespace LIBC_NAMESPACE_DECL {
 
+#if defined(__ARM_NEON)
 using uint128_t = generic_v128;
 using uint256_t = generic_v256;
 using uint512_t = generic_v512;
+using AlignT = uint256_t;
+using LoopT = uint512_t;
+#else
+using uint128_t = cpp::array<uint64_t, 2>;
+using uint256_t = cpp::array<uint64_t, 4>;
+using AlignT = uint128_t;
+using LoopT = uint256_t;
+#endif
 
-LIBC_INLINE void inline_memmove_aarch64(Ptr dst, CPtr src, size_t count) {
+LIBC_INLINE bool inline_memmove_small_size_aarch64(Ptr dst, CPtr src,
+                                                   size_t count) {
   if (count == 0)
-    return;
-  if (count == 1)
-    return generic::Memmove<uint8_t>::block(dst, src);
-  if (count <= 4)
-    return generic::Memmove<uint16_t>::head_tail(dst, src, count);
-  if (count <= 8)
-    return generic::Memmove<uint32_t>::head_tail(dst, src, count);
-  if (count <= 16)
-    return generic::Memmove<uint64_t>::head_tail(dst, src, count);
-  if (count <= 32)
-    return generic::Memmove<uint128_t>::head_tail(dst, src, count);
-  if (count <= 64)
-    return generic::Memmove<uint256_t>::head_tail(dst, src, count);
-  if (count <= 128)
-    return generic::Memmove<uint512_t>::head_tail(dst, src, count);
+    return true;
+  if (count == 1) {
+    generic::Memmove<uint8_t>::block(dst, src);
+    return true;
+  }
+  if (count <= 4) {
+    generic::Memmove<uint16_t>::head_tail(dst, src, count);
+    return true;
+  }
+  if (count <= 8) {
+    generic::Memmove<uint32_t>::head_tail(dst, src, count);
+    return true;
+  }
+  if (count <= 16) {
+    generic::Memmove<uint64_t>::head_tail(dst, src, count);
+    return true;
+  }
+  if (count <= 32) {
+    generic::Memmove<uint128_t>::head_tail(dst, src, count);
+    return true;
+  }
+  if (count <= 64) {
+    generic::Memmove<uint256_t>::head_tail(dst, src, count);
+    return true;
+  }
+#if defined(__ARM_NEON)
+  if (count <= 128) {
+    generic::Memmove<uint512_t>::head_tail(dst, src, count);
+    return true;
+  }
+#endif
+  return false;
+}
+
+LIBC_INLINE void inline_memmove_follow_up_aarch64(Ptr dst, CPtr src,
+                                                  size_t count) {
   if (dst < src) {
-    generic::Memmove<uint256_t>::align_forward<Arg::Src>(dst, src, count);
-    return generic::Memmove<uint512_t>::loop_and_tail_forward(dst, src, count);
+    generic::Memmove<AlignT>::align_forward<Arg::Src>(dst, src, count);
+    return generic::Memmove<LoopT>::loop_and_tail_forward(dst, src, count);
   } else {
-    generic::Memmove<uint256_t>::align_backward<Arg::Src>(dst, src, count);
-    return generic::Memmove<uint512_t>::loop_and_tail_backward(dst, src, count);
+    generic::Memmove<AlignT>::align_backward<Arg::Src>(dst, src, count);
+    return generic::Memmove<LoopT>::loop_and_tail_backward(dst, src, count);
   }
 }
 
diff --git a/libc/src/string/memory_utils/inline_memmove.h b/libc/src/string/memory_utils/inline_memmove.h
index c75100b48cc95b..d34013a3ce8b1b 100644
--- a/libc/src/string/memory_utils/inline_memmove.h
+++ b/libc/src/string/memory_utils/inline_memmove.h
@@ -27,8 +27,9 @@
 #elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
 #include "src/string/memory_utils/aarch64/inline_memmove.h"
 #define LIBC_SRC_STRING_MEMORY_UTILS_MEMMOVE_SMALL_SIZE                        \
-  inline_memmove_no_small_size
-#define LIBC_SRC_STRING_MEMORY_UTILS_MEMMOVE_FOLLOW_UP inline_memmove_aarch64
+  inline_memmove_small_size_aarch64
+#define LIBC_SRC_STRING_MEMORY_UTILS_MEMMOVE_FOLLOW_UP                         \
+  inline_memmove_follow_up_aarch64
 #elif defined(LIBC_TARGET_ARCH_IS_ANY_RISCV)
 #include "src/string/memory_utils/riscv/inline_memmove.h"
 #define LIBC_SRC_STRING_MEMORY_UTILS_MEMMOVE_SMALL_SIZE                        \