[libc-commits] [libc] c3b7d6f - [libc] Keep __restrict on the Cortex-M memcpy for MVE targets (#233575)

[via libc-commits]

Author: Prabhu Rajasekaran
Date: 2026-10-19T23:16:19-07:00
New Revision: c3b7d6f983cc280f17d7d269b812d2fe83539f6a

URL: https://github.com/llvm/llvm-project/commit/c3b7d6f983cc280f17d7d269b812d2fe83539f6a
DIFF: https://github.com/llvm/llvm-project/commit/c3b7d6f983cc280f17d7d269b812d2fe83539f6a.diff

LOG: [libc] Keep __restrict on the Cortex-M memcpy for MVE targets (#233575)

With #226955 the SLP vectorizer versions the Cortex-M memcpy copy loops
with runtime alias checks on MVE targets (270 -> 412 bytes on
Cortex-M55).
Keeping __restrict on MVE targets lets it vectorize without versioning
(222 bytes). Non-MVE targets are unchanged.

Fixes #233426

Added: 
    

Modified: 
    libc/src/string/memory_utils/arm/inline_memcpy.h

Removed: 
    


################################################################################
diff  --git a/libc/src/string/memory_utils/arm/inline_memcpy.h b/libc/src/string/memory_utils/arm/inline_memcpy.h
index 5eddb80d25506..5d226245d12ee 100644
--- a/libc/src/string/memory_utils/arm/inline_memcpy.h
+++ b/libc/src/string/memory_utils/arm/inline_memcpy.h
@@ -139,11 +139,32 @@ copy_bytes_and_bump_pointers(Ptr &dst, CPtr &src, size_t size) {
   copy_bytes_and_bump_pointers(dst, src, size);
 }
 
-// Implementation for Cortex-M3, M4, M7, M23, M33, M35P, M52 with hardware
-// support for unaligned loads and stores. It compiles down to 272 bytes when
-// used through `memcpy` that also needs to return the `dst` ptr.
-[[maybe_unused]] LIBC_INLINE void inline_memcpy_arm_mid_end(Ptr dst, CPtr src,
-                                                            size_t size) {
+// `memcpy` guarantees that `dst` and `src` do not overlap. Whether the
+// implementation gets to know it is a code size trade-off:
+// - With MVE, the compiler then vectorizes the word by word copies with
+//   `vldrb.u8`/`vstrb.8`. Without `__restrict`, the SLP vectorizer versions
+//   the copy loops with runtime alias checks and a scalar fallback instead,
+//   which is about 50% larger.
+// - Without MVE, the compiler would reorder the word by word loads and stores,
+//   which needs more registers and adds push/pop. Hiding `__restrict` keeps
+//   the tuned load/store sequence.
+#ifdef __ARM_FEATURE_MVE
+#define LIBC_ARM_MEMCPY_RESTRICT __restrict
+#else
+#define LIBC_ARM_MEMCPY_RESTRICT
+#endif
+
+// Implementation for Cortex-M3, M4, M7, M23, M33, M35P, M52, M55, M85 with
+// hardware support for unaligned loads and stores. It compiles down to 272
+// bytes (222 bytes with MVE) when used through `memcpy` that also needs to
+// return the `dst` ptr.
+[[maybe_unused]] LIBC_INLINE void
+inline_memcpy_arm_mid_end(Ptr LIBC_ARM_MEMCPY_RESTRICT dst_,
+                          CPtr LIBC_ARM_MEMCPY_RESTRICT src_, size_t size) {
+  // The helpers take the pointers by reference, which a `__restrict` pointer
+  // cannot bind to.
+  Ptr dst = dst_;
+  CPtr src = src_;
   if (misaligned(bitwise_or(src, dst)))
     LIBC_ATTR_UNLIKELY {
       if (size < 8)
@@ -194,14 +215,11 @@ copy_bytes_and_bump_pointers(Ptr &dst, CPtr &src, size_t size) {
 
 [[maybe_unused]] LIBC_INLINE void inline_memcpy_arm(Ptr dst, CPtr src,
                                                     size_t size) {
-  // The compiler performs alias analysis and is able to prove that `dst` and
-  // `src` do not alias by propagating the `__restrict` keyword from the
-  // `memcpy` prototype. This allows the compiler to merge consecutive
-  // load/store (LDR, STR) instructions generated in
-  // `copy_block_and_bump_pointers` with `BlockOp::kByWord` into load/store
-  // double (LDRD, STRD) instructions, this is undesirable so we prevent the
-  // compiler from inferring `__restrict` with the following line.
+#ifndef __ARM_FEATURE_MVE
+  // Hide the `__restrict` of the `memcpy` prototype from the implementation,
+  // see `LIBC_ARM_MEMCPY_RESTRICT`.
   asm volatile("" : "+r"(dst), "+r"(src));
+#endif
 #ifdef __ARM_FEATURE_UNALIGNED
   return inline_memcpy_arm_mid_end(dst, src, size);
 #else