[libc-commits] [libc] [libc][baremetal][BigInt] fix codegen issue for UInt128 (PR #220389)
Schrodinger ZHU Yifan via libc-commits
libc-commits at lists.llvm.org
Thu Sep 3 08:57:51 PDT 2026
https://github.com/SchrodingerZhu updated https://github.com/llvm/llvm-project/pull/220389
>From 6c326745beb90514c62389e19c3df69840e41264 Mon Sep 17 00:00:00 2001
From: Schrodinger ZHU Yifan <i at zhuyi.fan>
Date: Tue, 1 Sep 2026 17:11:52 -0400
Subject: [PATCH 1/2] [libc][baremetal][BigInt] fix codegen issue for UInt128
BigInt::operator* always computes the full 2N-word product (ful_mul,
i.e. multiword::multiply_with_carry) and then truncates. For the
double-word case (UInt<128> with 64-bit words, WORD_COUNT == 2) the
Accumulator/DoubleWide temporaries are cpp::array objects that SROA
fails to scalarize on targets without a native __int128, so the whole
multiply round-trips through the stack and the carry bits are reduced
through in-memory arrays. Add a straight-line truncating specialization
of operator* for the double-word case built from multiword::mul2 plus
two truncating word multiplies; wider WORD_COUNTs keep the existing
path.
On 64-bit targets the codegen is roughly okay either way (aarch64 is
byte-identical before/after; x86-64 gets one mov shorter). After the
patch, 128x128 -> 128 (lo/hi in memory):
aarch64: x86-64:
ldp x8, x12, [x0] movq (%rdi), %r9
ldp x9, x10, [x1] movq (%rsi), %r8
mul x10, x10, x8 movq %r8, %rax
mul x11, x9, x8 mulq %r9
umulh x8, x9, x8 imulq 8(%rsi), %r9
madd x9, x9, x12, x10 imulq 8(%rdi), %r8
add x8, x9, x8 addq %r9, %r8
stp x11, x8, [x2] addq %rdx, %r8
ret ...
On 32-bit targets without native __int128 such as cortex-m85 the old
codegen is bloated in a pretty poor form: a 64-byte stack frame, carry
selection through IT blocks, and a call into an outlined ful_mul
(folded multiply kernel: lo ^ hi of the UInt<128> product):
before: after:
sub sp, #64 umull r12, lr, r2, r0
mov.w r12, #0 umull r2, r4, r2, r1
strd r12, r12, [sp, #24] umull r0, r5, r3, r0
orrs.w lr, r0, r1 umaal r4, r5, r3, r1
it ne adds.w r1, lr, r2
strdne r0, r1, [sp, #16] adcs r2, r4, #0
... (5 more stores) adc r3, r5, #0
bl BigInt<128>::ful_mul adds r0, r0, r1
ldm r3, {r0, r1, r2, r3} adcs r2, r2, #0
eors r1, r3 adc r1, r3, #0
eors r0, r2 eors r1, r0
add sp, #64 eor.w r0, r2, r12
llvm-mca (cortex-m85): 114 -> 11 cycles per folded multiply (10.4x).
Measured on hardware (EK-RA8M2, Cortex-M85 r1p1, DWT cycle counter,
I-cache on): 250 -> 20 cycles per multiply (12.5x), identical results.
wasm32 under wasmer 7.3, with __int128 disabled (-U__SIZEOF_INT128__)
so BigInt exercises the same multiword path as such targets: 100M
multiplies in 0.70 s -> 0.28 s (2.5x). Bench kernel (dependent chain):
using B = LIBC_NAMESPACE::BigInt<128, false, uint64_t>;
B x = (B(a) << 64) | B(~a);
B y = (B(b) << 64) | B(~b);
for (int i = 0; i < n; i++) {
x = x * y;
y.val[0] ^= x.val[1];
}
return x.val[0] ^ x.val[1];
Correctness: 2,000,000 random 128x128 truncating multiplies match
native __int128 on the host, boundary operands included; constexpr
evaluation still works through the new path.
Meanwhile, fix a typo in the add_with_carry / sub_with_borrow builtin
dispatch: the #if/#elif chain over __has_builtin makes the preprocessor
drop every branch except the first one on clang (where __builtin_addcb
always exists), so types wider than unsigned char never reach their
intrinsic. This was semantically wrong, although it barely affects
codegen: at -O2 clang already folds the generic fallback into the same
adc/adcs carry chains (measured identical on x86-64, aarch64 and
cortex-m85).
Notes:
1. Another workaround for such targets is to force-enable native
128-bit integers (clang's -fforce-enable-int128): mul2 then takes
its builtin path and reaches roughly the same codegen (~10 cycles
per folded multiply by llvm-mca on cortex-m85). This patch gets
there by construction, without requiring a non-default compiler
flag, and composes with that option.
2. The wasm number above is benched with 128-bit integers forcibly
disabled (-U__SIZEOF_INT128__). Stock wasm32 clang does provide
__int128, in which case UInt128 aliases the builtin type and mul2
already takes the native path, making old and new codegen identical
there. Disabling it makes BigInt exercise the generic multiword
path this patch fixes, matching 32-bit targets without __int128.
Co-Authored-By: Claude Fable 5 <noreply at anthropic.com>
Claude-Session: https://claude.ai/code/session_017rFwCVRr81nSMxUYgvR92m
---
libc/src/__support/big_int.h | 15 +++++++++++++--
libc/src/__support/math_extras.h | 24 ++++++++++++++++--------
2 files changed, 29 insertions(+), 10 deletions(-)
diff --git a/libc/src/__support/big_int.h b/libc/src/__support/big_int.h
index 5d2675909f2fb..8802fdcb5ab21 100644
--- a/libc/src/__support/big_int.h
+++ b/libc/src/__support/big_int.h
@@ -617,8 +617,19 @@ struct BigInt {
}
LIBC_INLINE constexpr BigInt operator*(const BigInt &other) const {
- // Perform full mul and truncate.
- return BigInt(ful_mul(other));
+ if constexpr (!Signed && WORD_COUNT == 2) {
+ // Straight-line truncating multiply for the double-word case.
+ BigInt result{};
+ multiword::DoubleWide<word_type> lo_prod =
+ multiword::mul2(val[0], other.val[0]);
+ result.val[0] = multiword::lo(lo_prod);
+ result.val[1] = multiword::hi(lo_prod) + val[0] * other.val[1] +
+ val[1] * other.val[0];
+ return result;
+ } else {
+ // Perform full mul and truncate.
+ return BigInt(ful_mul(other));
+ }
}
// Fast hi part of the full product. The normal product `operator*` returns
diff --git a/libc/src/__support/math_extras.h b/libc/src/__support/math_extras.h
index b8abf8041c8e7..8de5c50d0f12a 100644
--- a/libc/src/__support/math_extras.h
+++ b/libc/src/__support/math_extras.h
@@ -108,13 +108,17 @@ add_with_carry(T a, T b, T carry_in, T &carry_out) {
if (!cpp::is_constant_evaluated()) {
#if __has_builtin(__builtin_addcb)
RETURN_IF(unsigned char, __builtin_addcb)
-#elif __has_builtin(__builtin_addcs)
+#endif
+#if __has_builtin(__builtin_addcs)
RETURN_IF(unsigned short, __builtin_addcs)
-#elif __has_builtin(__builtin_addc)
+#endif
+#if __has_builtin(__builtin_addc)
RETURN_IF(unsigned int, __builtin_addc)
-#elif __has_builtin(__builtin_addcl)
+#endif
+#if __has_builtin(__builtin_addcl)
RETURN_IF(unsigned long, __builtin_addcl)
-#elif __has_builtin(__builtin_addcll)
+#endif
+#if __has_builtin(__builtin_addcll)
RETURN_IF(unsigned long long, __builtin_addcll)
#endif
}
@@ -134,13 +138,17 @@ sub_with_borrow(T a, T b, T carry_in, T &carry_out) {
if (!cpp::is_constant_evaluated()) {
#if __has_builtin(__builtin_subcb)
RETURN_IF(unsigned char, __builtin_subcb)
-#elif __has_builtin(__builtin_subcs)
+#endif
+#if __has_builtin(__builtin_subcs)
RETURN_IF(unsigned short, __builtin_subcs)
-#elif __has_builtin(__builtin_subc)
+#endif
+#if __has_builtin(__builtin_subc)
RETURN_IF(unsigned int, __builtin_subc)
-#elif __has_builtin(__builtin_subcl)
+#endif
+#if __has_builtin(__builtin_subcl)
RETURN_IF(unsigned long, __builtin_subcl)
-#elif __has_builtin(__builtin_subcll)
+#endif
+#if __has_builtin(__builtin_subcll)
RETURN_IF(unsigned long long, __builtin_subcll)
#endif
}
>From 09c1aafa1afdc812af02bc135b89c2aca17e4782 Mon Sep 17 00:00:00 2001
From: Schrodinger ZHU Yifan <i at zhuyi.fan>
Date: Thu, 3 Sep 2026 11:57:38 -0400
Subject: [PATCH 2/2] [libc][BigInt] unroll small multiword multiplication
instead of specializing operator*
---
libc/src/__support/big_int.h | 29 ++++++++++++++---------------
1 file changed, 14 insertions(+), 15 deletions(-)
diff --git a/libc/src/__support/big_int.h b/libc/src/__support/big_int.h
index 8802fdcb5ab21..118f36de8f089 100644
--- a/libc/src/__support/big_int.h
+++ b/libc/src/__support/big_int.h
@@ -17,7 +17,7 @@
#include "src/__support/CPP/type_traits.h"
#include "src/__support/macros/attributes.h" // LIBC_INLINE
#include "src/__support/macros/config.h"
-#include "src/__support/macros/optimization.h" // LIBC_UNLIKELY
+#include "src/__support/macros/optimization.h" // LIBC_UNLIKELY, LIBC_LOOP_UNROLL
#include "src/__support/macros/properties/compiler.h" // LIBC_COMPILER_IS_CLANG
#include "src/__support/macros/properties/types.h" // LIBC_TYPES_HAS_INT128, LIBC_TYPES_HAS_INT64
#include "src/__support/math_extras.h" // add_with_carry, sub_with_borrow
@@ -214,13 +214,23 @@ LIBC_INLINE constexpr word multiply_with_carry(cpp::array<word, O> &dst,
const cpp::array<word, N> &rhs) {
static_assert(O >= M + N);
Accumulator<word> acc;
- for (size_t i = 0; i < O; ++i) {
+ auto step = [&](size_t i) {
const size_t lower_idx = i < N ? 0 : i - N + 1;
const size_t upper_idx = i < M ? i : M - 1;
word carry = 0;
for (size_t j = lower_idx; j <= upper_idx; ++j)
carry += mul_add_with_carry(acc, lhs[j], rhs[i - j]);
dst[i] = acc.advance(carry);
+ };
+ // Unrolling up to 4 words is enough for UInt128, the most used large integer
+ // type; larger integers may bloat the code too much.
+ if constexpr (O <= 4) {
+ LIBC_LOOP_UNROLL
+ for (size_t i = 0; i < O; ++i)
+ step(i);
+ } else {
+ for (size_t i = 0; i < O; ++i)
+ step(i);
}
return acc.carry();
}
@@ -617,19 +627,8 @@ struct BigInt {
}
LIBC_INLINE constexpr BigInt operator*(const BigInt &other) const {
- if constexpr (!Signed && WORD_COUNT == 2) {
- // Straight-line truncating multiply for the double-word case.
- BigInt result{};
- multiword::DoubleWide<word_type> lo_prod =
- multiword::mul2(val[0], other.val[0]);
- result.val[0] = multiword::lo(lo_prod);
- result.val[1] = multiword::hi(lo_prod) + val[0] * other.val[1] +
- val[1] * other.val[0];
- return result;
- } else {
- // Perform full mul and truncate.
- return BigInt(ful_mul(other));
- }
+ // Perform full mul and truncate.
+ return BigInt(ful_mul(other));
}
// Fast hi part of the full product. The normal product `operator*` returns
More information about the libc-commits
mailing list