[llvm] [ExpandMemCmp] Only narrow load sizes for targets that opt in (PR #215186)
Pengcheng Wang via llvm-commits
llvm-commits at lists.llvm.org
Mon Aug 10 00:07:07 PDT 2026
https://github.com/wangpc-pp created https://github.com/llvm/llvm-project/pull/215186
Commit e0e64570812f ("Query allowsMisalignedMemoryAccesses instead of a
target flag") replaced the opt-in MemCmpExpansionOptions::RequireNaturalAlignment
knob with an unconditional per-call-site filter that erases every load size
not accessible at the statically-known common pointer alignment. That filter
runs for every target, which regresses strict-alignment targets whose backend
legalizes a wide unaligned load into a branch-free byte sequence.
BPF is the concrete example (see llvm/llvm-project#209738): it reports
LoadSizes = {8, 4, 2, 1}, does not allow misaligned scalar access, uses the
default NumLoadsPerBlock of 1, and has a very large MaxLoadsPerMemcmp. For an
align-1 memcmp the filter drops 8/4/2 and keeps only i8, so the compare is
expanded byte-by-byte into one block per byte. A 32-byte compare becomes 33
nested branches; a real BPF program then exceeds the in-kernel verifier's 1M
instruction limit and is rejected.
The wide load is not the problem: `load i64, align 1` is legal IR, and the
BPF backend lowers it into naturally aligned byte loads with no branches
inside the block, so the machine code performs only naturally aligned
accesses. Narrowing the load sizes in the middle end is what turns a 5-branch
wide-granularity compare into a 33-branch byte-granularity one.
Restore the original opt-in design: reintroduce RequireNaturalAlignment and
run the filter only when the target sets it. RISC-V opts in when it cannot
perform unaligned scalar access, keeping its aligned-expansion / libcall
fallback behavior. Targets that permit unaligned access, or that rely on the
backend to legalize a wide unaligned load, leave it false and keep the wider
loads. The per-load overlapping/tail alignment checks in MemCmpExpansion are
unchanged, so CommonAlign is still computed and passed unconditionally.
Assisted-by: TRAE CLI (Opus 4.8)
>From fb247ec077994e6b69896f3cba043516ae16d98a Mon Sep 17 00:00:00 2001
From: Pengcheng Wang <wangpengcheng.pp at bytedance.com>
Date: Mon, 10 Aug 2026 14:57:39 +0800
Subject: [PATCH] [ExpandMemCmp] Only narrow load sizes for targets that opt in
Commit e0e64570812f ("Query allowsMisalignedMemoryAccesses instead of a
target flag") replaced the opt-in MemCmpExpansionOptions::RequireNaturalAlignment
knob with an unconditional per-call-site filter that erases every load size
not accessible at the statically-known common pointer alignment. That filter
runs for every target, which regresses strict-alignment targets whose backend
legalizes a wide unaligned load into a branch-free byte sequence.
BPF is the concrete example (see llvm/llvm-project#209738): it reports
LoadSizes = {8, 4, 2, 1}, does not allow misaligned scalar access, uses the
default NumLoadsPerBlock of 1, and has a very large MaxLoadsPerMemcmp. For an
align-1 memcmp the filter drops 8/4/2 and keeps only i8, so the compare is
expanded byte-by-byte into one block per byte. A 32-byte compare becomes 33
nested branches; a real BPF program then exceeds the in-kernel verifier's 1M
instruction limit and is rejected.
The wide load is not the problem: `load i64, align 1` is legal IR, and the
BPF backend lowers it into naturally aligned byte loads with no branches
inside the block, so the machine code performs only naturally aligned
accesses. Narrowing the load sizes in the middle end is what turns a 5-branch
wide-granularity compare into a 33-branch byte-granularity one.
Restore the original opt-in design: reintroduce RequireNaturalAlignment and
run the filter only when the target sets it. RISC-V opts in when it cannot
perform unaligned scalar access, keeping its aligned-expansion / libcall
fallback behavior. Targets that permit unaligned access, or that rely on the
backend to legalize a wide unaligned load, leave it false and keep the wider
loads. The per-load overlapping/tail alignment checks in MemCmpExpansion are
unchanged, so CommonAlign is still computed and passed unconditionally.
Assisted-by: TRAE CLI (Opus 4.8)
---
.../llvm/Analysis/TargetTransformInfo.h | 10 ++
.../Target/RISCV/RISCVTargetTransformInfo.cpp | 7 ++
llvm/lib/Transforms/Scalar/ExpandMemCmp.cpp | 26 +++--
.../Transforms/ExpandMemCmp/BPF/lit.local.cfg | 2 +
.../Transforms/ExpandMemCmp/BPF/memcmp.ll | 101 ++++++++++++++++++
5 files changed, 137 insertions(+), 9 deletions(-)
create mode 100644 llvm/test/Transforms/ExpandMemCmp/BPF/lit.local.cfg
create mode 100644 llvm/test/Transforms/ExpandMemCmp/BPF/memcmp.ll
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfo.h b/llvm/include/llvm/Analysis/TargetTransformInfo.h
index 107ae4dba5075..8ddda74c1fc20 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfo.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfo.h
@@ -1135,6 +1135,16 @@ class TargetTransformInfo {
// requires all loads in LoadSizes to be doable in an unaligned way.
bool AllowOverlappingLoads = false;
+ // Set to true if the expansion may only emit naturally aligned loads.
+ // Strict-alignment targets that cannot perform unaligned scalar loads opt
+ // into this so MemCmpExpansion keeps only the load sizes covered by the
+ // statically-known common alignment of both pointers at the call site (or
+ // that the target still reports accessible via
+ // allowsMisalignedMemoryAccesses), falling back to the libcall when no load
+ // size fits. Targets that allow unaligned access leave this false, keeping
+ // the wider loads their backend legalizes into a branch-free sequence.
+ bool RequireNaturalAlignment = false;
+
// Sometimes, the amount of data that needs to be compared is smaller than
// the standard register size, but it cannot be loaded with just one load
// instruction. For example, if the size of the memory comparison is 6
diff --git a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
index 9868e4d02f905..5d50f5a14fdad 100644
--- a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
+++ b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
@@ -3678,6 +3678,13 @@ RISCVTTIImpl::enableMemCmpExpansion(bool OptSize, bool IsZeroCmp) const {
if (!ST->hasStdExtZbb() && !ST->hasStdExtZbkb() && !IsZeroCmp)
return Options;
+ // When the target cannot perform unaligned scalar accesses, opt into the
+ // natural-alignment filter in ExpandMemCmp: it keeps only the load sizes
+ // covered by the statically-known common pointer alignment (and falls back to
+ // the libcall when none fits), so expansion still happens for sufficiently
+ // aligned pointers without emitting unaligned loads.
+ Options.RequireNaturalAlignment = !ST->enableUnalignedScalarMem();
+
Options.AllowOverlappingLoads = true;
Options.MaxNumLoads = TLI->getMaxExpandSizeMemcmp(OptSize);
Options.NumLoadsPerBlock = Options.MaxNumLoads;
diff --git a/llvm/lib/Transforms/Scalar/ExpandMemCmp.cpp b/llvm/lib/Transforms/Scalar/ExpandMemCmp.cpp
index 62e9c82ea7756..397d0bc03b605 100644
--- a/llvm/lib/Transforms/Scalar/ExpandMemCmp.cpp
+++ b/llvm/lib/Transforms/Scalar/ExpandMemCmp.cpp
@@ -933,17 +933,25 @@ static bool expandMemCmp(CallInst *CI, const TargetTransformInfo *TTI,
// kept here is always accessible in that sequence; overlapping loads and
// merged tail expansions are checked separately against their actual offsets
// in MemCmpExpansion.
+ //
+ // Only targets that opt in via RequireNaturalAlignment are narrowed. Targets
+ // that permit unaligned access (or, like BPF, legalize a wide unaligned load
+ // into a branch-free byte sequence in the backend) keep the wider load sizes:
+ // narrowing them there would replace a single wide compare with a long chain
+ // of per-byte compare blocks.
const Align CommonAlign = std::min(getMemCmpArgAlignment(CI, 0, *DL),
getMemCmpArgAlignment(CI, 1, *DL));
- llvm::erase_if(Options.LoadSizes, [&](unsigned LoadSize) {
- return !isAccessAllowed(CI, *TTI, CommonAlign, LoadSize, /*Offset=*/0);
- });
- // If the filter removed every load size, bail out to the libcall: the
- // MemCmpExpansion constructor asserts that at least one load size remains.
- // In practice all in-tree targets include a byte load size, which is
- // accessible at any alignment and therefore always survives the filter.
- if (Options.LoadSizes.empty())
- return false;
+ if (Options.RequireNaturalAlignment) {
+ llvm::erase_if(Options.LoadSizes, [&](unsigned LoadSize) {
+ return !isAccessAllowed(CI, *TTI, CommonAlign, LoadSize, /*Offset=*/0);
+ });
+ // If the filter removed every load size, bail out to the libcall: the
+ // MemCmpExpansion constructor asserts that at least one load size remains.
+ // In practice all in-tree targets include a byte load size, which is
+ // accessible at any alignment and therefore always survives the filter.
+ if (Options.LoadSizes.empty())
+ return false;
+ }
MemCmpExpansion Expansion(CI, SizeVal, Options, IsUsedForZeroCmp, *DL, DTU,
*TTI, CommonAlign);
diff --git a/llvm/test/Transforms/ExpandMemCmp/BPF/lit.local.cfg b/llvm/test/Transforms/ExpandMemCmp/BPF/lit.local.cfg
new file mode 100644
index 0000000000000..0fe841da7d0e7
--- /dev/null
+++ b/llvm/test/Transforms/ExpandMemCmp/BPF/lit.local.cfg
@@ -0,0 +1,2 @@
+if not "BPF" in config.root.targets:
+ config.unsupported = True
diff --git a/llvm/test/Transforms/ExpandMemCmp/BPF/memcmp.ll b/llvm/test/Transforms/ExpandMemCmp/BPF/memcmp.ll
new file mode 100644
index 0000000000000..59a44df9d85e1
--- /dev/null
+++ b/llvm/test/Transforms/ExpandMemCmp/BPF/memcmp.ll
@@ -0,0 +1,101 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 5
+; RUN: opt -S -passes=expand-memcmp -mtriple=bpf < %s | FileCheck %s
+
+; BPF does not allow unaligned scalar accesses, but its backend legalizes a wide
+; unaligned load into a branch-free byte sequence. ExpandMemCmp must therefore
+; keep the wide load sizes here instead of narrowing them to i8: narrowing would
+; emit one compare block per byte, which explodes the number of branches (a BPF
+; program with too many nested branches fails the in-kernel verifier). See
+; https://github.com/llvm/llvm-project/pull/209738.
+
+target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n32:64-S128"
+
+declare i32 @memcmp(ptr, ptr, i64)
+declare i32 @bcmp(ptr, ptr, i64)
+
+; A 32-byte equality compare against an unaligned pointer must still be expanded
+; with i64 loads (4 compare blocks), not 32 single-byte compare blocks.
+define i1 @memcmp_size_32_align_1(ptr %a, ptr %b) {
+; CHECK-LABEL: define i1 @memcmp_size_32_align_1(
+; CHECK-SAME: ptr [[A:%.*]], ptr [[B:%.*]]) {
+; CHECK-NEXT: br label %[[LOADBB4:.*]]
+; CHECK: [[LOADBB:.*]]:
+; CHECK-NEXT: br label %[[ENDBLOCK:.*]]
+; CHECK: [[LOADBB4]]:
+; CHECK-NEXT: [[TMP1:%.*]] = load i64, ptr [[A]], align 1
+; CHECK-NEXT: [[TMP2:%.*]] = load i64, ptr [[B]], align 1
+; CHECK-NEXT: [[TMP3:%.*]] = icmp ne i64 [[TMP1]], [[TMP2]]
+; CHECK-NEXT: br i1 [[TMP3]], label %[[LOADBB]], label %[[LOADBB1:.*]]
+; CHECK: [[LOADBB1]]:
+; CHECK-NEXT: [[TMP4:%.*]] = getelementptr i8, ptr [[A]], i64 8
+; CHECK-NEXT: [[TMP5:%.*]] = getelementptr i8, ptr [[B]], i64 8
+; CHECK-NEXT: [[TMP6:%.*]] = load i64, ptr [[TMP4]], align 1
+; CHECK-NEXT: [[TMP7:%.*]] = load i64, ptr [[TMP5]], align 1
+; CHECK-NEXT: [[TMP8:%.*]] = icmp ne i64 [[TMP6]], [[TMP7]]
+; CHECK-NEXT: br i1 [[TMP8]], label %[[LOADBB]], label %[[LOADBB2:.*]]
+; CHECK: [[LOADBB2]]:
+; CHECK-NEXT: [[TMP9:%.*]] = getelementptr i8, ptr [[A]], i64 16
+; CHECK-NEXT: [[TMP10:%.*]] = getelementptr i8, ptr [[B]], i64 16
+; CHECK-NEXT: [[TMP11:%.*]] = load i64, ptr [[TMP9]], align 1
+; CHECK-NEXT: [[TMP12:%.*]] = load i64, ptr [[TMP10]], align 1
+; CHECK-NEXT: [[TMP13:%.*]] = icmp ne i64 [[TMP11]], [[TMP12]]
+; CHECK-NEXT: br i1 [[TMP13]], label %[[LOADBB]], label %[[LOADBB3:.*]]
+; CHECK: [[LOADBB3]]:
+; CHECK-NEXT: [[TMP14:%.*]] = getelementptr i8, ptr [[A]], i64 24
+; CHECK-NEXT: [[TMP15:%.*]] = getelementptr i8, ptr [[B]], i64 24
+; CHECK-NEXT: [[TMP16:%.*]] = load i64, ptr [[TMP14]], align 1
+; CHECK-NEXT: [[TMP17:%.*]] = load i64, ptr [[TMP15]], align 1
+; CHECK-NEXT: [[TMP18:%.*]] = icmp ne i64 [[TMP16]], [[TMP17]]
+; CHECK-NEXT: br i1 [[TMP18]], label %[[LOADBB]], label %[[ENDBLOCK]]
+; CHECK: [[ENDBLOCK]]:
+; CHECK-NEXT: [[PHI_RES:%.*]] = phi i32 [ 0, %[[LOADBB3]] ], [ 1, %[[LOADBB]] ]
+; CHECK-NEXT: [[C:%.*]] = icmp eq i32 [[PHI_RES]], 0
+; CHECK-NEXT: ret i1 [[C]]
+;
+ %call = call i32 @memcmp(ptr %a, ptr %b, i64 32)
+ %c = icmp eq i32 %call, 0
+ ret i1 %c
+}
+
+; Same for bcmp.
+define i1 @bcmp_size_32_align_1(ptr %a, ptr %b) {
+; CHECK-LABEL: define i1 @bcmp_size_32_align_1(
+; CHECK-SAME: ptr [[A:%.*]], ptr [[B:%.*]]) {
+; CHECK-NEXT: br label %[[LOADBB4:.*]]
+; CHECK: [[LOADBB:.*]]:
+; CHECK-NEXT: br label %[[ENDBLOCK:.*]]
+; CHECK: [[LOADBB4]]:
+; CHECK-NEXT: [[TMP1:%.*]] = load i64, ptr [[A]], align 1
+; CHECK-NEXT: [[TMP2:%.*]] = load i64, ptr [[B]], align 1
+; CHECK-NEXT: [[TMP3:%.*]] = icmp ne i64 [[TMP1]], [[TMP2]]
+; CHECK-NEXT: br i1 [[TMP3]], label %[[LOADBB]], label %[[LOADBB1:.*]]
+; CHECK: [[LOADBB1]]:
+; CHECK-NEXT: [[TMP4:%.*]] = getelementptr i8, ptr [[A]], i64 8
+; CHECK-NEXT: [[TMP5:%.*]] = getelementptr i8, ptr [[B]], i64 8
+; CHECK-NEXT: [[TMP6:%.*]] = load i64, ptr [[TMP4]], align 1
+; CHECK-NEXT: [[TMP7:%.*]] = load i64, ptr [[TMP5]], align 1
+; CHECK-NEXT: [[TMP8:%.*]] = icmp ne i64 [[TMP6]], [[TMP7]]
+; CHECK-NEXT: br i1 [[TMP8]], label %[[LOADBB]], label %[[LOADBB2:.*]]
+; CHECK: [[LOADBB2]]:
+; CHECK-NEXT: [[TMP9:%.*]] = getelementptr i8, ptr [[A]], i64 16
+; CHECK-NEXT: [[TMP10:%.*]] = getelementptr i8, ptr [[B]], i64 16
+; CHECK-NEXT: [[TMP11:%.*]] = load i64, ptr [[TMP9]], align 1
+; CHECK-NEXT: [[TMP12:%.*]] = load i64, ptr [[TMP10]], align 1
+; CHECK-NEXT: [[TMP13:%.*]] = icmp ne i64 [[TMP11]], [[TMP12]]
+; CHECK-NEXT: br i1 [[TMP13]], label %[[LOADBB]], label %[[LOADBB3:.*]]
+; CHECK: [[LOADBB3]]:
+; CHECK-NEXT: [[TMP14:%.*]] = getelementptr i8, ptr [[A]], i64 24
+; CHECK-NEXT: [[TMP15:%.*]] = getelementptr i8, ptr [[B]], i64 24
+; CHECK-NEXT: [[TMP16:%.*]] = load i64, ptr [[TMP14]], align 1
+; CHECK-NEXT: [[TMP17:%.*]] = load i64, ptr [[TMP15]], align 1
+; CHECK-NEXT: [[TMP18:%.*]] = icmp ne i64 [[TMP16]], [[TMP17]]
+; CHECK-NEXT: br i1 [[TMP18]], label %[[LOADBB]], label %[[ENDBLOCK]]
+; CHECK: [[ENDBLOCK]]:
+; CHECK-NEXT: [[PHI_RES:%.*]] = phi i32 [ 0, %[[LOADBB3]] ], [ 1, %[[LOADBB]] ]
+; CHECK-NEXT: [[C:%.*]] = icmp eq i32 [[PHI_RES]], 0
+; CHECK-NEXT: ret i1 [[C]]
+;
+ %call = call i32 @bcmp(ptr %a, ptr %b, i64 32)
+ %c = icmp eq i32 %call, 0
+ ret i1 %c
+}
More information about the llvm-commits
mailing list