[llvm] [X86] Reuse already-materialized values when forming LEAs (alternative) (PR #210739)
Nikita Taranov via llvm-commits
llvm-commits at lists.llvm.org
Tue Jul 21 03:15:50 PDT 2026
https://github.com/nickitat updated https://github.com/llvm/llvm-project/pull/210739
>From 7bcbe8677981ff3b9c12eec150b1dbd6e0d98b25 Mon Sep 17 00:00:00 2001
From: Nikita Taranov <nikita.taranov at clickhouse.com>
Date: Mon, 20 Jul 2026 21:18:27 +0700
Subject: [PATCH 1/4] [X86] Reuse an already-materialized value instead of
splitting it into an LEA
When selecting the address for an LEA, matchAdd would look through a multi-use
operand to sink one of its parts into the address (e.g. a constant into the
displacement). When that operand is already materialized in a register this
just recomputes it. For example, for
add1 = (a + b) + 17 ; stored, multi-use
add2 = add1 + b
the LEA for add2 was matched as base=(a+b), index=b, disp=17, rematerializing
a+b, rather than reusing add1: base=add1, index=b.
Avoid decomposing a multi-use operand when forming an LEA (a memory operand
folds the arithmetic for free, so this only applies to LEAs) and only in the
ADD-parent context (matchAdd), so the SUB matching path is left untouched and
does not start forming slow negated-index 3-source LEAs. This covers add-like
operands (ADD, and OR/XOR that are adds) and shl-by-1/2/3 (which fold to a
scaled index); mul-by-3/5/9 and real subtracts are already handled, because
they consume both base and index, so matchAdd's fallback keeps them whole.
"Already materialized" is determined by hasMaterializingUse(): the operand has
a use that puts it in a register as a value - a stored value operand or a
CopyToReg (return value, call argument, or a value live out of the block). A
multi-use value consumed only by foldable-address uses (GEP indices, load/store
addresses) is not materialized and must not be reused, as that would add a
redundant materialization. Selection may already have turned an ISD::STORE into
a machine store by the time the matcher runs, so both the ISD::STORE form and
an already-selected machine store are recognized; for the latter the stored
value follows the address operands at X86::AddrNumOperands. Because the guard
only fires on a genuinely materialized value, it cannot regress: not firing is
the pre-existing behavior.
Fixes #51707.
---
llvm/lib/Target/X86/X86ISelDAGToDAG.cpp | 85 +++++++-
llvm/test/CodeGen/X86/lea-opt-cse1.ll | 13 +-
llvm/test/CodeGen/X86/lea-opt-cse2.ll | 29 ++-
llvm/test/CodeGen/X86/lea-opt-cse4.ll | 53 +++--
llvm/test/CodeGen/X86/lea-recursion.ll | 30 ++-
llvm/test/CodeGen/X86/pr51707.ll | 251 ++++++++++++++++++++++++
6 files changed, 379 insertions(+), 82 deletions(-)
create mode 100644 llvm/test/CodeGen/X86/pr51707.ll
diff --git a/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp b/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
index 9a0045367a8bf..d46b11ef60a18 100644
--- a/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
+++ b/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
@@ -79,6 +79,11 @@ namespace {
Align Alignment; // CP alignment.
unsigned char SymbolFlags = X86II::MO_NO_FLAG; // X86II::MO_*
bool NegateIndex = false;
+ // True when this address is being matched to be emitted as a LEA rather
+ // than folded into a memory operand. Unlike a memory operand, a LEA turns
+ // the folded arithmetic into real instructions, so it is not profitable to
+ // split an already-materialized (multi-use) value here. (Issue #51707)
+ bool IsForLEA = false;
X86ISelAddressMode() = default;
@@ -206,6 +211,7 @@ namespace {
bool matchAddress(SDValue N, X86ISelAddressMode &AM);
bool matchVectorAddress(SDValue N, X86ISelAddressMode &AM);
bool matchAdd(SDValue &N, X86ISelAddressMode &AM, unsigned Depth);
+ bool hasMaterializingUse(SDValue V) const;
SDValue matchIndexRecursively(SDValue N, X86ISelAddressMode &AM,
unsigned Depth);
bool matchAddressRecursively(SDValue N, X86ISelAddressMode &AM,
@@ -2061,22 +2067,90 @@ bool X86DAGToDAGISel::matchAddress(SDValue N, X86ISelAddressMode &AM) {
return false;
}
+// Returns true if V has a use that materializes it in a register as a value -
+// a stored value operand or a CopyToReg (a return value, call argument, or a
+// value that is live out of the block). Such a use means V will be in a
+// register regardless, so reusing it when forming an LEA is free. Uses where V
+// is only an address (a load/store pointer, or folded into another address
+// computation) do not materialize it. This is a more precise replacement for
+// the !hasOneUse() proxy: an address-only multi-use value is not materialized.
+bool X86DAGToDAGISel::hasMaterializingUse(SDValue V) const {
+ const TargetInstrInfo *TII = Subtarget->getInstrInfo();
+ for (SDUse &U : V->uses()) {
+ if (U.getResNo() != V.getResNo())
+ continue;
+ SDNode *User = U.getUser();
+ // A return value, call argument, or a value live out of the block.
+ if (User->getOpcode() == ISD::CopyToReg)
+ return true;
+ // A stored value materializes V (V as a store *address* does not).
+ if (auto *St = dyn_cast<StoreSDNode>(User)) {
+ if (St->getValue() == V)
+ return true;
+ continue;
+ }
+ // Selection may already have turned the ISD::STORE into a machine store by
+ // the time we get here. For a store the memory reference comes first, so
+ // the stored value is the operand at X86::AddrNumOperands (as in e.g.
+ // X86AvoidStoreForwardingBlocks). Note there is no getOperandBias() here:
+ // unlike a MachineInstr, an SDNode's operand list has no leading defs.
+ if (User->isMachineOpcode() &&
+ TII->get(User->getMachineOpcode()).mayStore() &&
+ User->getNumOperands() > X86::AddrNumOperands &&
+ User->getOperand(X86::AddrNumOperands) == V)
+ return true;
+ }
+ return false;
+}
+
bool X86DAGToDAGISel::matchAdd(SDValue &N, X86ISelAddressMode &AM,
unsigned Depth) {
// Add an artificial use to this node so that we can keep track of
// it if it gets CSE'd with a different node.
HandleSDNode Handle(N);
+ auto IsAddLike = [&](SDValue V) {
+ return V.getOpcode() == ISD::ADD || CurDAG->isADDLike(V);
+ };
+
+ // When forming a LEA, avoid splitting an already-materialized value: use the
+ // operand directly as a base/index register instead. hasMaterializingUse()
+ // decides whether the operand is genuinely materialized - it has a use that
+ // puts it in a register as a value. A value used only as an address is not
+ // materialized, and splitting it there would only add a redundant
+ // materialization (see the two_ptrs test).
+ auto SplitsMaterializedValue = [&](SDValue Op) {
+ if (!AM.IsForLEA || !hasMaterializingUse(Op))
+ return false;
+
+ // add-like: decomposes to base + index (+ disp)
+ if (IsAddLike(Op))
+ return IsAddLike(Op.getOperand(0)) || IsAddLike(Op.getOperand(1));
+
+ // shl by 1/2/3 folds to a scaled index
+ if (Op.getOpcode() == ISD::SHL)
+ if (auto *C = dyn_cast<ConstantSDNode>(Op.getOperand(1)))
+ return C->getZExtValue() >= 1 && C->getZExtValue() <= 3 &&
+ IsAddLike(Op.getOperand(0));
+
+ return false;
+ };
+
+ auto MatchOperand = [&](SDValue Op) {
+ if (SplitsMaterializedValue(Op))
+ return matchAddressBase(Op, AM);
+ return matchAddressRecursively(Op, AM, Depth + 1);
+ };
+
X86ISelAddressMode Backup = AM;
- if (!matchAddressRecursively(N.getOperand(0), AM, Depth+1) &&
- !matchAddressRecursively(Handle.getValue().getOperand(1), AM, Depth+1))
+ if (!MatchOperand(N.getOperand(0)) &&
+ !MatchOperand(Handle.getValue().getOperand(1)))
return false;
AM = Backup;
// Try again after commutating the operands.
- if (!matchAddressRecursively(Handle.getValue().getOperand(1), AM,
- Depth + 1) &&
- !matchAddressRecursively(Handle.getValue().getOperand(0), AM, Depth + 1))
+ if (!MatchOperand(Handle.getValue().getOperand(1)) &&
+ !MatchOperand(Handle.getValue().getOperand(0)))
return false;
AM = Backup;
@@ -3153,6 +3227,7 @@ bool X86DAGToDAGISel::selectLEAAddr(SDValue N,
SDValue &Index, SDValue &Disp,
SDValue &Segment) {
X86ISelAddressMode AM;
+ AM.IsForLEA = true;
// Save the DL and VT before calling matchAddress, it can invalidate N.
SDLoc DL(N);
diff --git a/llvm/test/CodeGen/X86/lea-opt-cse1.ll b/llvm/test/CodeGen/X86/lea-opt-cse1.ll
index 5ceca9fbd9b5f..88fcd09612048 100644
--- a/llvm/test/CodeGen/X86/lea-opt-cse1.ll
+++ b/llvm/test/CodeGen/X86/lea-opt-cse1.ll
@@ -9,28 +9,21 @@ define void @test_func(ptr nocapture %ctx, i32 %n) local_unnamed_addr {
; X64: # %bb.0: # %entry
; X64-NEXT: movl (%rdi), %eax
; X64-NEXT: movl 16(%rdi), %ecx
-; X64-NEXT: leal (%rax,%rcx), %edx
; X64-NEXT: leal 1(%rax,%rcx), %eax
; X64-NEXT: movl %eax, 12(%rdi)
-; X64-NEXT: leal 1(%rcx,%rdx), %eax
+; X64-NEXT: addl %ecx, %eax
; X64-NEXT: movl %eax, 16(%rdi)
; X64-NEXT: retq
;
; X86-LABEL: test_func:
; X86: # %bb.0: # %entry
-; X86-NEXT: pushl %esi
-; X86-NEXT: .cfi_def_cfa_offset 8
-; X86-NEXT: .cfi_offset %esi, -8
; X86-NEXT: movl {{[0-9]+}}(%esp), %eax
; X86-NEXT: movl (%eax), %ecx
; X86-NEXT: movl 16(%eax), %edx
-; X86-NEXT: leal 1(%ecx,%edx), %esi
+; X86-NEXT: leal 1(%ecx,%edx), %ecx
+; X86-NEXT: movl %ecx, 12(%eax)
; X86-NEXT: addl %edx, %ecx
-; X86-NEXT: movl %esi, 12(%eax)
-; X86-NEXT: leal 1(%edx,%ecx), %ecx
; X86-NEXT: movl %ecx, 16(%eax)
-; X86-NEXT: popl %esi
-; X86-NEXT: .cfi_def_cfa_offset 4
; X86-NEXT: retl
entry:
%0 = load i32, ptr %ctx, align 8
diff --git a/llvm/test/CodeGen/X86/lea-opt-cse2.ll b/llvm/test/CodeGen/X86/lea-opt-cse2.ll
index e39d01f1447f8..fbee135df0b0c 100644
--- a/llvm/test/CodeGen/X86/lea-opt-cse2.ll
+++ b/llvm/test/CodeGen/X86/lea-opt-cse2.ll
@@ -10,26 +10,22 @@ define void @foo(ptr nocapture %ctx, i32 %n) local_unnamed_addr #0 {
; X64-NEXT: .p2align 4
; X64-NEXT: .LBB0_1: # %loop
; X64-NEXT: # =>This Inner Loop Header: Depth=1
-; X64-NEXT: movl (%rdi), %eax
-; X64-NEXT: movl 16(%rdi), %ecx
-; X64-NEXT: leal 1(%rax,%rcx), %edx
-; X64-NEXT: movl %edx, 12(%rdi)
+; X64-NEXT: movl (%rdi), %ecx
+; X64-NEXT: movl 16(%rdi), %eax
+; X64-NEXT: leal 1(%rcx,%rax), %ecx
+; X64-NEXT: movl %ecx, 12(%rdi)
; X64-NEXT: decl %esi
; X64-NEXT: jne .LBB0_1
; X64-NEXT: # %bb.2: # %exit
-; X64-NEXT: addl %ecx, %eax
-; X64-NEXT: leal 1(%rcx,%rax), %eax
-; X64-NEXT: movl %eax, 16(%rdi)
+; X64-NEXT: addl %eax, %ecx
+; X64-NEXT: movl %ecx, 16(%rdi)
; X64-NEXT: retq
;
; X86-LABEL: foo:
; X86: # %bb.0: # %entry
-; X86-NEXT: pushl %edi
-; X86-NEXT: .cfi_def_cfa_offset 8
; X86-NEXT: pushl %esi
-; X86-NEXT: .cfi_def_cfa_offset 12
-; X86-NEXT: .cfi_offset %esi, -12
-; X86-NEXT: .cfi_offset %edi, -8
+; X86-NEXT: .cfi_def_cfa_offset 8
+; X86-NEXT: .cfi_offset %esi, -8
; X86-NEXT: movl {{[0-9]+}}(%esp), %ecx
; X86-NEXT: movl {{[0-9]+}}(%esp), %eax
; X86-NEXT: .p2align 4
@@ -37,17 +33,14 @@ define void @foo(ptr nocapture %ctx, i32 %n) local_unnamed_addr #0 {
; X86-NEXT: # =>This Inner Loop Header: Depth=1
; X86-NEXT: movl (%eax), %edx
; X86-NEXT: movl 16(%eax), %esi
-; X86-NEXT: leal 1(%edx,%esi), %edi
-; X86-NEXT: movl %edi, 12(%eax)
+; X86-NEXT: leal 1(%edx,%esi), %edx
+; X86-NEXT: movl %edx, 12(%eax)
; X86-NEXT: decl %ecx
; X86-NEXT: jne .LBB0_1
; X86-NEXT: # %bb.2: # %exit
; X86-NEXT: addl %esi, %edx
-; X86-NEXT: leal 1(%esi,%edx), %ecx
-; X86-NEXT: movl %ecx, 16(%eax)
+; X86-NEXT: movl %edx, 16(%eax)
; X86-NEXT: popl %esi
-; X86-NEXT: .cfi_def_cfa_offset 8
-; X86-NEXT: popl %edi
; X86-NEXT: .cfi_def_cfa_offset 4
; X86-NEXT: retl
entry:
diff --git a/llvm/test/CodeGen/X86/lea-opt-cse4.ll b/llvm/test/CodeGen/X86/lea-opt-cse4.ll
index 4fa9acd99bb2f..40868d11eb7d5 100644
--- a/llvm/test/CodeGen/X86/lea-opt-cse4.ll
+++ b/llvm/test/CodeGen/X86/lea-opt-cse4.ll
@@ -12,11 +12,10 @@ define void @foo(ptr nocapture %ctx, i32 %n) local_unnamed_addr #0 {
; X64-NEXT: addl %eax, %ecx
; X64-NEXT: addl %eax, %ecx
; X64-NEXT: addl %eax, %ecx
-; X64-NEXT: leal (%rcx,%rax), %edx
; X64-NEXT: leal 1(%rax,%rcx), %ecx
; X64-NEXT: movl %ecx, 12(%rdi)
-; X64-NEXT: leal 1(%rax,%rdx), %eax
-; X64-NEXT: movl %eax, 16(%rdi)
+; X64-NEXT: addl %eax, %ecx
+; X64-NEXT: movl %ecx, 16(%rdi)
; X64-NEXT: retq
;
; X86-LABEL: foo:
@@ -30,11 +29,10 @@ define void @foo(ptr nocapture %ctx, i32 %n) local_unnamed_addr #0 {
; X86-NEXT: addl %ecx, %edx
; X86-NEXT: addl %ecx, %edx
; X86-NEXT: addl %ecx, %edx
-; X86-NEXT: leal 1(%ecx,%edx), %esi
+; X86-NEXT: leal 1(%ecx,%edx), %edx
+; X86-NEXT: movl %edx, 12(%eax)
; X86-NEXT: addl %ecx, %edx
-; X86-NEXT: movl %esi, 12(%eax)
-; X86-NEXT: leal 1(%ecx,%edx), %ecx
-; X86-NEXT: movl %ecx, 16(%eax)
+; X86-NEXT: movl %edx, 16(%eax)
; X86-NEXT: popl %esi
; X86-NEXT: .cfi_def_cfa_offset 4
; X86-NEXT: retl
@@ -64,50 +62,43 @@ define void @foo_loop(ptr nocapture %ctx, i32 %n) local_unnamed_addr #0 {
; X64-NEXT: # =>This Inner Loop Header: Depth=1
; X64-NEXT: movl (%rdi), %ecx
; X64-NEXT: movl 16(%rdi), %eax
-; X64-NEXT: leal 1(%rcx,%rax), %edx
-; X64-NEXT: movl %edx, 12(%rdi)
+; X64-NEXT: leal 1(%rcx,%rax), %ecx
+; X64-NEXT: movl %ecx, 12(%rdi)
; X64-NEXT: decl %esi
; X64-NEXT: jne .LBB1_1
; X64-NEXT: # %bb.2: # %exit
-; X64-NEXT: addl %eax, %ecx
-; X64-NEXT: leal 1(%rax,%rcx), %ecx
; X64-NEXT: leal (%rax,%rax), %edx
-; X64-NEXT: addl %eax, %edx
; X64-NEXT: addl %edx, %ecx
-; X64-NEXT: addl %edx, %ecx
-; X64-NEXT: movl %ecx, 16(%rdi)
+; X64-NEXT: addl %edx, %eax
+; X64-NEXT: addl %ecx, %eax
+; X64-NEXT: addl %edx, %eax
+; X64-NEXT: movl %eax, 16(%rdi)
; X64-NEXT: retq
;
; X86-LABEL: foo_loop:
; X86: # %bb.0: # %entry
-; X86-NEXT: pushl %edi
-; X86-NEXT: .cfi_def_cfa_offset 8
; X86-NEXT: pushl %esi
-; X86-NEXT: .cfi_def_cfa_offset 12
-; X86-NEXT: .cfi_offset %esi, -12
-; X86-NEXT: .cfi_offset %edi, -8
-; X86-NEXT: movl {{[0-9]+}}(%esp), %edx
+; X86-NEXT: .cfi_def_cfa_offset 8
+; X86-NEXT: .cfi_offset %esi, -8
+; X86-NEXT: movl {{[0-9]+}}(%esp), %esi
; X86-NEXT: movl {{[0-9]+}}(%esp), %eax
; X86-NEXT: .p2align 4
; X86-NEXT: .LBB1_1: # %loop
; X86-NEXT: # =>This Inner Loop Header: Depth=1
-; X86-NEXT: movl (%eax), %esi
+; X86-NEXT: movl (%eax), %edx
; X86-NEXT: movl 16(%eax), %ecx
-; X86-NEXT: leal 1(%esi,%ecx), %edi
-; X86-NEXT: movl %edi, 12(%eax)
-; X86-NEXT: decl %edx
+; X86-NEXT: leal 1(%edx,%ecx), %edx
+; X86-NEXT: movl %edx, 12(%eax)
+; X86-NEXT: decl %esi
; X86-NEXT: jne .LBB1_1
; X86-NEXT: # %bb.2: # %exit
-; X86-NEXT: addl %ecx, %esi
-; X86-NEXT: leal 1(%ecx,%esi), %edx
; X86-NEXT: leal (%ecx,%ecx), %esi
-; X86-NEXT: addl %ecx, %esi
; X86-NEXT: addl %esi, %edx
-; X86-NEXT: addl %esi, %edx
-; X86-NEXT: movl %edx, 16(%eax)
+; X86-NEXT: addl %esi, %ecx
+; X86-NEXT: addl %edx, %ecx
+; X86-NEXT: addl %esi, %ecx
+; X86-NEXT: movl %ecx, 16(%eax)
; X86-NEXT: popl %esi
-; X86-NEXT: .cfi_def_cfa_offset 8
-; X86-NEXT: popl %edi
; X86-NEXT: .cfi_def_cfa_offset 4
; X86-NEXT: retl
entry:
diff --git a/llvm/test/CodeGen/X86/lea-recursion.ll b/llvm/test/CodeGen/X86/lea-recursion.ll
index 07a550fa394d6..a8bfe8b3b675c 100644
--- a/llvm/test/CodeGen/X86/lea-recursion.ll
+++ b/llvm/test/CodeGen/X86/lea-recursion.ll
@@ -17,31 +17,25 @@ define dso_local void @foo() {
; CHECK: # %bb.0: # %entry
; CHECK-NEXT: movl g0(%rip), %eax
; CHECK-NEXT: movl g1(%rip), %ecx
-; CHECK-NEXT: leal (%rax,%rcx), %edx
; CHECK-NEXT: leal 1(%rax,%rcx), %eax
; CHECK-NEXT: movl %eax, g0+4(%rip)
-; CHECK-NEXT: movl g1+4(%rip), %eax
-; CHECK-NEXT: leal 1(%rax,%rdx), %ecx
-; CHECK-NEXT: leal 2(%rax,%rdx), %eax
+; CHECK-NEXT: movl g1+4(%rip), %ecx
+; CHECK-NEXT: leal 1(%rax,%rcx), %eax
; CHECK-NEXT: movl %eax, g0+8(%rip)
-; CHECK-NEXT: movl g1+8(%rip), %eax
-; CHECK-NEXT: leal 1(%rax,%rcx), %edx
-; CHECK-NEXT: leal 2(%rax,%rcx), %eax
+; CHECK-NEXT: movl g1+8(%rip), %ecx
+; CHECK-NEXT: leal 1(%rax,%rcx), %eax
; CHECK-NEXT: movl %eax, g0+12(%rip)
-; CHECK-NEXT: movl g1+12(%rip), %eax
-; CHECK-NEXT: leal 1(%rax,%rdx), %ecx
-; CHECK-NEXT: leal 2(%rax,%rdx), %eax
+; CHECK-NEXT: movl g1+12(%rip), %ecx
+; CHECK-NEXT: leal 1(%rax,%rcx), %eax
; CHECK-NEXT: movl %eax, g0+16(%rip)
-; CHECK-NEXT: movl g1+16(%rip), %eax
-; CHECK-NEXT: leal 1(%rax,%rcx), %edx
-; CHECK-NEXT: leal 2(%rax,%rcx), %eax
+; CHECK-NEXT: movl g1+16(%rip), %ecx
+; CHECK-NEXT: leal 1(%rax,%rcx), %eax
; CHECK-NEXT: movl %eax, g0+20(%rip)
-; CHECK-NEXT: movl g1+20(%rip), %eax
-; CHECK-NEXT: leal 1(%rax,%rdx), %ecx
-; CHECK-NEXT: leal 2(%rax,%rdx), %eax
+; CHECK-NEXT: movl g1+20(%rip), %ecx
+; CHECK-NEXT: leal 1(%rax,%rcx), %eax
; CHECK-NEXT: movl %eax, g0+24(%rip)
-; CHECK-NEXT: movl g1+24(%rip), %eax
-; CHECK-NEXT: leal 2(%rax,%rcx), %eax
+; CHECK-NEXT: movl g1+24(%rip), %ecx
+; CHECK-NEXT: leal 1(%rax,%rcx), %eax
; CHECK-NEXT: movl %eax, g0+28(%rip)
; CHECK-NEXT: retq
entry:
diff --git a/llvm/test/CodeGen/X86/pr51707.ll b/llvm/test/CodeGen/X86/pr51707.ll
new file mode 100644
index 0000000000000..ffe4b7b34aaaa
--- /dev/null
+++ b/llvm/test/CodeGen/X86/pr51707.ll
@@ -0,0 +1,251 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
+; RUN: llc -mtriple=x86_64-- < %s | FileCheck %s
+
+; Issue #51707: add2 = add1 + b should reuse the already-materialized value of
+; add1 (a + b + 17) instead of rematerializing a + b.
+
+define i32 @reduced(i32 %a, i32 %b, ptr %p) {
+; CHECK-LABEL: reduced:
+; CHECK: # %bb.0:
+; CHECK-NEXT: # kill: def $esi killed $esi def $rsi
+; CHECK-NEXT: # kill: def $edi killed $edi def $rdi
+; CHECK-NEXT: leal 17(%rdi,%rsi), %eax
+; CHECK-NEXT: movl %eax, (%rdx)
+; CHECK-NEXT: addl %esi, %eax
+; CHECK-NEXT: retq
+ %add = add i32 %a, 17
+ %add1 = add i32 %add, %b
+ store i32 %add1, ptr %p, align 4
+ %add2 = add nsw i32 %add1, %b
+ ret i32 %add2
+}
+
+; int32_t f(int32_t & __restrict a, const int32_t & __restrict b) {
+; a += b + 17;
+; return a + b;
+; }
+define i32 @f(ptr noalias %a, ptr noalias readonly %b) {
+; CHECK-LABEL: f:
+; CHECK: # %bb.0:
+; CHECK-NEXT: movl (%rsi), %ecx
+; CHECK-NEXT: movl (%rdi), %eax
+; CHECK-NEXT: leal 17(%rcx,%rax), %eax
+; CHECK-NEXT: movl %eax, (%rdi)
+; CHECK-NEXT: addl %ecx, %eax
+; CHECK-NEXT: retq
+ %lb = load i32, ptr %b, align 4
+ %la = load i32, ptr %a, align 4
+ %t = add i32 %lb, 17
+ %sum = add i32 %t, %la
+ store i32 %sum, ptr %a, align 4
+ %ret = add i32 %sum, %lb
+ ret i32 %ret
+}
+
+; Commuted: the reused value is the second operand of the final add.
+define i32 @commuted(i32 %a, i32 %b, i32 %c, ptr %p) {
+; CHECK-LABEL: commuted:
+; CHECK: # %bb.0:
+; CHECK-NEXT: # kill: def $esi killed $esi def $rsi
+; CHECK-NEXT: # kill: def $edi killed $edi def $rdi
+; CHECK-NEXT: leal 17(%rdi,%rsi), %eax
+; CHECK-NEXT: movl %eax, (%rcx)
+; CHECK-NEXT: addl %edx, %eax
+; CHECK-NEXT: retq
+ %s = add i32 %a, %b
+ %add1 = add i32 %s, 17
+ store i32 %add1, ptr %p
+ %r = add i32 %c, %add1
+ ret i32 %r
+}
+
+; Deeper nesting: v = ((a + b) + c) + 17 is materialized, then reused for v + d.
+define i32 @deeper(i32 %a, i32 %b, i32 %c, i32 %d, ptr %p) {
+; CHECK-LABEL: deeper:
+; CHECK: # %bb.0:
+; CHECK-NEXT: # kill: def $edx killed $edx def $rdx
+; CHECK-NEXT: # kill: def $edi killed $edi def $rdi
+; CHECK-NEXT: addl %esi, %edi
+; CHECK-NEXT: leal 17(%rdx,%rdi), %eax
+; CHECK-NEXT: movl %eax, (%r8)
+; CHECK-NEXT: addl %ecx, %eax
+; CHECK-NEXT: retq
+ %s1 = add i32 %a, %b
+ %s2 = add i32 %s1, %c
+ %v = add i32 %s2, 17
+ store i32 %v, ptr %p
+ %r = add i32 %v, %d
+ ret i32 %r
+}
+
+; A multi-use shl-by-constant of an add folds to a scaled index; the materialized
+; value m = (a + b) << 2 should be reused for m + c, not recomputed.
+define i32 @shl_reuse(i32 %a, i32 %b, i32 %c, ptr %p) {
+; CHECK-LABEL: shl_reuse:
+; CHECK: # %bb.0:
+; CHECK-NEXT: # kill: def $esi killed $esi def $rsi
+; CHECK-NEXT: # kill: def $edi killed $edi def $rdi
+; CHECK-NEXT: leal (%rdi,%rsi), %eax
+; CHECK-NEXT: shll $2, %eax
+; CHECK-NEXT: movl %eax, (%rcx)
+; CHECK-NEXT: addl %edx, %eax
+; CHECK-NEXT: retq
+ %s = add i32 %a, %b
+ %m = shl i32 %s, 2
+ store i32 %m, ptr %p
+ %r = add i32 %m, %c
+ ret i32 %r
+}
+
+; Boundary/other-operand coverage: the following already produce optimal code
+; (with or without the reuse fix); they guard against a future matcher change
+; re-introducing the de-CSE.
+
+; A shift by 4 cannot fold into an LEA scale (max is <<3 == scale 8), so m is
+; kept whole regardless - nothing to split.
+define i32 @shl_by_4(i32 %a, i32 %b, i32 %c, ptr %p) {
+; CHECK-LABEL: shl_by_4:
+; CHECK: # %bb.0:
+; CHECK-NEXT: # kill: def $esi killed $esi def $rsi
+; CHECK-NEXT: # kill: def $edi killed $edi def $rdi
+; CHECK-NEXT: leal (%rdi,%rsi), %eax
+; CHECK-NEXT: shll $4, %eax
+; CHECK-NEXT: movl %eax, (%rcx)
+; CHECK-NEXT: addl %edx, %eax
+; CHECK-NEXT: retq
+ %s = add i32 %a, %b
+ %m = shl i32 %s, 4
+ store i32 %m, ptr %p
+ %r = add i32 %m, %c
+ ret i32 %r
+}
+
+; mul by 3/5/9 folds to lea (X, X, {2,4,8}), consuming both base and index, so
+; the sibling add operand cannot be folded and m is reused whole.
+define i32 @mul_3(i32 %a, i32 %b, i32 %c, ptr %p) {
+; CHECK-LABEL: mul_3:
+; CHECK: # %bb.0:
+; CHECK-NEXT: # kill: def $edi killed $edi def $rdi
+; CHECK-NEXT: addl %esi, %edi
+; CHECK-NEXT: leal (%rdi,%rdi,2), %eax
+; CHECK-NEXT: movl %eax, (%rcx)
+; CHECK-NEXT: addl %edx, %eax
+; CHECK-NEXT: retq
+ %s = add i32 %a, %b
+ %m = mul i32 %s, 3
+ store i32 %m, ptr %p
+ %r = add i32 %m, %c
+ ret i32 %r
+}
+
+define i32 @mul_5(i32 %a, i32 %b, i32 %c, ptr %p) {
+; CHECK-LABEL: mul_5:
+; CHECK: # %bb.0:
+; CHECK-NEXT: # kill: def $edi killed $edi def $rdi
+; CHECK-NEXT: addl %esi, %edi
+; CHECK-NEXT: leal (%rdi,%rdi,4), %eax
+; CHECK-NEXT: movl %eax, (%rcx)
+; CHECK-NEXT: addl %edx, %eax
+; CHECK-NEXT: retq
+ %s = add i32 %a, %b
+ %m = mul i32 %s, 5
+ store i32 %m, ptr %p
+ %r = add i32 %m, %c
+ ret i32 %r
+}
+
+define i32 @mul_9(i32 %a, i32 %b, i32 %c, ptr %p) {
+; CHECK-LABEL: mul_9:
+; CHECK: # %bb.0:
+; CHECK-NEXT: # kill: def $edi killed $edi def $rdi
+; CHECK-NEXT: addl %esi, %edi
+; CHECK-NEXT: leal (%rdi,%rdi,8), %eax
+; CHECK-NEXT: movl %eax, (%rcx)
+; CHECK-NEXT: addl %edx, %eax
+; CHECK-NEXT: retq
+ %s = add i32 %a, %b
+ %m = mul i32 %s, 9
+ store i32 %m, ptr %p
+ %r = add i32 %m, %c
+ ret i32 %r
+}
+
+; A real subtract (variable subtrahend) folds to base + neg-index, again
+; consuming both slots, so m = (a + b) - c is reused whole for m + d.
+define i32 @sub_var(i32 %a, i32 %b, i32 %c, i32 %d, ptr %p) {
+; CHECK-LABEL: sub_var:
+; CHECK: # %bb.0:
+; CHECK-NEXT: # kill: def $esi killed $esi def $rsi
+; CHECK-NEXT: # kill: def $edi killed $edi def $rdi
+; CHECK-NEXT: leal (%rdi,%rsi), %eax
+; CHECK-NEXT: subl %edx, %eax
+; CHECK-NEXT: movl %eax, (%r8)
+; CHECK-NEXT: addl %ecx, %eax
+; CHECK-NEXT: retq
+ %s = add i32 %a, %b
+ %m = sub i32 %s, %c
+ store i32 %m, ptr %p
+ %r = add i32 %m, %d
+ ret i32 %r
+}
+
+; Subtracting a constant is canonicalized to an add of a negative constant, so
+; this is the add-like case with a negative displacement and is covered by it.
+define i32 @sub_const(i32 %a, i32 %b, ptr %p) {
+; CHECK-LABEL: sub_const:
+; CHECK: # %bb.0:
+; CHECK-NEXT: # kill: def $esi killed $esi def $rsi
+; CHECK-NEXT: # kill: def $edi killed $edi def $rdi
+; CHECK-NEXT: leal -17(%rdi,%rsi), %eax
+; CHECK-NEXT: movl %eax, (%rdx)
+; CHECK-NEXT: addl %esi, %eax
+; CHECK-NEXT: retq
+ %s = add i32 %a, %b
+ %m = sub i32 %s, 17
+ store i32 %m, ptr %p
+ %r = add i32 %m, %b
+ ret i32 %r
+}
+
+; The materialized value used as the minuend of a subtract is reused directly.
+define i32 @minuend(i32 %a, i32 %b, i32 %c, ptr %p) {
+; CHECK-LABEL: minuend:
+; CHECK: # %bb.0:
+; CHECK-NEXT: # kill: def $esi killed $esi def $rsi
+; CHECK-NEXT: # kill: def $edi killed $edi def $rdi
+; CHECK-NEXT: leal 17(%rdi,%rsi), %eax
+; CHECK-NEXT: movl %eax, (%rcx)
+; CHECK-NEXT: subl %edx, %eax
+; CHECK-NEXT: retq
+ %s = add i32 %a, %b
+ %m = add i32 %s, 17
+ store i32 %m, ptr %p
+ %r = sub i32 %m, %c
+ ret i32 %r
+}
+
+; idx = (a+b)+4 is multi-use but consumed only by foldable-address uses (the two
+; returned GEP pointers and the load), so it is never materialized in a register
+; as a value. hasMaterializingUse() sees no such use, so idx is not reused; each
+; pointer folds base+idx+disp fresh (no redundant materialization).
+define { ptr, ptr } @two_ptrs(i64 %a, i64 %b, ptr %base, ptr %sink) nounwind {
+; CHECK-LABEL: two_ptrs:
+; CHECK: # %bb.0:
+; CHECK-NEXT: addq %rsi, %rdi
+; CHECK-NEXT: leaq 4(%rdx,%rdi), %rax
+; CHECK-NEXT: leaq 136(%rdx,%rdi), %rsi
+; CHECK-NEXT: movl 4(%rdx,%rdi), %edx
+; CHECK-NEXT: movl %edx, (%rcx)
+; CHECK-NEXT: movq %rsi, %rdx
+; CHECK-NEXT: retq
+ %s = add i64 %a, %b
+ %idx = add i64 %s, 4
+ %p0 = getelementptr i8, ptr %base, i64 %idx
+ %idx1 = add i64 %idx, 132
+ %p1 = getelementptr i8, ptr %base, i64 %idx1
+ %v0 = load i32, ptr %p0
+ store i32 %v0, ptr %sink
+ %r0 = insertvalue { ptr, ptr } poison, ptr %p0, 0
+ %r1 = insertvalue { ptr, ptr } %r0, ptr %p1, 1
+ ret { ptr, ptr } %r1
+}
>From bb750fcf60ab62ce449f8f30dbcb93cb250aa36f Mon Sep 17 00:00:00 2001
From: Nikita Taranov <nikita.taranov at clickhouse.com>
Date: Tue, 21 Jul 2026 16:26:58 +0700
Subject: [PATCH 2/4] [X86] Rename IsAddLike lambda to IsAddOrAddLike
Address review comment: the lambda returns true for a plain ISD::ADD as well
as an isADDLike OR/XOR, so IsAddOrAddLike describes it more precisely.
---
llvm/lib/Target/X86/X86ISelDAGToDAG.cpp | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp b/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
index d46b11ef60a18..68f8d4e1b630a 100644
--- a/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
+++ b/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
@@ -2109,7 +2109,7 @@ bool X86DAGToDAGISel::matchAdd(SDValue &N, X86ISelAddressMode &AM,
// it if it gets CSE'd with a different node.
HandleSDNode Handle(N);
- auto IsAddLike = [&](SDValue V) {
+ auto IsAddOrAddLike = [&](SDValue V) {
return V.getOpcode() == ISD::ADD || CurDAG->isADDLike(V);
};
@@ -2124,14 +2124,15 @@ bool X86DAGToDAGISel::matchAdd(SDValue &N, X86ISelAddressMode &AM,
return false;
// add-like: decomposes to base + index (+ disp)
- if (IsAddLike(Op))
- return IsAddLike(Op.getOperand(0)) || IsAddLike(Op.getOperand(1));
+ if (IsAddOrAddLike(Op))
+ return IsAddOrAddLike(Op.getOperand(0)) ||
+ IsAddOrAddLike(Op.getOperand(1));
// shl by 1/2/3 folds to a scaled index
if (Op.getOpcode() == ISD::SHL)
if (auto *C = dyn_cast<ConstantSDNode>(Op.getOperand(1)))
return C->getZExtValue() >= 1 && C->getZExtValue() <= 3 &&
- IsAddLike(Op.getOperand(0));
+ IsAddOrAddLike(Op.getOperand(0));
return false;
};
>From 2dd3cc9fd0ca8340e72cc83facc0a509f0d69d3a Mon Sep 17 00:00:00 2001
From: Nikita Taranov <nikita.taranov at clickhouse.com>
Date: Tue, 21 Jul 2026 16:52:47 +0700
Subject: [PATCH 3/4] [X86] Locate a machine store's value operand robustly in
hasMaterializingUse
Address review comment: the stored value is not always the operand right after
the memory reference, because the memory reference is not always the first
operand. Use X86II::getMemoryOperandNo() to find the address operands and treat
any other (non chain/glue) operand equal to V as the stored value, instead of
assuming it sits at X86::AddrNumOperands.
No functional change on the existing tests (equivalent on the whole
test/CodeGen/X86 corpus); this only fixes the assumed operand layout for store
forms that do not put the value after the memory operands.
---
llvm/lib/Target/X86/X86ISelDAGToDAG.cpp | 33 ++++++++++++++++++-------
1 file changed, 24 insertions(+), 9 deletions(-)
diff --git a/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp b/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
index 68f8d4e1b630a..827e8af60f694 100644
--- a/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
+++ b/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
@@ -2090,15 +2090,30 @@ bool X86DAGToDAGISel::hasMaterializingUse(SDValue V) const {
continue;
}
// Selection may already have turned the ISD::STORE into a machine store by
- // the time we get here. For a store the memory reference comes first, so
- // the stored value is the operand at X86::AddrNumOperands (as in e.g.
- // X86AvoidStoreForwardingBlocks). Note there is no getOperandBias() here:
- // unlike a MachineInstr, an SDNode's operand list has no leading defs.
- if (User->isMachineOpcode() &&
- TII->get(User->getMachineOpcode()).mayStore() &&
- User->getNumOperands() > X86::AddrNumOperands &&
- User->getOperand(X86::AddrNumOperands) == V)
- return true;
+ // the time we get here. V materializes it if it is a stored value, i.e. an
+ // operand that is neither part of the memory reference (the address
+ // operands) nor the chain/glue. The memory reference is not always the
+ // first operand, so locate it via the instruction's memory-operand info
+ // rather than assuming a fixed layout. (No getOperandBias() is needed:
+ // unlike a MachineInstr, an SDNode's operand list has no leading defs.)
+ if (!User->isMachineOpcode())
+ continue;
+ const MCInstrDesc &Desc = TII->get(User->getMachineOpcode());
+ if (!Desc.mayStore())
+ continue;
+ int MemRefBegin = X86II::getMemoryOperandNo(Desc.TSFlags);
+ if (MemRefBegin < 0)
+ continue;
+ unsigned MemRefEnd = MemRefBegin + X86::AddrNumOperands;
+ for (unsigned I = 0, E = User->getNumOperands(); I != E; ++I) {
+ if (I >= static_cast<unsigned>(MemRefBegin) && I < MemRefEnd)
+ continue; // an address operand
+ SDValue Opnd = User->getOperand(I);
+ if (Opnd.getValueType() == MVT::Other || Opnd.getValueType() == MVT::Glue)
+ continue; // chain / glue
+ if (Opnd == V)
+ return true; // a stored value operand
+ }
}
return false;
}
>From 2d18fbf2a869b3be75f3a7ae3ca61ee85ddf496d Mon Sep 17 00:00:00 2001
From: Nikita Taranov <nikita.taranov at clickhouse.com>
Date: Tue, 21 Jul 2026 16:53:56 +0700
Subject: [PATCH 4/4] [X86] Comment why the reuse check lives in matchAdd, not
matchAddressRecursively
Address review comment. Explain that the check is scoped to matchAdd's per-add
operand folding on purpose: matchAddressRecursively is also reached for the LEA
root and from the SUB case's operand fold, and firing there regresses code.
---
llvm/lib/Target/X86/X86ISelDAGToDAG.cpp | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp b/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
index 827e8af60f694..94cc1e49e79fe 100644
--- a/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
+++ b/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
@@ -2152,6 +2152,11 @@ bool X86DAGToDAGISel::matchAdd(SDValue &N, X86ISelAddressMode &AM,
return false;
};
+ // The check is applied here, per add operand, rather than inside
+ // matchAddressRecursively, so that it only fires when an add directly
+ // consumes the value. matchAddressRecursively is also entered for the LEA
+ // root itself and from the SUB case's operand fold.
+ // Firing there produces worse code.
auto MatchOperand = [&](SDValue Op) {
if (SplitsMaterializedValue(Op))
return matchAddressBase(Op, AM);
More information about the llvm-commits
mailing list