[llvm] [AArch64] Pair far-offset LDP/STP via base-address CSE and base-register adjustment (PR #222646)
via llvm-commits
llvm-commits at lists.llvm.org
Thu Sep 10 06:12:21 PDT 2026
llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-llvm-globalisel
Author: kk_compiler (whokeke)
<details>
<summary>Changes</summary>
LDP/STP take a 7-bit signed scaled immediate, so adjacent scaled
loads/stores with offsets outside [-64, +63] cannot pair and stay as two
independent LDR/STR. This wastes load-port pressure for struct-field
access at large offsets and arena-allocator patterns.
Far offsets fail to pair for two different reasons depending on sign,
addressed by two independent commits (no file overlap, each revertable
on its own):
**[AArch64MIPeepholeOpt] Coalesce sibling base-address materializations**
(negative offsets): ISel materializes an independent SUBXri base for
each negative far-offset access outside LDUR's 9-bit signed range, so
the accesses end up with different base registers. A pre-RA peephole
coalesces sibling single-use SUBXri/ADDXri materializations onto the
minimum-offset one and rewrites the other accesses' offsets, restoring
the shared base so the existing LoadStoreOptimizer pairing applies.
Gated by -aarch64-base-address-cse (default on).
**[AArch64LoadStoreOptimizer] Pair far-offset loads/stores via
base-register adjustment** (positive offsets): the base is already
shared but the offset exceeds the pair range. LSO inserts an ADDXri to
materialize a scratch base and emits the pair off it with offset 0.
Gated by -aarch64-ldp-stp-base-adjust (default on), restricted to
scaled accesses and loop-invariant bases (the per-pair ADDXri cannot be
hoisted out of a loop, where independent LDRs also overlap better).
For GPR64 loads the first load destination is reused as the adjusted
base (dead-by-construction); otherwise a scratch is scavenged from
X9..X15, excluding X16/X17 (IP0/IP1) which the linker may clobber via
veneers/PLT entries. Scavenging failure leaves the accesses unpaired
(missed optimization, not a miscompile).
Example (i64, p[100]/p[101]):
; before ; after
ldr x8, x0, #<!-- -->800 add x8, x0, #<!-- -->800
ldr x9, x0, #<!-- -->808 ldp x8, x9, x8
Tests: new ldp-far-offset.ll / stp-far-offset.ll (dest-reuse orderings,
X16-destination exclusion, Wn/Xn scratch aliasing, 4k-aligned ADDXri
encoding, loop-invariant vs loop-variant gating) and
base-address-cse.mir / base-address-cse.ll; existing tests whose
codegen changed are regenerated. Full AArch64 CodeGen suite shows no
regressions; all new tests run with -verify-machineinstrs.
---
Patch is 158.63 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/222646.diff
20 Files Affected:
- (modified) llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp (+214-5)
- (modified) llvm/lib/Target/AArch64/AArch64MIPeepholeOpt.cpp (+118)
- (modified) llvm/test/CodeGen/AArch64/GlobalISel/split-offsets-for-stp.ll (+6-6)
- (modified) llvm/test/CodeGen/AArch64/aarch64-gep-opt.ll (+2-3)
- (modified) llvm/test/CodeGen/AArch64/aarch64-ldst-opt-instr-ref.mir (+6)
- (modified) llvm/test/CodeGen/AArch64/aarch64-sve-fill-spill-pair.ll (+4-4)
- (added) llvm/test/CodeGen/AArch64/base-address-cse.ll (+207)
- (added) llvm/test/CodeGen/AArch64/base-address-cse.mir (+101)
- (added) llvm/test/CodeGen/AArch64/ldp-far-offset.ll (+291)
- (modified) llvm/test/CodeGen/AArch64/ldst-opt.ll (+6-6)
- (modified) llvm/test/CodeGen/AArch64/preserve-all-large-csr.ll (+6-6)
- (modified) llvm/test/CodeGen/AArch64/sme-streaming-checkvl.ll (+37-37)
- (modified) llvm/test/CodeGen/AArch64/stack-hazard.ll (+170-170)
- (added) llvm/test/CodeGen/AArch64/stp-far-offset.ll (+176)
- (modified) llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-fp-to-int.ll (+8-8)
- (modified) llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-int-extends.ll (+43-43)
- (modified) llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-int-to-fp.ll (+3-3)
- (modified) llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-masked-expandload.ll (+19-19)
- (modified) llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-masked-load.ll (+19-19)
- (modified) llvm/test/CodeGen/AArch64/sve-streaming-mode-fixed-length-trunc.ll (+354-353)
``````````diff
diff --git a/llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp b/llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp
index 289552a8d9411..e8292ffd5454c 100644
--- a/llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp
+++ b/llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp
@@ -21,6 +21,7 @@
#include "AArch64MachineFunctionInfo.h"
#include "AArch64Subtarget.h"
#include "MCTargetDesc/AArch64AddressingModes.h"
+#include "llvm/ADT/BitVector.h"
#include "llvm/ADT/SetVector.h"
#include "llvm/ADT/SmallVector.h"
#include "llvm/ADT/Statistic.h"
@@ -32,8 +33,10 @@
#include "llvm/CodeGen/MachineFunctionPass.h"
#include "llvm/CodeGen/MachineInstr.h"
#include "llvm/CodeGen/MachineInstrBuilder.h"
+#include "llvm/CodeGen/MachineLoopInfo.h"
#include "llvm/CodeGen/MachineOperand.h"
#include "llvm/CodeGen/MachineRegisterInfo.h"
+#include "llvm/CodeGen/RegisterScavenging.h"
#include "llvm/CodeGen/TargetRegisterInfo.h"
#include "llvm/IR/DebugLoc.h"
#include "llvm/MC/MCAsmInfo.h"
@@ -67,6 +70,10 @@ STATISTIC(NumConstOffsetFolded,
"Number of const offset of index address folded");
STATISTIC(NumUMOVFoldedToFPRStore,
"Number of UMOV + GPR stores folded to FPR stores");
+STATISTIC(NumBaseAdjustLdpCreated,
+ "Number of LDP created with base register adjustment");
+STATISTIC(NumBaseAdjustStpCreated,
+ "Number of STP created with base register adjustment");
DEBUG_COUNTER(RegRenamingCounter, DEBUG_TYPE "-reg-renaming",
"Controls which pairs are considered for renaming");
@@ -94,6 +101,14 @@ static cl::opt<unsigned> UMOVFoldLimit("aarch64-umov-fold-scan-limit",
static cl::opt<bool> EnableRenaming("aarch64-load-store-renaming",
cl::init(true), cl::Hidden);
+// Allow LDP/STP pairing for far-offset scaled loads/stores by inserting an
+// ADDXri to adjust the base register. Restricted to scaled accesses with
+// loop-invariant bases.
+static cl::opt<bool> EnableLdpStpBaseAdjust(
+ "aarch64-ldp-stp-base-adjust", cl::init(true), cl::Hidden,
+ cl::desc("Allow LDP/STP pairing for far-offset scaled loads/stores by "
+ "adjusting the base register"));
+
#define AARCH64_LOAD_STORE_OPT_NAME "AArch64 load / store optimization pass"
namespace {
@@ -115,6 +130,14 @@ using LdStPairFlags = struct LdStPairFlags {
// forward.
std::optional<MCPhysReg> RenameReg;
+ // If true, the LDP/STP offset is out of range and we need to insert an
+ // ADDXri to compute an adjusted base register before the pair.
+ bool RequiresBaseAdjust = false;
+ // Byte offset added to the base register (encoded in ADDXri).
+ int64_t BaseAdjustByteOffset = 0;
+ // Scratch register used as the adjusted base for the pair.
+ Register BaseAdjustScratchReg = AArch64::NoRegister;
+
LdStPairFlags() = default;
void setMergeForward(bool V = true) { MergeForward = V; }
@@ -126,13 +149,31 @@ using LdStPairFlags = struct LdStPairFlags {
void setRenameReg(MCPhysReg R) { RenameReg = R; }
void clearRenameReg() { RenameReg = std::nullopt; }
std::optional<MCPhysReg> getRenameReg() const { return RenameReg; }
+
+ void setRequiresBaseAdjust(bool V = true) { RequiresBaseAdjust = V; }
+ bool getRequiresBaseAdjust() const { return RequiresBaseAdjust; }
+ void setBaseAdjustByteOffset(int64_t V) { BaseAdjustByteOffset = V; }
+ int64_t getBaseAdjustByteOffset() const { return BaseAdjustByteOffset; }
+ void setBaseAdjustScratchReg(Register R) { BaseAdjustScratchReg = R; }
+ Register getBaseAdjustScratchReg() const { return BaseAdjustScratchReg; }
};
+// Check whether Val can be encoded as the immediate operand of an ADDXri
+// instruction (12-bit unsigned immediate with shift 0 or 12).
+static bool canEncodeAddXriImm(int64_t Val) {
+ if (Val < 0)
+ return false;
+ if (Val <= 0xFFF)
+ return true;
+ return (Val & 0xFFF) == 0 && (Val >> 12) <= 0xFFF;
+}
+
struct AArch64LoadStoreOpt {
AliasAnalysis *AA;
const AArch64InstrInfo *TII;
const TargetRegisterInfo *TRI;
const AArch64Subtarget *Subtarget;
+ MachineLoopInfo *MLI = nullptr;
// Track which register units have been modified and used.
LiveRegUnits ModifiedRegUnits, UsedRegUnits;
@@ -229,6 +270,11 @@ struct AArch64LoadStoreOpt {
// Replace a UMOV (lane 0) + GPR store with a direct FPR sub-register store.
bool tryToReplaceUMOVStore(MachineBasicBlock::iterator &MBBI);
+ // Returns true if the base register of MI is not modified anywhere in the
+ // enclosing loop (or MI is not in a loop). LSO runs post-RA with no SSA
+ // def-use chain, so scan the loop body for writes to the base physreg.
+ bool isBaseLoopInvariant(const MachineInstr &MI);
+
bool optimizeBlock(MachineBasicBlock &MBB, bool EnableNarrowZeroStOpt);
bool runOnMachineFunction(MachineFunction &MF);
@@ -243,6 +289,8 @@ struct AArch64LoadStoreOptLegacy : public MachineFunctionPass {
void getAnalysisUsage(AnalysisUsage &AU) const override {
AU.addRequired<AAResultsWrapperPass>();
+ AU.addRequired<MachineLoopInfoWrapperPass>();
+ AU.addPreserved<MachineLoopInfoWrapperPass>();
MachineFunctionPass::getAnalysisUsage(AU);
}
@@ -1146,6 +1194,29 @@ AArch64LoadStoreOpt::mergePairedInsns(MachineBasicBlock::iterator I,
#endif
}
+ // --- Base register adjustment for far-offset LDP/STP ---
+ if (Flags.getRequiresBaseAdjust()) {
+ Register Scratch = Flags.getBaseAdjustScratchReg();
+ int64_t ByteOffset = Flags.getBaseAdjustByteOffset();
+ unsigned Imm12, Shift;
+ if (ByteOffset <= 0xFFF) {
+ Imm12 = ByteOffset;
+ Shift = 0;
+ } else {
+ Imm12 = ByteOffset >> 12;
+ Shift = 12;
+ }
+ MachineBasicBlock *AdjMBB = I->getParent();
+ DebugLoc AdjDL = I->getDebugLoc();
+ Register BaseReg = AArch64InstrInfo::getLdStBaseOp(*I).getReg();
+ BuildMI(*AdjMBB, I, AdjDL, TII->get(AArch64::ADDXri))
+ .addDef(Scratch)
+ .addUse(BaseReg)
+ .addImm(Imm12)
+ .addImm(Shift);
+ MergeForward = false;
+ }
+
// Insert our new paired instruction after whichever of the paired
// instructions MergeForward indicates.
MachineBasicBlock::iterator InsertionPoint = MergeForward ? Paired : I;
@@ -1154,6 +1225,17 @@ AArch64LoadStoreOpt::mergePairedInsns(MachineBasicBlock::iterator I,
const MachineOperand &BaseRegOp =
MergeForward ? AArch64InstrInfo::getLdStBaseOp(*Paired)
: AArch64InstrInfo::getLdStBaseOp(*I);
+ // If base adjustment is used, override the base register to the scratch
+ // register.
+ MachineOperand BaseRegOpForLDP = BaseRegOp;
+ if (Flags.getRequiresBaseAdjust()) {
+ Register Scratch = Flags.getBaseAdjustScratchReg();
+ BaseRegOpForLDP =
+ MachineOperand::CreateReg(Scratch, /*isDef=*/false, /*isImp=*/false,
+ /*isKill=*/false, /*isDead=*/false,
+ /*isUndef=*/false, /*isEarlyClobber=*/false,
+ /*isImplicit=*/false);
+ }
int Offset = AArch64InstrInfo::getLdStOffsetOp(*I).getImm();
int PairedOffset = AArch64InstrInfo::getLdStOffsetOp(*Paired).getImm();
@@ -1198,6 +1280,10 @@ AArch64LoadStoreOpt::mergePairedInsns(MachineBasicBlock::iterator I,
"Unscaled offset cannot be scaled.");
OffsetImm /= TII->getMemScale(*RtMI);
}
+ // With base adjustment the pair address is scratch + 0; the byte offset
+ // is carried by the ADDXri.
+ if (Flags.getRequiresBaseAdjust())
+ OffsetImm = 0;
// Construct the new instruction.
MachineInstrBuilder MIB;
@@ -1243,7 +1329,7 @@ AArch64LoadStoreOpt::mergePairedInsns(MachineBasicBlock::iterator I,
MIB.add(RegOp0)
.add(RegOp1)
- .add(BaseRegOp)
+ .add(BaseRegOpForLDP)
.addImm(OffsetImm)
.cloneMergedMemRefs({&*I, &*Paired})
.setMIFlags(I->mergeFlagsWith(*Paired));
@@ -2047,6 +2133,31 @@ AArch64LoadStoreOpt::findMatchingInsn(MachineBasicBlock::iterator I,
// Remember any instructions that read/write memory between FirstMI and MI.
SmallVector<MachineInstr *, 4> MemInsns;
+ // Scratch availability at FirstMI, computed lazily and at most once per
+ // call. ModifiedRegUnits/UsedRegUnits only cover the region between the
+ // paired insns, so the base-adjust path needs the scavenger for sound
+ // liveness at FirstMI; without it, no scratch is picked.
+ BitVector AvailableScratchRegs;
+ bool ScavengerReady = false;
+ bool ScavengerComputed = false;
+ auto computeScratchAvailability = [&] {
+ if (ScavengerComputed)
+ return;
+ ScavengerComputed = true;
+ const MachineFunction &MF = *FirstMI.getParent()->getParent();
+ if (MF.getRegInfo().tracksLiveness()) {
+ RegScavenger RS;
+ RS.enterBasicBlockEnd(*I->getParent());
+ // backward(I) stops at FirstMI, so the live set is taken between
+ // FirstMI and the next instruction; a store's Rt looks dead there,
+ // which the regsOverlap skip below compensates for.
+ RS.backward(I);
+ AvailableScratchRegs =
+ RS.getRegsAvailable(&AArch64::GPR64commonRegClass);
+ ScavengerReady = true;
+ }
+ };
+
LLVM_DEBUG(dbgs() << "Find match for: "; FirstMI.dump());
for (unsigned Count = 0; MBBI != E && Count < Limit;
MBBI = next_nodbg(MBBI, E)) {
@@ -2146,15 +2257,74 @@ AArch64LoadStoreOpt::findMatchingInsn(MachineBasicBlock::iterator I,
// Pairwise instructions have a 7-bit signed offset field. Single
// insns have a 12-bit unsigned offset field. If the resultant
// immediate offset of merging these instructions is out of range for
- // a pairwise instruction, bail and keep looking.
+ // a pairwise instruction, try base register adjustment.
if (!inBoundsForPair(IsUnscaled, MinOffset, OffsetStride)) {
+ if (EnableLdpStpBaseAdjust) {
+ int MemScale = TII->getMemScale(FirstMI);
+ int64_t ByteOffset = static_cast<int64_t>(MinOffset) *
+ (IsUnscaled ? 1 : MemScale);
+ if (ByteOffset >= 0 && canEncodeAddXriImm(ByteOffset)) {
+ Register Reg0 = getLdStRegOp(FirstMI).getReg();
+ Register Reg1 = getLdStRegOp(MI).getReg();
+ Register Scratch = AArch64::NoRegister;
+
+ // For GPR64 loads, reuse the first destination register as
+ // the adjusted base. It is dead-by-construction (about to
+ // be written by the load), so no liveness analysis is
+ // needed, and `ldp Rt1, Rt2, [Rt1]` is legal: the base is
+ // read before Rt1 is written (same model as
+ // `ldr x0, [x0, #8]`). X16/X17 are skipped here too; see
+ // the scavenger path below.
+ if (MayLoad && AArch64::GPR64RegClass.contains(Reg0) &&
+ !TRI->regsOverlap(Reg0, Reg1) &&
+ Reg0 != AArch64::X16 && Reg0 != AArch64::X17)
+ Scratch = Reg0;
+ else {
+ // Stores and sub-64-bit loads cannot reuse a dest/source
+ // register (it holds the value to store, or is not a
+ // 64-bit base), so scavenge from caller-saved temporaries
+ // x9-x15. X16/X17 (IP0/IP1) are excluded: the linker may
+ // clobber them with veneers/PLT entries, and holding a
+ // live adjusted base in IP0/IP1 across the add/ldp is
+ // unsound under CSPGO, where long-range branch veneers
+ // are common. If no register is free, leave the accesses
+ // unpaired; a missed optimization beats a miscompile.
+ computeScratchAvailability();
+ static const MCPhysReg ScratchCandidates[] = {
+ AArch64::X9, AArch64::X10, AArch64::X11, AArch64::X12,
+ AArch64::X13, AArch64::X14, AArch64::X15};
+ // Candidates are 64-bit while Reg0/Reg1 may be 32-bit (Wn)
+ // for sub-64-bit accesses, so skip by register overlap: an
+ // exact compare would miss that Xn aliases Wn and let the
+ // ADDXri clobber the stored value before the STP reads it.
+ if (ScavengerReady) {
+ for (MCPhysReg Reg : ScratchCandidates) {
+ if (TRI->regsOverlap(Reg, Reg0) ||
+ TRI->regsOverlap(Reg, Reg1))
+ continue;
+ if (AvailableScratchRegs[Reg]) {
+ Scratch = Reg;
+ break;
+ }
+ }
+ }
+ }
+ if (Scratch != AArch64::NoRegister) {
+ Flags.setRequiresBaseAdjust(true);
+ Flags.setBaseAdjustByteOffset(ByteOffset);
+ Flags.setBaseAdjustScratchReg(Scratch);
+ goto base_adjust_accepted;
+ }
+ }
+ }
LiveRegUnits::accumulateUsedDefed(MI, ModifiedRegUnits,
UsedRegUnits, TRI);
MemInsns.push_back(&MI);
- LLVM_DEBUG(dbgs() << "Offset doesn't fit in immediate, "
+ LLVM_DEBUG(dbgs() << "No base adjust opportunity, "
<< "keep looking.\n");
continue;
}
+ base_adjust_accepted:
// If the alignment requirements of the paired (scaled) instruction
// can't express the offset of the unscaled input, bail and keep
// looking.
@@ -2843,6 +3013,20 @@ bool AArch64LoadStoreOpt::tryToMergeZeroStInst(
return false;
}
+bool AArch64LoadStoreOpt::isBaseLoopInvariant(const MachineInstr &MI) {
+ if (!MLI)
+ return true;
+ const MachineLoop *L = MLI->getLoopFor(MI.getParent());
+ if (!L)
+ return true;
+ Register BaseReg = AArch64InstrInfo::getLdStBaseOp(MI).getReg();
+ for (const MachineBasicBlock *MBB : L->blocks())
+ for (const MachineInstr &LoopMI : *MBB)
+ if (LoopMI.modifiesRegister(BaseReg, TRI))
+ return false;
+ return true;
+}
+
// Find loads and stores that can be merged into a single load or store pair
// instruction.
bool AArch64LoadStoreOpt::tryToPairLdStInst(MachineBasicBlock::iterator &MBBI) {
@@ -2869,8 +3053,25 @@ bool AArch64LoadStoreOpt::tryToPairLdStInst(MachineBasicBlock::iterator &MBBI) {
// Allow one more for offset.
if (Offset > 0)
Offset -= OffsetStride;
- if (!inBoundsForPair(IsUnscaled, Offset, OffsetStride))
- return false;
+ if (!inBoundsForPair(IsUnscaled, Offset, OffsetStride)) {
+ // Base adjustment is restricted to scaled accesses: an unscaled far
+ // offset cannot occur in practice (LDUR's signed 9-bit range covers the
+ // LDP/STP pair range), and pairing an unscaled access whose byte offset
+ // is not a multiple of the scale would truncate it during the merge.
+ if (!EnableLdpStpBaseAdjust || IsUnscaled)
+ return false;
+ // Base-adjust is net negative for loop-variant bases: the per-pair
+ // ADDXri cannot be hoisted out of the loop.
+ if (!isBaseLoopInvariant(MI))
+ return false;
+ int MemScale = TII->getMemScale(MI);
+ int64_t ByteOffset =
+ static_cast<int64_t>(Offset) * (IsUnscaled ? 1 : MemScale);
+ if (ByteOffset < 0 || !canEncodeAddXriImm(ByteOffset))
+ return false;
+ LLVM_DEBUG(dbgs() << "Offset " << Offset
+ << " out of LDP/STP range, may use base adjustment\n");
+ }
// Look ahead up to LdStLimit instructions for a pairable instruction.
LdStPairFlags Flags;
@@ -2914,6 +3115,12 @@ bool AArch64LoadStoreOpt::tryToPairLdStInst(MachineBasicBlock::iterator &MBBI) {
++NumPairCreated;
if (TII->hasUnscaledLdStOffset(MI))
++NumUnscaledPairCreated;
+ if (Flags.getRequiresBaseAdjust()) {
+ if (MI.mayLoad())
+ ++NumBaseAdjustLdpCreated;
+ else
+ ++NumBaseAdjustStpCreated;
+ }
MBBI = mergePairedInsns(MBBI, Paired, Flags);
// Collect liveness info for instructions between Prev and the new position
@@ -3333,6 +3540,7 @@ bool AArch64LoadStoreOptLegacy::runOnMachineFunction(MachineFunction &MF) {
return false;
AArch64LoadStoreOpt Impl;
Impl.AA = &getAnalysis<AAResultsWrapperPass>().getAAResults();
+ Impl.MLI = &getAnalysis<MachineLoopInfoWrapperPass>().getLI();
return Impl.runOnMachineFunction(MF);
}
@@ -3349,6 +3557,7 @@ AArch64LoadStoreOptPass::run(MachineFunction &MF,
Impl.AA = &MFAM.getResult<FunctionAnalysisManagerMachineFunctionProxy>(MF)
.getManager()
.getResult<AAManager>(MF.getFunction());
+ Impl.MLI = &MFAM.getResult<MachineLoopAnalysis>(MF);
bool Changed = Impl.runOnMachineFunction(MF);
if (!Changed)
return PreservedAnalyses::all();
diff --git a/llvm/lib/Target/AArch64/AArch64MIPeepholeOpt.cpp b/llvm/lib/Target/AArch64/AArch64MIPeepholeOpt.cpp
index 554d5938cf2cd..b4f37d4f2032e 100644
--- a/llvm/lib/Target/AArch64/AArch64MIPeepholeOpt.cpp
+++ b/llvm/lib/Target/AArch64/AArch64MIPeepholeOpt.cpp
@@ -72,13 +72,20 @@
#include "AArch64ExpandImm.h"
#include "AArch64InstrInfo.h"
#include "MCTargetDesc/AArch64AddressingModes.h"
+#include "llvm/ADT/MapVector.h"
#include "llvm/CodeGen/MachineDominators.h"
#include "llvm/CodeGen/MachineLoopInfo.h"
+#include "llvm/Support/CommandLine.h"
using namespace llvm;
#define DEBUG_TYPE "aarch64-mi-peephole-opt"
+static cl::opt<bool> EnableBaseAddressCSE(
+ "aarch64-base-address-cse", cl::init(true), cl::Hidden,
+ cl::desc("Coalesce sibling base-address materializations to enable "
+ "LDP/STP pairing"));
+
namespace {
class AArch64MIPeepholeOptImpl {
@@ -143,6 +150,8 @@ class AArch64MIPeepholeOptImpl {
bool visitFMOVDr(MachineInstr &MI);
bool visitUBFMXri(MachineInstr &MI);
bool visitCopy(MachineInstr &MI);
+
+ bool shareBaseAddresses(MachineBasicBlock &MBB);
};
struct AArch64MIPeepholeOptLegacy : public MachineFunctionPass {
@@ -957,6 +966,114 @@ bool AArch64MIPeepholeOptImpl::visitCopy(MachineInstr &MI) {
return true;
}
+bool AArch64MIPeepholeOptImpl::shareBaseAddresses(MachineBasicBlock &MBB) {
+ if (!EnableBaseAddressCSE)
+ return false;
+
+ bool Changed = false;
+
+ DenseMap<MachineInstr *, unsigned> Pos;
+ unsigned Idx = 0;
+ for (MachineInstr &MI : MBB.instrs())
+ Pos[&MI] = Idx++;
+
+ struct Cand {
+ MachineInstr *DefMI;
+ Register SrcReg;
+ unsigned Opc;
+ int64_t C;
+ Register BaseReg;
+ MachineInstr *UserMI;
+ int Scale;
+ int64_t OldOff;
+ };
+ SmallVector<Cand, 8> Cands;
+ using Key = std::pair<Register, unsigned>;
+ MapVector<Key, SmallVector<unsigned, 4>> Groups;
+
+ for (MachineInstr &MI : MBB.instrs()) {
+ unsigned Opc = MI.getOpcode();
+ if (Opc != AArch64::ADDXri && Opc != AArch64::SUBXri)
+ continue;
+ if (!MI.getOperand(2).isImm() || !MI.getOperand(3).isImm())
+ continue;
+ if (MI.getOperand(3).getImm() != 0) // shift must be 0 (no LSL #12)
+ continue;
+ Register SrcReg = MI.getOperand(1).getReg();
+ Register BaseReg = MI.getOperand(0).getReg();
+ if (!SrcReg.isVirtual() || !BaseReg.isVirtual())
+ continue;
+ if (!MRI->hasOneUse(BaseReg)) // secondary def is erased below
+ continue;
+ MachineInstr &User = *MRI->use_instr_begin(BaseReg);
+ if (!User.mayLoadOrStore())
+ continue;
+ if (AArch64InstrInfo::isPairedLdSt(User) ||
+ AArch64InstrInfo::isPreLdSt(User)) // different operand layout
+ continue;
+ if (TII->hasUnscaledLdStOffset(User)) // rewrite below assumes scaled
+ continue;
+ if (!AArch64InstrInfo::getLdStOffsetOp(User).isImm())
+ continue;
+ const MachineOperand &BaseOp = AArch64InstrInfo::getLdStBaseOp(User);
+ if (!BaseOp.isReg() || BaseOp.getReg() != BaseReg)
+ continue;
+ unsigned CIdx = Cands.size();
+ Cands.push_back({&MI, SrcReg, Opc, MI.getOperand(2).getImm(), BaseReg,
+ &User, TII->getMemScale(User),
+ AArch64InstrInfo::getLdStOffsetOp(User).getImm()});
+ Groups[{SrcReg, Opc}].push_back(CIdx);
+ }
+
+ auto Eff = [](const Cand &C) {
+ return C.Opc == AArch64::ADDXri ? C.C : -C.C;
+ };
+
+ for (auto &Group : Groups) {
+ auto &Idxs = Group.second;
+ if (Idxs.size() ...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/222646
More information about the llvm-commits
mailing list