[llvm-branch-commits] [llvm] [LoongArch] Avoid store forwarding blocks in small copies (PR #231948)
via llvm-branch-commits
llvm-branch-commits at lists.llvm.org
Wed Oct 14 07:58:09 PDT 2026
https://github.com/heiher created https://github.com/llvm/llvm-project/pull/231948
Store-to-load forwarding fails when a load is wider than the store that wrote its bytes, for GPR, FPR, LSX and LASX accesses alike. On LA664 such a load takes 14-17 cycles instead of 7-10.
Small memcpy/memmove are lowered to ld.d/st.d, fld.d/fst.d, vld/vst or xvld/xvst copies. Add a pass that breaks such a copy into smaller copies when its load is likely to read data from in-flight narrower stores:
- A narrower store shortly before the load writes inside the copied bytes through the same base register or frame index: break the copy around the store.
- In a loop, a store of the previous iteration of at least 8 bytes may write the same buffer, as in LZ77 back-reference copies: break the copy into 8-byte pieces.
The pass runs at -O1 and above and can be disabled with -loongarch-disable-avoid-sfb.
rustc-perf bufreader_snappy on a Loongson-3A6000 with rustc's default loongarch64 target features (+f,+d,+lsx,+relax): 282.8M -> 241.0M cycles (-14.8%).
This patch refers to X86AvoidStoreForwardingBlocks.
>From 3e18e0294686d02ef02f10ad5ba04d90f90b0e13 Mon Sep 17 00:00:00 2001
From: WANG Rui <wangrui at loongson.cn>
Date: Wed, 14 Oct 2026 10:51:14 +0800
Subject: [PATCH] [LoongArch] Avoid store forwarding blocks in small copies
Store-to-load forwarding fails when a load is wider than the store that
wrote its bytes, for GPR, FPR, LSX and LASX accesses alike. On LA664 such
a load takes 14-17 cycles instead of 7-10.
Small memcpy/memmove are lowered to ld.d/st.d, fld.d/fst.d, vld/vst or
xvld/xvst copies. Add a pass that breaks such a copy into smaller copies
when its load is likely to read data from in-flight narrower stores:
- A narrower store shortly before the load writes inside the copied bytes
through the same base register or frame index: break the copy around
the store.
- In a loop, a store of the previous iteration of at least 8 bytes may
write the same buffer, as in LZ77 back-reference copies: break the copy
into 8-byte pieces.
The pass runs at -O1 and above and can be disabled with
-loongarch-disable-avoid-sfb.
rustc-perf bufreader_snappy on a Loongson-3A6000 with rustc's default
loongarch64 target features (+f,+d,+lsx,+relax): 282.8M -> 241.0M cycles
(-14.8%).
This patch refers to X86AvoidStoreForwardingBlocks.
---
llvm/lib/Target/LoongArch/CMakeLists.txt | 1 +
llvm/lib/Target/LoongArch/LoongArch.h | 2 +
.../LoongArchAvoidStoreForwardingBlocks.cpp | 555 ++++++++++++++++++
.../LoongArch/LoongArchTargetMachine.cpp | 3 +
llvm/test/CodeGen/LoongArch/avoid-sfb.ll | 361 ++++++++++--
llvm/test/CodeGen/LoongArch/opt-pipeline.ll | 1 +
.../llvm/lib/Target/LoongArch/BUILD.gn | 1 +
7 files changed, 876 insertions(+), 48 deletions(-)
create mode 100644 llvm/lib/Target/LoongArch/LoongArchAvoidStoreForwardingBlocks.cpp
diff --git a/llvm/lib/Target/LoongArch/CMakeLists.txt b/llvm/lib/Target/LoongArch/CMakeLists.txt
index 26da74c926df87..83d07cacbcbc06 100644
--- a/llvm/lib/Target/LoongArch/CMakeLists.txt
+++ b/llvm/lib/Target/LoongArch/CMakeLists.txt
@@ -17,6 +17,7 @@ add_public_tablegen_target(LoongArchCommonTableGen)
add_llvm_target(LoongArchCodeGen
LoongArchAsmPrinter.cpp
+ LoongArchAvoidStoreForwardingBlocks.cpp
LoongArchDeadRegisterDefinitions.cpp
LoongArchExpandAtomicPseudoInsts.cpp
LoongArchExpandPseudoInsts.cpp
diff --git a/llvm/lib/Target/LoongArch/LoongArch.h b/llvm/lib/Target/LoongArch/LoongArch.h
index 1926420440a0cd..69a11dbb0afa6c 100644
--- a/llvm/lib/Target/LoongArch/LoongArch.h
+++ b/llvm/lib/Target/LoongArch/LoongArch.h
@@ -33,6 +33,7 @@ bool lowerLoongArchMachineOperandToMCOperand(const MachineOperand &MO,
MCOperand &MCOp,
const AsmPrinter &AP);
+FunctionPass *createLoongArchAvoidStoreForwardingBlocksPass();
FunctionPass *createLoongArchDeadRegisterDefinitionsPass();
FunctionPass *createLoongArchExpandAtomicPseudoPass();
FunctionPass *createLoongArchISelDag(LoongArchTargetMachine &TM,
@@ -44,6 +45,7 @@ FunctionPass *createLoongArchPreRAExpandPseudoPass();
FunctionPass *createLoongArchExpandPseudoPass();
FunctionPass *createLoongArchLateBranchOptPass();
void initializeLoongArchAsmPrinterPass(PassRegistry &);
+void initializeLoongArchAvoidSFBPass(PassRegistry &);
void initializeLoongArchDAGToDAGISelLegacyPass(PassRegistry &);
void initializeLoongArchDeadRegisterDefinitionsPass(PassRegistry &);
void initializeLoongArchExpandAtomicPseudoPass(PassRegistry &);
diff --git a/llvm/lib/Target/LoongArch/LoongArchAvoidStoreForwardingBlocks.cpp b/llvm/lib/Target/LoongArch/LoongArchAvoidStoreForwardingBlocks.cpp
new file mode 100644
index 00000000000000..24f9291fd9fc0e
--- /dev/null
+++ b/llvm/lib/Target/LoongArch/LoongArchAvoidStoreForwardingBlocks.cpp
@@ -0,0 +1,555 @@
+//===- LoongArchAvoidStoreForwardingBlocks.cpp - Avoid HW Store Forward Block //
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass breaks small memcpy-like load/store pairs (LD_D/ST_D, FLD_D/FST_D,
+// VLD/VST and XVLD/XVST) into smaller copies when the load is likely to read
+// data from in-flight narrower stores. Store-to-load forwarding fails when the
+// load is wider than the store, which delays the load by several cycles.
+//
+// A store blocks a copy if it is executed shortly before the copy's load and
+// either:
+// - is narrower than the load and writes inside the loaded bytes through the
+// same base register or frame index. The copy is broken around it.
+// - belongs to the previous loop iteration, is at least 8 bytes wide and may
+// write the same underlying object, as in LZ77 back-reference copies:
+// do { memcpy(dst, dst - offset, 16); dst += 16; } while (dst < end);
+// The copy is broken into 8-byte pieces.
+//
+// All partial loads are emitted before all partial stores, so overlapping
+// copies (memmove) remain correct.
+//
+//===----------------------------------------------------------------------===//
+
+#include "LoongArch.h"
+#include "LoongArchInstrInfo.h"
+#include "LoongArchSubtarget.h"
+#include "llvm/ADT/SmallPtrSet.h"
+#include "llvm/ADT/Statistic.h"
+#include "llvm/Analysis/AliasAnalysis.h"
+#include "llvm/Analysis/ValueTracking.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/MachineInstrBuilder.h"
+#include "llvm/CodeGen/MachineLoopInfo.h"
+#include "llvm/CodeGen/MachineRegisterInfo.h"
+#include "llvm/InitializePasses.h"
+#include <map>
+
+using namespace llvm;
+
+#define DEBUG_TYPE "loongarch-avoid-sfb"
+#define LOONGARCH_AVOID_SFB_NAME "LoongArch Avoid Store Forwarding Blocks"
+
+STATISTIC(NumBrokenCopies, "Number of copies broken into smaller copies");
+STATISTIC(NumBrokenScalarCopies,
+ "Number of 8-byte scalar copies broken into smaller copies");
+STATISTIC(NumLoopCarriedBlocks,
+ "Number of copies broken because of loop-carried blocking stores");
+
+static cl::opt<bool> DisableAvoidSFB("loongarch-disable-avoid-sfb", cl::Hidden,
+ cl::desc("LoongArch: Disable Store "
+ "Forwarding Blocks fixup."),
+ cl::init(false));
+
+static cl::opt<unsigned> SFBInspectionLimit(
+ "loongarch-sfb-inspection-limit", cl::Hidden, cl::init(20),
+ cl::desc("LoongArch: Number of instructions backward to inspect for store "
+ "forwarding blocks."));
+
+namespace {
+
+using DisplacementSizeMap = std::map<int64_t, unsigned>;
+
+class LoongArchAvoidSFB : public MachineFunctionPass {
+public:
+ static char ID;
+ LoongArchAvoidSFB() : MachineFunctionPass(ID) {}
+
+ StringRef getPassName() const override { return LOONGARCH_AVOID_SFB_NAME; }
+
+ bool runOnMachineFunction(MachineFunction &MF) override;
+
+ void getAnalysisUsage(AnalysisUsage &AU) const override {
+ AU.setPreservesCFG();
+ AU.addRequired<AAResultsWrapperPass>();
+ AU.addRequired<MachineLoopInfoWrapperPass>();
+ AU.addPreserved<MachineLoopInfoWrapperPass>();
+ MachineFunctionPass::getAnalysisUsage(AU);
+ }
+
+private:
+ MachineRegisterInfo *MRI = nullptr;
+ const LoongArchInstrInfo *TII = nullptr;
+ AliasAnalysis *AA = nullptr;
+ MachineLoopInfo *MLI = nullptr;
+ SmallVector<std::pair<MachineInstr *, MachineInstr *>, 2>
+ BlockedLoadsStoresPairs;
+ SmallVector<MachineInstr *, 2> ForRemoval;
+
+ /// A store that may block a load, and whether it was reached through a
+ /// loop back edge (i.e. it belongs to a previous iteration).
+ struct PotentialBlocker {
+ MachineInstr *MI;
+ bool ViaBackEdge;
+ };
+
+ /// Returns couples of Load then Store to memory which look like a memcpy.
+ void findPotentiallyBlockedCopies(MachineFunction &MF);
+ SmallVector<PotentialBlocker, 4>
+ findPotentialBlockers(MachineInstr *LoadInst);
+ bool mayAliasAcrossIterations(const MachineMemOperand &LdMMO,
+ const MachineMemOperand &StMMO) const;
+ /// Break the memcpy's load and store into smaller copies such that each
+ /// memory load that was blocked by a smaller store would now be copied
+ /// separately.
+ void breakBlockedCopies(MachineInstr *LoadInst, MachineInstr *StoreInst,
+ const DisplacementSizeMap &BlockingStoresDispSizeMap);
+ /// Break a copy of size Size to smaller copies.
+ void buildCopies(int Size, MachineInstr *LoadInst, int64_t LdDispImm,
+ MachineInstr *StoreInst, int64_t StDispImm, int64_t Offset);
+ void buildCopy(MachineInstr *LoadInst, unsigned NLoadOpcode, int64_t LoadDisp,
+ MachineInstr *StoreInst, unsigned NStoreOpcode,
+ int64_t StoreDisp, unsigned Size, int64_t Offset);
+};
+
+} // end anonymous namespace
+
+char LoongArchAvoidSFB::ID = 0;
+
+INITIALIZE_PASS_BEGIN(LoongArchAvoidSFB, DEBUG_TYPE, LOONGARCH_AVOID_SFB_NAME,
+ false, false)
+INITIALIZE_PASS_DEPENDENCY(AAResultsWrapperPass)
+INITIALIZE_PASS_DEPENDENCY(MachineLoopInfoWrapperPass)
+INITIALIZE_PASS_END(LoongArchAvoidSFB, DEBUG_TYPE, LOONGARCH_AVOID_SFB_NAME,
+ false, false)
+
+FunctionPass *llvm::createLoongArchAvoidStoreForwardingBlocksPass() {
+ return new LoongArchAvoidSFB();
+}
+
+static bool isPotentialBlockedMemCpyLd(unsigned Opcode) {
+ return Opcode == LoongArch::VLD || Opcode == LoongArch::XVLD ||
+ Opcode == LoongArch::LD_D || Opcode == LoongArch::FLD_D;
+}
+
+static bool isPotentialBlockedMemCpyPair(unsigned LdOpcode, unsigned StOpcode) {
+ switch (LdOpcode) {
+ case LoongArch::VLD:
+ return StOpcode == LoongArch::VST;
+ case LoongArch::XVLD:
+ return StOpcode == LoongArch::XVST;
+ case LoongArch::LD_D:
+ return StOpcode == LoongArch::ST_D;
+ case LoongArch::FLD_D:
+ return StOpcode == LoongArch::FST_D;
+ default:
+ return false;
+ }
+}
+
+static bool isPotentialBlockingStoreInst(unsigned Opcode, unsigned LoadOpcode) {
+ bool PBlock = false;
+ PBlock |= Opcode == LoongArch::ST_D || Opcode == LoongArch::ST_W ||
+ Opcode == LoongArch::ST_H || Opcode == LoongArch::ST_B ||
+ Opcode == LoongArch::STPTR_D || Opcode == LoongArch::STPTR_W ||
+ Opcode == LoongArch::FST_D || Opcode == LoongArch::FST_S;
+ if (LoadOpcode == LoongArch::XVLD)
+ PBlock |= Opcode == LoongArch::VST;
+ return PBlock;
+}
+
+static const int MOV256SZ = 32;
+static const int MOV128SZ = 16;
+static const int MOV64SZ = 8;
+static const int MOV32SZ = 4;
+static const int MOV16SZ = 2;
+static const int MOV8SZ = 1;
+
+static unsigned getRegSizeInBytes(const MachineInstr *LoadInst) {
+ switch (LoadInst->getOpcode()) {
+ case LoongArch::XVLD:
+ return MOV256SZ;
+ case LoongArch::VLD:
+ return MOV128SZ;
+ default:
+ return MOV64SZ;
+ }
+}
+
+// All instructions handled here use the (reg, base, imm) operand layout.
+static MachineOperand &getBaseOperand(MachineInstr *MI) {
+ return MI->getOperand(1);
+}
+
+static MachineOperand &getDispOperand(MachineInstr *MI) {
+ return MI->getOperand(2);
+}
+
+// Relevant addressing modes contain only base register and immediate
+// displacement or frameindex and immediate displacement.
+static bool isRelevantAddressingMode(MachineInstr *MI) {
+ const MachineOperand &Base = getBaseOperand(MI);
+ const MachineOperand &Disp = getDispOperand(MI);
+ return (Base.isReg() || Base.isFI()) && Disp.isImm();
+}
+
+// Every piece of the broken copy must still be encodable as reg+simm12.
+static bool hasEncodableDisps(MachineInstr *MI, unsigned Size) {
+ int64_t Disp = getDispOperand(MI).getImm();
+ return isInt<12>(Disp) && isInt<12>(Disp + Size - 1);
+}
+
+// Collect instructions executed shortly before LoadInst: those before it in
+// its block and in its predecessors and, if it is in a loop, those after it in
+// its block (previous iteration). Stop at calls and after SFBInspectionLimit
+// instructions, beyond which a blocked store has likely completed.
+SmallVector<LoongArchAvoidSFB::PotentialBlocker, 4>
+LoongArchAvoidSFB::findPotentialBlockers(MachineInstr *LoadInst) {
+ SmallVector<PotentialBlocker, 4> PotentialBlockers;
+ MachineBasicBlock *MBB = LoadInst->getParent();
+ MachineLoop *L = MLI->getLoopFor(MBB);
+ unsigned InspectionLimit = SFBInspectionLimit;
+
+ // Scan a range of instructions backwards; return false on a call.
+ auto ScanBackwards = [&](auto Begin, auto End, unsigned Limit,
+ bool ViaBackEdge, unsigned &Count) {
+ for (auto I = Begin; I != End; ++I) {
+ if (I->isMetaInstruction())
+ continue;
+ if (++Count >= Limit)
+ break;
+ if (I->isCall())
+ return false;
+ PotentialBlockers.push_back({&*I, ViaBackEdge});
+ }
+ return true;
+ };
+
+ unsigned BlockCount = 0;
+ if (!ScanBackwards(std::next(MachineBasicBlock::reverse_iterator(LoadInst)),
+ MBB->rend(), InspectionLimit, false, BlockCount))
+ return PotentialBlockers;
+ // If we didn't get to the instructions limit try predecessing blocks.
+ if (BlockCount >= InspectionLimit)
+ return PotentialBlockers;
+ unsigned LimitLeft = InspectionLimit - BlockCount;
+
+ if (L) {
+ unsigned Count = 0;
+ ScanBackwards(MBB->rbegin(), MachineBasicBlock::reverse_iterator(LoadInst),
+ LimitLeft, true, Count);
+ }
+ for (MachineBasicBlock *PMBB : MBB->predecessors()) {
+ // The tail of MBB itself was handled above.
+ if (PMBB == MBB)
+ continue;
+ bool ViaBackEdge = L && L->getHeader() == MBB && L->contains(PMBB);
+ unsigned Count = 0;
+ ScanBackwards(PMBB->rbegin(), PMBB->rend(), LimitLeft, ViaBackEdge, Count);
+ }
+ return PotentialBlockers;
+}
+
+// Whether the store may write bytes that the load reads in a later iteration.
+// Also require a common underlying object, so that copies between different
+// buffers keep using vector accesses.
+bool LoongArchAvoidSFB::mayAliasAcrossIterations(
+ const MachineMemOperand &LdMMO, const MachineMemOperand &StMMO) const {
+ const Value *LdVal = LdMMO.getValue();
+ const Value *StVal = StMMO.getValue();
+ if (!LdVal || !StVal)
+ return false;
+ if (AA->isNoAlias(MemoryLocation::getBeforeOrAfter(LdVal, LdMMO.getAAInfo()),
+ MemoryLocation::getBeforeOrAfter(StVal, StMMO.getAAInfo())))
+ return false;
+ // Look through loop header phis.
+ SmallVector<const Value *, 4> LdObjs, StObjs;
+ getUnderlyingObjects(LdVal, LdObjs);
+ getUnderlyingObjects(StVal, StObjs);
+ SmallPtrSet<const Value *, 4> LdSet(llvm::from_range, LdObjs);
+ return any_of(StObjs, [&](const Value *V) { return LdSet.contains(V); });
+}
+
+void LoongArchAvoidSFB::buildCopy(MachineInstr *LoadInst, unsigned NLoadOpcode,
+ int64_t LoadDisp, MachineInstr *StoreInst,
+ unsigned NStoreOpcode, int64_t StoreDisp,
+ unsigned Size, int64_t Offset) {
+ MachineOperand &LoadBase = getBaseOperand(LoadInst);
+ MachineOperand &StoreBase = getBaseOperand(StoreInst);
+ MachineBasicBlock *MBB = LoadInst->getParent();
+ MachineFunction *MF = MBB->getParent();
+ MachineMemOperand *LMMO = *LoadInst->memoperands_begin();
+ MachineMemOperand *SMMO = *StoreInst->memoperands_begin();
+
+ const TargetRegisterClass *RC = NLoadOpcode == LoongArch::VLD
+ ? &LoongArch::LSX128RegClass
+ : &LoongArch::GPRRegClass;
+ Register Reg = MRI->createVirtualRegister(RC);
+ // Loads go to the original load and stores to the original store, which
+ // keeps overlapping copies correct.
+ MachineInstr *NewLoad =
+ BuildMI(*MBB, LoadInst, LoadInst->getDebugLoc(), TII->get(NLoadOpcode),
+ Reg)
+ .add(LoadBase)
+ .addImm(LoadDisp)
+ .addMemOperand(MF->getMachineMemOperand(LMMO, Offset, Size));
+ if (LoadBase.isReg())
+ getBaseOperand(NewLoad).setIsKill(false);
+ LLVM_DEBUG(NewLoad->dump());
+ MachineInstr *NewStore =
+ BuildMI(*MBB, StoreInst, StoreInst->getDebugLoc(), TII->get(NStoreOpcode))
+ .addReg(Reg, RegState::Kill)
+ .add(StoreBase)
+ .addImm(StoreDisp)
+ .addMemOperand(MF->getMachineMemOperand(SMMO, Offset, Size));
+ if (StoreBase.isReg())
+ getBaseOperand(NewStore).setIsKill(false);
+ LLVM_DEBUG(NewStore->dump());
+}
+
+void LoongArchAvoidSFB::buildCopies(int Size, MachineInstr *LoadInst,
+ int64_t LdDispImm, MachineInstr *StoreInst,
+ int64_t StDispImm, int64_t Offset) {
+ static const struct {
+ int Size;
+ unsigned LoadOpc, StoreOpc;
+ } Pieces[] = {{MOV128SZ, LoongArch::VLD, LoongArch::VST},
+ {MOV64SZ, LoongArch::LD_D, LoongArch::ST_D},
+ {MOV32SZ, LoongArch::LD_W, LoongArch::ST_W},
+ {MOV16SZ, LoongArch::LD_H, LoongArch::ST_H},
+ {MOV8SZ, LoongArch::LD_B, LoongArch::ST_B}};
+ int64_t LdDisp = LdDispImm;
+ int64_t StDisp = StDispImm;
+ while (Size > 0) {
+ for (const auto &P : Pieces) {
+ // 128-bit pieces are only used to break 256-bit copies.
+ if (P.Size == MOV128SZ && LoadInst->getOpcode() != LoongArch::XVLD)
+ continue;
+ if (Size < P.Size)
+ continue;
+ buildCopy(LoadInst, P.LoadOpc, LdDisp, StoreInst, P.StoreOpc, StDisp,
+ P.Size, Offset);
+ Size -= P.Size;
+ LdDisp += P.Size;
+ StDisp += P.Size;
+ Offset += P.Size;
+ break;
+ }
+ }
+ assert(Size == 0 && "Wrong size division");
+}
+
+void LoongArchAvoidSFB::findPotentiallyBlockedCopies(MachineFunction &MF) {
+ for (auto &MBB : MF)
+ for (auto &MI : MBB) {
+ if (!isPotentialBlockedMemCpyLd(MI.getOpcode()))
+ continue;
+ Register DefVR = MI.getOperand(0).getReg();
+ if (!MRI->hasOneNonDBGUse(DefVR))
+ continue;
+ MachineInstr &StoreMI = *MRI->use_instr_nodbg_begin(DefVR);
+ if (StoreMI.getParent() != MI.getParent() ||
+ !isPotentialBlockedMemCpyPair(MI.getOpcode(), StoreMI.getOpcode()) ||
+ StoreMI.getOperand(0).getReg() != DefVR ||
+ !isRelevantAddressingMode(&MI) ||
+ !isRelevantAddressingMode(&StoreMI) ||
+ !hasEncodableDisps(&MI, getRegSizeInBytes(&MI)) ||
+ !hasEncodableDisps(&StoreMI, getRegSizeInBytes(&MI)) ||
+ !MI.hasOneMemOperand() || !StoreMI.hasOneMemOperand())
+ continue;
+ // Don't split volatile or atomic accesses.
+ const MachineMemOperand *LMMO = *MI.memoperands_begin();
+ const MachineMemOperand *SMMO = *StoreMI.memoperands_begin();
+ if (LMMO->isVolatile() || LMMO->isAtomic() || SMMO->isVolatile() ||
+ SMMO->isAtomic())
+ continue;
+ BlockedLoadsStoresPairs.push_back(std::make_pair(&MI, &StoreMI));
+ }
+}
+
+void LoongArchAvoidSFB::breakBlockedCopies(
+ MachineInstr *LoadInst, MachineInstr *StoreInst,
+ const DisplacementSizeMap &BlockingStoresDispSizeMap) {
+ int64_t LdDispImm = getDispOperand(LoadInst).getImm();
+ int64_t StDispImm = getDispOperand(StoreInst).getImm();
+ int64_t Offset = 0;
+
+ int64_t LdDisp1 = LdDispImm;
+ int64_t LdDisp2 = 0;
+ int64_t StDisp1 = StDispImm;
+ int64_t StDisp2 = 0;
+ unsigned Size1 = 0;
+ unsigned Size2 = 0;
+ int64_t LdStDelta = StDispImm - LdDispImm;
+
+ for (auto DispSizePair : BlockingStoresDispSizeMap) {
+ LdDisp2 = DispSizePair.first;
+ StDisp2 = DispSizePair.first + LdStDelta;
+ Size2 = DispSizePair.second;
+ // Avoid copying overlapping areas.
+ if (LdDisp2 < LdDisp1) {
+ int OverlapDelta = LdDisp1 - LdDisp2;
+ LdDisp2 += OverlapDelta;
+ StDisp2 += OverlapDelta;
+ Size2 -= OverlapDelta;
+ }
+ Size1 = LdDisp2 - LdDisp1;
+
+ // Build a copy for the point until the current blocking store's
+ // displacement.
+ buildCopies(Size1, LoadInst, LdDisp1, StoreInst, StDisp1, Offset);
+ // Build a copy for the current blocking store.
+ buildCopies(Size2, LoadInst, LdDisp2, StoreInst, StDisp2, Offset + Size1);
+ LdDisp1 = LdDisp2 + Size2;
+ StDisp1 = StDisp2 + Size2;
+ Offset += Size1 + Size2;
+ }
+ unsigned Size3 = (LdDispImm + getRegSizeInBytes(LoadInst)) - LdDisp1;
+ buildCopies(Size3, LoadInst, LdDisp1, StoreInst, StDisp1, Offset);
+}
+
+static bool hasSameBaseOpValue(MachineInstr *LoadInst,
+ MachineInstr *StoreInst) {
+ const MachineOperand &LoadBase = getBaseOperand(LoadInst);
+ const MachineOperand &StoreBase = getBaseOperand(StoreInst);
+ if (LoadBase.isReg() != StoreBase.isReg())
+ return false;
+ if (LoadBase.isReg())
+ return LoadBase.getReg() == StoreBase.getReg();
+ return LoadBase.getIndex() == StoreBase.getIndex();
+}
+
+static bool isBlockingStore(int64_t LoadDispImm, unsigned LoadSize,
+ int64_t StoreDispImm, unsigned StoreSize) {
+ return ((StoreDispImm >= LoadDispImm) &&
+ (StoreDispImm <= LoadDispImm + (LoadSize - StoreSize)));
+}
+
+// Keep track of all stores blocking a load
+static void
+updateBlockingStoresDispSizeMap(DisplacementSizeMap &BlockingStoresDispSizeMap,
+ int64_t DispImm, unsigned Size) {
+ auto [It, Inserted] = BlockingStoresDispSizeMap.try_emplace(DispImm, Size);
+ // Choose the smallest blocking store starting at this displacement.
+ if (!Inserted && It->second > Size)
+ It->second = Size;
+}
+
+// Remove blocking stores contained in each other.
+static void
+removeRedundantBlockingStores(DisplacementSizeMap &BlockingStoresDispSizeMap) {
+ if (BlockingStoresDispSizeMap.size() <= 1)
+ return;
+
+ SmallVector<std::pair<int64_t, unsigned>, 0> DispSizeStack;
+ for (auto DispSizePair : BlockingStoresDispSizeMap) {
+ int64_t CurrDisp = DispSizePair.first;
+ unsigned CurrSize = DispSizePair.second;
+ while (DispSizeStack.size()) {
+ int64_t PrevDisp = DispSizeStack.back().first;
+ unsigned PrevSize = DispSizeStack.back().second;
+ if (CurrDisp + CurrSize > PrevDisp + PrevSize)
+ break;
+ DispSizeStack.pop_back();
+ }
+ DispSizeStack.push_back(DispSizePair);
+ }
+ BlockingStoresDispSizeMap.clear();
+ for (auto Disp : DispSizeStack)
+ BlockingStoresDispSizeMap.insert(Disp);
+}
+
+bool LoongArchAvoidSFB::runOnMachineFunction(MachineFunction &MF) {
+ if (skipFunction(MF.getFunction()) || MF.getFunction().hasOptSize())
+ return false;
+
+ const auto &ST = MF.getSubtarget<LoongArchSubtarget>();
+ if (DisableAvoidSFB || !ST.is64Bit())
+ return false;
+
+ MRI = &MF.getRegInfo();
+ assert(MRI->isSSA() && "Expected MIR to be in SSA form");
+ TII = ST.getInstrInfo();
+ AA = &getAnalysis<AAResultsWrapperPass>().getAAResults();
+ MLI = &getAnalysis<MachineLoopInfoWrapperPass>().getLI();
+ LLVM_DEBUG(dbgs() << "Start LoongArchAvoidStoreForwardBlocks\n";);
+ // Look for a load then a store which look like a memcpy.
+ findPotentiallyBlockedCopies(MF);
+
+ bool Changed = false;
+ for (auto LoadStoreInstPair : BlockedLoadsStoresPairs) {
+ MachineInstr *LoadInst = LoadStoreInstPair.first;
+ MachineInstr *StoreInst = LoadStoreInstPair.second;
+ int64_t LdDispImm = getDispOperand(LoadInst).getImm();
+ unsigned LdSize = getRegSizeInBytes(LoadInst);
+ const MachineMemOperand &LMMO = **LoadInst->memoperands_begin();
+ DisplacementSizeMap BlockingStoresDispSizeMap;
+ bool LoopCarriedBlock = false;
+
+ for (auto [PBInst, ViaBackEdge] : findPotentialBlockers(LoadInst)) {
+ if (!PBInst->mayStore() || !PBInst->hasOneMemOperand())
+ continue;
+ const MachineMemOperand &PBMMO = **PBInst->memoperands_begin();
+ if (isPotentialBlockingStoreInst(PBInst->getOpcode(),
+ LoadInst->getOpcode()) &&
+ isRelevantAddressingMode(PBInst) &&
+ hasSameBaseOpValue(LoadInst, PBInst)) {
+ // Known displacement: a narrower store inside the loaded bytes.
+ int64_t PBstDispImm = getDispOperand(PBInst).getImm();
+ unsigned PBstSize = PBMMO.getSize().getValue();
+ if (!ViaBackEdge && PBstSize < LdSize &&
+ isBlockingStore(LdDispImm, LdSize, PBstDispImm, PBstSize))
+ updateBlockingStoresDispSizeMap(BlockingStoresDispSizeMap,
+ PBstDispImm, PBstSize);
+ continue;
+ }
+ // Unknown displacement: a store of the previous iteration to the same
+ // object, wide enough to forward to 8-byte pieces.
+ if (ViaBackEdge && LdSize > MOV64SZ && PBMMO.getSize().hasValue() &&
+ PBMMO.getSize().getValue() >= MOV64SZ &&
+ mayAliasAcrossIterations(LMMO, PBMMO))
+ LoopCarriedBlock = true;
+ }
+
+ if (LoopCarriedBlock) {
+ // The store may hit any part of the load: copy 8 bytes at a time.
+ BlockingStoresDispSizeMap.clear();
+ for (unsigned I = 0; I < LdSize; I += MOV64SZ)
+ BlockingStoresDispSizeMap[LdDispImm + I] = MOV64SZ;
+ }
+
+ if (BlockingStoresDispSizeMap.empty())
+ continue;
+
+ // Break the copy around the blocking stores.
+ LLVM_DEBUG(dbgs() << "Blocked load and store instructions: \n");
+ LLVM_DEBUG(LoadInst->dump());
+ LLVM_DEBUG(StoreInst->dump());
+ LLVM_DEBUG(dbgs() << "Replaced with:\n");
+ removeRedundantBlockingStores(BlockingStoresDispSizeMap);
+ breakBlockedCopies(LoadInst, StoreInst, BlockingStoresDispSizeMap);
+ // The base registers are read again by the new instructions.
+ for (MachineInstr *MI : {LoadInst, StoreInst})
+ if (getBaseOperand(MI).isReg())
+ MRI->clearKillFlags(getBaseOperand(MI).getReg());
+ ForRemoval.push_back(LoadInst);
+ ForRemoval.push_back(StoreInst);
+ ++NumBrokenCopies;
+ if (LdSize == MOV64SZ)
+ ++NumBrokenScalarCopies;
+ if (LoopCarriedBlock)
+ ++NumLoopCarriedBlocks;
+ Changed = true;
+ }
+ for (auto *RemovedInst : ForRemoval)
+ RemovedInst->eraseFromParent();
+ ForRemoval.clear();
+ BlockedLoadsStoresPairs.clear();
+ LLVM_DEBUG(dbgs() << "End LoongArchAvoidStoreForwardBlocks\n";);
+
+ return Changed;
+}
diff --git a/llvm/lib/Target/LoongArch/LoongArchTargetMachine.cpp b/llvm/lib/Target/LoongArch/LoongArchTargetMachine.cpp
index bb3825392a1b7b..519ad4ece33cda 100644
--- a/llvm/lib/Target/LoongArch/LoongArchTargetMachine.cpp
+++ b/llvm/lib/Target/LoongArch/LoongArchTargetMachine.cpp
@@ -36,6 +36,7 @@ LLVMInitializeLoongArchTarget() {
RegisterTargetMachine<LoongArchTargetMachine> X(getTheLoongArch32Target());
RegisterTargetMachine<LoongArchTargetMachine> Y(getTheLoongArch64Target());
auto *PR = PassRegistry::getPassRegistry();
+ initializeLoongArchAvoidSFBPass(*PR);
initializeLoongArchDeadRegisterDefinitionsPass(*PR);
initializeLoongArchMemoryBarrierOptPass(*PR);
initializeLoongArchMergeBaseOffsetOptPass(*PR);
@@ -219,6 +220,8 @@ void LoongArchPassConfig::addPreRegAlloc() {
addPass(createLoongArchPreRAExpandPseudoPass());
if (TM->getOptLevel() != CodeGenOptLevel::None && EnableMergeBaseOffset)
addPass(createLoongArchMergeBaseOffsetOptPass());
+ if (TM->getOptLevel() != CodeGenOptLevel::None)
+ addPass(createLoongArchAvoidStoreForwardingBlocksPass());
}
bool LoongArchPassConfig::addRegAssignAndRewriteFast() {
diff --git a/llvm/test/CodeGen/LoongArch/avoid-sfb.ll b/llvm/test/CodeGen/LoongArch/avoid-sfb.ll
index 795d057a3c4744..68158af8a47a2f 100644
--- a/llvm/test/CodeGen/LoongArch/avoid-sfb.ll
+++ b/llvm/test/CodeGen/LoongArch/avoid-sfb.ll
@@ -1,6 +1,8 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
; RUN: llc --mtriple=loongarch64 --mattr=+lsx < %s | FileCheck %s --check-prefix=LSX
; RUN: llc --mtriple=loongarch64 --mattr=+lasx < %s | FileCheck %s --check-prefix=LASX
+; RUN: llc --mtriple=loongarch64 --mattr=+lsx --loongarch-disable-avoid-sfb < %s \
+; RUN: | FileCheck %s --check-prefix=DISABLE
;; Small memcpy/memmove are lowered to ld.d/st.d, fld.d/fst.d, vld/vst or
;; xvld/xvst copies. A copy whose load reads data from in-flight narrower
@@ -18,8 +20,10 @@ define ptr @pattern_extend(ptr %dst, i64 %diff0) {
; LSX-NEXT: .p2align 4, , 16
; LSX-NEXT: .LBB0_1: # %loop
; LSX-NEXT: # =>This Inner Loop Header: Depth=1
-; LSX-NEXT: vld $vr0, $a2, 0
-; LSX-NEXT: vst $vr0, $a0, 0
+; LSX-NEXT: ld.d $a4, $a2, 0
+; LSX-NEXT: ld.d $a5, $a2, 8
+; LSX-NEXT: st.d $a4, $a0, 0
+; LSX-NEXT: st.d $a5, $a0, 8
; LSX-NEXT: add.d $a0, $a0, $a1
; LSX-NEXT: sub.d $a1, $a0, $a2
; LSX-NEXT: bltu $a1, $a3, .LBB0_1
@@ -32,13 +36,29 @@ define ptr @pattern_extend(ptr %dst, i64 %diff0) {
; LASX-NEXT: .p2align 4, , 16
; LASX-NEXT: .LBB0_1: # %loop
; LASX-NEXT: # =>This Inner Loop Header: Depth=1
-; LASX-NEXT: vld $vr0, $a2, 0
-; LASX-NEXT: vst $vr0, $a0, 0
+; LASX-NEXT: ld.d $a4, $a2, 0
+; LASX-NEXT: ld.d $a5, $a2, 8
+; LASX-NEXT: st.d $a4, $a0, 0
+; LASX-NEXT: st.d $a5, $a0, 8
; LASX-NEXT: add.d $a0, $a0, $a1
; LASX-NEXT: sub.d $a1, $a0, $a2
; LASX-NEXT: bltu $a1, $a3, .LBB0_1
; LASX-NEXT: # %bb.2: # %exit
; LASX-NEXT: ret
+; DISABLE-LABEL: pattern_extend:
+; DISABLE: # %bb.0: # %entry
+; DISABLE-NEXT: sub.d $a2, $a0, $a1
+; DISABLE-NEXT: ori $a3, $zero, 16
+; DISABLE-NEXT: .p2align 4, , 16
+; DISABLE-NEXT: .LBB0_1: # %loop
+; DISABLE-NEXT: # =>This Inner Loop Header: Depth=1
+; DISABLE-NEXT: vld $vr0, $a2, 0
+; DISABLE-NEXT: vst $vr0, $a0, 0
+; DISABLE-NEXT: add.d $a0, $a0, $a1
+; DISABLE-NEXT: sub.d $a1, $a0, $a2
+; DISABLE-NEXT: bltu $a1, $a3, .LBB0_1
+; DISABLE-NEXT: # %bb.2: # %exit
+; DISABLE-NEXT: ret
entry:
%neg = sub i64 0, %diff0
%src = getelementptr inbounds i8, ptr %dst, i64 %neg
@@ -88,6 +108,19 @@ define void @overlap_copy(ptr %dst, i64 %offset, i64 %len) {
; LASX-NEXT: bltu $a3, $a2, .LBB1_1
; LASX-NEXT: # %bb.2: # %exit
; LASX-NEXT: ret
+; DISABLE-LABEL: overlap_copy:
+; DISABLE: # %bb.0: # %entry
+; DISABLE-NEXT: move $a3, $zero
+; DISABLE-NEXT: sub.d $a1, $a0, $a1
+; DISABLE-NEXT: .p2align 4, , 16
+; DISABLE-NEXT: .LBB1_1: # %loop
+; DISABLE-NEXT: # =>This Inner Loop Header: Depth=1
+; DISABLE-NEXT: vldx $vr0, $a1, $a3
+; DISABLE-NEXT: vstx $vr0, $a0, $a3
+; DISABLE-NEXT: addi.d $a3, $a3, 16
+; DISABLE-NEXT: bltu $a3, $a2, .LBB1_1
+; DISABLE-NEXT: # %bb.2: # %exit
+; DISABLE-NEXT: ret
entry:
%neg = sub i64 0, %offset
%src = getelementptr inbounds i8, ptr %dst, i64 %neg
@@ -117,8 +150,10 @@ define void @overlap_copy_imm(ptr %buf, i64 %n) {
; LSX-NEXT: .p2align 4, , 16
; LSX-NEXT: .LBB2_1: # %loop
; LSX-NEXT: # =>This Inner Loop Header: Depth=1
-; LSX-NEXT: vld $vr0, $a0, -24
-; LSX-NEXT: vst $vr0, $a0, 0
+; LSX-NEXT: ld.d $a3, $a0, -24
+; LSX-NEXT: ld.d $a4, $a0, -16
+; LSX-NEXT: st.d $a3, $a0, 0
+; LSX-NEXT: st.d $a4, $a0, 8
; LSX-NEXT: addi.d $a2, $a2, 1
; LSX-NEXT: addi.d $a0, $a0, 16
; LSX-NEXT: bltu $a2, $a1, .LBB2_1
@@ -131,13 +166,29 @@ define void @overlap_copy_imm(ptr %buf, i64 %n) {
; LASX-NEXT: .p2align 4, , 16
; LASX-NEXT: .LBB2_1: # %loop
; LASX-NEXT: # =>This Inner Loop Header: Depth=1
-; LASX-NEXT: vld $vr0, $a0, -24
-; LASX-NEXT: vst $vr0, $a0, 0
+; LASX-NEXT: ld.d $a3, $a0, -24
+; LASX-NEXT: ld.d $a4, $a0, -16
+; LASX-NEXT: st.d $a3, $a0, 0
+; LASX-NEXT: st.d $a4, $a0, 8
; LASX-NEXT: addi.d $a2, $a2, 1
; LASX-NEXT: addi.d $a0, $a0, 16
; LASX-NEXT: bltu $a2, $a1, .LBB2_1
; LASX-NEXT: # %bb.2: # %exit
; LASX-NEXT: ret
+; DISABLE-LABEL: overlap_copy_imm:
+; DISABLE: # %bb.0: # %entry
+; DISABLE-NEXT: move $a2, $zero
+; DISABLE-NEXT: addi.d $a0, $a0, 32
+; DISABLE-NEXT: .p2align 4, , 16
+; DISABLE-NEXT: .LBB2_1: # %loop
+; DISABLE-NEXT: # =>This Inner Loop Header: Depth=1
+; DISABLE-NEXT: vld $vr0, $a0, -24
+; DISABLE-NEXT: vst $vr0, $a0, 0
+; DISABLE-NEXT: addi.d $a2, $a2, 1
+; DISABLE-NEXT: addi.d $a0, $a0, 16
+; DISABLE-NEXT: bltu $a2, $a1, .LBB2_1
+; DISABLE-NEXT: # %bb.2: # %exit
+; DISABLE-NEXT: ret
entry:
br label %loop
@@ -183,6 +234,18 @@ define void @independent_copy(ptr noalias %dst, ptr noalias %src, i64 %len) {
; LASX-NEXT: bltu $a3, $a2, .LBB3_1
; LASX-NEXT: # %bb.2: # %exit
; LASX-NEXT: ret
+; DISABLE-LABEL: independent_copy:
+; DISABLE: # %bb.0: # %entry
+; DISABLE-NEXT: move $a3, $zero
+; DISABLE-NEXT: .p2align 4, , 16
+; DISABLE-NEXT: .LBB3_1: # %loop
+; DISABLE-NEXT: # =>This Inner Loop Header: Depth=1
+; DISABLE-NEXT: vldx $vr0, $a1, $a3
+; DISABLE-NEXT: vstx $vr0, $a0, $a3
+; DISABLE-NEXT: addi.d $a3, $a3, 16
+; DISABLE-NEXT: bltu $a3, $a2, .LBB3_1
+; DISABLE-NEXT: # %bb.2: # %exit
+; DISABLE-NEXT: ret
entry:
br label %loop
@@ -213,6 +276,12 @@ define void @straight_line(ptr %buf, i64 %offset) {
; LASX-NEXT: vld $vr0, $a1, 0
; LASX-NEXT: vst $vr0, $a0, 0
; LASX-NEXT: ret
+; DISABLE-LABEL: straight_line:
+; DISABLE: # %bb.0:
+; DISABLE-NEXT: sub.d $a1, $a0, $a1
+; DISABLE-NEXT: vld $vr0, $a1, 0
+; DISABLE-NEXT: vst $vr0, $a0, 0
+; DISABLE-NEXT: ret
%neg = sub i64 0, %offset
%src = getelementptr inbounds i8, ptr %buf, i64 %neg
tail call void @llvm.memcpy.p0.p0.i64(ptr align 1 %buf, ptr align 1 %src, i64 16, i1 false)
@@ -248,6 +317,19 @@ define void @offset_out_of_range(ptr %buf, i64 %n) {
; LASX-NEXT: bltu $a2, $a1, .LBB5_1
; LASX-NEXT: # %bb.2: # %exit
; LASX-NEXT: ret
+; DISABLE-LABEL: offset_out_of_range:
+; DISABLE: # %bb.0: # %entry
+; DISABLE-NEXT: move $a2, $zero
+; DISABLE-NEXT: .p2align 4, , 16
+; DISABLE-NEXT: .LBB5_1: # %loop
+; DISABLE-NEXT: # =>This Inner Loop Header: Depth=1
+; DISABLE-NEXT: vld $vr0, $a0, 2040
+; DISABLE-NEXT: vst $vr0, $a0, 0
+; DISABLE-NEXT: addi.d $a2, $a2, 1
+; DISABLE-NEXT: addi.d $a0, $a0, 16
+; DISABLE-NEXT: bltu $a2, $a1, .LBB5_1
+; DISABLE-NEXT: # %bb.2: # %exit
+; DISABLE-NEXT: ret
entry:
br label %loop
@@ -291,6 +373,18 @@ define void @overlap_copy_optsize(ptr %dst, i64 %offset, i64 %len) optsize {
; LASX-NEXT: bltu $a3, $a2, .LBB6_1
; LASX-NEXT: # %bb.2: # %exit
; LASX-NEXT: ret
+; DISABLE-LABEL: overlap_copy_optsize:
+; DISABLE: # %bb.0: # %entry
+; DISABLE-NEXT: move $a3, $zero
+; DISABLE-NEXT: sub.d $a1, $a0, $a1
+; DISABLE-NEXT: .LBB6_1: # %loop
+; DISABLE-NEXT: # =>This Inner Loop Header: Depth=1
+; DISABLE-NEXT: vldx $vr0, $a1, $a3
+; DISABLE-NEXT: vstx $vr0, $a0, $a3
+; DISABLE-NEXT: addi.d $a3, $a3, 16
+; DISABLE-NEXT: bltu $a3, $a2, .LBB6_1
+; DISABLE-NEXT: # %bb.2: # %exit
+; DISABLE-NEXT: ret
entry:
%neg = sub i64 0, %offset
%src = getelementptr inbounds i8, ptr %dst, i64 %neg
@@ -322,10 +416,14 @@ define void @conditional_store(ptr noalias %s1, ptr noalias %s2, i32 %x, ptr noa
; LSX-NEXT: # %bb.1: # %if.then
; LSX-NEXT: st.w $a2, $a0, 4
; LSX-NEXT: .LBB7_2: # %if.end
-; LSX-NEXT: vld $vr0, $a0, 0
-; LSX-NEXT: vld $vr1, $a4, 0
-; LSX-NEXT: vst $vr0, $a3, 0
-; LSX-NEXT: vst $vr1, $a1, 0
+; LSX-NEXT: ld.w $a2, $a0, 0
+; LSX-NEXT: ld.w $a5, $a0, 4
+; LSX-NEXT: ld.d $a0, $a0, 8
+; LSX-NEXT: vld $vr0, $a4, 0
+; LSX-NEXT: st.w $a2, $a3, 0
+; LSX-NEXT: st.w $a5, $a3, 4
+; LSX-NEXT: st.d $a0, $a3, 8
+; LSX-NEXT: vst $vr0, $a1, 0
; LSX-NEXT: ret
; LASX-LABEL: conditional_store:
; LASX: # %bb.0: # %entry
@@ -335,11 +433,28 @@ define void @conditional_store(ptr noalias %s1, ptr noalias %s2, i32 %x, ptr noa
; LASX-NEXT: # %bb.1: # %if.then
; LASX-NEXT: st.w $a2, $a0, 4
; LASX-NEXT: .LBB7_2: # %if.end
-; LASX-NEXT: vld $vr0, $a0, 0
-; LASX-NEXT: vld $vr1, $a4, 0
-; LASX-NEXT: vst $vr0, $a3, 0
-; LASX-NEXT: vst $vr1, $a1, 0
+; LASX-NEXT: ld.w $a2, $a0, 0
+; LASX-NEXT: ld.w $a5, $a0, 4
+; LASX-NEXT: ld.d $a0, $a0, 8
+; LASX-NEXT: vld $vr0, $a4, 0
+; LASX-NEXT: st.w $a2, $a3, 0
+; LASX-NEXT: st.w $a5, $a3, 4
+; LASX-NEXT: st.d $a0, $a3, 8
+; LASX-NEXT: vst $vr0, $a1, 0
; LASX-NEXT: ret
+; DISABLE-LABEL: conditional_store:
+; DISABLE: # %bb.0: # %entry
+; DISABLE-NEXT: addi.w $a5, $a2, 0
+; DISABLE-NEXT: ori $a6, $zero, 18
+; DISABLE-NEXT: blt $a5, $a6, .LBB7_2
+; DISABLE-NEXT: # %bb.1: # %if.then
+; DISABLE-NEXT: st.w $a2, $a0, 4
+; DISABLE-NEXT: .LBB7_2: # %if.end
+; DISABLE-NEXT: vld $vr0, $a0, 0
+; DISABLE-NEXT: vld $vr1, $a4, 0
+; DISABLE-NEXT: vst $vr0, $a3, 0
+; DISABLE-NEXT: vst $vr1, $a1, 0
+; DISABLE-NEXT: ret
entry:
%cmp = icmp sgt i32 %x, 17
br i1 %cmp, label %if.then, label %if.end
@@ -366,9 +481,11 @@ define void @conditional_store_32(ptr noalias %s1, ptr noalias %s2, i64 %x, i1 %
; LSX-NEXT: st.d $a2, $a0, 8
; LSX-NEXT: .LBB8_2: # %if.end
; LSX-NEXT: vld $vr0, $a0, 16
-; LSX-NEXT: vld $vr1, $a0, 0
+; LSX-NEXT: ld.d $a2, $a0, 0
+; LSX-NEXT: ld.d $a0, $a0, 8
; LSX-NEXT: vst $vr0, $a1, 16
-; LSX-NEXT: vst $vr1, $a1, 0
+; LSX-NEXT: st.d $a2, $a1, 0
+; LSX-NEXT: st.d $a0, $a1, 8
; LSX-NEXT: ret
; LASX-LABEL: conditional_store_32:
; LASX: # %bb.0: # %entry
@@ -377,9 +494,25 @@ define void @conditional_store_32(ptr noalias %s1, ptr noalias %s2, i64 %x, i1 %
; LASX-NEXT: # %bb.1: # %if.then
; LASX-NEXT: st.d $a2, $a0, 8
; LASX-NEXT: .LBB8_2: # %if.end
-; LASX-NEXT: xvld $xr0, $a0, 0
-; LASX-NEXT: xvst $xr0, $a1, 0
+; LASX-NEXT: ld.d $a2, $a0, 0
+; LASX-NEXT: ld.d $a3, $a0, 8
+; LASX-NEXT: vld $vr0, $a0, 16
+; LASX-NEXT: st.d $a2, $a1, 0
+; LASX-NEXT: st.d $a3, $a1, 8
+; LASX-NEXT: vst $vr0, $a1, 16
; LASX-NEXT: ret
+; DISABLE-LABEL: conditional_store_32:
+; DISABLE: # %bb.0: # %entry
+; DISABLE-NEXT: andi $a3, $a3, 1
+; DISABLE-NEXT: beqz $a3, .LBB8_2
+; DISABLE-NEXT: # %bb.1: # %if.then
+; DISABLE-NEXT: st.d $a2, $a0, 8
+; DISABLE-NEXT: .LBB8_2: # %if.end
+; DISABLE-NEXT: vld $vr0, $a0, 16
+; DISABLE-NEXT: vld $vr1, $a0, 0
+; DISABLE-NEXT: vst $vr0, $a1, 16
+; DISABLE-NEXT: vst $vr1, $a1, 0
+; DISABLE-NEXT: ret
entry:
br i1 %c, label %if.then, label %if.end
@@ -402,8 +535,12 @@ define void @fp_store_vector_copy(ptr noalias %s1, ptr noalias %s2, float %f, i1
; LSX-NEXT: # %bb.1: # %if.then
; LSX-NEXT: fst.s $fa0, $a0, 4
; LSX-NEXT: .LBB9_2: # %if.end
-; LSX-NEXT: vld $vr0, $a0, 0
-; LSX-NEXT: vst $vr0, $a1, 0
+; LSX-NEXT: ld.w $a2, $a0, 0
+; LSX-NEXT: ld.w $a3, $a0, 4
+; LSX-NEXT: ld.d $a0, $a0, 8
+; LSX-NEXT: st.w $a2, $a1, 0
+; LSX-NEXT: st.w $a3, $a1, 4
+; LSX-NEXT: st.d $a0, $a1, 8
; LSX-NEXT: ret
; LASX-LABEL: fp_store_vector_copy:
; LASX: # %bb.0: # %entry
@@ -412,9 +549,23 @@ define void @fp_store_vector_copy(ptr noalias %s1, ptr noalias %s2, float %f, i1
; LASX-NEXT: # %bb.1: # %if.then
; LASX-NEXT: fst.s $fa0, $a0, 4
; LASX-NEXT: .LBB9_2: # %if.end
-; LASX-NEXT: vld $vr0, $a0, 0
-; LASX-NEXT: vst $vr0, $a1, 0
+; LASX-NEXT: ld.w $a2, $a0, 0
+; LASX-NEXT: ld.w $a3, $a0, 4
+; LASX-NEXT: ld.d $a0, $a0, 8
+; LASX-NEXT: st.w $a2, $a1, 0
+; LASX-NEXT: st.w $a3, $a1, 4
+; LASX-NEXT: st.d $a0, $a1, 8
; LASX-NEXT: ret
+; DISABLE-LABEL: fp_store_vector_copy:
+; DISABLE: # %bb.0: # %entry
+; DISABLE-NEXT: andi $a2, $a2, 1
+; DISABLE-NEXT: beqz $a2, .LBB9_2
+; DISABLE-NEXT: # %bb.1: # %if.then
+; DISABLE-NEXT: fst.s $fa0, $a0, 4
+; DISABLE-NEXT: .LBB9_2: # %if.end
+; DISABLE-NEXT: vld $vr0, $a0, 0
+; DISABLE-NEXT: vst $vr0, $a1, 0
+; DISABLE-NEXT: ret
entry:
br i1 %c, label %if.then, label %if.end
@@ -436,8 +587,10 @@ define void @fp_double_store_vector_copy(ptr noalias %s1, ptr noalias %s2, doubl
; LSX-NEXT: # %bb.1: # %if.then
; LSX-NEXT: fst.d $fa0, $a0, 8
; LSX-NEXT: .LBB10_2: # %if.end
-; LSX-NEXT: vld $vr0, $a0, 0
-; LSX-NEXT: vst $vr0, $a1, 0
+; LSX-NEXT: ld.d $a2, $a0, 0
+; LSX-NEXT: ld.d $a0, $a0, 8
+; LSX-NEXT: st.d $a2, $a1, 0
+; LSX-NEXT: st.d $a0, $a1, 8
; LSX-NEXT: ret
; LASX-LABEL: fp_double_store_vector_copy:
; LASX: # %bb.0: # %entry
@@ -446,9 +599,21 @@ define void @fp_double_store_vector_copy(ptr noalias %s1, ptr noalias %s2, doubl
; LASX-NEXT: # %bb.1: # %if.then
; LASX-NEXT: fst.d $fa0, $a0, 8
; LASX-NEXT: .LBB10_2: # %if.end
-; LASX-NEXT: vld $vr0, $a0, 0
-; LASX-NEXT: vst $vr0, $a1, 0
+; LASX-NEXT: ld.d $a2, $a0, 0
+; LASX-NEXT: ld.d $a0, $a0, 8
+; LASX-NEXT: st.d $a2, $a1, 0
+; LASX-NEXT: st.d $a0, $a1, 8
; LASX-NEXT: ret
+; DISABLE-LABEL: fp_double_store_vector_copy:
+; DISABLE: # %bb.0: # %entry
+; DISABLE-NEXT: andi $a2, $a2, 1
+; DISABLE-NEXT: beqz $a2, .LBB10_2
+; DISABLE-NEXT: # %bb.1: # %if.then
+; DISABLE-NEXT: fst.d $fa0, $a0, 8
+; DISABLE-NEXT: .LBB10_2: # %if.end
+; DISABLE-NEXT: vld $vr0, $a0, 0
+; DISABLE-NEXT: vst $vr0, $a1, 0
+; DISABLE-NEXT: ret
entry:
br i1 %c, label %if.then, label %if.end
@@ -471,8 +636,10 @@ define void @int_store_scalar_copy(ptr noalias %s1, ptr noalias %s2, i32 %x, i1
; LSX-NEXT: # %bb.1: # %if.then
; LSX-NEXT: st.w $a2, $a0, 4
; LSX-NEXT: .LBB11_2: # %if.end
-; LSX-NEXT: ld.d $a0, $a0, 0
-; LSX-NEXT: st.d $a0, $a1, 0
+; LSX-NEXT: ld.w $a2, $a0, 0
+; LSX-NEXT: ld.w $a0, $a0, 4
+; LSX-NEXT: st.w $a2, $a1, 0
+; LSX-NEXT: st.w $a0, $a1, 4
; LSX-NEXT: ret
; LASX-LABEL: int_store_scalar_copy:
; LASX: # %bb.0: # %entry
@@ -481,9 +648,21 @@ define void @int_store_scalar_copy(ptr noalias %s1, ptr noalias %s2, i32 %x, i1
; LASX-NEXT: # %bb.1: # %if.then
; LASX-NEXT: st.w $a2, $a0, 4
; LASX-NEXT: .LBB11_2: # %if.end
-; LASX-NEXT: ld.d $a0, $a0, 0
-; LASX-NEXT: st.d $a0, $a1, 0
+; LASX-NEXT: ld.w $a2, $a0, 0
+; LASX-NEXT: ld.w $a0, $a0, 4
+; LASX-NEXT: st.w $a2, $a1, 0
+; LASX-NEXT: st.w $a0, $a1, 4
; LASX-NEXT: ret
+; DISABLE-LABEL: int_store_scalar_copy:
+; DISABLE: # %bb.0: # %entry
+; DISABLE-NEXT: andi $a3, $a3, 1
+; DISABLE-NEXT: beqz $a3, .LBB11_2
+; DISABLE-NEXT: # %bb.1: # %if.then
+; DISABLE-NEXT: st.w $a2, $a0, 4
+; DISABLE-NEXT: .LBB11_2: # %if.end
+; DISABLE-NEXT: ld.d $a0, $a0, 0
+; DISABLE-NEXT: st.d $a0, $a1, 0
+; DISABLE-NEXT: ret
entry:
br i1 %c, label %if.then, label %if.end
@@ -505,8 +684,14 @@ define void @byte_store_scalar_copy(ptr noalias %s1, ptr noalias %s2, i8 %x, i1
; LSX-NEXT: # %bb.1: # %if.then
; LSX-NEXT: st.b $a2, $a0, 3
; LSX-NEXT: .LBB12_2: # %if.end
-; LSX-NEXT: ld.d $a0, $a0, 0
-; LSX-NEXT: st.d $a0, $a1, 0
+; LSX-NEXT: ld.h $a2, $a0, 0
+; LSX-NEXT: ld.b $a3, $a0, 2
+; LSX-NEXT: ld.b $a4, $a0, 3
+; LSX-NEXT: ld.w $a0, $a0, 4
+; LSX-NEXT: st.h $a2, $a1, 0
+; LSX-NEXT: st.b $a3, $a1, 2
+; LSX-NEXT: st.b $a4, $a1, 3
+; LSX-NEXT: st.w $a0, $a1, 4
; LSX-NEXT: ret
; LASX-LABEL: byte_store_scalar_copy:
; LASX: # %bb.0: # %entry
@@ -515,9 +700,25 @@ define void @byte_store_scalar_copy(ptr noalias %s1, ptr noalias %s2, i8 %x, i1
; LASX-NEXT: # %bb.1: # %if.then
; LASX-NEXT: st.b $a2, $a0, 3
; LASX-NEXT: .LBB12_2: # %if.end
-; LASX-NEXT: ld.d $a0, $a0, 0
-; LASX-NEXT: st.d $a0, $a1, 0
+; LASX-NEXT: ld.h $a2, $a0, 0
+; LASX-NEXT: ld.b $a3, $a0, 2
+; LASX-NEXT: ld.b $a4, $a0, 3
+; LASX-NEXT: ld.w $a0, $a0, 4
+; LASX-NEXT: st.h $a2, $a1, 0
+; LASX-NEXT: st.b $a3, $a1, 2
+; LASX-NEXT: st.b $a4, $a1, 3
+; LASX-NEXT: st.w $a0, $a1, 4
; LASX-NEXT: ret
+; DISABLE-LABEL: byte_store_scalar_copy:
+; DISABLE: # %bb.0: # %entry
+; DISABLE-NEXT: andi $a3, $a3, 1
+; DISABLE-NEXT: beqz $a3, .LBB12_2
+; DISABLE-NEXT: # %bb.1: # %if.then
+; DISABLE-NEXT: st.b $a2, $a0, 3
+; DISABLE-NEXT: .LBB12_2: # %if.end
+; DISABLE-NEXT: ld.d $a0, $a0, 0
+; DISABLE-NEXT: st.d $a0, $a1, 0
+; DISABLE-NEXT: ret
entry:
br i1 %c, label %if.then, label %if.end
@@ -540,8 +741,10 @@ define void @fp_store_fp_copy(ptr noalias %s1, ptr noalias %s2, float %f, i1 %c)
; LSX-NEXT: # %bb.1: # %if.then
; LSX-NEXT: fst.s $fa0, $a0, 0
; LSX-NEXT: .LBB13_2: # %if.end
-; LSX-NEXT: fld.d $fa0, $a0, 0
-; LSX-NEXT: fst.d $fa0, $a1, 0
+; LSX-NEXT: ld.w $a2, $a0, 0
+; LSX-NEXT: ld.w $a0, $a0, 4
+; LSX-NEXT: st.w $a2, $a1, 0
+; LSX-NEXT: st.w $a0, $a1, 4
; LSX-NEXT: ret
; LASX-LABEL: fp_store_fp_copy:
; LASX: # %bb.0: # %entry
@@ -550,9 +753,21 @@ define void @fp_store_fp_copy(ptr noalias %s1, ptr noalias %s2, float %f, i1 %c)
; LASX-NEXT: # %bb.1: # %if.then
; LASX-NEXT: fst.s $fa0, $a0, 0
; LASX-NEXT: .LBB13_2: # %if.end
-; LASX-NEXT: fld.d $fa0, $a0, 0
-; LASX-NEXT: fst.d $fa0, $a1, 0
+; LASX-NEXT: ld.w $a2, $a0, 0
+; LASX-NEXT: ld.w $a0, $a0, 4
+; LASX-NEXT: st.w $a2, $a1, 0
+; LASX-NEXT: st.w $a0, $a1, 4
; LASX-NEXT: ret
+; DISABLE-LABEL: fp_store_fp_copy:
+; DISABLE: # %bb.0: # %entry
+; DISABLE-NEXT: andi $a2, $a2, 1
+; DISABLE-NEXT: beqz $a2, .LBB13_2
+; DISABLE-NEXT: # %bb.1: # %if.then
+; DISABLE-NEXT: fst.s $fa0, $a0, 0
+; DISABLE-NEXT: .LBB13_2: # %if.end
+; DISABLE-NEXT: fld.d $fa0, $a0, 0
+; DISABLE-NEXT: fst.d $fa0, $a1, 0
+; DISABLE-NEXT: ret
entry:
br i1 %c, label %if.then, label %if.end
@@ -588,6 +803,16 @@ define void @same_size_store_scalar_copy(ptr noalias %s1, ptr noalias %s2, i64 %
; LASX-NEXT: ld.d $a0, $a0, 0
; LASX-NEXT: st.d $a0, $a1, 0
; LASX-NEXT: ret
+; DISABLE-LABEL: same_size_store_scalar_copy:
+; DISABLE: # %bb.0: # %entry
+; DISABLE-NEXT: andi $a3, $a3, 1
+; DISABLE-NEXT: beqz $a3, .LBB14_2
+; DISABLE-NEXT: # %bb.1: # %if.then
+; DISABLE-NEXT: st.d $a2, $a0, 0
+; DISABLE-NEXT: .LBB14_2: # %if.end
+; DISABLE-NEXT: ld.d $a0, $a0, 0
+; DISABLE-NEXT: st.d $a0, $a1, 0
+; DISABLE-NEXT: ret
entry:
br i1 %c, label %if.then, label %if.end
@@ -621,9 +846,23 @@ define void @vector_store_lasx_copy(ptr noalias %s1, ptr noalias %s2, <4 x i32>
; LASX-NEXT: # %bb.1: # %if.then
; LASX-NEXT: vst $vr0, $a0, 16
; LASX-NEXT: .LBB15_2: # %if.end
-; LASX-NEXT: xvld $xr0, $a0, 0
-; LASX-NEXT: xvst $xr0, $a1, 0
+; LASX-NEXT: vld $vr0, $a0, 0
+; LASX-NEXT: vld $vr1, $a0, 16
+; LASX-NEXT: vst $vr0, $a1, 0
+; LASX-NEXT: vst $vr1, $a1, 16
; LASX-NEXT: ret
+; DISABLE-LABEL: vector_store_lasx_copy:
+; DISABLE: # %bb.0: # %entry
+; DISABLE-NEXT: andi $a2, $a2, 1
+; DISABLE-NEXT: beqz $a2, .LBB15_2
+; DISABLE-NEXT: # %bb.1: # %if.then
+; DISABLE-NEXT: vst $vr0, $a0, 16
+; DISABLE-NEXT: .LBB15_2: # %if.end
+; DISABLE-NEXT: vld $vr0, $a0, 16
+; DISABLE-NEXT: vld $vr1, $a0, 0
+; DISABLE-NEXT: vst $vr0, $a1, 16
+; DISABLE-NEXT: vst $vr1, $a1, 0
+; DISABLE-NEXT: ret
entry:
br i1 %c, label %if.then, label %if.end
@@ -647,10 +886,14 @@ define void @overlap_copy_32(ptr %buf, i64 %n) {
; LSX-NEXT: .p2align 4, , 16
; LSX-NEXT: .LBB16_1: # %loop
; LSX-NEXT: # =>This Inner Loop Header: Depth=1
-; LSX-NEXT: vld $vr0, $a0, -24
-; LSX-NEXT: vld $vr1, $a0, -40
-; LSX-NEXT: vst $vr0, $a0, 16
-; LSX-NEXT: vst $vr1, $a0, 0
+; LSX-NEXT: ld.d $a3, $a0, -24
+; LSX-NEXT: ld.d $a4, $a0, -16
+; LSX-NEXT: ld.d $a5, $a0, -40
+; LSX-NEXT: ld.d $a6, $a0, -32
+; LSX-NEXT: st.d $a3, $a0, 16
+; LSX-NEXT: st.d $a4, $a0, 24
+; LSX-NEXT: st.d $a5, $a0, 0
+; LSX-NEXT: st.d $a6, $a0, 8
; LSX-NEXT: addi.d $a2, $a2, 1
; LSX-NEXT: addi.d $a0, $a0, 32
; LSX-NEXT: bltu $a2, $a1, .LBB16_1
@@ -663,13 +906,35 @@ define void @overlap_copy_32(ptr %buf, i64 %n) {
; LASX-NEXT: .p2align 4, , 16
; LASX-NEXT: .LBB16_1: # %loop
; LASX-NEXT: # =>This Inner Loop Header: Depth=1
-; LASX-NEXT: xvld $xr0, $a0, -40
-; LASX-NEXT: xvst $xr0, $a0, 0
+; LASX-NEXT: ld.d $a3, $a0, -40
+; LASX-NEXT: ld.d $a4, $a0, -32
+; LASX-NEXT: ld.d $a5, $a0, -24
+; LASX-NEXT: ld.d $a6, $a0, -16
+; LASX-NEXT: st.d $a3, $a0, 0
+; LASX-NEXT: st.d $a4, $a0, 8
+; LASX-NEXT: st.d $a5, $a0, 16
+; LASX-NEXT: st.d $a6, $a0, 24
; LASX-NEXT: addi.d $a2, $a2, 1
; LASX-NEXT: addi.d $a0, $a0, 32
; LASX-NEXT: bltu $a2, $a1, .LBB16_1
; LASX-NEXT: # %bb.2: # %exit
; LASX-NEXT: ret
+; DISABLE-LABEL: overlap_copy_32:
+; DISABLE: # %bb.0: # %entry
+; DISABLE-NEXT: move $a2, $zero
+; DISABLE-NEXT: addi.d $a0, $a0, 48
+; DISABLE-NEXT: .p2align 4, , 16
+; DISABLE-NEXT: .LBB16_1: # %loop
+; DISABLE-NEXT: # =>This Inner Loop Header: Depth=1
+; DISABLE-NEXT: vld $vr0, $a0, -24
+; DISABLE-NEXT: vld $vr1, $a0, -40
+; DISABLE-NEXT: vst $vr0, $a0, 16
+; DISABLE-NEXT: vst $vr1, $a0, 0
+; DISABLE-NEXT: addi.d $a2, $a2, 1
+; DISABLE-NEXT: addi.d $a0, $a0, 32
+; DISABLE-NEXT: bltu $a2, $a1, .LBB16_1
+; DISABLE-NEXT: # %bb.2: # %exit
+; DISABLE-NEXT: ret
entry:
br label %loop
diff --git a/llvm/test/CodeGen/LoongArch/opt-pipeline.ll b/llvm/test/CodeGen/LoongArch/opt-pipeline.ll
index 334dfa39081c0c..0f6f8cadf1aebd 100644
--- a/llvm/test/CodeGen/LoongArch/opt-pipeline.ll
+++ b/llvm/test/CodeGen/LoongArch/opt-pipeline.ll
@@ -113,6 +113,7 @@
; LA64-NEXT: LoongArch Optimize W Instructions
; LAXX-NEXT: LoongArch Pre-RA pseudo instruction expansion pass
; LAXX-NEXT: LoongArch Merge Base Offset
+; LAXX-NEXT: LoongArch Avoid Store Forwarding Blocks
; LAXX-NEXT: Detect Dead Lanes
; LAXX-NEXT: Init Undef Pass
; LAXX-NEXT: Process Implicit Definitions
diff --git a/llvm/utils/gn/secondary/llvm/lib/Target/LoongArch/BUILD.gn b/llvm/utils/gn/secondary/llvm/lib/Target/LoongArch/BUILD.gn
index ce2f84eea690b8..43b5ebcef6198c 100644
--- a/llvm/utils/gn/secondary/llvm/lib/Target/LoongArch/BUILD.gn
+++ b/llvm/utils/gn/secondary/llvm/lib/Target/LoongArch/BUILD.gn
@@ -41,6 +41,7 @@ static_library("LLVMLoongArchCodeGen") {
include_dirs = [ "." ]
sources = [
"LoongArchAsmPrinter.cpp",
+ "LoongArchAvoidStoreForwardingBlocks.cpp",
"LoongArchDeadRegisterDefinitions.cpp",
"LoongArchExpandAtomicPseudoInsts.cpp",
"LoongArchExpandPseudoInsts.cpp",
More information about the llvm-branch-commits
mailing list