[llvm] [Draft] Data flow based shrink wrapping (PR #191784)
via llvm-commits
llvm-commits at lists.llvm.org
Mon Apr 13 02:58:46 PDT 2026
llvmbot wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-backend-risc-v
Author: Elizaveta Noskova (enoskova-sc)
<details>
<summary>Changes</summary>
This patch introduces a new ShrinkWrapping pass that implements a data-flow-based shrink-wrapping optimization, originally described by Fred C. Chow in PLDI '88. Patch mainly consist of the code deleted many years ago from llvm (dbec9d9 removing commit hash) with the number of improvements, reducing conservativity of approach. The main improvement is reducing the conservativeness of the original approach by allowing the pass to split edges and insert new basic blocks when the existing CFG lacks optimal insertion points for save and restore instructions.
SPEC2006 (ref) runs on p550 show consistent improvements when replacing the current shrink-wrap implementation with the proposed pass:
| Benchmark | % Diff |
|-----------|----------------------------|
| **400.perlbench** | **‑1.20 %** |
| **401.bzip2** | **≈ 0 %** (all runs within ±0.01 %) |
| **403.gcc** | **‑1.77 %** |
| **429.mcf** | **≈ 0 %** |
| **445.gobmk** | **‑1.92 %** |
| **456.hmmer** | **‑0.06 %** |
| **458.sjeng** | **+0.02 %** |
| **462.libquantum**| **≈ 0 %** |
| **464.h264ref** | **‑0.09 %** |
| **471.omnetpp** | **‑0.45 %** |
| **473.astar** | **‑6.73 %** |
| **483.xalancbmk** | **+0.31 %** |
The purpose of this pull request is to discuss if data flow approach is suitable. If the direction is approved, I will split the implementation into a series of smaller patches.
It is also worth mentioning, that ShrinkWrappingPass from my patch can produce only multiple save points and can't be switched to single save/restore point mode. It is definitely should be fixed, and I plan to do it if approach is accepted.
---
Patch is 133.63 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/191784.diff
27 Files Affected:
- (modified) llvm/include/llvm/CodeGen/MachineFrameInfo.h (+47-2)
- (modified) llvm/include/llvm/CodeGen/Passes.h (+2)
- (added) llvm/include/llvm/CodeGen/ShrinkWrapping.h (+29)
- (modified) llvm/include/llvm/CodeGen/TargetFrameLowering.h (+4)
- (modified) llvm/include/llvm/CodeGen/TargetRegisterInfo.h (+6)
- (modified) llvm/include/llvm/InitializePasses.h (+1)
- (modified) llvm/lib/CodeGen/CMakeLists.txt (+1)
- (modified) llvm/lib/CodeGen/CodeGen.cpp (+1)
- (modified) llvm/lib/CodeGen/MIRPrinter.cpp (+1)
- (modified) llvm/lib/CodeGen/MachineFrameInfo.cpp (+1)
- (modified) llvm/lib/CodeGen/PrologEpilogInserter.cpp (+197-106)
- (modified) llvm/lib/CodeGen/ShrinkWrap.cpp (+1)
- (added) llvm/lib/CodeGen/ShrinkWrapping.cpp (+1734)
- (modified) llvm/lib/CodeGen/TargetPassConfig.cpp (+2-1)
- (modified) llvm/lib/Passes/PassBuilder.cpp (+1)
- (modified) llvm/lib/Target/AMDGPU/SILowerSGPRSpills.cpp (+14-13)
- (modified) llvm/lib/Target/PowerPC/PPCFrameLowering.cpp (-2)
- (modified) llvm/lib/Target/RISCV/RISCVFrameLowering.cpp (+160-19)
- (modified) llvm/lib/Target/RISCV/RISCVFrameLowering.h (+6)
- (modified) llvm/lib/Target/RISCV/RISCVRegisterInfo.cpp (+28-1)
- (modified) llvm/lib/Target/RISCV/RISCVRegisterInfo.h (+3)
- (modified) llvm/test/CodeGen/AArch64/live-debugvalues-sve.mir (+10-2)
- (modified) llvm/test/CodeGen/PowerPC/common-chain.ll (+33-33)
- (modified) llvm/test/CodeGen/PowerPC/more-dq-form-prepare.ll (+36-27)
- (modified) llvm/test/CodeGen/PowerPC/pr43527.ll (+2-2)
- (added) llvm/test/CodeGen/RISCV/shrinkwrap-split.mir (+284)
- (modified) llvm/tools/llvm-reduce/ReducerWorkItem.cpp (-6)
``````````diff
diff --git a/llvm/include/llvm/CodeGen/MachineFrameInfo.h b/llvm/include/llvm/CodeGen/MachineFrameInfo.h
index c4852f84be3524..1da9737d2b9d35 100644
--- a/llvm/include/llvm/CodeGen/MachineFrameInfo.h
+++ b/llvm/include/llvm/CodeGen/MachineFrameInfo.h
@@ -339,9 +339,16 @@ class MachineFrameInfo {
/// stack objects like arguments so we can't treat them as immutable.
bool HasTailCall = false;
- /// Not empty, if shrink-wrapping found a better place for the prologue.
+ /// Not null, if shrink-wrapping found a better place for the prologue.
+ MachineBasicBlock *Prolog = nullptr;
+ /// Not null, if shrink-wrapping found a better place for the epilogue.
+ MachineBasicBlock *Epilog = nullptr;
+
+ /// Not empty, if shrink-wrapping found a better place for saving callee
+ /// saves.
SaveRestorePoints SavePoints;
- /// Not empty, if shrink-wrapping found a better place for the epilogue.
+ /// Not empty, if shrink-wrapping found a better place for restoring callee
+ /// saves.
SaveRestorePoints RestorePoints;
/// Size of the UnsafeStack Frame
@@ -844,6 +851,39 @@ class MachineFrameInfo {
/// \copydoc getCalleeSavedInfo()
std::vector<CalleeSavedInfo> &getCalleeSavedInfo() { return CSInfo; }
+ /// Returns callee saved info vector for provided save point in
+ /// the current function.
+ std::vector<CalleeSavedInfo> getSaveCSInfo(MachineBasicBlock *MBB) const {
+ return SavePoints.lookup(MBB);
+ }
+
+ /// Returns callee saved info vector for provided restore point
+ /// in the current function.
+ const std::vector<CalleeSavedInfo>
+ getRestoreCSInfo(MachineBasicBlock *MBB) const {
+ return RestorePoints.lookup(MBB);
+ }
+
+ MachineBasicBlock *findSpilledIn(const CalleeSavedInfo &Match) const {
+ for (auto [BB, CSIV] : SavePoints) {
+ for (auto &CSI : CSIV) {
+ if (CSI.getReg() == Match.getReg())
+ return BB;
+ }
+ }
+ return nullptr;
+ }
+
+ MachineBasicBlock *findRestoredIn(const CalleeSavedInfo &Match) const {
+ for (auto [BB, CSIV] : RestorePoints) {
+ for (auto &CSI : CSIV) {
+ if (CSI.getReg() == Match.getReg())
+ return BB;
+ }
+ }
+ return nullptr;
+ }
+
/// Used by prolog/epilog inserter to set the function's callee saved
/// information.
void setCalleeSavedInfo(std::vector<CalleeSavedInfo> CSI) {
@@ -867,6 +907,11 @@ class MachineFrameInfo {
RestorePoints = std::move(NewRestorePoints);
}
+ MachineBasicBlock *getProlog() const { return Prolog; }
+ void setProlog(MachineBasicBlock *BB) { Prolog = BB; }
+ MachineBasicBlock *getEpilog() const { return Epilog; }
+ void setEpilog(MachineBasicBlock *BB) { Epilog = BB; }
+
void clearSavePoints() { SavePoints.clear(); }
void clearRestorePoints() { RestorePoints.clear(); }
diff --git a/llvm/include/llvm/CodeGen/Passes.h b/llvm/include/llvm/CodeGen/Passes.h
index 396c40d360eef8..a649279009802c 100644
--- a/llvm/include/llvm/CodeGen/Passes.h
+++ b/llvm/include/llvm/CodeGen/Passes.h
@@ -191,6 +191,8 @@ LLVM_ABI extern char &SpillPlacementID;
// instruction and update the MachineFunctionInfo with that information.
LLVM_ABI extern char &ShrinkWrapID;
+LLVM_ABI extern char &ShrinkWrappingID;
+
/// LiveRangeShrink pass. Move instruction close to its definition to shrink
/// the definition's live range.
LLVM_ABI extern char &LiveRangeShrinkID;
diff --git a/llvm/include/llvm/CodeGen/ShrinkWrapping.h b/llvm/include/llvm/CodeGen/ShrinkWrapping.h
new file mode 100644
index 00000000000000..d2dd4a96c00211
--- /dev/null
+++ b/llvm/include/llvm/CodeGen/ShrinkWrapping.h
@@ -0,0 +1,29 @@
+//===- llvm/CodeGen/ShrinkWrapping.h --------------------------------*- C++
+//-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CODEGEN_SHRINKWRAPPING_H
+#define LLVM_CODEGEN_SHRINKWRAPPING_H
+
+#include "llvm/CodeGen/MachinePassManager.h"
+
+namespace llvm {
+
+class ShrinkWrappingPass : public PassInfoMixin<ShrinkWrappingPass> {
+public:
+ PreservedAnalyses run(MachineFunction &MF,
+ MachineFunctionAnalysisManager &MFAM);
+
+ MachineFunctionProperties getRequiredProperties() const {
+ return MachineFunctionProperties().setNoVRegs();
+ }
+};
+
+} // namespace llvm
+
+#endif // LLVM_CODEGEN_SHRINKWRAPPING_H
diff --git a/llvm/include/llvm/CodeGen/TargetFrameLowering.h b/llvm/include/llvm/CodeGen/TargetFrameLowering.h
index 99034754e466b5..2e9b638358558b 100644
--- a/llvm/include/llvm/CodeGen/TargetFrameLowering.h
+++ b/llvm/include/llvm/CodeGen/TargetFrameLowering.h
@@ -193,6 +193,10 @@ class LLVM_ABI TargetFrameLowering {
return false;
}
+ /// enableCSRSaveRestorePointsSplit - Returns true if the target support
+ /// multiple save/restore points in shrink wrapping.
+ virtual bool enableCSRSaveRestorePointsSplit() const { return false; }
+
/// Returns true if the stack slot holes in the fixed and callee-save stack
/// area should be used when allocating other stack locations to reduce stack
/// size.
diff --git a/llvm/include/llvm/CodeGen/TargetRegisterInfo.h b/llvm/include/llvm/CodeGen/TargetRegisterInfo.h
index 5902fcdfcf1729..00a3dfad53d789 100644
--- a/llvm/include/llvm/CodeGen/TargetRegisterInfo.h
+++ b/llvm/include/llvm/CodeGen/TargetRegisterInfo.h
@@ -1172,6 +1172,12 @@ class LLVM_ABI TargetRegisterInfo : public MCRegisterInfo {
return false;
}
+ virtual bool isCSIFrameIndex(MachineFunction *MF, int FrameIndex) const {
+ return false;
+ }
+
+ virtual int64_t getCSIFrameOffset(MachineFunction *MF) const { return 0; }
+
/// Process frame indices in reverse block order. This changes the behavior of
/// the RegScavenger passed to eliminateFrameIndex. If this is true targets
/// should scavengeRegisterBackwards in eliminateFrameIndex. New targets
diff --git a/llvm/include/llvm/InitializePasses.h b/llvm/include/llvm/InitializePasses.h
index 48e4ecd8ee2af8..4d9ebbcfd59739 100644
--- a/llvm/include/llvm/InitializePasses.h
+++ b/llvm/include/llvm/InitializePasses.h
@@ -308,6 +308,7 @@ LLVM_ABI void
initializeSeparateConstOffsetFromGEPLegacyPassPass(PassRegistry &);
LLVM_ABI void initializeShadowStackGCLoweringPass(PassRegistry &);
LLVM_ABI void initializeShrinkWrapLegacyPass(PassRegistry &);
+LLVM_ABI void initializeShrinkWrappingLegacyPass(PassRegistry &);
LLVM_ABI void initializeSingleLoopExtractorPass(PassRegistry &);
LLVM_ABI void initializeSinkingLegacyPassPass(PassRegistry &);
LLVM_ABI void initializeSjLjEHPreparePass(PassRegistry &);
diff --git a/llvm/lib/CodeGen/CMakeLists.txt b/llvm/lib/CodeGen/CMakeLists.txt
index 7620d546091c8a..0182ef0a9833e0 100644
--- a/llvm/lib/CodeGen/CMakeLists.txt
+++ b/llvm/lib/CodeGen/CMakeLists.txt
@@ -222,6 +222,7 @@ add_llvm_component_library(LLVMCodeGen
SelectOptimize.cpp
ShadowStackGCLowering.cpp
ShrinkWrap.cpp
+ ShrinkWrapping.cpp
SjLjEHPrepare.cpp
SlotIndexes.cpp
SpillPlacement.cpp
diff --git a/llvm/lib/CodeGen/CodeGen.cpp b/llvm/lib/CodeGen/CodeGen.cpp
index a325c31faf4176..6f0dfbb39783f2 100644
--- a/llvm/lib/CodeGen/CodeGen.cpp
+++ b/llvm/lib/CodeGen/CodeGen.cpp
@@ -146,6 +146,7 @@ void llvm::initializeCodeGen(PassRegistry &Registry) {
initializeSelectOptimizePass(Registry);
initializeShadowStackGCLoweringPass(Registry);
initializeShrinkWrapLegacyPass(Registry);
+ initializeShrinkWrappingLegacyPass(Registry);
initializeSjLjEHPreparePass(Registry);
initializeSlotIndexesWrapperPassPass(Registry);
initializeStackColoringLegacyPass(Registry);
diff --git a/llvm/lib/CodeGen/MIRPrinter.cpp b/llvm/lib/CodeGen/MIRPrinter.cpp
index 250b40863c9f34..e78c22c477b207 100644
--- a/llvm/lib/CodeGen/MIRPrinter.cpp
+++ b/llvm/lib/CodeGen/MIRPrinter.cpp
@@ -162,6 +162,7 @@ convertSRPoints(ModuleSlotTracker &MST,
std::vector<yaml::SaveRestorePointEntry> &YamlSRPoints,
const llvm::SaveRestorePoints &SRPoints,
const TargetRegisterInfo *TRI);
+
static void convertStackObjects(yaml::MachineFunction &YMF,
const MachineFunction &MF,
ModuleSlotTracker &MST, MFPrintState &State);
diff --git a/llvm/lib/CodeGen/MachineFrameInfo.cpp b/llvm/lib/CodeGen/MachineFrameInfo.cpp
index aed68afb4eb1b4..c6658d2e9eba88 100644
--- a/llvm/lib/CodeGen/MachineFrameInfo.cpp
+++ b/llvm/lib/CodeGen/MachineFrameInfo.cpp
@@ -244,6 +244,7 @@ void MachineFrameInfo::print(const MachineFunction &MF, raw_ostream &OS) const{
}
OS << "\n";
}
+
OS << "save/restore points:\n";
if (!SavePoints.empty()) {
diff --git a/llvm/lib/CodeGen/PrologEpilogInserter.cpp b/llvm/lib/CodeGen/PrologEpilogInserter.cpp
index 68fd54cf00146e..697f023502b41c 100644
--- a/llvm/lib/CodeGen/PrologEpilogInserter.cpp
+++ b/llvm/lib/CodeGen/PrologEpilogInserter.cpp
@@ -80,8 +80,12 @@ namespace {
class PEIImpl {
RegScavenger *RS = nullptr;
- // Save and Restore blocks of the current function. Typically there is a
- // single save block, unless Windows EH funclets are involved.
+ // Prolog and Epilog blocks of the current function. Typically there is a
+ // single Prolog block, unless Windows EH funclets are involved.
+ MBBVector PrologBlocks;
+ MBBVector EpilogBlocks;
+
+ // Save and Restore blocks of the current function.
MBBVector SaveBlocks;
MBBVector RestoreBlocks;
@@ -99,6 +103,7 @@ class PEIImpl {
void calculateCallFrameInfo(MachineFunction &MF);
void calculateSaveRestoreBlocks(MachineFunction &MF);
+ void calculatePrologEpilogBlocks(MachineFunction &MF);
void spillCalleeSavedRegs(MachineFunction &MF);
void calculateFrameObjectOffsets(MachineFunction &MF);
@@ -229,14 +234,17 @@ bool PEIImpl::run(MachineFunction &MF) {
// information. Also eliminates call frame pseudo instructions.
calculateCallFrameInfo(MF);
- // Determine placement of CSR spill/restore code and prolog/epilog code:
+ // Determine placement of CSR spill/restore code:
// place all spills in the entry block, all restores in return blocks.
calculateSaveRestoreBlocks(MF);
+ // Determine placement of prolog/epilog code.
+ calculatePrologEpilogBlocks(MF);
+
// Stash away DBG_VALUEs that should not be moved by insertion of prolog code.
SavedDbgValuesMap EntryDbgValues;
- for (MachineBasicBlock *SaveBlock : SaveBlocks)
- stashEntryDbgValues(*SaveBlock, EntryDbgValues);
+ for (MachineBasicBlock *PrologBlock : PrologBlocks)
+ stashEntryDbgValues(*PrologBlock, EntryDbgValues);
// Handle CSR spilling and restoring, for targets that need it.
if (MF.getTarget().usesPhysRegsForValues())
@@ -344,6 +352,8 @@ bool PEIImpl::run(MachineFunction &MF) {
delete RS;
SaveBlocks.clear();
RestoreBlocks.clear();
+ PrologBlocks.clear();
+ EpilogBlocks.clear();
MFI.clearSavePoints();
MFI.clearRestorePoints();
return true;
@@ -412,6 +422,18 @@ void PEIImpl::calculateCallFrameInfo(MachineFunction &MF) {
}
}
+/// Compute two sets of blocks for placing prolog and epilog code respectively.
+void PEIImpl::calculatePrologEpilogBlocks(MachineFunction &MF) {
+ // Save refs to entry and return blocks.
+ PrologBlocks.push_back(&MF.front());
+ for (MachineBasicBlock &MBB : MF) {
+ if (MBB.isEHFuncletEntry())
+ PrologBlocks.push_back(&MBB);
+ if (MBB.isReturnBlock())
+ EpilogBlocks.push_back(&MBB);
+ }
+}
+
/// Compute the sets of entry and return blocks for saving and restoring
/// callee-saved registers, and placing prolog and epilog code.
void PEIImpl::calculateSaveRestoreBlocks(MachineFunction &MF) {
@@ -422,19 +444,19 @@ void PEIImpl::calculateSaveRestoreBlocks(MachineFunction &MF) {
// Use the points found by shrink-wrapping, if any.
if (!MFI.getSavePoints().empty()) {
- assert(MFI.getSavePoints().size() == 1 &&
- "Multiple save points are not yet supported!");
- const auto &SavePoint = *MFI.getSavePoints().begin();
- SaveBlocks.push_back(SavePoint.first);
- assert(MFI.getRestorePoints().size() == 1 &&
- "Multiple restore points are not yet supported!");
- const auto &RestorePoint = *MFI.getRestorePoints().begin();
- MachineBasicBlock *RestoreBlock = RestorePoint.first;
- // If RestoreBlock does not have any successor and is not a return block
- // then the end point is unreachable and we do not need to insert any
- // epilogue.
- if (!RestoreBlock->succ_empty() || RestoreBlock->isReturnBlock())
- RestoreBlocks.push_back(RestoreBlock);
+ assert(!MFI.getRestorePoints().empty() &&
+ "Both restore and save must be set");
+ for (auto &item : MFI.getSavePoints())
+ SaveBlocks.push_back(item.first);
+
+ for (auto &item : MFI.getRestorePoints()) {
+ MachineBasicBlock *RestoreBlock = item.first;
+ // If RestoreBlock does not have any successor and is not a return block
+ // then the end point is unreachable and we do not need to insert any
+ // epilogue.
+ if (!RestoreBlock->succ_empty() || RestoreBlock->isReturnBlock())
+ RestoreBlocks.push_back(RestoreBlock);
+ }
return;
}
@@ -539,8 +561,7 @@ static void assignCalleeSavedSpillSlots(MachineFunction &F,
/// Helper function to update the liveness information for the callee-saved
/// registers.
-static void updateLiveness(MachineFunction &MF) {
- MachineFrameInfo &MFI = MF.getFrameInfo();
+static void updateLiveness(MachineFunction &MF, CalleeSavedInfo &Info) {
// Visited will contain all the basic blocks that are in the region
// where the callee saved registers are alive:
// - Anything that is not Save or Restore -> LiveThrough.
@@ -548,73 +569,58 @@ static void updateLiveness(MachineFunction &MF) {
// - Restore -> LiveOut.
// The live-out is not attached to the block, so no need to keep
// Restore in this set.
+ bool AddSucc = true;
SmallPtrSet<MachineBasicBlock *, 8> Visited;
SmallVector<MachineBasicBlock *, 8> WorkList;
MachineBasicBlock *Entry = &MF.front();
+ MachineFrameInfo &MFI = MF.getFrameInfo();
- assert(MFI.getSavePoints().size() < 2 &&
- "Multiple save points not yet supported!");
- MachineBasicBlock *Save = MFI.getSavePoints().empty()
- ? nullptr
- : (*MFI.getSavePoints().begin()).first;
-
- if (!Save)
- Save = Entry;
-
- if (Entry != Save) {
- WorkList.push_back(Entry);
- Visited.insert(Entry);
- }
- Visited.insert(Save);
-
- assert(MFI.getRestorePoints().size() < 2 &&
- "Multiple restore points not yet supported!");
- MachineBasicBlock *Restore = MFI.getRestorePoints().empty()
- ? nullptr
- : (*MFI.getRestorePoints().begin()).first;
- if (Restore)
- // By construction Restore cannot be visited, otherwise it
- // means there exists a path to Restore that does not go
- // through Save.
- WorkList.push_back(Restore);
-
+ WorkList.push_back(Entry);
+ Visited.insert(Entry);
while (!WorkList.empty()) {
+ AddSucc = true;
const MachineBasicBlock *CurBB = WorkList.pop_back_val();
- // By construction, the region that is after the save point is
- // dominated by the Save and post-dominated by the Restore.
- if (CurBB == Save && Save != Restore)
- continue;
- // Enqueue all the successors not already visited.
- // Those are by construction either before Save or after Restore.
- for (MachineBasicBlock *SuccBB : CurBB->successors())
- if (Visited.insert(SuccBB).second)
- WorkList.push_back(SuccBB);
- }
+ auto SaveIt = MFI.getSavePoints().find(CurBB);
+ auto RestoreIt = MFI.getRestorePoints().find(CurBB);
+ if (SaveIt != MFI.getSavePoints().end() &&
+ count_if(SaveIt->second, [&Info](CalleeSavedInfo Other) {
+ return Other.getReg() == Info.getReg();
+ })) {
+ AddSucc = false;
+ if (RestoreIt != MFI.getRestorePoints().end() &&
+ count_if(RestoreIt->second, [&Info](CalleeSavedInfo Other) {
+ return Other.getReg() == Info.getReg();
+ }))
+ AddSucc = true;
+ }
- const std::vector<CalleeSavedInfo> &CSI = MFI.getCalleeSavedInfo();
+ if (AddSucc) {
+ for (MachineBasicBlock *SuccBB : CurBB->successors())
+ if (Visited.insert(SuccBB).second)
+ WorkList.push_back(SuccBB);
+ }
+ }
MachineRegisterInfo &MRI = MF.getRegInfo();
- for (const CalleeSavedInfo &I : CSI) {
- for (MachineBasicBlock *MBB : Visited) {
- MCRegister Reg = I.getReg();
- // Add the callee-saved register as live-in.
- // It's killed at the spill.
- if (!MRI.isReserved(Reg) && !MBB->isLiveIn(Reg))
- MBB->addLiveIn(Reg);
- }
- // If callee-saved register is spilled to another register rather than
- // spilling to stack, the destination register has to be marked as live for
- // each MBB between the prologue and epilogue so that it is not clobbered
- // before it is reloaded in the epilogue. The Visited set contains all
- // blocks outside of the region delimited by prologue/epilogue.
- if (I.isSpilledToReg()) {
- for (MachineBasicBlock &MBB : MF) {
- if (Visited.count(&MBB))
- continue;
- MCRegister DstReg = I.getDstReg();
- if (!MBB.isLiveIn(DstReg))
- MBB.addLiveIn(DstReg);
- }
+ for (MachineBasicBlock *MBB : Visited) {
+ MCPhysReg Reg = Info.getReg();
+ // Add the callee-saved register as live-in.
+ // It's killed at the spill.
+ if (!MRI.isReserved(Reg) && !MBB->isLiveIn(Reg))
+ MBB->addLiveIn(Reg);
+ }
+ // If callee-saved register is spilled to another register rather than
+ // spilling to stack, the destination register has to be marked as live for
+ // each MBB between the save and restore point so that it is not clobbered
+ // before it is reloaded in the restore point. The Visited set contains all
+ // blocks outside of the region delimited by save/restore.
+ if (Info.isSpilledToReg()) {
+ for (MachineBasicBlock &MBB : MF) {
+ if (Visited.count(&MBB))
+ continue;
+ MCPhysReg DstReg = Info.getDstReg();
+ if (!MBB.isLiveIn(DstReg))
+ MBB.addLiveIn(DstReg);
}
}
}
@@ -637,7 +643,7 @@ static void insertCSRSaves(MachineBasicBlock &SaveBlock,
/// Insert restore code for the callee-saved registers used in the function.
static void insertCSRRestores(MachineBasicBlock &RestoreBlock,
- std::vector<CalleeSavedInfo> &CSI) {
+ std::vector<CalleeSavedInfo> CSI) {
MachineFunction &MF = *RestoreBlock.getParent();
const TargetInstrInfo *TII = MF.getSubtarget().getInstrInfo();
const TargetFrameLowering *TFI = MF.getSubtarget().getFrameLowering();
@@ -654,6 +660,64 @@ static void insertCSRRestores(MachineBasicBlock &RestoreBlock,
}
}
+static void fillCSInfoPerBB(MachineFrameInfo &MFI,
+ DenseMap<MCRegister, CalleeSavedInfo *> &RegToInfo,
+ MBBVector &PrologEpilogBlocks, bool isSave) {
+ // Global CalleeSavedInfo list aggregating CSIVs for all points
+ std::vector<CalleeSavedInfo> GCSIV;
+ const SaveRestorePoints &SRPoints =
+ isSave ? MFI.getSavePoints() : MFI.getRestorePoints();
+ SaveRestorePoints Inner;
+ for (auto [BB, Regs] : SRPoints) {
+ // CalleeSavedInfo list for each point
+ std::vector<CalleeSavedInfo> CSIV;
+ for (auto &Reg : Regs) {
+ auto It = RegToInfo.find(Reg.getReg());
+ if (It == RegToInfo.end())
+ continue;
+ CSIV.push_back(*RegToInfo.at(Reg.getReg()));
+ GCSIV.push_back(*RegToInfo.at(Reg.getReg()));
+ }
+ // We need to sort CSIV, because Aarch64 expect CSI list to come sorted by
+ // frame index
+ sort(CSIV, [](const CalleeSavedInfo &Lhs, const CalleeSavedInfo &Rhs) {
+ return Lhs.getFrameIdx() < Rhs.getFrameIdx();
+ });
+ Inner.try_emplace(BB, std::move(CSIV));
+ }
+
+ // If in any case not all CSRs listed in MFI.getCalleeSavedInfo are in the
+ // list of spilled/restored registers (for example AArch64 backend add VG
+ // registers in the list of CalleeSavedRegs during spill slot assignment), we
+ // should add t...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/191784
More information about the llvm-commits
mailing list