[llvm] [RISCV] Move RISCVInsertVSETVLI to after phi elimination (PR #91440)
via llvm-commits
llvm-commits at lists.llvm.org
Wed May 8 00:48:45 PDT 2024
llvmbot wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-backend-risc-v
Author: Luke Lau (lukel97)
<details>
<summary>Changes</summary>
Split off from #<!-- -->70549, this patch moves RISCVInsertVSETVLI to after phi elimination where we exit SSA and need to move to LiveVariables.
The motivation for splitting this off is to avoid the large scheduling diffs from moving completely to after regalloc, and instead focus on converting the pass to work on LiveIntervals.
This limits the test diff to vsetvlis moving between csr and fsr instructions, due to RISCVInsertVSETVLI now taking place after RISCVInsertReadWriteCSR and RISCVInsertWriteVXRM.
The two main changes required are updating VSETVLIInfo to store VNInfos instead of MachineInstrs, which allows us to still check for PHI defs in needVSETVLIPHI, and fixing up the live intervals of any AVL operands after inserting new instructions.
On O3 the pass is inserted after the register coalescer, otherwise we end up with a bunch of COPYs around eliminated PHIs that trip up needVSETVLIPHI.
Note that this manually fixes up the LiveIntervals instead of recomputing them as is currently done in #<!-- -->70549, since it seems to avoid most of the changes in spills and reloads that we were seeing. Specifically LIS->handleMove seemed to change the SlotIndex of instructions which might have affected regalloc.
After this patch moving to post regalloc should be a matter of moving the pass in RISCVTargetMachines wihout any further changes to RISCVInsertVSETVLI itself.
Co-authored-by: Piyou Chen <piyou.chen@<!-- -->sifive.com>
Co-authored-by: Luke Lau <luke@<!-- -->igalia.com>
---
Patch is 1.73 MiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/91440.diff
110 Files Affected:
- (modified) llvm/lib/Target/RISCV/RISCVInsertVSETVLI.cpp (+157-103)
- (modified) llvm/lib/Target/RISCV/RISCVTargetMachine.cpp (+8-1)
- (modified) llvm/test/CodeGen/RISCV/O0-pipeline.ll (+4-2)
- (modified) llvm/test/CodeGen/RISCV/O3-pipeline.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/ceil-vp.ll (+18-18)
- (modified) llvm/test/CodeGen/RISCV/rvv/combine-vmv.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/commutable.ll (+6-6)
- (modified) llvm/test/CodeGen/RISCV/rvv/concat-vectors-constant-stride.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/ctlz-sdnode.ll (+28-28)
- (modified) llvm/test/CodeGen/RISCV/rvv/ctlz-vp.ll (+176-176)
- (modified) llvm/test/CodeGen/RISCV/rvv/cttz-sdnode.ll (+16-16)
- (modified) llvm/test/CodeGen/RISCV/rvv/dont-sink-splat-operands.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/double-round-conv.ll (+16-16)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-ceil-vp.ll (+19-19)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-ctlz.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-cttz.ll (+8-8)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-floor-vp.ll (+19-19)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-int.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-masked-gather.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-round-vp.ll (+19-19)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-roundeven-vp.ll (+19-19)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-roundtozero-vp.ll (+19-19)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-shuffle-vslide1up.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-strided-load-combine.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-strided-load-store-asm.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-trunc-vp.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vaaddu.ll (+19-19)
- (modified) llvm/test/CodeGen/RISCV/rvv/float-round-conv.ll (+24-24)
- (modified) llvm/test/CodeGen/RISCV/rvv/floor-vp.ll (+18-18)
- (modified) llvm/test/CodeGen/RISCV/rvv/fold-scalar-load-crash.ll (+2-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/fpclamptosat_vec.ll (+6-6)
- (modified) llvm/test/CodeGen/RISCV/rvv/frm-insert.ll (+42-42)
- (modified) llvm/test/CodeGen/RISCV/rvv/half-round-conv.ll (+12-12)
- (modified) llvm/test/CodeGen/RISCV/rvv/masked-tama.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/masked-tamu.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/masked-tuma.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/masked-tumu.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/round-vp.ll (+26-26)
- (modified) llvm/test/CodeGen/RISCV/rvv/roundeven-vp.ll (+26-26)
- (modified) llvm/test/CodeGen/RISCV/rvv/roundtozero-vp.ll (+26-26)
- (modified) llvm/test/CodeGen/RISCV/rvv/rv32-spill-vector-csr.ll (+2-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/rv32-spill-zvlsseg.ll (+5-5)
- (modified) llvm/test/CodeGen/RISCV/rvv/rv64-spill-vector-csr.ll (+2-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/rv64-spill-zvlsseg.ll (+5-5)
- (modified) llvm/test/CodeGen/RISCV/rvv/rvv-peephole-vmerge-masked-vops.ll (+2-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/rvv-peephole-vmerge-vops.ll (+11-11)
- (modified) llvm/test/CodeGen/RISCV/rvv/sf_vfnrclip_x_f_qf.ll (+20-20)
- (modified) llvm/test/CodeGen/RISCV/rvv/sf_vfnrclip_xu_f_qf.ll (+20-20)
- (modified) llvm/test/CodeGen/RISCV/rvv/sink-splat-operands.ll (+22-22)
- (modified) llvm/test/CodeGen/RISCV/rvv/undef-earlyclobber-chain.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/unmasked-tu.ll (+13-13)
- (modified) llvm/test/CodeGen/RISCV/rvv/vaadd.ll (+88-88)
- (modified) llvm/test/CodeGen/RISCV/rvv/vaaddu-sdnode.ll (+18-18)
- (modified) llvm/test/CodeGen/RISCV/rvv/vaaddu.ll (+88-88)
- (modified) llvm/test/CodeGen/RISCV/rvv/vasub.ll (+88-88)
- (modified) llvm/test/CodeGen/RISCV/rvv/vasubu.ll (+88-88)
- (modified) llvm/test/CodeGen/RISCV/rvv/vector-reassociations.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfadd.ll (+117-117)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfcvt-f-x.ll (+60-60)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfcvt-f-xu.ll (+60-60)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfcvt-x-f.ll (+60-60)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfcvt-xu-f.ll (+60-60)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfdiv.ll (+117-117)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfmacc.ll (+96-96)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfmadd.ll (+96-96)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfmsac.ll (+96-96)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfmsub.ll (+96-96)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfmul.ll (+117-117)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfncvt-f-f.ll (+36-36)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfncvt-f-x.ll (+36-36)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfncvt-f-xu.ll (+36-36)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfncvt-x-f.ll (+60-60)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfncvt-xu-f.ll (+60-60)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfnmacc.ll (+96-96)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfnmadd.ll (+96-96)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfnmsac.ll (+96-96)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfnmsub.ll (+96-96)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfrdiv.ll (+60-60)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfrec7.ll (+60-60)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfredosum.ll (+60-60)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfredusum.ll (+60-60)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfsqrt.ll (+60-60)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfwadd.ll (+72-72)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfwadd.w.ll (+122-122)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfwcvt-x-f.ll (+36-36)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfwcvt-xu-f.ll (+36-36)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfwmacc-vp.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfwmacc.ll (+72-72)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfwmsac.ll (+72-72)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfwmul.ll (+72-72)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfwnmacc.ll (+72-72)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfwnmsac.ll (+72-72)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfwredosum.ll (+44-44)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfwredusum.ll (+44-44)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfwsub.ll (+72-72)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfwsub.w.ll (+122-122)
- (modified) llvm/test/CodeGen/RISCV/rvv/vnclip.ll (+90-90)
- (modified) llvm/test/CodeGen/RISCV/rvv/vnclipu.ll (+90-90)
- (modified) llvm/test/CodeGen/RISCV/rvv/vrgatherei16-subreg-liveness.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/vsetvli-insert-crossbb.ll (-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/vsetvli-insert-crossbb.mir (+56-80)
- (modified) llvm/test/CodeGen/RISCV/rvv/vsetvli-insert.ll (+5-7)
- (modified) llvm/test/CodeGen/RISCV/rvv/vsmul-rv32.ll (+80-80)
- (modified) llvm/test/CodeGen/RISCV/rvv/vsmul-rv64.ll (+88-88)
- (modified) llvm/test/CodeGen/RISCV/rvv/vssra-rv32.ll (+88-88)
- (modified) llvm/test/CodeGen/RISCV/rvv/vssra-rv64.ll (+88-88)
- (modified) llvm/test/CodeGen/RISCV/rvv/vssrl-rv32.ll (+88-88)
- (modified) llvm/test/CodeGen/RISCV/rvv/vssrl-rv64.ll (+88-88)
- (modified) llvm/test/CodeGen/RISCV/rvv/vxrm-insert.ll (+12-12)
- (modified) llvm/test/CodeGen/RISCV/rvv/vxrm.mir (+3-3)
``````````diff
diff --git a/llvm/lib/Target/RISCV/RISCVInsertVSETVLI.cpp b/llvm/lib/Target/RISCV/RISCVInsertVSETVLI.cpp
index 5f8b610e52338..f2cf40483fcd4 100644
--- a/llvm/lib/Target/RISCV/RISCVInsertVSETVLI.cpp
+++ b/llvm/lib/Target/RISCV/RISCVInsertVSETVLI.cpp
@@ -47,6 +47,14 @@ static cl::opt<bool> DisableInsertVSETVLPHIOpt(
namespace {
+static VNInfo *getVNInfoFromReg(Register Reg, const MachineInstr &MI,
+ const LiveIntervals *LIS) {
+ auto &LI = LIS->getInterval(Reg);
+ SlotIndex SI = LIS->getSlotIndexes()->getInstructionIndex(MI);
+ VNInfo *VNI = LI.getVNInfoBefore(SI);
+ return VNI;
+}
+
static unsigned getVLOpNum(const MachineInstr &MI) {
return RISCVII::getVLOpNum(MI.getDesc());
}
@@ -337,9 +345,7 @@ static bool areCompatibleVTYPEs(uint64_t CurVType, uint64_t NewVType,
}
/// Return the fields and properties demanded by the provided instruction.
-DemandedFields getDemanded(const MachineInstr &MI,
- const MachineRegisterInfo *MRI,
- const RISCVSubtarget *ST) {
+DemandedFields getDemanded(const MachineInstr &MI, const RISCVSubtarget *ST) {
// Warning: This function has to work on both the lowered (i.e. post
// emitVSETVLIs) and pre-lowering forms. The main implication of this is
// that it can't use the value of a SEW, VL, or Policy operand as they might
@@ -428,7 +434,7 @@ DemandedFields getDemanded(const MachineInstr &MI,
/// values of the VL and VTYPE registers after insertion.
class VSETVLIInfo {
struct AVLDef {
- const MachineInstr *DefMI;
+ const VNInfo *AVLVNInfo;
Register DefReg;
};
union {
@@ -467,9 +473,9 @@ class VSETVLIInfo {
void setUnknown() { State = Unknown; }
bool isUnknown() const { return State == Unknown; }
- void setAVLRegDef(const MachineInstr *DefMI, Register AVLReg) {
- assert(DefMI && AVLReg.isVirtual());
- AVLRegDef.DefMI = DefMI;
+ void setAVLRegDef(const VNInfo *VNInfo, Register AVLReg) {
+ assert(VNInfo && AVLReg.isVirtual());
+ AVLRegDef.AVLVNInfo = VNInfo;
AVLRegDef.DefReg = AVLReg;
State = AVLIsReg;
}
@@ -495,9 +501,9 @@ class VSETVLIInfo {
assert(hasAVLImm());
return AVLImm;
}
- const MachineInstr &getAVLDefMI() const {
- assert(hasAVLReg() && AVLRegDef.DefMI);
- return *AVLRegDef.DefMI;
+ const VNInfo *getAVLVNInfo() const {
+ assert(hasAVLReg());
+ return AVLRegDef.AVLVNInfo;
}
void setAVL(VSETVLIInfo Info) {
@@ -505,7 +511,7 @@ class VSETVLIInfo {
if (Info.isUnknown())
setUnknown();
else if (Info.hasAVLReg())
- setAVLRegDef(&Info.getAVLDefMI(), Info.getAVLReg());
+ setAVLRegDef(Info.getAVLVNInfo(), Info.getAVLReg());
else if (Info.hasAVLVLMAX())
setAVLVLMAX();
else if (Info.hasAVLIgnored())
@@ -521,11 +527,13 @@ class VSETVLIInfo {
bool getTailAgnostic() const { return TailAgnostic; }
bool getMaskAgnostic() const { return MaskAgnostic; }
- bool hasNonZeroAVL() const {
+ bool hasNonZeroAVL(const LiveIntervals *LIS) const {
if (hasAVLImm())
return getAVLImm() > 0;
- if (hasAVLReg())
- return isNonZeroLoadImmediate(getAVLDefMI());
+ if (hasAVLReg()) {
+ if (auto *DefMI = LIS->getInstructionFromIndex(getAVLVNInfo()->def))
+ return isNonZeroLoadImmediate(*DefMI);
+ }
if (hasAVLVLMAX())
return true;
if (hasAVLIgnored())
@@ -533,16 +541,17 @@ class VSETVLIInfo {
return false;
}
- bool hasEquallyZeroAVL(const VSETVLIInfo &Other) const {
+ bool hasEquallyZeroAVL(const VSETVLIInfo &Other,
+ const LiveIntervals *LIS) const {
if (hasSameAVL(Other))
return true;
- return (hasNonZeroAVL() && Other.hasNonZeroAVL());
+ return (hasNonZeroAVL(LIS) && Other.hasNonZeroAVL(LIS));
}
bool hasSameAVL(const VSETVLIInfo &Other) const {
if (hasAVLReg() && Other.hasAVLReg())
- return AVLRegDef.DefMI == Other.AVLRegDef.DefMI &&
- AVLRegDef.DefReg == Other.AVLRegDef.DefReg;
+ return getAVLVNInfo()->id == Other.getAVLVNInfo()->id &&
+ getAVLReg() == Other.getAVLReg();
if (hasAVLImm() && Other.hasAVLImm())
return getAVLImm() == Other.getAVLImm();
@@ -622,7 +631,7 @@ class VSETVLIInfo {
// Require are compatible with the previous vsetvli instruction represented
// by this. MI is the instruction whose requirements we're considering.
bool isCompatible(const DemandedFields &Used, const VSETVLIInfo &Require,
- const MachineRegisterInfo &MRI) const {
+ const LiveIntervals *LIS) const {
assert(isValid() && Require.isValid() &&
"Can't compare invalid VSETVLIInfos");
assert(!Require.SEWLMULRatioOnly &&
@@ -638,7 +647,7 @@ class VSETVLIInfo {
if (Used.VLAny && !(hasSameAVL(Require) && hasSameVLMAX(Require)))
return false;
- if (Used.VLZeroness && !hasEquallyZeroAVL(Require))
+ if (Used.VLZeroness && !hasEquallyZeroAVL(Require, LIS))
return false;
return hasCompatibleVTYPE(Used, Require);
@@ -767,6 +776,7 @@ class RISCVInsertVSETVLI : public MachineFunctionPass {
const RISCVSubtarget *ST;
const TargetInstrInfo *TII;
MachineRegisterInfo *MRI;
+ LiveIntervals *LIS;
std::vector<BlockData> BlockInfo;
std::queue<const MachineBasicBlock *> WorkList;
@@ -779,6 +789,14 @@ class RISCVInsertVSETVLI : public MachineFunctionPass {
void getAnalysisUsage(AnalysisUsage &AU) const override {
AU.setPreservesCFG();
+
+ AU.addRequired<LiveIntervals>();
+ AU.addPreserved<LiveIntervals>();
+ AU.addRequired<SlotIndexes>();
+ AU.addPreserved<SlotIndexes>();
+ AU.addPreserved<LiveDebugVariables>();
+ AU.addPreserved<LiveStacks>();
+
MachineFunctionPass::getAnalysisUsage(AU);
}
@@ -850,7 +868,7 @@ INITIALIZE_PASS(RISCVCoalesceVSETVLI, "riscv-coalesce-vsetvli",
// Return a VSETVLIInfo representing the changes made by this VSETVLI or
// VSETIVLI instruction.
static VSETVLIInfo getInfoForVSETVLI(const MachineInstr &MI,
- const MachineRegisterInfo &MRI) {
+ const LiveIntervals *LIS) {
VSETVLIInfo NewInfo;
if (MI.getOpcode() == RISCV::PseudoVSETIVLI) {
NewInfo.setAVLImm(MI.getOperand(1).getImm());
@@ -863,7 +881,7 @@ static VSETVLIInfo getInfoForVSETVLI(const MachineInstr &MI,
if (AVLReg == RISCV::X0)
NewInfo.setAVLVLMAX();
else
- NewInfo.setAVLRegDef(MRI.getUniqueVRegDef(AVLReg), AVLReg);
+ NewInfo.setAVLRegDef(getVNInfoFromReg(AVLReg, MI, LIS), AVLReg);
}
NewInfo.setVTYPE(MI.getOperand(2).getImm());
@@ -882,7 +900,7 @@ static unsigned computeVLMAX(unsigned VLEN, unsigned SEW,
static VSETVLIInfo computeInfoForInstr(const MachineInstr &MI, uint64_t TSFlags,
const RISCVSubtarget &ST,
- const MachineRegisterInfo *MRI) {
+ const LiveIntervals *LIS) {
VSETVLIInfo InstrInfo;
bool TailAgnostic = true;
@@ -935,7 +953,7 @@ static VSETVLIInfo computeInfoForInstr(const MachineInstr &MI, uint64_t TSFlags,
else
InstrInfo.setAVLImm(Imm);
} else {
- InstrInfo.setAVLRegDef(MRI->getUniqueVRegDef(VLOp.getReg()),
+ InstrInfo.setAVLRegDef(getVNInfoFromReg(VLOp.getReg(), MI, LIS),
VLOp.getReg());
}
} else {
@@ -957,9 +975,10 @@ static VSETVLIInfo computeInfoForInstr(const MachineInstr &MI, uint64_t TSFlags,
// register AVLs to avoid extending live ranges without being sure we can
// kill the original source reg entirely.
if (InstrInfo.hasAVLReg()) {
- const MachineInstr &DefMI = InstrInfo.getAVLDefMI();
- if (isVectorConfigInstr(DefMI)) {
- VSETVLIInfo DefInstrInfo = getInfoForVSETVLI(DefMI, *MRI);
+ const MachineInstr *DefMI =
+ LIS->getInstructionFromIndex(InstrInfo.getAVLVNInfo()->def);
+ if (DefMI && isVectorConfigInstr(*DefMI)) {
+ VSETVLIInfo DefInstrInfo = getInfoForVSETVLI(*DefMI, LIS);
if (DefInstrInfo.hasSameVLMAX(InstrInfo) &&
(DefInstrInfo.hasAVLImm() || DefInstrInfo.hasAVLVLMAX()))
InstrInfo.setAVL(DefInstrInfo);
@@ -985,11 +1004,12 @@ void RISCVInsertVSETVLI::insertVSETVLI(MachineBasicBlock &MBB,
// Use X0, X0 form if the AVL is the same and the SEW+LMUL gives the same
// VLMAX.
if (Info.hasSameAVL(PrevInfo) && Info.hasSameVLMAX(PrevInfo)) {
- BuildMI(MBB, InsertPt, DL, TII->get(RISCV::PseudoVSETVLIX0))
- .addReg(RISCV::X0, RegState::Define | RegState::Dead)
- .addReg(RISCV::X0, RegState::Kill)
- .addImm(Info.encodeVTYPE())
- .addReg(RISCV::VL, RegState::Implicit);
+ auto MI = BuildMI(MBB, InsertPt, DL, TII->get(RISCV::PseudoVSETVLIX0))
+ .addReg(RISCV::X0, RegState::Define | RegState::Dead)
+ .addReg(RISCV::X0, RegState::Kill)
+ .addImm(Info.encodeVTYPE())
+ .addReg(RISCV::VL, RegState::Implicit);
+ LIS->InsertMachineInstrInMaps(*MI);
return;
}
@@ -997,15 +1017,17 @@ void RISCVInsertVSETVLI::insertVSETVLI(MachineBasicBlock &MBB,
// it has the same VLMAX we want and the last VL/VTYPE we observed is the
// same, we can use the X0, X0 form.
if (Info.hasSameVLMAX(PrevInfo) && Info.hasAVLReg()) {
- const MachineInstr &DefMI = Info.getAVLDefMI();
- if (isVectorConfigInstr(DefMI)) {
- VSETVLIInfo DefInfo = getInfoForVSETVLI(DefMI, *MRI);
+ const MachineInstr *DefMI =
+ LIS->getInstructionFromIndex(Info.getAVLVNInfo()->def);
+ if (DefMI && isVectorConfigInstr(*DefMI)) {
+ VSETVLIInfo DefInfo = getInfoForVSETVLI(*DefMI, LIS);
if (DefInfo.hasSameAVL(PrevInfo) && DefInfo.hasSameVLMAX(PrevInfo)) {
- BuildMI(MBB, InsertPt, DL, TII->get(RISCV::PseudoVSETVLIX0))
- .addReg(RISCV::X0, RegState::Define | RegState::Dead)
- .addReg(RISCV::X0, RegState::Kill)
- .addImm(Info.encodeVTYPE())
- .addReg(RISCV::VL, RegState::Implicit);
+ auto MI = BuildMI(MBB, InsertPt, DL, TII->get(RISCV::PseudoVSETVLIX0))
+ .addReg(RISCV::X0, RegState::Define | RegState::Dead)
+ .addReg(RISCV::X0, RegState::Kill)
+ .addImm(Info.encodeVTYPE())
+ .addReg(RISCV::VL, RegState::Implicit);
+ LIS->InsertMachineInstrInMaps(*MI);
return;
}
}
@@ -1013,10 +1035,11 @@ void RISCVInsertVSETVLI::insertVSETVLI(MachineBasicBlock &MBB,
}
if (Info.hasAVLImm()) {
- BuildMI(MBB, InsertPt, DL, TII->get(RISCV::PseudoVSETIVLI))
- .addReg(RISCV::X0, RegState::Define | RegState::Dead)
- .addImm(Info.getAVLImm())
- .addImm(Info.encodeVTYPE());
+ auto MI = BuildMI(MBB, InsertPt, DL, TII->get(RISCV::PseudoVSETIVLI))
+ .addReg(RISCV::X0, RegState::Define | RegState::Dead)
+ .addImm(Info.getAVLImm())
+ .addImm(Info.encodeVTYPE());
+ LIS->InsertMachineInstrInMaps(*MI);
return;
}
@@ -1025,36 +1048,43 @@ void RISCVInsertVSETVLI::insertVSETVLI(MachineBasicBlock &MBB,
// the previous vl to become invalid.
if (PrevInfo.isValid() && !PrevInfo.isUnknown() &&
Info.hasSameVLMAX(PrevInfo)) {
- BuildMI(MBB, InsertPt, DL, TII->get(RISCV::PseudoVSETVLIX0))
- .addReg(RISCV::X0, RegState::Define | RegState::Dead)
- .addReg(RISCV::X0, RegState::Kill)
- .addImm(Info.encodeVTYPE())
- .addReg(RISCV::VL, RegState::Implicit);
+ auto MI = BuildMI(MBB, InsertPt, DL, TII->get(RISCV::PseudoVSETVLIX0))
+ .addReg(RISCV::X0, RegState::Define | RegState::Dead)
+ .addReg(RISCV::X0, RegState::Kill)
+ .addImm(Info.encodeVTYPE())
+ .addReg(RISCV::VL, RegState::Implicit);
+ LIS->InsertMachineInstrInMaps(*MI);
return;
}
// Otherwise use an AVL of 1 to avoid depending on previous vl.
- BuildMI(MBB, InsertPt, DL, TII->get(RISCV::PseudoVSETIVLI))
- .addReg(RISCV::X0, RegState::Define | RegState::Dead)
- .addImm(1)
- .addImm(Info.encodeVTYPE());
+ auto MI = BuildMI(MBB, InsertPt, DL, TII->get(RISCV::PseudoVSETIVLI))
+ .addReg(RISCV::X0, RegState::Define | RegState::Dead)
+ .addImm(1)
+ .addImm(Info.encodeVTYPE());
+ LIS->InsertMachineInstrInMaps(*MI);
return;
}
if (Info.hasAVLVLMAX()) {
Register DestReg = MRI->createVirtualRegister(&RISCV::GPRRegClass);
- BuildMI(MBB, InsertPt, DL, TII->get(RISCV::PseudoVSETVLIX0))
- .addReg(DestReg, RegState::Define | RegState::Dead)
- .addReg(RISCV::X0, RegState::Kill)
- .addImm(Info.encodeVTYPE());
+ auto MI = BuildMI(MBB, InsertPt, DL, TII->get(RISCV::PseudoVSETVLIX0))
+ .addReg(DestReg, RegState::Define | RegState::Dead)
+ .addReg(RISCV::X0, RegState::Kill)
+ .addImm(Info.encodeVTYPE());
+ LIS->InsertMachineInstrInMaps(*MI);
+ LIS->createAndComputeVirtRegInterval(DestReg);
return;
}
Register AVLReg = Info.getAVLReg();
MRI->constrainRegClass(AVLReg, &RISCV::GPRNoX0RegClass);
- BuildMI(MBB, InsertPt, DL, TII->get(RISCV::PseudoVSETVLI))
- .addReg(RISCV::X0, RegState::Define | RegState::Dead)
- .addReg(AVLReg)
- .addImm(Info.encodeVTYPE());
+ auto MI = BuildMI(MBB, InsertPt, DL, TII->get(RISCV::PseudoVSETVLI))
+ .addReg(RISCV::X0, RegState::Define | RegState::Dead)
+ .addReg(AVLReg)
+ .addImm(Info.encodeVTYPE());
+ LIS->InsertMachineInstrInMaps(*MI);
+ LIS->getInterval(AVLReg).extendInBlock(
+ LIS->getMBBStartIdx(&MBB), LIS->getInstructionIndex(*MI).getRegSlot());
}
static bool isLMUL1OrSmaller(RISCVII::VLMUL LMUL) {
@@ -1067,12 +1097,12 @@ static bool isLMUL1OrSmaller(RISCVII::VLMUL LMUL) {
bool RISCVInsertVSETVLI::needVSETVLI(const MachineInstr &MI,
const VSETVLIInfo &Require,
const VSETVLIInfo &CurInfo) const {
- assert(Require == computeInfoForInstr(MI, MI.getDesc().TSFlags, *ST, MRI));
+ assert(Require == computeInfoForInstr(MI, MI.getDesc().TSFlags, *ST, LIS));
if (!CurInfo.isValid() || CurInfo.isUnknown() || CurInfo.hasSEWLMULRatioOnly())
return true;
- DemandedFields Used = getDemanded(MI, MRI, ST);
+ DemandedFields Used = getDemanded(MI, ST);
// A slidedown/slideup with an *undefined* merge op can freely clobber
// elements not copied from the source vector (e.g. masked off, tail, or
@@ -1108,7 +1138,7 @@ bool RISCVInsertVSETVLI::needVSETVLI(const MachineInstr &MI,
Used.TailPolicy = false;
}
- if (CurInfo.isCompatible(Used, Require, *MRI))
+ if (CurInfo.isCompatible(Used, Require, LIS))
return false;
// We didn't find a compatible value. If our AVL is a virtual register,
@@ -1116,9 +1146,10 @@ bool RISCVInsertVSETVLI::needVSETVLI(const MachineInstr &MI,
// and the last VL/VTYPE we observed is the same, we don't need a
// VSETVLI here.
if (Require.hasAVLReg() && CurInfo.hasCompatibleVTYPE(Used, Require)) {
- const MachineInstr &DefMI = Require.getAVLDefMI();
- if (isVectorConfigInstr(DefMI)) {
- VSETVLIInfo DefInfo = getInfoForVSETVLI(DefMI, *MRI);
+ const MachineInstr *DefMI =
+ LIS->getInstructionFromIndex(Require.getAVLVNInfo()->def);
+ if (DefMI && isVectorConfigInstr(*DefMI)) {
+ VSETVLIInfo DefInfo = getInfoForVSETVLI(*DefMI, LIS);
if (DefInfo.hasSameAVL(CurInfo) && DefInfo.hasSameVLMAX(CurInfo))
return false;
}
@@ -1154,7 +1185,7 @@ void RISCVInsertVSETVLI::transferBefore(VSETVLIInfo &Info,
if (!RISCVII::hasSEWOp(TSFlags))
return;
- const VSETVLIInfo NewInfo = computeInfoForInstr(MI, TSFlags, *ST, MRI);
+ const VSETVLIInfo NewInfo = computeInfoForInstr(MI, TSFlags, *ST, LIS);
assert(NewInfo.isValid() && !NewInfo.isUnknown());
if (Info.isValid() && !needVSETVLI(MI, NewInfo, Info))
return;
@@ -1163,7 +1194,7 @@ void RISCVInsertVSETVLI::transferBefore(VSETVLIInfo &Info,
if (!Info.isValid() || Info.isUnknown())
Info = NewInfo;
- DemandedFields Demanded = getDemanded(MI, MRI, ST);
+ DemandedFields Demanded = getDemanded(MI, ST);
const VSETVLIInfo IncomingInfo = adjustIncoming(PrevInfo, NewInfo, Demanded);
// If MI only demands that VL has the same zeroness, we only need to set the
@@ -1173,7 +1204,7 @@ void RISCVInsertVSETVLI::transferBefore(VSETVLIInfo &Info,
// variant, so we avoid the transform to prevent extending live range of an
// avl register operand.
// TODO: We can probably relax this for immediates.
- bool EquallyZero = IncomingInfo.hasEquallyZeroAVL(PrevInfo) &&
+ bool EquallyZero = IncomingInfo.hasEquallyZeroAVL(PrevInfo, LIS) &&
IncomingInfo.hasSameVLMAX(PrevInfo);
if (Demanded.VLAny || (Demanded.VLZeroness && !EquallyZero))
Info.setAVL(IncomingInfo);
@@ -1204,14 +1235,17 @@ void RISCVInsertVSETVLI::transferBefore(VSETVLIInfo &Info,
void RISCVInsertVSETVLI::transferAfter(VSETVLIInfo &Info,
const MachineInstr &MI) const {
if (isVectorConfigInstr(MI)) {
- Info = getInfoForVSETVLI(MI, *MRI);
+ Info = getInfoForVSETVLI(MI, LIS);
return;
}
if (RISCV::isFaultFirstLoad(MI)) {
// Update AVL to vl-output of the fault first load.
- Info.setAVLRegDef(MRI->getUniqueVRegDef(MI.getOperand(1).getReg()),
- MI.getOperand(1).getReg());
+ assert(MI.getOperand(1).getReg().isVirtual());
+ auto &LI = LIS->getInterval(MI.getOperand(1).getReg());
+ SlotIndex SI = LIS->getSlotIndexes()->getInstructionIndex(MI).getRegSlot();
+ VNInfo *VNI = LI.getVNInfoAt(SI);
+ Info.setAVLRegDef(VNI, MI.getOperand(1).getReg());
return;
}
@@ -1295,7 +1329,7 @@ void RISCVInsertVSETVLI::computeIncomingVLVTYPE(const MachineBasicBlock &MBB) {
}
// If we weren't able to prove a vsetvli was directly unneeded, it might still
-// be unneeded if the AVL is a phi node where all incoming values are VL
+// be unneeded if the AVL was a phi node where all incoming values are VL
// outputs from the last VSETVLI in their respective basic blocks.
bool RISCVInsertVSETVLI::needVSETVLIPHI(const VSETVLIInfo &Require,
const MachineBasicBlock &MBB) const {
@@ -1305,26 +1339,27 @@ bool RISCVInsertVSETVLI::needVSETVLIPHI(const VSETVLIInfo &Require,
if (!Require.hasAVLReg())
return true;
- // We need the AVL to be produce by a PHI node in this basic block.
- const MachineInstr *PHI = &Require.getAVLDefMI();
- if (PHI->getOpcode() != RISCV::PHI || PHI->getParent() != &MBB)
+ // We need the AVL to have been produced by a PHI node in this basic block.
+ const VNInfo *Valno = Require.getAVLVNInfo();
+ if (!Valno->isPHIDef() || LIS->getMBBFromIndex(Valno->def) != &MBB)
return true;
- for (unsigned PHIOp = 1, NumOps = PHI->getNumOperands(); PHIOp != NumOps;
- PHIOp += 2) {
- Register InReg = PHI->getOperand(PHIOp).getReg();
- MachineBasicBlock *PBB = PHI->getOperand(PHIOp + 1).getMBB();
+ const LiveRange &LR = LIS->getInterval(Require.getAVLReg());
+
+ for (auto *PBB : MBB.predecessors()) {
const VSETVLIInfo &PBBExit = BlockInfo[PBB->getNumber()].Exit;
// We need the PHI input to the be the output of a VSET(I)VLI.
- MachineInstr *DefMI = MRI->getUniqueVRegDef(InReg);
- assert(DefMI);
- if (!isVectorConfigInstr(*DefMI))
+ const VNInfo *Value = LR.getVNInfoBefore(LIS->getMBBEndIdx(PBB));
+ if (!Value)
+ return true;
+ MachineInstr *DefMI = LIS->getInstructionFromIndex(Value->def);
+ if (!DefMI || !isVectorConfigInstr(*DefMI))
return true;
// We found a VSET(I)VLI make sure it matches the output of the
// predecessor block.
- VSETVLIInfo DefInfo = getInfoForVSETVLI(*DefMI, *MRI);
+ VSETVLIInfo DefInfo = getInfoForVSETVLI(*DefMI, LIS);
if (DefInfo != PBBExit)
return true;
@@ -1379,19 +1414,28 @@ void RISCVInsertVSETVLI::emitVSETVLIs(MachineBasicBlock &MBB) {
MachineOperand &VLOp = MI.getOperand(getVLOpNum(MI));
if (VLOp.isR...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/91440
More information about the llvm-commits
mailing list