[llvm] [RISCV] Don't use V0 directly in patterns (PR #88496)
via llvm-commits
llvm-commits at lists.llvm.org
Fri Apr 12 03:31:32 PDT 2024
llvmbot wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-llvm-globalisel
Author: Pengcheng Wang (wangpc-pp)
<details>
<summary>Changes</summary>
I had thought about this when I tried to optimize MatcherTable
last year, and this unimpressive use of V0 has become an issue
when I try to make more vector pseudos commutable.
The issue is, for masked vector pseudos, we can't do MachineCSE on
two equal instructions because `MachineCSE::hasLivePhysRegDefUses`
returns `true`. IMO, we will miss some more oppotinuties to optimize
as using physical register directly breaks use-def/value numbering
mechinisms.
This patch replace all `V0` in patterns with a `VMV0` operand and
let the RA do the register allocation.
The benefits are:
* The MatcherTable decreases by about 7.8%, from 2218043 to 2052389.
* We seem to be able optimize some tests in tree (192 files changed,
8022 insertions(+), 10527 deletions(-)`)
The correctness checking is ongoing, and feedbacks are welcome.
---
Patch is 1.71 MiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/88496.diff
192 Files Affected:
- (modified) llvm/lib/Target/RISCV/RISCVFoldMasks.cpp (+19-31)
- (modified) llvm/lib/Target/RISCV/RISCVISelDAGToDAG.cpp (+15-71)
- (modified) llvm/lib/Target/RISCV/RISCVInstrInfo.h (+1)
- (modified) llvm/lib/Target/RISCV/RISCVInstrInfoVPseudos.td (+46-46)
- (modified) llvm/lib/Target/RISCV/RISCVInstrInfoVSDPatterns.td (+17-17)
- (modified) llvm/lib/Target/RISCV/RISCVInstrInfoVVLPatterns.td (+215-215)
- (modified) llvm/lib/Target/RISCV/RISCVInstrInfoZvk.td (+33-33)
- (modified) llvm/lib/Target/RISCV/RISCVRegisterInfo.h (+2)
- (modified) llvm/lib/Target/RISCV/RISCVRegisterInfo.td (+1-1)
- (modified) llvm/test/CodeGen/RISCV/GlobalISel/instruction-select/rvv/select.mir (+44-66)
- (modified) llvm/test/CodeGen/RISCV/early-clobber-tied-def-subreg-liveness.ll (+7-7)
- (modified) llvm/test/CodeGen/RISCV/intrinsic-cttz-elts-vscale.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/abd.ll (+12-12)
- (modified) llvm/test/CodeGen/RISCV/rvv/abs-vp.ll (+6-30)
- (modified) llvm/test/CodeGen/RISCV/rvv/bitreverse-vp.ll (+109-125)
- (modified) llvm/test/CodeGen/RISCV/rvv/bswap-vp.ll (+78-94)
- (modified) llvm/test/CodeGen/RISCV/rvv/ceil-vp.ll (+87-110)
- (modified) llvm/test/CodeGen/RISCV/rvv/constant-folding-crash.ll (+8-8)
- (modified) llvm/test/CodeGen/RISCV/rvv/ctlz-vp.ll (+6-6)
- (modified) llvm/test/CodeGen/RISCV/rvv/ctpop-vp.ll (+60-181)
- (modified) llvm/test/CodeGen/RISCV/rvv/cttz-sdnode.ll (+20-20)
- (modified) llvm/test/CodeGen/RISCV/rvv/cttz-vp.ll (+69-225)
- (modified) llvm/test/CodeGen/RISCV/rvv/extractelt-i1.ll (+18-18)
- (modified) llvm/test/CodeGen/RISCV/rvv/fceil-constrained-sdnode.ll (+10-10)
- (modified) llvm/test/CodeGen/RISCV/rvv/ffloor-constrained-sdnode.ll (+10-10)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-bitreverse-vp.ll (+98-114)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-bswap-vp.ll (+77-93)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-buildvec-of-binop.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-ceil-vp.ll (+79-103)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-ctlz-vp.ll (+194-422)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-ctpop-vp.ll (+47-93)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-cttz-vp.ll (+184-412)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-cttz.ll (+8-8)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-extract-i1.ll (+38-38)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-fceil-constrained-sdnode.ll (+10-10)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-ffloor-constrained-sdnode.ll (+10-10)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-floor-vp.ll (+79-103)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-fmaximum.ll (+69-89)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-fminimum.ll (+69-89)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-fnearbyint-constrained-sdnode.ll (+8-8)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-fp-buildvec.ll (+2-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-fp-interleave.ll (+9-8)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-fp2i-sat.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-fround-costrained-sdnode.ll (+10-10)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-froundeven-constrained-sdnode.ll (+10-10)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-fshr-fshl-vp.ll (+30-18)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-ftrunc-constrained-sdnode.ll (+10-10)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-insert-subvector.ll (+28-28)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-int-interleave.ll (+11-10)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-int-shuffles.ll (+15-15)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-int.ll (+19-19)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-interleaved-access.ll (+344-268)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-mask-buildvec.ll (+12-12)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-mask-splat.ll (+3-3)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-masked-gather.ll (+15-15)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-masked-load-fp.ll (+28-28)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-masked-load-int.ll (+27-27)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-masked-store-fp.ll (+36-124)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-masked-store-int.ll (+50-151)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-nearbyint-vp.ll (+63-78)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-reduction-fp-vp.ll (+12-12)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-reduction-fp.ll (+84-192)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-reduction-int-vp.ll (+6-6)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-rint-vp.ll (+65-89)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-round-vp.ll (+79-103)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-roundeven-vp.ll (+79-103)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-roundtozero-vp.ll (+79-103)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-setcc-fp-vp.ll (+70-29)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-setcc-int-vp.ll (+55-39)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-shuffle-transpose.ll (+11-11)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-strided-vpload.ll (+26-22)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-trunc-vp.ll (-277)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vaaddu.ll (+8-8)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vadd-vp.ll (+2-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vand-vp.ll (+11-11)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vcopysign-vp.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfcmp-constrained-sdnode.ll (+648-648)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfma-vp.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfmax-vp.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfmin-vp.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vfmuladd-vp.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vmax-vp.ll (+2-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vmaxu-vp.ll (+2-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vmin-vp.ll (+2-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vminu-vp.ll (+2-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vpgather.ll (+2-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vpload.ll (+19-19)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vpscatter.ll (+32-32)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vsadd-vp.ll (+2-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vsaddu-vp.ll (+2-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vselect-vp.ll (+27-36)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vselect.ll (+12-12)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vssub-vp.ll (+6-6)
- (modified) llvm/test/CodeGen/RISCV/rvv/fixed-vectors-vssubu-vp.ll (+6-6)
- (modified) llvm/test/CodeGen/RISCV/rvv/floor-vp.ll (+87-110)
- (modified) llvm/test/CodeGen/RISCV/rvv/fmaximum-sdnode.ll (+77-113)
- (modified) llvm/test/CodeGen/RISCV/rvv/fminimum-sdnode.ll (+77-113)
- (modified) llvm/test/CodeGen/RISCV/rvv/fnearbyint-constrained-sdnode.ll (+10-10)
- (modified) llvm/test/CodeGen/RISCV/rvv/fround-costrained-sdnode.ll (+10-10)
- (modified) llvm/test/CodeGen/RISCV/rvv/froundeven-constrained-sdnode.ll (+10-10)
- (modified) llvm/test/CodeGen/RISCV/rvv/fshr-fshl-vp.ll (+97-419)
- (modified) llvm/test/CodeGen/RISCV/rvv/ftrunc-constrained-sdnode.ll (+10-10)
- (modified) llvm/test/CodeGen/RISCV/rvv/implicit-def-copy.ll (+2-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/insert-subvector.ll (+10-10)
- (modified) llvm/test/CodeGen/RISCV/rvv/llrint-vp.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/lrint-vp.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/mgather-sdnode.ll (+10-9)
- (modified) llvm/test/CodeGen/RISCV/rvv/mutate-prior-vsetvli-avl.ll (+6-6)
- (modified) llvm/test/CodeGen/RISCV/rvv/nearbyint-vp.ll (+139-200)
- (modified) llvm/test/CodeGen/RISCV/rvv/pass-fast-math-flags-sdnode.ll (+2-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/rint-vp.ll (+140-191)
- (modified) llvm/test/CodeGen/RISCV/rvv/round-vp.ll (+140-191)
- (modified) llvm/test/CodeGen/RISCV/rvv/roundeven-vp.ll (+140-191)
- (modified) llvm/test/CodeGen/RISCV/rvv/roundtozero-vp.ll (+140-191)
- (modified) llvm/test/CodeGen/RISCV/rvv/rvv-peephole-vmerge-to-vmv.mir (+2-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/rvv-peephole-vmerge-vops-mir.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/setcc-fp-vp.ll (+98-129)
- (modified) llvm/test/CodeGen/RISCV/rvv/setcc-int-vp.ll (+32-42)
- (modified) llvm/test/CodeGen/RISCV/rvv/shuffle-reverse.ll (+19-19)
- (modified) llvm/test/CodeGen/RISCV/rvv/sink-splat-operands.ll (+12-12)
- (modified) llvm/test/CodeGen/RISCV/rvv/sshl_sat_vec.ll (+48-56)
- (modified) llvm/test/CodeGen/RISCV/rvv/strided-vpload-vpstore-output.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/strided-vpload.ll (+14-12)
- (modified) llvm/test/CodeGen/RISCV/rvv/strided-vpstore.ll (+5-20)
- (modified) llvm/test/CodeGen/RISCV/rvv/umulo-sdnode.ll (+88-88)
- (modified) llvm/test/CodeGen/RISCV/rvv/vadd-vp.ll (+6-6)
- (modified) llvm/test/CodeGen/RISCV/rvv/vcopysign-vp.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/vector-deinterleave-load.ll (+5-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/vector-deinterleave.ll (+25-12)
- (modified) llvm/test/CodeGen/RISCV/rvv/vector-interleave-store.ll (+13-13)
- (modified) llvm/test/CodeGen/RISCV/rvv/vector-interleave.ll (+45-44)
- (modified) llvm/test/CodeGen/RISCV/rvv/vector-splice.ll (+80-80)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfabs-vp.ll (+9-9)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfadd-vp.ll (+10-10)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfcmp-constrained-sdnode.ll (+972-972)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfdiv-vp.ll (+10-10)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfma-vp.ll (+63-26)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfmax-vp.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfmin-vp.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfmul-vp.ll (+10-10)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfmuladd-vp.ll (+63-26)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfneg-vp.ll (+9-9)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfpext-vp.ll (+1-1)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfptosi-vp.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfptoui-vp.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfptrunc-vp.ll (+41-54)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfsqrt-vp.ll (+9-9)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfsub-vp.ll (+10-10)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfwnmacc-vp.ll (+20-6)
- (modified) llvm/test/CodeGen/RISCV/rvv/vfwnmsac-vp.ll (+20-6)
- (modified) llvm/test/CodeGen/RISCV/rvv/vleff-vlseg2ff-output.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/vmax-vp.ll (+6-6)
- (modified) llvm/test/CodeGen/RISCV/rvv/vmaxu-vp.ll (+6-6)
- (modified) llvm/test/CodeGen/RISCV/rvv/vmfeq.ll (+30-42)
- (modified) llvm/test/CodeGen/RISCV/rvv/vmfge.ll (+30-42)
- (modified) llvm/test/CodeGen/RISCV/rvv/vmfgt.ll (+30-42)
- (modified) llvm/test/CodeGen/RISCV/rvv/vmfle.ll (+30-42)
- (modified) llvm/test/CodeGen/RISCV/rvv/vmflt.ll (+30-42)
- (modified) llvm/test/CodeGen/RISCV/rvv/vmfne.ll (+30-42)
- (modified) llvm/test/CodeGen/RISCV/rvv/vmin-vp.ll (+6-6)
- (modified) llvm/test/CodeGen/RISCV/rvv/vminu-vp.ll (+6-6)
- (modified) llvm/test/CodeGen/RISCV/rvv/vmseq.ll (+48-66)
- (modified) llvm/test/CodeGen/RISCV/rvv/vmsgt.ll (+48-66)
- (modified) llvm/test/CodeGen/RISCV/rvv/vmsgtu.ll (+48-66)
- (modified) llvm/test/CodeGen/RISCV/rvv/vmsle.ll (+48-66)
- (modified) llvm/test/CodeGen/RISCV/rvv/vmsleu.ll (+48-66)
- (modified) llvm/test/CodeGen/RISCV/rvv/vmslt.ll (+48-66)
- (modified) llvm/test/CodeGen/RISCV/rvv/vmsltu.ll (+48-66)
- (modified) llvm/test/CodeGen/RISCV/rvv/vmsne.ll (+48-66)
- (modified) llvm/test/CodeGen/RISCV/rvv/vp-reverse-int.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/vp-reverse-mask-fixed-vectors.ll (+18-18)
- (modified) llvm/test/CodeGen/RISCV/rvv/vp-reverse-mask.ll (+34-33)
- (modified) llvm/test/CodeGen/RISCV/rvv/vp-splice-mask-fixed-vectors.ll (+84-84)
- (modified) llvm/test/CodeGen/RISCV/rvv/vp-splice-mask-vectors.ll (+150-150)
- (modified) llvm/test/CodeGen/RISCV/rvv/vpgather-sdnode.ll (+24-23)
- (modified) llvm/test/CodeGen/RISCV/rvv/vpload.ll (+19-19)
- (modified) llvm/test/CodeGen/RISCV/rvv/vpmerge-sdnode.ll (+22-13)
- (modified) llvm/test/CodeGen/RISCV/rvv/vpstore.ll (+5-20)
- (modified) llvm/test/CodeGen/RISCV/rvv/vreductions-fp-vp.ll (+12-12)
- (modified) llvm/test/CodeGen/RISCV/rvv/vreductions-int-vp.ll (+17-17)
- (modified) llvm/test/CodeGen/RISCV/rvv/vsadd-vp.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/vsaddu-vp.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/vscale-vw-web-simplification.ll (+100-100)
- (modified) llvm/test/CodeGen/RISCV/rvv/vselect-fp.ll (+14-23)
- (modified) llvm/test/CodeGen/RISCV/rvv/vselect-vp.ll (+29-59)
- (modified) llvm/test/CodeGen/RISCV/rvv/vsext-vp.ll (+2-2)
- (modified) llvm/test/CodeGen/RISCV/rvv/vsitofp-vp.ll (+5-5)
- (modified) llvm/test/CodeGen/RISCV/rvv/vssub-vp.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/vssubu-vp.ll (+4-4)
- (modified) llvm/test/CodeGen/RISCV/rvv/vtrunc-vp.ll (+48-47)
- (modified) llvm/test/CodeGen/RISCV/rvv/vuitofp-vp.ll (+5-5)
- (modified) llvm/test/CodeGen/RISCV/rvv/vzext-vp.ll (+2-2)
``````````diff
diff --git a/llvm/lib/Target/RISCV/RISCVFoldMasks.cpp b/llvm/lib/Target/RISCV/RISCVFoldMasks.cpp
index 2089f5dda6fe52..0425e1411107e0 100644
--- a/llvm/lib/Target/RISCV/RISCVFoldMasks.cpp
+++ b/llvm/lib/Target/RISCV/RISCVFoldMasks.cpp
@@ -50,10 +50,7 @@ class RISCVFoldMasks : public MachineFunctionPass {
bool convertToUnmasked(MachineInstr &MI) const;
bool convertVMergeToVMv(MachineInstr &MI) const;
- bool isAllOnesMask(const MachineInstr *MaskDef) const;
-
- /// Maps uses of V0 to the corresponding def of V0.
- DenseMap<const MachineInstr *, const MachineInstr *> V0Defs;
+ bool isAllOnesMask(const MachineOperand& MaskOp) const;
};
} // namespace
@@ -62,12 +59,22 @@ char RISCVFoldMasks::ID = 0;
INITIALIZE_PASS(RISCVFoldMasks, DEBUG_TYPE, "RISC-V Fold Masks", false, false)
-bool RISCVFoldMasks::isAllOnesMask(const MachineInstr *MaskDef) const {
- assert(MaskDef && MaskDef->isCopy() &&
- MaskDef->getOperand(0).getReg() == RISCV::V0);
+bool RISCVFoldMasks::isAllOnesMask(const MachineOperand &MaskOp) const {
+ if (!MaskOp.isReg())
+ return false;
+
+ Register MaskReg = MaskOp.getReg();
+ if (!MaskReg.isVirtual())
+ return false;
+
+ MachineInstr *MaskDef = MRI->getVRegDef(MaskReg);
+ if (!MaskDef || !MaskDef->isCopy())
+ return false;
+
Register SrcReg = TRI->lookThruCopyLike(MaskDef->getOperand(1).getReg(), MRI);
if (!SrcReg.isVirtual())
return false;
+
MaskDef = MRI->getVRegDef(SrcReg);
if (!MaskDef)
return false;
@@ -116,8 +123,8 @@ bool RISCVFoldMasks::convertVMergeToVMv(MachineInstr &MI) const {
TRI->lookThruCopyLike(FalseReg, MRI))
return false;
- assert(MI.getOperand(4).isReg() && MI.getOperand(4).getReg() == RISCV::V0);
- if (!isAllOnesMask(V0Defs.lookup(&MI)))
+ // assert(MI.getOperand(4).isReg() && MI.getOperand(4).getReg() == RISCV::V0);
+ if (!isAllOnesMask(MI.getOperand(4)))
return false;
MI.setDesc(TII->get(NewOpc));
@@ -140,7 +147,9 @@ bool RISCVFoldMasks::convertToUnmasked(MachineInstr &MI) const {
if (!I)
return false;
- if (!isAllOnesMask(V0Defs.lookup(&MI)))
+ // TODO: Increment all MaskOpIdxs in tablegen by num of explicit defs?
+ unsigned MaskOpIdx = I->MaskOpIdx + MI.getNumExplicitDefs();
+ if (!isAllOnesMask(MI.getOperand(MaskOpIdx)))
return false;
// There are two classes of pseudos in the table - compares and
@@ -160,9 +169,6 @@ bool RISCVFoldMasks::convertToUnmasked(MachineInstr &MI) const {
(void)HasPolicyOp;
MI.setDesc(MCID);
-
- // TODO: Increment all MaskOpIdxs in tablegen by num of explicit defs?
- unsigned MaskOpIdx = I->MaskOpIdx + MI.getNumExplicitDefs();
MI.removeOperand(MaskOpIdx);
// The unmasked pseudo will no longer be constrained to the vrnov0 reg class,
@@ -193,24 +199,6 @@ bool RISCVFoldMasks::runOnMachineFunction(MachineFunction &MF) {
bool Changed = false;
- // Masked pseudos coming out of isel will have their mask operand in the form:
- //
- // $v0:vr = COPY %mask:vr
- // %x:vr = Pseudo_MASK %a:vr, %b:br, $v0:vr
- //
- // Because $v0 isn't in SSA, keep track of its definition at each use so we
- // can check mask operands.
- for (const MachineBasicBlock &MBB : MF) {
- const MachineInstr *CurrentV0Def = nullptr;
- for (const MachineInstr &MI : MBB) {
- if (MI.readsRegister(RISCV::V0, TRI))
- V0Defs[&MI] = CurrentV0Def;
-
- if (MI.definesRegister(RISCV::V0, TRI))
- CurrentV0Def = &MI;
- }
- }
-
for (MachineBasicBlock &MBB : MF) {
for (MachineInstr &MI : MBB) {
Changed |= convertToUnmasked(MI);
diff --git a/llvm/lib/Target/RISCV/RISCVISelDAGToDAG.cpp b/llvm/lib/Target/RISCV/RISCVISelDAGToDAG.cpp
index f99dc0b8576368..81543768d6a646 100644
--- a/llvm/lib/Target/RISCV/RISCVISelDAGToDAG.cpp
+++ b/llvm/lib/Target/RISCV/RISCVISelDAGToDAG.cpp
@@ -296,7 +296,6 @@ void RISCVDAGToDAGISel::addVectorLoadStoreOperands(
bool IsMasked, bool IsStridedOrIndexed, SmallVectorImpl<SDValue> &Operands,
bool IsLoad, MVT *IndexVT) {
SDValue Chain = Node->getOperand(0);
- SDValue Glue;
Operands.push_back(Node->getOperand(CurOp++)); // Base pointer.
@@ -307,11 +306,8 @@ void RISCVDAGToDAGISel::addVectorLoadStoreOperands(
}
if (IsMasked) {
- // Mask needs to be copied to V0.
SDValue Mask = Node->getOperand(CurOp++);
- Chain = CurDAG->getCopyToReg(Chain, DL, RISCV::V0, Mask, SDValue());
- Glue = Chain.getValue(1);
- Operands.push_back(CurDAG->getRegister(RISCV::V0, Mask.getValueType()));
+ Operands.push_back(Mask);
}
SDValue VL;
selectVLOp(Node->getOperand(CurOp++), VL);
@@ -333,8 +329,6 @@ void RISCVDAGToDAGISel::addVectorLoadStoreOperands(
}
Operands.push_back(Chain); // Chain.
- if (Glue)
- Operands.push_back(Glue);
}
void RISCVDAGToDAGISel::selectVLSEG(SDNode *Node, bool IsMasked,
@@ -1670,12 +1664,6 @@ void RISCVDAGToDAGISel::Select(SDNode *Node) {
return;
}
- // Mask needs to be copied to V0.
- SDValue Chain = CurDAG->getCopyToReg(CurDAG->getEntryNode(), DL,
- RISCV::V0, Mask, SDValue());
- SDValue Glue = Chain.getValue(1);
- SDValue V0 = CurDAG->getRegister(RISCV::V0, VT);
-
// Otherwise use
// vmslt{u}.vx vd, va, x, v0.t; vmxor.mm vd, vd, v0
// The result is mask undisturbed.
@@ -1683,7 +1671,7 @@ void RISCVDAGToDAGISel::Select(SDNode *Node) {
// the agnostic result can be either undisturbed or all 1.
SDValue Cmp = SDValue(
CurDAG->getMachineNode(VMSLTMaskOpcode, DL, VT,
- {MaskedOff, Src1, Src2, V0, VL, SEW, Glue}),
+ {MaskedOff, Src1, Src2, Mask, VL, SEW}),
0);
// vmxor.mm vd, vd, v0 is used to update active value.
ReplaceNode(Node, CurDAG->getMachineNode(VMXOROpcode, DL, VT,
@@ -3426,32 +3414,7 @@ bool RISCVDAGToDAGISel::doPeepholeSExtW(SDNode *N) {
return false;
}
-static bool usesAllOnesMask(SDValue MaskOp, SDValue GlueOp) {
- // Check that we're using V0 as a mask register.
- if (!isa<RegisterSDNode>(MaskOp) ||
- cast<RegisterSDNode>(MaskOp)->getReg() != RISCV::V0)
- return false;
-
- // The glued user defines V0.
- const auto *Glued = GlueOp.getNode();
-
- if (!Glued || Glued->getOpcode() != ISD::CopyToReg)
- return false;
-
- // Check that we're defining V0 as a mask register.
- if (!isa<RegisterSDNode>(Glued->getOperand(1)) ||
- cast<RegisterSDNode>(Glued->getOperand(1))->getReg() != RISCV::V0)
- return false;
-
- // Check the instruction defining V0; it needs to be a VMSET pseudo.
- SDValue MaskSetter = Glued->getOperand(2);
-
- // Sometimes the VMSET is wrapped in a COPY_TO_REGCLASS, e.g. if the mask came
- // from an extract_subvector or insert_subvector.
- if (MaskSetter->isMachineOpcode() &&
- MaskSetter->getMachineOpcode() == RISCV::COPY_TO_REGCLASS)
- MaskSetter = MaskSetter->getOperand(0);
-
+static bool usesAllOnesMask(SDValue MaskOp) {
const auto IsVMSet = [](unsigned Opc) {
return Opc == RISCV::PseudoVMSET_M_B1 || Opc == RISCV::PseudoVMSET_M_B16 ||
Opc == RISCV::PseudoVMSET_M_B2 || Opc == RISCV::PseudoVMSET_M_B32 ||
@@ -3462,14 +3425,13 @@ static bool usesAllOnesMask(SDValue MaskOp, SDValue GlueOp) {
// TODO: Check that the VMSET is the expected bitwidth? The pseudo has
// undefined behaviour if it's the wrong bitwidth, so we could choose to
// assume that it's all-ones? Same applies to its VL.
- return MaskSetter->isMachineOpcode() &&
- IsVMSet(MaskSetter.getMachineOpcode());
+ return MaskOp->isMachineOpcode() &&
+ IsVMSet(MaskOp.getMachineOpcode());
}
// Return true if we can make sure mask of N is all-ones mask.
static bool usesAllOnesMask(SDNode *N, unsigned MaskOpIdx) {
- return usesAllOnesMask(N->getOperand(MaskOpIdx),
- N->getOperand(N->getNumOperands() - 1));
+ return usesAllOnesMask(N->getOperand(MaskOpIdx));
}
static bool isImplicitDef(SDValue V) {
@@ -3515,10 +3477,10 @@ bool RISCVDAGToDAGISel::doPeepholeMaskedRVV(MachineSDNode *N) {
Ops.push_back(Op);
}
- // Transitively apply any node glued to our new node.
- const auto *Glued = N->getGluedNode();
- if (auto *TGlued = Glued->getGluedNode())
- Ops.push_back(SDValue(TGlued, TGlued->getNumValues() - 1));
+ // // Transitively apply any node glued to our new node.
+ // const auto *Glued = N->getGluedNode();
+ // if (auto *TGlued = Glued->getGluedNode())
+ // Ops.push_back(SDValue(TGlued, TGlued->getNumValues() - 1));
MachineSDNode *Result =
CurDAG->getMachineNode(Opc, SDLoc(N), N->getVTList(), Ops);
@@ -3584,7 +3546,7 @@ static unsigned GetVMSetForLMul(RISCVII::VLMUL LMUL) {
// The resulting policy is the effective policy the vmerge would have had,
// i.e. whether or not it's merge operand was implicit-def.
bool RISCVDAGToDAGISel::performCombineVMergeAndVOps(SDNode *N) {
- SDValue Merge, False, True, VL, Mask, Glue;
+ SDValue Merge, False, True, VL, Mask;
// A vmv.v.v is equivalent to a vmerge with an all-ones mask.
if (IsVMv(N)) {
Merge = N->getOperand(0);
@@ -3600,11 +3562,7 @@ bool RISCVDAGToDAGISel::performCombineVMergeAndVOps(SDNode *N) {
True = N->getOperand(2);
Mask = N->getOperand(3);
VL = N->getOperand(4);
- // We always have a glue node for the mask at v0.
- Glue = N->getOperand(N->getNumOperands() - 1);
}
- assert(!Mask || cast<RegisterSDNode>(Mask)->getReg() == RISCV::V0);
- assert(!Glue || Glue.getValueType() == MVT::Glue);
// We require that either merge and false are the same, or that merge
// is undefined.
@@ -3639,7 +3597,7 @@ bool RISCVDAGToDAGISel::performCombineVMergeAndVOps(SDNode *N) {
// When Mask is not a true mask, this transformation is illegal for some
// operations whose results are affected by mask, like viota.m.
- if (Info->MaskAffectsResult && Mask && !usesAllOnesMask(Mask, Glue))
+ if (Info->MaskAffectsResult && Mask && !usesAllOnesMask(Mask))
return false;
// If True has a merge operand then it needs to be the same as vmerge's False,
@@ -3664,7 +3622,7 @@ bool RISCVDAGToDAGISel::performCombineVMergeAndVOps(SDNode *N) {
return false;
// FIXME: Support mask agnostic True instruction which would have an
// undef merge operand.
- if (Mask && !usesAllOnesMask(Mask, Glue))
+ if (Mask && !usesAllOnesMask(Mask))
return false;
}
@@ -3691,8 +3649,6 @@ bool RISCVDAGToDAGISel::performCombineVMergeAndVOps(SDNode *N) {
if (Mask)
LoopWorklist.push_back(Mask.getNode());
LoopWorklist.push_back(VL.getNode());
- if (Glue)
- LoopWorklist.push_back(Glue.getNode());
if (SDNode::hasPredecessorHelper(True.getNode(), Visited, LoopWorklist))
return false;
}
@@ -3737,11 +3693,8 @@ bool RISCVDAGToDAGISel::performCombineVMergeAndVOps(SDNode *N) {
// From the preconditions we checked above, we know the mask and thus glue
// for the result node will be taken from True.
- if (IsMasked) {
+ if (IsMasked)
Mask = True->getOperand(Info->MaskOpIdx);
- Glue = True->getOperand(True->getNumOperands() - 1);
- assert(Glue.getValueType() == MVT::Glue);
- }
// If we end up using the vmerge mask the vmerge is actually a vmv.v.v, create
// an all-ones mask to use.
else if (IsVMv(N)) {
@@ -3749,13 +3702,7 @@ bool RISCVDAGToDAGISel::performCombineVMergeAndVOps(SDNode *N) {
unsigned VMSetOpc = GetVMSetForLMul(RISCVII::getLMul(TSFlags));
ElementCount EC = N->getValueType(0).getVectorElementCount();
MVT MaskVT = MVT::getVectorVT(MVT::i1, EC);
-
- SDValue AllOnesMask =
- SDValue(CurDAG->getMachineNode(VMSetOpc, DL, MaskVT, VL, SEW), 0);
- SDValue MaskCopy = CurDAG->getCopyToReg(CurDAG->getEntryNode(), DL,
- RISCV::V0, AllOnesMask, SDValue());
- Mask = CurDAG->getRegister(RISCV::V0, MaskVT);
- Glue = MaskCopy.getValue(1);
+ Mask = SDValue(CurDAG->getMachineNode(VMSetOpc, DL, MaskVT, VL, SEW), 0);
}
unsigned MaskedOpc = Info->MaskedPseudo;
@@ -3806,9 +3753,6 @@ bool RISCVDAGToDAGISel::performCombineVMergeAndVOps(SDNode *N) {
if (HasChainOp)
Ops.push_back(True.getOperand(TrueChainOpIdx));
- // Add the glue for the CopyToReg of mask->v0.
- Ops.push_back(Glue);
-
MachineSDNode *Result =
CurDAG->getMachineNode(MaskedOpc, DL, True->getVTList(), Ops);
Result->setFlags(True->getFlags());
diff --git a/llvm/lib/Target/RISCV/RISCVInstrInfo.h b/llvm/lib/Target/RISCV/RISCVInstrInfo.h
index 70fe7da85be0e7..804b9e8ffb2fdd 100644
--- a/llvm/lib/Target/RISCV/RISCVInstrInfo.h
+++ b/llvm/lib/Target/RISCV/RISCVInstrInfo.h
@@ -279,6 +279,7 @@ class RISCVInstrInfo : public RISCVGenInstrInfo {
unsigned getUndefInitOpcode(unsigned RegClassID) const override {
switch (RegClassID) {
+ case RISCV::VMV0RegClassID:
case RISCV::VRRegClassID:
return RISCV::PseudoRVVInitUndefM1;
case RISCV::VRM2RegClassID:
diff --git a/llvm/lib/Target/RISCV/RISCVInstrInfoVPseudos.td b/llvm/lib/Target/RISCV/RISCVInstrInfoVPseudos.td
index cf9a31c23a06e0..8f5467ff8604cc 100644
--- a/llvm/lib/Target/RISCV/RISCVInstrInfoVPseudos.td
+++ b/llvm/lib/Target/RISCV/RISCVInstrInfoVPseudos.td
@@ -3972,12 +3972,12 @@ class VPatUnaryMask<string intrinsic_name,
Pat<(result_type (!cast<Intrinsic>(intrinsic_name#"_mask")
(result_type result_reg_class:$merge),
(op2_type op2_reg_class:$rs2),
- (mask_type V0),
+ (mask_type VMV0:$vm),
VLOpFrag, (XLenVT timm:$policy))),
(!cast<Instruction>(inst#"_"#kind#"_"#vlmul.MX#"_MASK")
(result_type result_reg_class:$merge),
(op2_type op2_reg_class:$rs2),
- (mask_type V0), GPR:$vl, log2sew, (XLenVT timm:$policy))>;
+ (mask_type VMV0:$vm), GPR:$vl, log2sew, (XLenVT timm:$policy))>;
class VPatUnaryMaskRoundingMode<string intrinsic_name,
string inst,
@@ -3993,7 +3993,7 @@ class VPatUnaryMaskRoundingMode<string intrinsic_name,
Pat<(result_type (!cast<Intrinsic>(intrinsic_name#"_mask")
(result_type result_reg_class:$merge),
(op2_type op2_reg_class:$rs2),
- (mask_type V0),
+ (mask_type VMV0:$vm),
(XLenVT timm:$round),
VLOpFrag, (XLenVT timm:$policy))),
(!cast<Instruction>(
@@ -4002,7 +4002,7 @@ class VPatUnaryMaskRoundingMode<string intrinsic_name,
inst#"_"#kind#"_"#vlmul.MX#"_MASK"))
(result_type result_reg_class:$merge),
(op2_type op2_reg_class:$rs2),
- (mask_type V0),
+ (mask_type VMV0:$vm),
(XLenVT timm:$round),
GPR:$vl, log2sew, (XLenVT timm:$policy))>;
@@ -4024,12 +4024,12 @@ class VPatMaskUnaryMask<string intrinsic_name,
Pat<(mti.Mask (!cast<Intrinsic>(intrinsic_name#"_mask")
(mti.Mask VR:$merge),
(mti.Mask VR:$rs2),
- (mti.Mask V0),
+ (mti.Mask VMV0:$vm),
VLOpFrag)),
(!cast<Instruction>(inst#"_M_"#mti.BX#"_MASK")
(mti.Mask VR:$merge),
(mti.Mask VR:$rs2),
- (mti.Mask V0), GPR:$vl, mti.Log2SEW, TU_MU)>;
+ (mti.Mask VMV0:$vm), GPR:$vl, mti.Log2SEW, TU_MU)>;
class VPatUnaryAnyMask<string intrinsic,
string inst,
@@ -4165,13 +4165,13 @@ class VPatBinaryMask<string intrinsic_name,
(result_type result_reg_class:$merge),
(op1_type op1_reg_class:$rs1),
(op2_type op2_kind:$rs2),
- (mask_type V0),
+ (mask_type VMV0:$vm),
VLOpFrag)),
(!cast<Instruction>(inst#"_MASK")
(result_type result_reg_class:$merge),
(op1_type op1_reg_class:$rs1),
(op2_type op2_kind:$rs2),
- (mask_type V0), GPR:$vl, sew)>;
+ (mask_type VMV0:$vm), GPR:$vl, sew)>;
class VPatBinaryMaskTA<string intrinsic_name,
string inst,
@@ -4187,13 +4187,13 @@ class VPatBinaryMaskTA<string intrinsic_name,
(result_type result_reg_class:$merge),
(op1_type op1_reg_class:$rs1),
(op2_type op2_kind:$rs2),
- (mask_type V0),
+ (mask_type VMV0:$vm),
VLOpFrag, (XLenVT timm:$policy))),
(!cast<Instruction>(inst#"_MASK")
(result_type result_reg_class:$merge),
(op1_type op1_reg_class:$rs1),
(op2_type op2_kind:$rs2),
- (mask_type V0), GPR:$vl, sew, (XLenVT timm:$policy))>;
+ (mask_type VMV0:$vm), GPR:$vl, sew, (XLenVT timm:$policy))>;
class VPatBinaryMaskTARoundingMode<string intrinsic_name,
string inst,
@@ -4209,14 +4209,14 @@ class VPatBinaryMaskTARoundingMode<string intrinsic_name,
(result_type result_reg_class:$merge),
(op1_type op1_reg_class:$rs1),
(op2_type op2_kind:$rs2),
- (mask_type V0),
+ (mask_type VMV0:$vm),
(XLenVT timm:$round),
VLOpFrag, (XLenVT timm:$policy))),
(!cast<Instruction>(inst#"_MASK")
(result_type result_reg_class:$merge),
(op1_type op1_reg_class:$rs1),
(op2_type op2_kind:$rs2),
- (mask_type V0),
+ (mask_type VMV0:$vm),
(XLenVT timm:$round),
GPR:$vl, sew, (XLenVT timm:$policy))>;
@@ -4235,13 +4235,13 @@ class VPatBinaryMaskSwapped<string intrinsic_name,
(result_type result_reg_class:$merge),
(op2_type op2_kind:$rs2),
(op1_type op1_reg_class:$rs1),
- (mask_type V0),
+ (mask_type VMV0:$vm),
VLOpFrag)),
(!cast<Instruction>(inst#"_MASK")
(result_type result_reg_class:$merge),
(op1_type op1_reg_class:$rs1),
(op2_type op2_kind:$rs2),
- (mask_type V0), GPR:$vl, sew)>;
+ (mask_type VMV0:$vm), GPR:$vl, sew)>;
class VPatTiedBinaryNoMask<string intrinsic_name,
string inst,
@@ -4327,12 +4327,12 @@ class VPatTiedBinaryMask<string intrinsic_name,
(result_type result_reg_class:$merge),
(result_type result_reg_class:$merge),
(op2_type op2_kind:$rs2),
- (mask_type V0),
+ (mask_type VMV0:$vm),
VLOpFrag, (XLenVT timm:$policy))),
(!cast<Instruction>(inst#"_MASK_TIED")
(result_type result_reg_class:$merge),
(op2_type op2_kind:$rs2),
- (mask_type V0), GPR:$vl, sew, (XLenVT timm:$policy))>;
+ (mask_type VMV0:$vm), GPR:$vl, sew, (XLenVT timm:$policy))>;
class VPatTiedBinaryMaskRoundingMode<string intrinsic_name,
string inst,
@@ -4346,13 +4346,13 @@ class VPatTiedBinaryMaskRoundingMode<string intrinsic_name,
(result_type result_reg_class:$merge),
(result_type result_reg_class:$merge),
(op2_type op2_kind:$rs2),
- (mask_type V0),
+ (mask_type VMV0:$vm),
(XLenVT timm:$round),
VLOpFrag, (XLenVT timm:$policy))),
(!cast<Instruction>(inst#"_MASK_TIED")
(result_type result_reg_class:$merge),
(op2_type op2_kind:$rs2),
- (mask_type V0),
+ (mask_type VMV0:$vm),
(XLenVT timm:$round),
GPR:$vl, sew, (XLenVT timm:$policy))>;
@@ -4486,13 +4486,13 @@ class VPatTer...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/88496
More information about the llvm-commits
mailing list