[llvm-branch-commits] [llvm] [AMDGPU] Lower loads and stores for address space 13 (PR #209541)

[Matt Arsenault via llvm-branch-commits]

================
@@ -1061,6 +1061,105 @@ def SI_INDIRECT_DST_V32 : SI_INDIRECT_DST<VReg_1024>;
 
 } // End Uses = [EXEC], Defs = [M0, EXEC]
 
+//===----------------------------------------------------------------------===//
+// VGPR "as memory" indexed load/store pseudos (address space 13)
+//===----------------------------------------------------------------------===//
+
+// V_LOAD_IDX_B<N> / V_STORE_IDX_B<N> load or store N bits from/to the wave's
+// view of its vector registers, at a dword index of ($idx + $offset). $idx is
+// a 32-bit value that may be uniform (SGPR) or divergent (VGPR); $offset is a
+// constant dword offset folded in at selection.
+//
+// AMDGPULowerVGPREncoding lowers each into an M0-relative move over the wave's
+// vector registers: v_movrels_b32 (load) / v_movreld_b32 (store) where the
+// subtarget has movrel, and a v_mov_b32 under the VGPR indexing mode otherwise.
+// It writes $idx to M0 there, beside the move that reads it.
+
+// Populates the VLdStIdxOpcodeInfo searchable table (see AMDGPUMachineInstrs.h
+// and AMDGPU::getVLdStIdxOpcodeInfo*), mapping each pseudo to its bit width and
+// load/store direction.
+class VLdStIdxOpcodeInfo<int size, bit isStore> {
+  Instruction Opcode = !cast<Instruction>(NAME);
+  bits<12> BitWidth = size;
+  bit IsStore = isStore;
+}
+
+foreach rc = [VGPR_32, VReg_64, VReg_96, VReg_128, VReg_160, VReg_192,
+              VReg_224, VReg_256, VReg_288, VReg_320, VReg_352, VReg_384,
+              VReg_512, VReg_1024] in {
+  // The units of $idx and $offset are in dwords.
+  //
+  // The index reaches the hardware through M0, so these define it: either by
+  // writing M0 for v_movrel[sd], or through s_set_gpr_idx_on where the subtarget
+  // indexes with the VGPR indexing mode. Declaring that here also keeps a
+  // divergent-index access pinned inside its waterfall loop, since an
+  // instruction defining a physical register is not hoisted or sunk.
+  def V_LOAD_IDX_B#rc.Size : VPseudoInstSI <
+    (outs rc:$data),
+    (ins SReg_32:$idx, i32imm:$offset)>,
+    VLdStIdxOpcodeInfo<rc.Size, 0> {
+      let mayLoad = 1;
+      let VALU = 1;
+      let UseNamedOperandTable = 1;
+      let hasSideEffects = 0;
+      let Defs = [M0];
+  }
+  def V_STORE_IDX_B#rc.Size : VPseudoInstSI <
+    (outs),
+    (ins rc:$data, SReg_32:$idx, i32imm:$offset)>,
+    VLdStIdxOpcodeInfo<rc.Size, 1> {
+      let mayStore = 1;
+      let VALU = 1;
+      let UseNamedOperandTable = 1;
+      let hasSideEffects = 0;
+      let Defs = [M0];
+  }
+}
+
+// Copies of v_movrel[sd]_b32 for the moves the pseudos above expand into.
+//
+// The storage a movrel indexes is normally a register tuple, and the machine
+// verifier requires an implicit use of that tuple naming the register the
+// instruction really touches. Here the storage is the whole register file,
+// which no operand can name, so these carry their own opcodes and leave those
+// rules to the tuple form. AMDGPUMCInstLower maps them back for encoding.
+let VALU = 1, VOP1 = 1, Uses = [M0, EXEC], hasSideEffects = 0,
----------------
arsenm wrote:

The VALU should be redundant with using VPseudoInstSI

https://github.com/llvm/llvm-project/pull/209541