[llvm] [AMDGPU] Add initial version of gfx950 MFMA co-exec rules (PR #214682)

[Alexey Sachkov via llvm-commits]

================
@@ -346,12 +346,180 @@ inline CoExecInfo CoExecInfo::build(unsigned TotalWindow, const char *Pattern) {
   return Info;
 }
 
+/// Get co-execution info for a gfx950 MFMA instruction.
+/// The occupancy (cycles until the next MFMA may issue) is expressed as the
+/// first stage carrying the WMMA bit.
+inline CoExecInfo getMFMACoExecInfo(const MachineInstr &MI) {
+  CoExecInfo Res;
+  for (unsigned I = 0; I < MaxCoExecStages; ++I)
+    Res.Slots[I].Mask = CoExecMask::None;
+
+  // TODO: Implement proper patterns support (for debugging purposes).
+  // Existing pattern letters are WMMA-specific and will probably be confusing
+  // if used as-is for MFMA. Inventing new MFMA-specific letters is an option,
+  // but perhaps the pattern should be instead dynamically reconstructed when
+  // needed by printing specific slots in full instead of a key for them.
+  Res.Pattern = "undefinedundefinedundefinedundefined";
+
+  // MFMA co-exec slots are incremental, i.e. for every slot N it supports all
+  // instructions which were supported by the previous slot N-1 and may support
+  // something extra.
+  auto AllowCoExec = [](CoExecInfo &Info, CoExecMaskT ExtraBits,
+                        unsigned StartIndex) {
+    for (unsigned Index = StartIndex; Index < Info.TotalWindow; ++Index)
+      Info.Slots[Index].Mask |= ExtraBits;
+  };
+
+  switch (MI.getOpcode()) {
+  // 4-cycle occupancy, 8-cycle window.
+  case V_MFMA_F32_16X16X128_F8F6F4_f4_f4_e64:
+  case V_MFMA_F32_16X16X128_F8F6F4_f4_f4_vgprcd_e64:
+  case V_MFMA_F32_16X16X128_F8F6F4_f4_f6_e64:
+  case V_MFMA_F32_16X16X128_F8F6F4_f4_f6_vgprcd_e64:
+  case V_MFMA_F32_16X16X128_F8F6F4_f6_f4_e64:
+  case V_MFMA_F32_16X16X128_F8F6F4_f6_f4_vgprcd_e64:
+  case V_MFMA_F32_16X16X128_F8F6F4_f6_f6_e64:
+  case V_MFMA_F32_16X16X128_F8F6F4_f6_f6_vgprcd_e64:
+  case V_MFMA_F32_16X16X32_BF16_e64:
+  case V_MFMA_F32_16X16X32_BF16_vgprcd_e64:
+  case V_MFMA_I32_16X16X64_I8_e64:
+  case V_MFMA_I32_16X16X64_I8_vgprcd_e64:
+  case V_MFMA_F32_16X16X32_F16_e64:
+  case V_MFMA_F32_16X16X32_F16_vgprcd_e64:
+    // Distinction marker to simplify mapping to the programming guide
+  case V_SMFMAC_F32_16X16X64_BF16_e64:
+  case V_SMFMAC_I32_16X16X128_I8_e64:
+  case V_SMFMAC_F32_16X16X128_BF8_BF8_e64:
+  case V_SMFMAC_F32_16X16X128_BF8_FP8_e64:
+  case V_SMFMAC_F32_16X16X128_FP8_BF8_e64:
+  case V_SMFMAC_F32_16X16X128_FP8_FP8_e64:
+  case V_SMFMAC_F32_16X16X64_F16_e64:
+    Res.TotalWindow = 8;
+    AllowCoExec(Res, CoExecMask::SALU, 1);
+    AllowCoExec(Res, CoExecMask::DS | CoExecMask::VALU, 2);
----------------
AlexeySachkov wrote:

That is correct, the proposed model is simplified. For example, there are also different kinds of MFMA and they can only co-execute at different stages, whilst for now we consider them all to be the same.

The strategy for now is to start with some baseline and then fine-tune where we see a benefit.

https://github.com/llvm/llvm-project/pull/214682