[llvm-branch-commits] [llvm] [AMDGPU] Measure MFMA hazard windows at their own producers (PR #225989)

[Michael Halkenhäuser via llvm-branch-commits]

================
@@ -2557,3 +2557,139 @@ body:             |
     S_NOP 2
     $vgpr0 = V_MOV_B32_e32 1, implicit $exec
 ...
+
+# Window and distance must come from one producer: srcA after a 16-pass XDL
+# needs NumPasses + 3, one more on gfx950, after a 2-pass non-XDL NumPasses + 2.
+
+# GCN-LABEL: name: mfma_srca_read_mixed_xdl_and_smfma
+# GCN:      bb.3:
+# GCN-NEXT: S_NOP 15
+# GFX942-NEXT: S_NOP 1
+# GFX950-NEXT: S_NOP 2
+# GCN-NEXT: $vgpr40_vgpr41_vgpr42_vgpr43 = V_MFMA_F32_4X4X1F32
+name:            mfma_srca_read_mixed_xdl_and_smfma
+body:             |
+  bb.0:
+    successors: %bb.1, %bb.2
+    S_CBRANCH_SCC1 %bb.2, implicit $scc
+    S_BRANCH %bb.1
+
+  bb.1:
+    successors: %bb.3
+    $vgpr0_vgpr1_vgpr2_vgpr3_vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17_vgpr18_vgpr19_vgpr20_vgpr21_vgpr22_vgpr23_vgpr24_vgpr25_vgpr26_vgpr27_vgpr28_vgpr29_vgpr30_vgpr31 = V_MFMA_F32_32X32X4F16_vgprcd_e64 $vgpr34_vgpr35, $vgpr36_vgpr37, $vgpr0_vgpr1_vgpr2_vgpr3_vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17_vgpr18_vgpr19_vgpr20_vgpr21_vgpr22_vgpr23_vgpr24_vgpr25_vgpr26_vgpr27_vgpr28_vgpr29_vgpr30_vgpr31, 0, 0, 0, implicit $mode, implicit $exec
+    S_BRANCH %bb.3
+
+  bb.2:
+    successors: %bb.3
+    $vgpr0_vgpr1_vgpr2_vgpr3 = V_MFMA_F32_4X4X1F32_vgprcd_e64 $vgpr38, $vgpr39, $vgpr0_vgpr1_vgpr2_vgpr3, 0, 0, 0, implicit $mode, implicit $exec
+    S_BRANCH %bb.3
+
+  bb.3:
+    $vgpr40_vgpr41_vgpr42_vgpr43 = V_MFMA_F32_4X4X1F32_vgprcd_e64 $vgpr0, $vgpr39, $vgpr40_vgpr41_vgpr42_vgpr43, 0, 0, 0, implicit $mode, implicit $exec
+    S_ENDPGM 0
+...
+
+# The same for a srcC read, whose windows come from another table: 8-pass XDL
+# needs NumPasses + 1, one more on gfx950, the 2-pass non-XDL one 2.
+
+# GCN-LABEL: name: mfma_srcc_read_mixed_xdl_and_smfma
+# GCN:      bb.3:
+# GFX942-NEXT: S_NOP 7
+# GFX950-NEXT: S_NOP 8
+# GCN-NEXT: $agpr20_agpr21_agpr22_agpr23 = V_MFMA_F32_4X4X1F32
+name:            mfma_srcc_read_mixed_xdl_and_smfma
+body:             |
+  bb.0:
+    successors: %bb.1, %bb.2
+    S_CBRANCH_SCC1 %bb.2, implicit $scc
+    S_BRANCH %bb.1
+
+  bb.1:
+    successors: %bb.3
+    $agpr0_agpr1_agpr2_agpr3_agpr4_agpr5_agpr6_agpr7_agpr8_agpr9_agpr10_agpr11_agpr12_agpr13_agpr14_agpr15 = V_MFMA_F32_32X32X8F16_e64 $vgpr2_vgpr3, $vgpr4_vgpr5, $agpr0_agpr1_agpr2_agpr3_agpr4_agpr5_agpr6_agpr7_agpr8_agpr9_agpr10_agpr11_agpr12_agpr13_agpr14_agpr15, 0, 0, 0, implicit $mode, implicit $exec
+    S_BRANCH %bb.3
+
+  bb.2:
+    successors: %bb.3
+    $agpr0_agpr1_agpr2_agpr3 = V_MFMA_F32_4X4X1F32_e64 $vgpr1, $vgpr0, $agpr0_agpr1_agpr2_agpr3, 0, 0, 0, implicit $mode, implicit $exec
+    S_BRANCH %bb.3
+
+  bb.3:
+    $agpr20_agpr21_agpr22_agpr23 = V_MFMA_F32_4X4X1F32_e64 $vgpr1, $vgpr0, $agpr0_agpr1_agpr2_agpr3, 0, 0, 0, implicit $mode, implicit $exec
+    S_ENDPGM 0
+...
+# A 4-pass producer writing exactly the srcC register asks nothing and must not
+# end the search: the 8-pass XDL result 2 away needs 9, 10 on gfx950.
----------------
mhalk wrote:

Good catch. This comment needs to be improved.
"needs 9" is referring to the corresponding window and "2 away" -> 1 slot needs to be deducted.
Re-reading this: it is not clear (enough).

https://github.com/llvm/llvm-project/pull/225989