[llvm] 2dfb4b5 - [AMDGPU] Fix incorrect hazard mitigation

[Stephen Thomas via llvm-commits]

Author: Stephen Thomas
Date: 2023-07-04T14:42:51+01:00
New Revision: 2dfb4b56fe3c233b4f278a6d17b4ec5780007a8f

URL: https://github.com/llvm/llvm-project/commit/2dfb4b56fe3c233b4f278a6d17b4ec5780007a8f
DIFF: https://github.com/llvm/llvm-project/commit/2dfb4b56fe3c233b4f278a6d17b4ec5780007a8f.diff

LOG: [AMDGPU] Fix incorrect hazard mitigation

GCNHazardRecognizer::fixVcmpxExecWARHazard() mitigates a specific hazard
by inserting a wait on sa_sdst==0 if such a wait isn't already present.
Unfortunately, the check for an existing wait incorrectly checks for one
that doesn't actually care about sa_sdst itself, but requires that no
other counters are waited for.

Once the check is performed correctly, a lit test needs to be updated,
since it is currently testing for the incorrect behaviour.

Differential Revision: https://reviews.llvm.org/D154438

Added: 
    

Modified: 
    llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
    llvm/test/CodeGen/AMDGPU/vcmpx-exec-war-hazard.mir

Removed: 
    


################################################################################
diff  --git a/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp b/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
index 59dd1cd6f93c42..2d53b2a70dbe05 100644
--- a/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
@@ -1293,8 +1293,7 @@ bool GCNHazardRecognizer::fixVcmpxExecWARHazard(MachineInstr *MI) {
           return true;
     }
     if (MI.getOpcode() == AMDGPU::S_WAITCNT_DEPCTR &&
-        AMDGPU::DepCtr::encodeFieldSaSdst(MI.getOperand(0).getImm(), 0) ==
-            0xfffe)
+        AMDGPU::DepCtr::decodeFieldSaSdst(MI.getOperand(0).getImm()) == 0)
       return true;
     return false;
   };

diff  --git a/llvm/test/CodeGen/AMDGPU/vcmpx-exec-war-hazard.mir b/llvm/test/CodeGen/AMDGPU/vcmpx-exec-war-hazard.mir
index 7a802ed1cac2a6..b5ab6af991244e 100644
--- a/llvm/test/CodeGen/AMDGPU/vcmpx-exec-war-hazard.mir
+++ b/llvm/test/CodeGen/AMDGPU/vcmpx-exec-war-hazard.mir
@@ -126,12 +126,13 @@ body:            |
     S_ENDPGM 0
 ...
 
-# GCN-LABEL: name: no_hazard_vcmpx_smov_exec_lo_depctr_ffff
-# GCN:      $sgpr0 = S_MOV_B32 $exec_lo
-# GCN-NEXT: S_WAITCNT_DEPCTR 65535
-# GCN-NEXT: V_CMPX_LE_F32_nosdst_e32
+# GCN-LABEL: name: hazard_vcmpx_smov_exec_lo_depctr_ffff
+# GCN:        $sgpr0 = S_MOV_B32 $exec_lo
+# GCN-NEXT:   S_WAITCNT_DEPCTR 65535
+# GFX10-NEXT: S_WAITCNT_DEPCTR 65534
+# GCN-NEXT:   V_CMPX_LE_F32_nosdst_e32
 ---
-name:            no_hazard_vcmpx_smov_exec_lo_depctr_ffff
+name:            hazard_vcmpx_smov_exec_lo_depctr_ffff
 body:            |
   bb.0:
     successors: %bb.1
@@ -148,7 +149,6 @@ body:            |
 # GCN-LABEL: name: hazard_vcmpx_smov_exec_lo_depctr_effe
 # GCN:        $sgpr0 = S_MOV_B32 $exec_lo
 # GCN-NEXT:   S_WAITCNT_DEPCTR 61438
-# GFX10-NEXT: S_WAITCNT_DEPCTR 65534
 # GCN-NEXT:   V_CMPX_LE_F32_nosdst_e32
 ---
 name:            hazard_vcmpx_smov_exec_lo_depctr_effe