[PATCH] D104219: [AMDGPU] Limit runs of fixLdsBranchVmemWARHazard

[Piotr Sobczak via Phabricator via llvm-commits]

piotr created this revision.
Herald added subscribers: foad, kerbowa, hiraditya, t-tye, tpr, dstuttard, yaxunl, nhaehnle, jvesely, kzhuravl, arsenm.
piotr requested review of this revision.
Herald added subscribers: llvm-commits, wdng.
Herald added a project: LLVM.

The code in fixLdsBranchVmemWARHazard looks for patterns of a vmem/lds
access followed by a branch, followed by an lds/vmem access.

The handling of the hazard requires an arbitrary number of instructions
to process. In the worst case where a function has a vmem access, but no lds
accesses, all instructions are examined only to conclude that the hazard
cannot occur.

Add the pre-processing stage which detects if there is both lds and vmem
present in the function and only then does the more costly search.

This patch significantly improves compilation time in the cases the hazard
cannot happen. In one pathological case I looked at IsHazardInst is needlesly
called 88.6 milions times.

The numbers could also be improved by introducing a map around the
inner calls to ::getWaitStatesSince in fixLdsBranchVmemWARHazard, but
nothing will beat not running fixLdsBranchVmemWARHazard at all in the cases
detected by shouldRunLdsBranchVmemWARHazardFixup().


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D104219

Files:
  llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
  llvm/lib/Target/AMDGPU/GCNHazardRecognizer.h


Index: llvm/lib/Target/AMDGPU/GCNHazardRecognizer.h
===================================================================
--- llvm/lib/Target/AMDGPU/GCNHazardRecognizer.h
+++ llvm/lib/Target/AMDGPU/GCNHazardRecognizer.h
@@ -48,6 +48,7 @@
   const SIInstrInfo &TII;
   const SIRegisterInfo &TRI;
   TargetSchedModel TSchedModel;
+  bool RunLdsBranchVmemWARHazardFixup;
 
   /// RegUnits of uses in the current soft memory clause.
   BitVector ClauseUses;
Index: llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
===================================================================
--- llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
+++ llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
@@ -23,6 +23,9 @@
 // Hazard Recoginizer Implementation
 //===----------------------------------------------------------------------===//
 
+static bool shouldRunLdsBranchVmemWARHazardFixup(const MachineFunction &MF,
+                                                 const GCNSubtarget &ST);
+
 GCNHazardRecognizer::GCNHazardRecognizer(const MachineFunction &MF) :
   IsHazardRecognizerMode(false),
   CurrCycleInstr(nullptr),
@@ -34,6 +37,7 @@
   ClauseDefs(TRI.getNumRegUnits()) {
   MaxLookAhead = MF.getRegInfo().isPhysRegUsed(AMDGPU::AGPR0) ? 19 : 5;
   TSchedModel.init(&ST);
+  RunLdsBranchVmemWARHazardFixup = shouldRunLdsBranchVmemWARHazardFixup(MF, ST);
 }
 
 void GCNHazardRecognizer::Reset() {
@@ -1074,10 +1078,33 @@
   return true;
 }
 
-bool GCNHazardRecognizer::fixLdsBranchVmemWARHazard(MachineInstr *MI) {
+static bool shouldRunLdsBranchVmemWARHazardFixup(const MachineFunction &MF,
+                                                 const GCNSubtarget &ST) {
   if (!ST.hasLdsBranchVmemWARHazard())
     return false;
 
+  // Check if the necessary condition for the hazard is met: both LDS and VMEM
+  // instructions need to appear in the same function.
+  bool hasLDS = false;
+  bool hasVMEM = false;
+  for (auto &MBB : MF) {
+    for (auto &MI : MBB) {
+      hasLDS |= SIInstrInfo::isDS(MI);
+      hasVMEM |=
+          SIInstrInfo::isVMEM(MI) || SIInstrInfo::isSegmentSpecificFLAT(MI);
+      if (hasLDS && hasVMEM)
+        return true;
+    }
+  }
+  return false;
+}
+
+bool GCNHazardRecognizer::fixLdsBranchVmemWARHazard(MachineInstr *MI) {
+  if (!RunLdsBranchVmemWARHazardFixup)
+    return false;
+
+  assert(ST.hasLdsBranchVmemWARHazard());
+
   auto IsHazardInst = [](const MachineInstr &MI) {
     if (SIInstrInfo::isDS(MI))
       return 1;


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D104219.351832.patch
Type: text/x-patch
Size: 2455 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20210614/98f2de55/attachment.bin>