[llvm] [AMDGPU] Examine instructions in pending queues during scheduling (PR #147653)

[Jeffrey Byrnes via llvm-commits]

================
@@ -450,35 +534,46 @@ SUnit *GCNSchedStrategy::pickNode(bool &IsTopNode) {
            Bot.Available.empty() && Bot.Pending.empty() && "ReadyQ garbage");
     return nullptr;
   }
+  bool PickedPending;
   SUnit *SU;
   do {
+    PickedPending = false;
     if (RegionPolicy.OnlyTopDown) {
-      SU = Top.pickOnlyChoice();
+      SU = pickOnlyChoice(Top, SchedModel);
       if (!SU) {
         CandPolicy NoPolicy;
         TopCand.reset(NoPolicy);
         pickNodeFromQueue(Top, NoPolicy, DAG->getTopRPTracker(), TopCand,
+                          PickedPending,
                           /*IsBottomUp=*/false);
         assert(TopCand.Reason != NoCand && "failed to find a candidate");
         SU = TopCand.SU;
       }
       IsTopNode = true;
     } else if (RegionPolicy.OnlyBottomUp) {
-      SU = Bot.pickOnlyChoice();
+      SU = pickOnlyChoice(Bot, SchedModel);
       if (!SU) {
         CandPolicy NoPolicy;
         BotCand.reset(NoPolicy);
         pickNodeFromQueue(Bot, NoPolicy, DAG->getBotRPTracker(), BotCand,
+                          PickedPending,
                           /*IsBottomUp=*/true);
         assert(BotCand.Reason != NoCand && "failed to find a candidate");
         SU = BotCand.SU;
       }
       IsTopNode = false;
     } else {
-      SU = pickNodeBidirectional(IsTopNode);
+      SU = pickNodeBidirectional(IsTopNode, PickedPending);
     }
   } while (SU->isScheduled);
 
+  if (PickedPending) {
+    unsigned ReadyCycle = IsTopNode ? SU->TopReadyCycle : SU->BotReadyCycle;
+    SchedBoundary &Zone = IsTopNode ? Top : Bot;
+    Zone.bumpCycle(ReadyCycle);
----------------
jrbyrnes wrote:

We're making a mess of this test

llc -mtriple=amdgcn -mcpu=gfx908 --run-pass=machine-scheduler --misched-prera-direction=topdown <filename.mir>

```
--- |
  define amdgpu_kernel void @pending_queue_ready_cycle_bug(ptr addrspace(1) %arg) {
  bb:
    unreachable
  }

...
---
name:            pending_queue_ready_cycle_bug
tracksRegLiveness: true
body:             |
  bb.0.bb:
    liveins: $sgpr4_sgpr5

    %2:sgpr_128 = IMPLICIT_DEF
    %14:vgpr_32 = IMPLICIT_DEF
    %15:vgpr_32 = IMPLICIT_DEF
    %18:areg_512 = IMPLICIT_DEF
    %18:areg_512 = V_MFMA_F32_16X16X1F32_mac_e64 %15, %14, %18, 0, 0, 0, implicit $mode, implicit $exec
    %5:vreg_128 = BUFFER_LOAD_DWORDX4_OFFSET %2, 0, 0, 0, 0, implicit $exec
    %18:areg_512 = V_MFMA_F32_16X16X1F32_mac_e64 %15, %14, %18, 0, 0, 0, implicit $mode, implicit $exec
    undef %84.sub0:vreg_128_align2 = V_ADD_U32_e32 %5.sub0, %14, implicit $exec
    %7:vreg_512 = COPY %18
    SCHED_BARRIER 0
    S_NOP 0, implicit %18, implicit %7, %84
    S_ENDPGM 0
...
```


We schedule the first MFMA, then the LOAD, then the ADD, then the second MFMA. After scheduling the ADD, the Top zone has current cycle of 83 due to the long latency of the load. I'm not sure why, but I am seeing that the second MFMA is still in the pending queue at this point. Then, we call bumpCycle with cycle 9. This issue is probably more due to not releasing the MFMA from pending than it is due to improper cycle management.

https://github.com/llvm/llvm-project/pull/147653