[llvm] [AMDGPU] Less-constrained scheduling for ASYNCMARK (PR #197638)

via llvm-commits llvm-commits at lists.llvm.org
Thu May 14 01:43:11 PDT 2026


https://github.com/lijinpei-amd created https://github.com/llvm/llvm-project/pull/197638

ASYNCMARK previously acted as a global memory barrier in the scheduler because of its hasSideEffects bit, pinning all surrounding memory access (read/write, sync/async, vmem/lds, etc) in place.

This commit improves scheduling freedom for ASYNCMARK by

1. Stop reporting ASYNCMARK as a global memory object. WAIT_ASYNCMARK is still a global memory object so ordinary loads/stores remain anchored across waits.

2. Instead, express the strictly necessary ordering with an artificial-edge DAG mutation:

   2.1. Relative order of ASYNCMARKs and WAIT_ASYNCMARKs is kept. 2.2. Each async load/store gets an edge to the next ASYNCMARK and an edge from the previous ASYNCMARK.

   ASYNCMARKs are otherwise exempt from dependencies with non-async load/store. WAIT_ASYNCMARK anchoring is left to the natural memory dep that WAIT_ASYNCMARK provides as a global memory object.

The mutation is registered in every pre-RA scheduler stage and in the post-RA scheduler, and handles BUNDLEs created by SIPostRABundler / SIInsertHardClauses.

The new mutation is a must-have for correctness: once ASYNCMARK is no longer a global memory object, the scheduler is free to reorder it across async loads/stores and WAIT_ASYNCMARK in ways that break program semantics. Later scheduler stages must remember to register this mutation.

>From 874352efa3ef87ea329fc9f66fa2dbcda46612da Mon Sep 17 00:00:00 2001
From: Li Jinpei <jinpli at amd.com>
Date: Wed, 13 May 2026 14:30:15 +0800
Subject: [PATCH] [AMDGPU] Less-constrained scheduling for ASYNCMARK

ASYNCMARK previously acted as a global memory barrier in the scheduler
because of its hasSideEffects bit, pinning all surrounding memory
access (read/write, sync/async, vmem/lds, etc) in place.

This commit improves scheduling freedom for ASYNCMARK by

1. Stop reporting ASYNCMARK as a global memory object. WAIT_ASYNCMARK
   is still a global memory object so ordinary loads/stores remain
   anchored across waits.

2. Instead, express the strictly necessary ordering with an
   artificial-edge DAG mutation:

   2.1. Relative order of ASYNCMARKs and WAIT_ASYNCMARKs is kept.
   2.2. Each async load/store gets an edge to the next ASYNCMARK and an
        edge from the previous ASYNCMARK.

   ASYNCMARKs are otherwise exempt from dependencies with non-async
   load/store. WAIT_ASYNCMARK anchoring is left to the natural memory
   dep that WAIT_ASYNCMARK provides as a global memory object.

The mutation is registered in every pre-RA scheduler stage and in the
post-RA scheduler, and handles BUNDLEs created by SIPostRABundler /
SIInsertHardClauses.

The new mutation is a must-have for correctness: once ASYNCMARK is no
longer a global memory object, the scheduler is free to reorder it
across async loads/stores and WAIT_ASYNCMARK in ways that break
program semantics. Later scheduler stages must remember to register
this mutation.
---
 .../AMDGPU/AMDGPUAsyncMarkScheduling.cpp      | 156 ++++++++++++++++
 .../Target/AMDGPU/AMDGPUAsyncMarkScheduling.h |  27 +++
 .../lib/Target/AMDGPU/AMDGPUTargetMachine.cpp |   9 +
 llvm/lib/Target/AMDGPU/CMakeLists.txt         |   1 +
 .../Target/AMDGPU/GCNIterativeScheduler.cpp   |   2 +
 llvm/lib/Target/AMDGPU/GCNSchedStrategy.cpp   |   4 +
 llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp   |   8 +-
 llvm/lib/Target/AMDGPU/SIInstrInfo.cpp        |  17 ++
 llvm/lib/Target/AMDGPU/SIInstrInfo.h          |   5 +
 .../CodeGen/AMDGPU/asyncmark-gfx12plus.ll     |  25 ++-
 .../test/CodeGen/AMDGPU/asyncmark-pregfx12.ll |  73 ++++----
 .../CodeGen/AMDGPU/asyncmark-sched-postra.mir | 169 ++++++++++++++++++
 .../CodeGen/AMDGPU/asyncmark-sched-prera.mir  | 109 +++++++++++
 13 files changed, 549 insertions(+), 56 deletions(-)
 create mode 100644 llvm/lib/Target/AMDGPU/AMDGPUAsyncMarkScheduling.cpp
 create mode 100644 llvm/lib/Target/AMDGPU/AMDGPUAsyncMarkScheduling.h
 create mode 100644 llvm/test/CodeGen/AMDGPU/asyncmark-sched-postra.mir
 create mode 100644 llvm/test/CodeGen/AMDGPU/asyncmark-sched-prera.mir

diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAsyncMarkScheduling.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAsyncMarkScheduling.cpp
new file mode 100644
index 0000000000000..e8e2bba761bac
--- /dev/null
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAsyncMarkScheduling.cpp
@@ -0,0 +1,156 @@
+//===--- AMDGPUAsyncMarkScheduling.cpp - AMDGPU Async Mark Scheduling -----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+/// \file
+/// Adds the minimal artificial scheduling DAG edges that preserve the
+/// semantics of ASYNCMARK / WAIT_ASYNCMARK without modeling ASYNCMARK as a
+/// global memory barrier.
+///
+/// ASYNCMARK is a meta marker placed in the stream of asynchronous
+/// requests; WAIT_ASYNCMARK(N) waits for the Nth previous mark to retire.
+/// Treating ASYNCMARK as a global memory object would over-constrain the
+/// scheduler, because everything around it would be pinned. Instead, this
+/// mutation expresses the strictly necessary ordering as artificial edges:
+///
+///   1. Relative order of ASYNCMARKs and WAIT_ASYNCMARKs is kept.
+///
+///   2. Each async load/store gets an edge to the next ASYNCMARK and an
+///      edge from the previous ASYNCMARK.
+///
+/// This scheme exempts ASYNCMARKs from dependencies with non-async
+/// load/store.
+///
+/// WAIT_ASYNCMARK is intentionally left as a global-memory object so that
+/// ordinary loads/stores remain anchored across waits.
+///
+/// At post-RA scheduling the SU may represent a BUNDLE rather than a single
+/// instruction (created by SIPostRABundler / SIInsertHardClauses). When that
+/// happens the relevant async loads/stores and ASYNCMARKs are bundle members
+/// rather than top-level SUnits, so the helpers below classify the SU based
+/// on what is *contained* in it.
+//
+//===----------------------------------------------------------------------===//
+
+#include "AMDGPUAsyncMarkScheduling.h"
+#include "MCTargetDesc/AMDGPUMCTargetDesc.h"
+#include "SIInstrInfo.h"
+#include "llvm/ADT/BitmaskEnum.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/CodeGen/MachineInstrBundle.h"
+#include "llvm/CodeGen/ScheduleDAG.h"
+#include "llvm/CodeGen/ScheduleDAGInstrs.h"
+
+using namespace llvm;
+
+namespace {
+
+LLVM_ENABLE_BITMASK_ENUMS_IN_NAMESPACE();
+
+// Classification of an SU's relevance to async marking. A bundle SU may
+// hold any combination of async loads/stores, ASYNCMARKs, and
+// WAIT_ASYNCMARKs, so this is a bitmask rather than a single tag.
+enum class SUKind : unsigned {
+  None = 0,
+  // SU contains an async mem-op with no preceding ASYNCMARK in the same SU.
+  PreMarkAsyncMemOp = 1u << 0,
+  // SU contains an async mem-op with no following ASYNCMARK in the same SU.
+  PostMarkAsyncMemOp = 1u << 1,
+  AsyncMark = 1u << 2,
+  WaitAsyncMark = 1u << 3,
+  LLVM_MARK_AS_BITMASK_ENUM(/*LargestValue=*/WaitAsyncMark)
+};
+
+inline SUKind classify(const MachineInstr &MI, const SIInstrInfo &TII) {
+  unsigned Opc = MI.getOpcode();
+  if (Opc == AMDGPU::ASYNCMARK)
+    return SUKind::AsyncMark;
+  if (Opc == AMDGPU::WAIT_ASYNCMARK)
+    return SUKind::WaitAsyncMark;
+  if (TII.isAsyncLDSDMA(MI))
+    return SUKind::PreMarkAsyncMemOp | SUKind::PostMarkAsyncMemOp;
+  return SUKind::None;
+}
+
+inline SUKind classifySU(const SUnit &SU, const SIInstrInfo &TII) {
+  const MachineInstr *MI = SU.getInstr();
+  if (!MI)
+    return SUKind::None;
+
+  if (!MI->isBundle())
+    return classify(*MI, TII);
+
+  SUKind Bits = SUKind::None;
+  for (auto It = std::next(MI->getIterator()),
+            End = getBundleEnd(MI->getIterator());
+       It != End; ++It) {
+    SUKind New = classify(*It, TII);
+    // Mem-ops following an in-bundle ASYNCMARK are not "pre-mark".
+    if (any(Bits & SUKind::AsyncMark))
+      Bits |= New & ~SUKind::PreMarkAsyncMemOp;
+    else
+      Bits |= New;
+    // Once we see an ASYNCMARK in the bundle, any prior async mem-ops are
+    // tagged by it, so they no longer count as "post-mark".
+    if (any(New & SUKind::AsyncMark))
+      Bits &= ~SUKind::PostMarkAsyncMemOp;
+  }
+  return Bits;
+}
+
+class AsyncMarkSched : public ScheduleDAGMutation {
+public:
+  void apply(ScheduleDAGInstrs *DAG) override;
+};
+
+void AsyncMarkSched::apply(ScheduleDAGInstrs *DAG) {
+  const SIInstrInfo &TII = *static_cast<const SIInstrInfo *>(DAG->TII);
+
+  SmallVector<SUnit *, 8> PendingAsync;
+  SUnit *LastMark = nullptr;
+  SUnit *LastWait = nullptr;
+
+  for (SUnit &SU : DAG->SUnits) {
+    SUKind Kind = classifySU(SU, TII);
+    if (!any(Kind))
+      continue;
+
+    auto AddPred = [&](SUnit *Pred) {
+      DAG->addEdge(&SU, SDep(Pred, SDep::Artificial));
+    };
+
+    bool IsMark = any(Kind & SUKind::AsyncMark);
+    bool IsWait = any(Kind & SUKind::WaitAsyncMark);
+    bool IsPre = any(Kind & SUKind::PreMarkAsyncMemOp);
+    bool IsPost = any(Kind & SUKind::PostMarkAsyncMemOp);
+
+    if (LastMark && (IsMark || IsWait || IsPre))
+      AddPred(LastMark);
+
+    if (IsMark) {
+      for (SUnit *Pred : PendingAsync)
+        AddPred(Pred);
+      if (LastWait)
+        AddPred(LastWait);
+      PendingAsync.clear();
+      LastMark = &SU;
+    }
+
+    if (IsPost)
+      PendingAsync.push_back(&SU);
+
+    if (IsWait)
+      LastWait = &SU;
+  }
+}
+
+} // end anonymous namespace
+
+std::unique_ptr<ScheduleDAGMutation>
+llvm::createAMDGPUAsyncMarkSchedDAGMutation() {
+  return std::make_unique<AsyncMarkSched>();
+}
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAsyncMarkScheduling.h b/llvm/lib/Target/AMDGPU/AMDGPUAsyncMarkScheduling.h
new file mode 100644
index 0000000000000..62aef11c93cc4
--- /dev/null
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAsyncMarkScheduling.h
@@ -0,0 +1,27 @@
+//===--- AMDGPUAsyncMarkScheduling.h ---------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+/// \file Adds the minimal scheduling-DAG dependencies required to keep
+/// AMDGPU::ASYNCMARK / AMDGPU::WAIT_ASYNCMARK semantics intact while
+/// allowing the scheduler to freely reorder unrelated work around them.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIB_TARGET_AMDGPU_AMDGPUASYNCMARKSCHEDULING_H
+#define LLVM_LIB_TARGET_AMDGPU_AMDGPUASYNCMARKSCHEDULING_H
+
+#include "llvm/CodeGen/ScheduleDAGMutation.h"
+#include <memory>
+
+namespace llvm {
+
+std::unique_ptr<ScheduleDAGMutation> createAMDGPUAsyncMarkSchedDAGMutation();
+
+} // end namespace llvm
+
+#endif // LLVM_LIB_TARGET_AMDGPU_AMDGPUASYNCMARKSCHEDULING_H
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
index 65317016c6390..9381f6c6c7bdf 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
@@ -17,6 +17,7 @@
 #include "AMDGPUTargetMachine.h"
 #include "AMDGPU.h"
 #include "AMDGPUAliasAnalysis.h"
+#include "AMDGPUAsyncMarkScheduling.h"
 #include "AMDGPUBarrierLatency.h"
 #include "AMDGPUCoExecSchedStrategy.h"
 #include "AMDGPUCtorDtorLowering.h"
@@ -759,6 +760,7 @@ createGCNMaxOccupancyMachineScheduler(MachineSchedContext *C) {
   DAG->addMutation(createAMDGPUExportClusteringDAGMutation());
   DAG->addMutation(createAMDGPUBarrierLatencyDAGMutation(C->MF));
   DAG->addMutation(createAMDGPUHazardLatencyDAGMutation(C->MF));
+  DAG->addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
   return DAG;
 }
 
@@ -767,6 +769,7 @@ createGCNMaxILPMachineScheduler(MachineSchedContext *C) {
   ScheduleDAGMILive *DAG =
       new GCNScheduleDAGMILive(C, std::make_unique<GCNMaxILPSchedStrategy>(C));
   DAG->addMutation(createIGroupLPDAGMutation(AMDGPU::SchedulingPhase::Initial));
+  DAG->addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
   return DAG;
 }
 
@@ -781,6 +784,7 @@ createGCNMaxMemoryClauseMachineScheduler(MachineSchedContext *C) {
   DAG->addMutation(createAMDGPUExportClusteringDAGMutation());
   DAG->addMutation(createAMDGPUBarrierLatencyDAGMutation(C->MF));
   DAG->addMutation(createAMDGPUHazardLatencyDAGMutation(C->MF));
+  DAG->addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
   return DAG;
 }
 
@@ -793,6 +797,7 @@ createIterativeGCNMaxOccupancyMachineScheduler(MachineSchedContext *C) {
   if (ST.shouldClusterStores())
     DAG->addMutation(createStoreClusterDAGMutation(DAG->TII, DAG->TRI));
   DAG->addMutation(createIGroupLPDAGMutation(AMDGPU::SchedulingPhase::Initial));
+  DAG->addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
   return DAG;
 }
 
@@ -800,6 +805,7 @@ static ScheduleDAGInstrs *createMinRegScheduler(MachineSchedContext *C) {
   auto *DAG = new GCNIterativeScheduler(
       C, GCNIterativeScheduler::SCHEDULE_MINREGFORCED);
   DAG->addMutation(createIGroupLPDAGMutation(AMDGPU::SchedulingPhase::Initial));
+  DAG->addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
   return DAG;
 }
 
@@ -812,6 +818,7 @@ createIterativeILPMachineScheduler(MachineSchedContext *C) {
     DAG->addMutation(createStoreClusterDAGMutation(DAG->TII, DAG->TRI));
   DAG->addMutation(createAMDGPUMacroFusionDAGMutation());
   DAG->addMutation(createIGroupLPDAGMutation(AMDGPU::SchedulingPhase::Initial));
+  DAG->addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
   return DAG;
 }
 
@@ -910,6 +917,7 @@ AMDGPUTargetMachine::createMachineScheduler(MachineSchedContext *C) const {
   DAG->addMutation(createLoadClusterDAGMutation(DAG->TII, DAG->TRI));
   if (ST.shouldClusterStores())
     DAG->addMutation(createStoreClusterDAGMutation(DAG->TII, DAG->TRI));
+  DAG->addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
   return DAG;
 }
 
@@ -1332,6 +1340,7 @@ GCNTargetMachine::createPostMachineScheduler(MachineSchedContext *C) const {
   DAG->addMutation(createAMDGPUExportClusteringDAGMutation());
   DAG->addMutation(createAMDGPUBarrierLatencyDAGMutation(C->MF));
   DAG->addMutation(createAMDGPUHazardLatencyDAGMutation(C->MF));
+  DAG->addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
   return DAG;
 }
 //===----------------------------------------------------------------------===//
diff --git a/llvm/lib/Target/AMDGPU/CMakeLists.txt b/llvm/lib/Target/AMDGPU/CMakeLists.txt
index ae8f1c0fad5ba..45548cc22916b 100644
--- a/llvm/lib/Target/AMDGPU/CMakeLists.txt
+++ b/llvm/lib/Target/AMDGPU/CMakeLists.txt
@@ -47,6 +47,7 @@ add_llvm_target(AMDGPUCodeGen
   AMDGPUArgumentUsageInfo.cpp
   AMDGPUAsanInstrumentation.cpp
   AMDGPUAsmPrinter.cpp
+  AMDGPUAsyncMarkScheduling.cpp
   AMDGPUAtomicOptimizer.cpp
   AMDGPUAttributor.cpp
   AMDGPUBarrierLatency.cpp
diff --git a/llvm/lib/Target/AMDGPU/GCNIterativeScheduler.cpp b/llvm/lib/Target/AMDGPU/GCNIterativeScheduler.cpp
index dff153cebdd4c..03996b227a4a1 100644
--- a/llvm/lib/Target/AMDGPU/GCNIterativeScheduler.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNIterativeScheduler.cpp
@@ -12,6 +12,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "GCNIterativeScheduler.h"
+#include "AMDGPUAsyncMarkScheduling.h"
 #include "AMDGPUIGroupLP.h"
 #include "GCNSchedStrategy.h"
 #include "SIMachineFunctionInfo.h"
@@ -136,6 +137,7 @@ void GCNIterativeScheduler::swapIGLPMutations(const Region &R, bool IsReentry) {
                                 : AMDGPU::SchedulingPhase::Initial;
 
     addMutation(createIGroupLPDAGMutation(SchedPhase));
+    addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
   }
 }
 
diff --git a/llvm/lib/Target/AMDGPU/GCNSchedStrategy.cpp b/llvm/lib/Target/AMDGPU/GCNSchedStrategy.cpp
index 8a6c1d5e21e85..3da95a127874b 100644
--- a/llvm/lib/Target/AMDGPU/GCNSchedStrategy.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNSchedStrategy.cpp
@@ -24,6 +24,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "GCNSchedStrategy.h"
+#include "AMDGPUAsyncMarkScheduling.h"
 #include "AMDGPUIGroupLP.h"
 #include "GCNHazardRecognizer.h"
 #include "GCNRegPressure.h"
@@ -1387,6 +1388,7 @@ bool UnclusteredHighRPStage::initGCNSchedStage() {
   SavedMutations.swap(DAG.Mutations);
   DAG.addMutation(
       createIGroupLPDAGMutation(AMDGPU::SchedulingPhase::PreRAReentry));
+  DAG.addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
 
   InitialOccupancy = DAG.MinOccupancy;
   // Aggressively try to reduce register pressure in the unclustered high RP
@@ -1784,6 +1786,7 @@ bool GCNSchedStage::initGCNRegion() {
     DAG.addMutation(createIGroupLPDAGMutation(
         IsInitialStage ? AMDGPU::SchedulingPhase::Initial
                        : AMDGPU::SchedulingPhase::PreRAReentry));
+    DAG.addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
   }
 
   return true;
@@ -3092,6 +3095,7 @@ void GCNPostScheduleDAGMILive::schedule() {
     SavedMutations.clear();
     SavedMutations.swap(Mutations);
     addMutation(createIGroupLPDAGMutation(AMDGPU::SchedulingPhase::PostRA));
+    addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
   }
 
   ScheduleDAGMI::schedule();
diff --git a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
index e3d7b7a5a519e..8154330fee5bf 100644
--- a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
+++ b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
@@ -656,13 +656,7 @@ class SIInsertWaitcnts {
   getExpertSchedulingEventType(const MachineInstr &Inst) const;
 
   bool isAsync(const MachineInstr &MI) const {
-    if (!SIInstrInfo::isLDSDMA(MI))
-      return false;
-    if (SIInstrInfo::usesASYNC_CNT(MI))
-      return true;
-    const MachineOperand *Async =
-        TII.getNamedOperand(MI, AMDGPU::OpName::IsAsync);
-    return Async && (Async->getImm());
+    return TII.isAsyncLDSDMA(MI);
   }
 
   bool isNonAsyncLdsDmaWrite(const MachineInstr &MI) const {
diff --git a/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp b/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
index 64c070d933ff0..ba87e641cc60c 100644
--- a/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
@@ -11443,7 +11443,24 @@ void SIInstrInfo::enforceOperandRCAlignment(MachineInstr &MI,
   MI.addOperand(MachineOperand::CreateReg(NewVR, false, true));
 }
 
+bool SIInstrInfo::isAsyncLDSDMA(const MachineInstr &MI) const {
+  if (!isLDSDMA(MI))
+    return false;
+  if (usesASYNC_CNT(MI))
+    return true;
+  const MachineOperand *Async = getNamedOperand(MI, AMDGPU::OpName::IsAsync);
+  return Async && Async->getImm();
+}
+
 bool SIInstrInfo::isGlobalMemoryObject(const MachineInstr *MI) const {
+  // ASYNCMARK is a meta marker placed in the async-request stream. It must
+  // retain hasSideEffects = 1 so it is not DCE'd, but it is not a memory
+  // operation and should not act as a barrier in the scheduler. The required
+  // ordering against async loads/stores and WAIT_ASYNCMARK is supplied
+  // explicitly by the AsyncMark scheduling DAG mutation.
+  if (MI->getOpcode() == AMDGPU::ASYNCMARK)
+    return false;
+
   if (isIGLP(*MI))
     return false;
 
diff --git a/llvm/lib/Target/AMDGPU/SIInstrInfo.h b/llvm/lib/Target/AMDGPU/SIInstrInfo.h
index 7c081fbba8a33..0622aa4f1e986 100644
--- a/llvm/lib/Target/AMDGPU/SIInstrInfo.h
+++ b/llvm/lib/Target/AMDGPU/SIInstrInfo.h
@@ -638,6 +638,11 @@ class SIInstrInfo final : public AMDGPUGenInstrInfo {
            (get(Opcode).TSFlags & SIInstrFlags::TENSOR_CNT);
   }
 
+  /// True if MI is an LDS-DMA write that targets the async vmem counter,
+  /// either via the ASYNC_CNT TSFlag or the legacy IsAsync operand on
+  /// MUBUF/FLAT load-to-LDS instructions.
+  bool isAsyncLDSDMA(const MachineInstr &MI) const;
+
   static bool isGWS(const MachineInstr &MI) {
     return MI.getDesc().TSFlags & SIInstrFlags::GWS;
   }
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-gfx12plus.ll b/llvm/test/CodeGen/AMDGPU/asyncmark-gfx12plus.ll
index 4e5ad79b72468..1e70fbd9447c4 100644
--- a/llvm/test/CodeGen/AMDGPU/asyncmark-gfx12plus.ll
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-gfx12plus.ll
@@ -22,8 +22,8 @@ define void @interleaved_with_wave_barrier(ptr addrspace(1) %foo, ptr addrspace(
 ; SDAG-NEXT:    v_add_nc_u64_e32 v[4:5], 0x58, v[8:9]
 ; SDAG-NEXT:    v_add_nc_u32_e32 v3, 0x58, v2
 ; SDAG-NEXT:    ; wave barrier
-; SDAG-NEXT:    ; asyncmark
 ; SDAG-NEXT:    global_load_b32 v0, v[0:1], off offset:8
+; SDAG-NEXT:    ; asyncmark
 ; SDAG-NEXT:    ; wave barrier
 ; SDAG-NEXT:    global_load_async_to_lds_b32 v3, v[4:5], off offset:4 th:TH_LOAD_LU nv
 ; SDAG-NEXT:    ; wave barrier
@@ -63,8 +63,8 @@ define void @interleaved_with_wave_barrier(ptr addrspace(1) %foo, ptr addrspace(
 ; GISEL-NEXT:    v_add_co_ci_u32_e64 v7, null, 0, v9, vcc_lo
 ; GISEL-NEXT:    v_add_nc_u32_e32 v3, 0x58, v2
 ; GISEL-NEXT:    ; wave barrier
-; GISEL-NEXT:    ; asyncmark
 ; GISEL-NEXT:    global_load_b32 v0, v[0:1], off offset:8
+; GISEL-NEXT:    ; asyncmark
 ; GISEL-NEXT:    ; wave barrier
 ; GISEL-NEXT:    global_load_async_to_lds_b32 v3, v[6:7], off offset:4 th:TH_LOAD_LU nv
 ; GISEL-NEXT:    ; wave barrier
@@ -310,25 +310,24 @@ define amdgpu_kernel void @test_pipelined_loop_with_global(ptr addrspace(1) %foo
 ; SDAG-NEXT:    s_clause 0x1
 ; SDAG-NEXT:    s_load_b96 s[8:10], s[4:5], 0x24 nv
 ; SDAG-NEXT:    s_load_b128 s[0:3], s[4:5], 0x34 nv
-; SDAG-NEXT:    v_mov_b32_e32 v0, 0
 ; SDAG-NEXT:    s_wait_kmcnt 0x0
-; SDAG-NEXT:    s_load_b32 s6, s[8:9], 0x0
-; SDAG-NEXT:    s_load_b32 s7, s[0:1], 0x0
-; SDAG-NEXT:    v_mov_b32_e32 v1, s10
-; SDAG-NEXT:    s_add_co_i32 s11, s10, 4
+; SDAG-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s10
+; SDAG-NEXT:    s_add_co_i32 s6, s10, 4
 ; SDAG-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
-; SDAG-NEXT:    v_dual_mov_b32 v3, 4 :: v_dual_mov_b32 v4, s11
+; SDAG-NEXT:    v_dual_mov_b32 v3, 4 :: v_dual_mov_b32 v4, s6
+; SDAG-NEXT:    s_load_b32 s6, s[0:1], 0x0
+; SDAG-NEXT:    s_load_b32 s7, s[8:9], 0x0
 ; SDAG-NEXT:    s_load_b32 s11, s[4:5], 0x44 nv
 ; SDAG-NEXT:    s_clause 0x2
 ; SDAG-NEXT:    global_load_async_to_lds_b32 v1, v0, s[8:9] offset:4 nv
-; SDAG-NEXT:    ; asyncmark
 ; SDAG-NEXT:    global_load_b32 v1, v0, s[8:9] offset:4
 ; SDAG-NEXT:    global_load_b32 v2, v0, s[0:1] offset:4
+; SDAG-NEXT:    ; asyncmark
 ; SDAG-NEXT:    s_wait_xcnt 0x0
 ; SDAG-NEXT:    s_add_nc_u64 s[0:1], s[0:1], 8
 ; SDAG-NEXT:    s_add_nc_u64 s[4:5], s[8:9], 8
 ; SDAG-NEXT:    s_wait_kmcnt 0x0
-; SDAG-NEXT:    v_dual_mov_b32 v5, s6 :: v_dual_mov_b32 v6, s7
+; SDAG-NEXT:    v_dual_mov_b32 v6, s6 :: v_dual_mov_b32 v5, s7
 ; SDAG-NEXT:    s_mov_b64 s[6:7], s[2:3]
 ; SDAG-NEXT:    global_load_async_to_lds_b32 v4, v3, s[8:9] offset:4 nv
 ; SDAG-NEXT:    s_wait_loadcnt 0x0
@@ -392,13 +391,13 @@ define amdgpu_kernel void @test_pipelined_loop_with_global(ptr addrspace(1) %foo
 ; GISEL-LABEL: test_pipelined_loop_with_global:
 ; GISEL:       ; %bb.0: ; %prolog
 ; GISEL-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GISEL-NEXT:    s_clause 0x2
+; GISEL-NEXT:    s_clause 0x1
 ; GISEL-NEXT:    s_load_b96 s[8:10], s[4:5], 0x24 nv
 ; GISEL-NEXT:    s_load_b128 s[0:3], s[4:5], 0x34 nv
-; GISEL-NEXT:    s_load_b32 s11, s[4:5], 0x44 nv
 ; GISEL-NEXT:    v_mov_b32_e32 v0, 0
 ; GISEL-NEXT:    s_wait_kmcnt 0x0
 ; GISEL-NEXT:    s_load_b32 s14, s[8:9], 0x0
+; GISEL-NEXT:    s_load_b32 s11, s[4:5], 0x44 nv
 ; GISEL-NEXT:    s_load_b32 s15, s[0:1], 0x0
 ; GISEL-NEXT:    v_mov_b32_e32 v1, s10
 ; GISEL-NEXT:    s_add_co_u32 s6, s10, 4
@@ -407,9 +406,9 @@ define amdgpu_kernel void @test_pipelined_loop_with_global(ptr addrspace(1) %foo
 ; GISEL-NEXT:    s_mov_b64 s[6:7], s[2:3]
 ; GISEL-NEXT:    s_clause 0x2
 ; GISEL-NEXT:    global_load_async_to_lds_b32 v1, v0, s[8:9] offset:4 nv
-; GISEL-NEXT:    ; asyncmark
 ; GISEL-NEXT:    global_load_b32 v1, v0, s[8:9] offset:4
 ; GISEL-NEXT:    global_load_b32 v2, v0, s[0:1] offset:4
+; GISEL-NEXT:    ; asyncmark
 ; GISEL-NEXT:    s_wait_xcnt 0x0
 ; GISEL-NEXT:    s_add_co_u32 s0, s0, 8
 ; GISEL-NEXT:    s_add_co_ci_u32 s1, s1, 0
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-pregfx12.ll b/llvm/test/CodeGen/AMDGPU/asyncmark-pregfx12.ll
index 46af269d0a0a6..2fc3253d064c2 100644
--- a/llvm/test/CodeGen/AMDGPU/asyncmark-pregfx12.ll
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-pregfx12.ll
@@ -51,23 +51,22 @@ define void @interleaved_global_and_dma(ptr addrspace(1) %foo, ptr addrspace(3)
 ; SDAG-LABEL: interleaved_global_and_dma:
 ; SDAG:       ; %bb.0: ; %entry
 ; SDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; SDAG-NEXT:    v_readfirstlane_b32 s4, v2
 ; SDAG-NEXT:    global_load_dword v7, v[3:4], off
 ; SDAG-NEXT:    global_load_dword v8, v[0:1], off
-; SDAG-NEXT:    s_mov_b32 m0, s4
+; SDAG-NEXT:    v_readfirstlane_b32 s4, v2
 ; SDAG-NEXT:    ; wave barrier
+; SDAG-NEXT:    s_mov_b32 m0, s4
+; SDAG-NEXT:    global_load_dword v0, v[0:1], off
 ; SDAG-NEXT:    s_nop 0
 ; SDAG-NEXT:    global_load_dword v[3:4], off lds
 ; SDAG-NEXT:    ; asyncmark
-; SDAG-NEXT:    global_load_dword v0, v[0:1], off
 ; SDAG-NEXT:    ; wave barrier
-; SDAG-NEXT:    s_nop 0
 ; SDAG-NEXT:    global_load_dword v[3:4], off lds
 ; SDAG-NEXT:    ; wave barrier
 ; SDAG-NEXT:    global_load_dword v1, v[3:4], off
 ; SDAG-NEXT:    ; asyncmark
 ; SDAG-NEXT:    ; wait_asyncmark(1)
-; SDAG-NEXT:    s_waitcnt vmcnt(3)
+; SDAG-NEXT:    s_waitcnt vmcnt(2)
 ; SDAG-NEXT:    ds_read_b32 v3, v2
 ; SDAG-NEXT:    ; wait_asyncmark(0)
 ; SDAG-NEXT:    s_waitcnt vmcnt(1)
@@ -85,14 +84,14 @@ define void @interleaved_global_and_dma(ptr addrspace(1) %foo, ptr addrspace(3)
 ; GISEL:       ; %bb.0: ; %entry
 ; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GISEL-NEXT:    v_readfirstlane_b32 s4, v2
+; GISEL-NEXT:    s_mov_b32 m0, s4
 ; GISEL-NEXT:    global_load_dword v7, v[3:4], off
 ; GISEL-NEXT:    global_load_dword v8, v[0:1], off
-; GISEL-NEXT:    s_mov_b32 m0, s4
 ; GISEL-NEXT:    ; wave barrier
 ; GISEL-NEXT:    s_nop 0
 ; GISEL-NEXT:    global_load_dword v[3:4], off lds
-; GISEL-NEXT:    ; asyncmark
 ; GISEL-NEXT:    global_load_dword v0, v[0:1], off
+; GISEL-NEXT:    ; asyncmark
 ; GISEL-NEXT:    ; wave barrier
 ; GISEL-NEXT:    s_nop 0
 ; GISEL-NEXT:    global_load_dword v[3:4], off lds
@@ -154,15 +153,15 @@ define void @interleaved_buffer_and_dma(ptr addrspace(8) inreg %buf, ptr addrspa
 ; SDAG-LABEL: interleaved_buffer_and_dma:
 ; SDAG:       ; %bb.0: ; %entry
 ; SDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; SDAG-NEXT:    s_mov_b32 m0, s20
 ; SDAG-NEXT:    global_load_dword v6, v[2:3], off
 ; SDAG-NEXT:    global_load_dword v7, v[0:1], off
-; SDAG-NEXT:    v_mov_b32_e32 v8, 0x54
+; SDAG-NEXT:    s_mov_b32 m0, s20
 ; SDAG-NEXT:    ; wave barrier
-; SDAG-NEXT:    buffer_load_dword v8, s[16:19], 0 offen lds
-; SDAG-NEXT:    ; asyncmark
+; SDAG-NEXT:    v_mov_b32_e32 v8, 0x54
 ; SDAG-NEXT:    global_load_dword v0, v[0:1], off
 ; SDAG-NEXT:    v_mov_b32_e32 v1, 0x58
+; SDAG-NEXT:    buffer_load_dword v8, s[16:19], 0 offen lds
+; SDAG-NEXT:    ; asyncmark
 ; SDAG-NEXT:    ; wave barrier
 ; SDAG-NEXT:    buffer_load_dword v1, s[16:19], 0 offen lds
 ; SDAG-NEXT:    ; wave barrier
@@ -170,7 +169,7 @@ define void @interleaved_buffer_and_dma(ptr addrspace(8) inreg %buf, ptr addrspa
 ; SDAG-NEXT:    v_mov_b32_e32 v2, s20
 ; SDAG-NEXT:    ; asyncmark
 ; SDAG-NEXT:    ; wait_asyncmark(1)
-; SDAG-NEXT:    s_waitcnt vmcnt(3)
+; SDAG-NEXT:    s_waitcnt vmcnt(2)
 ; SDAG-NEXT:    ds_read_b32 v3, v2
 ; SDAG-NEXT:    ; wait_asyncmark(0)
 ; SDAG-NEXT:    s_waitcnt vmcnt(1)
@@ -188,14 +187,16 @@ define void @interleaved_buffer_and_dma(ptr addrspace(8) inreg %buf, ptr addrspa
 ; GISEL:       ; %bb.0: ; %entry
 ; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GISEL-NEXT:    s_mov_b32 m0, s20
+; GISEL-NEXT:    v_mov_b32_e32 v8, 0x54
 ; GISEL-NEXT:    global_load_dword v6, v[2:3], off
 ; GISEL-NEXT:    global_load_dword v7, v[0:1], off
-; GISEL-NEXT:    v_mov_b32_e32 v8, 0x54
 ; GISEL-NEXT:    ; wave barrier
+; GISEL-NEXT:    s_waitcnt vmcnt(0)
+; GISEL-NEXT:    v_add_u32_e32 v6, v7, v6
 ; GISEL-NEXT:    buffer_load_dword v8, s[16:19], 0 offen lds
-; GISEL-NEXT:    ; asyncmark
 ; GISEL-NEXT:    global_load_dword v0, v[0:1], off
 ; GISEL-NEXT:    v_mov_b32_e32 v1, 0x58
+; GISEL-NEXT:    ; asyncmark
 ; GISEL-NEXT:    ; wave barrier
 ; GISEL-NEXT:    buffer_load_dword v1, s[16:19], 0 offen lds
 ; GISEL-NEXT:    ; wave barrier
@@ -208,7 +209,6 @@ define void @interleaved_buffer_and_dma(ptr addrspace(8) inreg %buf, ptr addrspa
 ; GISEL-NEXT:    ; wait_asyncmark(0)
 ; GISEL-NEXT:    s_waitcnt vmcnt(1)
 ; GISEL-NEXT:    ds_read_b32 v2, v2
-; GISEL-NEXT:    v_add_u32_e32 v6, v7, v6
 ; GISEL-NEXT:    s_waitcnt lgkmcnt(1)
 ; GISEL-NEXT:    v_add3_u32 v0, v6, v3, v0
 ; GISEL-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
@@ -264,9 +264,9 @@ define void @fence_with_asyncmark(ptr addrspace(8) inreg %buf, ptr addrspace(1)
 ; SDAG-NEXT:    ; wave barrier
 ; SDAG-NEXT:    buffer_load_dword v8, s[16:19], 0 offen lds
 ; SDAG-NEXT:    s_waitcnt vmcnt(0)
-; SDAG-NEXT:    ; asyncmark
 ; SDAG-NEXT:    global_load_dword v0, v[0:1], off
 ; SDAG-NEXT:    v_mov_b32_e32 v1, 0x58
+; SDAG-NEXT:    ; asyncmark
 ; SDAG-NEXT:    ; wave barrier
 ; SDAG-NEXT:    buffer_load_dword v1, s[16:19], 0 offen lds
 ; SDAG-NEXT:    ; wave barrier
@@ -297,9 +297,9 @@ define void @fence_with_asyncmark(ptr addrspace(8) inreg %buf, ptr addrspace(1)
 ; GISEL-NEXT:    ; wave barrier
 ; GISEL-NEXT:    buffer_load_dword v8, s[16:19], 0 offen lds
 ; GISEL-NEXT:    s_waitcnt vmcnt(0)
-; GISEL-NEXT:    ; asyncmark
 ; GISEL-NEXT:    global_load_dword v0, v[0:1], off
 ; GISEL-NEXT:    v_mov_b32_e32 v1, 0x58
+; GISEL-NEXT:    ; asyncmark
 ; GISEL-NEXT:    ; wave barrier
 ; GISEL-NEXT:    buffer_load_dword v1, s[16:19], 0 offen lds
 ; GISEL-NEXT:    ; wave barrier
@@ -496,40 +496,41 @@ define void @test_pipelined_loop_with_global(ptr addrspace(1) %foo, ptr addrspac
 ; SDAG:       ; %bb.0: ; %prolog
 ; SDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; SDAG-NEXT:    v_readfirstlane_b32 s4, v2
+; SDAG-NEXT:    global_load_dword v14, v[0:1], off
+; SDAG-NEXT:    global_load_dword v8, v[0:1], off
+; SDAG-NEXT:    global_load_dword v15, v[3:4], off
+; SDAG-NEXT:    global_load_dword v9, v[3:4], off
 ; SDAG-NEXT:    s_mov_b32 m0, s4
-; SDAG-NEXT:    global_load_dword v10, v[0:1], off
-; SDAG-NEXT:    global_load_dword v14, v[3:4], off
 ; SDAG-NEXT:    s_mov_b32 s6, 2
 ; SDAG-NEXT:    global_load_dword v[0:1], off lds
 ; SDAG-NEXT:    ; asyncmark
-; SDAG-NEXT:    global_load_dword v8, v[0:1], off
-; SDAG-NEXT:    global_load_dword v9, v[3:4], off
-; SDAG-NEXT:    s_mov_b64 s[4:5], 0
 ; SDAG-NEXT:    global_load_dword v[0:1], off lds
+; SDAG-NEXT:    s_mov_b64 s[4:5], 0
 ; SDAG-NEXT:    ; asyncmark
+; SDAG-NEXT:    s_waitcnt vmcnt(4)
+; SDAG-NEXT:    v_mov_b32_e32 v12, v8
 ; SDAG-NEXT:    s_waitcnt vmcnt(2)
-; SDAG-NEXT:    v_mov_b32_e32 v13, v8
-; SDAG-NEXT:    s_waitcnt vmcnt(1)
-; SDAG-NEXT:    v_mov_b32_e32 v15, v9
+; SDAG-NEXT:    v_mov_b32_e32 v13, v9
 ; SDAG-NEXT:  .LBB5_1: ; %loop_body
 ; SDAG-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; SDAG-NEXT:    v_readfirstlane_b32 s7, v2
 ; SDAG-NEXT:    s_waitcnt vmcnt(1)
-; SDAG-NEXT:    v_mov_b32_e32 v12, v15
 ; SDAG-NEXT:    v_mov_b32_e32 v11, v13
-; SDAG-NEXT:    global_load_dword v13, v[0:1], off
-; SDAG-NEXT:    global_load_dword v15, v[3:4], off
+; SDAG-NEXT:    v_mov_b32_e32 v10, v12
+; SDAG-NEXT:    global_load_dword v12, v[0:1], off
+; SDAG-NEXT:    global_load_dword v13, v[3:4], off
 ; SDAG-NEXT:    s_mov_b32 m0, s7
 ; SDAG-NEXT:    s_add_i32 s6, s6, 1
 ; SDAG-NEXT:    global_load_dword v[0:1], off lds
 ; SDAG-NEXT:    v_cmp_ge_i32_e32 vcc, s6, v7
-; SDAG-NEXT:    v_mov_b32_e32 v16, v14
-; SDAG-NEXT:    v_mov_b32_e32 v17, v10
-; SDAG-NEXT:    v_mov_b32_e32 v10, v8
-; SDAG-NEXT:    v_mov_b32_e32 v14, v9
+; SDAG-NEXT:    v_mov_b32_e32 v16, v15
+; SDAG-NEXT:    v_mov_b32_e32 v17, v14
+; SDAG-NEXT:    v_mov_b32_e32 v14, v8
+; SDAG-NEXT:    v_mov_b32_e32 v15, v9
 ; SDAG-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
 ; SDAG-NEXT:    ; asyncmark
 ; SDAG-NEXT:    ; wait_asyncmark(2)
+; SDAG-NEXT:    s_waitcnt vmcnt(4)
 ; SDAG-NEXT:    s_andn2_b64 exec, exec, s[4:5]
 ; SDAG-NEXT:    s_cbranch_execnz .LBB5_1
 ; SDAG-NEXT:  ; %bb.2: ; %epilog
@@ -543,10 +544,10 @@ define void @test_pipelined_loop_with_global(ptr addrspace(1) %foo, ptr addrspac
 ; SDAG-NEXT:    ds_read_b32 v2, v2
 ; SDAG-NEXT:    v_add_u32_e32 v3, v17, v16
 ; SDAG-NEXT:    s_waitcnt lgkmcnt(2)
-; SDAG-NEXT:    v_add3_u32 v0, v3, v0, v12
+; SDAG-NEXT:    v_add3_u32 v0, v3, v0, v11
 ; SDAG-NEXT:    s_waitcnt lgkmcnt(1)
-; SDAG-NEXT:    v_add3_u32 v0, v11, v0, v1
-; SDAG-NEXT:    v_add_u32_e32 v1, v13, v15
+; SDAG-NEXT:    v_add3_u32 v0, v10, v0, v1
+; SDAG-NEXT:    v_add_u32_e32 v1, v12, v13
 ; SDAG-NEXT:    s_waitcnt lgkmcnt(0)
 ; SDAG-NEXT:    v_add3_u32 v0, v1, v2, v0
 ; SDAG-NEXT:    global_store_dword v[5:6], v0, off
@@ -562,9 +563,9 @@ define void @test_pipelined_loop_with_global(ptr addrspace(1) %foo, ptr addrspac
 ; GISEL-NEXT:    global_load_dword v14, v[3:4], off
 ; GISEL-NEXT:    s_mov_b32 s6, 2
 ; GISEL-NEXT:    global_load_dword v[0:1], off lds
-; GISEL-NEXT:    ; asyncmark
 ; GISEL-NEXT:    global_load_dword v8, v[0:1], off
 ; GISEL-NEXT:    global_load_dword v9, v[3:4], off
+; GISEL-NEXT:    ; asyncmark
 ; GISEL-NEXT:    s_mov_b64 s[4:5], 0
 ; GISEL-NEXT:    global_load_dword v[0:1], off lds
 ; GISEL-NEXT:    v_mov_b32_e32 v16, s6
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-sched-postra.mir b/llvm/test/CodeGen/AMDGPU/asyncmark-sched-postra.mir
new file mode 100644
index 0000000000000..c54523766e064
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-sched-postra.mir
@@ -0,0 +1,169 @@
+# RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1250 -passes=postmisched \
+# RUN:     -debug-only=machine-scheduler -o /dev/null %s 2>&1 \
+# RUN:     | FileCheck %s
+
+# REQUIRES: asserts
+
+# Pins the artificial edges added by the AMDGPU AsyncMark scheduling DAG
+# mutation across bundled and non-bundled shapes. Each test verifies the
+# `Predecessors:` block of the next ASYNCMARK / WAIT_ASYNCMARK in the
+# scheduler's SU dump.
+
+--- |
+  define void @bundle_mem_then_mark() { unreachable }
+  define void @bundle_mark_then_mem() { unreachable }
+  define void @nonbundle_two_batches() { unreachable }
+  define void @mixed_around_mark() { unreachable }
+...
+
+# A BUNDLE { async load, ASYNCMARK } is self-tagged: the in-bundle MARK
+# clears PostMarkAsyncMemOp, so the bundle is not pushed to the pending-
+# async list. The next ASYNCMARK gets exactly two artificial predecessors
+# (rule 1 from the bundle, rule 2 from the standalone load).
+#
+# CHECK-LABEL: bundle_mem_then_mark:%bb.0
+# CHECK-LABEL: SU(0):{{.*}}BUNDLE
+# CHECK-LABEL: SU(1):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(2):{{.*}}ASYNCMARK
+# CHECK:        Predecessors:
+# CHECK-NEXT:     SU(0):{{.*}}Artificial
+# CHECK-NEXT:     SU(1):{{.*}}Artificial
+# CHECK-NEXT:   Successors:
+# CHECK-LABEL: SU(3):{{.*}}WAIT_ASYNCMARK
+# CHECK:        Predecessors:
+# CHECK:          SU(2):{{.*}}Artificial
+# CHECK-NEXT: ExitSU:
+---
+name:            bundle_mem_then_mark
+tracksRegLiveness: true
+machineFunctionInfo:
+  occupancy:       8
+body:             |
+  bb.0:
+    liveins: $vgpr0_vgpr1, $vgpr2, $vgpr4_vgpr5, $vgpr3
+
+    BUNDLE implicit-def $asynccnt, implicit $vgpr0_vgpr1, implicit $vgpr2, implicit $exec, implicit $asynccnt {
+      GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+      ASYNCMARK
+    }
+    GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr3, $vgpr4_vgpr5, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+    ASYNCMARK
+    WAIT_ASYNCMARK 0
+    S_ENDPGM 0
+...
+
+# A BUNDLE { ASYNCMARK, async load }: the in-bundle MARK does not tag the
+# trailing in-bundle mem, so the bundle is classified Mark|Post and is
+# pushed to the pending-async list. The next external ASYNCMARK chains to
+# the bundle (the LastMark + drain edges fold into a single artificial
+# predecessor in the dump).
+#
+# CHECK-LABEL: bundle_mark_then_mem:%bb.0
+# CHECK-LABEL: SU(0):{{.*}}BUNDLE
+# CHECK-LABEL: SU(1):{{.*}}ASYNCMARK
+# CHECK:        Predecessors:
+# CHECK-NEXT:     SU(0):{{.*}}Artificial
+# CHECK-NEXT:   Successors:
+# CHECK-LABEL: SU(2):{{.*}}WAIT_ASYNCMARK
+# CHECK:        Predecessors:
+# CHECK:          SU(1):{{.*}}Artificial
+# CHECK-NEXT: ExitSU:
+---
+name:            bundle_mark_then_mem
+tracksRegLiveness: true
+machineFunctionInfo:
+  occupancy:       8
+body:             |
+  bb.0:
+    liveins: $vgpr0_vgpr1, $vgpr2
+
+    BUNDLE implicit-def $asynccnt, implicit $vgpr0_vgpr1, implicit $vgpr2, implicit $exec, implicit $asynccnt {
+      ASYNCMARK
+      GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+    }
+    ASYNCMARK
+    WAIT_ASYNCMARK 0
+    S_ENDPGM 0
+...
+
+# Two-batch non-bundled case: load, load, MARK1, load, MARK2, WAIT.
+# MARK1 drains both pending loads (SU(0), SU(1)). MARK2 chains to MARK1
+# (rule 1) and drains the third load (rule 2). WAIT chains to MARK2.
+#
+# CHECK-LABEL: nonbundle_two_batches:%bb.0
+# CHECK-LABEL: SU(0):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(1):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(2):{{.*}}ASYNCMARK
+# CHECK:        Predecessors:
+# CHECK-NEXT:     SU(0):{{.*}}Artificial
+# CHECK-NEXT:     SU(1):{{.*}}Artificial
+# CHECK-NEXT:   Successors:
+# CHECK-LABEL: SU(3):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(4):{{.*}}ASYNCMARK
+# CHECK:        Predecessors:
+# CHECK-NEXT:     SU(2):{{.*}}Artificial
+# CHECK-NEXT:     SU(3):{{.*}}Artificial
+# CHECK-NEXT:   Successors:
+# CHECK-LABEL: SU(5):{{.*}}WAIT_ASYNCMARK
+# CHECK:        Predecessors:
+# CHECK:          SU(4):{{.*}}Artificial
+# CHECK-NEXT: ExitSU:
+---
+name:            nonbundle_two_batches
+tracksRegLiveness: true
+machineFunctionInfo:
+  occupancy:       8
+body:             |
+  bb.0:
+    liveins: $vgpr0_vgpr1, $vgpr2, $vgpr4_vgpr5, $vgpr3
+
+    GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+    GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr3, $vgpr4_vgpr5, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+    ASYNCMARK
+    GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+    ASYNCMARK
+    WAIT_ASYNCMARK 0
+    S_ENDPGM 0
+...
+
+# ASYNCMARK has no dependencies with non-async memory ops. With async
+# loads and a mix of non-async reads/writes on both sides, MARK's full
+# pred/succ list contains only the async loads and the WAIT.
+#
+# CHECK-LABEL: mixed_around_mark:%bb.0
+# CHECK-LABEL: SU(0):{{.*}}GLOBAL_LOAD_DWORD
+# CHECK-LABEL: SU(1):{{.*}}DS_READ_B32
+# CHECK-LABEL: SU(2):{{.*}}GLOBAL_STORE_DWORD
+# CHECK-LABEL: SU(3):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(4):{{.*}}ASYNCMARK
+# CHECK:        Predecessors:
+# CHECK-NEXT:     SU(3):{{.*}}Artificial
+# CHECK-NEXT:   Successors:
+# CHECK-NEXT:     SU(8):{{.*}}Artificial
+# CHECK-NEXT:     SU(9):{{.*}}Artificial
+# CHECK-NEXT: SU(5):{{.*}}BUFFER_LOAD_DWORD_OFFSET
+# CHECK-LABEL: SU(6):{{.*}}DS_WRITE_B32
+# CHECK-LABEL: SU(7):{{.*}}BUFFER_STORE_DWORD_OFFSET
+# CHECK-LABEL: SU(8):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(9):{{.*}}WAIT_ASYNCMARK
+---
+name:            mixed_around_mark
+tracksRegLiveness: true
+machineFunctionInfo:
+  occupancy:       8
+body:             |
+  bb.0:
+    liveins: $vgpr0_vgpr1, $vgpr4_vgpr5, $vgpr2, $vgpr3, $sgpr0_sgpr1_sgpr2_sgpr3
+
+    $vgpr6 = GLOBAL_LOAD_DWORD $vgpr0_vgpr1, 0, 0, implicit $exec :: (load (s32), addrspace 1)
+    $vgpr7 = DS_READ_B32 $vgpr2, 0, 0, implicit $m0, implicit $exec :: (load (s32), addrspace 3)
+    GLOBAL_STORE_DWORD $vgpr0_vgpr1, $vgpr3, 0, 0, implicit $exec :: (store (s32), addrspace 1)
+    GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr3, $vgpr4_vgpr5, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+    ASYNCMARK
+    $vgpr8 = BUFFER_LOAD_DWORD_OFFSET $sgpr0_sgpr1_sgpr2_sgpr3, 0, 0, 0, 0, implicit $exec :: (load (s32))
+    DS_WRITE_B32 $vgpr2, $vgpr3, 0, 0, implicit $m0, implicit $exec :: (store (s32), addrspace 3)
+    BUFFER_STORE_DWORD_OFFSET $vgpr3, $sgpr0_sgpr1_sgpr2_sgpr3, 0, 0, 0, 0, implicit $exec :: (store (s32))
+    GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr3, $vgpr4_vgpr5, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+    WAIT_ASYNCMARK 0
+    S_ENDPGM 0
+...
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-sched-prera.mir b/llvm/test/CodeGen/AMDGPU/asyncmark-sched-prera.mir
new file mode 100644
index 0000000000000..654cd26412187
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-sched-prera.mir
@@ -0,0 +1,109 @@
+# RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1250 -run-pass=machine-scheduler \
+# RUN:     -debug-only=machine-scheduler -o /dev/null %s 2>&1 \
+# RUN:     | FileCheck %s
+
+# REQUIRES: asserts
+
+# Two-batch case: load, load, MARK, load, MARK, WAIT.
+# After 4 IMPLICIT_DEF set-up SUs (SU(0)..SU(3)), the async ops and
+# marks become SU(4)..SU(9):
+#   SU(4), SU(5), SU(7) = async loads
+#   SU(6), SU(8)        = ASYNCMARKs
+#   SU(9)               = WAIT_ASYNCMARK
+#
+# CHECK-LABEL: two_batches:%bb.0
+# CHECK-LABEL: SU(4):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(5):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(6):{{.*}}ASYNCMARK
+# CHECK:        Predecessors:
+# CHECK-NEXT:     SU(4):{{.*}}Artificial
+# CHECK-NEXT:     SU(5):{{.*}}Artificial
+# CHECK-NEXT:   Successors:
+# CHECK-LABEL: SU(7):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(8):{{.*}}ASYNCMARK
+# CHECK:        Predecessors:
+# CHECK-NEXT:     SU(6):{{.*}}Artificial
+# CHECK-NEXT:     SU(7):{{.*}}Artificial
+# CHECK-NEXT:   Successors:
+# CHECK-LABEL: SU(9):{{.*}}WAIT_ASYNCMARK
+# CHECK:        Predecessors:
+# CHECK:          SU(8):{{.*}}Artificial
+# CHECK-NEXT:   Pressure Diff
+---
+name:            two_batches
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    %0:vreg_64_align2 = IMPLICIT_DEF
+    %1:vgpr_32 = IMPLICIT_DEF
+    %2:vreg_64_align2 = IMPLICIT_DEF
+    %3:vgpr_32 = IMPLICIT_DEF
+    GLOBAL_LOAD_ASYNC_TO_LDS_B32 %1, %0, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+    GLOBAL_LOAD_ASYNC_TO_LDS_B32 %3, %2, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+    ASYNCMARK
+    GLOBAL_LOAD_ASYNC_TO_LDS_B32 %1, %0, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+    ASYNCMARK
+    WAIT_ASYNCMARK 0
+    S_ENDPGM 0
+...
+
+# ASYNCMARK is exempt from dependencies with non-async memory ops, but
+# still picks up rule-2 edges from the surrounding async LDS DMAs and a
+# rule-1 edge to WAIT_ASYNCMARK. Mix a non-async read, non-async write,
+# and LDS read/write on each side, plus an async load on each side.
+#
+# After 5 IMPLICIT_DEFs (SU(0)..SU(4)):
+#   SU(5)  = GLOBAL_LOAD_DWORD            (before MARK, non-async read)
+#   SU(6)  = DS_READ_B32                  (before MARK, lds read)
+#   SU(7)  = GLOBAL_STORE_DWORD           (before MARK, non-async write)
+#   SU(8)  = GLOBAL_LOAD_ASYNC_TO_LDS_B32 (before MARK, async)
+#   SU(9)  = ASYNCMARK
+#   SU(10) = BUFFER_LOAD_DWORD_OFFSET     (after MARK, non-async read)
+#   SU(11) = DS_WRITE_B32                 (after MARK, lds write)
+#   SU(12) = BUFFER_STORE_DWORD_OFFSET    (after MARK, non-async write)
+#   SU(13) = GLOBAL_LOAD_ASYNC_TO_LDS_B32 (after MARK, async)
+#   SU(14) = WAIT_ASYNCMARK
+#
+# Asserting MARK's full Predecessors/Successors with CHECK-NEXT
+# guarantees that none of the six non-async memory ops are connected to
+# the MARK - only the two async loads and the WAIT are.
+#
+# CHECK-LABEL: mixed_around_mark:%bb.0
+# CHECK-LABEL: SU(5):{{.*}}GLOBAL_LOAD_DWORD
+# CHECK-LABEL: SU(6):{{.*}}DS_READ_B32
+# CHECK-LABEL: SU(7):{{.*}}GLOBAL_STORE_DWORD
+# CHECK-LABEL: SU(8):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(9):{{.*}}ASYNCMARK
+# CHECK:        Predecessors:
+# CHECK-NEXT:     SU(8):{{.*}}Artificial
+# CHECK-NEXT:   Successors:
+# CHECK-NEXT:     SU(13):{{.*}}Artificial
+# CHECK-NEXT:     SU(14):{{.*}}Artificial
+# CHECK-NEXT:   Pressure Diff
+# CHECK-LABEL: SU(10):{{.*}}BUFFER_LOAD_DWORD_OFFSET
+# CHECK-LABEL: SU(11):{{.*}}DS_WRITE_B32
+# CHECK-LABEL: SU(12):{{.*}}BUFFER_STORE_DWORD_OFFSET
+# CHECK-LABEL: SU(13):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(14):{{.*}}WAIT_ASYNCMARK
+---
+name:            mixed_around_mark
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    %0:vreg_64_align2 = IMPLICIT_DEF
+    %1:vreg_64_align2 = IMPLICIT_DEF
+    %2:vgpr_32 = IMPLICIT_DEF
+    %3:vgpr_32 = IMPLICIT_DEF
+    %4:sgpr_128 = IMPLICIT_DEF
+    %5:vgpr_32 = GLOBAL_LOAD_DWORD %0, 0, 0, implicit $exec :: (load (s32), addrspace 1)
+    %6:vgpr_32 = DS_READ_B32 %2, 0, 0, implicit $m0, implicit $exec :: (load (s32), addrspace 3)
+    GLOBAL_STORE_DWORD %0, %3, 0, 0, implicit $exec :: (store (s32), addrspace 1)
+    GLOBAL_LOAD_ASYNC_TO_LDS_B32 %3, %1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+    ASYNCMARK
+    %7:vgpr_32 = BUFFER_LOAD_DWORD_OFFSET %4, 0, 0, 0, 0, implicit $exec :: (load (s32))
+    DS_WRITE_B32 %2, %3, 0, 0, implicit $m0, implicit $exec :: (store (s32), addrspace 3)
+    BUFFER_STORE_DWORD_OFFSET %3, %4, 0, 0, 0, 0, implicit $exec :: (store (s32))
+    GLOBAL_LOAD_ASYNC_TO_LDS_B32 %3, %1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+    WAIT_ASYNCMARK 0
+    S_ENDPGM 0
+...



More information about the llvm-commits mailing list