[llvm] [AMDGPU] Less-constrained scheduling for ASYNCMARK (PR #197638)
via llvm-commits
llvm-commits at lists.llvm.org
Thu May 14 01:43:11 PDT 2026
https://github.com/lijinpei-amd created https://github.com/llvm/llvm-project/pull/197638
ASYNCMARK previously acted as a global memory barrier in the scheduler because of its hasSideEffects bit, pinning all surrounding memory access (read/write, sync/async, vmem/lds, etc) in place.
This commit improves scheduling freedom for ASYNCMARK by
1. Stop reporting ASYNCMARK as a global memory object. WAIT_ASYNCMARK is still a global memory object so ordinary loads/stores remain anchored across waits.
2. Instead, express the strictly necessary ordering with an artificial-edge DAG mutation:
2.1. Relative order of ASYNCMARKs and WAIT_ASYNCMARKs is kept. 2.2. Each async load/store gets an edge to the next ASYNCMARK and an edge from the previous ASYNCMARK.
ASYNCMARKs are otherwise exempt from dependencies with non-async load/store. WAIT_ASYNCMARK anchoring is left to the natural memory dep that WAIT_ASYNCMARK provides as a global memory object.
The mutation is registered in every pre-RA scheduler stage and in the post-RA scheduler, and handles BUNDLEs created by SIPostRABundler / SIInsertHardClauses.
The new mutation is a must-have for correctness: once ASYNCMARK is no longer a global memory object, the scheduler is free to reorder it across async loads/stores and WAIT_ASYNCMARK in ways that break program semantics. Later scheduler stages must remember to register this mutation.
>From 874352efa3ef87ea329fc9f66fa2dbcda46612da Mon Sep 17 00:00:00 2001
From: Li Jinpei <jinpli at amd.com>
Date: Wed, 13 May 2026 14:30:15 +0800
Subject: [PATCH] [AMDGPU] Less-constrained scheduling for ASYNCMARK
ASYNCMARK previously acted as a global memory barrier in the scheduler
because of its hasSideEffects bit, pinning all surrounding memory
access (read/write, sync/async, vmem/lds, etc) in place.
This commit improves scheduling freedom for ASYNCMARK by
1. Stop reporting ASYNCMARK as a global memory object. WAIT_ASYNCMARK
is still a global memory object so ordinary loads/stores remain
anchored across waits.
2. Instead, express the strictly necessary ordering with an
artificial-edge DAG mutation:
2.1. Relative order of ASYNCMARKs and WAIT_ASYNCMARKs is kept.
2.2. Each async load/store gets an edge to the next ASYNCMARK and an
edge from the previous ASYNCMARK.
ASYNCMARKs are otherwise exempt from dependencies with non-async
load/store. WAIT_ASYNCMARK anchoring is left to the natural memory
dep that WAIT_ASYNCMARK provides as a global memory object.
The mutation is registered in every pre-RA scheduler stage and in the
post-RA scheduler, and handles BUNDLEs created by SIPostRABundler /
SIInsertHardClauses.
The new mutation is a must-have for correctness: once ASYNCMARK is no
longer a global memory object, the scheduler is free to reorder it
across async loads/stores and WAIT_ASYNCMARK in ways that break
program semantics. Later scheduler stages must remember to register
this mutation.
---
.../AMDGPU/AMDGPUAsyncMarkScheduling.cpp | 156 ++++++++++++++++
.../Target/AMDGPU/AMDGPUAsyncMarkScheduling.h | 27 +++
.../lib/Target/AMDGPU/AMDGPUTargetMachine.cpp | 9 +
llvm/lib/Target/AMDGPU/CMakeLists.txt | 1 +
.../Target/AMDGPU/GCNIterativeScheduler.cpp | 2 +
llvm/lib/Target/AMDGPU/GCNSchedStrategy.cpp | 4 +
llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp | 8 +-
llvm/lib/Target/AMDGPU/SIInstrInfo.cpp | 17 ++
llvm/lib/Target/AMDGPU/SIInstrInfo.h | 5 +
.../CodeGen/AMDGPU/asyncmark-gfx12plus.ll | 25 ++-
.../test/CodeGen/AMDGPU/asyncmark-pregfx12.ll | 73 ++++----
.../CodeGen/AMDGPU/asyncmark-sched-postra.mir | 169 ++++++++++++++++++
.../CodeGen/AMDGPU/asyncmark-sched-prera.mir | 109 +++++++++++
13 files changed, 549 insertions(+), 56 deletions(-)
create mode 100644 llvm/lib/Target/AMDGPU/AMDGPUAsyncMarkScheduling.cpp
create mode 100644 llvm/lib/Target/AMDGPU/AMDGPUAsyncMarkScheduling.h
create mode 100644 llvm/test/CodeGen/AMDGPU/asyncmark-sched-postra.mir
create mode 100644 llvm/test/CodeGen/AMDGPU/asyncmark-sched-prera.mir
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAsyncMarkScheduling.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAsyncMarkScheduling.cpp
new file mode 100644
index 0000000000000..e8e2bba761bac
--- /dev/null
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAsyncMarkScheduling.cpp
@@ -0,0 +1,156 @@
+//===--- AMDGPUAsyncMarkScheduling.cpp - AMDGPU Async Mark Scheduling -----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+/// \file
+/// Adds the minimal artificial scheduling DAG edges that preserve the
+/// semantics of ASYNCMARK / WAIT_ASYNCMARK without modeling ASYNCMARK as a
+/// global memory barrier.
+///
+/// ASYNCMARK is a meta marker placed in the stream of asynchronous
+/// requests; WAIT_ASYNCMARK(N) waits for the Nth previous mark to retire.
+/// Treating ASYNCMARK as a global memory object would over-constrain the
+/// scheduler, because everything around it would be pinned. Instead, this
+/// mutation expresses the strictly necessary ordering as artificial edges:
+///
+/// 1. Relative order of ASYNCMARKs and WAIT_ASYNCMARKs is kept.
+///
+/// 2. Each async load/store gets an edge to the next ASYNCMARK and an
+/// edge from the previous ASYNCMARK.
+///
+/// This scheme exempts ASYNCMARKs from dependencies with non-async
+/// load/store.
+///
+/// WAIT_ASYNCMARK is intentionally left as a global-memory object so that
+/// ordinary loads/stores remain anchored across waits.
+///
+/// At post-RA scheduling the SU may represent a BUNDLE rather than a single
+/// instruction (created by SIPostRABundler / SIInsertHardClauses). When that
+/// happens the relevant async loads/stores and ASYNCMARKs are bundle members
+/// rather than top-level SUnits, so the helpers below classify the SU based
+/// on what is *contained* in it.
+//
+//===----------------------------------------------------------------------===//
+
+#include "AMDGPUAsyncMarkScheduling.h"
+#include "MCTargetDesc/AMDGPUMCTargetDesc.h"
+#include "SIInstrInfo.h"
+#include "llvm/ADT/BitmaskEnum.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/CodeGen/MachineInstrBundle.h"
+#include "llvm/CodeGen/ScheduleDAG.h"
+#include "llvm/CodeGen/ScheduleDAGInstrs.h"
+
+using namespace llvm;
+
+namespace {
+
+LLVM_ENABLE_BITMASK_ENUMS_IN_NAMESPACE();
+
+// Classification of an SU's relevance to async marking. A bundle SU may
+// hold any combination of async loads/stores, ASYNCMARKs, and
+// WAIT_ASYNCMARKs, so this is a bitmask rather than a single tag.
+enum class SUKind : unsigned {
+ None = 0,
+ // SU contains an async mem-op with no preceding ASYNCMARK in the same SU.
+ PreMarkAsyncMemOp = 1u << 0,
+ // SU contains an async mem-op with no following ASYNCMARK in the same SU.
+ PostMarkAsyncMemOp = 1u << 1,
+ AsyncMark = 1u << 2,
+ WaitAsyncMark = 1u << 3,
+ LLVM_MARK_AS_BITMASK_ENUM(/*LargestValue=*/WaitAsyncMark)
+};
+
+inline SUKind classify(const MachineInstr &MI, const SIInstrInfo &TII) {
+ unsigned Opc = MI.getOpcode();
+ if (Opc == AMDGPU::ASYNCMARK)
+ return SUKind::AsyncMark;
+ if (Opc == AMDGPU::WAIT_ASYNCMARK)
+ return SUKind::WaitAsyncMark;
+ if (TII.isAsyncLDSDMA(MI))
+ return SUKind::PreMarkAsyncMemOp | SUKind::PostMarkAsyncMemOp;
+ return SUKind::None;
+}
+
+inline SUKind classifySU(const SUnit &SU, const SIInstrInfo &TII) {
+ const MachineInstr *MI = SU.getInstr();
+ if (!MI)
+ return SUKind::None;
+
+ if (!MI->isBundle())
+ return classify(*MI, TII);
+
+ SUKind Bits = SUKind::None;
+ for (auto It = std::next(MI->getIterator()),
+ End = getBundleEnd(MI->getIterator());
+ It != End; ++It) {
+ SUKind New = classify(*It, TII);
+ // Mem-ops following an in-bundle ASYNCMARK are not "pre-mark".
+ if (any(Bits & SUKind::AsyncMark))
+ Bits |= New & ~SUKind::PreMarkAsyncMemOp;
+ else
+ Bits |= New;
+ // Once we see an ASYNCMARK in the bundle, any prior async mem-ops are
+ // tagged by it, so they no longer count as "post-mark".
+ if (any(New & SUKind::AsyncMark))
+ Bits &= ~SUKind::PostMarkAsyncMemOp;
+ }
+ return Bits;
+}
+
+class AsyncMarkSched : public ScheduleDAGMutation {
+public:
+ void apply(ScheduleDAGInstrs *DAG) override;
+};
+
+void AsyncMarkSched::apply(ScheduleDAGInstrs *DAG) {
+ const SIInstrInfo &TII = *static_cast<const SIInstrInfo *>(DAG->TII);
+
+ SmallVector<SUnit *, 8> PendingAsync;
+ SUnit *LastMark = nullptr;
+ SUnit *LastWait = nullptr;
+
+ for (SUnit &SU : DAG->SUnits) {
+ SUKind Kind = classifySU(SU, TII);
+ if (!any(Kind))
+ continue;
+
+ auto AddPred = [&](SUnit *Pred) {
+ DAG->addEdge(&SU, SDep(Pred, SDep::Artificial));
+ };
+
+ bool IsMark = any(Kind & SUKind::AsyncMark);
+ bool IsWait = any(Kind & SUKind::WaitAsyncMark);
+ bool IsPre = any(Kind & SUKind::PreMarkAsyncMemOp);
+ bool IsPost = any(Kind & SUKind::PostMarkAsyncMemOp);
+
+ if (LastMark && (IsMark || IsWait || IsPre))
+ AddPred(LastMark);
+
+ if (IsMark) {
+ for (SUnit *Pred : PendingAsync)
+ AddPred(Pred);
+ if (LastWait)
+ AddPred(LastWait);
+ PendingAsync.clear();
+ LastMark = &SU;
+ }
+
+ if (IsPost)
+ PendingAsync.push_back(&SU);
+
+ if (IsWait)
+ LastWait = &SU;
+ }
+}
+
+} // end anonymous namespace
+
+std::unique_ptr<ScheduleDAGMutation>
+llvm::createAMDGPUAsyncMarkSchedDAGMutation() {
+ return std::make_unique<AsyncMarkSched>();
+}
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAsyncMarkScheduling.h b/llvm/lib/Target/AMDGPU/AMDGPUAsyncMarkScheduling.h
new file mode 100644
index 0000000000000..62aef11c93cc4
--- /dev/null
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAsyncMarkScheduling.h
@@ -0,0 +1,27 @@
+//===--- AMDGPUAsyncMarkScheduling.h ---------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+/// \file Adds the minimal scheduling-DAG dependencies required to keep
+/// AMDGPU::ASYNCMARK / AMDGPU::WAIT_ASYNCMARK semantics intact while
+/// allowing the scheduler to freely reorder unrelated work around them.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIB_TARGET_AMDGPU_AMDGPUASYNCMARKSCHEDULING_H
+#define LLVM_LIB_TARGET_AMDGPU_AMDGPUASYNCMARKSCHEDULING_H
+
+#include "llvm/CodeGen/ScheduleDAGMutation.h"
+#include <memory>
+
+namespace llvm {
+
+std::unique_ptr<ScheduleDAGMutation> createAMDGPUAsyncMarkSchedDAGMutation();
+
+} // end namespace llvm
+
+#endif // LLVM_LIB_TARGET_AMDGPU_AMDGPUASYNCMARKSCHEDULING_H
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
index 65317016c6390..9381f6c6c7bdf 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
@@ -17,6 +17,7 @@
#include "AMDGPUTargetMachine.h"
#include "AMDGPU.h"
#include "AMDGPUAliasAnalysis.h"
+#include "AMDGPUAsyncMarkScheduling.h"
#include "AMDGPUBarrierLatency.h"
#include "AMDGPUCoExecSchedStrategy.h"
#include "AMDGPUCtorDtorLowering.h"
@@ -759,6 +760,7 @@ createGCNMaxOccupancyMachineScheduler(MachineSchedContext *C) {
DAG->addMutation(createAMDGPUExportClusteringDAGMutation());
DAG->addMutation(createAMDGPUBarrierLatencyDAGMutation(C->MF));
DAG->addMutation(createAMDGPUHazardLatencyDAGMutation(C->MF));
+ DAG->addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
return DAG;
}
@@ -767,6 +769,7 @@ createGCNMaxILPMachineScheduler(MachineSchedContext *C) {
ScheduleDAGMILive *DAG =
new GCNScheduleDAGMILive(C, std::make_unique<GCNMaxILPSchedStrategy>(C));
DAG->addMutation(createIGroupLPDAGMutation(AMDGPU::SchedulingPhase::Initial));
+ DAG->addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
return DAG;
}
@@ -781,6 +784,7 @@ createGCNMaxMemoryClauseMachineScheduler(MachineSchedContext *C) {
DAG->addMutation(createAMDGPUExportClusteringDAGMutation());
DAG->addMutation(createAMDGPUBarrierLatencyDAGMutation(C->MF));
DAG->addMutation(createAMDGPUHazardLatencyDAGMutation(C->MF));
+ DAG->addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
return DAG;
}
@@ -793,6 +797,7 @@ createIterativeGCNMaxOccupancyMachineScheduler(MachineSchedContext *C) {
if (ST.shouldClusterStores())
DAG->addMutation(createStoreClusterDAGMutation(DAG->TII, DAG->TRI));
DAG->addMutation(createIGroupLPDAGMutation(AMDGPU::SchedulingPhase::Initial));
+ DAG->addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
return DAG;
}
@@ -800,6 +805,7 @@ static ScheduleDAGInstrs *createMinRegScheduler(MachineSchedContext *C) {
auto *DAG = new GCNIterativeScheduler(
C, GCNIterativeScheduler::SCHEDULE_MINREGFORCED);
DAG->addMutation(createIGroupLPDAGMutation(AMDGPU::SchedulingPhase::Initial));
+ DAG->addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
return DAG;
}
@@ -812,6 +818,7 @@ createIterativeILPMachineScheduler(MachineSchedContext *C) {
DAG->addMutation(createStoreClusterDAGMutation(DAG->TII, DAG->TRI));
DAG->addMutation(createAMDGPUMacroFusionDAGMutation());
DAG->addMutation(createIGroupLPDAGMutation(AMDGPU::SchedulingPhase::Initial));
+ DAG->addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
return DAG;
}
@@ -910,6 +917,7 @@ AMDGPUTargetMachine::createMachineScheduler(MachineSchedContext *C) const {
DAG->addMutation(createLoadClusterDAGMutation(DAG->TII, DAG->TRI));
if (ST.shouldClusterStores())
DAG->addMutation(createStoreClusterDAGMutation(DAG->TII, DAG->TRI));
+ DAG->addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
return DAG;
}
@@ -1332,6 +1340,7 @@ GCNTargetMachine::createPostMachineScheduler(MachineSchedContext *C) const {
DAG->addMutation(createAMDGPUExportClusteringDAGMutation());
DAG->addMutation(createAMDGPUBarrierLatencyDAGMutation(C->MF));
DAG->addMutation(createAMDGPUHazardLatencyDAGMutation(C->MF));
+ DAG->addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
return DAG;
}
//===----------------------------------------------------------------------===//
diff --git a/llvm/lib/Target/AMDGPU/CMakeLists.txt b/llvm/lib/Target/AMDGPU/CMakeLists.txt
index ae8f1c0fad5ba..45548cc22916b 100644
--- a/llvm/lib/Target/AMDGPU/CMakeLists.txt
+++ b/llvm/lib/Target/AMDGPU/CMakeLists.txt
@@ -47,6 +47,7 @@ add_llvm_target(AMDGPUCodeGen
AMDGPUArgumentUsageInfo.cpp
AMDGPUAsanInstrumentation.cpp
AMDGPUAsmPrinter.cpp
+ AMDGPUAsyncMarkScheduling.cpp
AMDGPUAtomicOptimizer.cpp
AMDGPUAttributor.cpp
AMDGPUBarrierLatency.cpp
diff --git a/llvm/lib/Target/AMDGPU/GCNIterativeScheduler.cpp b/llvm/lib/Target/AMDGPU/GCNIterativeScheduler.cpp
index dff153cebdd4c..03996b227a4a1 100644
--- a/llvm/lib/Target/AMDGPU/GCNIterativeScheduler.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNIterativeScheduler.cpp
@@ -12,6 +12,7 @@
//===----------------------------------------------------------------------===//
#include "GCNIterativeScheduler.h"
+#include "AMDGPUAsyncMarkScheduling.h"
#include "AMDGPUIGroupLP.h"
#include "GCNSchedStrategy.h"
#include "SIMachineFunctionInfo.h"
@@ -136,6 +137,7 @@ void GCNIterativeScheduler::swapIGLPMutations(const Region &R, bool IsReentry) {
: AMDGPU::SchedulingPhase::Initial;
addMutation(createIGroupLPDAGMutation(SchedPhase));
+ addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
}
}
diff --git a/llvm/lib/Target/AMDGPU/GCNSchedStrategy.cpp b/llvm/lib/Target/AMDGPU/GCNSchedStrategy.cpp
index 8a6c1d5e21e85..3da95a127874b 100644
--- a/llvm/lib/Target/AMDGPU/GCNSchedStrategy.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNSchedStrategy.cpp
@@ -24,6 +24,7 @@
//===----------------------------------------------------------------------===//
#include "GCNSchedStrategy.h"
+#include "AMDGPUAsyncMarkScheduling.h"
#include "AMDGPUIGroupLP.h"
#include "GCNHazardRecognizer.h"
#include "GCNRegPressure.h"
@@ -1387,6 +1388,7 @@ bool UnclusteredHighRPStage::initGCNSchedStage() {
SavedMutations.swap(DAG.Mutations);
DAG.addMutation(
createIGroupLPDAGMutation(AMDGPU::SchedulingPhase::PreRAReentry));
+ DAG.addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
InitialOccupancy = DAG.MinOccupancy;
// Aggressively try to reduce register pressure in the unclustered high RP
@@ -1784,6 +1786,7 @@ bool GCNSchedStage::initGCNRegion() {
DAG.addMutation(createIGroupLPDAGMutation(
IsInitialStage ? AMDGPU::SchedulingPhase::Initial
: AMDGPU::SchedulingPhase::PreRAReentry));
+ DAG.addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
}
return true;
@@ -3092,6 +3095,7 @@ void GCNPostScheduleDAGMILive::schedule() {
SavedMutations.clear();
SavedMutations.swap(Mutations);
addMutation(createIGroupLPDAGMutation(AMDGPU::SchedulingPhase::PostRA));
+ addMutation(createAMDGPUAsyncMarkSchedDAGMutation());
}
ScheduleDAGMI::schedule();
diff --git a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
index e3d7b7a5a519e..8154330fee5bf 100644
--- a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
+++ b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
@@ -656,13 +656,7 @@ class SIInsertWaitcnts {
getExpertSchedulingEventType(const MachineInstr &Inst) const;
bool isAsync(const MachineInstr &MI) const {
- if (!SIInstrInfo::isLDSDMA(MI))
- return false;
- if (SIInstrInfo::usesASYNC_CNT(MI))
- return true;
- const MachineOperand *Async =
- TII.getNamedOperand(MI, AMDGPU::OpName::IsAsync);
- return Async && (Async->getImm());
+ return TII.isAsyncLDSDMA(MI);
}
bool isNonAsyncLdsDmaWrite(const MachineInstr &MI) const {
diff --git a/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp b/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
index 64c070d933ff0..ba87e641cc60c 100644
--- a/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
@@ -11443,7 +11443,24 @@ void SIInstrInfo::enforceOperandRCAlignment(MachineInstr &MI,
MI.addOperand(MachineOperand::CreateReg(NewVR, false, true));
}
+bool SIInstrInfo::isAsyncLDSDMA(const MachineInstr &MI) const {
+ if (!isLDSDMA(MI))
+ return false;
+ if (usesASYNC_CNT(MI))
+ return true;
+ const MachineOperand *Async = getNamedOperand(MI, AMDGPU::OpName::IsAsync);
+ return Async && Async->getImm();
+}
+
bool SIInstrInfo::isGlobalMemoryObject(const MachineInstr *MI) const {
+ // ASYNCMARK is a meta marker placed in the async-request stream. It must
+ // retain hasSideEffects = 1 so it is not DCE'd, but it is not a memory
+ // operation and should not act as a barrier in the scheduler. The required
+ // ordering against async loads/stores and WAIT_ASYNCMARK is supplied
+ // explicitly by the AsyncMark scheduling DAG mutation.
+ if (MI->getOpcode() == AMDGPU::ASYNCMARK)
+ return false;
+
if (isIGLP(*MI))
return false;
diff --git a/llvm/lib/Target/AMDGPU/SIInstrInfo.h b/llvm/lib/Target/AMDGPU/SIInstrInfo.h
index 7c081fbba8a33..0622aa4f1e986 100644
--- a/llvm/lib/Target/AMDGPU/SIInstrInfo.h
+++ b/llvm/lib/Target/AMDGPU/SIInstrInfo.h
@@ -638,6 +638,11 @@ class SIInstrInfo final : public AMDGPUGenInstrInfo {
(get(Opcode).TSFlags & SIInstrFlags::TENSOR_CNT);
}
+ /// True if MI is an LDS-DMA write that targets the async vmem counter,
+ /// either via the ASYNC_CNT TSFlag or the legacy IsAsync operand on
+ /// MUBUF/FLAT load-to-LDS instructions.
+ bool isAsyncLDSDMA(const MachineInstr &MI) const;
+
static bool isGWS(const MachineInstr &MI) {
return MI.getDesc().TSFlags & SIInstrFlags::GWS;
}
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-gfx12plus.ll b/llvm/test/CodeGen/AMDGPU/asyncmark-gfx12plus.ll
index 4e5ad79b72468..1e70fbd9447c4 100644
--- a/llvm/test/CodeGen/AMDGPU/asyncmark-gfx12plus.ll
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-gfx12plus.ll
@@ -22,8 +22,8 @@ define void @interleaved_with_wave_barrier(ptr addrspace(1) %foo, ptr addrspace(
; SDAG-NEXT: v_add_nc_u64_e32 v[4:5], 0x58, v[8:9]
; SDAG-NEXT: v_add_nc_u32_e32 v3, 0x58, v2
; SDAG-NEXT: ; wave barrier
-; SDAG-NEXT: ; asyncmark
; SDAG-NEXT: global_load_b32 v0, v[0:1], off offset:8
+; SDAG-NEXT: ; asyncmark
; SDAG-NEXT: ; wave barrier
; SDAG-NEXT: global_load_async_to_lds_b32 v3, v[4:5], off offset:4 th:TH_LOAD_LU nv
; SDAG-NEXT: ; wave barrier
@@ -63,8 +63,8 @@ define void @interleaved_with_wave_barrier(ptr addrspace(1) %foo, ptr addrspace(
; GISEL-NEXT: v_add_co_ci_u32_e64 v7, null, 0, v9, vcc_lo
; GISEL-NEXT: v_add_nc_u32_e32 v3, 0x58, v2
; GISEL-NEXT: ; wave barrier
-; GISEL-NEXT: ; asyncmark
; GISEL-NEXT: global_load_b32 v0, v[0:1], off offset:8
+; GISEL-NEXT: ; asyncmark
; GISEL-NEXT: ; wave barrier
; GISEL-NEXT: global_load_async_to_lds_b32 v3, v[6:7], off offset:4 th:TH_LOAD_LU nv
; GISEL-NEXT: ; wave barrier
@@ -310,25 +310,24 @@ define amdgpu_kernel void @test_pipelined_loop_with_global(ptr addrspace(1) %foo
; SDAG-NEXT: s_clause 0x1
; SDAG-NEXT: s_load_b96 s[8:10], s[4:5], 0x24 nv
; SDAG-NEXT: s_load_b128 s[0:3], s[4:5], 0x34 nv
-; SDAG-NEXT: v_mov_b32_e32 v0, 0
; SDAG-NEXT: s_wait_kmcnt 0x0
-; SDAG-NEXT: s_load_b32 s6, s[8:9], 0x0
-; SDAG-NEXT: s_load_b32 s7, s[0:1], 0x0
-; SDAG-NEXT: v_mov_b32_e32 v1, s10
-; SDAG-NEXT: s_add_co_i32 s11, s10, 4
+; SDAG-NEXT: v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s10
+; SDAG-NEXT: s_add_co_i32 s6, s10, 4
; SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
-; SDAG-NEXT: v_dual_mov_b32 v3, 4 :: v_dual_mov_b32 v4, s11
+; SDAG-NEXT: v_dual_mov_b32 v3, 4 :: v_dual_mov_b32 v4, s6
+; SDAG-NEXT: s_load_b32 s6, s[0:1], 0x0
+; SDAG-NEXT: s_load_b32 s7, s[8:9], 0x0
; SDAG-NEXT: s_load_b32 s11, s[4:5], 0x44 nv
; SDAG-NEXT: s_clause 0x2
; SDAG-NEXT: global_load_async_to_lds_b32 v1, v0, s[8:9] offset:4 nv
-; SDAG-NEXT: ; asyncmark
; SDAG-NEXT: global_load_b32 v1, v0, s[8:9] offset:4
; SDAG-NEXT: global_load_b32 v2, v0, s[0:1] offset:4
+; SDAG-NEXT: ; asyncmark
; SDAG-NEXT: s_wait_xcnt 0x0
; SDAG-NEXT: s_add_nc_u64 s[0:1], s[0:1], 8
; SDAG-NEXT: s_add_nc_u64 s[4:5], s[8:9], 8
; SDAG-NEXT: s_wait_kmcnt 0x0
-; SDAG-NEXT: v_dual_mov_b32 v5, s6 :: v_dual_mov_b32 v6, s7
+; SDAG-NEXT: v_dual_mov_b32 v6, s6 :: v_dual_mov_b32 v5, s7
; SDAG-NEXT: s_mov_b64 s[6:7], s[2:3]
; SDAG-NEXT: global_load_async_to_lds_b32 v4, v3, s[8:9] offset:4 nv
; SDAG-NEXT: s_wait_loadcnt 0x0
@@ -392,13 +391,13 @@ define amdgpu_kernel void @test_pipelined_loop_with_global(ptr addrspace(1) %foo
; GISEL-LABEL: test_pipelined_loop_with_global:
; GISEL: ; %bb.0: ; %prolog
; GISEL-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GISEL-NEXT: s_clause 0x2
+; GISEL-NEXT: s_clause 0x1
; GISEL-NEXT: s_load_b96 s[8:10], s[4:5], 0x24 nv
; GISEL-NEXT: s_load_b128 s[0:3], s[4:5], 0x34 nv
-; GISEL-NEXT: s_load_b32 s11, s[4:5], 0x44 nv
; GISEL-NEXT: v_mov_b32_e32 v0, 0
; GISEL-NEXT: s_wait_kmcnt 0x0
; GISEL-NEXT: s_load_b32 s14, s[8:9], 0x0
+; GISEL-NEXT: s_load_b32 s11, s[4:5], 0x44 nv
; GISEL-NEXT: s_load_b32 s15, s[0:1], 0x0
; GISEL-NEXT: v_mov_b32_e32 v1, s10
; GISEL-NEXT: s_add_co_u32 s6, s10, 4
@@ -407,9 +406,9 @@ define amdgpu_kernel void @test_pipelined_loop_with_global(ptr addrspace(1) %foo
; GISEL-NEXT: s_mov_b64 s[6:7], s[2:3]
; GISEL-NEXT: s_clause 0x2
; GISEL-NEXT: global_load_async_to_lds_b32 v1, v0, s[8:9] offset:4 nv
-; GISEL-NEXT: ; asyncmark
; GISEL-NEXT: global_load_b32 v1, v0, s[8:9] offset:4
; GISEL-NEXT: global_load_b32 v2, v0, s[0:1] offset:4
+; GISEL-NEXT: ; asyncmark
; GISEL-NEXT: s_wait_xcnt 0x0
; GISEL-NEXT: s_add_co_u32 s0, s0, 8
; GISEL-NEXT: s_add_co_ci_u32 s1, s1, 0
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-pregfx12.ll b/llvm/test/CodeGen/AMDGPU/asyncmark-pregfx12.ll
index 46af269d0a0a6..2fc3253d064c2 100644
--- a/llvm/test/CodeGen/AMDGPU/asyncmark-pregfx12.ll
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-pregfx12.ll
@@ -51,23 +51,22 @@ define void @interleaved_global_and_dma(ptr addrspace(1) %foo, ptr addrspace(3)
; SDAG-LABEL: interleaved_global_and_dma:
; SDAG: ; %bb.0: ; %entry
; SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; SDAG-NEXT: v_readfirstlane_b32 s4, v2
; SDAG-NEXT: global_load_dword v7, v[3:4], off
; SDAG-NEXT: global_load_dword v8, v[0:1], off
-; SDAG-NEXT: s_mov_b32 m0, s4
+; SDAG-NEXT: v_readfirstlane_b32 s4, v2
; SDAG-NEXT: ; wave barrier
+; SDAG-NEXT: s_mov_b32 m0, s4
+; SDAG-NEXT: global_load_dword v0, v[0:1], off
; SDAG-NEXT: s_nop 0
; SDAG-NEXT: global_load_dword v[3:4], off lds
; SDAG-NEXT: ; asyncmark
-; SDAG-NEXT: global_load_dword v0, v[0:1], off
; SDAG-NEXT: ; wave barrier
-; SDAG-NEXT: s_nop 0
; SDAG-NEXT: global_load_dword v[3:4], off lds
; SDAG-NEXT: ; wave barrier
; SDAG-NEXT: global_load_dword v1, v[3:4], off
; SDAG-NEXT: ; asyncmark
; SDAG-NEXT: ; wait_asyncmark(1)
-; SDAG-NEXT: s_waitcnt vmcnt(3)
+; SDAG-NEXT: s_waitcnt vmcnt(2)
; SDAG-NEXT: ds_read_b32 v3, v2
; SDAG-NEXT: ; wait_asyncmark(0)
; SDAG-NEXT: s_waitcnt vmcnt(1)
@@ -85,14 +84,14 @@ define void @interleaved_global_and_dma(ptr addrspace(1) %foo, ptr addrspace(3)
; GISEL: ; %bb.0: ; %entry
; GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GISEL-NEXT: v_readfirstlane_b32 s4, v2
+; GISEL-NEXT: s_mov_b32 m0, s4
; GISEL-NEXT: global_load_dword v7, v[3:4], off
; GISEL-NEXT: global_load_dword v8, v[0:1], off
-; GISEL-NEXT: s_mov_b32 m0, s4
; GISEL-NEXT: ; wave barrier
; GISEL-NEXT: s_nop 0
; GISEL-NEXT: global_load_dword v[3:4], off lds
-; GISEL-NEXT: ; asyncmark
; GISEL-NEXT: global_load_dword v0, v[0:1], off
+; GISEL-NEXT: ; asyncmark
; GISEL-NEXT: ; wave barrier
; GISEL-NEXT: s_nop 0
; GISEL-NEXT: global_load_dword v[3:4], off lds
@@ -154,15 +153,15 @@ define void @interleaved_buffer_and_dma(ptr addrspace(8) inreg %buf, ptr addrspa
; SDAG-LABEL: interleaved_buffer_and_dma:
; SDAG: ; %bb.0: ; %entry
; SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; SDAG-NEXT: s_mov_b32 m0, s20
; SDAG-NEXT: global_load_dword v6, v[2:3], off
; SDAG-NEXT: global_load_dword v7, v[0:1], off
-; SDAG-NEXT: v_mov_b32_e32 v8, 0x54
+; SDAG-NEXT: s_mov_b32 m0, s20
; SDAG-NEXT: ; wave barrier
-; SDAG-NEXT: buffer_load_dword v8, s[16:19], 0 offen lds
-; SDAG-NEXT: ; asyncmark
+; SDAG-NEXT: v_mov_b32_e32 v8, 0x54
; SDAG-NEXT: global_load_dword v0, v[0:1], off
; SDAG-NEXT: v_mov_b32_e32 v1, 0x58
+; SDAG-NEXT: buffer_load_dword v8, s[16:19], 0 offen lds
+; SDAG-NEXT: ; asyncmark
; SDAG-NEXT: ; wave barrier
; SDAG-NEXT: buffer_load_dword v1, s[16:19], 0 offen lds
; SDAG-NEXT: ; wave barrier
@@ -170,7 +169,7 @@ define void @interleaved_buffer_and_dma(ptr addrspace(8) inreg %buf, ptr addrspa
; SDAG-NEXT: v_mov_b32_e32 v2, s20
; SDAG-NEXT: ; asyncmark
; SDAG-NEXT: ; wait_asyncmark(1)
-; SDAG-NEXT: s_waitcnt vmcnt(3)
+; SDAG-NEXT: s_waitcnt vmcnt(2)
; SDAG-NEXT: ds_read_b32 v3, v2
; SDAG-NEXT: ; wait_asyncmark(0)
; SDAG-NEXT: s_waitcnt vmcnt(1)
@@ -188,14 +187,16 @@ define void @interleaved_buffer_and_dma(ptr addrspace(8) inreg %buf, ptr addrspa
; GISEL: ; %bb.0: ; %entry
; GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GISEL-NEXT: s_mov_b32 m0, s20
+; GISEL-NEXT: v_mov_b32_e32 v8, 0x54
; GISEL-NEXT: global_load_dword v6, v[2:3], off
; GISEL-NEXT: global_load_dword v7, v[0:1], off
-; GISEL-NEXT: v_mov_b32_e32 v8, 0x54
; GISEL-NEXT: ; wave barrier
+; GISEL-NEXT: s_waitcnt vmcnt(0)
+; GISEL-NEXT: v_add_u32_e32 v6, v7, v6
; GISEL-NEXT: buffer_load_dword v8, s[16:19], 0 offen lds
-; GISEL-NEXT: ; asyncmark
; GISEL-NEXT: global_load_dword v0, v[0:1], off
; GISEL-NEXT: v_mov_b32_e32 v1, 0x58
+; GISEL-NEXT: ; asyncmark
; GISEL-NEXT: ; wave barrier
; GISEL-NEXT: buffer_load_dword v1, s[16:19], 0 offen lds
; GISEL-NEXT: ; wave barrier
@@ -208,7 +209,6 @@ define void @interleaved_buffer_and_dma(ptr addrspace(8) inreg %buf, ptr addrspa
; GISEL-NEXT: ; wait_asyncmark(0)
; GISEL-NEXT: s_waitcnt vmcnt(1)
; GISEL-NEXT: ds_read_b32 v2, v2
-; GISEL-NEXT: v_add_u32_e32 v6, v7, v6
; GISEL-NEXT: s_waitcnt lgkmcnt(1)
; GISEL-NEXT: v_add3_u32 v0, v6, v3, v0
; GISEL-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
@@ -264,9 +264,9 @@ define void @fence_with_asyncmark(ptr addrspace(8) inreg %buf, ptr addrspace(1)
; SDAG-NEXT: ; wave barrier
; SDAG-NEXT: buffer_load_dword v8, s[16:19], 0 offen lds
; SDAG-NEXT: s_waitcnt vmcnt(0)
-; SDAG-NEXT: ; asyncmark
; SDAG-NEXT: global_load_dword v0, v[0:1], off
; SDAG-NEXT: v_mov_b32_e32 v1, 0x58
+; SDAG-NEXT: ; asyncmark
; SDAG-NEXT: ; wave barrier
; SDAG-NEXT: buffer_load_dword v1, s[16:19], 0 offen lds
; SDAG-NEXT: ; wave barrier
@@ -297,9 +297,9 @@ define void @fence_with_asyncmark(ptr addrspace(8) inreg %buf, ptr addrspace(1)
; GISEL-NEXT: ; wave barrier
; GISEL-NEXT: buffer_load_dword v8, s[16:19], 0 offen lds
; GISEL-NEXT: s_waitcnt vmcnt(0)
-; GISEL-NEXT: ; asyncmark
; GISEL-NEXT: global_load_dword v0, v[0:1], off
; GISEL-NEXT: v_mov_b32_e32 v1, 0x58
+; GISEL-NEXT: ; asyncmark
; GISEL-NEXT: ; wave barrier
; GISEL-NEXT: buffer_load_dword v1, s[16:19], 0 offen lds
; GISEL-NEXT: ; wave barrier
@@ -496,40 +496,41 @@ define void @test_pipelined_loop_with_global(ptr addrspace(1) %foo, ptr addrspac
; SDAG: ; %bb.0: ; %prolog
; SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; SDAG-NEXT: v_readfirstlane_b32 s4, v2
+; SDAG-NEXT: global_load_dword v14, v[0:1], off
+; SDAG-NEXT: global_load_dword v8, v[0:1], off
+; SDAG-NEXT: global_load_dword v15, v[3:4], off
+; SDAG-NEXT: global_load_dword v9, v[3:4], off
; SDAG-NEXT: s_mov_b32 m0, s4
-; SDAG-NEXT: global_load_dword v10, v[0:1], off
-; SDAG-NEXT: global_load_dword v14, v[3:4], off
; SDAG-NEXT: s_mov_b32 s6, 2
; SDAG-NEXT: global_load_dword v[0:1], off lds
; SDAG-NEXT: ; asyncmark
-; SDAG-NEXT: global_load_dword v8, v[0:1], off
-; SDAG-NEXT: global_load_dword v9, v[3:4], off
-; SDAG-NEXT: s_mov_b64 s[4:5], 0
; SDAG-NEXT: global_load_dword v[0:1], off lds
+; SDAG-NEXT: s_mov_b64 s[4:5], 0
; SDAG-NEXT: ; asyncmark
+; SDAG-NEXT: s_waitcnt vmcnt(4)
+; SDAG-NEXT: v_mov_b32_e32 v12, v8
; SDAG-NEXT: s_waitcnt vmcnt(2)
-; SDAG-NEXT: v_mov_b32_e32 v13, v8
-; SDAG-NEXT: s_waitcnt vmcnt(1)
-; SDAG-NEXT: v_mov_b32_e32 v15, v9
+; SDAG-NEXT: v_mov_b32_e32 v13, v9
; SDAG-NEXT: .LBB5_1: ; %loop_body
; SDAG-NEXT: ; =>This Inner Loop Header: Depth=1
; SDAG-NEXT: v_readfirstlane_b32 s7, v2
; SDAG-NEXT: s_waitcnt vmcnt(1)
-; SDAG-NEXT: v_mov_b32_e32 v12, v15
; SDAG-NEXT: v_mov_b32_e32 v11, v13
-; SDAG-NEXT: global_load_dword v13, v[0:1], off
-; SDAG-NEXT: global_load_dword v15, v[3:4], off
+; SDAG-NEXT: v_mov_b32_e32 v10, v12
+; SDAG-NEXT: global_load_dword v12, v[0:1], off
+; SDAG-NEXT: global_load_dword v13, v[3:4], off
; SDAG-NEXT: s_mov_b32 m0, s7
; SDAG-NEXT: s_add_i32 s6, s6, 1
; SDAG-NEXT: global_load_dword v[0:1], off lds
; SDAG-NEXT: v_cmp_ge_i32_e32 vcc, s6, v7
-; SDAG-NEXT: v_mov_b32_e32 v16, v14
-; SDAG-NEXT: v_mov_b32_e32 v17, v10
-; SDAG-NEXT: v_mov_b32_e32 v10, v8
-; SDAG-NEXT: v_mov_b32_e32 v14, v9
+; SDAG-NEXT: v_mov_b32_e32 v16, v15
+; SDAG-NEXT: v_mov_b32_e32 v17, v14
+; SDAG-NEXT: v_mov_b32_e32 v14, v8
+; SDAG-NEXT: v_mov_b32_e32 v15, v9
; SDAG-NEXT: s_or_b64 s[4:5], vcc, s[4:5]
; SDAG-NEXT: ; asyncmark
; SDAG-NEXT: ; wait_asyncmark(2)
+; SDAG-NEXT: s_waitcnt vmcnt(4)
; SDAG-NEXT: s_andn2_b64 exec, exec, s[4:5]
; SDAG-NEXT: s_cbranch_execnz .LBB5_1
; SDAG-NEXT: ; %bb.2: ; %epilog
@@ -543,10 +544,10 @@ define void @test_pipelined_loop_with_global(ptr addrspace(1) %foo, ptr addrspac
; SDAG-NEXT: ds_read_b32 v2, v2
; SDAG-NEXT: v_add_u32_e32 v3, v17, v16
; SDAG-NEXT: s_waitcnt lgkmcnt(2)
-; SDAG-NEXT: v_add3_u32 v0, v3, v0, v12
+; SDAG-NEXT: v_add3_u32 v0, v3, v0, v11
; SDAG-NEXT: s_waitcnt lgkmcnt(1)
-; SDAG-NEXT: v_add3_u32 v0, v11, v0, v1
-; SDAG-NEXT: v_add_u32_e32 v1, v13, v15
+; SDAG-NEXT: v_add3_u32 v0, v10, v0, v1
+; SDAG-NEXT: v_add_u32_e32 v1, v12, v13
; SDAG-NEXT: s_waitcnt lgkmcnt(0)
; SDAG-NEXT: v_add3_u32 v0, v1, v2, v0
; SDAG-NEXT: global_store_dword v[5:6], v0, off
@@ -562,9 +563,9 @@ define void @test_pipelined_loop_with_global(ptr addrspace(1) %foo, ptr addrspac
; GISEL-NEXT: global_load_dword v14, v[3:4], off
; GISEL-NEXT: s_mov_b32 s6, 2
; GISEL-NEXT: global_load_dword v[0:1], off lds
-; GISEL-NEXT: ; asyncmark
; GISEL-NEXT: global_load_dword v8, v[0:1], off
; GISEL-NEXT: global_load_dword v9, v[3:4], off
+; GISEL-NEXT: ; asyncmark
; GISEL-NEXT: s_mov_b64 s[4:5], 0
; GISEL-NEXT: global_load_dword v[0:1], off lds
; GISEL-NEXT: v_mov_b32_e32 v16, s6
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-sched-postra.mir b/llvm/test/CodeGen/AMDGPU/asyncmark-sched-postra.mir
new file mode 100644
index 0000000000000..c54523766e064
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-sched-postra.mir
@@ -0,0 +1,169 @@
+# RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1250 -passes=postmisched \
+# RUN: -debug-only=machine-scheduler -o /dev/null %s 2>&1 \
+# RUN: | FileCheck %s
+
+# REQUIRES: asserts
+
+# Pins the artificial edges added by the AMDGPU AsyncMark scheduling DAG
+# mutation across bundled and non-bundled shapes. Each test verifies the
+# `Predecessors:` block of the next ASYNCMARK / WAIT_ASYNCMARK in the
+# scheduler's SU dump.
+
+--- |
+ define void @bundle_mem_then_mark() { unreachable }
+ define void @bundle_mark_then_mem() { unreachable }
+ define void @nonbundle_two_batches() { unreachable }
+ define void @mixed_around_mark() { unreachable }
+...
+
+# A BUNDLE { async load, ASYNCMARK } is self-tagged: the in-bundle MARK
+# clears PostMarkAsyncMemOp, so the bundle is not pushed to the pending-
+# async list. The next ASYNCMARK gets exactly two artificial predecessors
+# (rule 1 from the bundle, rule 2 from the standalone load).
+#
+# CHECK-LABEL: bundle_mem_then_mark:%bb.0
+# CHECK-LABEL: SU(0):{{.*}}BUNDLE
+# CHECK-LABEL: SU(1):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(2):{{.*}}ASYNCMARK
+# CHECK: Predecessors:
+# CHECK-NEXT: SU(0):{{.*}}Artificial
+# CHECK-NEXT: SU(1):{{.*}}Artificial
+# CHECK-NEXT: Successors:
+# CHECK-LABEL: SU(3):{{.*}}WAIT_ASYNCMARK
+# CHECK: Predecessors:
+# CHECK: SU(2):{{.*}}Artificial
+# CHECK-NEXT: ExitSU:
+---
+name: bundle_mem_then_mark
+tracksRegLiveness: true
+machineFunctionInfo:
+ occupancy: 8
+body: |
+ bb.0:
+ liveins: $vgpr0_vgpr1, $vgpr2, $vgpr4_vgpr5, $vgpr3
+
+ BUNDLE implicit-def $asynccnt, implicit $vgpr0_vgpr1, implicit $vgpr2, implicit $exec, implicit $asynccnt {
+ GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+ ASYNCMARK
+ }
+ GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr3, $vgpr4_vgpr5, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+ ASYNCMARK
+ WAIT_ASYNCMARK 0
+ S_ENDPGM 0
+...
+
+# A BUNDLE { ASYNCMARK, async load }: the in-bundle MARK does not tag the
+# trailing in-bundle mem, so the bundle is classified Mark|Post and is
+# pushed to the pending-async list. The next external ASYNCMARK chains to
+# the bundle (the LastMark + drain edges fold into a single artificial
+# predecessor in the dump).
+#
+# CHECK-LABEL: bundle_mark_then_mem:%bb.0
+# CHECK-LABEL: SU(0):{{.*}}BUNDLE
+# CHECK-LABEL: SU(1):{{.*}}ASYNCMARK
+# CHECK: Predecessors:
+# CHECK-NEXT: SU(0):{{.*}}Artificial
+# CHECK-NEXT: Successors:
+# CHECK-LABEL: SU(2):{{.*}}WAIT_ASYNCMARK
+# CHECK: Predecessors:
+# CHECK: SU(1):{{.*}}Artificial
+# CHECK-NEXT: ExitSU:
+---
+name: bundle_mark_then_mem
+tracksRegLiveness: true
+machineFunctionInfo:
+ occupancy: 8
+body: |
+ bb.0:
+ liveins: $vgpr0_vgpr1, $vgpr2
+
+ BUNDLE implicit-def $asynccnt, implicit $vgpr0_vgpr1, implicit $vgpr2, implicit $exec, implicit $asynccnt {
+ ASYNCMARK
+ GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+ }
+ ASYNCMARK
+ WAIT_ASYNCMARK 0
+ S_ENDPGM 0
+...
+
+# Two-batch non-bundled case: load, load, MARK1, load, MARK2, WAIT.
+# MARK1 drains both pending loads (SU(0), SU(1)). MARK2 chains to MARK1
+# (rule 1) and drains the third load (rule 2). WAIT chains to MARK2.
+#
+# CHECK-LABEL: nonbundle_two_batches:%bb.0
+# CHECK-LABEL: SU(0):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(1):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(2):{{.*}}ASYNCMARK
+# CHECK: Predecessors:
+# CHECK-NEXT: SU(0):{{.*}}Artificial
+# CHECK-NEXT: SU(1):{{.*}}Artificial
+# CHECK-NEXT: Successors:
+# CHECK-LABEL: SU(3):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(4):{{.*}}ASYNCMARK
+# CHECK: Predecessors:
+# CHECK-NEXT: SU(2):{{.*}}Artificial
+# CHECK-NEXT: SU(3):{{.*}}Artificial
+# CHECK-NEXT: Successors:
+# CHECK-LABEL: SU(5):{{.*}}WAIT_ASYNCMARK
+# CHECK: Predecessors:
+# CHECK: SU(4):{{.*}}Artificial
+# CHECK-NEXT: ExitSU:
+---
+name: nonbundle_two_batches
+tracksRegLiveness: true
+machineFunctionInfo:
+ occupancy: 8
+body: |
+ bb.0:
+ liveins: $vgpr0_vgpr1, $vgpr2, $vgpr4_vgpr5, $vgpr3
+
+ GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+ GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr3, $vgpr4_vgpr5, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+ ASYNCMARK
+ GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+ ASYNCMARK
+ WAIT_ASYNCMARK 0
+ S_ENDPGM 0
+...
+
+# ASYNCMARK has no dependencies with non-async memory ops. With async
+# loads and a mix of non-async reads/writes on both sides, MARK's full
+# pred/succ list contains only the async loads and the WAIT.
+#
+# CHECK-LABEL: mixed_around_mark:%bb.0
+# CHECK-LABEL: SU(0):{{.*}}GLOBAL_LOAD_DWORD
+# CHECK-LABEL: SU(1):{{.*}}DS_READ_B32
+# CHECK-LABEL: SU(2):{{.*}}GLOBAL_STORE_DWORD
+# CHECK-LABEL: SU(3):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(4):{{.*}}ASYNCMARK
+# CHECK: Predecessors:
+# CHECK-NEXT: SU(3):{{.*}}Artificial
+# CHECK-NEXT: Successors:
+# CHECK-NEXT: SU(8):{{.*}}Artificial
+# CHECK-NEXT: SU(9):{{.*}}Artificial
+# CHECK-NEXT: SU(5):{{.*}}BUFFER_LOAD_DWORD_OFFSET
+# CHECK-LABEL: SU(6):{{.*}}DS_WRITE_B32
+# CHECK-LABEL: SU(7):{{.*}}BUFFER_STORE_DWORD_OFFSET
+# CHECK-LABEL: SU(8):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(9):{{.*}}WAIT_ASYNCMARK
+---
+name: mixed_around_mark
+tracksRegLiveness: true
+machineFunctionInfo:
+ occupancy: 8
+body: |
+ bb.0:
+ liveins: $vgpr0_vgpr1, $vgpr4_vgpr5, $vgpr2, $vgpr3, $sgpr0_sgpr1_sgpr2_sgpr3
+
+ $vgpr6 = GLOBAL_LOAD_DWORD $vgpr0_vgpr1, 0, 0, implicit $exec :: (load (s32), addrspace 1)
+ $vgpr7 = DS_READ_B32 $vgpr2, 0, 0, implicit $m0, implicit $exec :: (load (s32), addrspace 3)
+ GLOBAL_STORE_DWORD $vgpr0_vgpr1, $vgpr3, 0, 0, implicit $exec :: (store (s32), addrspace 1)
+ GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr3, $vgpr4_vgpr5, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+ ASYNCMARK
+ $vgpr8 = BUFFER_LOAD_DWORD_OFFSET $sgpr0_sgpr1_sgpr2_sgpr3, 0, 0, 0, 0, implicit $exec :: (load (s32))
+ DS_WRITE_B32 $vgpr2, $vgpr3, 0, 0, implicit $m0, implicit $exec :: (store (s32), addrspace 3)
+ BUFFER_STORE_DWORD_OFFSET $vgpr3, $sgpr0_sgpr1_sgpr2_sgpr3, 0, 0, 0, 0, implicit $exec :: (store (s32))
+ GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr3, $vgpr4_vgpr5, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+ WAIT_ASYNCMARK 0
+ S_ENDPGM 0
+...
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-sched-prera.mir b/llvm/test/CodeGen/AMDGPU/asyncmark-sched-prera.mir
new file mode 100644
index 0000000000000..654cd26412187
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-sched-prera.mir
@@ -0,0 +1,109 @@
+# RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1250 -run-pass=machine-scheduler \
+# RUN: -debug-only=machine-scheduler -o /dev/null %s 2>&1 \
+# RUN: | FileCheck %s
+
+# REQUIRES: asserts
+
+# Two-batch case: load, load, MARK, load, MARK, WAIT.
+# After 4 IMPLICIT_DEF set-up SUs (SU(0)..SU(3)), the async ops and
+# marks become SU(4)..SU(9):
+# SU(4), SU(5), SU(7) = async loads
+# SU(6), SU(8) = ASYNCMARKs
+# SU(9) = WAIT_ASYNCMARK
+#
+# CHECK-LABEL: two_batches:%bb.0
+# CHECK-LABEL: SU(4):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(5):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(6):{{.*}}ASYNCMARK
+# CHECK: Predecessors:
+# CHECK-NEXT: SU(4):{{.*}}Artificial
+# CHECK-NEXT: SU(5):{{.*}}Artificial
+# CHECK-NEXT: Successors:
+# CHECK-LABEL: SU(7):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(8):{{.*}}ASYNCMARK
+# CHECK: Predecessors:
+# CHECK-NEXT: SU(6):{{.*}}Artificial
+# CHECK-NEXT: SU(7):{{.*}}Artificial
+# CHECK-NEXT: Successors:
+# CHECK-LABEL: SU(9):{{.*}}WAIT_ASYNCMARK
+# CHECK: Predecessors:
+# CHECK: SU(8):{{.*}}Artificial
+# CHECK-NEXT: Pressure Diff
+---
+name: two_batches
+tracksRegLiveness: true
+body: |
+ bb.0:
+ %0:vreg_64_align2 = IMPLICIT_DEF
+ %1:vgpr_32 = IMPLICIT_DEF
+ %2:vreg_64_align2 = IMPLICIT_DEF
+ %3:vgpr_32 = IMPLICIT_DEF
+ GLOBAL_LOAD_ASYNC_TO_LDS_B32 %1, %0, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+ GLOBAL_LOAD_ASYNC_TO_LDS_B32 %3, %2, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+ ASYNCMARK
+ GLOBAL_LOAD_ASYNC_TO_LDS_B32 %1, %0, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+ ASYNCMARK
+ WAIT_ASYNCMARK 0
+ S_ENDPGM 0
+...
+
+# ASYNCMARK is exempt from dependencies with non-async memory ops, but
+# still picks up rule-2 edges from the surrounding async LDS DMAs and a
+# rule-1 edge to WAIT_ASYNCMARK. Mix a non-async read, non-async write,
+# and LDS read/write on each side, plus an async load on each side.
+#
+# After 5 IMPLICIT_DEFs (SU(0)..SU(4)):
+# SU(5) = GLOBAL_LOAD_DWORD (before MARK, non-async read)
+# SU(6) = DS_READ_B32 (before MARK, lds read)
+# SU(7) = GLOBAL_STORE_DWORD (before MARK, non-async write)
+# SU(8) = GLOBAL_LOAD_ASYNC_TO_LDS_B32 (before MARK, async)
+# SU(9) = ASYNCMARK
+# SU(10) = BUFFER_LOAD_DWORD_OFFSET (after MARK, non-async read)
+# SU(11) = DS_WRITE_B32 (after MARK, lds write)
+# SU(12) = BUFFER_STORE_DWORD_OFFSET (after MARK, non-async write)
+# SU(13) = GLOBAL_LOAD_ASYNC_TO_LDS_B32 (after MARK, async)
+# SU(14) = WAIT_ASYNCMARK
+#
+# Asserting MARK's full Predecessors/Successors with CHECK-NEXT
+# guarantees that none of the six non-async memory ops are connected to
+# the MARK - only the two async loads and the WAIT are.
+#
+# CHECK-LABEL: mixed_around_mark:%bb.0
+# CHECK-LABEL: SU(5):{{.*}}GLOBAL_LOAD_DWORD
+# CHECK-LABEL: SU(6):{{.*}}DS_READ_B32
+# CHECK-LABEL: SU(7):{{.*}}GLOBAL_STORE_DWORD
+# CHECK-LABEL: SU(8):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(9):{{.*}}ASYNCMARK
+# CHECK: Predecessors:
+# CHECK-NEXT: SU(8):{{.*}}Artificial
+# CHECK-NEXT: Successors:
+# CHECK-NEXT: SU(13):{{.*}}Artificial
+# CHECK-NEXT: SU(14):{{.*}}Artificial
+# CHECK-NEXT: Pressure Diff
+# CHECK-LABEL: SU(10):{{.*}}BUFFER_LOAD_DWORD_OFFSET
+# CHECK-LABEL: SU(11):{{.*}}DS_WRITE_B32
+# CHECK-LABEL: SU(12):{{.*}}BUFFER_STORE_DWORD_OFFSET
+# CHECK-LABEL: SU(13):{{.*}}GLOBAL_LOAD_ASYNC_TO_LDS_B32
+# CHECK-LABEL: SU(14):{{.*}}WAIT_ASYNCMARK
+---
+name: mixed_around_mark
+tracksRegLiveness: true
+body: |
+ bb.0:
+ %0:vreg_64_align2 = IMPLICIT_DEF
+ %1:vreg_64_align2 = IMPLICIT_DEF
+ %2:vgpr_32 = IMPLICIT_DEF
+ %3:vgpr_32 = IMPLICIT_DEF
+ %4:sgpr_128 = IMPLICIT_DEF
+ %5:vgpr_32 = GLOBAL_LOAD_DWORD %0, 0, 0, implicit $exec :: (load (s32), addrspace 1)
+ %6:vgpr_32 = DS_READ_B32 %2, 0, 0, implicit $m0, implicit $exec :: (load (s32), addrspace 3)
+ GLOBAL_STORE_DWORD %0, %3, 0, 0, implicit $exec :: (store (s32), addrspace 1)
+ GLOBAL_LOAD_ASYNC_TO_LDS_B32 %3, %1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+ ASYNCMARK
+ %7:vgpr_32 = BUFFER_LOAD_DWORD_OFFSET %4, 0, 0, 0, 0, implicit $exec :: (load (s32))
+ DS_WRITE_B32 %2, %3, 0, 0, implicit $m0, implicit $exec :: (store (s32), addrspace 3)
+ BUFFER_STORE_DWORD_OFFSET %3, %4, 0, 0, 0, 0, implicit $exec :: (store (s32))
+ GLOBAL_LOAD_ASYNC_TO_LDS_B32 %3, %1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
+ WAIT_ASYNCMARK 0
+ S_ENDPGM 0
+...
More information about the llvm-commits
mailing list