[llvm] [AMDGPU][SIInsertWaitcnts] Fix iota_range assertion when OtherMarks is empty in mergeAsyncMarks() (PR #193499)
via llvm-commits
llvm-commits at lists.llvm.org
Fri May 1 06:39:10 PDT 2026
https://github.com/xiaohuguo2023 updated https://github.com/llvm/llvm-project/pull/193499
>From d4c3e0c41fc35ae605809d39f4c27c245f34e766 Mon Sep 17 00:00:00 2001
From: Xiaohu Guo <Xiaohu.Guo at amd.com>
Date: Mon, 20 Apr 2026 21:25:09 +0000
Subject: [PATCH 1/9] [AMDGPU][SIInsertWaitcnts] Fix iota_range assert when
MergeCount == 0
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
In mergeAsyncMarks(), when one branch of a conditional (e.g. EVEN_K=False
K-loop boundary) has no async memory ops, OtherMarks can be empty while
AsyncMarks is not. The existing early-exit only handles both-empty, so
MergeCount = min(0, N) = 0 reaches seq_inclusive<unsigned>(1, 0) which
asserts Begin <= End.
Fix: return early when MergeCount == 0 — nothing to merge.
Reproducer: compile a StreamK GEMM kernel on gfx950 (MI350X) with
BLOCK_SIZE_K=256, K=2880 (not divisible → EVEN_K=False),
num_stages=2, waves_per_eu=4, matrix_instr_nonkdim=32.
The assert fires deterministically even when compiled alone (not a race).
Fixes: iota_range<unsigned>(1, 0) → Assertion Begin <= End failed
---
llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp | 2 +
.../AMDGPU/asyncmark-merge-empty-other.mir | 75 +++++++++++++++++++
2 files changed, 77 insertions(+)
create mode 100644 llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
diff --git a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
index f00b99075b022..2ae46245b486f 100644
--- a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
+++ b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
@@ -3067,6 +3067,8 @@ bool WaitcntBrackets::mergeAsyncMarks(ArrayRef<MergeInfo> MergeInfos,
unsigned OtherSize = OtherMarks.size();
unsigned OurSize = AsyncMarks.size();
unsigned MergeCount = std::min(OtherSize, OurSize);
+ if (MergeCount == 0)
+ return StrictDom;
for (auto Idx : seq_inclusive<unsigned>(1, MergeCount)) {
for (auto T : inst_counter_types(Context->MaxCounter)) {
StrictDom |= mergeScore(MergeInfos[T], AsyncMarks[OurSize - Idx][T],
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
new file mode 100644
index 0000000000000..ec9c33f33a1f8
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
@@ -0,0 +1,75 @@
+# RUN: llc -mtriple=amdgcn -mcpu=gfx950 -verify-machineinstrs -run-pass=si-insert-waitcnts -o - %s | FileCheck %s
+
+# Regression test for mergeAsyncMarks() asserting when OtherMarks is empty.
+#
+# At a CFG join point where one predecessor has an ASYNCMARK (non-empty
+# AsyncMarks) and the other does not (empty OtherMarks), MergeCount becomes
+# min(0, N) = 0. Before the fix, seq_inclusive<unsigned>(1, 0) asserted
+# Begin <= End. After the fix the function returns early: the other predecessor
+# contributed no async marks so our marks are unchanged and no stricter waits
+# are needed.
+#
+# Key check: no S_WAITCNT is inserted at the join block (bb.3). The else
+# branch had no async operations so merging its empty marks into the then
+# branch's non-empty marks requires no additional waits.
+
+--- |
+ define void @asyncmark_merge_empty_other(i32 %cond) {
+ entry:
+ br i1 true, label %then, label %else
+ then:
+ br label %join
+ else:
+ br label %join
+ join:
+ ret void
+ }
+...
+
+---
+name: asyncmark_merge_empty_other
+tracksRegLiveness: true
+machineFunctionInfo:
+ occupancy: 8
+body: |
+ ; entry — conditional branch to then (async) or else (no async)
+ bb.0.entry:
+ successors: %bb.1(0x40000000), %bb.2(0x40000000)
+ liveins: $sgpr0, $sgpr1
+
+ ; CHECK-LABEL: name: asyncmark_merge_empty_other
+ ; CHECK: bb.0.entry:
+ ; CHECK: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
+ ; CHECK-NEXT: S_CBRANCH_SCC1 %bb.2, implicit killed $scc
+ S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
+ S_CBRANCH_SCC1 %bb.2, implicit killed $scc
+
+ ; then branch — ASYNCMARK records non-empty AsyncMarks
+ bb.1.then:
+ successors: %bb.3(0x80000000)
+
+ ; CHECK: bb.1.then:
+ ; CHECK: ASYNCMARK
+ ; CHECK-NEXT: S_BRANCH %bb.3
+ ASYNCMARK
+ S_BRANCH %bb.3
+
+ ; else branch — no async operations, OtherMarks is empty at join
+ bb.2.else:
+ successors: %bb.3(0x80000000)
+
+ ; CHECK: bb.2.else:
+ ; CHECK: S_BRANCH %bb.3
+ S_BRANCH %bb.3
+
+ ; join — mergeAsyncMarks() sees non-empty AsyncMarks (bb.1) and empty
+ ; OtherMarks (bb.2). Before the fix: assertion. After the fix: early
+ ; return, no waitcnt inserted.
+ bb.3.join:
+ ; CHECK: bb.3.join:
+ ; CHECK-NOT: S_WAITCNT
+ ; CHECK: S_ENDPGM 0
+ ; Verify no waitcnt is inserted before S_ENDPGM — the else predecessor
+ ; had no async marks so no stricter waits are needed at the join.
+ S_ENDPGM 0
+...
>From 553da6ecdbe1c2a8e5c0032626d46381a279a143 Mon Sep 17 00:00:00 2001
From: Xiaohu Guo <Xiaohu.Guo at amd.com>
Date: Wed, 22 Apr 2026 09:37:01 -0500
Subject: [PATCH 2/9] Address review feedback: - Regenerate CHECK lines using
update_mir_test_checks.py - Remove -verify-machineinstrs from RUN line
---
.../AMDGPU/asyncmark-merge-empty-other.mir | 37 ++++++++++++-------
1 file changed, 24 insertions(+), 13 deletions(-)
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
index ec9c33f33a1f8..93c9ae133ca77 100644
--- a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
@@ -1,4 +1,5 @@
-# RUN: llc -mtriple=amdgcn -mcpu=gfx950 -verify-machineinstrs -run-pass=si-insert-waitcnts -o - %s | FileCheck %s
+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 6
+# RUN: llc -mtriple=amdgcn -mcpu=gfx950 -run-pass=si-insert-waitcnts -o - %s | FileCheck %s
# Regression test for mergeAsyncMarks() asserting when OtherMarks is empty.
#
@@ -32,15 +33,33 @@ tracksRegLiveness: true
machineFunctionInfo:
occupancy: 8
body: |
+ ; CHECK-LABEL: name: asyncmark_merge_empty_other
+ ; CHECK: bb.0.entry:
+ ; CHECK-NEXT: successors: %bb.1(0x40000000), %bb.2(0x40000000)
+ ; CHECK-NEXT: liveins: $sgpr0, $sgpr1
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: S_WAITCNT 0
+ ; CHECK-NEXT: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
+ ; CHECK-NEXT: S_CBRANCH_SCC1 %bb.2, implicit killed $scc
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.1.then:
+ ; CHECK-NEXT: successors: %bb.3(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: ASYNCMARK
+ ; CHECK-NEXT: S_BRANCH %bb.3
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.2.else:
+ ; CHECK-NEXT: successors: %bb.3(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: S_BRANCH %bb.3
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.3.join:
+ ; CHECK-NEXT: S_ENDPGM 0
; entry — conditional branch to then (async) or else (no async)
bb.0.entry:
successors: %bb.1(0x40000000), %bb.2(0x40000000)
liveins: $sgpr0, $sgpr1
- ; CHECK-LABEL: name: asyncmark_merge_empty_other
- ; CHECK: bb.0.entry:
- ; CHECK: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
- ; CHECK-NEXT: S_CBRANCH_SCC1 %bb.2, implicit killed $scc
S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
S_CBRANCH_SCC1 %bb.2, implicit killed $scc
@@ -48,9 +67,6 @@ body: |
bb.1.then:
successors: %bb.3(0x80000000)
- ; CHECK: bb.1.then:
- ; CHECK: ASYNCMARK
- ; CHECK-NEXT: S_BRANCH %bb.3
ASYNCMARK
S_BRANCH %bb.3
@@ -58,17 +74,12 @@ body: |
bb.2.else:
successors: %bb.3(0x80000000)
- ; CHECK: bb.2.else:
- ; CHECK: S_BRANCH %bb.3
S_BRANCH %bb.3
; join — mergeAsyncMarks() sees non-empty AsyncMarks (bb.1) and empty
; OtherMarks (bb.2). Before the fix: assertion. After the fix: early
; return, no waitcnt inserted.
bb.3.join:
- ; CHECK: bb.3.join:
- ; CHECK-NOT: S_WAITCNT
- ; CHECK: S_ENDPGM 0
; Verify no waitcnt is inserted before S_ENDPGM — the else predecessor
; had no async marks so no stricter waits are needed at the join.
S_ENDPGM 0
>From e3dc3a875be3af7d77421aea00c9b1c7174f8e37 Mon Sep 17 00:00:00 2001
From: Xiaohu Guo <Xiaohu.Guo at amd.com>
Date: Fri, 24 Apr 2026 12:38:23 -0500
Subject: [PATCH 3/9] Address reviw feedback:
Change `&&` to `||` in the early-exit check so the function returns
immediately when either AsyncMarks or OtherMarks is empty, rather than
falling through to the resize/pad logic and hitting a seq_inclusive
assert (Begin > End) when MergeCount == 0. Removes the now-unreachable
mid-function MergeCount == 0 guard.
---
llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
index 2ae46245b486f..c3267bfc13a40 100644
--- a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
+++ b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
@@ -3025,8 +3025,11 @@ bool WaitcntBrackets::mergeAsyncMarks(ArrayRef<MergeInfo> MergeInfos,
bool StrictDom = false;
LLVM_DEBUG(dbgs() << "Merging async marks ...");
- // Early exit: both empty
- if (AsyncMarks.empty() && OtherMarks.empty()) {
+ // Early exit: nothing to merge when either side is empty.
+ // If OtherMarks is empty the join point has no async marks from that
+ // predecessor, so our existing marks are already the conservative result.
+ // If both are empty there is simply nothing to do.
+ if (AsyncMarks.empty() || OtherMarks.empty()) {
LLVM_DEBUG(dbgs() << " nothing to merge\n");
return false;
}
@@ -3067,8 +3070,6 @@ bool WaitcntBrackets::mergeAsyncMarks(ArrayRef<MergeInfo> MergeInfos,
unsigned OtherSize = OtherMarks.size();
unsigned OurSize = AsyncMarks.size();
unsigned MergeCount = std::min(OtherSize, OurSize);
- if (MergeCount == 0)
- return StrictDom;
for (auto Idx : seq_inclusive<unsigned>(1, MergeCount)) {
for (auto T : inst_counter_types(Context->MaxCounter)) {
StrictDom |= mergeScore(MergeInfos[T], AsyncMarks[OurSize - Idx][T],
>From 59989456fffb28f529942992736507f1e324cd9b Mon Sep 17 00:00:00 2001
From: Xiaohu Guo <Xiaohu.Guo at amd.com>
Date: Mon, 27 Apr 2026 05:13:12 -0500
Subject: [PATCH 4/9] add MIR regression test for mergeAsyncMarks with
one-side-empty marks
Test two CFG patterns where one predecessor carries an ASYNCMARK from a
BUFFER_LOAD and the other is a sync path (empty OtherMarks):
asyncmark_in_then: ASYNCMARK in then-successor, else is sync
asyncmark_in_else: ASYNCMARK in else-successor, then is sync
Before the fix, mergeAsyncMarks() asserted Begin <= End via
seq_inclusive<unsigned>(1, 0). After the fix the function returns
early when either side is empty and preserves the pending async marks,
so S_WAITCNT is correctly emitted before the consumer in the join block
---
.../AMDGPU/asyncmark-merge-empty-other.mir | 138 +++++++++++++++---
1 file changed, 117 insertions(+), 21 deletions(-)
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
index 93c9ae133ca77..2119c28bda70c 100644
--- a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
@@ -4,18 +4,33 @@
# Regression test for mergeAsyncMarks() asserting when OtherMarks is empty.
#
# At a CFG join point where one predecessor has an ASYNCMARK (non-empty
-# AsyncMarks) and the other does not (empty OtherMarks), MergeCount becomes
-# min(0, N) = 0. Before the fix, seq_inclusive<unsigned>(1, 0) asserted
-# Begin <= End. After the fix the function returns early: the other predecessor
-# contributed no async marks so our marks are unchanged and no stricter waits
-# are needed.
+# AsyncMarks) and the other does not (empty OtherMarks), before the fix
+# seq_inclusive<unsigned>(1, 0) would assert Begin <= End.
#
-# Key check: no S_WAITCNT is inserted at the join block (bb.3). The else
-# branch had no async operations so merging its empty marks into the then
-# branch's non-empty marks requires no additional waits.
+# After the fix the function returns early when either side is empty. The
+# pending async marks from the non-empty predecessor are preserved, so any
+# consumer instruction in the join block still receives the correct S_WAITCNT.
+#
+# Two patterns are tested:
+# asyncmark_in_then - ASYNCMARK in the then-successor, else-successor is sync
+# asyncmark_in_else - ASYNCMARK in the else-successor, then-successor is sync
+#
+# In both cases the join block must emit S_WAITCNT to drain the pending async
+# buffer load before the result is consumed.
--- |
- define void @asyncmark_merge_empty_other(i32 %cond) {
+ define void @asyncmark_in_then(i32 %cond) {
+ entry:
+ br i1 true, label %then, label %else
+ then:
+ br label %join
+ else:
+ br label %join
+ join:
+ ret void
+ }
+
+ define void @asyncmark_in_else(i32 %cond) {
entry:
br i1 true, label %then, label %else
then:
@@ -28,15 +43,17 @@
...
---
-name: asyncmark_merge_empty_other
+# Pattern 1: ASYNCMARK in then-successor, else-successor is sync.
+# The join block must wait for the async load from the then path.
+name: asyncmark_in_then
tracksRegLiveness: true
machineFunctionInfo:
occupancy: 8
body: |
- ; CHECK-LABEL: name: asyncmark_merge_empty_other
+ ; CHECK-LABEL: name: asyncmark_in_then
; CHECK: bb.0.entry:
; CHECK-NEXT: successors: %bb.1(0x40000000), %bb.2(0x40000000)
- ; CHECK-NEXT: liveins: $sgpr0, $sgpr1
+ ; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: S_WAITCNT 0
; CHECK-NEXT: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
@@ -44,43 +61,122 @@ body: |
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.1.then:
; CHECK-NEXT: successors: %bb.3(0x80000000)
+ ; CHECK-NEXT: liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, implicit $exec
; CHECK-NEXT: ASYNCMARK
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.2.else:
; CHECK-NEXT: successors: %bb.3(0x80000000)
; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: $vgpr0 = V_MOV_B32_e32 0, implicit $exec
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.3.join:
+ ; CHECK-NEXT: liveins: $vgpr0
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: S_WAITCNT 3952
+ ; CHECK-NEXT: $vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
; CHECK-NEXT: S_ENDPGM 0
- ; entry — conditional branch to then (async) or else (no async)
bb.0.entry:
successors: %bb.1(0x40000000), %bb.2(0x40000000)
- liveins: $sgpr0, $sgpr1
+ liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
S_CBRANCH_SCC1 %bb.2, implicit killed $scc
- ; then branch — ASYNCMARK records non-empty AsyncMarks
+ ; then branch — issues async buffer load + ASYNCMARK
bb.1.then:
successors: %bb.3(0x80000000)
+ liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
+ $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, implicit $exec
ASYNCMARK
S_BRANCH %bb.3
- ; else branch — no async operations, OtherMarks is empty at join
+ ; else branch — sync path, no async operations, OtherMarks is empty at join
bb.2.else:
successors: %bb.3(0x80000000)
+ $vgpr0 = V_MOV_B32_e32 0, implicit $exec
S_BRANCH %bb.3
- ; join — mergeAsyncMarks() sees non-empty AsyncMarks (bb.1) and empty
- ; OtherMarks (bb.2). Before the fix: assertion. After the fix: early
- ; return, no waitcnt inserted.
+ ; join — mergeAsyncMarks sees non-empty AsyncMarks (then) and empty OtherMarks (else).
+ ; Before fix: assertion. After fix: returns early, preserves async marks.
+ ; S_WAITCNT must be emitted before the use of $vgpr0 to drain the pending load.
+ ; (mergeAsyncMarks sees non-empty AsyncMarks from then, empty from else)
bb.3.join:
- ; Verify no waitcnt is inserted before S_ENDPGM — the else predecessor
- ; had no async marks so no stricter waits are needed at the join.
+ liveins: $vgpr0
+
+ $vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
+ S_ENDPGM 0
+...
+---
+# Pattern 2: ASYNCMARK in else-successor, then-successor is sync.
+# Mirror of asyncmark_in_then — exercises the opposite predecessor ordering.
+name: asyncmark_in_else
+tracksRegLiveness: true
+machineFunctionInfo:
+ occupancy: 8
+body: |
+ ; CHECK-LABEL: name: asyncmark_in_else
+ ; CHECK: bb.0.entry:
+ ; CHECK-NEXT: successors: %bb.1(0x40000000), %bb.2(0x40000000)
+ ; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: S_WAITCNT 0
+ ; CHECK-NEXT: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
+ ; CHECK-NEXT: S_CBRANCH_SCC1 %bb.2, implicit killed $scc
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.1.then:
+ ; CHECK-NEXT: successors: %bb.3(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: $vgpr0 = V_MOV_B32_e32 0, implicit $exec
+ ; CHECK-NEXT: S_BRANCH %bb.3
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.2.else:
+ ; CHECK-NEXT: successors: %bb.3(0x80000000)
+ ; CHECK-NEXT: liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, implicit $exec
+ ; CHECK-NEXT: ASYNCMARK
+ ; CHECK-NEXT: S_BRANCH %bb.3
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.3.join:
+ ; CHECK-NEXT: liveins: $vgpr0
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: S_WAITCNT 3952
+ ; CHECK-NEXT: $vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
+ ; CHECK-NEXT: S_ENDPGM 0
+ bb.0.entry:
+ successors: %bb.1(0x40000000), %bb.2(0x40000000)
+ liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
+
+ S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
+ S_CBRANCH_SCC1 %bb.2, implicit killed $scc
+
+ ; then branch — sync path, no async operations
+ bb.1.then:
+ successors: %bb.3(0x80000000)
+
+ $vgpr0 = V_MOV_B32_e32 0, implicit $exec
+ S_BRANCH %bb.3
+
+ ; else branch — issues async buffer load + ASYNCMARK
+ bb.2.else:
+ successors: %bb.3(0x80000000)
+ liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
+
+ $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, implicit $exec
+ ASYNCMARK
+ S_BRANCH %bb.3
+
+ ; join — mergeAsyncMarks sees empty AsyncMarks (then) and non-empty OtherMarks (else).
+ ; S_WAITCNT must be emitted before the use of $vgpr0 to drain the pending load.
+ bb.3.join:
+ liveins: $vgpr0
+
+ $vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
S_ENDPGM 0
...
>From 3816fcccc1a5f0cbe9bd197cf1885eed7983785a Mon Sep 17 00:00:00 2001
From: Xiaohu Guo <Xiaohu.Guo at amd.com>
Date: Mon, 27 Apr 2026 09:41:55 -0500
Subject: [PATCH 5/9] remove the IR section and plain BB name suffixes
(.entry/.then/.else/.join) as suggested in review
---
.../AMDGPU/asyncmark-merge-empty-other.mir | 56 ++++++-------------
1 file changed, 16 insertions(+), 40 deletions(-)
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
index 2119c28bda70c..0dd3738a71048 100644
--- a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
@@ -18,30 +18,6 @@
# In both cases the join block must emit S_WAITCNT to drain the pending async
# buffer load before the result is consumed.
---- |
- define void @asyncmark_in_then(i32 %cond) {
- entry:
- br i1 true, label %then, label %else
- then:
- br label %join
- else:
- br label %join
- join:
- ret void
- }
-
- define void @asyncmark_in_else(i32 %cond) {
- entry:
- br i1 true, label %then, label %else
- then:
- br label %join
- else:
- br label %join
- join:
- ret void
- }
-...
-
---
# Pattern 1: ASYNCMARK in then-successor, else-successor is sync.
# The join block must wait for the async load from the then path.
@@ -51,7 +27,7 @@ machineFunctionInfo:
occupancy: 8
body: |
; CHECK-LABEL: name: asyncmark_in_then
- ; CHECK: bb.0.entry:
+ ; CHECK: bb.0:
; CHECK-NEXT: successors: %bb.1(0x40000000), %bb.2(0x40000000)
; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
; CHECK-NEXT: {{ $}}
@@ -59,7 +35,7 @@ body: |
; CHECK-NEXT: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
; CHECK-NEXT: S_CBRANCH_SCC1 %bb.2, implicit killed $scc
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: bb.1.then:
+ ; CHECK-NEXT: bb.1:
; CHECK-NEXT: successors: %bb.3(0x80000000)
; CHECK-NEXT: liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
; CHECK-NEXT: {{ $}}
@@ -67,19 +43,19 @@ body: |
; CHECK-NEXT: ASYNCMARK
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: bb.2.else:
+ ; CHECK-NEXT: bb.2:
; CHECK-NEXT: successors: %bb.3(0x80000000)
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: $vgpr0 = V_MOV_B32_e32 0, implicit $exec
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: bb.3.join:
+ ; CHECK-NEXT: bb.3:
; CHECK-NEXT: liveins: $vgpr0
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: S_WAITCNT 3952
; CHECK-NEXT: $vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
; CHECK-NEXT: S_ENDPGM 0
- bb.0.entry:
+ bb.0:
successors: %bb.1(0x40000000), %bb.2(0x40000000)
liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
@@ -87,7 +63,7 @@ body: |
S_CBRANCH_SCC1 %bb.2, implicit killed $scc
; then branch — issues async buffer load + ASYNCMARK
- bb.1.then:
+ bb.1:
successors: %bb.3(0x80000000)
liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
@@ -96,7 +72,7 @@ body: |
S_BRANCH %bb.3
; else branch — sync path, no async operations, OtherMarks is empty at join
- bb.2.else:
+ bb.2:
successors: %bb.3(0x80000000)
$vgpr0 = V_MOV_B32_e32 0, implicit $exec
@@ -106,7 +82,7 @@ body: |
; Before fix: assertion. After fix: returns early, preserves async marks.
; S_WAITCNT must be emitted before the use of $vgpr0 to drain the pending load.
; (mergeAsyncMarks sees non-empty AsyncMarks from then, empty from else)
- bb.3.join:
+ bb.3:
liveins: $vgpr0
$vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
@@ -121,7 +97,7 @@ machineFunctionInfo:
occupancy: 8
body: |
; CHECK-LABEL: name: asyncmark_in_else
- ; CHECK: bb.0.entry:
+ ; CHECK: bb.0:
; CHECK-NEXT: successors: %bb.1(0x40000000), %bb.2(0x40000000)
; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
; CHECK-NEXT: {{ $}}
@@ -129,13 +105,13 @@ body: |
; CHECK-NEXT: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
; CHECK-NEXT: S_CBRANCH_SCC1 %bb.2, implicit killed $scc
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: bb.1.then:
+ ; CHECK-NEXT: bb.1:
; CHECK-NEXT: successors: %bb.3(0x80000000)
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: $vgpr0 = V_MOV_B32_e32 0, implicit $exec
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: bb.2.else:
+ ; CHECK-NEXT: bb.2:
; CHECK-NEXT: successors: %bb.3(0x80000000)
; CHECK-NEXT: liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
; CHECK-NEXT: {{ $}}
@@ -143,13 +119,13 @@ body: |
; CHECK-NEXT: ASYNCMARK
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: bb.3.join:
+ ; CHECK-NEXT: bb.3:
; CHECK-NEXT: liveins: $vgpr0
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: S_WAITCNT 3952
; CHECK-NEXT: $vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
; CHECK-NEXT: S_ENDPGM 0
- bb.0.entry:
+ bb.0:
successors: %bb.1(0x40000000), %bb.2(0x40000000)
liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
@@ -157,14 +133,14 @@ body: |
S_CBRANCH_SCC1 %bb.2, implicit killed $scc
; then branch — sync path, no async operations
- bb.1.then:
+ bb.1:
successors: %bb.3(0x80000000)
$vgpr0 = V_MOV_B32_e32 0, implicit $exec
S_BRANCH %bb.3
; else branch — issues async buffer load + ASYNCMARK
- bb.2.else:
+ bb.2:
successors: %bb.3(0x80000000)
liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
@@ -174,7 +150,7 @@ body: |
; join — mergeAsyncMarks sees empty AsyncMarks (then) and non-empty OtherMarks (else).
; S_WAITCNT must be emitted before the use of $vgpr0 to drain the pending load.
- bb.3.join:
+ bb.3:
liveins: $vgpr0
$vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
>From cdf2d67b2402b9dec986de8b3d82dca65ad47031 Mon Sep 17 00:00:00 2001
From: Xiaohu Guo <Xiaohu.Guo at amd.com>
Date: Wed, 29 Apr 2026 06:02:04 -0500
Subject: [PATCH 6/9] =?UTF-8?q?Address=20review=20feedback:=20=20=20-=20Re?=
=?UTF-8?q?placed=20BUFFER=5FLOAD=5FDWORD=5FOFFSET=20with=20BUFFER=5FLOAD?=
=?UTF-8?q?=5FDWORD=5FLDS=5FIDXEN=20+=20IsAsync=3D1=20=E2=80=94=20a=20real?=
=?UTF-8?q?=20async=20LDS=20DMA=20that=20isAsync()=20recognises,=20recordi?=
=?UTF-8?q?ng=20AsyncScore[LOAD=5FCNT]=20before=20ASYNCMARK=20pushes=20it?=
=?UTF-8?q?=20onto=20AsyncMarks=20=20=20-=20Added=20$m0=20=3D=20S=5FMOV=5F?=
=?UTF-8?q?B32=200=20in=20bb.0=20(required=20for=20LDS=20DMA)=20=20=20-=20?=
=?UTF-8?q?Added=20WAIT=5FASYNCMARK=200=20to=20the=20join=20block=20so=20t?=
=?UTF-8?q?he=20async=20mark=20machinery=20is=20actually=20consumed=20=20?=
=?UTF-8?q?=20-=20Changed=20join=20block=20consumer=20to=20DS=5FREAD=5FB32?=
=?UTF-8?q?=5Fgfx9=20(reads=20from=20LDS=20where=20the=20async=20DMA=20wro?=
=?UTF-8?q?te)=20=20=20-=20Regenerated=20CHECK=20lines?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
---
.../AMDGPU/asyncmark-merge-empty-other.mir | 85 ++++++++++---------
1 file changed, 46 insertions(+), 39 deletions(-)
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
index 0dd3738a71048..96c7e08c562d9 100644
--- a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
@@ -7,20 +7,23 @@
# AsyncMarks) and the other does not (empty OtherMarks), before the fix
# seq_inclusive<unsigned>(1, 0) would assert Begin <= End.
#
-# After the fix the function returns early when either side is empty. The
-# pending async marks from the non-empty predecessor are preserved, so any
-# consumer instruction in the join block still receives the correct S_WAITCNT.
+# After the fix the function returns early when either side is empty.
+#
+# BUFFER_LOAD_DWORD_LDS_IDXEN with IsAsync=1 loads from global memory
+# directly into LDS (bypassing VGPRs). isAsync() returns true for this
+# instruction, so the LOAD_CNT score is recorded into AsyncScore before
+# ASYNCMARK pushes it onto AsyncMarks.
+#
+# The join block contains WAIT_ASYNCMARK 0 to consume the pending mark.
+# Before the fix, mergeAsyncMarks() asserted before reaching the wait.
+# After the fix the function returns early and the pass completes without asserting.
#
# Two patterns are tested:
# asyncmark_in_then - ASYNCMARK in the then-successor, else-successor is sync
# asyncmark_in_else - ASYNCMARK in the else-successor, then-successor is sync
-#
-# In both cases the join block must emit S_WAITCNT to drain the pending async
-# buffer load before the result is consumed.
---
# Pattern 1: ASYNCMARK in then-successor, else-successor is sync.
-# The join block must wait for the async load from the then path.
name: asyncmark_in_then
tracksRegLiveness: true
machineFunctionInfo:
@@ -29,63 +32,64 @@ body: |
; CHECK-LABEL: name: asyncmark_in_then
; CHECK: bb.0:
; CHECK-NEXT: successors: %bb.1(0x40000000), %bb.2(0x40000000)
- ; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
+ ; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7, $vgpr0, $vgpr1
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: S_WAITCNT 0
+ ; CHECK-NEXT: $m0 = S_MOV_B32 0
; CHECK-NEXT: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
; CHECK-NEXT: S_CBRANCH_SCC1 %bb.2, implicit killed $scc
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.1:
; CHECK-NEXT: successors: %bb.3(0x80000000)
- ; CHECK-NEXT: liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
+ ; CHECK-NEXT: liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7, $vgpr0
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, implicit $exec
+ ; CHECK-NEXT: BUFFER_LOAD_DWORD_LDS_IDXEN $vgpr0, $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, 1, implicit $exec, implicit $m0 :: (load (s32) from `ptr addrspace(1) poison`, addrspace 1), (store (s32) into `ptr addrspace(3) poison`, addrspace 3)
; CHECK-NEXT: ASYNCMARK
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.2:
; CHECK-NEXT: successors: %bb.3(0x80000000)
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: $vgpr0 = V_MOV_B32_e32 0, implicit $exec
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.3:
- ; CHECK-NEXT: liveins: $vgpr0
+ ; CHECK-NEXT: liveins: $vgpr1
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: S_WAITCNT 3952
- ; CHECK-NEXT: $vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
+ ; CHECK-NEXT: WAIT_ASYNCMARK 0
+ ; CHECK-NEXT: $vgpr2 = DS_READ_B32_gfx9 $vgpr1, 0, 0, implicit $m0, implicit $exec :: (load (s32) from `ptr addrspace(3) poison`, addrspace 3)
; CHECK-NEXT: S_ENDPGM 0
bb.0:
successors: %bb.1(0x40000000), %bb.2(0x40000000)
- liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
+ liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7, $vgpr0, $vgpr1
+ $m0 = S_MOV_B32 0
S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
S_CBRANCH_SCC1 %bb.2, implicit killed $scc
- ; then branch — issues async buffer load + ASYNCMARK
+ ; then branch — issues async LDS DMA + ASYNCMARK
bb.1:
successors: %bb.3(0x80000000)
- liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
+ liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7, $vgpr0
- $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, implicit $exec
+ BUFFER_LOAD_DWORD_LDS_IDXEN $vgpr0, $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, 1, implicit $exec, implicit $m0 :: (load (s32) from `ptr addrspace(1) poison`), (store (s32) into `ptr addrspace(3) poison`)
ASYNCMARK
S_BRANCH %bb.3
- ; else branch — sync path, no async operations, OtherMarks is empty at join
+ ; else branch — sync path, no async operations; OtherMarks is empty at join
bb.2:
successors: %bb.3(0x80000000)
- $vgpr0 = V_MOV_B32_e32 0, implicit $exec
S_BRANCH %bb.3
; join — mergeAsyncMarks sees non-empty AsyncMarks (then) and empty OtherMarks (else).
- ; Before fix: assertion. After fix: returns early, preserves async marks.
- ; S_WAITCNT must be emitted before the use of $vgpr0 to drain the pending load.
- ; (mergeAsyncMarks sees non-empty AsyncMarks from then, empty from else)
+ ; Before fix: assertion. After fix: returns early, preserving async marks from the then-path.
+ ; The vmcnt score at the join is the min of both predecessors (0 from else-path),
+ ; so no S_WAITCNT is inserted — but the assertion no longer fires.
bb.3:
- liveins: $vgpr0
+ liveins: $vgpr1
- $vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
+ WAIT_ASYNCMARK 0
+ $vgpr2 = DS_READ_B32_gfx9 $vgpr1, 0, 0, implicit $m0, implicit $exec :: (load (s32) from `ptr addrspace(3) poison`)
S_ENDPGM 0
...
---
@@ -99,36 +103,38 @@ body: |
; CHECK-LABEL: name: asyncmark_in_else
; CHECK: bb.0:
; CHECK-NEXT: successors: %bb.1(0x40000000), %bb.2(0x40000000)
- ; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
+ ; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7, $vgpr0, $vgpr1
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: S_WAITCNT 0
+ ; CHECK-NEXT: $m0 = S_MOV_B32 0
; CHECK-NEXT: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
; CHECK-NEXT: S_CBRANCH_SCC1 %bb.2, implicit killed $scc
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.1:
; CHECK-NEXT: successors: %bb.3(0x80000000)
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: $vgpr0 = V_MOV_B32_e32 0, implicit $exec
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.2:
; CHECK-NEXT: successors: %bb.3(0x80000000)
- ; CHECK-NEXT: liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
+ ; CHECK-NEXT: liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7, $vgpr0
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, implicit $exec
+ ; CHECK-NEXT: BUFFER_LOAD_DWORD_LDS_IDXEN $vgpr0, $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, 1, implicit $exec, implicit $m0 :: (load (s32) from `ptr addrspace(1) poison`, addrspace 1), (store (s32) into `ptr addrspace(3) poison`, addrspace 3)
; CHECK-NEXT: ASYNCMARK
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.3:
- ; CHECK-NEXT: liveins: $vgpr0
+ ; CHECK-NEXT: liveins: $vgpr1
; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: WAIT_ASYNCMARK 0
; CHECK-NEXT: S_WAITCNT 3952
- ; CHECK-NEXT: $vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
+ ; CHECK-NEXT: $vgpr2 = DS_READ_B32_gfx9 $vgpr1, 0, 0, implicit $m0, implicit $exec :: (load (s32) from `ptr addrspace(3) poison`, addrspace 3)
; CHECK-NEXT: S_ENDPGM 0
bb.0:
successors: %bb.1(0x40000000), %bb.2(0x40000000)
- liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
+ liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7, $vgpr0, $vgpr1
+ $m0 = S_MOV_B32 0
S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
S_CBRANCH_SCC1 %bb.2, implicit killed $scc
@@ -136,23 +142,24 @@ body: |
bb.1:
successors: %bb.3(0x80000000)
- $vgpr0 = V_MOV_B32_e32 0, implicit $exec
S_BRANCH %bb.3
- ; else branch — issues async buffer load + ASYNCMARK
+ ; else branch — issues async LDS DMA + ASYNCMARK
bb.2:
successors: %bb.3(0x80000000)
- liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
+ liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7, $vgpr0
- $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, implicit $exec
+ BUFFER_LOAD_DWORD_LDS_IDXEN $vgpr0, $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, 1, implicit $exec, implicit $m0 :: (load (s32) from `ptr addrspace(1) poison`), (store (s32) into `ptr addrspace(3) poison`)
ASYNCMARK
S_BRANCH %bb.3
; join — mergeAsyncMarks sees empty AsyncMarks (then) and non-empty OtherMarks (else).
- ; S_WAITCNT must be emitted before the use of $vgpr0 to drain the pending load.
+ ; The pending vmcnt from the else-path's async load is preserved at the join;
+ ; S_WAITCNT vmcnt(0) is inserted before the DS_READ to drain it.
bb.3:
- liveins: $vgpr0
+ liveins: $vgpr1
- $vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
+ WAIT_ASYNCMARK 0
+ $vgpr2 = DS_READ_B32_gfx9 $vgpr1, 0, 0, implicit $m0, implicit $exec :: (load (s32) from `ptr addrspace(3) poison`)
S_ENDPGM 0
...
>From 5d9b8e4137549b8b29b0a7e2c00235e413dd0709 Mon Sep 17 00:00:00 2001
From: Xiaohu Guo <Xiaohu.Guo at amd.com>
Date: Thu, 30 Apr 2026 16:04:04 -0500
Subject: [PATCH 7/9] Address review feed back: Update MIR test to use
gfx1250 async LDS DMA
Switch from gfx950 + BUFFER_LOAD_DWORD_LDS_IDXEN to gfx1250 +
GLOBAL_LOAD_ASYNC_TO_LDS_B32 as suggested in review.
On GFX1250 the async load is tracked via ASYNC_CNT rather than VM_CNT,
which properly exercises the ASYNCMARK/WAIT_ASYNCMARK machinery. The
join block carries WAIT_ASYNCMARK 0; in asyncmark_in_else the pass
expands it to S_WAIT_ASYNCCNT 0 to drain the pending async count.
Rebased onto upstream main to pick up GFX1250 ASYNCMARK support
added in #185813.
Regenerate CHECK lines with update_mir_test_checks.py.
---
.../AMDGPU/asyncmark-merge-empty-other.mir | 74 +++++++------------
1 file changed, 28 insertions(+), 46 deletions(-)
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
index 96c7e08c562d9..572dc778c95be 100644
--- a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
@@ -1,22 +1,21 @@
# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 6
-# RUN: llc -mtriple=amdgcn -mcpu=gfx950 -run-pass=si-insert-waitcnts -o - %s | FileCheck %s
+# RUN: llc -mtriple=amdgcn -mcpu=gfx1250 -run-pass=si-insert-waitcnts -o - %s | FileCheck %s
# Regression test for mergeAsyncMarks() asserting when OtherMarks is empty.
#
# At a CFG join point where one predecessor has an ASYNCMARK (non-empty
-# AsyncMarks) and the other does not (empty OtherMarks), before the fix
-# seq_inclusive<unsigned>(1, 0) would assert Begin <= End.
+# AsyncMarks) and the other does not (empty OtherMarks), MergeCount becomes
+# min(0, N) = 0. Before the fix, seq_inclusive<unsigned>(1, 0) asserted
+# Begin <= End. After the fix the function returns early when either side
+# is empty.
#
-# After the fix the function returns early when either side is empty.
-#
-# BUFFER_LOAD_DWORD_LDS_IDXEN with IsAsync=1 loads from global memory
-# directly into LDS (bypassing VGPRs). isAsync() returns true for this
-# instruction, so the LOAD_CNT score is recorded into AsyncScore before
-# ASYNCMARK pushes it onto AsyncMarks.
+# GLOBAL_LOAD_ASYNC_TO_LDS_B32 is a GFX1250 async LDS DMA instruction tracked
+# via ASYNC_CNT. isAsync() returns true for it, so the score is recorded into
+# AsyncScore before ASYNCMARK pushes it onto AsyncMarks.
#
# The join block contains WAIT_ASYNCMARK 0 to consume the pending mark.
# Before the fix, mergeAsyncMarks() asserted before reaching the wait.
-# After the fix the function returns early and the pass completes without asserting.
+# After the fix the pass completes without asserting.
#
# Two patterns are tested:
# asyncmark_in_then - ASYNCMARK in the then-successor, else-successor is sync
@@ -32,18 +31,18 @@ body: |
; CHECK-LABEL: name: asyncmark_in_then
; CHECK: bb.0:
; CHECK-NEXT: successors: %bb.1(0x40000000), %bb.2(0x40000000)
- ; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7, $vgpr0, $vgpr1
+ ; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $vgpr0_vgpr1, $vgpr2
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: S_WAITCNT 0
- ; CHECK-NEXT: $m0 = S_MOV_B32 0
+ ; CHECK-NEXT: S_WAIT_LOADCNT_DSCNT 0
+ ; CHECK-NEXT: S_WAIT_KMCNT 0
; CHECK-NEXT: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
; CHECK-NEXT: S_CBRANCH_SCC1 %bb.2, implicit killed $scc
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.1:
; CHECK-NEXT: successors: %bb.3(0x80000000)
- ; CHECK-NEXT: liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7, $vgpr0
+ ; CHECK-NEXT: liveins: $vgpr0_vgpr1, $vgpr2
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: BUFFER_LOAD_DWORD_LDS_IDXEN $vgpr0, $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, 1, implicit $exec, implicit $m0 :: (load (s32) from `ptr addrspace(1) poison`, addrspace 1), (store (s32) into `ptr addrspace(3) poison`, addrspace 3)
+ ; CHECK-NEXT: GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32) from `ptr addrspace(1) poison`, addrspace 1), (store (s32) into `ptr addrspace(3) poison`, addrspace 3)
; CHECK-NEXT: ASYNCMARK
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
@@ -53,25 +52,21 @@ body: |
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.3:
- ; CHECK-NEXT: liveins: $vgpr1
- ; CHECK-NEXT: {{ $}}
; CHECK-NEXT: WAIT_ASYNCMARK 0
- ; CHECK-NEXT: $vgpr2 = DS_READ_B32_gfx9 $vgpr1, 0, 0, implicit $m0, implicit $exec :: (load (s32) from `ptr addrspace(3) poison`, addrspace 3)
; CHECK-NEXT: S_ENDPGM 0
bb.0:
successors: %bb.1(0x40000000), %bb.2(0x40000000)
- liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7, $vgpr0, $vgpr1
+ liveins: $sgpr0, $sgpr1, $vgpr0_vgpr1, $vgpr2
- $m0 = S_MOV_B32 0
S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
S_CBRANCH_SCC1 %bb.2, implicit killed $scc
; then branch — issues async LDS DMA + ASYNCMARK
bb.1:
successors: %bb.3(0x80000000)
- liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7, $vgpr0
+ liveins: $vgpr0_vgpr1, $vgpr2
- BUFFER_LOAD_DWORD_LDS_IDXEN $vgpr0, $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, 1, implicit $exec, implicit $m0 :: (load (s32) from `ptr addrspace(1) poison`), (store (s32) into `ptr addrspace(3) poison`)
+ GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32) from `ptr addrspace(1) poison`), (store (s32) into `ptr addrspace(3) poison`)
ASYNCMARK
S_BRANCH %bb.3
@@ -82,14 +77,9 @@ body: |
S_BRANCH %bb.3
; join — mergeAsyncMarks sees non-empty AsyncMarks (then) and empty OtherMarks (else).
- ; Before fix: assertion. After fix: returns early, preserving async marks from the then-path.
- ; The vmcnt score at the join is the min of both predecessors (0 from else-path),
- ; so no S_WAITCNT is inserted — but the assertion no longer fires.
+ ; Before fix: assertion. After fix: returns early, no spurious wait inserted.
bb.3:
- liveins: $vgpr1
-
WAIT_ASYNCMARK 0
- $vgpr2 = DS_READ_B32_gfx9 $vgpr1, 0, 0, implicit $m0, implicit $exec :: (load (s32) from `ptr addrspace(3) poison`)
S_ENDPGM 0
...
---
@@ -103,10 +93,10 @@ body: |
; CHECK-LABEL: name: asyncmark_in_else
; CHECK: bb.0:
; CHECK-NEXT: successors: %bb.1(0x40000000), %bb.2(0x40000000)
- ; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7, $vgpr0, $vgpr1
+ ; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $vgpr0_vgpr1, $vgpr2
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: S_WAITCNT 0
- ; CHECK-NEXT: $m0 = S_MOV_B32 0
+ ; CHECK-NEXT: S_WAIT_LOADCNT_DSCNT 0
+ ; CHECK-NEXT: S_WAIT_KMCNT 0
; CHECK-NEXT: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
; CHECK-NEXT: S_CBRANCH_SCC1 %bb.2, implicit killed $scc
; CHECK-NEXT: {{ $}}
@@ -117,24 +107,20 @@ body: |
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.2:
; CHECK-NEXT: successors: %bb.3(0x80000000)
- ; CHECK-NEXT: liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7, $vgpr0
+ ; CHECK-NEXT: liveins: $vgpr0_vgpr1, $vgpr2
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: BUFFER_LOAD_DWORD_LDS_IDXEN $vgpr0, $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, 1, implicit $exec, implicit $m0 :: (load (s32) from `ptr addrspace(1) poison`, addrspace 1), (store (s32) into `ptr addrspace(3) poison`, addrspace 3)
+ ; CHECK-NEXT: GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32) from `ptr addrspace(1) poison`, addrspace 1), (store (s32) into `ptr addrspace(3) poison`, addrspace 3)
; CHECK-NEXT: ASYNCMARK
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.3:
- ; CHECK-NEXT: liveins: $vgpr1
- ; CHECK-NEXT: {{ $}}
; CHECK-NEXT: WAIT_ASYNCMARK 0
- ; CHECK-NEXT: S_WAITCNT 3952
- ; CHECK-NEXT: $vgpr2 = DS_READ_B32_gfx9 $vgpr1, 0, 0, implicit $m0, implicit $exec :: (load (s32) from `ptr addrspace(3) poison`, addrspace 3)
+ ; CHECK-NEXT: S_WAIT_ASYNCCNT 0, implicit-def $asynccnt, implicit $asynccnt
; CHECK-NEXT: S_ENDPGM 0
bb.0:
successors: %bb.1(0x40000000), %bb.2(0x40000000)
- liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7, $vgpr0, $vgpr1
+ liveins: $sgpr0, $sgpr1, $vgpr0_vgpr1, $vgpr2
- $m0 = S_MOV_B32 0
S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
S_CBRANCH_SCC1 %bb.2, implicit killed $scc
@@ -147,19 +133,15 @@ body: |
; else branch — issues async LDS DMA + ASYNCMARK
bb.2:
successors: %bb.3(0x80000000)
- liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7, $vgpr0
+ liveins: $vgpr0_vgpr1, $vgpr2
- BUFFER_LOAD_DWORD_LDS_IDXEN $vgpr0, $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, 1, implicit $exec, implicit $m0 :: (load (s32) from `ptr addrspace(1) poison`), (store (s32) into `ptr addrspace(3) poison`)
+ GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32) from `ptr addrspace(1) poison`), (store (s32) into `ptr addrspace(3) poison`)
ASYNCMARK
S_BRANCH %bb.3
; join — mergeAsyncMarks sees empty AsyncMarks (then) and non-empty OtherMarks (else).
- ; The pending vmcnt from the else-path's async load is preserved at the join;
- ; S_WAITCNT vmcnt(0) is inserted before the DS_READ to drain it.
+ ; Before fix: assertion. After fix: returns early, no spurious wait inserted.
bb.3:
- liveins: $vgpr1
-
WAIT_ASYNCMARK 0
- $vgpr2 = DS_READ_B32_gfx9 $vgpr1, 0, 0, implicit $m0, implicit $exec :: (load (s32) from `ptr addrspace(3) poison`)
S_ENDPGM 0
...
>From f7f5c977b16f581645af8a7eacf9966cebd5de4d Mon Sep 17 00:00:00 2001
From: Xiaohu Guo <Xiaohu.Guo at amd.com>
Date: Fri, 1 May 2026 06:54:18 -0500
Subject: [PATCH 8/9] restore && and add MergeCount == 0 guard after padding
To consolidate the one-side-empty guard at the top of the function,
the early-exit was changed from && to ||. However, after rebasing onto
upstream/main, the || condition broke asyncmark-max-pregfx12.ll: the
padding logic now sits between the early-exit and the seq_inclusive loop,
so returning early when AsyncMarks.empty() && OtherMarks non-empty skips
the padding and silently drops OtherMarks.
The correct fix keeps && at the top (handles the both-empty case, leaves
padding intact for the AsyncMarks-empty case) and adds:
if (MergeCount == 0)
return StrictDom;
after the padding to handle the OtherMarks-empty case without touching
the padding logic or triggering seq_inclusive<unsigned>(1, 0).
Update asyncmark-merge-empty-other.mir: with OtherMarks no longer
dropped, asyncmark_in_else correctly emits S_WAIT_ASYNCCNT 0 at the
join block.
---
llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp | 12 +++++++-----
.../CodeGen/AMDGPU/asyncmark-merge-empty-other.mir | 1 +
2 files changed, 8 insertions(+), 5 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
index c3267bfc13a40..a466eb632d5c5 100644
--- a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
+++ b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
@@ -3025,11 +3025,8 @@ bool WaitcntBrackets::mergeAsyncMarks(ArrayRef<MergeInfo> MergeInfos,
bool StrictDom = false;
LLVM_DEBUG(dbgs() << "Merging async marks ...");
- // Early exit: nothing to merge when either side is empty.
- // If OtherMarks is empty the join point has no async marks from that
- // predecessor, so our existing marks are already the conservative result.
- // If both are empty there is simply nothing to do.
- if (AsyncMarks.empty() || OtherMarks.empty()) {
+ // Early exit: nothing to merge when both sides are empty.
+ if (AsyncMarks.empty() && OtherMarks.empty()) {
LLVM_DEBUG(dbgs() << " nothing to merge\n");
return false;
}
@@ -3070,6 +3067,11 @@ bool WaitcntBrackets::mergeAsyncMarks(ArrayRef<MergeInfo> MergeInfos,
unsigned OtherSize = OtherMarks.size();
unsigned OurSize = AsyncMarks.size();
unsigned MergeCount = std::min(OtherSize, OurSize);
+ // OtherMarks is empty → OtherSize == 0 → MergeCount == 0.
+ // Our existing marks are the conservative result; return early to avoid
+ // passing MergeCount == 0 to seq_inclusive which asserts Begin <= End.
+ if (MergeCount == 0)
+ return StrictDom;
for (auto Idx : seq_inclusive<unsigned>(1, MergeCount)) {
for (auto T : inst_counter_types(Context->MaxCounter)) {
StrictDom |= mergeScore(MergeInfos[T], AsyncMarks[OurSize - Idx][T],
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
index 572dc778c95be..1639aa3870782 100644
--- a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
@@ -53,6 +53,7 @@ body: |
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.3:
; CHECK-NEXT: WAIT_ASYNCMARK 0
+ ; CHECK-NEXT: S_WAIT_ASYNCCNT 0, implicit-def $asynccnt, implicit $asynccnt
; CHECK-NEXT: S_ENDPGM 0
bb.0:
successors: %bb.1(0x40000000), %bb.2(0x40000000)
>From 1b4097f37cf6ed73439a13f3c589b5ba3e66c122 Mon Sep 17 00:00:00 2001
From: Xiaohu Guo <Xiaohu.Guo at amd.com>
Date: Fri, 1 May 2026 08:38:45 -0500
Subject: [PATCH 9/9] Address review: remove input successors and simplify
memoperands
---
.../AMDGPU/asyncmark-merge-empty-other.mir | 23 ++++++-------------
1 file changed, 7 insertions(+), 16 deletions(-)
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
index 1639aa3870782..c271de4b456bb 100644
--- a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
@@ -30,7 +30,7 @@ machineFunctionInfo:
body: |
; CHECK-LABEL: name: asyncmark_in_then
; CHECK: bb.0:
- ; CHECK-NEXT: successors: %bb.1(0x40000000), %bb.2(0x40000000)
+ ; CHECK-NEXT: successors: %bb.2(0x40000000), %bb.1(0x40000000)
; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $vgpr0_vgpr1, $vgpr2
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: S_WAIT_LOADCNT_DSCNT 0
@@ -42,7 +42,7 @@ body: |
; CHECK-NEXT: successors: %bb.3(0x80000000)
; CHECK-NEXT: liveins: $vgpr0_vgpr1, $vgpr2
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32) from `ptr addrspace(1) poison`, addrspace 1), (store (s32) into `ptr addrspace(3) poison`, addrspace 3)
+ ; CHECK-NEXT: GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
; CHECK-NEXT: ASYNCMARK
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
@@ -56,7 +56,6 @@ body: |
; CHECK-NEXT: S_WAIT_ASYNCCNT 0, implicit-def $asynccnt, implicit $asynccnt
; CHECK-NEXT: S_ENDPGM 0
bb.0:
- successors: %bb.1(0x40000000), %bb.2(0x40000000)
liveins: $sgpr0, $sgpr1, $vgpr0_vgpr1, $vgpr2
S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
@@ -64,17 +63,14 @@ body: |
; then branch — issues async LDS DMA + ASYNCMARK
bb.1:
- successors: %bb.3(0x80000000)
liveins: $vgpr0_vgpr1, $vgpr2
- GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32) from `ptr addrspace(1) poison`), (store (s32) into `ptr addrspace(3) poison`)
+ GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
ASYNCMARK
S_BRANCH %bb.3
; else branch — sync path, no async operations; OtherMarks is empty at join
bb.2:
- successors: %bb.3(0x80000000)
-
S_BRANCH %bb.3
; join — mergeAsyncMarks sees non-empty AsyncMarks (then) and empty OtherMarks (else).
@@ -93,7 +89,7 @@ machineFunctionInfo:
body: |
; CHECK-LABEL: name: asyncmark_in_else
; CHECK: bb.0:
- ; CHECK-NEXT: successors: %bb.1(0x40000000), %bb.2(0x40000000)
+ ; CHECK-NEXT: successors: %bb.2(0x40000000), %bb.1(0x40000000)
; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $vgpr0_vgpr1, $vgpr2
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: S_WAIT_LOADCNT_DSCNT 0
@@ -110,7 +106,7 @@ body: |
; CHECK-NEXT: successors: %bb.3(0x80000000)
; CHECK-NEXT: liveins: $vgpr0_vgpr1, $vgpr2
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32) from `ptr addrspace(1) poison`, addrspace 1), (store (s32) into `ptr addrspace(3) poison`, addrspace 3)
+ ; CHECK-NEXT: GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
; CHECK-NEXT: ASYNCMARK
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
@@ -119,7 +115,6 @@ body: |
; CHECK-NEXT: S_WAIT_ASYNCCNT 0, implicit-def $asynccnt, implicit $asynccnt
; CHECK-NEXT: S_ENDPGM 0
bb.0:
- successors: %bb.1(0x40000000), %bb.2(0x40000000)
liveins: $sgpr0, $sgpr1, $vgpr0_vgpr1, $vgpr2
S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
@@ -127,21 +122,17 @@ body: |
; then branch — sync path, no async operations
bb.1:
- successors: %bb.3(0x80000000)
-
S_BRANCH %bb.3
; else branch — issues async LDS DMA + ASYNCMARK
bb.2:
- successors: %bb.3(0x80000000)
liveins: $vgpr0_vgpr1, $vgpr2
- GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32) from `ptr addrspace(1) poison`), (store (s32) into `ptr addrspace(3) poison`)
+ GLOBAL_LOAD_ASYNC_TO_LDS_B32 $vgpr2, $vgpr0_vgpr1, 0, 0, implicit-def $asynccnt, implicit $exec, implicit $asynccnt :: (load (s32), addrspace 1), (store (s32), addrspace 3)
ASYNCMARK
S_BRANCH %bb.3
- ; join — mergeAsyncMarks sees empty AsyncMarks (then) and non-empty OtherMarks (else).
- ; Before fix: assertion. After fix: returns early, no spurious wait inserted.
+ ; join block
bb.3:
WAIT_ASYNCMARK 0
S_ENDPGM 0
More information about the llvm-commits
mailing list