[llvm] [AMDGPU][SIInsertWaitcnts] Fix iota_range assertion when OtherMarks is empty in mergeAsyncMarks() (PR #193499)
via llvm-commits
llvm-commits at lists.llvm.org
Mon Apr 27 07:43:18 PDT 2026
https://github.com/xiaohuguo2023 updated https://github.com/llvm/llvm-project/pull/193499
>From 13de1196fb266c8c2ed15cd582376753054eaba8 Mon Sep 17 00:00:00 2001
From: Xiaohu Guo <Xiaohu.Guo at amd.com>
Date: Mon, 20 Apr 2026 21:25:09 +0000
Subject: [PATCH 1/5] [AMDGPU][SIInsertWaitcnts] Fix iota_range assert when
MergeCount == 0
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
In mergeAsyncMarks(), when one branch of a conditional (e.g. EVEN_K=False
K-loop boundary) has no async memory ops, OtherMarks can be empty while
AsyncMarks is not. The existing early-exit only handles both-empty, so
MergeCount = min(0, N) = 0 reaches seq_inclusive<unsigned>(1, 0) which
asserts Begin <= End.
Fix: return early when MergeCount == 0 — nothing to merge.
Reproducer: compile a StreamK GEMM kernel on gfx950 (MI350X) with
BLOCK_SIZE_K=256, K=2880 (not divisible → EVEN_K=False),
num_stages=2, waves_per_eu=4, matrix_instr_nonkdim=32.
The assert fires deterministically even when compiled alone (not a race).
Fixes: iota_range<unsigned>(1, 0) → Assertion Begin <= End failed
---
llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp | 2 +
.../AMDGPU/asyncmark-merge-empty-other.mir | 75 +++++++++++++++++++
2 files changed, 77 insertions(+)
create mode 100644 llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
diff --git a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
index abd03d5f9bf73..120faaa82a315 100644
--- a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
+++ b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
@@ -3024,6 +3024,8 @@ bool WaitcntBrackets::mergeAsyncMarks(ArrayRef<MergeInfo> MergeInfos,
unsigned OtherSize = OtherMarks.size();
unsigned OurSize = AsyncMarks.size();
unsigned MergeCount = std::min(OtherSize, OurSize);
+ if (MergeCount == 0)
+ return StrictDom;
for (auto Idx : seq_inclusive<unsigned>(1, MergeCount)) {
for (auto T : inst_counter_types(Context->MaxCounter)) {
StrictDom |= mergeScore(MergeInfos[T], AsyncMarks[OurSize - Idx][T],
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
new file mode 100644
index 0000000000000..ec9c33f33a1f8
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
@@ -0,0 +1,75 @@
+# RUN: llc -mtriple=amdgcn -mcpu=gfx950 -verify-machineinstrs -run-pass=si-insert-waitcnts -o - %s | FileCheck %s
+
+# Regression test for mergeAsyncMarks() asserting when OtherMarks is empty.
+#
+# At a CFG join point where one predecessor has an ASYNCMARK (non-empty
+# AsyncMarks) and the other does not (empty OtherMarks), MergeCount becomes
+# min(0, N) = 0. Before the fix, seq_inclusive<unsigned>(1, 0) asserted
+# Begin <= End. After the fix the function returns early: the other predecessor
+# contributed no async marks so our marks are unchanged and no stricter waits
+# are needed.
+#
+# Key check: no S_WAITCNT is inserted at the join block (bb.3). The else
+# branch had no async operations so merging its empty marks into the then
+# branch's non-empty marks requires no additional waits.
+
+--- |
+ define void @asyncmark_merge_empty_other(i32 %cond) {
+ entry:
+ br i1 true, label %then, label %else
+ then:
+ br label %join
+ else:
+ br label %join
+ join:
+ ret void
+ }
+...
+
+---
+name: asyncmark_merge_empty_other
+tracksRegLiveness: true
+machineFunctionInfo:
+ occupancy: 8
+body: |
+ ; entry — conditional branch to then (async) or else (no async)
+ bb.0.entry:
+ successors: %bb.1(0x40000000), %bb.2(0x40000000)
+ liveins: $sgpr0, $sgpr1
+
+ ; CHECK-LABEL: name: asyncmark_merge_empty_other
+ ; CHECK: bb.0.entry:
+ ; CHECK: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
+ ; CHECK-NEXT: S_CBRANCH_SCC1 %bb.2, implicit killed $scc
+ S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
+ S_CBRANCH_SCC1 %bb.2, implicit killed $scc
+
+ ; then branch — ASYNCMARK records non-empty AsyncMarks
+ bb.1.then:
+ successors: %bb.3(0x80000000)
+
+ ; CHECK: bb.1.then:
+ ; CHECK: ASYNCMARK
+ ; CHECK-NEXT: S_BRANCH %bb.3
+ ASYNCMARK
+ S_BRANCH %bb.3
+
+ ; else branch — no async operations, OtherMarks is empty at join
+ bb.2.else:
+ successors: %bb.3(0x80000000)
+
+ ; CHECK: bb.2.else:
+ ; CHECK: S_BRANCH %bb.3
+ S_BRANCH %bb.3
+
+ ; join — mergeAsyncMarks() sees non-empty AsyncMarks (bb.1) and empty
+ ; OtherMarks (bb.2). Before the fix: assertion. After the fix: early
+ ; return, no waitcnt inserted.
+ bb.3.join:
+ ; CHECK: bb.3.join:
+ ; CHECK-NOT: S_WAITCNT
+ ; CHECK: S_ENDPGM 0
+ ; Verify no waitcnt is inserted before S_ENDPGM — the else predecessor
+ ; had no async marks so no stricter waits are needed at the join.
+ S_ENDPGM 0
+...
>From 0ea21daca30e02df2e35b6b00f94d5978205ff8e Mon Sep 17 00:00:00 2001
From: Xiaohu Guo <Xiaohu.Guo at amd.com>
Date: Wed, 22 Apr 2026 09:37:01 -0500
Subject: [PATCH 2/5] Address review feedback: - Regenerate CHECK lines using
update_mir_test_checks.py - Remove -verify-machineinstrs from RUN line
---
.../AMDGPU/asyncmark-merge-empty-other.mir | 37 ++++++++++++-------
1 file changed, 24 insertions(+), 13 deletions(-)
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
index ec9c33f33a1f8..93c9ae133ca77 100644
--- a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
@@ -1,4 +1,5 @@
-# RUN: llc -mtriple=amdgcn -mcpu=gfx950 -verify-machineinstrs -run-pass=si-insert-waitcnts -o - %s | FileCheck %s
+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 6
+# RUN: llc -mtriple=amdgcn -mcpu=gfx950 -run-pass=si-insert-waitcnts -o - %s | FileCheck %s
# Regression test for mergeAsyncMarks() asserting when OtherMarks is empty.
#
@@ -32,15 +33,33 @@ tracksRegLiveness: true
machineFunctionInfo:
occupancy: 8
body: |
+ ; CHECK-LABEL: name: asyncmark_merge_empty_other
+ ; CHECK: bb.0.entry:
+ ; CHECK-NEXT: successors: %bb.1(0x40000000), %bb.2(0x40000000)
+ ; CHECK-NEXT: liveins: $sgpr0, $sgpr1
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: S_WAITCNT 0
+ ; CHECK-NEXT: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
+ ; CHECK-NEXT: S_CBRANCH_SCC1 %bb.2, implicit killed $scc
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.1.then:
+ ; CHECK-NEXT: successors: %bb.3(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: ASYNCMARK
+ ; CHECK-NEXT: S_BRANCH %bb.3
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.2.else:
+ ; CHECK-NEXT: successors: %bb.3(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: S_BRANCH %bb.3
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.3.join:
+ ; CHECK-NEXT: S_ENDPGM 0
; entry — conditional branch to then (async) or else (no async)
bb.0.entry:
successors: %bb.1(0x40000000), %bb.2(0x40000000)
liveins: $sgpr0, $sgpr1
- ; CHECK-LABEL: name: asyncmark_merge_empty_other
- ; CHECK: bb.0.entry:
- ; CHECK: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
- ; CHECK-NEXT: S_CBRANCH_SCC1 %bb.2, implicit killed $scc
S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
S_CBRANCH_SCC1 %bb.2, implicit killed $scc
@@ -48,9 +67,6 @@ body: |
bb.1.then:
successors: %bb.3(0x80000000)
- ; CHECK: bb.1.then:
- ; CHECK: ASYNCMARK
- ; CHECK-NEXT: S_BRANCH %bb.3
ASYNCMARK
S_BRANCH %bb.3
@@ -58,17 +74,12 @@ body: |
bb.2.else:
successors: %bb.3(0x80000000)
- ; CHECK: bb.2.else:
- ; CHECK: S_BRANCH %bb.3
S_BRANCH %bb.3
; join — mergeAsyncMarks() sees non-empty AsyncMarks (bb.1) and empty
; OtherMarks (bb.2). Before the fix: assertion. After the fix: early
; return, no waitcnt inserted.
bb.3.join:
- ; CHECK: bb.3.join:
- ; CHECK-NOT: S_WAITCNT
- ; CHECK: S_ENDPGM 0
; Verify no waitcnt is inserted before S_ENDPGM — the else predecessor
; had no async marks so no stricter waits are needed at the join.
S_ENDPGM 0
>From 8e1fb2e1a17b596ca893e901e80214ebb11f7ab8 Mon Sep 17 00:00:00 2001
From: Xiaohu Guo <Xiaohu.Guo at amd.com>
Date: Fri, 24 Apr 2026 12:38:23 -0500
Subject: [PATCH 3/5] Address reviw feedback:
Change `&&` to `||` in the early-exit check so the function returns
immediately when either AsyncMarks or OtherMarks is empty, rather than
falling through to the resize/pad logic and hitting a seq_inclusive
assert (Begin > End) when MergeCount == 0. Removes the now-unreachable
mid-function MergeCount == 0 guard.
---
llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
index 120faaa82a315..dab440e22b7ae 100644
--- a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
+++ b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
@@ -2982,8 +2982,11 @@ bool WaitcntBrackets::mergeAsyncMarks(ArrayRef<MergeInfo> MergeInfos,
bool StrictDom = false;
LLVM_DEBUG(dbgs() << "Merging async marks ...");
- // Early exit: both empty
- if (AsyncMarks.empty() && OtherMarks.empty()) {
+ // Early exit: nothing to merge when either side is empty.
+ // If OtherMarks is empty the join point has no async marks from that
+ // predecessor, so our existing marks are already the conservative result.
+ // If both are empty there is simply nothing to do.
+ if (AsyncMarks.empty() || OtherMarks.empty()) {
LLVM_DEBUG(dbgs() << " nothing to merge\n");
return false;
}
@@ -3024,8 +3027,6 @@ bool WaitcntBrackets::mergeAsyncMarks(ArrayRef<MergeInfo> MergeInfos,
unsigned OtherSize = OtherMarks.size();
unsigned OurSize = AsyncMarks.size();
unsigned MergeCount = std::min(OtherSize, OurSize);
- if (MergeCount == 0)
- return StrictDom;
for (auto Idx : seq_inclusive<unsigned>(1, MergeCount)) {
for (auto T : inst_counter_types(Context->MaxCounter)) {
StrictDom |= mergeScore(MergeInfos[T], AsyncMarks[OurSize - Idx][T],
>From 00af50579662423a0ef5dcb74f1867c6d6ee44f6 Mon Sep 17 00:00:00 2001
From: Xiaohu Guo <Xiaohu.Guo at amd.com>
Date: Mon, 27 Apr 2026 05:13:12 -0500
Subject: [PATCH 4/5] add MIR regression test for mergeAsyncMarks with
one-side-empty marks
Test two CFG patterns where one predecessor carries an ASYNCMARK from a
BUFFER_LOAD and the other is a sync path (empty OtherMarks):
asyncmark_in_then: ASYNCMARK in then-successor, else is sync
asyncmark_in_else: ASYNCMARK in else-successor, then is sync
Before the fix, mergeAsyncMarks() asserted Begin <= End via
seq_inclusive<unsigned>(1, 0). After the fix the function returns
early when either side is empty and preserves the pending async marks,
so S_WAITCNT is correctly emitted before the consumer in the join block
---
.../AMDGPU/asyncmark-merge-empty-other.mir | 138 +++++++++++++++---
1 file changed, 117 insertions(+), 21 deletions(-)
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
index 93c9ae133ca77..2119c28bda70c 100644
--- a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
@@ -4,18 +4,33 @@
# Regression test for mergeAsyncMarks() asserting when OtherMarks is empty.
#
# At a CFG join point where one predecessor has an ASYNCMARK (non-empty
-# AsyncMarks) and the other does not (empty OtherMarks), MergeCount becomes
-# min(0, N) = 0. Before the fix, seq_inclusive<unsigned>(1, 0) asserted
-# Begin <= End. After the fix the function returns early: the other predecessor
-# contributed no async marks so our marks are unchanged and no stricter waits
-# are needed.
+# AsyncMarks) and the other does not (empty OtherMarks), before the fix
+# seq_inclusive<unsigned>(1, 0) would assert Begin <= End.
#
-# Key check: no S_WAITCNT is inserted at the join block (bb.3). The else
-# branch had no async operations so merging its empty marks into the then
-# branch's non-empty marks requires no additional waits.
+# After the fix the function returns early when either side is empty. The
+# pending async marks from the non-empty predecessor are preserved, so any
+# consumer instruction in the join block still receives the correct S_WAITCNT.
+#
+# Two patterns are tested:
+# asyncmark_in_then - ASYNCMARK in the then-successor, else-successor is sync
+# asyncmark_in_else - ASYNCMARK in the else-successor, then-successor is sync
+#
+# In both cases the join block must emit S_WAITCNT to drain the pending async
+# buffer load before the result is consumed.
--- |
- define void @asyncmark_merge_empty_other(i32 %cond) {
+ define void @asyncmark_in_then(i32 %cond) {
+ entry:
+ br i1 true, label %then, label %else
+ then:
+ br label %join
+ else:
+ br label %join
+ join:
+ ret void
+ }
+
+ define void @asyncmark_in_else(i32 %cond) {
entry:
br i1 true, label %then, label %else
then:
@@ -28,15 +43,17 @@
...
---
-name: asyncmark_merge_empty_other
+# Pattern 1: ASYNCMARK in then-successor, else-successor is sync.
+# The join block must wait for the async load from the then path.
+name: asyncmark_in_then
tracksRegLiveness: true
machineFunctionInfo:
occupancy: 8
body: |
- ; CHECK-LABEL: name: asyncmark_merge_empty_other
+ ; CHECK-LABEL: name: asyncmark_in_then
; CHECK: bb.0.entry:
; CHECK-NEXT: successors: %bb.1(0x40000000), %bb.2(0x40000000)
- ; CHECK-NEXT: liveins: $sgpr0, $sgpr1
+ ; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: S_WAITCNT 0
; CHECK-NEXT: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
@@ -44,43 +61,122 @@ body: |
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.1.then:
; CHECK-NEXT: successors: %bb.3(0x80000000)
+ ; CHECK-NEXT: liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, implicit $exec
; CHECK-NEXT: ASYNCMARK
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.2.else:
; CHECK-NEXT: successors: %bb.3(0x80000000)
; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: $vgpr0 = V_MOV_B32_e32 0, implicit $exec
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: bb.3.join:
+ ; CHECK-NEXT: liveins: $vgpr0
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: S_WAITCNT 3952
+ ; CHECK-NEXT: $vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
; CHECK-NEXT: S_ENDPGM 0
- ; entry — conditional branch to then (async) or else (no async)
bb.0.entry:
successors: %bb.1(0x40000000), %bb.2(0x40000000)
- liveins: $sgpr0, $sgpr1
+ liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
S_CBRANCH_SCC1 %bb.2, implicit killed $scc
- ; then branch — ASYNCMARK records non-empty AsyncMarks
+ ; then branch — issues async buffer load + ASYNCMARK
bb.1.then:
successors: %bb.3(0x80000000)
+ liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
+ $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, implicit $exec
ASYNCMARK
S_BRANCH %bb.3
- ; else branch — no async operations, OtherMarks is empty at join
+ ; else branch — sync path, no async operations, OtherMarks is empty at join
bb.2.else:
successors: %bb.3(0x80000000)
+ $vgpr0 = V_MOV_B32_e32 0, implicit $exec
S_BRANCH %bb.3
- ; join — mergeAsyncMarks() sees non-empty AsyncMarks (bb.1) and empty
- ; OtherMarks (bb.2). Before the fix: assertion. After the fix: early
- ; return, no waitcnt inserted.
+ ; join — mergeAsyncMarks sees non-empty AsyncMarks (then) and empty OtherMarks (else).
+ ; Before fix: assertion. After fix: returns early, preserves async marks.
+ ; S_WAITCNT must be emitted before the use of $vgpr0 to drain the pending load.
+ ; (mergeAsyncMarks sees non-empty AsyncMarks from then, empty from else)
bb.3.join:
- ; Verify no waitcnt is inserted before S_ENDPGM — the else predecessor
- ; had no async marks so no stricter waits are needed at the join.
+ liveins: $vgpr0
+
+ $vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
+ S_ENDPGM 0
+...
+---
+# Pattern 2: ASYNCMARK in else-successor, then-successor is sync.
+# Mirror of asyncmark_in_then — exercises the opposite predecessor ordering.
+name: asyncmark_in_else
+tracksRegLiveness: true
+machineFunctionInfo:
+ occupancy: 8
+body: |
+ ; CHECK-LABEL: name: asyncmark_in_else
+ ; CHECK: bb.0.entry:
+ ; CHECK-NEXT: successors: %bb.1(0x40000000), %bb.2(0x40000000)
+ ; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: S_WAITCNT 0
+ ; CHECK-NEXT: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
+ ; CHECK-NEXT: S_CBRANCH_SCC1 %bb.2, implicit killed $scc
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.1.then:
+ ; CHECK-NEXT: successors: %bb.3(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: $vgpr0 = V_MOV_B32_e32 0, implicit $exec
+ ; CHECK-NEXT: S_BRANCH %bb.3
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.2.else:
+ ; CHECK-NEXT: successors: %bb.3(0x80000000)
+ ; CHECK-NEXT: liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, implicit $exec
+ ; CHECK-NEXT: ASYNCMARK
+ ; CHECK-NEXT: S_BRANCH %bb.3
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: bb.3.join:
+ ; CHECK-NEXT: liveins: $vgpr0
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: S_WAITCNT 3952
+ ; CHECK-NEXT: $vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
+ ; CHECK-NEXT: S_ENDPGM 0
+ bb.0.entry:
+ successors: %bb.1(0x40000000), %bb.2(0x40000000)
+ liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
+
+ S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
+ S_CBRANCH_SCC1 %bb.2, implicit killed $scc
+
+ ; then branch — sync path, no async operations
+ bb.1.then:
+ successors: %bb.3(0x80000000)
+
+ $vgpr0 = V_MOV_B32_e32 0, implicit $exec
+ S_BRANCH %bb.3
+
+ ; else branch — issues async buffer load + ASYNCMARK
+ bb.2.else:
+ successors: %bb.3(0x80000000)
+ liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
+
+ $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr4_sgpr5_sgpr6_sgpr7, $sgpr0, 0, 0, 0, implicit $exec
+ ASYNCMARK
+ S_BRANCH %bb.3
+
+ ; join — mergeAsyncMarks sees empty AsyncMarks (then) and non-empty OtherMarks (else).
+ ; S_WAITCNT must be emitted before the use of $vgpr0 to drain the pending load.
+ bb.3.join:
+ liveins: $vgpr0
+
+ $vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
S_ENDPGM 0
...
>From a8a4114681e1c71ca5d2bed6ea787bb4aa0ea629 Mon Sep 17 00:00:00 2001
From: Xiaohu Guo <Xiaohu.Guo at amd.com>
Date: Mon, 27 Apr 2026 09:41:55 -0500
Subject: [PATCH 5/5] remove the IR section and plain BB name suffixes
(.entry/.then/.else/.join) as suggested in review
---
.../AMDGPU/asyncmark-merge-empty-other.mir | 56 ++++++-------------
1 file changed, 16 insertions(+), 40 deletions(-)
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
index 2119c28bda70c..0dd3738a71048 100644
--- a/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-merge-empty-other.mir
@@ -18,30 +18,6 @@
# In both cases the join block must emit S_WAITCNT to drain the pending async
# buffer load before the result is consumed.
---- |
- define void @asyncmark_in_then(i32 %cond) {
- entry:
- br i1 true, label %then, label %else
- then:
- br label %join
- else:
- br label %join
- join:
- ret void
- }
-
- define void @asyncmark_in_else(i32 %cond) {
- entry:
- br i1 true, label %then, label %else
- then:
- br label %join
- else:
- br label %join
- join:
- ret void
- }
-...
-
---
# Pattern 1: ASYNCMARK in then-successor, else-successor is sync.
# The join block must wait for the async load from the then path.
@@ -51,7 +27,7 @@ machineFunctionInfo:
occupancy: 8
body: |
; CHECK-LABEL: name: asyncmark_in_then
- ; CHECK: bb.0.entry:
+ ; CHECK: bb.0:
; CHECK-NEXT: successors: %bb.1(0x40000000), %bb.2(0x40000000)
; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
; CHECK-NEXT: {{ $}}
@@ -59,7 +35,7 @@ body: |
; CHECK-NEXT: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
; CHECK-NEXT: S_CBRANCH_SCC1 %bb.2, implicit killed $scc
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: bb.1.then:
+ ; CHECK-NEXT: bb.1:
; CHECK-NEXT: successors: %bb.3(0x80000000)
; CHECK-NEXT: liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
; CHECK-NEXT: {{ $}}
@@ -67,19 +43,19 @@ body: |
; CHECK-NEXT: ASYNCMARK
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: bb.2.else:
+ ; CHECK-NEXT: bb.2:
; CHECK-NEXT: successors: %bb.3(0x80000000)
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: $vgpr0 = V_MOV_B32_e32 0, implicit $exec
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: bb.3.join:
+ ; CHECK-NEXT: bb.3:
; CHECK-NEXT: liveins: $vgpr0
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: S_WAITCNT 3952
; CHECK-NEXT: $vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
; CHECK-NEXT: S_ENDPGM 0
- bb.0.entry:
+ bb.0:
successors: %bb.1(0x40000000), %bb.2(0x40000000)
liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
@@ -87,7 +63,7 @@ body: |
S_CBRANCH_SCC1 %bb.2, implicit killed $scc
; then branch — issues async buffer load + ASYNCMARK
- bb.1.then:
+ bb.1:
successors: %bb.3(0x80000000)
liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
@@ -96,7 +72,7 @@ body: |
S_BRANCH %bb.3
; else branch — sync path, no async operations, OtherMarks is empty at join
- bb.2.else:
+ bb.2:
successors: %bb.3(0x80000000)
$vgpr0 = V_MOV_B32_e32 0, implicit $exec
@@ -106,7 +82,7 @@ body: |
; Before fix: assertion. After fix: returns early, preserves async marks.
; S_WAITCNT must be emitted before the use of $vgpr0 to drain the pending load.
; (mergeAsyncMarks sees non-empty AsyncMarks from then, empty from else)
- bb.3.join:
+ bb.3:
liveins: $vgpr0
$vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
@@ -121,7 +97,7 @@ machineFunctionInfo:
occupancy: 8
body: |
; CHECK-LABEL: name: asyncmark_in_else
- ; CHECK: bb.0.entry:
+ ; CHECK: bb.0:
; CHECK-NEXT: successors: %bb.1(0x40000000), %bb.2(0x40000000)
; CHECK-NEXT: liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
; CHECK-NEXT: {{ $}}
@@ -129,13 +105,13 @@ body: |
; CHECK-NEXT: S_CMP_LG_U32 $sgpr0, $sgpr1, implicit-def $scc
; CHECK-NEXT: S_CBRANCH_SCC1 %bb.2, implicit killed $scc
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: bb.1.then:
+ ; CHECK-NEXT: bb.1:
; CHECK-NEXT: successors: %bb.3(0x80000000)
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: $vgpr0 = V_MOV_B32_e32 0, implicit $exec
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: bb.2.else:
+ ; CHECK-NEXT: bb.2:
; CHECK-NEXT: successors: %bb.3(0x80000000)
; CHECK-NEXT: liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
; CHECK-NEXT: {{ $}}
@@ -143,13 +119,13 @@ body: |
; CHECK-NEXT: ASYNCMARK
; CHECK-NEXT: S_BRANCH %bb.3
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: bb.3.join:
+ ; CHECK-NEXT: bb.3:
; CHECK-NEXT: liveins: $vgpr0
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: S_WAITCNT 3952
; CHECK-NEXT: $vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
; CHECK-NEXT: S_ENDPGM 0
- bb.0.entry:
+ bb.0:
successors: %bb.1(0x40000000), %bb.2(0x40000000)
liveins: $sgpr0, $sgpr1, $sgpr4_sgpr5_sgpr6_sgpr7
@@ -157,14 +133,14 @@ body: |
S_CBRANCH_SCC1 %bb.2, implicit killed $scc
; then branch — sync path, no async operations
- bb.1.then:
+ bb.1:
successors: %bb.3(0x80000000)
$vgpr0 = V_MOV_B32_e32 0, implicit $exec
S_BRANCH %bb.3
; else branch — issues async buffer load + ASYNCMARK
- bb.2.else:
+ bb.2:
successors: %bb.3(0x80000000)
liveins: $sgpr0, $sgpr4_sgpr5_sgpr6_sgpr7
@@ -174,7 +150,7 @@ body: |
; join — mergeAsyncMarks sees empty AsyncMarks (then) and non-empty OtherMarks (else).
; S_WAITCNT must be emitted before the use of $vgpr0 to drain the pending load.
- bb.3.join:
+ bb.3:
liveins: $vgpr0
$vgpr1 = V_ADD_U32_e32 $vgpr0, $vgpr0, implicit $exec
More information about the llvm-commits
mailing list