[llvm] [AMDGPU] Add dynamic threshold for DPP atomic optimizer on integer LDS atomics (PR #186762)
YaFan Tao via llvm-commits
llvm-commits at lists.llvm.org
Tue Jun 30 02:10:06 PDT 2026
https://github.com/taoyafan updated https://github.com/llvm/llvm-project/pull/186762
>From 03ba3d87a73faaf5bbc9d3510b683077fab3d2e5 Mon Sep 17 00:00:00 2001
From: YaFan <YaFan.Tao at amd.com>
Date: Mon, 16 Mar 2026 17:23:06 +0800
Subject: [PATCH 01/17] [AMDGPU] Add dynamic threshold for DPP atomic optimizer
on integer LDS atomics
When only a few lanes are active, the DPP wavefront scan overhead can
outweigh the benefit of reducing LDS atomic contention. Profiling a
DX12 application showed that the affected shader ran ~20 percent faster
(and the whole application ~2.7 percent faster) with the atomic
optimizer disabled entirely.
Add a hidden command-line option -amdgpu-atomic-optimizer-dpp-lds-threshold
that emits a dynamic branch: when the active lane count exceeds the
threshold the existing DPP scan path is taken; otherwise each lane
independently issues its own atomic.
Only integer LDS atomics are affected because DX12 does not expose
floating-point LDS atomic instructions, so the float path is untested.
The default threshold is 0 (always use DPP) because unconditionally
enabling the threshold can regress DPP-friendly workloads by ~4 percent.
Implementation details:
- Extract buildMbcnt and buildDPPScanAndReduce as reusable helpers.
- Extract optimizeAtomicImpl from optimizeAtomic so the threshold path
can share the pixel-shader and replacement logic.
- Add optimizeAtomicWithDynamicThreshold which builds the two-path CFG.
- Add atomic_optimizations_dpp_lds_threshold.ll testing threshold=5
(dynamic branch) and threshold>=wavefront-size (DPP fully disabled).
---
.../Target/AMDGPU/AMDGPUAtomicOptimizer.cpp | 308 +++-
.../atomic_optimizations_dpp_lds_threshold.ll | 1528 +++++++++++++++++
2 files changed, 1784 insertions(+), 52 deletions(-)
create mode 100644 llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index 784ee36d55c1e..52996dd2ddc66 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -38,6 +38,23 @@
using namespace llvm;
using namespace llvm::AMDGPU;
+// When only a few lanes are active, the DPP scan overhead can outweigh the
+// benefit of reducing LDS atomic contention. This threshold lets the compiler
+// emit a dynamic branch so that small-active-lane cases fall back to
+// unoptimized per-lane atomics while large-active-lane cases still benefit from
+// DPP.
+//
+// Only integer LDS atomics are affected because floating-point path is
+// untested. The default is 0 (always use DPP) because unconditionally enabling
+// the threshold can regress DPP-friendly workloads by ~4% (tested on RX
+// 7900XT).
+static cl::opt<unsigned> AMDGPUAtomicOptimizerDPPLdsThreshold(
+ "amdgpu-atomic-optimizer-dpp-lds-threshold",
+ cl::desc("Use DPP scan for integer LDS atomics only when active lanes > "
+ "this value; floating-point LDS atomics are not affected "
+ "(0 = always use DPP, >= wavefront size = never use DPP)"),
+ cl::init(0), cl::Hidden);
+
namespace {
struct ReplacementInfo {
@@ -45,6 +62,7 @@ struct ReplacementInfo {
AtomicRMWInst::BinOp Op;
unsigned ValIdx;
bool ValDivergent;
+ bool IsLDS;
};
class AMDGPUAtomicOptimizer : public FunctionPass {
@@ -86,8 +104,22 @@ class AMDGPUAtomicOptimizerImpl
Value *const Identity, Value *V, Instruction &I,
BasicBlock *ComputeLoop, BasicBlock *ComputeEnd) const;
+ Value *buildMbcnt(IRBuilder<> &B, Value *Ballot) const;
+
+ std::pair<Value *, Value *>
+ buildDPPScanAndReduce(IRBuilder<> &B, AtomicRMWInst::BinOp ScanOp, Type *Ty,
+ Value *V, Value *Identity, bool NeedResult) const;
+
void optimizeAtomic(Instruction &I, AtomicRMWInst::BinOp Op, unsigned ValIdx,
- bool ValDivergent) const;
+ bool ValDivergent, bool IsLDS) const;
+
+ Value *optimizeAtomicImpl(IRBuilder<> &B, Instruction &I,
+ AtomicRMWInst::BinOp Op, unsigned ValIdx,
+ bool ValDivergent) const;
+
+ Value *optimizeAtomicWithDynamicThreshold(IRBuilder<> &B, Instruction &I,
+ AtomicRMWInst::BinOp Op,
+ unsigned ValIdx) const;
public:
AMDGPUAtomicOptimizerImpl() = delete;
@@ -159,8 +191,8 @@ bool AMDGPUAtomicOptimizerImpl::run() {
if (ToReplace.empty())
return false;
- for (auto &[I, Op, ValIdx, ValDivergent] : ToReplace)
- optimizeAtomic(*I, Op, ValIdx, ValDivergent);
+ for (auto &[I, Op, ValIdx, ValDivergent, IsLDS] : ToReplace)
+ optimizeAtomic(*I, Op, ValIdx, ValDivergent, IsLDS);
ToReplace.clear();
return true;
}
@@ -239,10 +271,12 @@ void AMDGPUAtomicOptimizerImpl::visitAtomicRMWInst(AtomicRMWInst &I) {
return;
}
+ const bool IsLDS = I.getPointerAddressSpace() == AMDGPUAS::LOCAL_ADDRESS;
+
// If we get here, we can optimize the atomic using a single wavefront-wide
// atomic operation to do the calculation for the entire wavefront, so
// remember the instruction so we can come back to it.
- ToReplace.push_back({&I, Op, ValIdx, ValDivergent});
+ ToReplace.push_back({&I, Op, ValIdx, ValDivergent, IsLDS});
}
void AMDGPUAtomicOptimizerImpl::visitIntrinsicInst(IntrinsicInst &I) {
@@ -333,7 +367,8 @@ void AMDGPUAtomicOptimizerImpl::visitIntrinsicInst(IntrinsicInst &I) {
// If we get here, we can optimize the atomic using a single wavefront-wide
// atomic operation to do the calculation for the entire wavefront, so
// remember the instruction so we can come back to it.
- ToReplace.push_back({&I, Op, ValIdx, ValDivergent});
+ // Buffer atomics are never LDS.
+ ToReplace.push_back({&I, Op, ValIdx, ValDivergent, /*IsLDS=*/false});
}
// Use the builder to create the non-atomic counterpart of the specified
@@ -602,6 +637,47 @@ std::pair<Value *, Value *> AMDGPUAtomicOptimizerImpl::buildScanIteratively(
return {OldValue, NewAccumulator};
}
+// Use the builder to create an mbcnt intrinsic call to count the number of
+// active lanes below the current lane.
+Value *AMDGPUAtomicOptimizerImpl::buildMbcnt(IRBuilder<> &B,
+ Value *Ballot) const {
+ Type *Int32Ty = B.getInt32Ty();
+ if (ST.isWave32()) {
+ return B.CreateIntrinsic(Intrinsic::amdgcn_mbcnt_lo,
+ {Ballot, B.getInt32(0)});
+ }
+ Value *const ExtractLo = B.CreateTrunc(Ballot, Int32Ty);
+ Value *const ExtractHi = B.CreateTrunc(B.CreateLShr(Ballot, 32), Int32Ty);
+ Value *Mbcnt =
+ B.CreateIntrinsic(Intrinsic::amdgcn_mbcnt_lo, {ExtractLo, B.getInt32(0)});
+ return B.CreateIntrinsic(Intrinsic::amdgcn_mbcnt_hi, {ExtractHi, Mbcnt});
+}
+
+// Use the builder to create a DPP-based scan or reduction across the
+// wavefront on divergent values. Sets inactive lanes to the identity value,
+// then either builds a full scan (when the result is needed) or just a
+// reduction. Returns {ReducedValue, ExclusiveScan}; ExclusiveScan is nullptr
+// when !NeedResult && ST.hasPermLaneX16().
+std::pair<Value *, Value *> AMDGPUAtomicOptimizerImpl::buildDPPScanAndReduce(
+ IRBuilder<> &B, AtomicRMWInst::BinOp ScanOp, Type *Ty, Value *V,
+ Value *Identity, bool NeedResult) const {
+ Value *NewV =
+ B.CreateIntrinsic(Intrinsic::amdgcn_set_inactive, Ty, {V, Identity});
+ Value *ExclScan = nullptr;
+ if (!NeedResult && ST.hasPermLaneX16()) {
+ NewV = buildReduction(B, ScanOp, NewV, Identity);
+ } else {
+ NewV = buildScan(B, ScanOp, NewV, Identity);
+ if (NeedResult)
+ ExclScan = buildShiftRight(B, NewV, Identity);
+ Value *const LastLaneIdx = B.getInt32(ST.getWavefrontSize() - 1);
+ NewV =
+ B.CreateIntrinsic(Ty, Intrinsic::amdgcn_readlane, {NewV, LastLaneIdx});
+ }
+ NewV = B.CreateIntrinsic(Intrinsic::amdgcn_strict_wwm, Ty, NewV);
+ return {NewV, ExclScan};
+}
+
static Constant *getIdentityValueForAtomicOp(Type *const Ty,
AtomicRMWInst::BinOp Op) {
LLVMContext &C = Ty->getContext();
@@ -644,7 +720,8 @@ static Value *buildMul(IRBuilder<> &B, Value *LHS, Value *RHS) {
void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
AtomicRMWInst::BinOp Op,
unsigned ValIdx,
- bool ValDivergent) const {
+ bool ValDivergent,
+ bool IsLDS) const {
// Start building just before the instruction.
IRBuilder<> B(&I);
@@ -677,6 +754,47 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
B.SetInsertPoint(&I);
}
+ // Decide which implementation path to use.
+ // For DPP strategy with integer, divergent-value LDS atomics, when a
+ // threshold is set, use dynamic branching based on active lane count.
+ const unsigned Threshold = AMDGPUAtomicOptimizerDPPLdsThreshold;
+ Value *Result = nullptr;
+ if (IsLDS && ValDivergent && ScanImpl == ScanOptions::DPP &&
+ !AtomicRMWInst::isFPOperation(Op) && Threshold > 0 &&
+ Threshold < ST.getWavefrontSize()) {
+ Result = optimizeAtomicWithDynamicThreshold(B, I, Op, ValIdx);
+ } else {
+ Result = optimizeAtomicImpl(B, I, Op, ValIdx, ValDivergent);
+ }
+
+ // Handle pixel shader reconvergence and replace original instruction.
+ if (Result) {
+ if (IsPixelShader) {
+ // Need a final PHI to reconverge to above the helper lane branch mask.
+ B.SetInsertPoint(PixelExitBB, PixelExitBB->getFirstNonPHIIt());
+
+ PHINode *const PHI = B.CreatePHI(I.getType(), 2);
+ PHI->addIncoming(PoisonValue::get(I.getType()), PixelEntryBB);
+ PHI->addIncoming(Result, I.getParent());
+ I.replaceAllUsesWith(PHI);
+ } else {
+ // Replace the original atomic instruction with the new one.
+ I.replaceAllUsesWith(Result);
+ }
+ }
+
+ // And delete the original.
+ I.eraseFromParent();
+}
+
+// Replace a single atomic with a wavefront-wide scan/reduction followed by a
+// single-lane atomic, then reconstruct per-lane results. Returns the per-lane
+// result, or nullptr when the result is unused.
+Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
+ Instruction &I,
+ AtomicRMWInst::BinOp Op,
+ unsigned ValIdx,
+ bool ValDivergent) const {
Type *const Ty = I.getType();
Type *Int32Ty = B.getInt32Ty();
bool isAtomicFloatingPointTy = Ty->isFloatingPointTy();
@@ -696,17 +814,7 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
// below us. If we counted each lane linearly starting from 0, a lane is
// below us only if its associated index was less than ours. We do this by
// using the mbcnt intrinsic.
- Value *Mbcnt;
- if (ST.isWave32()) {
- Mbcnt =
- B.CreateIntrinsic(Intrinsic::amdgcn_mbcnt_lo, {Ballot, B.getInt32(0)});
- } else {
- Value *const ExtractLo = B.CreateTrunc(Ballot, Int32Ty);
- Value *const ExtractHi = B.CreateTrunc(B.CreateLShr(Ballot, 32), Int32Ty);
- Mbcnt = B.CreateIntrinsic(Intrinsic::amdgcn_mbcnt_lo,
- {ExtractLo, B.getInt32(0)});
- Mbcnt = B.CreateIntrinsic(Intrinsic::amdgcn_mbcnt_hi, {ExtractHi, Mbcnt});
- }
+ Value *Mbcnt = buildMbcnt(B, Ballot);
Function *F = I.getFunction();
LLVMContext &C = F->getContext();
@@ -732,28 +840,8 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
// using DPP.
if (ValDivergent) {
if (ScanImpl == ScanOptions::DPP) {
- // First we need to set all inactive invocations to the identity value, so
- // that they can correctly contribute to the final result.
- NewV =
- B.CreateIntrinsic(Intrinsic::amdgcn_set_inactive, Ty, {V, Identity});
- if (!NeedResult && ST.hasPermLaneX16()) {
- // On GFX10 the permlanex16 instruction helps us build a reduction
- // without too many readlanes and writelanes, which are generally bad
- // for performance.
- NewV = buildReduction(B, ScanOp, NewV, Identity);
- } else {
- NewV = buildScan(B, ScanOp, NewV, Identity);
- if (NeedResult)
- ExclScan = buildShiftRight(B, NewV, Identity);
- // Read the value from the last lane, which has accumulated the values
- // of each active lane in the wavefront. This will be our new value
- // which we will provide to the atomic operation.
- Value *const LastLaneIdx = B.getInt32(ST.getWavefrontSize() - 1);
- NewV = B.CreateIntrinsic(Ty, Intrinsic::amdgcn_readlane,
- {NewV, LastLaneIdx});
- }
- // Finally mark the readlanes in the WWM section.
- NewV = B.CreateIntrinsic(Intrinsic::amdgcn_strict_wwm, Ty, NewV);
+ std::tie(NewV, ExclScan) =
+ buildDPPScanAndReduce(B, ScanOp, Ty, V, Identity, NeedResult);
} else if (ScanImpl == ScanOptions::Iterative) {
// Alternative implementation for scan
ComputeLoop = BasicBlock::Create(C, "ComputeLoop", F);
@@ -955,23 +1043,139 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
// first active lane.
Result = B.CreateSelect(Cond, BroadcastI, Result);
}
+ return Result;
+ }
+ return nullptr;
+}
- if (IsPixelShader) {
- // Need a final PHI to reconverge to above the helper lane branch mask.
- B.SetInsertPoint(PixelExitBB, PixelExitBB->getFirstNonPHIIt());
+// Generate a dynamic branch based on the active lane count so that the
+// DPP scan path is only used when enough lanes are active to amortise its
+// overhead. When the active count is at or below the threshold, each lane
+// independently issues its own atomic, which is cheaper for small groups.
+//
+// CFG produced:
+//
+// EntryBB:
+// Ballot / Ctpop -> ActiveCount
+// cmp ActiveCount >= Threshold
+// br -> DppBB or NoOptBB
+//
+// DppBB:
+// DPP scan / reduction, single-lane gate
+// br -> SingleLaneBB or DppExitBB
+//
+// SingleLaneBB:
+// atomic with reduced value
+// br -> DppExitBB
+//
+// DppExitBB:
+// readfirstlane + per-lane result
+// br -> MergeBB
+//
+// NoOptBB:
+// original atomic (unoptimized)
+// br -> MergeBB
+//
+// MergeBB:
+// PHI merges results from both paths
+Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicWithDynamicThreshold(
+ IRBuilder<> &B, Instruction &I, AtomicRMWInst::BinOp Op,
+ unsigned ValIdx) const {
+ Type *const Ty = I.getType();
+ const unsigned TyBitWidth = DL.getTypeSizeInBits(Ty);
+ const bool NeedResult = !I.use_empty();
- PHINode *const PHI = B.CreatePHI(Ty, 2);
- PHI->addIncoming(PoisonValue::get(Ty), PixelEntryBB);
- PHI->addIncoming(Result, I.getParent());
- I.replaceAllUsesWith(PHI);
- } else {
- // Replace the original atomic instruction with the new one.
- I.replaceAllUsesWith(Result);
- }
+ Type *const WaveTy = B.getIntNTy(ST.getWavefrontSize());
+ CallInst *const Ballot =
+ B.CreateIntrinsic(Intrinsic::amdgcn_ballot, WaveTy, B.getTrue());
+
+ Value *Mbcnt = buildMbcnt(B, Ballot);
+
+ // Count active lanes.
+ Value *const Ctpop = B.CreateIntCast(
+ B.CreateUnaryIntrinsic(Intrinsic::ctpop, Ballot), B.getInt32Ty(), false);
+
+ // Branch: if active lanes > threshold, use DPP; otherwise no-opt.
+ const unsigned Threshold = AMDGPUAtomicOptimizerDPPLdsThreshold;
+ Value *const ThresholdCond = B.CreateICmpUGT(Ctpop, B.getInt32(Threshold));
+
+ BasicBlock *const EntryBB = I.getParent();
+ Instruction *const SplitPt = &I;
+ Instruction *const ThenTerm = SplitBlockAndInsertIfThen(
+ ThresholdCond, SplitPt, false, nullptr, &DTU, nullptr);
+ BasicBlock *const DppBB = ThenTerm->getParent();
+ BasicBlock *const MergeBB = I.getParent();
+
+ // Also create NoOptBB between DppBB and MergeBB.
+ BasicBlock *const NoOptBB = BasicBlock::Create(
+ I.getContext(), "atomicrmw.no_opt", I.getFunction(), MergeBB);
+ // Fix CFG: EntryBB's conditional branch false edge -> NoOptBB -> MergeBB.
+ BranchInst *const EntryBr = cast<BranchInst>(EntryBB->getTerminator());
+ EntryBr->setSuccessor(1, NoOptBB);
+ IRBuilder<> NoOptBuilder(NoOptBB);
+ NoOptBuilder.CreateBr(MergeBB);
+ DTU.applyUpdates({{DominatorTree::Insert, EntryBB, NoOptBB},
+ {DominatorTree::Delete, EntryBB, MergeBB},
+ {DominatorTree::Insert, NoOptBB, MergeBB}});
+
+ // DppBB: perform DPP scan/reduction and single-lane atomic.
+ B.SetInsertPoint(ThenTerm);
+
+ AtomicRMWInst::BinOp ScanOp = Op;
+ if (Op == AtomicRMWInst::Sub)
+ ScanOp = AtomicRMWInst::Add;
+
+ Value *Identity = getIdentityValueForAtomicOp(Ty, ScanOp);
+ Value *V = I.getOperand(ValIdx);
+
+ auto [NewV, ExclScan] =
+ buildDPPScanAndReduce(B, ScanOp, Ty, V, Identity, NeedResult);
+
+ Value *const DppIsFirst = B.CreateICmpEQ(Mbcnt, B.getInt32(0));
+
+ BasicBlock *const DppOrigBB = B.GetInsertBlock();
+ Instruction *const DppSingleTerm = SplitBlockAndInsertIfThen(
+ DppIsFirst, ThenTerm, false, nullptr, &DTU, nullptr);
+ B.SetInsertPoint(DppSingleTerm);
+ Instruction *const DppNewI = I.clone();
+ B.Insert(DppNewI);
+ DppNewI->setOperand(ValIdx, NewV);
+
+ Value *DppResult = nullptr;
+ if (NeedResult) {
+ B.SetInsertPoint(ThenTerm);
+ PHINode *const DppPHI = B.CreatePHI(Ty, 2);
+ DppPHI->addIncoming(PoisonValue::get(Ty), DppOrigBB);
+ DppPHI->addIncoming(DppNewI, DppSingleTerm->getParent());
+
+ Value *ReadlaneVal = DppPHI;
+ if (TyBitWidth < 32)
+ ReadlaneVal = B.CreateZExt(DppPHI, B.getInt32Ty());
+ Value *BroadcastI = B.CreateIntrinsic(
+ ReadlaneVal->getType(), Intrinsic::amdgcn_readfirstlane, ReadlaneVal);
+ if (TyBitWidth < 32)
+ BroadcastI = B.CreateTrunc(BroadcastI, Ty);
+
+ Value *LaneOffset =
+ B.CreateIntrinsic(Intrinsic::amdgcn_strict_wwm, Ty, ExclScan);
+ DppResult = buildNonAtomicBinOp(B, Op, BroadcastI, LaneOffset);
}
- // And delete the original.
- I.eraseFromParent();
+ // NoOptBB: each lane independently issues its own atomic.
+ NoOptBuilder.SetInsertPoint(NoOptBB->getTerminator());
+ Instruction *const NoOptNewI = I.clone();
+ NoOptBuilder.Insert(NoOptNewI);
+ Value *NoOptResult = NeedResult ? static_cast<Value *>(NoOptNewI) : nullptr;
+
+ // MergeBB: PHI-merge results from DppBB and NoOptBB.
+ if (NeedResult) {
+ B.SetInsertPoint(MergeBB, MergeBB->getFirstNonPHIIt());
+ PHINode *const MergePHI = B.CreatePHI(Ty, 2);
+ MergePHI->addIncoming(DppResult, ThenTerm->getParent());
+ MergePHI->addIncoming(NoOptResult, NoOptBB);
+ return MergePHI;
+ }
+ return nullptr;
}
INITIALIZE_PASS_BEGIN(AMDGPUAtomicOptimizer, DEBUG_TYPE,
diff --git a/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll b/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
new file mode 100644
index 0000000000000..0ef74d7ca2c02
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
@@ -0,0 +1,1528 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+;
+; Test the -amdgpu-atomic-optimizer-dpp-lds-threshold option which controls
+; dynamic DPP vs no-opt branching for integer LDS atomics.
+;
+; Threshold=5: use DPP only when active lanes > 5, otherwise each lane does
+; its own atomic.
+; Threshold=32: on wave32 this disables DPP entirely (>= wavefront size).
+; Threshold=64: on wave64 this disables DPP entirely (>= wavefront size).
+
+; --- Threshold=5 tests (dynamic branch expected) ---
+; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP -amdgpu-atomic-optimizer-dpp-lds-threshold=5 < %s | FileCheck -enable-var-scope -check-prefixes=GFX1032-THRESH5 %s
+; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP -amdgpu-atomic-optimizer-dpp-lds-threshold=5 < %s | FileCheck -enable-var-scope -check-prefixes=GFX1064-THRESH5 %s
+; RUN: llc -mtriple=amdgcn -mcpu=gfx900 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP -amdgpu-atomic-optimizer-dpp-lds-threshold=5 < %s | FileCheck -enable-var-scope -check-prefixes=GFX900-THRESH5 %s
+
+; --- Threshold >= wavefront size tests (DPP fully disabled for int LDS) ---
+; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP -amdgpu-atomic-optimizer-dpp-lds-threshold=32 < %s | FileCheck -enable-var-scope -check-prefixes=GFX1032-DISABLED %s
+; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP -amdgpu-atomic-optimizer-dpp-lds-threshold=64 < %s | FileCheck -enable-var-scope -check-prefixes=GFX1064-DISABLED %s
+
+declare i32 @llvm.amdgcn.workitem.id.x()
+
+ at local_var32 = addrspace(3) global i32 undef, align 4
+
+; Test 1: divergent i32 add with result used -- dynamic threshold branch expected
+define amdgpu_kernel void @add_i32_varying(ptr addrspace(1) %out) {
+; GFX1032-THRESH5-LABEL: add_i32_varying:
+; GFX1032-THRESH5: ; %bb.0: ; %entry
+; GFX1032-THRESH5-NEXT: s_bcnt1_i32_b32 s0, exec_lo
+; GFX1032-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
+; GFX1032-THRESH5-NEXT: s_cbranch_scc0 .LBB0_2
+; GFX1032-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v4, 0
+; GFX1032-THRESH5-NEXT: ds_add_rtn_u32 v4, v4, v0
+; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-THRESH5-NEXT: buffer_gl0_inv
+; GFX1032-THRESH5-NEXT: s_cbranch_execz .LBB0_3
+; GFX1032-THRESH5-NEXT: s_branch .LBB0_6
+; GFX1032-THRESH5-NEXT: .LBB0_2:
+; GFX1032-THRESH5-NEXT: ; implicit-def: $vgpr4
+; GFX1032-THRESH5-NEXT: .LBB0_3:
+; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v3, 0
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-THRESH5-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1032-THRESH5-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-THRESH5-NEXT: v_readlane_b32 s1, v1, 15
+; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, 0
+; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-THRESH5-NEXT: v_writelane_b32 v3, s1, 16
+; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
+; GFX1032-THRESH5-NEXT: ; implicit-def: $vgpr4
+; GFX1032-THRESH5-NEXT: s_and_saveexec_b32 s0, vcc_lo
+; GFX1032-THRESH5-NEXT: s_cbranch_execz .LBB0_5
+; GFX1032-THRESH5-NEXT: ; %bb.4:
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v4, s2
+; GFX1032-THRESH5-NEXT: s_waitcnt_vscnt null, 0x0
+; GFX1032-THRESH5-NEXT: ds_add_rtn_u32 v4, v0, v4
+; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-THRESH5-NEXT: buffer_gl0_inv
+; GFX1032-THRESH5-NEXT: .LBB0_5:
+; GFX1032-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-THRESH5-NEXT: s_or_b32 exec_lo, exec_lo, s0
+; GFX1032-THRESH5-NEXT: v_readfirstlane_b32 s0, v4
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, v3
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_e32 v4, s0, v0
+; GFX1032-THRESH5-NEXT: .LBB0_6:
+; GFX1032-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1032-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1032-THRESH5-NEXT: s_mov_b32 s2, -1
+; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-THRESH5-NEXT: buffer_store_dword v4, off, s[0:3], 0
+; GFX1032-THRESH5-NEXT: s_endpgm
+;
+; GFX1064-THRESH5-LABEL: add_i32_varying:
+; GFX1064-THRESH5: ; %bb.0: ; %entry
+; GFX1064-THRESH5-NEXT: s_bcnt1_i32_b64 s0, exec
+; GFX1064-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
+; GFX1064-THRESH5-NEXT: s_cbranch_scc0 .LBB0_2
+; GFX1064-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v4, 0
+; GFX1064-THRESH5-NEXT: ds_add_rtn_u32 v4, v4, v0
+; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-THRESH5-NEXT: buffer_gl0_inv
+; GFX1064-THRESH5-NEXT: s_cbranch_execz .LBB0_3
+; GFX1064-THRESH5-NEXT: s_branch .LBB0_6
+; GFX1064-THRESH5-NEXT: .LBB0_2:
+; GFX1064-THRESH5-NEXT: ; implicit-def: $vgpr4
+; GFX1064-THRESH5-NEXT: .LBB0_3:
+; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v3, 0
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-THRESH5-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v2, s2
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 15
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s3, v1, 31
+; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s2, 16
+; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 63
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s6, v1, 47
+; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s3, 32
+; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, 0
+; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s6, 48
+; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX1064-THRESH5-NEXT: ; implicit-def: $vgpr4
+; GFX1064-THRESH5-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1064-THRESH5-NEXT: s_cbranch_execz .LBB0_5
+; GFX1064-THRESH5-NEXT: ; %bb.4:
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v4, s2
+; GFX1064-THRESH5-NEXT: s_waitcnt_vscnt null, 0x0
+; GFX1064-THRESH5-NEXT: ds_add_rtn_u32 v4, v0, v4
+; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-THRESH5-NEXT: buffer_gl0_inv
+; GFX1064-THRESH5-NEXT: .LBB0_5:
+; GFX1064-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-THRESH5-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1064-THRESH5-NEXT: v_readfirstlane_b32 s0, v4
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, v3
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_e32 v4, s0, v0
+; GFX1064-THRESH5-NEXT: .LBB0_6:
+; GFX1064-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-THRESH5-NEXT: s_mov_b32 s2, -1
+; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-THRESH5-NEXT: buffer_store_dword v4, off, s[0:3], 0
+; GFX1064-THRESH5-NEXT: s_endpgm
+;
+; GFX900-THRESH5-LABEL: add_i32_varying:
+; GFX900-THRESH5: ; %bb.0: ; %entry
+; GFX900-THRESH5-NEXT: s_bcnt1_i32_b64 s0, exec
+; GFX900-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
+; GFX900-THRESH5-NEXT: s_cbranch_scc0 .LBB0_2
+; GFX900-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v3, 0
+; GFX900-THRESH5-NEXT: ds_add_rtn_u32 v3, v3, v0
+; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-THRESH5-NEXT: s_cbranch_execz .LBB0_3
+; GFX900-THRESH5-NEXT: s_branch .LBB0_6
+; GFX900-THRESH5-NEXT: .LBB0_2:
+; GFX900-THRESH5-NEXT: ; implicit-def: $vgpr3
+; GFX900-THRESH5-NEXT: .LBB0_3:
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v3, 0
+; GFX900-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX900-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v4
+; GFX900-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX900-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v2, 0
+; GFX900-THRESH5-NEXT: s_nop 0
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
+; GFX900-THRESH5-NEXT: v_readlane_b32 s2, v1, 63
+; GFX900-THRESH5-NEXT: s_nop 0
+; GFX900-THRESH5-NEXT: v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
+; GFX900-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
+; GFX900-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX900-THRESH5-NEXT: ; implicit-def: $vgpr0
+; GFX900-THRESH5-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX900-THRESH5-NEXT: s_cbranch_execz .LBB0_5
+; GFX900-THRESH5-NEXT: ; %bb.4:
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, s2
+; GFX900-THRESH5-NEXT: ds_add_rtn_u32 v0, v3, v0
+; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-THRESH5-NEXT: .LBB0_5:
+; GFX900-THRESH5-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX900-THRESH5-NEXT: v_readfirstlane_b32 s0, v0
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, v2
+; GFX900-THRESH5-NEXT: v_add_u32_e32 v3, s0, v0
+; GFX900-THRESH5-NEXT: .LBB0_6:
+; GFX900-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX900-THRESH5-NEXT: s_mov_b32 s3, 0xf000
+; GFX900-THRESH5-NEXT: s_mov_b32 s2, -1
+; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-THRESH5-NEXT: buffer_store_dword v3, off, s[0:3], 0
+; GFX900-THRESH5-NEXT: s_endpgm
+;
+; GFX1032-DISABLED-LABEL: add_i32_varying:
+; GFX1032-DISABLED: ; %bb.0: ; %entry
+; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-DISABLED-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v3, 0
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-DISABLED-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1032-DISABLED-NEXT: v_readlane_b32 s1, v1, 15
+; GFX1032-DISABLED-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v4, 0
+; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-DISABLED-NEXT: v_writelane_b32 v3, s1, 16
+; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
+; GFX1032-DISABLED-NEXT: s_mov_b32 s0, s2
+; GFX1032-DISABLED-NEXT: s_mov_b32 s2, -1
+; GFX1032-DISABLED-NEXT: ; implicit-def: $vgpr0
+; GFX1032-DISABLED-NEXT: s_and_saveexec_b32 s1, vcc_lo
+; GFX1032-DISABLED-NEXT: s_cbranch_execz .LBB0_2
+; GFX1032-DISABLED-NEXT: ; %bb.1:
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, s0
+; GFX1032-DISABLED-NEXT: ds_add_rtn_u32 v0, v4, v0
+; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-DISABLED-NEXT: buffer_gl0_inv
+; GFX1032-DISABLED-NEXT: .LBB0_2:
+; GFX1032-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-DISABLED-NEXT: s_or_b32 exec_lo, exec_lo, s1
+; GFX1032-DISABLED-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1032-DISABLED-NEXT: v_readfirstlane_b32 s3, v0
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, v3
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_e32 v0, s3, v0
+; GFX1032-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1032-DISABLED-NEXT: s_endpgm
+;
+; GFX1064-DISABLED-LABEL: add_i32_varying:
+; GFX1064-DISABLED: ; %bb.0: ; %entry
+; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-DISABLED-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v3, 0
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-DISABLED-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v2, s2
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 15
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s3, v1, 31
+; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s2, 16
+; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 47
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s6, v1, 63
+; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s3, 32
+; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-DISABLED-NEXT: v_mbcnt_hi_u32_b32 v0, exec_hi, v0
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v4, 0
+; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s2, 48
+; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX1064-DISABLED-NEXT: s_mov_b32 s2, -1
+; GFX1064-DISABLED-NEXT: ; implicit-def: $vgpr0
+; GFX1064-DISABLED-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1064-DISABLED-NEXT: s_cbranch_execz .LBB0_2
+; GFX1064-DISABLED-NEXT: ; %bb.1:
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, s6
+; GFX1064-DISABLED-NEXT: ds_add_rtn_u32 v0, v4, v0
+; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-DISABLED-NEXT: buffer_gl0_inv
+; GFX1064-DISABLED-NEXT: .LBB0_2:
+; GFX1064-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-DISABLED-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1064-DISABLED-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-DISABLED-NEXT: v_readfirstlane_b32 s3, v0
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, v3
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_e32 v0, s3, v0
+; GFX1064-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1064-DISABLED-NEXT: s_endpgm
+entry:
+ %lane = call i32 @llvm.amdgcn.workitem.id.x()
+ %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel
+ store i32 %old, ptr addrspace(1) %out
+ ret void
+}
+
+; Test 2: divergent i32 sub with result used
+define amdgpu_kernel void @sub_i32_varying(ptr addrspace(1) %out) {
+; GFX1032-THRESH5-LABEL: sub_i32_varying:
+; GFX1032-THRESH5: ; %bb.0: ; %entry
+; GFX1032-THRESH5-NEXT: s_bcnt1_i32_b32 s0, exec_lo
+; GFX1032-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
+; GFX1032-THRESH5-NEXT: s_cbranch_scc0 .LBB1_2
+; GFX1032-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v4, 0
+; GFX1032-THRESH5-NEXT: ds_sub_rtn_u32 v4, v4, v0
+; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-THRESH5-NEXT: buffer_gl0_inv
+; GFX1032-THRESH5-NEXT: s_cbranch_execz .LBB1_3
+; GFX1032-THRESH5-NEXT: s_branch .LBB1_6
+; GFX1032-THRESH5-NEXT: .LBB1_2:
+; GFX1032-THRESH5-NEXT: ; implicit-def: $vgpr4
+; GFX1032-THRESH5-NEXT: .LBB1_3:
+; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v3, 0
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-THRESH5-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1032-THRESH5-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-THRESH5-NEXT: v_readlane_b32 s1, v1, 15
+; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, 0
+; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-THRESH5-NEXT: v_writelane_b32 v3, s1, 16
+; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
+; GFX1032-THRESH5-NEXT: ; implicit-def: $vgpr4
+; GFX1032-THRESH5-NEXT: s_and_saveexec_b32 s0, vcc_lo
+; GFX1032-THRESH5-NEXT: s_cbranch_execz .LBB1_5
+; GFX1032-THRESH5-NEXT: ; %bb.4:
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v4, s2
+; GFX1032-THRESH5-NEXT: s_waitcnt_vscnt null, 0x0
+; GFX1032-THRESH5-NEXT: ds_sub_rtn_u32 v4, v0, v4
+; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-THRESH5-NEXT: buffer_gl0_inv
+; GFX1032-THRESH5-NEXT: .LBB1_5:
+; GFX1032-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-THRESH5-NEXT: s_or_b32 exec_lo, exec_lo, s0
+; GFX1032-THRESH5-NEXT: v_readfirstlane_b32 s0, v4
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, v3
+; GFX1032-THRESH5-NEXT: v_sub_nc_u32_e32 v4, s0, v0
+; GFX1032-THRESH5-NEXT: .LBB1_6:
+; GFX1032-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1032-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1032-THRESH5-NEXT: s_mov_b32 s2, -1
+; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-THRESH5-NEXT: buffer_store_dword v4, off, s[0:3], 0
+; GFX1032-THRESH5-NEXT: s_endpgm
+;
+; GFX1064-THRESH5-LABEL: sub_i32_varying:
+; GFX1064-THRESH5: ; %bb.0: ; %entry
+; GFX1064-THRESH5-NEXT: s_bcnt1_i32_b64 s0, exec
+; GFX1064-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
+; GFX1064-THRESH5-NEXT: s_cbranch_scc0 .LBB1_2
+; GFX1064-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v4, 0
+; GFX1064-THRESH5-NEXT: ds_sub_rtn_u32 v4, v4, v0
+; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-THRESH5-NEXT: buffer_gl0_inv
+; GFX1064-THRESH5-NEXT: s_cbranch_execz .LBB1_3
+; GFX1064-THRESH5-NEXT: s_branch .LBB1_6
+; GFX1064-THRESH5-NEXT: .LBB1_2:
+; GFX1064-THRESH5-NEXT: ; implicit-def: $vgpr4
+; GFX1064-THRESH5-NEXT: .LBB1_3:
+; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v3, 0
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-THRESH5-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v2, s2
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 15
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s3, v1, 31
+; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s2, 16
+; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 63
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s6, v1, 47
+; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s3, 32
+; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, 0
+; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s6, 48
+; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX1064-THRESH5-NEXT: ; implicit-def: $vgpr4
+; GFX1064-THRESH5-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1064-THRESH5-NEXT: s_cbranch_execz .LBB1_5
+; GFX1064-THRESH5-NEXT: ; %bb.4:
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v4, s2
+; GFX1064-THRESH5-NEXT: s_waitcnt_vscnt null, 0x0
+; GFX1064-THRESH5-NEXT: ds_sub_rtn_u32 v4, v0, v4
+; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-THRESH5-NEXT: buffer_gl0_inv
+; GFX1064-THRESH5-NEXT: .LBB1_5:
+; GFX1064-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-THRESH5-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1064-THRESH5-NEXT: v_readfirstlane_b32 s0, v4
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, v3
+; GFX1064-THRESH5-NEXT: v_sub_nc_u32_e32 v4, s0, v0
+; GFX1064-THRESH5-NEXT: .LBB1_6:
+; GFX1064-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-THRESH5-NEXT: s_mov_b32 s2, -1
+; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-THRESH5-NEXT: buffer_store_dword v4, off, s[0:3], 0
+; GFX1064-THRESH5-NEXT: s_endpgm
+;
+; GFX900-THRESH5-LABEL: sub_i32_varying:
+; GFX900-THRESH5: ; %bb.0: ; %entry
+; GFX900-THRESH5-NEXT: s_bcnt1_i32_b64 s0, exec
+; GFX900-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
+; GFX900-THRESH5-NEXT: s_cbranch_scc0 .LBB1_2
+; GFX900-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v3, 0
+; GFX900-THRESH5-NEXT: ds_sub_rtn_u32 v3, v3, v0
+; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-THRESH5-NEXT: s_cbranch_execz .LBB1_3
+; GFX900-THRESH5-NEXT: s_branch .LBB1_6
+; GFX900-THRESH5-NEXT: .LBB1_2:
+; GFX900-THRESH5-NEXT: ; implicit-def: $vgpr3
+; GFX900-THRESH5-NEXT: .LBB1_3:
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v3, 0
+; GFX900-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX900-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v4
+; GFX900-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX900-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v2, 0
+; GFX900-THRESH5-NEXT: s_nop 0
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
+; GFX900-THRESH5-NEXT: v_readlane_b32 s2, v1, 63
+; GFX900-THRESH5-NEXT: s_nop 0
+; GFX900-THRESH5-NEXT: v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
+; GFX900-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
+; GFX900-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX900-THRESH5-NEXT: ; implicit-def: $vgpr0
+; GFX900-THRESH5-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX900-THRESH5-NEXT: s_cbranch_execz .LBB1_5
+; GFX900-THRESH5-NEXT: ; %bb.4:
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, s2
+; GFX900-THRESH5-NEXT: ds_sub_rtn_u32 v0, v3, v0
+; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-THRESH5-NEXT: .LBB1_5:
+; GFX900-THRESH5-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX900-THRESH5-NEXT: v_readfirstlane_b32 s0, v0
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, v2
+; GFX900-THRESH5-NEXT: v_sub_u32_e32 v3, s0, v0
+; GFX900-THRESH5-NEXT: .LBB1_6:
+; GFX900-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX900-THRESH5-NEXT: s_mov_b32 s3, 0xf000
+; GFX900-THRESH5-NEXT: s_mov_b32 s2, -1
+; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-THRESH5-NEXT: buffer_store_dword v3, off, s[0:3], 0
+; GFX900-THRESH5-NEXT: s_endpgm
+;
+; GFX1032-DISABLED-LABEL: sub_i32_varying:
+; GFX1032-DISABLED: ; %bb.0: ; %entry
+; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-DISABLED-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v3, 0
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-DISABLED-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1032-DISABLED-NEXT: v_readlane_b32 s1, v1, 15
+; GFX1032-DISABLED-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v4, 0
+; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-DISABLED-NEXT: v_writelane_b32 v3, s1, 16
+; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
+; GFX1032-DISABLED-NEXT: s_mov_b32 s0, s2
+; GFX1032-DISABLED-NEXT: s_mov_b32 s2, -1
+; GFX1032-DISABLED-NEXT: ; implicit-def: $vgpr0
+; GFX1032-DISABLED-NEXT: s_and_saveexec_b32 s1, vcc_lo
+; GFX1032-DISABLED-NEXT: s_cbranch_execz .LBB1_2
+; GFX1032-DISABLED-NEXT: ; %bb.1:
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, s0
+; GFX1032-DISABLED-NEXT: ds_sub_rtn_u32 v0, v4, v0
+; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-DISABLED-NEXT: buffer_gl0_inv
+; GFX1032-DISABLED-NEXT: .LBB1_2:
+; GFX1032-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-DISABLED-NEXT: s_or_b32 exec_lo, exec_lo, s1
+; GFX1032-DISABLED-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1032-DISABLED-NEXT: v_readfirstlane_b32 s3, v0
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, v3
+; GFX1032-DISABLED-NEXT: v_sub_nc_u32_e32 v0, s3, v0
+; GFX1032-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1032-DISABLED-NEXT: s_endpgm
+;
+; GFX1064-DISABLED-LABEL: sub_i32_varying:
+; GFX1064-DISABLED: ; %bb.0: ; %entry
+; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-DISABLED-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v3, 0
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-DISABLED-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v2, s2
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 15
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s3, v1, 31
+; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s2, 16
+; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 47
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s6, v1, 63
+; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s3, 32
+; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-DISABLED-NEXT: v_mbcnt_hi_u32_b32 v0, exec_hi, v0
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v4, 0
+; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s2, 48
+; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX1064-DISABLED-NEXT: s_mov_b32 s2, -1
+; GFX1064-DISABLED-NEXT: ; implicit-def: $vgpr0
+; GFX1064-DISABLED-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1064-DISABLED-NEXT: s_cbranch_execz .LBB1_2
+; GFX1064-DISABLED-NEXT: ; %bb.1:
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, s6
+; GFX1064-DISABLED-NEXT: ds_sub_rtn_u32 v0, v4, v0
+; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-DISABLED-NEXT: buffer_gl0_inv
+; GFX1064-DISABLED-NEXT: .LBB1_2:
+; GFX1064-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-DISABLED-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1064-DISABLED-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-DISABLED-NEXT: v_readfirstlane_b32 s3, v0
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, v3
+; GFX1064-DISABLED-NEXT: v_sub_nc_u32_e32 v0, s3, v0
+; GFX1064-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1064-DISABLED-NEXT: s_endpgm
+entry:
+ %lane = call i32 @llvm.amdgcn.workitem.id.x()
+ %old = atomicrmw sub ptr addrspace(3) @local_var32, i32 %lane acq_rel
+ store i32 %old, ptr addrspace(1) %out
+ ret void
+}
+
+; Test 3: divergent i32 add without result used (NeedResult=false)
+define amdgpu_kernel void @add_i32_varying_nouse() {
+; GFX1032-THRESH5-LABEL: add_i32_varying_nouse:
+; GFX1032-THRESH5: ; %bb.0: ; %entry
+; GFX1032-THRESH5-NEXT: s_bcnt1_i32_b32 s0, exec_lo
+; GFX1032-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
+; GFX1032-THRESH5-NEXT: s_mov_b32 s0, -1
+; GFX1032-THRESH5-NEXT: s_cbranch_scc0 .LBB2_2
+; GFX1032-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v3, 0
+; GFX1032-THRESH5-NEXT: s_mov_b32 s0, 0
+; GFX1032-THRESH5-NEXT: ds_add_u32 v3, v0
+; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-THRESH5-NEXT: buffer_gl0_inv
+; GFX1032-THRESH5-NEXT: .LBB2_2: ; %Flow1
+; GFX1032-THRESH5-NEXT: s_andn2_b32 vcc_lo, exec_lo, s0
+; GFX1032-THRESH5-NEXT: s_cbranch_vccnz .LBB2_5
+; GFX1032-THRESH5-NEXT: ; %bb.3:
+; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-THRESH5-NEXT: v_permlanex16_b32 v2, v1, 0, 0
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_e32 v1, v1, v2
+; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, 0
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v3, v1
+; GFX1032-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
+; GFX1032-THRESH5-NEXT: s_and_saveexec_b32 s0, vcc_lo
+; GFX1032-THRESH5-NEXT: s_cbranch_execz .LBB2_5
+; GFX1032-THRESH5-NEXT: ; %bb.4:
+; GFX1032-THRESH5-NEXT: s_waitcnt_vscnt null, 0x0
+; GFX1032-THRESH5-NEXT: ds_add_u32 v0, v3
+; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-THRESH5-NEXT: buffer_gl0_inv
+; GFX1032-THRESH5-NEXT: .LBB2_5:
+; GFX1032-THRESH5-NEXT: s_endpgm
+;
+; GFX1064-THRESH5-LABEL: add_i32_varying_nouse:
+; GFX1064-THRESH5: ; %bb.0: ; %entry
+; GFX1064-THRESH5-NEXT: s_bcnt1_i32_b64 s0, exec
+; GFX1064-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
+; GFX1064-THRESH5-NEXT: s_mov_b64 s[0:1], -1
+; GFX1064-THRESH5-NEXT: s_cbranch_scc0 .LBB2_2
+; GFX1064-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v3, 0
+; GFX1064-THRESH5-NEXT: s_mov_b64 s[0:1], 0
+; GFX1064-THRESH5-NEXT: ds_add_u32 v3, v0
+; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-THRESH5-NEXT: buffer_gl0_inv
+; GFX1064-THRESH5-NEXT: .LBB2_2: ; %Flow1
+; GFX1064-THRESH5-NEXT: s_andn2_b64 vcc, exec, s[0:1]
+; GFX1064-THRESH5-NEXT: s_cbranch_vccnz .LBB2_5
+; GFX1064-THRESH5-NEXT: ; %bb.3:
+; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-THRESH5-NEXT: v_permlanex16_b32 v2, v1, 0, 0
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_e32 v1, v1, v2
+; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 0
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s3, v1, 32
+; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v3, exec_hi, v0
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, 0
+; GFX1064-THRESH5-NEXT: s_add_i32 s0, s2, s3
+; GFX1064-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX1064-THRESH5-NEXT: s_and_saveexec_b64 s[2:3], vcc
+; GFX1064-THRESH5-NEXT: s_cbranch_execz .LBB2_5
+; GFX1064-THRESH5-NEXT: ; %bb.4:
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v3, s0
+; GFX1064-THRESH5-NEXT: s_waitcnt_vscnt null, 0x0
+; GFX1064-THRESH5-NEXT: ds_add_u32 v0, v3
+; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-THRESH5-NEXT: buffer_gl0_inv
+; GFX1064-THRESH5-NEXT: .LBB2_5:
+; GFX1064-THRESH5-NEXT: s_endpgm
+;
+; GFX900-THRESH5-LABEL: add_i32_varying_nouse:
+; GFX900-THRESH5: ; %bb.0: ; %entry
+; GFX900-THRESH5-NEXT: s_bcnt1_i32_b64 s0, exec
+; GFX900-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
+; GFX900-THRESH5-NEXT: s_mov_b64 s[0:1], -1
+; GFX900-THRESH5-NEXT: s_cbranch_scc0 .LBB2_2
+; GFX900-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v2, 0
+; GFX900-THRESH5-NEXT: ds_add_u32 v2, v0
+; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-THRESH5-NEXT: s_mov_b64 s[0:1], 0
+; GFX900-THRESH5-NEXT: .LBB2_2: ; %Flow1
+; GFX900-THRESH5-NEXT: s_andn2_b64 vcc, exec, s[0:1]
+; GFX900-THRESH5-NEXT: s_cbranch_vccnz .LBB2_5
+; GFX900-THRESH5-NEXT: ; %bb.3:
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v2, 0
+; GFX900-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v3, exec_lo, 0
+; GFX900-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v3, exec_hi, v3
+; GFX900-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX900-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
+; GFX900-THRESH5-NEXT: v_readlane_b32 s2, v1, 63
+; GFX900-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
+; GFX900-THRESH5-NEXT: s_mov_b32 s0, s2
+; GFX900-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX900-THRESH5-NEXT: s_and_saveexec_b64 s[2:3], vcc
+; GFX900-THRESH5-NEXT: s_cbranch_execz .LBB2_5
+; GFX900-THRESH5-NEXT: ; %bb.4:
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, s0
+; GFX900-THRESH5-NEXT: ds_add_u32 v2, v0
+; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-THRESH5-NEXT: .LBB2_5:
+; GFX900-THRESH5-NEXT: s_endpgm
+;
+; GFX1032-DISABLED-LABEL: add_i32_varying_nouse:
+; GFX1032-DISABLED: ; %bb.0: ; %entry
+; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-DISABLED-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-DISABLED-NEXT: v_permlanex16_b32 v2, v1, 0, 0
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_e32 v1, v1, v2
+; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, 0
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v3, v1
+; GFX1032-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
+; GFX1032-DISABLED-NEXT: s_and_saveexec_b32 s0, vcc_lo
+; GFX1032-DISABLED-NEXT: s_cbranch_execz .LBB2_2
+; GFX1032-DISABLED-NEXT: ; %bb.1:
+; GFX1032-DISABLED-NEXT: ds_add_u32 v0, v3
+; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-DISABLED-NEXT: buffer_gl0_inv
+; GFX1032-DISABLED-NEXT: .LBB2_2:
+; GFX1032-DISABLED-NEXT: s_endpgm
+;
+; GFX1064-DISABLED-LABEL: add_i32_varying_nouse:
+; GFX1064-DISABLED: ; %bb.0: ; %entry
+; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-DISABLED-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-DISABLED-NEXT: v_permlanex16_b32 v2, v1, 0, 0
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_e32 v1, v1, v2
+; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 0
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s3, v1, 32
+; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-DISABLED-NEXT: v_mbcnt_hi_u32_b32 v3, exec_hi, v0
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, 0
+; GFX1064-DISABLED-NEXT: s_add_i32 s0, s2, s3
+; GFX1064-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX1064-DISABLED-NEXT: s_and_saveexec_b64 s[2:3], vcc
+; GFX1064-DISABLED-NEXT: s_cbranch_execz .LBB2_2
+; GFX1064-DISABLED-NEXT: ; %bb.1:
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v3, s0
+; GFX1064-DISABLED-NEXT: ds_add_u32 v0, v3
+; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-DISABLED-NEXT: buffer_gl0_inv
+; GFX1064-DISABLED-NEXT: .LBB2_2:
+; GFX1064-DISABLED-NEXT: s_endpgm
+entry:
+ %lane = call i32 @llvm.amdgcn.workitem.id.x()
+ %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel
+ ret void
+}
+
+; Test 4: uniform i32 add -- should NOT be affected by threshold (not divergent)
+define amdgpu_kernel void @add_i32_uniform(ptr addrspace(1) %out, i32 %additive) {
+; GFX1032-THRESH5-LABEL: add_i32_uniform:
+; GFX1032-THRESH5: ; %bb.0: ; %entry
+; GFX1032-THRESH5-NEXT: s_load_dword s0, s[4:5], 0x2c
+; GFX1032-THRESH5-NEXT: s_mov_b32 s2, exec_lo
+; GFX1032-THRESH5-NEXT: ; implicit-def: $vgpr1
+; GFX1032-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, s2, 0
+; GFX1032-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
+; GFX1032-THRESH5-NEXT: s_and_saveexec_b32 s1, vcc_lo
+; GFX1032-THRESH5-NEXT: s_cbranch_execz .LBB3_2
+; GFX1032-THRESH5-NEXT: ; %bb.1:
+; GFX1032-THRESH5-NEXT: s_bcnt1_i32_b32 s2, s2
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v1, 0
+; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-THRESH5-NEXT: s_mul_i32 s2, s0, s2
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v2, s2
+; GFX1032-THRESH5-NEXT: ds_add_rtn_u32 v1, v1, v2
+; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-THRESH5-NEXT: buffer_gl0_inv
+; GFX1032-THRESH5-NEXT: .LBB3_2:
+; GFX1032-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-THRESH5-NEXT: s_or_b32 exec_lo, exec_lo, s1
+; GFX1032-THRESH5-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x24
+; GFX1032-THRESH5-NEXT: v_readfirstlane_b32 s2, v1
+; GFX1032-THRESH5-NEXT: s_mov_b32 s11, 0x31016000
+; GFX1032-THRESH5-NEXT: s_mov_b32 s10, -1
+; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-THRESH5-NEXT: v_mad_u64_u32 v[0:1], s0, s0, v0, s[2:3]
+; GFX1032-THRESH5-NEXT: buffer_store_dword v0, off, s[8:11], 0
+; GFX1032-THRESH5-NEXT: s_endpgm
+;
+; GFX1064-THRESH5-LABEL: add_i32_uniform:
+; GFX1064-THRESH5: ; %bb.0: ; %entry
+; GFX1064-THRESH5-NEXT: s_load_dword s6, s[4:5], 0x2c
+; GFX1064-THRESH5-NEXT: s_mov_b64 s[2:3], exec
+; GFX1064-THRESH5-NEXT: ; implicit-def: $vgpr1
+; GFX1064-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, s2, 0
+; GFX1064-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v0, s3, v0
+; GFX1064-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX1064-THRESH5-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1064-THRESH5-NEXT: s_cbranch_execz .LBB3_2
+; GFX1064-THRESH5-NEXT: ; %bb.1:
+; GFX1064-THRESH5-NEXT: s_bcnt1_i32_b64 s2, s[2:3]
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v1, 0
+; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-THRESH5-NEXT: s_mul_i32 s2, s6, s2
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v2, s2
+; GFX1064-THRESH5-NEXT: ds_add_rtn_u32 v1, v1, v2
+; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-THRESH5-NEXT: buffer_gl0_inv
+; GFX1064-THRESH5-NEXT: .LBB3_2:
+; GFX1064-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-THRESH5-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1064-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-THRESH5-NEXT: v_readfirstlane_b32 s2, v1
+; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-THRESH5-NEXT: v_mad_u64_u32 v[0:1], s[2:3], s6, v0, s[2:3]
+; GFX1064-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-THRESH5-NEXT: s_mov_b32 s2, -1
+; GFX1064-THRESH5-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1064-THRESH5-NEXT: s_endpgm
+;
+; GFX900-THRESH5-LABEL: add_i32_uniform:
+; GFX900-THRESH5: ; %bb.0: ; %entry
+; GFX900-THRESH5-NEXT: s_load_dword s6, s[4:5], 0x2c
+; GFX900-THRESH5-NEXT: s_mov_b64 s[2:3], exec
+; GFX900-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, s2, 0
+; GFX900-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v0, s3, v0
+; GFX900-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX900-THRESH5-NEXT: ; implicit-def: $vgpr1
+; GFX900-THRESH5-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX900-THRESH5-NEXT: s_cbranch_execz .LBB3_2
+; GFX900-THRESH5-NEXT: ; %bb.1:
+; GFX900-THRESH5-NEXT: s_bcnt1_i32_b64 s2, s[2:3]
+; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-THRESH5-NEXT: s_mul_i32 s2, s6, s2
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v1, 0
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v2, s2
+; GFX900-THRESH5-NEXT: ds_add_rtn_u32 v1, v1, v2
+; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-THRESH5-NEXT: .LBB3_2:
+; GFX900-THRESH5-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX900-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-THRESH5-NEXT: v_mul_lo_u32 v0, s6, v0
+; GFX900-THRESH5-NEXT: v_readfirstlane_b32 s4, v1
+; GFX900-THRESH5-NEXT: s_mov_b32 s3, 0xf000
+; GFX900-THRESH5-NEXT: s_mov_b32 s2, -1
+; GFX900-THRESH5-NEXT: v_add_u32_e32 v0, s4, v0
+; GFX900-THRESH5-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX900-THRESH5-NEXT: s_endpgm
+;
+; GFX1032-DISABLED-LABEL: add_i32_uniform:
+; GFX1032-DISABLED: ; %bb.0: ; %entry
+; GFX1032-DISABLED-NEXT: s_load_dword s0, s[4:5], 0x2c
+; GFX1032-DISABLED-NEXT: s_mov_b32 s2, exec_lo
+; GFX1032-DISABLED-NEXT: ; implicit-def: $vgpr1
+; GFX1032-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, s2, 0
+; GFX1032-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
+; GFX1032-DISABLED-NEXT: s_and_saveexec_b32 s1, vcc_lo
+; GFX1032-DISABLED-NEXT: s_cbranch_execz .LBB3_2
+; GFX1032-DISABLED-NEXT: ; %bb.1:
+; GFX1032-DISABLED-NEXT: s_bcnt1_i32_b32 s2, s2
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v1, 0
+; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-DISABLED-NEXT: s_mul_i32 s2, s0, s2
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v2, s2
+; GFX1032-DISABLED-NEXT: ds_add_rtn_u32 v1, v1, v2
+; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-DISABLED-NEXT: buffer_gl0_inv
+; GFX1032-DISABLED-NEXT: .LBB3_2:
+; GFX1032-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-DISABLED-NEXT: s_or_b32 exec_lo, exec_lo, s1
+; GFX1032-DISABLED-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x24
+; GFX1032-DISABLED-NEXT: v_readfirstlane_b32 s2, v1
+; GFX1032-DISABLED-NEXT: s_mov_b32 s11, 0x31016000
+; GFX1032-DISABLED-NEXT: s_mov_b32 s10, -1
+; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-DISABLED-NEXT: v_mad_u64_u32 v[0:1], s0, s0, v0, s[2:3]
+; GFX1032-DISABLED-NEXT: buffer_store_dword v0, off, s[8:11], 0
+; GFX1032-DISABLED-NEXT: s_endpgm
+;
+; GFX1064-DISABLED-LABEL: add_i32_uniform:
+; GFX1064-DISABLED: ; %bb.0: ; %entry
+; GFX1064-DISABLED-NEXT: s_load_dword s6, s[4:5], 0x2c
+; GFX1064-DISABLED-NEXT: s_mov_b64 s[2:3], exec
+; GFX1064-DISABLED-NEXT: ; implicit-def: $vgpr1
+; GFX1064-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, s2, 0
+; GFX1064-DISABLED-NEXT: v_mbcnt_hi_u32_b32 v0, s3, v0
+; GFX1064-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX1064-DISABLED-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1064-DISABLED-NEXT: s_cbranch_execz .LBB3_2
+; GFX1064-DISABLED-NEXT: ; %bb.1:
+; GFX1064-DISABLED-NEXT: s_bcnt1_i32_b64 s2, s[2:3]
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v1, 0
+; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-DISABLED-NEXT: s_mul_i32 s2, s6, s2
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v2, s2
+; GFX1064-DISABLED-NEXT: ds_add_rtn_u32 v1, v1, v2
+; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-DISABLED-NEXT: buffer_gl0_inv
+; GFX1064-DISABLED-NEXT: .LBB3_2:
+; GFX1064-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-DISABLED-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1064-DISABLED-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-DISABLED-NEXT: v_readfirstlane_b32 s2, v1
+; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-DISABLED-NEXT: v_mad_u64_u32 v[0:1], s[2:3], s6, v0, s[2:3]
+; GFX1064-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-DISABLED-NEXT: s_mov_b32 s2, -1
+; GFX1064-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1064-DISABLED-NEXT: s_endpgm
+entry:
+ %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %additive acq_rel
+ store i32 %old, ptr addrspace(1) %out
+ ret void
+}
+
+; Test 5: divergent float fadd -- should NOT be affected by threshold (FP excluded)
+define amdgpu_kernel void @fadd_f32_varying(ptr addrspace(1) %out) {
+; GFX1032-THRESH5-LABEL: fadd_f32_varying:
+; GFX1032-THRESH5: ; %bb.0: ; %entry
+; GFX1032-THRESH5-NEXT: v_cvt_f32_i32_e32 v0, v0
+; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-THRESH5-NEXT: v_bfrev_b32_e32 v1, 1
+; GFX1032-THRESH5-NEXT: v_cndmask_b32_e64 v2, 0x80000000, v0, s0
+; GFX1032-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-THRESH5-NEXT: v_add_f32_e32 v1, v2, v1
+; GFX1032-THRESH5-NEXT: v_bfrev_b32_e32 v2, 1
+; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:2 row_mask:0xf bank_mask:0xf
+; GFX1032-THRESH5-NEXT: v_add_f32_e32 v1, v1, v3
+; GFX1032-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v2, v1 row_shr:4 row_mask:0xf bank_mask:0xf
+; GFX1032-THRESH5-NEXT: v_add_f32_e32 v1, v1, v2
+; GFX1032-THRESH5-NEXT: v_bfrev_b32_e32 v2, 1
+; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:8 row_mask:0xf bank_mask:0xf
+; GFX1032-THRESH5-NEXT: v_add_f32_e32 v1, v1, v3
+; GFX1032-THRESH5-NEXT: v_permlanex16_b32 v3, v1, -1, -1
+; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1032-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1032-THRESH5-NEXT: v_add_f32_e32 v1, v1, v2
+; GFX1032-THRESH5-NEXT: v_readlane_b32 s1, v1, 15
+; GFX1032-THRESH5-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-THRESH5-NEXT: v_writelane_b32 v3, s1, 16
+; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
+; GFX1032-THRESH5-NEXT: s_mov_b32 s0, s2
+; GFX1032-THRESH5-NEXT: s_mov_b32 s2, -1
+; GFX1032-THRESH5-NEXT: ; implicit-def: $vgpr0
+; GFX1032-THRESH5-NEXT: s_and_saveexec_b32 s1, vcc_lo
+; GFX1032-THRESH5-NEXT: s_cbranch_execz .LBB4_2
+; GFX1032-THRESH5-NEXT: ; %bb.1:
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, 0
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v4, s0
+; GFX1032-THRESH5-NEXT: ds_add_rtn_f32 v0, v0, v4
+; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-THRESH5-NEXT: buffer_gl0_inv
+; GFX1032-THRESH5-NEXT: .LBB4_2:
+; GFX1032-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-THRESH5-NEXT: s_or_b32 exec_lo, exec_lo, s1
+; GFX1032-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1032-THRESH5-NEXT: v_readfirstlane_b32 s3, v0
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, v3
+; GFX1032-THRESH5-NEXT: v_add_f32_e32 v0, s3, v0
+; GFX1032-THRESH5-NEXT: v_cndmask_b32_e64 v0, v0, s3, vcc_lo
+; GFX1032-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-THRESH5-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1032-THRESH5-NEXT: s_endpgm
+;
+; GFX1064-THRESH5-LABEL: fadd_f32_varying:
+; GFX1064-THRESH5: ; %bb.0: ; %entry
+; GFX1064-THRESH5-NEXT: v_cvt_f32_i32_e32 v0, v0
+; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-THRESH5-NEXT: v_bfrev_b32_e32 v1, 1
+; GFX1064-THRESH5-NEXT: v_cndmask_b32_e64 v2, 0x80000000, v0, s[0:1]
+; GFX1064-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-THRESH5-NEXT: v_add_f32_e32 v1, v2, v1
+; GFX1064-THRESH5-NEXT: v_bfrev_b32_e32 v2, 1
+; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:2 row_mask:0xf bank_mask:0xf
+; GFX1064-THRESH5-NEXT: v_add_f32_e32 v1, v1, v3
+; GFX1064-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v2, v1 row_shr:4 row_mask:0xf bank_mask:0xf
+; GFX1064-THRESH5-NEXT: v_add_f32_e32 v1, v1, v2
+; GFX1064-THRESH5-NEXT: v_bfrev_b32_e32 v2, 1
+; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:8 row_mask:0xf bank_mask:0xf
+; GFX1064-THRESH5-NEXT: v_add_f32_e32 v1, v1, v3
+; GFX1064-THRESH5-NEXT: v_permlanex16_b32 v3, v1, -1, -1
+; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1064-THRESH5-NEXT: v_add_f32_e32 v1, v1, v2
+; GFX1064-THRESH5-NEXT: v_bfrev_b32_e32 v2, 1
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v3, s2
+; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1064-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1064-THRESH5-NEXT: v_add_f32_e32 v1, v1, v2
+; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 15
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s3, v1, 31
+; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s2, 16
+; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 47
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s6, v1, 63
+; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s3, 32
+; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v0, exec_hi, v0
+; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s2, 48
+; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX1064-THRESH5-NEXT: s_mov_b32 s2, -1
+; GFX1064-THRESH5-NEXT: ; implicit-def: $vgpr0
+; GFX1064-THRESH5-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1064-THRESH5-NEXT: s_cbranch_execz .LBB4_2
+; GFX1064-THRESH5-NEXT: ; %bb.1:
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, 0
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v4, s6
+; GFX1064-THRESH5-NEXT: ds_add_rtn_f32 v0, v0, v4
+; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-THRESH5-NEXT: buffer_gl0_inv
+; GFX1064-THRESH5-NEXT: .LBB4_2:
+; GFX1064-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-THRESH5-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1064-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-THRESH5-NEXT: v_readfirstlane_b32 s3, v0
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, v3
+; GFX1064-THRESH5-NEXT: v_add_f32_e32 v0, s3, v0
+; GFX1064-THRESH5-NEXT: v_cndmask_b32_e64 v0, v0, s3, vcc
+; GFX1064-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-THRESH5-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1064-THRESH5-NEXT: s_endpgm
+;
+; GFX900-THRESH5-LABEL: fadd_f32_varying:
+; GFX900-THRESH5: ; %bb.0: ; %entry
+; GFX900-THRESH5-NEXT: v_cvt_f32_i32_e32 v0, v0
+; GFX900-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX900-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v4
+; GFX900-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX900-THRESH5-NEXT: v_bfrev_b32_e32 v1, 1
+; GFX900-THRESH5-NEXT: v_cndmask_b32_e64 v2, v1, v0, s[0:1]
+; GFX900-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_mov_b32_dpp v3, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX900-THRESH5-NEXT: v_add_f32_e32 v2, v2, v3
+; GFX900-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_mov_b32_dpp v3, v2 row_shr:2 row_mask:0xf bank_mask:0xf
+; GFX900-THRESH5-NEXT: v_add_f32_e32 v2, v2, v3
+; GFX900-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_mov_b32_dpp v3, v2 row_shr:4 row_mask:0xf bank_mask:0xf
+; GFX900-THRESH5-NEXT: v_add_f32_e32 v2, v2, v3
+; GFX900-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_mov_b32_dpp v3, v2 row_shr:8 row_mask:0xf bank_mask:0xf
+; GFX900-THRESH5-NEXT: v_add_f32_e32 v2, v2, v3
+; GFX900-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_mov_b32_dpp v3, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
+; GFX900-THRESH5-NEXT: v_add_f32_e32 v2, v2, v3
+; GFX900-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_mov_b32_dpp v3, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
+; GFX900-THRESH5-NEXT: v_add_f32_e32 v2, v2, v3
+; GFX900-THRESH5-NEXT: v_readlane_b32 s2, v2, 63
+; GFX900-THRESH5-NEXT: s_nop 0
+; GFX900-THRESH5-NEXT: v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
+; GFX900-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
+; GFX900-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX900-THRESH5-NEXT: ; implicit-def: $vgpr0
+; GFX900-THRESH5-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX900-THRESH5-NEXT: s_cbranch_execz .LBB4_2
+; GFX900-THRESH5-NEXT: ; %bb.1:
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, 0
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v4, s2
+; GFX900-THRESH5-NEXT: ds_add_rtn_f32 v0, v0, v4
+; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-THRESH5-NEXT: .LBB4_2:
+; GFX900-THRESH5-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX900-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX900-THRESH5-NEXT: v_readfirstlane_b32 s4, v0
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, v1
+; GFX900-THRESH5-NEXT: v_add_f32_e32 v0, s4, v0
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v4, s4
+; GFX900-THRESH5-NEXT: s_mov_b32 s3, 0xf000
+; GFX900-THRESH5-NEXT: s_mov_b32 s2, -1
+; GFX900-THRESH5-NEXT: v_cndmask_b32_e32 v0, v0, v4, vcc
+; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-THRESH5-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX900-THRESH5-NEXT: s_endpgm
+;
+; GFX1032-DISABLED-LABEL: fadd_f32_varying:
+; GFX1032-DISABLED: ; %bb.0: ; %entry
+; GFX1032-DISABLED-NEXT: v_cvt_f32_i32_e32 v0, v0
+; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-DISABLED-NEXT: v_bfrev_b32_e32 v1, 1
+; GFX1032-DISABLED-NEXT: v_cndmask_b32_e64 v2, 0x80000000, v0, s0
+; GFX1032-DISABLED-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-DISABLED-NEXT: v_add_f32_e32 v1, v2, v1
+; GFX1032-DISABLED-NEXT: v_bfrev_b32_e32 v2, 1
+; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:2 row_mask:0xf bank_mask:0xf
+; GFX1032-DISABLED-NEXT: v_add_f32_e32 v1, v1, v3
+; GFX1032-DISABLED-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v2, v1 row_shr:4 row_mask:0xf bank_mask:0xf
+; GFX1032-DISABLED-NEXT: v_add_f32_e32 v1, v1, v2
+; GFX1032-DISABLED-NEXT: v_bfrev_b32_e32 v2, 1
+; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:8 row_mask:0xf bank_mask:0xf
+; GFX1032-DISABLED-NEXT: v_add_f32_e32 v1, v1, v3
+; GFX1032-DISABLED-NEXT: v_permlanex16_b32 v3, v1, -1, -1
+; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1032-DISABLED-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1032-DISABLED-NEXT: v_add_f32_e32 v1, v1, v2
+; GFX1032-DISABLED-NEXT: v_readlane_b32 s1, v1, 15
+; GFX1032-DISABLED-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-DISABLED-NEXT: v_writelane_b32 v3, s1, 16
+; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
+; GFX1032-DISABLED-NEXT: s_mov_b32 s0, s2
+; GFX1032-DISABLED-NEXT: s_mov_b32 s2, -1
+; GFX1032-DISABLED-NEXT: ; implicit-def: $vgpr0
+; GFX1032-DISABLED-NEXT: s_and_saveexec_b32 s1, vcc_lo
+; GFX1032-DISABLED-NEXT: s_cbranch_execz .LBB4_2
+; GFX1032-DISABLED-NEXT: ; %bb.1:
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, 0
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v4, s0
+; GFX1032-DISABLED-NEXT: ds_add_rtn_f32 v0, v0, v4
+; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-DISABLED-NEXT: buffer_gl0_inv
+; GFX1032-DISABLED-NEXT: .LBB4_2:
+; GFX1032-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-DISABLED-NEXT: s_or_b32 exec_lo, exec_lo, s1
+; GFX1032-DISABLED-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1032-DISABLED-NEXT: v_readfirstlane_b32 s3, v0
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, v3
+; GFX1032-DISABLED-NEXT: v_add_f32_e32 v0, s3, v0
+; GFX1032-DISABLED-NEXT: v_cndmask_b32_e64 v0, v0, s3, vcc_lo
+; GFX1032-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1032-DISABLED-NEXT: s_endpgm
+;
+; GFX1064-DISABLED-LABEL: fadd_f32_varying:
+; GFX1064-DISABLED: ; %bb.0: ; %entry
+; GFX1064-DISABLED-NEXT: v_cvt_f32_i32_e32 v0, v0
+; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-DISABLED-NEXT: v_bfrev_b32_e32 v1, 1
+; GFX1064-DISABLED-NEXT: v_cndmask_b32_e64 v2, 0x80000000, v0, s[0:1]
+; GFX1064-DISABLED-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-DISABLED-NEXT: v_add_f32_e32 v1, v2, v1
+; GFX1064-DISABLED-NEXT: v_bfrev_b32_e32 v2, 1
+; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:2 row_mask:0xf bank_mask:0xf
+; GFX1064-DISABLED-NEXT: v_add_f32_e32 v1, v1, v3
+; GFX1064-DISABLED-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v2, v1 row_shr:4 row_mask:0xf bank_mask:0xf
+; GFX1064-DISABLED-NEXT: v_add_f32_e32 v1, v1, v2
+; GFX1064-DISABLED-NEXT: v_bfrev_b32_e32 v2, 1
+; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:8 row_mask:0xf bank_mask:0xf
+; GFX1064-DISABLED-NEXT: v_add_f32_e32 v1, v1, v3
+; GFX1064-DISABLED-NEXT: v_permlanex16_b32 v3, v1, -1, -1
+; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1064-DISABLED-NEXT: v_add_f32_e32 v1, v1, v2
+; GFX1064-DISABLED-NEXT: v_bfrev_b32_e32 v2, 1
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v3, s2
+; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1064-DISABLED-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1064-DISABLED-NEXT: v_add_f32_e32 v1, v1, v2
+; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 15
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s3, v1, 31
+; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s2, 16
+; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 47
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s6, v1, 63
+; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s3, 32
+; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-DISABLED-NEXT: v_mbcnt_hi_u32_b32 v0, exec_hi, v0
+; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s2, 48
+; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX1064-DISABLED-NEXT: s_mov_b32 s2, -1
+; GFX1064-DISABLED-NEXT: ; implicit-def: $vgpr0
+; GFX1064-DISABLED-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1064-DISABLED-NEXT: s_cbranch_execz .LBB4_2
+; GFX1064-DISABLED-NEXT: ; %bb.1:
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, 0
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v4, s6
+; GFX1064-DISABLED-NEXT: ds_add_rtn_f32 v0, v0, v4
+; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-DISABLED-NEXT: buffer_gl0_inv
+; GFX1064-DISABLED-NEXT: .LBB4_2:
+; GFX1064-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-DISABLED-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1064-DISABLED-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-DISABLED-NEXT: v_readfirstlane_b32 s3, v0
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, v3
+; GFX1064-DISABLED-NEXT: v_add_f32_e32 v0, s3, v0
+; GFX1064-DISABLED-NEXT: v_cndmask_b32_e64 v0, v0, s3, vcc
+; GFX1064-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1064-DISABLED-NEXT: s_endpgm
+entry:
+ %lane = call i32 @llvm.amdgcn.workitem.id.x()
+ %conv = sitofp i32 %lane to float
+ %rmw = atomicrmw fadd ptr addrspace(3) @local_var32, float %conv syncscope("agent") acq_rel
+ store float %rmw, ptr addrspace(1) %out
+ ret void
+}
+
+; Test 6: divergent i32 add on GLOBAL memory -- should NOT be affected by threshold
+define amdgpu_kernel void @add_i32_varying_global(ptr addrspace(1) %out, ptr addrspace(1) %inout) {
+; GFX1032-THRESH5-LABEL: add_i32_varying_global:
+; GFX1032-THRESH5: ; %bb.0: ; %entry
+; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-THRESH5-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-THRESH5-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
+; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s4, -1
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v3, 0
+; GFX1032-THRESH5-NEXT: v_readlane_b32 s6, v1, 31
+; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-THRESH5-NEXT: v_readlane_b32 s5, v1, 15
+; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s4
+; GFX1032-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s4, -1
+; GFX1032-THRESH5-NEXT: v_writelane_b32 v3, s5, 16
+; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s4
+; GFX1032-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
+; GFX1032-THRESH5-NEXT: s_mov_b32 s4, s6
+; GFX1032-THRESH5-NEXT: s_mov_b32 s6, -1
+; GFX1032-THRESH5-NEXT: ; implicit-def: $vgpr0
+; GFX1032-THRESH5-NEXT: s_and_saveexec_b32 s8, vcc_lo
+; GFX1032-THRESH5-NEXT: s_cbranch_execz .LBB5_2
+; GFX1032-THRESH5-NEXT: ; %bb.1:
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, s4
+; GFX1032-THRESH5-NEXT: s_mov_b32 s7, 0x31016000
+; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-THRESH5-NEXT: s_mov_b32 s4, s2
+; GFX1032-THRESH5-NEXT: s_mov_b32 s5, s3
+; GFX1032-THRESH5-NEXT: buffer_atomic_add v0, off, s[4:7], 0 glc
+; GFX1032-THRESH5-NEXT: s_waitcnt vmcnt(0)
+; GFX1032-THRESH5-NEXT: buffer_gl1_inv
+; GFX1032-THRESH5-NEXT: buffer_gl0_inv
+; GFX1032-THRESH5-NEXT: .LBB5_2:
+; GFX1032-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-THRESH5-NEXT: s_or_b32 exec_lo, exec_lo, s8
+; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-THRESH5-NEXT: v_readfirstlane_b32 s2, v0
+; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, v3
+; GFX1032-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1032-THRESH5-NEXT: v_add_nc_u32_e32 v0, s2, v0
+; GFX1032-THRESH5-NEXT: s_mov_b32 s2, s6
+; GFX1032-THRESH5-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1032-THRESH5-NEXT: s_endpgm
+;
+; GFX1064-THRESH5-LABEL: add_i32_varying_global:
+; GFX1064-THRESH5: ; %bb.0: ; %entry
+; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v3, 0
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-THRESH5-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v2, s2
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s6, v1, 15
+; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-THRESH5-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
+; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[4:5], -1
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s7, v1, 31
+; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s6, 16
+; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[4:5]
+; GFX1064-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[4:5], -1
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s8, v1, 47
+; GFX1064-THRESH5-NEXT: v_readlane_b32 s9, v1, 63
+; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s7, 32
+; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[4:5]
+; GFX1064-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v0, exec_hi, v0
+; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[6:7], -1
+; GFX1064-THRESH5-NEXT: s_mov_b32 s4, s9
+; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s8, 48
+; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[6:7]
+; GFX1064-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX1064-THRESH5-NEXT: s_mov_b32 s6, -1
+; GFX1064-THRESH5-NEXT: ; implicit-def: $vgpr0
+; GFX1064-THRESH5-NEXT: s_and_saveexec_b64 s[8:9], vcc
+; GFX1064-THRESH5-NEXT: s_cbranch_execz .LBB5_2
+; GFX1064-THRESH5-NEXT: ; %bb.1:
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, s4
+; GFX1064-THRESH5-NEXT: s_mov_b32 s7, 0x31016000
+; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-THRESH5-NEXT: s_mov_b32 s4, s2
+; GFX1064-THRESH5-NEXT: s_mov_b32 s5, s3
+; GFX1064-THRESH5-NEXT: buffer_atomic_add v0, off, s[4:7], 0 glc
+; GFX1064-THRESH5-NEXT: s_waitcnt vmcnt(0)
+; GFX1064-THRESH5-NEXT: buffer_gl1_inv
+; GFX1064-THRESH5-NEXT: buffer_gl0_inv
+; GFX1064-THRESH5-NEXT: .LBB5_2:
+; GFX1064-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-THRESH5-NEXT: s_or_b64 exec, exec, s[8:9]
+; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-THRESH5-NEXT: v_readfirstlane_b32 s2, v0
+; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, v3
+; GFX1064-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-THRESH5-NEXT: v_add_nc_u32_e32 v0, s2, v0
+; GFX1064-THRESH5-NEXT: s_mov_b32 s2, s6
+; GFX1064-THRESH5-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1064-THRESH5-NEXT: s_endpgm
+;
+; GFX900-THRESH5-LABEL: add_i32_varying_global:
+; GFX900-THRESH5: ; %bb.0: ; %entry
+; GFX900-THRESH5-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
+; GFX900-THRESH5-NEXT: s_or_saveexec_b64 s[4:5], -1
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v1, 0
+; GFX900-THRESH5-NEXT: s_mov_b64 exec, s[4:5]
+; GFX900-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v3, exec_lo, 0
+; GFX900-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v3, exec_hi, v3
+; GFX900-THRESH5-NEXT: s_or_saveexec_b64 s[4:5], -1
+; GFX900-THRESH5-NEXT: v_cndmask_b32_e64 v2, 0, v0, s[4:5]
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v2, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v2, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v2, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
+; GFX900-THRESH5-NEXT: s_nop 1
+; GFX900-THRESH5-NEXT: v_add_u32_dpp v2, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
+; GFX900-THRESH5-NEXT: v_readlane_b32 s6, v2, 63
+; GFX900-THRESH5-NEXT: s_nop 0
+; GFX900-THRESH5-NEXT: v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
+; GFX900-THRESH5-NEXT: s_mov_b64 exec, s[4:5]
+; GFX900-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX900-THRESH5-NEXT: ; implicit-def: $vgpr0
+; GFX900-THRESH5-NEXT: s_and_saveexec_b64 s[4:5], vcc
+; GFX900-THRESH5-NEXT: s_cbranch_execz .LBB5_2
+; GFX900-THRESH5-NEXT: ; %bb.1:
+; GFX900-THRESH5-NEXT: s_mov_b32 s11, 0xf000
+; GFX900-THRESH5-NEXT: s_mov_b32 s10, -1
+; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-THRESH5-NEXT: s_mov_b32 s8, s2
+; GFX900-THRESH5-NEXT: s_mov_b32 s9, s3
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, s6
+; GFX900-THRESH5-NEXT: buffer_atomic_add v0, off, s[8:11], 0 glc
+; GFX900-THRESH5-NEXT: s_waitcnt vmcnt(0)
+; GFX900-THRESH5-NEXT: buffer_wbinvl1_vol
+; GFX900-THRESH5-NEXT: .LBB5_2:
+; GFX900-THRESH5-NEXT: s_or_b64 exec, exec, s[4:5]
+; GFX900-THRESH5-NEXT: v_readfirstlane_b32 s4, v0
+; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, v1
+; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-THRESH5-NEXT: s_mov_b32 s3, 0xf000
+; GFX900-THRESH5-NEXT: s_mov_b32 s2, -1
+; GFX900-THRESH5-NEXT: v_add_u32_e32 v0, s4, v0
+; GFX900-THRESH5-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX900-THRESH5-NEXT: s_endpgm
+;
+; GFX1032-DISABLED-LABEL: add_i32_varying_global:
+; GFX1032-DISABLED: ; %bb.0: ; %entry
+; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-DISABLED-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-DISABLED-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-DISABLED-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
+; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s4, -1
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v3, 0
+; GFX1032-DISABLED-NEXT: v_readlane_b32 s6, v1, 31
+; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-DISABLED-NEXT: v_readlane_b32 s5, v1, 15
+; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s4
+; GFX1032-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s4, -1
+; GFX1032-DISABLED-NEXT: v_writelane_b32 v3, s5, 16
+; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s4
+; GFX1032-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
+; GFX1032-DISABLED-NEXT: s_mov_b32 s4, s6
+; GFX1032-DISABLED-NEXT: s_mov_b32 s6, -1
+; GFX1032-DISABLED-NEXT: ; implicit-def: $vgpr0
+; GFX1032-DISABLED-NEXT: s_and_saveexec_b32 s8, vcc_lo
+; GFX1032-DISABLED-NEXT: s_cbranch_execz .LBB5_2
+; GFX1032-DISABLED-NEXT: ; %bb.1:
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, s4
+; GFX1032-DISABLED-NEXT: s_mov_b32 s7, 0x31016000
+; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-DISABLED-NEXT: s_mov_b32 s4, s2
+; GFX1032-DISABLED-NEXT: s_mov_b32 s5, s3
+; GFX1032-DISABLED-NEXT: buffer_atomic_add v0, off, s[4:7], 0 glc
+; GFX1032-DISABLED-NEXT: s_waitcnt vmcnt(0)
+; GFX1032-DISABLED-NEXT: buffer_gl1_inv
+; GFX1032-DISABLED-NEXT: buffer_gl0_inv
+; GFX1032-DISABLED-NEXT: .LBB5_2:
+; GFX1032-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-DISABLED-NEXT: s_or_b32 exec_lo, exec_lo, s8
+; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-DISABLED-NEXT: v_readfirstlane_b32 s2, v0
+; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, v3
+; GFX1032-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1032-DISABLED-NEXT: v_add_nc_u32_e32 v0, s2, v0
+; GFX1032-DISABLED-NEXT: s_mov_b32 s2, s6
+; GFX1032-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1032-DISABLED-NEXT: s_endpgm
+;
+; GFX1064-DISABLED-LABEL: add_i32_varying_global:
+; GFX1064-DISABLED: ; %bb.0: ; %entry
+; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-DISABLED-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v3, 0
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-DISABLED-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v2, s2
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s6, v1, 15
+; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-DISABLED-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
+; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[4:5], -1
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s7, v1, 31
+; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s6, 16
+; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[4:5]
+; GFX1064-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[4:5], -1
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s8, v1, 47
+; GFX1064-DISABLED-NEXT: v_readlane_b32 s9, v1, 63
+; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s7, 32
+; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[4:5]
+; GFX1064-DISABLED-NEXT: v_mbcnt_hi_u32_b32 v0, exec_hi, v0
+; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[6:7], -1
+; GFX1064-DISABLED-NEXT: s_mov_b32 s4, s9
+; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s8, 48
+; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[6:7]
+; GFX1064-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX1064-DISABLED-NEXT: s_mov_b32 s6, -1
+; GFX1064-DISABLED-NEXT: ; implicit-def: $vgpr0
+; GFX1064-DISABLED-NEXT: s_and_saveexec_b64 s[8:9], vcc
+; GFX1064-DISABLED-NEXT: s_cbranch_execz .LBB5_2
+; GFX1064-DISABLED-NEXT: ; %bb.1:
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, s4
+; GFX1064-DISABLED-NEXT: s_mov_b32 s7, 0x31016000
+; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-DISABLED-NEXT: s_mov_b32 s4, s2
+; GFX1064-DISABLED-NEXT: s_mov_b32 s5, s3
+; GFX1064-DISABLED-NEXT: buffer_atomic_add v0, off, s[4:7], 0 glc
+; GFX1064-DISABLED-NEXT: s_waitcnt vmcnt(0)
+; GFX1064-DISABLED-NEXT: buffer_gl1_inv
+; GFX1064-DISABLED-NEXT: buffer_gl0_inv
+; GFX1064-DISABLED-NEXT: .LBB5_2:
+; GFX1064-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-DISABLED-NEXT: s_or_b64 exec, exec, s[8:9]
+; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-DISABLED-NEXT: v_readfirstlane_b32 s2, v0
+; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, v3
+; GFX1064-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-DISABLED-NEXT: v_add_nc_u32_e32 v0, s2, v0
+; GFX1064-DISABLED-NEXT: s_mov_b32 s2, s6
+; GFX1064-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1064-DISABLED-NEXT: s_endpgm
+entry:
+ %lane = call i32 @llvm.amdgcn.workitem.id.x()
+ %old = atomicrmw add ptr addrspace(1) %inout, i32 %lane acq_rel
+ store i32 %old, ptr addrspace(1) %out
+ ret void
+}
>From fe5b5529af1b8665d052a9405dce92f8008500c5 Mon Sep 17 00:00:00 2001
From: YaFan <YaFan.Tao at amd.com>
Date: Fri, 20 Mar 2026 15:03:21 +0800
Subject: [PATCH 02/17] Remove product name from comment per review feedback
---
llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index f3cde05cee673..2aea1e59d26de 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -46,8 +46,7 @@ using namespace llvm::AMDGPU;
//
// Only integer LDS atomics are affected because floating-point path is
// untested. The default is 0 (always use DPP) because unconditionally enabling
-// the threshold can regress DPP-friendly workloads by ~4% (tested on RX
-// 7900XT).
+// the threshold can regress DPP-friendly workloads by ~4%.
static cl::opt<unsigned> AMDGPUAtomicOptimizerDPPLdsThreshold(
"amdgpu-atomic-optimizer-dpp-lds-threshold",
cl::desc("Use DPP scan for integer LDS atomics only when active lanes > "
>From d6f5399675de49a6155a2dbb54ab6e4a0e63f170 Mon Sep 17 00:00:00 2001
From: YaFan <YaFan.Tao at amd.com>
Date: Mon, 23 Mar 2026 15:46:15 +0800
Subject: [PATCH 03/17] Move RUN lines to top of test file per review feedback
---
.../atomic_optimizations_dpp_lds_threshold.ll | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)
diff --git a/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll b/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
index 0ef74d7ca2c02..0d52229a34edf 100644
--- a/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
+++ b/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
@@ -1,21 +1,21 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
;
-; Test the -amdgpu-atomic-optimizer-dpp-lds-threshold option which controls
-; dynamic DPP vs no-opt branching for integer LDS atomics.
-;
-; Threshold=5: use DPP only when active lanes > 5, otherwise each lane does
-; its own atomic.
-; Threshold=32: on wave32 this disables DPP entirely (>= wavefront size).
-; Threshold=64: on wave64 this disables DPP entirely (>= wavefront size).
-
; --- Threshold=5 tests (dynamic branch expected) ---
; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP -amdgpu-atomic-optimizer-dpp-lds-threshold=5 < %s | FileCheck -enable-var-scope -check-prefixes=GFX1032-THRESH5 %s
; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP -amdgpu-atomic-optimizer-dpp-lds-threshold=5 < %s | FileCheck -enable-var-scope -check-prefixes=GFX1064-THRESH5 %s
; RUN: llc -mtriple=amdgcn -mcpu=gfx900 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP -amdgpu-atomic-optimizer-dpp-lds-threshold=5 < %s | FileCheck -enable-var-scope -check-prefixes=GFX900-THRESH5 %s
-
+;
; --- Threshold >= wavefront size tests (DPP fully disabled for int LDS) ---
; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP -amdgpu-atomic-optimizer-dpp-lds-threshold=32 < %s | FileCheck -enable-var-scope -check-prefixes=GFX1032-DISABLED %s
; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP -amdgpu-atomic-optimizer-dpp-lds-threshold=64 < %s | FileCheck -enable-var-scope -check-prefixes=GFX1064-DISABLED %s
+;
+; Test the -amdgpu-atomic-optimizer-dpp-lds-threshold option which controls
+; dynamic DPP vs no-opt branching for integer LDS atomics.
+;
+; Threshold=5: use DPP only when active lanes > 5, otherwise each lane does
+; its own atomic.
+; Threshold=32: on wave32 this disables DPP entirely (>= wavefront size).
+; Threshold=64: on wave64 this disables DPP entirely (>= wavefront size).
declare i32 @llvm.amdgcn.workitem.id.x()
>From 4c6ea98afb5b95ff6ede96f338e449cacc069344 Mon Sep 17 00:00:00 2001
From: YaFan <YaFan.Tao at amd.com>
Date: Mon, 23 Mar 2026 16:32:30 +0800
Subject: [PATCH 04/17] Support per-function attribute for DPP LDS threshold
and pass threshold as parameter
---
.../Target/AMDGPU/AMDGPUAtomicOptimizer.cpp | 23 ++++++++++++++-----
1 file changed, 17 insertions(+), 6 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index 2aea1e59d26de..556c0c1a99fe4 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -118,7 +118,8 @@ class AMDGPUAtomicOptimizerImpl
Value *optimizeAtomicWithDynamicThreshold(IRBuilder<> &B, Instruction &I,
AtomicRMWInst::BinOp Op,
- unsigned ValIdx) const;
+ unsigned ValIdx,
+ unsigned Threshold) const;
public:
AMDGPUAtomicOptimizerImpl() = delete;
@@ -756,12 +757,23 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
// Decide which implementation path to use.
// For DPP strategy with integer, divergent-value LDS atomics, when a
// threshold is set, use dynamic branching based on active lane count.
- const unsigned Threshold = AMDGPUAtomicOptimizerDPPLdsThreshold;
+ // The command-line option takes priority; otherwise fall back to a
+ // per-function attribute.
+ unsigned Threshold = 0;
+ if (AMDGPUAtomicOptimizerDPPLdsThreshold.getNumOccurrences() > 0) {
+ Threshold = AMDGPUAtomicOptimizerDPPLdsThreshold;
+ } else {
+ int AttrVal = I.getFunction()->getFnAttributeAsParsedInteger(
+ "amdgpu-atomic-optimizer-dpp-lds-threshold", 0);
+ if (AttrVal > 0)
+ Threshold = static_cast<unsigned>(AttrVal);
+ }
+
Value *Result = nullptr;
if (IsLDS && ValDivergent && ScanImpl == ScanOptions::DPP &&
!AtomicRMWInst::isFPOperation(Op) && Threshold > 0 &&
Threshold < ST.getWavefrontSize()) {
- Result = optimizeAtomicWithDynamicThreshold(B, I, Op, ValIdx);
+ Result = optimizeAtomicWithDynamicThreshold(B, I, Op, ValIdx, Threshold);
} else {
Result = optimizeAtomicImpl(B, I, Op, ValIdx, ValDivergent);
}
@@ -1078,8 +1090,8 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
// MergeBB:
// PHI merges results from both paths
Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicWithDynamicThreshold(
- IRBuilder<> &B, Instruction &I, AtomicRMWInst::BinOp Op,
- unsigned ValIdx) const {
+ IRBuilder<> &B, Instruction &I, AtomicRMWInst::BinOp Op, unsigned ValIdx,
+ unsigned Threshold) const {
Type *const Ty = I.getType();
const unsigned TyBitWidth = DL.getTypeSizeInBits(Ty);
const bool NeedResult = !I.use_empty();
@@ -1095,7 +1107,6 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicWithDynamicThreshold(
B.CreateUnaryIntrinsic(Intrinsic::ctpop, Ballot), B.getInt32Ty(), false);
// Branch: if active lanes > threshold, use DPP; otherwise no-opt.
- const unsigned Threshold = AMDGPUAtomicOptimizerDPPLdsThreshold;
Value *const ThresholdCond = B.CreateICmpUGT(Ctpop, B.getInt32(Threshold));
BasicBlock *const EntryBB = I.getParent();
>From ae05d5f71927db9da07b1db59669d420bc5d3526 Mon Sep 17 00:00:00 2001
From: YaFan <YaFan.Tao at amd.com>
Date: Mon, 23 Mar 2026 16:43:38 +0800
Subject: [PATCH 05/17] Invert NeedResult check to early return style
---
llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index 556c0c1a99fe4..5ed7f2cde7ccd 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -1178,14 +1178,14 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicWithDynamicThreshold(
Value *NoOptResult = NeedResult ? static_cast<Value *>(NoOptNewI) : nullptr;
// MergeBB: PHI-merge results from DppBB and NoOptBB.
- if (NeedResult) {
- B.SetInsertPoint(MergeBB, MergeBB->getFirstNonPHIIt());
- PHINode *const MergePHI = B.CreatePHI(Ty, 2);
- MergePHI->addIncoming(DppResult, ThenTerm->getParent());
- MergePHI->addIncoming(NoOptResult, NoOptBB);
- return MergePHI;
- }
- return nullptr;
+ if (!NeedResult)
+ return nullptr;
+
+ B.SetInsertPoint(MergeBB, MergeBB->getFirstNonPHIIt());
+ PHINode *const MergePHI = B.CreatePHI(Ty, 2);
+ MergePHI->addIncoming(DppResult, ThenTerm->getParent());
+ MergePHI->addIncoming(NoOptResult, NoOptBB);
+ return MergePHI;
}
INITIALIZE_PASS_BEGIN(AMDGPUAtomicOptimizer, DEBUG_TYPE,
>From f64c7dacf70aba4427f96d4e48f76471cdae9581 Mon Sep 17 00:00:00 2001
From: YaFan <YaFan.Tao at amd.com>
Date: Fri, 27 Mar 2026 16:47:52 +0800
Subject: [PATCH 06/17] [AMDGPU] Refactor optimizeAtomicWithDynamicThreshold to
reuse optimizeAtomicImpl
Instead of duplicating the DPP scan/reduction logic, delegate the actual
atomic optimization to optimizeAtomicImpl. Use SplitBlockAndInsertIfThenElse
to build the outer threshold CFG (OptBB/NoOptBB/TailBB), move I into OptBB,
and let optimizeAtomicImpl handle the rest.
NFC.
---
.../Target/AMDGPU/AMDGPUAtomicOptimizer.cpp | 135 ++++++------------
1 file changed, 40 insertions(+), 95 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index 5ed7f2cde7ccd..a9caeab069c7c 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -1060,131 +1060,76 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
}
// Generate a dynamic branch based on the active lane count so that the
-// DPP scan path is only used when enough lanes are active to amortise its
-// overhead. When the active count is at or below the threshold, each lane
-// independently issues its own atomic, which is cheaper for small groups.
+// optimized scan path is only used when enough lanes are active to amortise
+// its overhead. When the active count is at or below the threshold, each
+// lane independently issues its own atomic, which is cheaper for small
+// groups.
//
-// CFG produced:
+// This function builds the outer threshold CFG and delegates the actual
+// atomic optimization to optimizeAtomicImpl.
+//
+// CFG produced (the OptimizedPath sub-CFG is built by optimizeAtomicImpl):
//
// EntryBB:
// Ballot / Ctpop -> ActiveCount
-// cmp ActiveCount >= Threshold
-// br -> DppBB or NoOptBB
-//
-// DppBB:
-// DPP scan / reduction, single-lane gate
-// br -> SingleLaneBB or DppExitBB
+// cmp ActiveCount > Threshold
+// br -> OptBB or NoOptBB
//
-// SingleLaneBB:
-// atomic with reduced value
-// br -> DppExitBB
-//
-// DppExitBB:
-// readfirstlane + per-lane result
-// br -> MergeBB
+// OptBB:
+// [optimizeAtomicImpl rewrites I here, creating its own internal CFG]
+// br -> TailBB
//
// NoOptBB:
-// original atomic (unoptimized)
-// br -> MergeBB
+// original atomic (unoptimized, each lane issues its own)
+// br -> TailBB
//
-// MergeBB:
+// TailBB:
// PHI merges results from both paths
Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicWithDynamicThreshold(
IRBuilder<> &B, Instruction &I, AtomicRMWInst::BinOp Op, unsigned ValIdx,
unsigned Threshold) const {
Type *const Ty = I.getType();
- const unsigned TyBitWidth = DL.getTypeSizeInBits(Ty);
const bool NeedResult = !I.use_empty();
+ // Count active lanes and build the threshold condition.
Type *const WaveTy = B.getIntNTy(ST.getWavefrontSize());
CallInst *const Ballot =
B.CreateIntrinsic(Intrinsic::amdgcn_ballot, WaveTy, B.getTrue());
-
- Value *Mbcnt = buildMbcnt(B, Ballot);
-
- // Count active lanes.
Value *const Ctpop = B.CreateIntCast(
B.CreateUnaryIntrinsic(Intrinsic::ctpop, Ballot), B.getInt32Ty(), false);
-
- // Branch: if active lanes > threshold, use DPP; otherwise no-opt.
Value *const ThresholdCond = B.CreateICmpUGT(Ctpop, B.getInt32(Threshold));
- BasicBlock *const EntryBB = I.getParent();
- Instruction *const SplitPt = &I;
- Instruction *const ThenTerm = SplitBlockAndInsertIfThen(
- ThresholdCond, SplitPt, false, nullptr, &DTU, nullptr);
- BasicBlock *const DppBB = ThenTerm->getParent();
- BasicBlock *const MergeBB = I.getParent();
-
- // Also create NoOptBB between DppBB and MergeBB.
- BasicBlock *const NoOptBB = BasicBlock::Create(
- I.getContext(), "atomicrmw.no_opt", I.getFunction(), MergeBB);
- // Fix CFG: EntryBB's conditional branch false edge -> NoOptBB -> MergeBB.
- BranchInst *const EntryBr = cast<BranchInst>(EntryBB->getTerminator());
- EntryBr->setSuccessor(1, NoOptBB);
- IRBuilder<> NoOptBuilder(NoOptBB);
- NoOptBuilder.CreateBr(MergeBB);
- DTU.applyUpdates({{DominatorTree::Insert, EntryBB, NoOptBB},
- {DominatorTree::Delete, EntryBB, MergeBB},
- {DominatorTree::Insert, NoOptBB, MergeBB}});
-
- // DppBB: perform DPP scan/reduction and single-lane atomic.
- B.SetInsertPoint(ThenTerm);
-
- AtomicRMWInst::BinOp ScanOp = Op;
- if (Op == AtomicRMWInst::Sub)
- ScanOp = AtomicRMWInst::Add;
-
- Value *Identity = getIdentityValueForAtomicOp(Ty, ScanOp);
- Value *V = I.getOperand(ValIdx);
-
- auto [NewV, ExclScan] =
- buildDPPScanAndReduce(B, ScanOp, Ty, V, Identity, NeedResult);
+ // Split at I into: EntryBB -> OptBB (then) / NoOptBB (else) -> TailBB.
+ Instruction *ThenTerm = nullptr, *ElseTerm = nullptr;
+ SplitBlockAndInsertIfThenElse(ThresholdCond, &I, &ThenTerm, &ElseTerm,
+ nullptr, &DTU, nullptr);
+ BasicBlock *const NoOptBB = ElseTerm->getParent();
+ BasicBlock *const TailBB = I.getParent();
- Value *const DppIsFirst = B.CreateICmpEQ(Mbcnt, B.getInt32(0));
-
- BasicBlock *const DppOrigBB = B.GetInsertBlock();
- Instruction *const DppSingleTerm = SplitBlockAndInsertIfThen(
- DppIsFirst, ThenTerm, false, nullptr, &DTU, nullptr);
- B.SetInsertPoint(DppSingleTerm);
- Instruction *const DppNewI = I.clone();
- B.Insert(DppNewI);
- DppNewI->setOperand(ValIdx, NewV);
-
- Value *DppResult = nullptr;
- if (NeedResult) {
- B.SetInsertPoint(ThenTerm);
- PHINode *const DppPHI = B.CreatePHI(Ty, 2);
- DppPHI->addIncoming(PoisonValue::get(Ty), DppOrigBB);
- DppPHI->addIncoming(DppNewI, DppSingleTerm->getParent());
-
- Value *ReadlaneVal = DppPHI;
- if (TyBitWidth < 32)
- ReadlaneVal = B.CreateZExt(DppPHI, B.getInt32Ty());
- Value *BroadcastI = B.CreateIntrinsic(
- ReadlaneVal->getType(), Intrinsic::amdgcn_readfirstlane, ReadlaneVal);
- if (TyBitWidth < 32)
- BroadcastI = B.CreateTrunc(BroadcastI, Ty);
+ // NoOptBB: each lane independently issues its own atomic (unoptimized).
+ Instruction *const NoOptNewI = I.clone();
+ NoOptNewI->insertBefore(ElseTerm->getIterator());
- Value *LaneOffset =
- B.CreateIntrinsic(Intrinsic::amdgcn_strict_wwm, Ty, ExclScan);
- DppResult = buildNonAtomicBinOp(B, Op, BroadcastI, LaneOffset);
- }
+ // Move I into OptBB so that optimizeAtomicImpl rewrites only the optimized
+ // path. I's uses are temporarily non-dominating; they will be fixed when
+ // optimizeAtomic does replaceAllUsesWith after we return.
+ I.moveBefore(ThenTerm->getIterator());
+ B.SetInsertPoint(&I);
- // NoOptBB: each lane independently issues its own atomic.
- NoOptBuilder.SetInsertPoint(NoOptBB->getTerminator());
- Instruction *const NoOptNewI = I.clone();
- NoOptBuilder.Insert(NoOptNewI);
- Value *NoOptResult = NeedResult ? static_cast<Value *>(NoOptNewI) : nullptr;
+ // Delegate the full scan/reduction + single-lane atomic + result
+ // reconstruction to optimizeAtomicImpl.
+ Value *OptResult =
+ optimizeAtomicImpl(B, I, Op, ValIdx, /*ValDivergent=*/true);
- // MergeBB: PHI-merge results from DppBB and NoOptBB.
if (!NeedResult)
return nullptr;
- B.SetInsertPoint(MergeBB, MergeBB->getFirstNonPHIIt());
+ // After optimizeAtomicImpl, I sits in the exit block of the optimized
+ // sub-CFG. Merge the optimized and no-opt results in TailBB.
+ B.SetInsertPoint(TailBB, TailBB->getFirstNonPHIIt());
PHINode *const MergePHI = B.CreatePHI(Ty, 2);
- MergePHI->addIncoming(DppResult, ThenTerm->getParent());
- MergePHI->addIncoming(NoOptResult, NoOptBB);
+ MergePHI->addIncoming(OptResult, I.getParent());
+ MergePHI->addIncoming(NoOptNewI, NoOptBB);
return MergePHI;
}
>From bbc229a56c52c18d18b3b3fc75451f799c13ff0e Mon Sep 17 00:00:00 2001
From: YaFan <YaFan.Tao at amd.com>
Date: Thu, 2 Apr 2026 11:01:01 +0800
Subject: [PATCH 07/17] Delete east const.
---
.../Target/AMDGPU/AMDGPUAtomicOptimizer.cpp | 48 +++++++++----------
1 file changed, 24 insertions(+), 24 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index a9caeab069c7c..a99009df27c84 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -743,8 +743,8 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
// Record I's original position as the entry block.
PixelEntryBB = I.getParent();
- Value *const Cond = B.CreateIntrinsic(Intrinsic::amdgcn_ps_live, {});
- Instruction *const NonHelperTerminator =
+ Value *Cond = B.CreateIntrinsic(Intrinsic::amdgcn_ps_live, {});
+ Instruction *NonHelperTerminator =
SplitBlockAndInsertIfThen(Cond, &I, false, nullptr, &DTU, nullptr);
// Record I's new position as the exit block.
@@ -784,7 +784,7 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
// Need a final PHI to reconverge to above the helper lane branch mask.
B.SetInsertPoint(PixelExitBB, PixelExitBB->getFirstNonPHIIt());
- PHINode *const PHI = B.CreatePHI(I.getType(), 2);
+ PHINode *PHI = B.CreatePHI(I.getType(), 2);
PHI->addIncoming(PoisonValue::get(I.getType()), PixelEntryBB);
PHI->addIncoming(Result, I.getParent());
I.replaceAllUsesWith(PHI);
@@ -806,7 +806,7 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
AtomicRMWInst::BinOp Op,
unsigned ValIdx,
bool ValDivergent) const {
- Type *const Ty = I.getType();
+ Type *Ty = I.getType();
Type *Int32Ty = B.getInt32Ty();
bool isAtomicFloatingPointTy = Ty->isFloatingPointTy();
[[maybe_unused]] const unsigned TyBitWidth = DL.getTypeSizeInBits(Ty);
@@ -817,8 +817,8 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
// We need to know how many lanes are active within the wavefront, and we do
// this by doing a ballot of active lanes.
- Type *const WaveTy = B.getIntNTy(ST.getWavefrontSize());
- CallInst *const Ballot =
+ Type *WaveTy = B.getIntNTy(ST.getWavefrontSize());
+ CallInst *Ballot =
B.CreateIntrinsic(Intrinsic::amdgcn_ballot, WaveTy, B.getTrue());
// We need to know how many lanes are active within the wavefront that are
@@ -871,16 +871,16 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
case AtomicRMWInst::Sub: {
// The new value we will be contributing to the atomic operation is the
// old value times the number of active lanes.
- Value *const Ctpop = B.CreateIntCast(
+ Value *Ctpop = B.CreateIntCast(
B.CreateUnaryIntrinsic(Intrinsic::ctpop, Ballot), Ty, false);
NewV = buildMul(B, V, Ctpop);
break;
}
case AtomicRMWInst::FAdd:
case AtomicRMWInst::FSub: {
- Value *const Ctpop = B.CreateIntCast(
+ Value *Ctpop = B.CreateIntCast(
B.CreateUnaryIntrinsic(Intrinsic::ctpop, Ballot), Int32Ty, false);
- Value *const CtpopFP = B.CreateUIToFP(Ctpop, Ty);
+ Value *CtpopFP = B.CreateUIToFP(Ctpop, Ty);
NewV = B.CreateFMul(V, CtpopFP);
break;
}
@@ -900,7 +900,7 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
case AtomicRMWInst::Xor:
// The new value we will be contributing to the atomic operation is the
// old value times the parity of the number of active lanes.
- Value *const Ctpop = B.CreateIntCast(
+ Value *Ctpop = B.CreateIntCast(
B.CreateUnaryIntrinsic(Intrinsic::ctpop, Ballot), Ty, false);
NewV = buildMul(B, V, B.CreateAnd(Ctpop, 1));
break;
@@ -910,17 +910,17 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
// We only want a single lane to enter our new control flow, and we do this
// by checking if there are any active lanes below us. Only one lane will
// have 0 active lanes below us, so that will be the only one to progress.
- Value *const Cond = B.CreateICmpEQ(Mbcnt, B.getInt32(0));
+ Value *Cond = B.CreateICmpEQ(Mbcnt, B.getInt32(0));
// Store I's original basic block before we split the block.
- BasicBlock *const OriginalBB = I.getParent();
+ BasicBlock *OriginalBB = I.getParent();
// We need to introduce some new control flow to force a single lane to be
// active. We do this by splitting I's basic block at I, and introducing the
// new block such that:
// entry --> single_lane -\
// \------------------> exit
- Instruction *const SingleLaneTerminator =
+ Instruction *SingleLaneTerminator =
SplitBlockAndInsertIfThen(Cond, &I, false, nullptr, &DTU, nullptr);
// At this point, we have split the I's block to allow one lane in wavefront
@@ -970,7 +970,7 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
// Clone the original atomic operation into single lane, replacing the
// original value with our newly created one.
- Instruction *const NewI = I.clone();
+ Instruction *NewI = I.clone();
B.Insert(NewI);
NewI->setOperand(ValIdx, NewV);
@@ -980,7 +980,7 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
if (NeedResult) {
// Create a PHI node to get our new atomic result into the exit block.
- PHINode *const PHI = B.CreatePHI(Ty, 2);
+ PHINode *PHI = B.CreatePHI(Ty, 2);
PHI->addIncoming(PoisonValue::get(Ty), Predecessor);
PHI->addIncoming(NewI, SingleLaneTerminator->getParent());
@@ -1088,26 +1088,26 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicWithDynamicThreshold(
IRBuilder<> &B, Instruction &I, AtomicRMWInst::BinOp Op, unsigned ValIdx,
unsigned Threshold) const {
- Type *const Ty = I.getType();
+ Type *Ty = I.getType();
const bool NeedResult = !I.use_empty();
// Count active lanes and build the threshold condition.
- Type *const WaveTy = B.getIntNTy(ST.getWavefrontSize());
- CallInst *const Ballot =
+ Type *WaveTy = B.getIntNTy(ST.getWavefrontSize());
+ CallInst *Ballot =
B.CreateIntrinsic(Intrinsic::amdgcn_ballot, WaveTy, B.getTrue());
- Value *const Ctpop = B.CreateIntCast(
+ Value *Ctpop = B.CreateIntCast(
B.CreateUnaryIntrinsic(Intrinsic::ctpop, Ballot), B.getInt32Ty(), false);
- Value *const ThresholdCond = B.CreateICmpUGT(Ctpop, B.getInt32(Threshold));
+ Value *ThresholdCond = B.CreateICmpUGT(Ctpop, B.getInt32(Threshold));
// Split at I into: EntryBB -> OptBB (then) / NoOptBB (else) -> TailBB.
Instruction *ThenTerm = nullptr, *ElseTerm = nullptr;
SplitBlockAndInsertIfThenElse(ThresholdCond, &I, &ThenTerm, &ElseTerm,
nullptr, &DTU, nullptr);
- BasicBlock *const NoOptBB = ElseTerm->getParent();
- BasicBlock *const TailBB = I.getParent();
+ BasicBlock *NoOptBB = ElseTerm->getParent();
+ BasicBlock *TailBB = I.getParent();
// NoOptBB: each lane independently issues its own atomic (unoptimized).
- Instruction *const NoOptNewI = I.clone();
+ Instruction *NoOptNewI = I.clone();
NoOptNewI->insertBefore(ElseTerm->getIterator());
// Move I into OptBB so that optimizeAtomicImpl rewrites only the optimized
@@ -1127,7 +1127,7 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicWithDynamicThreshold(
// After optimizeAtomicImpl, I sits in the exit block of the optimized
// sub-CFG. Merge the optimized and no-opt results in TailBB.
B.SetInsertPoint(TailBB, TailBB->getFirstNonPHIIt());
- PHINode *const MergePHI = B.CreatePHI(Ty, 2);
+ PHINode *MergePHI = B.CreatePHI(Ty, 2);
MergePHI->addIncoming(OptResult, I.getParent());
MergePHI->addIncoming(NoOptNewI, NoOptBB);
return MergePHI;
>From cbf95a61328c62be93c773b66eb53e1cc757159b Mon Sep 17 00:00:00 2001
From: YaFan <YaFan.Tao at amd.com>
Date: Thu, 2 Apr 2026 14:33:38 +0800
Subject: [PATCH 08/17] Remove cl::opt and add document
---
llvm/docs/AMDGPUUsage.rst | 10 +++++++
.../Target/AMDGPU/AMDGPUAtomicOptimizer.cpp | 28 +++----------------
2 files changed, 14 insertions(+), 24 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 1ede5ca2d4cf6..81f5380b89e67 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -2205,6 +2205,16 @@ The AMDGPU backend supports the following LLVM IR attributes.
"amdgpu-no-fwd-progress" Disable forward progress mode for wave priority
(enabled by default).
+ "amdgpu-atomic-optimizer-dpp-lds-threshold"="n" Threshold that controls when the AMDGPUAtomicOptimizer uses a DPP-based
+ scan for integer LDS atomics. When set to a positive integer n, the
+ optimizer emits a dynamic branch: if the number of active lanes exceeds
+ n, the DPP scan path is taken; otherwise each lane independently issues
+ its own atomic. A value of 0 (the default) disables the dynamic branch
+ and always uses the DPP scan path when the DPP strategy is selected.
+ Values greater than or equal to the wavefront size effectively disable
+ DPP optimization for LDS atomics. Floating-point LDS atomics are not
+ affected by this attribute.
+
Relevant for GFX10+.
================================================ ==========================================================
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index a99009df27c84..cee2ab05203a0 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -38,22 +38,6 @@
using namespace llvm;
using namespace llvm::AMDGPU;
-// When only a few lanes are active, the DPP scan overhead can outweigh the
-// benefit of reducing LDS atomic contention. This threshold lets the compiler
-// emit a dynamic branch so that small-active-lane cases fall back to
-// unoptimized per-lane atomics while large-active-lane cases still benefit from
-// DPP.
-//
-// Only integer LDS atomics are affected because floating-point path is
-// untested. The default is 0 (always use DPP) because unconditionally enabling
-// the threshold can regress DPP-friendly workloads by ~4%.
-static cl::opt<unsigned> AMDGPUAtomicOptimizerDPPLdsThreshold(
- "amdgpu-atomic-optimizer-dpp-lds-threshold",
- cl::desc("Use DPP scan for integer LDS atomics only when active lanes > "
- "this value; floating-point LDS atomics are not affected "
- "(0 = always use DPP, >= wavefront size = never use DPP)"),
- cl::init(0), cl::Hidden);
-
namespace {
struct ReplacementInfo {
@@ -760,14 +744,10 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
// The command-line option takes priority; otherwise fall back to a
// per-function attribute.
unsigned Threshold = 0;
- if (AMDGPUAtomicOptimizerDPPLdsThreshold.getNumOccurrences() > 0) {
- Threshold = AMDGPUAtomicOptimizerDPPLdsThreshold;
- } else {
- int AttrVal = I.getFunction()->getFnAttributeAsParsedInteger(
- "amdgpu-atomic-optimizer-dpp-lds-threshold", 0);
- if (AttrVal > 0)
- Threshold = static_cast<unsigned>(AttrVal);
- }
+ int AttrVal = I.getFunction()->getFnAttributeAsParsedInteger(
+ "amdgpu-atomic-optimizer-dpp-lds-threshold", 0);
+ if (AttrVal > 0)
+ Threshold = static_cast<unsigned>(AttrVal);
Value *Result = nullptr;
if (IsLDS && ValDivergent && ScanImpl == ScanOptions::DPP &&
>From 8d8306cf564843196a4b837925b2daba042e4645 Mon Sep 17 00:00:00 2001
From: YaFan <YaFan.Tao at amd.com>
Date: Thu, 2 Apr 2026 16:02:46 +0800
Subject: [PATCH 09/17] Update Lit test, Fix that DPP should be skiped when
threshold >= wave size
---
.../Target/AMDGPU/AMDGPUAtomicOptimizer.cpp | 28 +-
.../atomic_optimizations_dpp_lds_threshold.ll | 2568 +++++++----------
2 files changed, 1127 insertions(+), 1469 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index cee2ab05203a0..84d1f73a76e17 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -706,6 +706,20 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
unsigned ValIdx,
bool ValDivergent,
bool IsLDS) const {
+ // Read the per-function threshold attribute early so we can bail out
+ // before any CFG modifications when the threshold disables optimization.
+ unsigned Threshold = 0;
+ int AttrVal = I.getFunction()->getFnAttributeAsParsedInteger(
+ "amdgpu-atomic-optimizer-dpp-lds-threshold", 0);
+ if (AttrVal > 0)
+ Threshold = static_cast<unsigned>(AttrVal);
+
+ // If the threshold is >= wavefront size, DPP is never profitable — skip
+ // the atomic optimization entirely for this instruction.
+ if (IsLDS && ValDivergent && ScanImpl == ScanOptions::DPP &&
+ !AtomicRMWInst::isFPOperation(Op) && Threshold >= ST.getWavefrontSize())
+ return;
+
// Start building just before the instruction.
IRBuilder<> B(&I);
@@ -738,21 +752,9 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
B.SetInsertPoint(&I);
}
- // Decide which implementation path to use.
- // For DPP strategy with integer, divergent-value LDS atomics, when a
- // threshold is set, use dynamic branching based on active lane count.
- // The command-line option takes priority; otherwise fall back to a
- // per-function attribute.
- unsigned Threshold = 0;
- int AttrVal = I.getFunction()->getFnAttributeAsParsedInteger(
- "amdgpu-atomic-optimizer-dpp-lds-threshold", 0);
- if (AttrVal > 0)
- Threshold = static_cast<unsigned>(AttrVal);
-
Value *Result = nullptr;
if (IsLDS && ValDivergent && ScanImpl == ScanOptions::DPP &&
- !AtomicRMWInst::isFPOperation(Op) && Threshold > 0 &&
- Threshold < ST.getWavefrontSize()) {
+ !AtomicRMWInst::isFPOperation(Op) && Threshold > 0) {
Result = optimizeAtomicWithDynamicThreshold(B, I, Op, ValIdx, Threshold);
} else {
Result = optimizeAtomicImpl(B, I, Op, ValIdx, ValDivergent);
diff --git a/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll b/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
index 0d52229a34edf..7540cc64bc89d 100644
--- a/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
+++ b/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
@@ -1,16 +1,11 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
;
-; --- Threshold=5 tests (dynamic branch expected) ---
-; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP -amdgpu-atomic-optimizer-dpp-lds-threshold=5 < %s | FileCheck -enable-var-scope -check-prefixes=GFX1032-THRESH5 %s
-; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP -amdgpu-atomic-optimizer-dpp-lds-threshold=5 < %s | FileCheck -enable-var-scope -check-prefixes=GFX1064-THRESH5 %s
-; RUN: llc -mtriple=amdgcn -mcpu=gfx900 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP -amdgpu-atomic-optimizer-dpp-lds-threshold=5 < %s | FileCheck -enable-var-scope -check-prefixes=GFX900-THRESH5 %s
+; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP < %s | FileCheck -enable-var-scope -check-prefixes=GFX1032 %s
+; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP < %s | FileCheck -enable-var-scope -check-prefixes=GFX1064 %s
+; RUN: llc -mtriple=amdgcn -mcpu=gfx900 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP < %s | FileCheck -enable-var-scope -check-prefixes=GFX900 %s
;
-; --- Threshold >= wavefront size tests (DPP fully disabled for int LDS) ---
-; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP -amdgpu-atomic-optimizer-dpp-lds-threshold=32 < %s | FileCheck -enable-var-scope -check-prefixes=GFX1032-DISABLED %s
-; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP -amdgpu-atomic-optimizer-dpp-lds-threshold=64 < %s | FileCheck -enable-var-scope -check-prefixes=GFX1064-DISABLED %s
-;
-; Test the -amdgpu-atomic-optimizer-dpp-lds-threshold option which controls
-; dynamic DPP vs no-opt branching for integer LDS atomics.
+; Test the "amdgpu-atomic-optimizer-dpp-lds-threshold" function attribute,
+; which controls dynamic DPP vs no-opt branching for integer LDS atomics.
;
; Threshold=5: use DPP only when active lanes > 5, otherwise each lane does
; its own atomic.
@@ -21,283 +16,192 @@ declare i32 @llvm.amdgcn.workitem.id.x()
@local_var32 = addrspace(3) global i32 undef, align 4
+attributes #0 = { "amdgpu-atomic-optimizer-dpp-lds-threshold"="5" }
+attributes #1 = { "amdgpu-atomic-optimizer-dpp-lds-threshold"="32" }
+attributes #2 = { "amdgpu-atomic-optimizer-dpp-lds-threshold"="64" }
+
; Test 1: divergent i32 add with result used -- dynamic threshold branch expected
-define amdgpu_kernel void @add_i32_varying(ptr addrspace(1) %out) {
-; GFX1032-THRESH5-LABEL: add_i32_varying:
-; GFX1032-THRESH5: ; %bb.0: ; %entry
-; GFX1032-THRESH5-NEXT: s_bcnt1_i32_b32 s0, exec_lo
-; GFX1032-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
-; GFX1032-THRESH5-NEXT: s_cbranch_scc0 .LBB0_2
-; GFX1032-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v4, 0
-; GFX1032-THRESH5-NEXT: ds_add_rtn_u32 v4, v4, v0
-; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-THRESH5-NEXT: buffer_gl0_inv
-; GFX1032-THRESH5-NEXT: s_cbranch_execz .LBB0_3
-; GFX1032-THRESH5-NEXT: s_branch .LBB0_6
-; GFX1032-THRESH5-NEXT: .LBB0_2:
-; GFX1032-THRESH5-NEXT: ; implicit-def: $vgpr4
-; GFX1032-THRESH5-NEXT: .LBB0_3:
-; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v3, 0
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-THRESH5-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-THRESH5-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-THRESH5-NEXT: v_readlane_b32 s1, v1, 15
-; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, 0
-; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-THRESH5-NEXT: v_writelane_b32 v3, s1, 16
-; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX1032-THRESH5-NEXT: ; implicit-def: $vgpr4
-; GFX1032-THRESH5-NEXT: s_and_saveexec_b32 s0, vcc_lo
-; GFX1032-THRESH5-NEXT: s_cbranch_execz .LBB0_5
-; GFX1032-THRESH5-NEXT: ; %bb.4:
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v4, s2
-; GFX1032-THRESH5-NEXT: s_waitcnt_vscnt null, 0x0
-; GFX1032-THRESH5-NEXT: ds_add_rtn_u32 v4, v0, v4
-; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-THRESH5-NEXT: buffer_gl0_inv
-; GFX1032-THRESH5-NEXT: .LBB0_5:
-; GFX1032-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1032-THRESH5-NEXT: s_or_b32 exec_lo, exec_lo, s0
-; GFX1032-THRESH5-NEXT: v_readfirstlane_b32 s0, v4
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, v3
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_e32 v4, s0, v0
-; GFX1032-THRESH5-NEXT: .LBB0_6:
-; GFX1032-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1032-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1032-THRESH5-NEXT: s_mov_b32 s2, -1
-; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-THRESH5-NEXT: buffer_store_dword v4, off, s[0:3], 0
-; GFX1032-THRESH5-NEXT: s_endpgm
-;
-; GFX1064-THRESH5-LABEL: add_i32_varying:
-; GFX1064-THRESH5: ; %bb.0: ; %entry
-; GFX1064-THRESH5-NEXT: s_bcnt1_i32_b64 s0, exec
-; GFX1064-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
-; GFX1064-THRESH5-NEXT: s_cbranch_scc0 .LBB0_2
-; GFX1064-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v4, 0
-; GFX1064-THRESH5-NEXT: ds_add_rtn_u32 v4, v4, v0
-; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-THRESH5-NEXT: buffer_gl0_inv
-; GFX1064-THRESH5-NEXT: s_cbranch_execz .LBB0_3
-; GFX1064-THRESH5-NEXT: s_branch .LBB0_6
-; GFX1064-THRESH5-NEXT: .LBB0_2:
-; GFX1064-THRESH5-NEXT: ; implicit-def: $vgpr4
-; GFX1064-THRESH5-NEXT: .LBB0_3:
-; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v3, 0
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-THRESH5-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v2, s2
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 15
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s3, v1, 31
-; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s2, 16
-; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 63
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s6, v1, 47
-; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s3, 32
-; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, 0
-; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s6, 48
-; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX1064-THRESH5-NEXT: ; implicit-def: $vgpr4
-; GFX1064-THRESH5-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX1064-THRESH5-NEXT: s_cbranch_execz .LBB0_5
-; GFX1064-THRESH5-NEXT: ; %bb.4:
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v4, s2
-; GFX1064-THRESH5-NEXT: s_waitcnt_vscnt null, 0x0
-; GFX1064-THRESH5-NEXT: ds_add_rtn_u32 v4, v0, v4
-; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-THRESH5-NEXT: buffer_gl0_inv
-; GFX1064-THRESH5-NEXT: .LBB0_5:
-; GFX1064-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1064-THRESH5-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX1064-THRESH5-NEXT: v_readfirstlane_b32 s0, v4
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, v3
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_e32 v4, s0, v0
-; GFX1064-THRESH5-NEXT: .LBB0_6:
-; GFX1064-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1064-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1064-THRESH5-NEXT: s_mov_b32 s2, -1
-; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-THRESH5-NEXT: buffer_store_dword v4, off, s[0:3], 0
-; GFX1064-THRESH5-NEXT: s_endpgm
-;
-; GFX900-THRESH5-LABEL: add_i32_varying:
-; GFX900-THRESH5: ; %bb.0: ; %entry
-; GFX900-THRESH5-NEXT: s_bcnt1_i32_b64 s0, exec
-; GFX900-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
-; GFX900-THRESH5-NEXT: s_cbranch_scc0 .LBB0_2
-; GFX900-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v3, 0
-; GFX900-THRESH5-NEXT: ds_add_rtn_u32 v3, v3, v0
-; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-THRESH5-NEXT: s_cbranch_execz .LBB0_3
-; GFX900-THRESH5-NEXT: s_branch .LBB0_6
-; GFX900-THRESH5-NEXT: .LBB0_2:
-; GFX900-THRESH5-NEXT: ; implicit-def: $vgpr3
-; GFX900-THRESH5-NEXT: .LBB0_3:
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v3, 0
-; GFX900-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX900-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX900-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX900-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v2, 0
-; GFX900-THRESH5-NEXT: s_nop 0
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX900-THRESH5-NEXT: v_readlane_b32 s2, v1, 63
-; GFX900-THRESH5-NEXT: s_nop 0
-; GFX900-THRESH5-NEXT: v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX900-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
-; GFX900-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX900-THRESH5-NEXT: ; implicit-def: $vgpr0
-; GFX900-THRESH5-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX900-THRESH5-NEXT: s_cbranch_execz .LBB0_5
-; GFX900-THRESH5-NEXT: ; %bb.4:
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, s2
-; GFX900-THRESH5-NEXT: ds_add_rtn_u32 v0, v3, v0
-; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-THRESH5-NEXT: .LBB0_5:
-; GFX900-THRESH5-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX900-THRESH5-NEXT: v_readfirstlane_b32 s0, v0
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, v2
-; GFX900-THRESH5-NEXT: v_add_u32_e32 v3, s0, v0
-; GFX900-THRESH5-NEXT: .LBB0_6:
-; GFX900-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX900-THRESH5-NEXT: s_mov_b32 s3, 0xf000
-; GFX900-THRESH5-NEXT: s_mov_b32 s2, -1
-; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-THRESH5-NEXT: buffer_store_dword v3, off, s[0:3], 0
-; GFX900-THRESH5-NEXT: s_endpgm
+define amdgpu_kernel void @add_i32_varying(ptr addrspace(1) %out) #0 {
+; GFX1032-LABEL: add_i32_varying:
+; GFX1032: ; %bb.0: ; %entry
+; GFX1032-NEXT: s_bcnt1_i32_b32 s0, exec_lo
+; GFX1032-NEXT: s_cmp_lt_u32 s0, 6
+; GFX1032-NEXT: s_cbranch_scc0 .LBB0_2
+; GFX1032-NEXT: ; %bb.1:
+; GFX1032-NEXT: v_mov_b32_e32 v4, 0
+; GFX1032-NEXT: ds_add_rtn_u32 v4, v4, v0
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_gl0_inv
+; GFX1032-NEXT: s_cbranch_execz .LBB0_3
+; GFX1032-NEXT: s_branch .LBB0_6
+; GFX1032-NEXT: .LBB0_2:
+; GFX1032-NEXT: ; implicit-def: $vgpr4
+; GFX1032-NEXT: .LBB0_3:
+; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1032-NEXT: v_mov_b32_e32 v3, 0
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1032-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1032-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-NEXT: v_readlane_b32 s1, v1, 15
+; GFX1032-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1032-NEXT: v_mov_b32_e32 v0, 0
+; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-NEXT: v_writelane_b32 v3, s1, 16
+; GFX1032-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
+; GFX1032-NEXT: ; implicit-def: $vgpr4
+; GFX1032-NEXT: s_and_saveexec_b32 s0, vcc_lo
+; GFX1032-NEXT: s_cbranch_execz .LBB0_5
+; GFX1032-NEXT: ; %bb.4:
+; GFX1032-NEXT: v_mov_b32_e32 v4, s2
+; GFX1032-NEXT: s_waitcnt_vscnt null, 0x0
+; GFX1032-NEXT: ds_add_rtn_u32 v4, v0, v4
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_gl0_inv
+; GFX1032-NEXT: .LBB0_5:
+; GFX1032-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-NEXT: s_or_b32 exec_lo, exec_lo, s0
+; GFX1032-NEXT: v_readfirstlane_b32 s0, v4
+; GFX1032-NEXT: v_mov_b32_e32 v0, v3
+; GFX1032-NEXT: v_add_nc_u32_e32 v4, s0, v0
+; GFX1032-NEXT: .LBB0_6:
+; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1032-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1032-NEXT: s_mov_b32 s2, -1
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_store_dword v4, off, s[0:3], 0
+; GFX1032-NEXT: s_endpgm
;
-; GFX1032-DISABLED-LABEL: add_i32_varying:
-; GFX1032-DISABLED: ; %bb.0: ; %entry
-; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-DISABLED-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v3, 0
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-DISABLED-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-DISABLED-NEXT: v_readlane_b32 s1, v1, 15
-; GFX1032-DISABLED-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v4, 0
-; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-DISABLED-NEXT: v_writelane_b32 v3, s1, 16
-; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-DISABLED-NEXT: s_mov_b32 s0, s2
-; GFX1032-DISABLED-NEXT: s_mov_b32 s2, -1
-; GFX1032-DISABLED-NEXT: ; implicit-def: $vgpr0
-; GFX1032-DISABLED-NEXT: s_and_saveexec_b32 s1, vcc_lo
-; GFX1032-DISABLED-NEXT: s_cbranch_execz .LBB0_2
-; GFX1032-DISABLED-NEXT: ; %bb.1:
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, s0
-; GFX1032-DISABLED-NEXT: ds_add_rtn_u32 v0, v4, v0
-; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-DISABLED-NEXT: buffer_gl0_inv
-; GFX1032-DISABLED-NEXT: .LBB0_2:
-; GFX1032-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1032-DISABLED-NEXT: s_or_b32 exec_lo, exec_lo, s1
-; GFX1032-DISABLED-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1032-DISABLED-NEXT: v_readfirstlane_b32 s3, v0
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, v3
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_e32 v0, s3, v0
-; GFX1032-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1032-DISABLED-NEXT: s_endpgm
+; GFX1064-LABEL: add_i32_varying:
+; GFX1064: ; %bb.0: ; %entry
+; GFX1064-NEXT: s_bcnt1_i32_b64 s0, exec
+; GFX1064-NEXT: s_cmp_lt_u32 s0, 6
+; GFX1064-NEXT: s_cbranch_scc0 .LBB0_2
+; GFX1064-NEXT: ; %bb.1:
+; GFX1064-NEXT: v_mov_b32_e32 v4, 0
+; GFX1064-NEXT: ds_add_rtn_u32 v4, v4, v0
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: s_cbranch_execz .LBB0_3
+; GFX1064-NEXT: s_branch .LBB0_6
+; GFX1064-NEXT: .LBB0_2:
+; GFX1064-NEXT: ; implicit-def: $vgpr4
+; GFX1064-NEXT: .LBB0_3:
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1064-NEXT: v_mov_b32_e32 v3, 0
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1064-NEXT: v_mov_b32_e32 v2, s2
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 15
+; GFX1064-NEXT: v_readlane_b32 s3, v1, 31
+; GFX1064-NEXT: v_writelane_b32 v3, s2, 16
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 63
+; GFX1064-NEXT: v_readlane_b32 s6, v1, 47
+; GFX1064-NEXT: v_writelane_b32 v3, s3, 32
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
+; GFX1064-NEXT: v_mov_b32_e32 v0, 0
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_writelane_b32 v3, s6, 48
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX1064-NEXT: ; implicit-def: $vgpr4
+; GFX1064-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1064-NEXT: s_cbranch_execz .LBB0_5
+; GFX1064-NEXT: ; %bb.4:
+; GFX1064-NEXT: v_mov_b32_e32 v4, s2
+; GFX1064-NEXT: s_waitcnt_vscnt null, 0x0
+; GFX1064-NEXT: ds_add_rtn_u32 v4, v0, v4
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: .LBB0_5:
+; GFX1064-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1064-NEXT: v_readfirstlane_b32 s0, v4
+; GFX1064-NEXT: v_mov_b32_e32 v0, v3
+; GFX1064-NEXT: v_add_nc_u32_e32 v4, s0, v0
+; GFX1064-NEXT: .LBB0_6:
+; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-NEXT: s_mov_b32 s2, -1
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_store_dword v4, off, s[0:3], 0
+; GFX1064-NEXT: s_endpgm
;
-; GFX1064-DISABLED-LABEL: add_i32_varying:
-; GFX1064-DISABLED: ; %bb.0: ; %entry
-; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-DISABLED-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v3, 0
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-DISABLED-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v2, s2
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 15
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s3, v1, 31
-; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s2, 16
-; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 47
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s6, v1, 63
-; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s3, 32
-; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-DISABLED-NEXT: v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v4, 0
-; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s2, 48
-; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-DISABLED-NEXT: s_mov_b32 s2, -1
-; GFX1064-DISABLED-NEXT: ; implicit-def: $vgpr0
-; GFX1064-DISABLED-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX1064-DISABLED-NEXT: s_cbranch_execz .LBB0_2
-; GFX1064-DISABLED-NEXT: ; %bb.1:
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, s6
-; GFX1064-DISABLED-NEXT: ds_add_rtn_u32 v0, v4, v0
-; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-DISABLED-NEXT: buffer_gl0_inv
-; GFX1064-DISABLED-NEXT: .LBB0_2:
-; GFX1064-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1064-DISABLED-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX1064-DISABLED-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1064-DISABLED-NEXT: v_readfirstlane_b32 s3, v0
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, v3
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_e32 v0, s3, v0
-; GFX1064-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1064-DISABLED-NEXT: s_endpgm
+; GFX900-LABEL: add_i32_varying:
+; GFX900: ; %bb.0: ; %entry
+; GFX900-NEXT: s_bcnt1_i32_b64 s0, exec
+; GFX900-NEXT: s_cmp_lt_u32 s0, 6
+; GFX900-NEXT: s_cbranch_scc0 .LBB0_2
+; GFX900-NEXT: ; %bb.1:
+; GFX900-NEXT: v_mov_b32_e32 v3, 0
+; GFX900-NEXT: ds_add_rtn_u32 v3, v3, v0
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: s_cbranch_execz .LBB0_3
+; GFX900-NEXT: s_branch .LBB0_6
+; GFX900-NEXT: .LBB0_2:
+; GFX900-NEXT: ; implicit-def: $vgpr3
+; GFX900-NEXT: .LBB0_3:
+; GFX900-NEXT: v_mov_b32_e32 v3, 0
+; GFX900-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX900-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v4
+; GFX900-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX900-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX900-NEXT: v_mov_b32_e32 v2, 0
+; GFX900-NEXT: s_nop 0
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
+; GFX900-NEXT: v_readlane_b32 s2, v1, 63
+; GFX900-NEXT: s_nop 0
+; GFX900-NEXT: v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
+; GFX900-NEXT: s_mov_b64 exec, s[0:1]
+; GFX900-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX900-NEXT: ; implicit-def: $vgpr0
+; GFX900-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX900-NEXT: s_cbranch_execz .LBB0_5
+; GFX900-NEXT: ; %bb.4:
+; GFX900-NEXT: v_mov_b32_e32 v0, s2
+; GFX900-NEXT: ds_add_rtn_u32 v0, v3, v0
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: .LBB0_5:
+; GFX900-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX900-NEXT: v_readfirstlane_b32 s0, v0
+; GFX900-NEXT: v_mov_b32_e32 v0, v2
+; GFX900-NEXT: v_add_u32_e32 v3, s0, v0
+; GFX900-NEXT: .LBB0_6:
+; GFX900-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX900-NEXT: s_mov_b32 s3, 0xf000
+; GFX900-NEXT: s_mov_b32 s2, -1
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: buffer_store_dword v3, off, s[0:3], 0
+; GFX900-NEXT: s_endpgm
entry:
%lane = call i32 @llvm.amdgcn.workitem.id.x()
%old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel
@@ -306,282 +210,187 @@ entry:
}
; Test 2: divergent i32 sub with result used
-define amdgpu_kernel void @sub_i32_varying(ptr addrspace(1) %out) {
-; GFX1032-THRESH5-LABEL: sub_i32_varying:
-; GFX1032-THRESH5: ; %bb.0: ; %entry
-; GFX1032-THRESH5-NEXT: s_bcnt1_i32_b32 s0, exec_lo
-; GFX1032-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
-; GFX1032-THRESH5-NEXT: s_cbranch_scc0 .LBB1_2
-; GFX1032-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v4, 0
-; GFX1032-THRESH5-NEXT: ds_sub_rtn_u32 v4, v4, v0
-; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-THRESH5-NEXT: buffer_gl0_inv
-; GFX1032-THRESH5-NEXT: s_cbranch_execz .LBB1_3
-; GFX1032-THRESH5-NEXT: s_branch .LBB1_6
-; GFX1032-THRESH5-NEXT: .LBB1_2:
-; GFX1032-THRESH5-NEXT: ; implicit-def: $vgpr4
-; GFX1032-THRESH5-NEXT: .LBB1_3:
-; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v3, 0
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-THRESH5-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-THRESH5-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-THRESH5-NEXT: v_readlane_b32 s1, v1, 15
-; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, 0
-; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-THRESH5-NEXT: v_writelane_b32 v3, s1, 16
-; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX1032-THRESH5-NEXT: ; implicit-def: $vgpr4
-; GFX1032-THRESH5-NEXT: s_and_saveexec_b32 s0, vcc_lo
-; GFX1032-THRESH5-NEXT: s_cbranch_execz .LBB1_5
-; GFX1032-THRESH5-NEXT: ; %bb.4:
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v4, s2
-; GFX1032-THRESH5-NEXT: s_waitcnt_vscnt null, 0x0
-; GFX1032-THRESH5-NEXT: ds_sub_rtn_u32 v4, v0, v4
-; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-THRESH5-NEXT: buffer_gl0_inv
-; GFX1032-THRESH5-NEXT: .LBB1_5:
-; GFX1032-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1032-THRESH5-NEXT: s_or_b32 exec_lo, exec_lo, s0
-; GFX1032-THRESH5-NEXT: v_readfirstlane_b32 s0, v4
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, v3
-; GFX1032-THRESH5-NEXT: v_sub_nc_u32_e32 v4, s0, v0
-; GFX1032-THRESH5-NEXT: .LBB1_6:
-; GFX1032-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1032-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1032-THRESH5-NEXT: s_mov_b32 s2, -1
-; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-THRESH5-NEXT: buffer_store_dword v4, off, s[0:3], 0
-; GFX1032-THRESH5-NEXT: s_endpgm
+define amdgpu_kernel void @sub_i32_varying(ptr addrspace(1) %out) #0 {
+; GFX1032-LABEL: sub_i32_varying:
+; GFX1032: ; %bb.0: ; %entry
+; GFX1032-NEXT: s_bcnt1_i32_b32 s0, exec_lo
+; GFX1032-NEXT: s_cmp_lt_u32 s0, 6
+; GFX1032-NEXT: s_cbranch_scc0 .LBB1_2
+; GFX1032-NEXT: ; %bb.1:
+; GFX1032-NEXT: v_mov_b32_e32 v4, 0
+; GFX1032-NEXT: ds_sub_rtn_u32 v4, v4, v0
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_gl0_inv
+; GFX1032-NEXT: s_cbranch_execz .LBB1_3
+; GFX1032-NEXT: s_branch .LBB1_6
+; GFX1032-NEXT: .LBB1_2:
+; GFX1032-NEXT: ; implicit-def: $vgpr4
+; GFX1032-NEXT: .LBB1_3:
+; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1032-NEXT: v_mov_b32_e32 v3, 0
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1032-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1032-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-NEXT: v_readlane_b32 s1, v1, 15
+; GFX1032-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1032-NEXT: v_mov_b32_e32 v0, 0
+; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-NEXT: v_writelane_b32 v3, s1, 16
+; GFX1032-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
+; GFX1032-NEXT: ; implicit-def: $vgpr4
+; GFX1032-NEXT: s_and_saveexec_b32 s0, vcc_lo
+; GFX1032-NEXT: s_cbranch_execz .LBB1_5
+; GFX1032-NEXT: ; %bb.4:
+; GFX1032-NEXT: v_mov_b32_e32 v4, s2
+; GFX1032-NEXT: s_waitcnt_vscnt null, 0x0
+; GFX1032-NEXT: ds_sub_rtn_u32 v4, v0, v4
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_gl0_inv
+; GFX1032-NEXT: .LBB1_5:
+; GFX1032-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-NEXT: s_or_b32 exec_lo, exec_lo, s0
+; GFX1032-NEXT: v_readfirstlane_b32 s0, v4
+; GFX1032-NEXT: v_mov_b32_e32 v0, v3
+; GFX1032-NEXT: v_sub_nc_u32_e32 v4, s0, v0
+; GFX1032-NEXT: .LBB1_6:
+; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1032-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1032-NEXT: s_mov_b32 s2, -1
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_store_dword v4, off, s[0:3], 0
+; GFX1032-NEXT: s_endpgm
;
-; GFX1064-THRESH5-LABEL: sub_i32_varying:
-; GFX1064-THRESH5: ; %bb.0: ; %entry
-; GFX1064-THRESH5-NEXT: s_bcnt1_i32_b64 s0, exec
-; GFX1064-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
-; GFX1064-THRESH5-NEXT: s_cbranch_scc0 .LBB1_2
-; GFX1064-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v4, 0
-; GFX1064-THRESH5-NEXT: ds_sub_rtn_u32 v4, v4, v0
-; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-THRESH5-NEXT: buffer_gl0_inv
-; GFX1064-THRESH5-NEXT: s_cbranch_execz .LBB1_3
-; GFX1064-THRESH5-NEXT: s_branch .LBB1_6
-; GFX1064-THRESH5-NEXT: .LBB1_2:
-; GFX1064-THRESH5-NEXT: ; implicit-def: $vgpr4
-; GFX1064-THRESH5-NEXT: .LBB1_3:
-; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v3, 0
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-THRESH5-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v2, s2
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 15
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s3, v1, 31
-; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s2, 16
-; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 63
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s6, v1, 47
-; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s3, 32
-; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, 0
-; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s6, 48
-; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX1064-THRESH5-NEXT: ; implicit-def: $vgpr4
-; GFX1064-THRESH5-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX1064-THRESH5-NEXT: s_cbranch_execz .LBB1_5
-; GFX1064-THRESH5-NEXT: ; %bb.4:
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v4, s2
-; GFX1064-THRESH5-NEXT: s_waitcnt_vscnt null, 0x0
-; GFX1064-THRESH5-NEXT: ds_sub_rtn_u32 v4, v0, v4
-; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-THRESH5-NEXT: buffer_gl0_inv
-; GFX1064-THRESH5-NEXT: .LBB1_5:
-; GFX1064-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1064-THRESH5-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX1064-THRESH5-NEXT: v_readfirstlane_b32 s0, v4
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, v3
-; GFX1064-THRESH5-NEXT: v_sub_nc_u32_e32 v4, s0, v0
-; GFX1064-THRESH5-NEXT: .LBB1_6:
-; GFX1064-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1064-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1064-THRESH5-NEXT: s_mov_b32 s2, -1
-; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-THRESH5-NEXT: buffer_store_dword v4, off, s[0:3], 0
-; GFX1064-THRESH5-NEXT: s_endpgm
+; GFX1064-LABEL: sub_i32_varying:
+; GFX1064: ; %bb.0: ; %entry
+; GFX1064-NEXT: s_bcnt1_i32_b64 s0, exec
+; GFX1064-NEXT: s_cmp_lt_u32 s0, 6
+; GFX1064-NEXT: s_cbranch_scc0 .LBB1_2
+; GFX1064-NEXT: ; %bb.1:
+; GFX1064-NEXT: v_mov_b32_e32 v4, 0
+; GFX1064-NEXT: ds_sub_rtn_u32 v4, v4, v0
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: s_cbranch_execz .LBB1_3
+; GFX1064-NEXT: s_branch .LBB1_6
+; GFX1064-NEXT: .LBB1_2:
+; GFX1064-NEXT: ; implicit-def: $vgpr4
+; GFX1064-NEXT: .LBB1_3:
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1064-NEXT: v_mov_b32_e32 v3, 0
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1064-NEXT: v_mov_b32_e32 v2, s2
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 15
+; GFX1064-NEXT: v_readlane_b32 s3, v1, 31
+; GFX1064-NEXT: v_writelane_b32 v3, s2, 16
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 63
+; GFX1064-NEXT: v_readlane_b32 s6, v1, 47
+; GFX1064-NEXT: v_writelane_b32 v3, s3, 32
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
+; GFX1064-NEXT: v_mov_b32_e32 v0, 0
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_writelane_b32 v3, s6, 48
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX1064-NEXT: ; implicit-def: $vgpr4
+; GFX1064-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1064-NEXT: s_cbranch_execz .LBB1_5
+; GFX1064-NEXT: ; %bb.4:
+; GFX1064-NEXT: v_mov_b32_e32 v4, s2
+; GFX1064-NEXT: s_waitcnt_vscnt null, 0x0
+; GFX1064-NEXT: ds_sub_rtn_u32 v4, v0, v4
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: .LBB1_5:
+; GFX1064-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1064-NEXT: v_readfirstlane_b32 s0, v4
+; GFX1064-NEXT: v_mov_b32_e32 v0, v3
+; GFX1064-NEXT: v_sub_nc_u32_e32 v4, s0, v0
+; GFX1064-NEXT: .LBB1_6:
+; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-NEXT: s_mov_b32 s2, -1
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_store_dword v4, off, s[0:3], 0
+; GFX1064-NEXT: s_endpgm
;
-; GFX900-THRESH5-LABEL: sub_i32_varying:
-; GFX900-THRESH5: ; %bb.0: ; %entry
-; GFX900-THRESH5-NEXT: s_bcnt1_i32_b64 s0, exec
-; GFX900-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
-; GFX900-THRESH5-NEXT: s_cbranch_scc0 .LBB1_2
-; GFX900-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v3, 0
-; GFX900-THRESH5-NEXT: ds_sub_rtn_u32 v3, v3, v0
-; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-THRESH5-NEXT: s_cbranch_execz .LBB1_3
-; GFX900-THRESH5-NEXT: s_branch .LBB1_6
-; GFX900-THRESH5-NEXT: .LBB1_2:
-; GFX900-THRESH5-NEXT: ; implicit-def: $vgpr3
-; GFX900-THRESH5-NEXT: .LBB1_3:
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v3, 0
-; GFX900-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX900-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX900-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX900-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v2, 0
-; GFX900-THRESH5-NEXT: s_nop 0
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX900-THRESH5-NEXT: v_readlane_b32 s2, v1, 63
-; GFX900-THRESH5-NEXT: s_nop 0
-; GFX900-THRESH5-NEXT: v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX900-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
-; GFX900-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX900-THRESH5-NEXT: ; implicit-def: $vgpr0
-; GFX900-THRESH5-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX900-THRESH5-NEXT: s_cbranch_execz .LBB1_5
-; GFX900-THRESH5-NEXT: ; %bb.4:
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, s2
-; GFX900-THRESH5-NEXT: ds_sub_rtn_u32 v0, v3, v0
-; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-THRESH5-NEXT: .LBB1_5:
-; GFX900-THRESH5-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX900-THRESH5-NEXT: v_readfirstlane_b32 s0, v0
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, v2
-; GFX900-THRESH5-NEXT: v_sub_u32_e32 v3, s0, v0
-; GFX900-THRESH5-NEXT: .LBB1_6:
-; GFX900-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX900-THRESH5-NEXT: s_mov_b32 s3, 0xf000
-; GFX900-THRESH5-NEXT: s_mov_b32 s2, -1
-; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-THRESH5-NEXT: buffer_store_dword v3, off, s[0:3], 0
-; GFX900-THRESH5-NEXT: s_endpgm
-;
-; GFX1032-DISABLED-LABEL: sub_i32_varying:
-; GFX1032-DISABLED: ; %bb.0: ; %entry
-; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-DISABLED-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v3, 0
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-DISABLED-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-DISABLED-NEXT: v_readlane_b32 s1, v1, 15
-; GFX1032-DISABLED-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v4, 0
-; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-DISABLED-NEXT: v_writelane_b32 v3, s1, 16
-; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-DISABLED-NEXT: s_mov_b32 s0, s2
-; GFX1032-DISABLED-NEXT: s_mov_b32 s2, -1
-; GFX1032-DISABLED-NEXT: ; implicit-def: $vgpr0
-; GFX1032-DISABLED-NEXT: s_and_saveexec_b32 s1, vcc_lo
-; GFX1032-DISABLED-NEXT: s_cbranch_execz .LBB1_2
-; GFX1032-DISABLED-NEXT: ; %bb.1:
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, s0
-; GFX1032-DISABLED-NEXT: ds_sub_rtn_u32 v0, v4, v0
-; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-DISABLED-NEXT: buffer_gl0_inv
-; GFX1032-DISABLED-NEXT: .LBB1_2:
-; GFX1032-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1032-DISABLED-NEXT: s_or_b32 exec_lo, exec_lo, s1
-; GFX1032-DISABLED-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1032-DISABLED-NEXT: v_readfirstlane_b32 s3, v0
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, v3
-; GFX1032-DISABLED-NEXT: v_sub_nc_u32_e32 v0, s3, v0
-; GFX1032-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1032-DISABLED-NEXT: s_endpgm
-;
-; GFX1064-DISABLED-LABEL: sub_i32_varying:
-; GFX1064-DISABLED: ; %bb.0: ; %entry
-; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-DISABLED-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v3, 0
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-DISABLED-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v2, s2
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 15
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s3, v1, 31
-; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s2, 16
-; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 47
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s6, v1, 63
-; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s3, 32
-; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-DISABLED-NEXT: v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v4, 0
-; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s2, 48
-; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-DISABLED-NEXT: s_mov_b32 s2, -1
-; GFX1064-DISABLED-NEXT: ; implicit-def: $vgpr0
-; GFX1064-DISABLED-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX1064-DISABLED-NEXT: s_cbranch_execz .LBB1_2
-; GFX1064-DISABLED-NEXT: ; %bb.1:
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, s6
-; GFX1064-DISABLED-NEXT: ds_sub_rtn_u32 v0, v4, v0
-; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-DISABLED-NEXT: buffer_gl0_inv
-; GFX1064-DISABLED-NEXT: .LBB1_2:
-; GFX1064-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1064-DISABLED-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX1064-DISABLED-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1064-DISABLED-NEXT: v_readfirstlane_b32 s3, v0
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, v3
-; GFX1064-DISABLED-NEXT: v_sub_nc_u32_e32 v0, s3, v0
-; GFX1064-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1064-DISABLED-NEXT: s_endpgm
+; GFX900-LABEL: sub_i32_varying:
+; GFX900: ; %bb.0: ; %entry
+; GFX900-NEXT: s_bcnt1_i32_b64 s0, exec
+; GFX900-NEXT: s_cmp_lt_u32 s0, 6
+; GFX900-NEXT: s_cbranch_scc0 .LBB1_2
+; GFX900-NEXT: ; %bb.1:
+; GFX900-NEXT: v_mov_b32_e32 v3, 0
+; GFX900-NEXT: ds_sub_rtn_u32 v3, v3, v0
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: s_cbranch_execz .LBB1_3
+; GFX900-NEXT: s_branch .LBB1_6
+; GFX900-NEXT: .LBB1_2:
+; GFX900-NEXT: ; implicit-def: $vgpr3
+; GFX900-NEXT: .LBB1_3:
+; GFX900-NEXT: v_mov_b32_e32 v3, 0
+; GFX900-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX900-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v4
+; GFX900-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX900-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX900-NEXT: v_mov_b32_e32 v2, 0
+; GFX900-NEXT: s_nop 0
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
+; GFX900-NEXT: v_readlane_b32 s2, v1, 63
+; GFX900-NEXT: s_nop 0
+; GFX900-NEXT: v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
+; GFX900-NEXT: s_mov_b64 exec, s[0:1]
+; GFX900-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX900-NEXT: ; implicit-def: $vgpr0
+; GFX900-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX900-NEXT: s_cbranch_execz .LBB1_5
+; GFX900-NEXT: ; %bb.4:
+; GFX900-NEXT: v_mov_b32_e32 v0, s2
+; GFX900-NEXT: ds_sub_rtn_u32 v0, v3, v0
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: .LBB1_5:
+; GFX900-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX900-NEXT: v_readfirstlane_b32 s0, v0
+; GFX900-NEXT: v_mov_b32_e32 v0, v2
+; GFX900-NEXT: v_sub_u32_e32 v3, s0, v0
+; GFX900-NEXT: .LBB1_6:
+; GFX900-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX900-NEXT: s_mov_b32 s3, 0xf000
+; GFX900-NEXT: s_mov_b32 s2, -1
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: buffer_store_dword v3, off, s[0:3], 0
+; GFX900-NEXT: s_endpgm
entry:
%lane = call i32 @llvm.amdgcn.workitem.id.x()
%old = atomicrmw sub ptr addrspace(3) @local_var32, i32 %lane acq_rel
@@ -590,189 +399,135 @@ entry:
}
; Test 3: divergent i32 add without result used (NeedResult=false)
-define amdgpu_kernel void @add_i32_varying_nouse() {
-; GFX1032-THRESH5-LABEL: add_i32_varying_nouse:
-; GFX1032-THRESH5: ; %bb.0: ; %entry
-; GFX1032-THRESH5-NEXT: s_bcnt1_i32_b32 s0, exec_lo
-; GFX1032-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
-; GFX1032-THRESH5-NEXT: s_mov_b32 s0, -1
-; GFX1032-THRESH5-NEXT: s_cbranch_scc0 .LBB2_2
-; GFX1032-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v3, 0
-; GFX1032-THRESH5-NEXT: s_mov_b32 s0, 0
-; GFX1032-THRESH5-NEXT: ds_add_u32 v3, v0
-; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-THRESH5-NEXT: buffer_gl0_inv
-; GFX1032-THRESH5-NEXT: .LBB2_2: ; %Flow1
-; GFX1032-THRESH5-NEXT: s_andn2_b32 vcc_lo, exec_lo, s0
-; GFX1032-THRESH5-NEXT: s_cbranch_vccnz .LBB2_5
-; GFX1032-THRESH5-NEXT: ; %bb.3:
-; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-THRESH5-NEXT: v_permlanex16_b32 v2, v1, 0, 0
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_e32 v1, v1, v2
-; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, 0
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v3, v1
-; GFX1032-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX1032-THRESH5-NEXT: s_and_saveexec_b32 s0, vcc_lo
-; GFX1032-THRESH5-NEXT: s_cbranch_execz .LBB2_5
-; GFX1032-THRESH5-NEXT: ; %bb.4:
-; GFX1032-THRESH5-NEXT: s_waitcnt_vscnt null, 0x0
-; GFX1032-THRESH5-NEXT: ds_add_u32 v0, v3
-; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-THRESH5-NEXT: buffer_gl0_inv
-; GFX1032-THRESH5-NEXT: .LBB2_5:
-; GFX1032-THRESH5-NEXT: s_endpgm
-;
-; GFX1064-THRESH5-LABEL: add_i32_varying_nouse:
-; GFX1064-THRESH5: ; %bb.0: ; %entry
-; GFX1064-THRESH5-NEXT: s_bcnt1_i32_b64 s0, exec
-; GFX1064-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
-; GFX1064-THRESH5-NEXT: s_mov_b64 s[0:1], -1
-; GFX1064-THRESH5-NEXT: s_cbranch_scc0 .LBB2_2
-; GFX1064-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v3, 0
-; GFX1064-THRESH5-NEXT: s_mov_b64 s[0:1], 0
-; GFX1064-THRESH5-NEXT: ds_add_u32 v3, v0
-; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-THRESH5-NEXT: buffer_gl0_inv
-; GFX1064-THRESH5-NEXT: .LBB2_2: ; %Flow1
-; GFX1064-THRESH5-NEXT: s_andn2_b64 vcc, exec, s[0:1]
-; GFX1064-THRESH5-NEXT: s_cbranch_vccnz .LBB2_5
-; GFX1064-THRESH5-NEXT: ; %bb.3:
-; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-THRESH5-NEXT: v_permlanex16_b32 v2, v1, 0, 0
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_e32 v1, v1, v2
-; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 0
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s3, v1, 32
-; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v3, exec_hi, v0
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, 0
-; GFX1064-THRESH5-NEXT: s_add_i32 s0, s2, s3
-; GFX1064-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v3
-; GFX1064-THRESH5-NEXT: s_and_saveexec_b64 s[2:3], vcc
-; GFX1064-THRESH5-NEXT: s_cbranch_execz .LBB2_5
-; GFX1064-THRESH5-NEXT: ; %bb.4:
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v3, s0
-; GFX1064-THRESH5-NEXT: s_waitcnt_vscnt null, 0x0
-; GFX1064-THRESH5-NEXT: ds_add_u32 v0, v3
-; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-THRESH5-NEXT: buffer_gl0_inv
-; GFX1064-THRESH5-NEXT: .LBB2_5:
-; GFX1064-THRESH5-NEXT: s_endpgm
-;
-; GFX900-THRESH5-LABEL: add_i32_varying_nouse:
-; GFX900-THRESH5: ; %bb.0: ; %entry
-; GFX900-THRESH5-NEXT: s_bcnt1_i32_b64 s0, exec
-; GFX900-THRESH5-NEXT: s_cmp_lt_u32 s0, 6
-; GFX900-THRESH5-NEXT: s_mov_b64 s[0:1], -1
-; GFX900-THRESH5-NEXT: s_cbranch_scc0 .LBB2_2
-; GFX900-THRESH5-NEXT: ; %bb.1: ; %atomicrmw.no_opt
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v2, 0
-; GFX900-THRESH5-NEXT: ds_add_u32 v2, v0
-; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-THRESH5-NEXT: s_mov_b64 s[0:1], 0
-; GFX900-THRESH5-NEXT: .LBB2_2: ; %Flow1
-; GFX900-THRESH5-NEXT: s_andn2_b64 vcc, exec, s[0:1]
-; GFX900-THRESH5-NEXT: s_cbranch_vccnz .LBB2_5
-; GFX900-THRESH5-NEXT: ; %bb.3:
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v2, 0
-; GFX900-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX900-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX900-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX900-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX900-THRESH5-NEXT: v_readlane_b32 s2, v1, 63
-; GFX900-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
-; GFX900-THRESH5-NEXT: s_mov_b32 s0, s2
-; GFX900-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v3
-; GFX900-THRESH5-NEXT: s_and_saveexec_b64 s[2:3], vcc
-; GFX900-THRESH5-NEXT: s_cbranch_execz .LBB2_5
-; GFX900-THRESH5-NEXT: ; %bb.4:
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, s0
-; GFX900-THRESH5-NEXT: ds_add_u32 v2, v0
-; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-THRESH5-NEXT: .LBB2_5:
-; GFX900-THRESH5-NEXT: s_endpgm
+define amdgpu_kernel void @add_i32_varying_nouse() #0 {
+; GFX1032-LABEL: add_i32_varying_nouse:
+; GFX1032: ; %bb.0: ; %entry
+; GFX1032-NEXT: s_bcnt1_i32_b32 s0, exec_lo
+; GFX1032-NEXT: s_cmp_lt_u32 s0, 6
+; GFX1032-NEXT: s_mov_b32 s0, -1
+; GFX1032-NEXT: s_cbranch_scc0 .LBB2_2
+; GFX1032-NEXT: ; %bb.1:
+; GFX1032-NEXT: v_mov_b32_e32 v3, 0
+; GFX1032-NEXT: s_mov_b32 s0, 0
+; GFX1032-NEXT: ds_add_u32 v3, v0
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_gl0_inv
+; GFX1032-NEXT: .LBB2_2: ; %Flow1
+; GFX1032-NEXT: s_andn2_b32 vcc_lo, exec_lo, s0
+; GFX1032-NEXT: s_cbranch_vccnz .LBB2_5
+; GFX1032-NEXT: ; %bb.3:
+; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_permlanex16_b32 v2, v1, 0, 0
+; GFX1032-NEXT: v_add_nc_u32_e32 v1, v1, v2
+; GFX1032-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1032-NEXT: v_mov_b32_e32 v0, 0
+; GFX1032-NEXT: v_mov_b32_e32 v3, v1
+; GFX1032-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
+; GFX1032-NEXT: s_and_saveexec_b32 s0, vcc_lo
+; GFX1032-NEXT: s_cbranch_execz .LBB2_5
+; GFX1032-NEXT: ; %bb.4:
+; GFX1032-NEXT: s_waitcnt_vscnt null, 0x0
+; GFX1032-NEXT: ds_add_u32 v0, v3
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_gl0_inv
+; GFX1032-NEXT: .LBB2_5:
+; GFX1032-NEXT: s_endpgm
;
-; GFX1032-DISABLED-LABEL: add_i32_varying_nouse:
-; GFX1032-DISABLED: ; %bb.0: ; %entry
-; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-DISABLED-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-DISABLED-NEXT: v_permlanex16_b32 v2, v1, 0, 0
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_e32 v1, v1, v2
-; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, 0
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v3, v1
-; GFX1032-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX1032-DISABLED-NEXT: s_and_saveexec_b32 s0, vcc_lo
-; GFX1032-DISABLED-NEXT: s_cbranch_execz .LBB2_2
-; GFX1032-DISABLED-NEXT: ; %bb.1:
-; GFX1032-DISABLED-NEXT: ds_add_u32 v0, v3
-; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-DISABLED-NEXT: buffer_gl0_inv
-; GFX1032-DISABLED-NEXT: .LBB2_2:
-; GFX1032-DISABLED-NEXT: s_endpgm
+; GFX1064-LABEL: add_i32_varying_nouse:
+; GFX1064: ; %bb.0: ; %entry
+; GFX1064-NEXT: s_bcnt1_i32_b64 s0, exec
+; GFX1064-NEXT: s_cmp_lt_u32 s0, 6
+; GFX1064-NEXT: s_mov_b64 s[0:1], -1
+; GFX1064-NEXT: s_cbranch_scc0 .LBB2_2
+; GFX1064-NEXT: ; %bb.1:
+; GFX1064-NEXT: v_mov_b32_e32 v3, 0
+; GFX1064-NEXT: s_mov_b64 s[0:1], 0
+; GFX1064-NEXT: ds_add_u32 v3, v0
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: .LBB2_2: ; %Flow1
+; GFX1064-NEXT: s_andn2_b64 vcc, exec, s[0:1]
+; GFX1064-NEXT: s_cbranch_vccnz .LBB2_5
+; GFX1064-NEXT: ; %bb.3:
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_permlanex16_b32 v2, v1, 0, 0
+; GFX1064-NEXT: v_add_nc_u32_e32 v1, v1, v2
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 0
+; GFX1064-NEXT: v_readlane_b32 s3, v1, 32
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v3, exec_hi, v0
+; GFX1064-NEXT: v_mov_b32_e32 v0, 0
+; GFX1064-NEXT: s_add_i32 s0, s2, s3
+; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX1064-NEXT: s_and_saveexec_b64 s[2:3], vcc
+; GFX1064-NEXT: s_cbranch_execz .LBB2_5
+; GFX1064-NEXT: ; %bb.4:
+; GFX1064-NEXT: v_mov_b32_e32 v3, s0
+; GFX1064-NEXT: s_waitcnt_vscnt null, 0x0
+; GFX1064-NEXT: ds_add_u32 v0, v3
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: .LBB2_5:
+; GFX1064-NEXT: s_endpgm
;
-; GFX1064-DISABLED-LABEL: add_i32_varying_nouse:
-; GFX1064-DISABLED: ; %bb.0: ; %entry
-; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-DISABLED-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-DISABLED-NEXT: v_permlanex16_b32 v2, v1, 0, 0
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_e32 v1, v1, v2
-; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 0
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s3, v1, 32
-; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-DISABLED-NEXT: v_mbcnt_hi_u32_b32 v3, exec_hi, v0
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, 0
-; GFX1064-DISABLED-NEXT: s_add_i32 s0, s2, s3
-; GFX1064-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc, 0, v3
-; GFX1064-DISABLED-NEXT: s_and_saveexec_b64 s[2:3], vcc
-; GFX1064-DISABLED-NEXT: s_cbranch_execz .LBB2_2
-; GFX1064-DISABLED-NEXT: ; %bb.1:
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v3, s0
-; GFX1064-DISABLED-NEXT: ds_add_u32 v0, v3
-; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-DISABLED-NEXT: buffer_gl0_inv
-; GFX1064-DISABLED-NEXT: .LBB2_2:
-; GFX1064-DISABLED-NEXT: s_endpgm
+; GFX900-LABEL: add_i32_varying_nouse:
+; GFX900: ; %bb.0: ; %entry
+; GFX900-NEXT: s_bcnt1_i32_b64 s0, exec
+; GFX900-NEXT: s_cmp_lt_u32 s0, 6
+; GFX900-NEXT: s_mov_b64 s[0:1], -1
+; GFX900-NEXT: s_cbranch_scc0 .LBB2_2
+; GFX900-NEXT: ; %bb.1:
+; GFX900-NEXT: v_mov_b32_e32 v2, 0
+; GFX900-NEXT: ds_add_u32 v2, v0
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: s_mov_b64 s[0:1], 0
+; GFX900-NEXT: .LBB2_2: ; %Flow1
+; GFX900-NEXT: s_andn2_b64 vcc, exec, s[0:1]
+; GFX900-NEXT: s_cbranch_vccnz .LBB2_5
+; GFX900-NEXT: ; %bb.3:
+; GFX900-NEXT: v_mov_b32_e32 v2, 0
+; GFX900-NEXT: v_mbcnt_lo_u32_b32 v3, exec_lo, 0
+; GFX900-NEXT: v_mbcnt_hi_u32_b32 v3, exec_hi, v3
+; GFX900-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX900-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
+; GFX900-NEXT: v_readlane_b32 s2, v1, 63
+; GFX900-NEXT: s_mov_b64 exec, s[0:1]
+; GFX900-NEXT: s_mov_b32 s0, s2
+; GFX900-NEXT: v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX900-NEXT: s_and_saveexec_b64 s[2:3], vcc
+; GFX900-NEXT: s_cbranch_execz .LBB2_5
+; GFX900-NEXT: ; %bb.4:
+; GFX900-NEXT: v_mov_b32_e32 v0, s0
+; GFX900-NEXT: ds_add_u32 v2, v0
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: .LBB2_5:
+; GFX900-NEXT: s_endpgm
entry:
%lane = call i32 @llvm.amdgcn.workitem.id.x()
%old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel
@@ -780,158 +535,97 @@ entry:
}
; Test 4: uniform i32 add -- should NOT be affected by threshold (not divergent)
-define amdgpu_kernel void @add_i32_uniform(ptr addrspace(1) %out, i32 %additive) {
-; GFX1032-THRESH5-LABEL: add_i32_uniform:
-; GFX1032-THRESH5: ; %bb.0: ; %entry
-; GFX1032-THRESH5-NEXT: s_load_dword s0, s[4:5], 0x2c
-; GFX1032-THRESH5-NEXT: s_mov_b32 s2, exec_lo
-; GFX1032-THRESH5-NEXT: ; implicit-def: $vgpr1
-; GFX1032-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, s2, 0
-; GFX1032-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-THRESH5-NEXT: s_and_saveexec_b32 s1, vcc_lo
-; GFX1032-THRESH5-NEXT: s_cbranch_execz .LBB3_2
-; GFX1032-THRESH5-NEXT: ; %bb.1:
-; GFX1032-THRESH5-NEXT: s_bcnt1_i32_b32 s2, s2
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v1, 0
-; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-THRESH5-NEXT: s_mul_i32 s2, s0, s2
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v2, s2
-; GFX1032-THRESH5-NEXT: ds_add_rtn_u32 v1, v1, v2
-; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-THRESH5-NEXT: buffer_gl0_inv
-; GFX1032-THRESH5-NEXT: .LBB3_2:
-; GFX1032-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1032-THRESH5-NEXT: s_or_b32 exec_lo, exec_lo, s1
-; GFX1032-THRESH5-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x24
-; GFX1032-THRESH5-NEXT: v_readfirstlane_b32 s2, v1
-; GFX1032-THRESH5-NEXT: s_mov_b32 s11, 0x31016000
-; GFX1032-THRESH5-NEXT: s_mov_b32 s10, -1
-; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-THRESH5-NEXT: v_mad_u64_u32 v[0:1], s0, s0, v0, s[2:3]
-; GFX1032-THRESH5-NEXT: buffer_store_dword v0, off, s[8:11], 0
-; GFX1032-THRESH5-NEXT: s_endpgm
+define amdgpu_kernel void @add_i32_uniform(ptr addrspace(1) %out, i32 %additive) #0 {
+; GFX1032-LABEL: add_i32_uniform:
+; GFX1032: ; %bb.0: ; %entry
+; GFX1032-NEXT: s_load_dword s0, s[4:5], 0x2c
+; GFX1032-NEXT: s_mov_b32 s2, exec_lo
+; GFX1032-NEXT: ; implicit-def: $vgpr1
+; GFX1032-NEXT: v_mbcnt_lo_u32_b32 v0, s2, 0
+; GFX1032-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
+; GFX1032-NEXT: s_and_saveexec_b32 s1, vcc_lo
+; GFX1032-NEXT: s_cbranch_execz .LBB3_2
+; GFX1032-NEXT: ; %bb.1:
+; GFX1032-NEXT: s_bcnt1_i32_b32 s2, s2
+; GFX1032-NEXT: v_mov_b32_e32 v1, 0
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: s_mul_i32 s2, s0, s2
+; GFX1032-NEXT: v_mov_b32_e32 v2, s2
+; GFX1032-NEXT: ds_add_rtn_u32 v1, v1, v2
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_gl0_inv
+; GFX1032-NEXT: .LBB3_2:
+; GFX1032-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-NEXT: s_or_b32 exec_lo, exec_lo, s1
+; GFX1032-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x24
+; GFX1032-NEXT: v_readfirstlane_b32 s2, v1
+; GFX1032-NEXT: s_mov_b32 s11, 0x31016000
+; GFX1032-NEXT: s_mov_b32 s10, -1
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: v_mad_u64_u32 v[0:1], s0, s0, v0, s[2:3]
+; GFX1032-NEXT: buffer_store_dword v0, off, s[8:11], 0
+; GFX1032-NEXT: s_endpgm
;
-; GFX1064-THRESH5-LABEL: add_i32_uniform:
-; GFX1064-THRESH5: ; %bb.0: ; %entry
-; GFX1064-THRESH5-NEXT: s_load_dword s6, s[4:5], 0x2c
-; GFX1064-THRESH5-NEXT: s_mov_b64 s[2:3], exec
-; GFX1064-THRESH5-NEXT: ; implicit-def: $vgpr1
-; GFX1064-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, s2, 0
-; GFX1064-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v0, s3, v0
-; GFX1064-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-THRESH5-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX1064-THRESH5-NEXT: s_cbranch_execz .LBB3_2
-; GFX1064-THRESH5-NEXT: ; %bb.1:
-; GFX1064-THRESH5-NEXT: s_bcnt1_i32_b64 s2, s[2:3]
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v1, 0
-; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-THRESH5-NEXT: s_mul_i32 s2, s6, s2
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v2, s2
-; GFX1064-THRESH5-NEXT: ds_add_rtn_u32 v1, v1, v2
-; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-THRESH5-NEXT: buffer_gl0_inv
-; GFX1064-THRESH5-NEXT: .LBB3_2:
-; GFX1064-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1064-THRESH5-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX1064-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1064-THRESH5-NEXT: v_readfirstlane_b32 s2, v1
-; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-THRESH5-NEXT: v_mad_u64_u32 v[0:1], s[2:3], s6, v0, s[2:3]
-; GFX1064-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1064-THRESH5-NEXT: s_mov_b32 s2, -1
-; GFX1064-THRESH5-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1064-THRESH5-NEXT: s_endpgm
+; GFX1064-LABEL: add_i32_uniform:
+; GFX1064: ; %bb.0: ; %entry
+; GFX1064-NEXT: s_load_dword s6, s[4:5], 0x2c
+; GFX1064-NEXT: s_mov_b64 s[2:3], exec
+; GFX1064-NEXT: ; implicit-def: $vgpr1
+; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, s2, 0
+; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v0, s3, v0
+; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX1064-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1064-NEXT: s_cbranch_execz .LBB3_2
+; GFX1064-NEXT: ; %bb.1:
+; GFX1064-NEXT: s_bcnt1_i32_b64 s2, s[2:3]
+; GFX1064-NEXT: v_mov_b32_e32 v1, 0
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: s_mul_i32 s2, s6, s2
+; GFX1064-NEXT: v_mov_b32_e32 v2, s2
+; GFX1064-NEXT: ds_add_rtn_u32 v1, v1, v2
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: .LBB3_2:
+; GFX1064-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-NEXT: v_readfirstlane_b32 s2, v1
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: v_mad_u64_u32 v[0:1], s[2:3], s6, v0, s[2:3]
+; GFX1064-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-NEXT: s_mov_b32 s2, -1
+; GFX1064-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1064-NEXT: s_endpgm
;
-; GFX900-THRESH5-LABEL: add_i32_uniform:
-; GFX900-THRESH5: ; %bb.0: ; %entry
-; GFX900-THRESH5-NEXT: s_load_dword s6, s[4:5], 0x2c
-; GFX900-THRESH5-NEXT: s_mov_b64 s[2:3], exec
-; GFX900-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, s2, 0
-; GFX900-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v0, s3, v0
-; GFX900-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX900-THRESH5-NEXT: ; implicit-def: $vgpr1
-; GFX900-THRESH5-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX900-THRESH5-NEXT: s_cbranch_execz .LBB3_2
-; GFX900-THRESH5-NEXT: ; %bb.1:
-; GFX900-THRESH5-NEXT: s_bcnt1_i32_b64 s2, s[2:3]
-; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-THRESH5-NEXT: s_mul_i32 s2, s6, s2
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v1, 0
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v2, s2
-; GFX900-THRESH5-NEXT: ds_add_rtn_u32 v1, v1, v2
-; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-THRESH5-NEXT: .LBB3_2:
-; GFX900-THRESH5-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX900-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-THRESH5-NEXT: v_mul_lo_u32 v0, s6, v0
-; GFX900-THRESH5-NEXT: v_readfirstlane_b32 s4, v1
-; GFX900-THRESH5-NEXT: s_mov_b32 s3, 0xf000
-; GFX900-THRESH5-NEXT: s_mov_b32 s2, -1
-; GFX900-THRESH5-NEXT: v_add_u32_e32 v0, s4, v0
-; GFX900-THRESH5-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX900-THRESH5-NEXT: s_endpgm
-;
-; GFX1032-DISABLED-LABEL: add_i32_uniform:
-; GFX1032-DISABLED: ; %bb.0: ; %entry
-; GFX1032-DISABLED-NEXT: s_load_dword s0, s[4:5], 0x2c
-; GFX1032-DISABLED-NEXT: s_mov_b32 s2, exec_lo
-; GFX1032-DISABLED-NEXT: ; implicit-def: $vgpr1
-; GFX1032-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, s2, 0
-; GFX1032-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-DISABLED-NEXT: s_and_saveexec_b32 s1, vcc_lo
-; GFX1032-DISABLED-NEXT: s_cbranch_execz .LBB3_2
-; GFX1032-DISABLED-NEXT: ; %bb.1:
-; GFX1032-DISABLED-NEXT: s_bcnt1_i32_b32 s2, s2
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v1, 0
-; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-DISABLED-NEXT: s_mul_i32 s2, s0, s2
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v2, s2
-; GFX1032-DISABLED-NEXT: ds_add_rtn_u32 v1, v1, v2
-; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-DISABLED-NEXT: buffer_gl0_inv
-; GFX1032-DISABLED-NEXT: .LBB3_2:
-; GFX1032-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1032-DISABLED-NEXT: s_or_b32 exec_lo, exec_lo, s1
-; GFX1032-DISABLED-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x24
-; GFX1032-DISABLED-NEXT: v_readfirstlane_b32 s2, v1
-; GFX1032-DISABLED-NEXT: s_mov_b32 s11, 0x31016000
-; GFX1032-DISABLED-NEXT: s_mov_b32 s10, -1
-; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-DISABLED-NEXT: v_mad_u64_u32 v[0:1], s0, s0, v0, s[2:3]
-; GFX1032-DISABLED-NEXT: buffer_store_dword v0, off, s[8:11], 0
-; GFX1032-DISABLED-NEXT: s_endpgm
-;
-; GFX1064-DISABLED-LABEL: add_i32_uniform:
-; GFX1064-DISABLED: ; %bb.0: ; %entry
-; GFX1064-DISABLED-NEXT: s_load_dword s6, s[4:5], 0x2c
-; GFX1064-DISABLED-NEXT: s_mov_b64 s[2:3], exec
-; GFX1064-DISABLED-NEXT: ; implicit-def: $vgpr1
-; GFX1064-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, s2, 0
-; GFX1064-DISABLED-NEXT: v_mbcnt_hi_u32_b32 v0, s3, v0
-; GFX1064-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-DISABLED-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX1064-DISABLED-NEXT: s_cbranch_execz .LBB3_2
-; GFX1064-DISABLED-NEXT: ; %bb.1:
-; GFX1064-DISABLED-NEXT: s_bcnt1_i32_b64 s2, s[2:3]
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v1, 0
-; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-DISABLED-NEXT: s_mul_i32 s2, s6, s2
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v2, s2
-; GFX1064-DISABLED-NEXT: ds_add_rtn_u32 v1, v1, v2
-; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-DISABLED-NEXT: buffer_gl0_inv
-; GFX1064-DISABLED-NEXT: .LBB3_2:
-; GFX1064-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1064-DISABLED-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX1064-DISABLED-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1064-DISABLED-NEXT: v_readfirstlane_b32 s2, v1
-; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-DISABLED-NEXT: v_mad_u64_u32 v[0:1], s[2:3], s6, v0, s[2:3]
-; GFX1064-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1064-DISABLED-NEXT: s_mov_b32 s2, -1
-; GFX1064-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1064-DISABLED-NEXT: s_endpgm
+; GFX900-LABEL: add_i32_uniform:
+; GFX900: ; %bb.0: ; %entry
+; GFX900-NEXT: s_load_dword s6, s[4:5], 0x2c
+; GFX900-NEXT: s_mov_b64 s[2:3], exec
+; GFX900-NEXT: v_mbcnt_lo_u32_b32 v0, s2, 0
+; GFX900-NEXT: v_mbcnt_hi_u32_b32 v0, s3, v0
+; GFX900-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX900-NEXT: ; implicit-def: $vgpr1
+; GFX900-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX900-NEXT: s_cbranch_execz .LBB3_2
+; GFX900-NEXT: ; %bb.1:
+; GFX900-NEXT: s_bcnt1_i32_b64 s2, s[2:3]
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: s_mul_i32 s2, s6, s2
+; GFX900-NEXT: v_mov_b32_e32 v1, 0
+; GFX900-NEXT: v_mov_b32_e32 v2, s2
+; GFX900-NEXT: ds_add_rtn_u32 v1, v1, v2
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: .LBB3_2:
+; GFX900-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX900-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: v_mul_lo_u32 v0, s6, v0
+; GFX900-NEXT: v_readfirstlane_b32 s4, v1
+; GFX900-NEXT: s_mov_b32 s3, 0xf000
+; GFX900-NEXT: s_mov_b32 s2, -1
+; GFX900-NEXT: v_add_u32_e32 v0, s4, v0
+; GFX900-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX900-NEXT: s_endpgm
entry:
%old = atomicrmw add ptr addrspace(3) @local_var32, i32 %additive acq_rel
store i32 %old, ptr addrspace(1) %out
@@ -939,307 +633,186 @@ entry:
}
; Test 5: divergent float fadd -- should NOT be affected by threshold (FP excluded)
-define amdgpu_kernel void @fadd_f32_varying(ptr addrspace(1) %out) {
-; GFX1032-THRESH5-LABEL: fadd_f32_varying:
-; GFX1032-THRESH5: ; %bb.0: ; %entry
-; GFX1032-THRESH5-NEXT: v_cvt_f32_i32_e32 v0, v0
-; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-THRESH5-NEXT: v_bfrev_b32_e32 v1, 1
-; GFX1032-THRESH5-NEXT: v_cndmask_b32_e64 v2, 0x80000000, v0, s0
-; GFX1032-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-THRESH5-NEXT: v_add_f32_e32 v1, v2, v1
-; GFX1032-THRESH5-NEXT: v_bfrev_b32_e32 v2, 1
-; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1032-THRESH5-NEXT: v_add_f32_e32 v1, v1, v3
-; GFX1032-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v2, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1032-THRESH5-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1032-THRESH5-NEXT: v_bfrev_b32_e32 v2, 1
-; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1032-THRESH5-NEXT: v_add_f32_e32 v1, v1, v3
-; GFX1032-THRESH5-NEXT: v_permlanex16_b32 v3, v1, -1, -1
-; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1032-THRESH5-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1032-THRESH5-NEXT: v_readlane_b32 s1, v1, 15
-; GFX1032-THRESH5-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-THRESH5-NEXT: v_writelane_b32 v3, s1, 16
-; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-THRESH5-NEXT: s_mov_b32 s0, s2
-; GFX1032-THRESH5-NEXT: s_mov_b32 s2, -1
-; GFX1032-THRESH5-NEXT: ; implicit-def: $vgpr0
-; GFX1032-THRESH5-NEXT: s_and_saveexec_b32 s1, vcc_lo
-; GFX1032-THRESH5-NEXT: s_cbranch_execz .LBB4_2
-; GFX1032-THRESH5-NEXT: ; %bb.1:
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, 0
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v4, s0
-; GFX1032-THRESH5-NEXT: ds_add_rtn_f32 v0, v0, v4
-; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-THRESH5-NEXT: buffer_gl0_inv
-; GFX1032-THRESH5-NEXT: .LBB4_2:
-; GFX1032-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1032-THRESH5-NEXT: s_or_b32 exec_lo, exec_lo, s1
-; GFX1032-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1032-THRESH5-NEXT: v_readfirstlane_b32 s3, v0
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, v3
-; GFX1032-THRESH5-NEXT: v_add_f32_e32 v0, s3, v0
-; GFX1032-THRESH5-NEXT: v_cndmask_b32_e64 v0, v0, s3, vcc_lo
-; GFX1032-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-THRESH5-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1032-THRESH5-NEXT: s_endpgm
-;
-; GFX1064-THRESH5-LABEL: fadd_f32_varying:
-; GFX1064-THRESH5: ; %bb.0: ; %entry
-; GFX1064-THRESH5-NEXT: v_cvt_f32_i32_e32 v0, v0
-; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-THRESH5-NEXT: v_bfrev_b32_e32 v1, 1
-; GFX1064-THRESH5-NEXT: v_cndmask_b32_e64 v2, 0x80000000, v0, s[0:1]
-; GFX1064-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-THRESH5-NEXT: v_add_f32_e32 v1, v2, v1
-; GFX1064-THRESH5-NEXT: v_bfrev_b32_e32 v2, 1
-; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1064-THRESH5-NEXT: v_add_f32_e32 v1, v1, v3
-; GFX1064-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v2, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1064-THRESH5-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1064-THRESH5-NEXT: v_bfrev_b32_e32 v2, 1
-; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1064-THRESH5-NEXT: v_add_f32_e32 v1, v1, v3
-; GFX1064-THRESH5-NEXT: v_permlanex16_b32 v3, v1, -1, -1
-; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-THRESH5-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1064-THRESH5-NEXT: v_bfrev_b32_e32 v2, 1
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v3, s2
-; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1064-THRESH5-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 15
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s3, v1, 31
-; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s2, 16
-; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 47
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s6, v1, 63
-; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s3, 32
-; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s2, 48
-; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-THRESH5-NEXT: s_mov_b32 s2, -1
-; GFX1064-THRESH5-NEXT: ; implicit-def: $vgpr0
-; GFX1064-THRESH5-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX1064-THRESH5-NEXT: s_cbranch_execz .LBB4_2
-; GFX1064-THRESH5-NEXT: ; %bb.1:
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, 0
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v4, s6
-; GFX1064-THRESH5-NEXT: ds_add_rtn_f32 v0, v0, v4
-; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-THRESH5-NEXT: buffer_gl0_inv
-; GFX1064-THRESH5-NEXT: .LBB4_2:
-; GFX1064-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1064-THRESH5-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX1064-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1064-THRESH5-NEXT: v_readfirstlane_b32 s3, v0
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, v3
-; GFX1064-THRESH5-NEXT: v_add_f32_e32 v0, s3, v0
-; GFX1064-THRESH5-NEXT: v_cndmask_b32_e64 v0, v0, s3, vcc
-; GFX1064-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-THRESH5-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1064-THRESH5-NEXT: s_endpgm
-;
-; GFX900-THRESH5-LABEL: fadd_f32_varying:
-; GFX900-THRESH5: ; %bb.0: ; %entry
-; GFX900-THRESH5-NEXT: v_cvt_f32_i32_e32 v0, v0
-; GFX900-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX900-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX900-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX900-THRESH5-NEXT: v_bfrev_b32_e32 v1, 1
-; GFX900-THRESH5-NEXT: v_cndmask_b32_e64 v2, v1, v0, s[0:1]
-; GFX900-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_mov_b32_dpp v3, v2 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX900-THRESH5-NEXT: v_add_f32_e32 v2, v2, v3
-; GFX900-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_mov_b32_dpp v3, v2 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX900-THRESH5-NEXT: v_add_f32_e32 v2, v2, v3
-; GFX900-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_mov_b32_dpp v3, v2 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX900-THRESH5-NEXT: v_add_f32_e32 v2, v2, v3
-; GFX900-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_mov_b32_dpp v3, v2 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX900-THRESH5-NEXT: v_add_f32_e32 v2, v2, v3
-; GFX900-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_mov_b32_dpp v3, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX900-THRESH5-NEXT: v_add_f32_e32 v2, v2, v3
-; GFX900-THRESH5-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_mov_b32_dpp v3, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX900-THRESH5-NEXT: v_add_f32_e32 v2, v2, v3
-; GFX900-THRESH5-NEXT: v_readlane_b32 s2, v2, 63
-; GFX900-THRESH5-NEXT: s_nop 0
-; GFX900-THRESH5-NEXT: v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX900-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
-; GFX900-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX900-THRESH5-NEXT: ; implicit-def: $vgpr0
-; GFX900-THRESH5-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX900-THRESH5-NEXT: s_cbranch_execz .LBB4_2
-; GFX900-THRESH5-NEXT: ; %bb.1:
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, 0
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v4, s2
-; GFX900-THRESH5-NEXT: ds_add_rtn_f32 v0, v0, v4
-; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-THRESH5-NEXT: .LBB4_2:
-; GFX900-THRESH5-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX900-THRESH5-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX900-THRESH5-NEXT: v_readfirstlane_b32 s4, v0
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, v1
-; GFX900-THRESH5-NEXT: v_add_f32_e32 v0, s4, v0
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v4, s4
-; GFX900-THRESH5-NEXT: s_mov_b32 s3, 0xf000
-; GFX900-THRESH5-NEXT: s_mov_b32 s2, -1
-; GFX900-THRESH5-NEXT: v_cndmask_b32_e32 v0, v0, v4, vcc
-; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-THRESH5-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX900-THRESH5-NEXT: s_endpgm
+define amdgpu_kernel void @fadd_f32_varying(ptr addrspace(1) %out) #0 {
+; GFX1032-LABEL: fadd_f32_varying:
+; GFX1032: ; %bb.0: ; %entry
+; GFX1032-NEXT: v_cvt_f32_i32_e32 v0, v0
+; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-NEXT: v_bfrev_b32_e32 v1, 1
+; GFX1032-NEXT: v_cndmask_b32_e64 v2, 0x80000000, v0, s0
+; GFX1032-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1032-NEXT: v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-NEXT: v_add_f32_e32 v1, v2, v1
+; GFX1032-NEXT: v_bfrev_b32_e32 v2, 1
+; GFX1032-NEXT: v_mov_b32_dpp v3, v1 row_shr:2 row_mask:0xf bank_mask:0xf
+; GFX1032-NEXT: v_add_f32_e32 v1, v1, v3
+; GFX1032-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1032-NEXT: v_mov_b32_dpp v2, v1 row_shr:4 row_mask:0xf bank_mask:0xf
+; GFX1032-NEXT: v_add_f32_e32 v1, v1, v2
+; GFX1032-NEXT: v_bfrev_b32_e32 v2, 1
+; GFX1032-NEXT: v_mov_b32_dpp v3, v1 row_shr:8 row_mask:0xf bank_mask:0xf
+; GFX1032-NEXT: v_add_f32_e32 v1, v1, v3
+; GFX1032-NEXT: v_permlanex16_b32 v3, v1, -1, -1
+; GFX1032-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1032-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1032-NEXT: v_add_f32_e32 v1, v1, v2
+; GFX1032-NEXT: v_readlane_b32 s1, v1, 15
+; GFX1032-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1032-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-NEXT: v_writelane_b32 v3, s1, 16
+; GFX1032-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
+; GFX1032-NEXT: s_mov_b32 s0, s2
+; GFX1032-NEXT: s_mov_b32 s2, -1
+; GFX1032-NEXT: ; implicit-def: $vgpr0
+; GFX1032-NEXT: s_and_saveexec_b32 s1, vcc_lo
+; GFX1032-NEXT: s_cbranch_execz .LBB4_2
+; GFX1032-NEXT: ; %bb.1:
+; GFX1032-NEXT: v_mov_b32_e32 v0, 0
+; GFX1032-NEXT: v_mov_b32_e32 v4, s0
+; GFX1032-NEXT: ds_add_rtn_f32 v0, v0, v4
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_gl0_inv
+; GFX1032-NEXT: .LBB4_2:
+; GFX1032-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-NEXT: s_or_b32 exec_lo, exec_lo, s1
+; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1032-NEXT: v_readfirstlane_b32 s3, v0
+; GFX1032-NEXT: v_mov_b32_e32 v0, v3
+; GFX1032-NEXT: v_add_f32_e32 v0, s3, v0
+; GFX1032-NEXT: v_cndmask_b32_e64 v0, v0, s3, vcc_lo
+; GFX1032-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1032-NEXT: s_endpgm
;
-; GFX1032-DISABLED-LABEL: fadd_f32_varying:
-; GFX1032-DISABLED: ; %bb.0: ; %entry
-; GFX1032-DISABLED-NEXT: v_cvt_f32_i32_e32 v0, v0
-; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-DISABLED-NEXT: v_bfrev_b32_e32 v1, 1
-; GFX1032-DISABLED-NEXT: v_cndmask_b32_e64 v2, 0x80000000, v0, s0
-; GFX1032-DISABLED-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-DISABLED-NEXT: v_add_f32_e32 v1, v2, v1
-; GFX1032-DISABLED-NEXT: v_bfrev_b32_e32 v2, 1
-; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1032-DISABLED-NEXT: v_add_f32_e32 v1, v1, v3
-; GFX1032-DISABLED-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v2, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1032-DISABLED-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1032-DISABLED-NEXT: v_bfrev_b32_e32 v2, 1
-; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1032-DISABLED-NEXT: v_add_f32_e32 v1, v1, v3
-; GFX1032-DISABLED-NEXT: v_permlanex16_b32 v3, v1, -1, -1
-; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-DISABLED-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1032-DISABLED-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1032-DISABLED-NEXT: v_readlane_b32 s1, v1, 15
-; GFX1032-DISABLED-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-DISABLED-NEXT: v_writelane_b32 v3, s1, 16
-; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-DISABLED-NEXT: s_mov_b32 s0, s2
-; GFX1032-DISABLED-NEXT: s_mov_b32 s2, -1
-; GFX1032-DISABLED-NEXT: ; implicit-def: $vgpr0
-; GFX1032-DISABLED-NEXT: s_and_saveexec_b32 s1, vcc_lo
-; GFX1032-DISABLED-NEXT: s_cbranch_execz .LBB4_2
-; GFX1032-DISABLED-NEXT: ; %bb.1:
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, 0
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v4, s0
-; GFX1032-DISABLED-NEXT: ds_add_rtn_f32 v0, v0, v4
-; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-DISABLED-NEXT: buffer_gl0_inv
-; GFX1032-DISABLED-NEXT: .LBB4_2:
-; GFX1032-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1032-DISABLED-NEXT: s_or_b32 exec_lo, exec_lo, s1
-; GFX1032-DISABLED-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1032-DISABLED-NEXT: v_readfirstlane_b32 s3, v0
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, v3
-; GFX1032-DISABLED-NEXT: v_add_f32_e32 v0, s3, v0
-; GFX1032-DISABLED-NEXT: v_cndmask_b32_e64 v0, v0, s3, vcc_lo
-; GFX1032-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1032-DISABLED-NEXT: s_endpgm
+; GFX1064-LABEL: fadd_f32_varying:
+; GFX1064: ; %bb.0: ; %entry
+; GFX1064-NEXT: v_cvt_f32_i32_e32 v0, v0
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_bfrev_b32_e32 v1, 1
+; GFX1064-NEXT: v_cndmask_b32_e64 v2, 0x80000000, v0, s[0:1]
+; GFX1064-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1064-NEXT: v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT: v_add_f32_e32 v1, v2, v1
+; GFX1064-NEXT: v_bfrev_b32_e32 v2, 1
+; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:2 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT: v_add_f32_e32 v1, v1, v3
+; GFX1064-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1064-NEXT: v_mov_b32_dpp v2, v1 row_shr:4 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT: v_add_f32_e32 v1, v1, v2
+; GFX1064-NEXT: v_bfrev_b32_e32 v2, 1
+; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:8 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT: v_add_f32_e32 v1, v1, v3
+; GFX1064-NEXT: v_permlanex16_b32 v3, v1, -1, -1
+; GFX1064-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1064-NEXT: v_add_f32_e32 v1, v1, v2
+; GFX1064-NEXT: v_bfrev_b32_e32 v2, 1
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1064-NEXT: v_mov_b32_e32 v3, s2
+; GFX1064-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1064-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX1064-NEXT: v_add_f32_e32 v1, v1, v2
+; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 15
+; GFX1064-NEXT: v_readlane_b32 s3, v1, 31
+; GFX1064-NEXT: v_writelane_b32 v3, s2, 16
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 47
+; GFX1064-NEXT: v_readlane_b32 s6, v1, 63
+; GFX1064-NEXT: v_writelane_b32 v3, s3, 32
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v0, exec_hi, v0
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_writelane_b32 v3, s2, 48
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX1064-NEXT: s_mov_b32 s2, -1
+; GFX1064-NEXT: ; implicit-def: $vgpr0
+; GFX1064-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1064-NEXT: s_cbranch_execz .LBB4_2
+; GFX1064-NEXT: ; %bb.1:
+; GFX1064-NEXT: v_mov_b32_e32 v0, 0
+; GFX1064-NEXT: v_mov_b32_e32 v4, s6
+; GFX1064-NEXT: ds_add_rtn_f32 v0, v0, v4
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: .LBB4_2:
+; GFX1064-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-NEXT: v_readfirstlane_b32 s3, v0
+; GFX1064-NEXT: v_mov_b32_e32 v0, v3
+; GFX1064-NEXT: v_add_f32_e32 v0, s3, v0
+; GFX1064-NEXT: v_cndmask_b32_e64 v0, v0, s3, vcc
+; GFX1064-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1064-NEXT: s_endpgm
;
-; GFX1064-DISABLED-LABEL: fadd_f32_varying:
-; GFX1064-DISABLED: ; %bb.0: ; %entry
-; GFX1064-DISABLED-NEXT: v_cvt_f32_i32_e32 v0, v0
-; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-DISABLED-NEXT: v_bfrev_b32_e32 v1, 1
-; GFX1064-DISABLED-NEXT: v_cndmask_b32_e64 v2, 0x80000000, v0, s[0:1]
-; GFX1064-DISABLED-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-DISABLED-NEXT: v_add_f32_e32 v1, v2, v1
-; GFX1064-DISABLED-NEXT: v_bfrev_b32_e32 v2, 1
-; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1064-DISABLED-NEXT: v_add_f32_e32 v1, v1, v3
-; GFX1064-DISABLED-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v2, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1064-DISABLED-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1064-DISABLED-NEXT: v_bfrev_b32_e32 v2, 1
-; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1064-DISABLED-NEXT: v_add_f32_e32 v1, v1, v3
-; GFX1064-DISABLED-NEXT: v_permlanex16_b32 v3, v1, -1, -1
-; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-DISABLED-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1064-DISABLED-NEXT: v_bfrev_b32_e32 v2, 1
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v3, s2
-; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-DISABLED-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1064-DISABLED-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 15
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s3, v1, 31
-; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s2, 16
-; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 47
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s6, v1, 63
-; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s3, 32
-; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-DISABLED-NEXT: v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s2, 48
-; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-DISABLED-NEXT: s_mov_b32 s2, -1
-; GFX1064-DISABLED-NEXT: ; implicit-def: $vgpr0
-; GFX1064-DISABLED-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX1064-DISABLED-NEXT: s_cbranch_execz .LBB4_2
-; GFX1064-DISABLED-NEXT: ; %bb.1:
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, 0
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v4, s6
-; GFX1064-DISABLED-NEXT: ds_add_rtn_f32 v0, v0, v4
-; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-DISABLED-NEXT: buffer_gl0_inv
-; GFX1064-DISABLED-NEXT: .LBB4_2:
-; GFX1064-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1064-DISABLED-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX1064-DISABLED-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1064-DISABLED-NEXT: v_readfirstlane_b32 s3, v0
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, v3
-; GFX1064-DISABLED-NEXT: v_add_f32_e32 v0, s3, v0
-; GFX1064-DISABLED-NEXT: v_cndmask_b32_e64 v0, v0, s3, vcc
-; GFX1064-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1064-DISABLED-NEXT: s_endpgm
+; GFX900-LABEL: fadd_f32_varying:
+; GFX900: ; %bb.0: ; %entry
+; GFX900-NEXT: v_cvt_f32_i32_e32 v0, v0
+; GFX900-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX900-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v4
+; GFX900-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX900-NEXT: v_bfrev_b32_e32 v1, 1
+; GFX900-NEXT: v_cndmask_b32_e64 v2, v1, v0, s[0:1]
+; GFX900-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_mov_b32_dpp v3, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX900-NEXT: v_add_f32_e32 v2, v2, v3
+; GFX900-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_mov_b32_dpp v3, v2 row_shr:2 row_mask:0xf bank_mask:0xf
+; GFX900-NEXT: v_add_f32_e32 v2, v2, v3
+; GFX900-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_mov_b32_dpp v3, v2 row_shr:4 row_mask:0xf bank_mask:0xf
+; GFX900-NEXT: v_add_f32_e32 v2, v2, v3
+; GFX900-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_mov_b32_dpp v3, v2 row_shr:8 row_mask:0xf bank_mask:0xf
+; GFX900-NEXT: v_add_f32_e32 v2, v2, v3
+; GFX900-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_mov_b32_dpp v3, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
+; GFX900-NEXT: v_add_f32_e32 v2, v2, v3
+; GFX900-NEXT: v_bfrev_b32_e32 v3, 1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_mov_b32_dpp v3, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
+; GFX900-NEXT: v_add_f32_e32 v2, v2, v3
+; GFX900-NEXT: v_readlane_b32 s2, v2, 63
+; GFX900-NEXT: s_nop 0
+; GFX900-NEXT: v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
+; GFX900-NEXT: s_mov_b64 exec, s[0:1]
+; GFX900-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX900-NEXT: ; implicit-def: $vgpr0
+; GFX900-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX900-NEXT: s_cbranch_execz .LBB4_2
+; GFX900-NEXT: ; %bb.1:
+; GFX900-NEXT: v_mov_b32_e32 v0, 0
+; GFX900-NEXT: v_mov_b32_e32 v4, s2
+; GFX900-NEXT: ds_add_rtn_f32 v0, v0, v4
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: .LBB4_2:
+; GFX900-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX900-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX900-NEXT: v_readfirstlane_b32 s4, v0
+; GFX900-NEXT: v_mov_b32_e32 v0, v1
+; GFX900-NEXT: v_add_f32_e32 v0, s4, v0
+; GFX900-NEXT: v_mov_b32_e32 v4, s4
+; GFX900-NEXT: s_mov_b32 s3, 0xf000
+; GFX900-NEXT: s_mov_b32 s2, -1
+; GFX900-NEXT: v_cndmask_b32_e32 v0, v0, v4, vcc
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX900-NEXT: s_endpgm
entry:
%lane = call i32 @llvm.amdgcn.workitem.id.x()
%conv = sitofp i32 %lane to float
@@ -1249,280 +822,363 @@ entry:
}
; Test 6: divergent i32 add on GLOBAL memory -- should NOT be affected by threshold
-define amdgpu_kernel void @add_i32_varying_global(ptr addrspace(1) %out, ptr addrspace(1) %inout) {
-; GFX1032-THRESH5-LABEL: add_i32_varying_global:
-; GFX1032-THRESH5: ; %bb.0: ; %entry
-; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-THRESH5-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-THRESH5-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
-; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s4, -1
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v3, 0
-; GFX1032-THRESH5-NEXT: v_readlane_b32 s6, v1, 31
-; GFX1032-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-THRESH5-NEXT: v_readlane_b32 s5, v1, 15
-; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s4
-; GFX1032-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-THRESH5-NEXT: s_or_saveexec_b32 s4, -1
-; GFX1032-THRESH5-NEXT: v_writelane_b32 v3, s5, 16
-; GFX1032-THRESH5-NEXT: s_mov_b32 exec_lo, s4
-; GFX1032-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-THRESH5-NEXT: s_mov_b32 s4, s6
-; GFX1032-THRESH5-NEXT: s_mov_b32 s6, -1
-; GFX1032-THRESH5-NEXT: ; implicit-def: $vgpr0
-; GFX1032-THRESH5-NEXT: s_and_saveexec_b32 s8, vcc_lo
-; GFX1032-THRESH5-NEXT: s_cbranch_execz .LBB5_2
-; GFX1032-THRESH5-NEXT: ; %bb.1:
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, s4
-; GFX1032-THRESH5-NEXT: s_mov_b32 s7, 0x31016000
-; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-THRESH5-NEXT: s_mov_b32 s4, s2
-; GFX1032-THRESH5-NEXT: s_mov_b32 s5, s3
-; GFX1032-THRESH5-NEXT: buffer_atomic_add v0, off, s[4:7], 0 glc
-; GFX1032-THRESH5-NEXT: s_waitcnt vmcnt(0)
-; GFX1032-THRESH5-NEXT: buffer_gl1_inv
-; GFX1032-THRESH5-NEXT: buffer_gl0_inv
-; GFX1032-THRESH5-NEXT: .LBB5_2:
-; GFX1032-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1032-THRESH5-NEXT: s_or_b32 exec_lo, exec_lo, s8
-; GFX1032-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-THRESH5-NEXT: v_readfirstlane_b32 s2, v0
-; GFX1032-THRESH5-NEXT: v_mov_b32_e32 v0, v3
-; GFX1032-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1032-THRESH5-NEXT: v_add_nc_u32_e32 v0, s2, v0
-; GFX1032-THRESH5-NEXT: s_mov_b32 s2, s6
-; GFX1032-THRESH5-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1032-THRESH5-NEXT: s_endpgm
+define amdgpu_kernel void @add_i32_varying_global(ptr addrspace(1) %out, ptr addrspace(1) %inout) #0 {
+; GFX1032-LABEL: add_i32_varying_global:
+; GFX1032: ; %bb.0: ; %entry
+; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1032-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
+; GFX1032-NEXT: s_or_saveexec_b32 s4, -1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1032-NEXT: v_mov_b32_e32 v3, 0
+; GFX1032-NEXT: v_readlane_b32 s6, v1, 31
+; GFX1032-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-NEXT: v_readlane_b32 s5, v1, 15
+; GFX1032-NEXT: s_mov_b32 exec_lo, s4
+; GFX1032-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1032-NEXT: s_or_saveexec_b32 s4, -1
+; GFX1032-NEXT: v_writelane_b32 v3, s5, 16
+; GFX1032-NEXT: s_mov_b32 exec_lo, s4
+; GFX1032-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
+; GFX1032-NEXT: s_mov_b32 s4, s6
+; GFX1032-NEXT: s_mov_b32 s6, -1
+; GFX1032-NEXT: ; implicit-def: $vgpr0
+; GFX1032-NEXT: s_and_saveexec_b32 s8, vcc_lo
+; GFX1032-NEXT: s_cbranch_execz .LBB5_2
+; GFX1032-NEXT: ; %bb.1:
+; GFX1032-NEXT: v_mov_b32_e32 v0, s4
+; GFX1032-NEXT: s_mov_b32 s7, 0x31016000
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: s_mov_b32 s4, s2
+; GFX1032-NEXT: s_mov_b32 s5, s3
+; GFX1032-NEXT: buffer_atomic_add v0, off, s[4:7], 0 glc
+; GFX1032-NEXT: s_waitcnt vmcnt(0)
+; GFX1032-NEXT: buffer_gl1_inv
+; GFX1032-NEXT: buffer_gl0_inv
+; GFX1032-NEXT: .LBB5_2:
+; GFX1032-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-NEXT: s_or_b32 exec_lo, exec_lo, s8
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: v_readfirstlane_b32 s2, v0
+; GFX1032-NEXT: v_mov_b32_e32 v0, v3
+; GFX1032-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1032-NEXT: v_add_nc_u32_e32 v0, s2, v0
+; GFX1032-NEXT: s_mov_b32 s2, s6
+; GFX1032-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1032-NEXT: s_endpgm
+;
+; GFX1064-LABEL: add_i32_varying_global:
+; GFX1064: ; %bb.0: ; %entry
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1064-NEXT: v_mov_b32_e32 v3, 0
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1064-NEXT: v_mov_b32_e32 v2, s2
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1064-NEXT: v_readlane_b32 s6, v1, 15
+; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
+; GFX1064-NEXT: s_or_saveexec_b64 s[4:5], -1
+; GFX1064-NEXT: v_readlane_b32 s7, v1, 31
+; GFX1064-NEXT: v_writelane_b32 v3, s6, 16
+; GFX1064-NEXT: s_mov_b64 exec, s[4:5]
+; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-NEXT: s_or_saveexec_b64 s[4:5], -1
+; GFX1064-NEXT: v_readlane_b32 s8, v1, 47
+; GFX1064-NEXT: v_readlane_b32 s9, v1, 63
+; GFX1064-NEXT: v_writelane_b32 v3, s7, 32
+; GFX1064-NEXT: s_mov_b64 exec, s[4:5]
+; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v0, exec_hi, v0
+; GFX1064-NEXT: s_or_saveexec_b64 s[6:7], -1
+; GFX1064-NEXT: s_mov_b32 s4, s9
+; GFX1064-NEXT: v_writelane_b32 v3, s8, 48
+; GFX1064-NEXT: s_mov_b64 exec, s[6:7]
+; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX1064-NEXT: s_mov_b32 s6, -1
+; GFX1064-NEXT: ; implicit-def: $vgpr0
+; GFX1064-NEXT: s_and_saveexec_b64 s[8:9], vcc
+; GFX1064-NEXT: s_cbranch_execz .LBB5_2
+; GFX1064-NEXT: ; %bb.1:
+; GFX1064-NEXT: v_mov_b32_e32 v0, s4
+; GFX1064-NEXT: s_mov_b32 s7, 0x31016000
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: s_mov_b32 s4, s2
+; GFX1064-NEXT: s_mov_b32 s5, s3
+; GFX1064-NEXT: buffer_atomic_add v0, off, s[4:7], 0 glc
+; GFX1064-NEXT: s_waitcnt vmcnt(0)
+; GFX1064-NEXT: buffer_gl1_inv
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: .LBB5_2:
+; GFX1064-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-NEXT: s_or_b64 exec, exec, s[8:9]
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: v_readfirstlane_b32 s2, v0
+; GFX1064-NEXT: v_mov_b32_e32 v0, v3
+; GFX1064-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-NEXT: v_add_nc_u32_e32 v0, s2, v0
+; GFX1064-NEXT: s_mov_b32 s2, s6
+; GFX1064-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1064-NEXT: s_endpgm
+;
+; GFX900-LABEL: add_i32_varying_global:
+; GFX900: ; %bb.0: ; %entry
+; GFX900-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
+; GFX900-NEXT: s_or_saveexec_b64 s[4:5], -1
+; GFX900-NEXT: v_mov_b32_e32 v1, 0
+; GFX900-NEXT: s_mov_b64 exec, s[4:5]
+; GFX900-NEXT: v_mbcnt_lo_u32_b32 v3, exec_lo, 0
+; GFX900-NEXT: v_mbcnt_hi_u32_b32 v3, exec_hi, v3
+; GFX900-NEXT: s_or_saveexec_b64 s[4:5], -1
+; GFX900-NEXT: v_cndmask_b32_e64 v2, 0, v0, s[4:5]
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v2, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v2, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v2, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v2, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
+; GFX900-NEXT: v_readlane_b32 s6, v2, 63
+; GFX900-NEXT: s_nop 0
+; GFX900-NEXT: v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
+; GFX900-NEXT: s_mov_b64 exec, s[4:5]
+; GFX900-NEXT: v_cmp_eq_u32_e32 vcc, 0, v3
+; GFX900-NEXT: ; implicit-def: $vgpr0
+; GFX900-NEXT: s_and_saveexec_b64 s[4:5], vcc
+; GFX900-NEXT: s_cbranch_execz .LBB5_2
+; GFX900-NEXT: ; %bb.1:
+; GFX900-NEXT: s_mov_b32 s11, 0xf000
+; GFX900-NEXT: s_mov_b32 s10, -1
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: s_mov_b32 s8, s2
+; GFX900-NEXT: s_mov_b32 s9, s3
+; GFX900-NEXT: v_mov_b32_e32 v0, s6
+; GFX900-NEXT: buffer_atomic_add v0, off, s[8:11], 0 glc
+; GFX900-NEXT: s_waitcnt vmcnt(0)
+; GFX900-NEXT: buffer_wbinvl1_vol
+; GFX900-NEXT: .LBB5_2:
+; GFX900-NEXT: s_or_b64 exec, exec, s[4:5]
+; GFX900-NEXT: v_readfirstlane_b32 s4, v0
+; GFX900-NEXT: v_mov_b32_e32 v0, v1
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: s_mov_b32 s3, 0xf000
+; GFX900-NEXT: s_mov_b32 s2, -1
+; GFX900-NEXT: v_add_u32_e32 v0, s4, v0
+; GFX900-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX900-NEXT: s_endpgm
+entry:
+ %lane = call i32 @llvm.amdgcn.workitem.id.x()
+ %old = atomicrmw add ptr addrspace(1) %inout, i32 %lane acq_rel
+ store i32 %old, ptr addrspace(1) %out
+ ret void
+}
+
+; Test 7: threshold=32 on wave32 (>= wavefront size) should avoid the dynamic
+; threshold branch.
+define amdgpu_kernel void @add_i32_varying_thresh32(ptr addrspace(1) %out) #1 {
+; GFX1032-LABEL: add_i32_varying_thresh32:
+; GFX1032: ; %bb.0: ; %entry
+; GFX1032-NEXT: v_mov_b32_e32 v1, 0
+; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1032-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1032-NEXT: s_mov_b32 s2, -1
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: ds_add_rtn_u32 v0, v1, v0
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_gl0_inv
+; GFX1032-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1032-NEXT: s_endpgm
;
-; GFX1064-THRESH5-LABEL: add_i32_varying_global:
-; GFX1064-THRESH5: ; %bb.0: ; %entry
-; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-THRESH5-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v3, 0
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-THRESH5-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v2, s2
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s6, v1, 15
-; GFX1064-THRESH5-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-THRESH5-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
-; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[4:5], -1
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s7, v1, 31
-; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s6, 16
-; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[4:5]
-; GFX1064-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[4:5], -1
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s8, v1, 47
-; GFX1064-THRESH5-NEXT: v_readlane_b32 s9, v1, 63
-; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s7, 32
-; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[4:5]
-; GFX1064-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-THRESH5-NEXT: s_or_saveexec_b64 s[6:7], -1
-; GFX1064-THRESH5-NEXT: s_mov_b32 s4, s9
-; GFX1064-THRESH5-NEXT: v_writelane_b32 v3, s8, 48
-; GFX1064-THRESH5-NEXT: s_mov_b64 exec, s[6:7]
-; GFX1064-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-THRESH5-NEXT: s_mov_b32 s6, -1
-; GFX1064-THRESH5-NEXT: ; implicit-def: $vgpr0
-; GFX1064-THRESH5-NEXT: s_and_saveexec_b64 s[8:9], vcc
-; GFX1064-THRESH5-NEXT: s_cbranch_execz .LBB5_2
-; GFX1064-THRESH5-NEXT: ; %bb.1:
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, s4
-; GFX1064-THRESH5-NEXT: s_mov_b32 s7, 0x31016000
-; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-THRESH5-NEXT: s_mov_b32 s4, s2
-; GFX1064-THRESH5-NEXT: s_mov_b32 s5, s3
-; GFX1064-THRESH5-NEXT: buffer_atomic_add v0, off, s[4:7], 0 glc
-; GFX1064-THRESH5-NEXT: s_waitcnt vmcnt(0)
-; GFX1064-THRESH5-NEXT: buffer_gl1_inv
-; GFX1064-THRESH5-NEXT: buffer_gl0_inv
-; GFX1064-THRESH5-NEXT: .LBB5_2:
-; GFX1064-THRESH5-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1064-THRESH5-NEXT: s_or_b64 exec, exec, s[8:9]
-; GFX1064-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-THRESH5-NEXT: v_readfirstlane_b32 s2, v0
-; GFX1064-THRESH5-NEXT: v_mov_b32_e32 v0, v3
-; GFX1064-THRESH5-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1064-THRESH5-NEXT: v_add_nc_u32_e32 v0, s2, v0
-; GFX1064-THRESH5-NEXT: s_mov_b32 s2, s6
-; GFX1064-THRESH5-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1064-THRESH5-NEXT: s_endpgm
+; GFX1064-LABEL: add_i32_varying_thresh32:
+; GFX1064: ; %bb.0: ; %entry
+; GFX1064-NEXT: s_bcnt1_i32_b64 s0, exec
+; GFX1064-NEXT: s_cmp_lt_u32 s0, 33
+; GFX1064-NEXT: s_cbranch_scc0 .LBB6_2
+; GFX1064-NEXT: ; %bb.1:
+; GFX1064-NEXT: v_mov_b32_e32 v4, 0
+; GFX1064-NEXT: ds_add_rtn_u32 v4, v4, v0
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: s_cbranch_execz .LBB6_3
+; GFX1064-NEXT: s_branch .LBB6_6
+; GFX1064-NEXT: .LBB6_2:
+; GFX1064-NEXT: ; implicit-def: $vgpr4
+; GFX1064-NEXT: .LBB6_3:
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1064-NEXT: v_mov_b32_e32 v3, 0
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1064-NEXT: v_mov_b32_e32 v2, s2
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 15
+; GFX1064-NEXT: v_readlane_b32 s3, v1, 31
+; GFX1064-NEXT: v_writelane_b32 v3, s2, 16
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 63
+; GFX1064-NEXT: v_readlane_b32 s6, v1, 47
+; GFX1064-NEXT: v_writelane_b32 v3, s3, 32
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
+; GFX1064-NEXT: v_mov_b32_e32 v0, 0
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_writelane_b32 v3, s6, 48
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX1064-NEXT: ; implicit-def: $vgpr4
+; GFX1064-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1064-NEXT: s_cbranch_execz .LBB6_5
+; GFX1064-NEXT: ; %bb.4:
+; GFX1064-NEXT: v_mov_b32_e32 v4, s2
+; GFX1064-NEXT: s_waitcnt_vscnt null, 0x0
+; GFX1064-NEXT: ds_add_rtn_u32 v4, v0, v4
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: .LBB6_5:
+; GFX1064-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1064-NEXT: v_readfirstlane_b32 s0, v4
+; GFX1064-NEXT: v_mov_b32_e32 v0, v3
+; GFX1064-NEXT: v_add_nc_u32_e32 v4, s0, v0
+; GFX1064-NEXT: .LBB6_6:
+; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-NEXT: s_mov_b32 s2, -1
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_store_dword v4, off, s[0:3], 0
+; GFX1064-NEXT: s_endpgm
;
-; GFX900-THRESH5-LABEL: add_i32_varying_global:
-; GFX900-THRESH5: ; %bb.0: ; %entry
-; GFX900-THRESH5-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
-; GFX900-THRESH5-NEXT: s_or_saveexec_b64 s[4:5], -1
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v1, 0
-; GFX900-THRESH5-NEXT: s_mov_b64 exec, s[4:5]
-; GFX900-THRESH5-NEXT: v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX900-THRESH5-NEXT: v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX900-THRESH5-NEXT: s_or_saveexec_b64 s[4:5], -1
-; GFX900-THRESH5-NEXT: v_cndmask_b32_e64 v2, 0, v0, s[4:5]
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v2, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v2, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v2, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX900-THRESH5-NEXT: s_nop 1
-; GFX900-THRESH5-NEXT: v_add_u32_dpp v2, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX900-THRESH5-NEXT: v_readlane_b32 s6, v2, 63
-; GFX900-THRESH5-NEXT: s_nop 0
-; GFX900-THRESH5-NEXT: v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX900-THRESH5-NEXT: s_mov_b64 exec, s[4:5]
-; GFX900-THRESH5-NEXT: v_cmp_eq_u32_e32 vcc, 0, v3
-; GFX900-THRESH5-NEXT: ; implicit-def: $vgpr0
-; GFX900-THRESH5-NEXT: s_and_saveexec_b64 s[4:5], vcc
-; GFX900-THRESH5-NEXT: s_cbranch_execz .LBB5_2
-; GFX900-THRESH5-NEXT: ; %bb.1:
-; GFX900-THRESH5-NEXT: s_mov_b32 s11, 0xf000
-; GFX900-THRESH5-NEXT: s_mov_b32 s10, -1
-; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-THRESH5-NEXT: s_mov_b32 s8, s2
-; GFX900-THRESH5-NEXT: s_mov_b32 s9, s3
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, s6
-; GFX900-THRESH5-NEXT: buffer_atomic_add v0, off, s[8:11], 0 glc
-; GFX900-THRESH5-NEXT: s_waitcnt vmcnt(0)
-; GFX900-THRESH5-NEXT: buffer_wbinvl1_vol
-; GFX900-THRESH5-NEXT: .LBB5_2:
-; GFX900-THRESH5-NEXT: s_or_b64 exec, exec, s[4:5]
-; GFX900-THRESH5-NEXT: v_readfirstlane_b32 s4, v0
-; GFX900-THRESH5-NEXT: v_mov_b32_e32 v0, v1
-; GFX900-THRESH5-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-THRESH5-NEXT: s_mov_b32 s3, 0xf000
-; GFX900-THRESH5-NEXT: s_mov_b32 s2, -1
-; GFX900-THRESH5-NEXT: v_add_u32_e32 v0, s4, v0
-; GFX900-THRESH5-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX900-THRESH5-NEXT: s_endpgm
+; GFX900-LABEL: add_i32_varying_thresh32:
+; GFX900: ; %bb.0: ; %entry
+; GFX900-NEXT: s_bcnt1_i32_b64 s0, exec
+; GFX900-NEXT: s_cmp_lt_u32 s0, 33
+; GFX900-NEXT: s_cbranch_scc0 .LBB6_2
+; GFX900-NEXT: ; %bb.1:
+; GFX900-NEXT: v_mov_b32_e32 v3, 0
+; GFX900-NEXT: ds_add_rtn_u32 v3, v3, v0
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: s_cbranch_execz .LBB6_3
+; GFX900-NEXT: s_branch .LBB6_6
+; GFX900-NEXT: .LBB6_2:
+; GFX900-NEXT: ; implicit-def: $vgpr3
+; GFX900-NEXT: .LBB6_3:
+; GFX900-NEXT: v_mov_b32_e32 v3, 0
+; GFX900-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX900-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v4
+; GFX900-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX900-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX900-NEXT: v_mov_b32_e32 v2, 0
+; GFX900-NEXT: s_nop 0
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
+; GFX900-NEXT: s_nop 1
+; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
+; GFX900-NEXT: v_readlane_b32 s2, v1, 63
+; GFX900-NEXT: s_nop 0
+; GFX900-NEXT: v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
+; GFX900-NEXT: s_mov_b64 exec, s[0:1]
+; GFX900-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX900-NEXT: ; implicit-def: $vgpr0
+; GFX900-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX900-NEXT: s_cbranch_execz .LBB6_5
+; GFX900-NEXT: ; %bb.4:
+; GFX900-NEXT: v_mov_b32_e32 v0, s2
+; GFX900-NEXT: ds_add_rtn_u32 v0, v3, v0
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: .LBB6_5:
+; GFX900-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX900-NEXT: v_readfirstlane_b32 s0, v0
+; GFX900-NEXT: v_mov_b32_e32 v0, v2
+; GFX900-NEXT: v_add_u32_e32 v3, s0, v0
+; GFX900-NEXT: .LBB6_6:
+; GFX900-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX900-NEXT: s_mov_b32 s3, 0xf000
+; GFX900-NEXT: s_mov_b32 s2, -1
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: buffer_store_dword v3, off, s[0:3], 0
+; GFX900-NEXT: s_endpgm
+entry:
+ %lane = call i32 @llvm.amdgcn.workitem.id.x()
+ %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel
+ store i32 %old, ptr addrspace(1) %out
+ ret void
+}
+
+; Test 8: threshold=64 on wave64 (>= wavefront size) should avoid the dynamic
+; threshold branch.
+define amdgpu_kernel void @add_i32_varying_thresh64(ptr addrspace(1) %out) #2 {
+; GFX1032-LABEL: add_i32_varying_thresh64:
+; GFX1032: ; %bb.0: ; %entry
+; GFX1032-NEXT: v_mov_b32_e32 v1, 0
+; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1032-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1032-NEXT: s_mov_b32 s2, -1
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: ds_add_rtn_u32 v0, v1, v0
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_gl0_inv
+; GFX1032-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1032-NEXT: s_endpgm
;
-; GFX1032-DISABLED-LABEL: add_i32_varying_global:
-; GFX1032-DISABLED: ; %bb.0: ; %entry
-; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-DISABLED-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-DISABLED-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-DISABLED-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
-; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s4, -1
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v3, 0
-; GFX1032-DISABLED-NEXT: v_readlane_b32 s6, v1, 31
-; GFX1032-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-DISABLED-NEXT: v_readlane_b32 s5, v1, 15
-; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s4
-; GFX1032-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-DISABLED-NEXT: s_or_saveexec_b32 s4, -1
-; GFX1032-DISABLED-NEXT: v_writelane_b32 v3, s5, 16
-; GFX1032-DISABLED-NEXT: s_mov_b32 exec_lo, s4
-; GFX1032-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-DISABLED-NEXT: s_mov_b32 s4, s6
-; GFX1032-DISABLED-NEXT: s_mov_b32 s6, -1
-; GFX1032-DISABLED-NEXT: ; implicit-def: $vgpr0
-; GFX1032-DISABLED-NEXT: s_and_saveexec_b32 s8, vcc_lo
-; GFX1032-DISABLED-NEXT: s_cbranch_execz .LBB5_2
-; GFX1032-DISABLED-NEXT: ; %bb.1:
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, s4
-; GFX1032-DISABLED-NEXT: s_mov_b32 s7, 0x31016000
-; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-DISABLED-NEXT: s_mov_b32 s4, s2
-; GFX1032-DISABLED-NEXT: s_mov_b32 s5, s3
-; GFX1032-DISABLED-NEXT: buffer_atomic_add v0, off, s[4:7], 0 glc
-; GFX1032-DISABLED-NEXT: s_waitcnt vmcnt(0)
-; GFX1032-DISABLED-NEXT: buffer_gl1_inv
-; GFX1032-DISABLED-NEXT: buffer_gl0_inv
-; GFX1032-DISABLED-NEXT: .LBB5_2:
-; GFX1032-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1032-DISABLED-NEXT: s_or_b32 exec_lo, exec_lo, s8
-; GFX1032-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-DISABLED-NEXT: v_readfirstlane_b32 s2, v0
-; GFX1032-DISABLED-NEXT: v_mov_b32_e32 v0, v3
-; GFX1032-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1032-DISABLED-NEXT: v_add_nc_u32_e32 v0, s2, v0
-; GFX1032-DISABLED-NEXT: s_mov_b32 s2, s6
-; GFX1032-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1032-DISABLED-NEXT: s_endpgm
+; GFX1064-LABEL: add_i32_varying_thresh64:
+; GFX1064: ; %bb.0: ; %entry
+; GFX1064-NEXT: v_mov_b32_e32 v1, 0
+; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-NEXT: s_mov_b32 s2, -1
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: ds_add_rtn_u32 v0, v1, v0
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1064-NEXT: s_endpgm
;
-; GFX1064-DISABLED-LABEL: add_i32_varying_global:
-; GFX1064-DISABLED: ; %bb.0: ; %entry
-; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-DISABLED-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v3, 0
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-DISABLED-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v2, s2
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s6, v1, 15
-; GFX1064-DISABLED-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-DISABLED-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
-; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[4:5], -1
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s7, v1, 31
-; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s6, 16
-; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[4:5]
-; GFX1064-DISABLED-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[4:5], -1
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s8, v1, 47
-; GFX1064-DISABLED-NEXT: v_readlane_b32 s9, v1, 63
-; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s7, 32
-; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[4:5]
-; GFX1064-DISABLED-NEXT: v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-DISABLED-NEXT: s_or_saveexec_b64 s[6:7], -1
-; GFX1064-DISABLED-NEXT: s_mov_b32 s4, s9
-; GFX1064-DISABLED-NEXT: v_writelane_b32 v3, s8, 48
-; GFX1064-DISABLED-NEXT: s_mov_b64 exec, s[6:7]
-; GFX1064-DISABLED-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-DISABLED-NEXT: s_mov_b32 s6, -1
-; GFX1064-DISABLED-NEXT: ; implicit-def: $vgpr0
-; GFX1064-DISABLED-NEXT: s_and_saveexec_b64 s[8:9], vcc
-; GFX1064-DISABLED-NEXT: s_cbranch_execz .LBB5_2
-; GFX1064-DISABLED-NEXT: ; %bb.1:
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, s4
-; GFX1064-DISABLED-NEXT: s_mov_b32 s7, 0x31016000
-; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-DISABLED-NEXT: s_mov_b32 s4, s2
-; GFX1064-DISABLED-NEXT: s_mov_b32 s5, s3
-; GFX1064-DISABLED-NEXT: buffer_atomic_add v0, off, s[4:7], 0 glc
-; GFX1064-DISABLED-NEXT: s_waitcnt vmcnt(0)
-; GFX1064-DISABLED-NEXT: buffer_gl1_inv
-; GFX1064-DISABLED-NEXT: buffer_gl0_inv
-; GFX1064-DISABLED-NEXT: .LBB5_2:
-; GFX1064-DISABLED-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1064-DISABLED-NEXT: s_or_b64 exec, exec, s[8:9]
-; GFX1064-DISABLED-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-DISABLED-NEXT: v_readfirstlane_b32 s2, v0
-; GFX1064-DISABLED-NEXT: v_mov_b32_e32 v0, v3
-; GFX1064-DISABLED-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1064-DISABLED-NEXT: v_add_nc_u32_e32 v0, s2, v0
-; GFX1064-DISABLED-NEXT: s_mov_b32 s2, s6
-; GFX1064-DISABLED-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1064-DISABLED-NEXT: s_endpgm
+; GFX900-LABEL: add_i32_varying_thresh64:
+; GFX900: ; %bb.0: ; %entry
+; GFX900-NEXT: v_mov_b32_e32 v1, 0
+; GFX900-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: ds_add_rtn_u32 v0, v1, v0
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: s_mov_b32 s3, 0xf000
+; GFX900-NEXT: s_mov_b32 s2, -1
+; GFX900-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX900-NEXT: s_endpgm
entry:
%lane = call i32 @llvm.amdgcn.workitem.id.x()
- %old = atomicrmw add ptr addrspace(1) %inout, i32 %lane acq_rel
+ %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel
store i32 %old, ptr addrspace(1) %out
ret void
}
>From 2ceee829c928271cc6b60493ba632e7895fc4d3b Mon Sep 17 00:00:00 2001
From: YaFan <YaFan.Tao at amd.com>
Date: Thu, 2 Apr 2026 16:02:46 +0800
Subject: [PATCH 10/17] Update Lit test, Fix that DPP should be skiped when
threshold >= wave size
---
llvm/docs/AMDGPUUsage.rst | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 81f5380b89e67..46a22ad68664a 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -2209,10 +2209,10 @@ The AMDGPU backend supports the following LLVM IR attributes.
scan for integer LDS atomics. When set to a positive integer n, the
optimizer emits a dynamic branch: if the number of active lanes exceeds
n, the DPP scan path is taken; otherwise each lane independently issues
- its own atomic. A value of 0 (the default) disables the dynamic branch
- and always uses the DPP scan path when the DPP strategy is selected.
- Values greater than or equal to the wavefront size effectively disable
- DPP optimization for LDS atomics. Floating-point LDS atomics are not
+ its own atomic. A value of 0 (the default) disables the dynamic branch
+ and always uses the DPP scan path when the DPP strategy is selected.
+ Values greater than or equal to the wavefront size effectively disable
+ DPP optimization for LDS atomics. Floating-point LDS atomics are not
affected by this attribute.
Relevant for GFX10+.
>From a7c083dd8b984772d2d03e015a0c8047b96cefc9 Mon Sep 17 00:00:00 2001
From: YaFan <YaFan.Tao at amd.com>
Date: Tue, 7 Apr 2026 18:00:16 +0800
Subject: [PATCH 11/17] Fix ease const, parse attribute per function, Let
Dynamic DPP works for float too.
---
llvm/docs/AMDGPUUsage.rst | 15 +-
.../Target/AMDGPU/AMDGPUAtomicOptimizer.cpp | 140 ++++++++----------
.../atomic_optimizations_dpp_lds_threshold.ll | 134 +++++++++++------
3 files changed, 156 insertions(+), 133 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 46a22ad68664a..21fc425526d0b 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -2206,14 +2206,13 @@ The AMDGPU backend supports the following LLVM IR attributes.
(enabled by default).
"amdgpu-atomic-optimizer-dpp-lds-threshold"="n" Threshold that controls when the AMDGPUAtomicOptimizer uses a DPP-based
- scan for integer LDS atomics. When set to a positive integer n, the
- optimizer emits a dynamic branch: if the number of active lanes exceeds
- n, the DPP scan path is taken; otherwise each lane independently issues
- its own atomic. A value of 0 (the default) disables the dynamic branch
- and always uses the DPP scan path when the DPP strategy is selected.
- Values greater than or equal to the wavefront size effectively disable
- DPP optimization for LDS atomics. Floating-point LDS atomics are not
- affected by this attribute.
+ scan for LDS atomics with divergent values. When set to a positive
+ integer n, the optimizer emits a dynamic branch: if the number of
+ active lanes exceeds n, the DPP scan path is taken; otherwise each
+ lane independently issues its own atomic. A value of 0 (the default)
+ disables the dynamic branch and always uses the DPP scan path when
+ the DPP strategy is selected. Values greater than or equal to the
+ wavefront size effectively disable DPP optimization for LDS atomics.
Relevant for GFX10+.
================================================ ==========================================================
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index 84d1f73a76e17..f6e2bc0c37b3c 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -87,14 +87,9 @@ class AMDGPUAtomicOptimizerImpl
Value *const Identity, Value *V, Instruction &I,
BasicBlock *ComputeLoop, BasicBlock *ComputeEnd) const;
- Value *buildMbcnt(IRBuilder<> &B, Value *Ballot) const;
-
- std::pair<Value *, Value *>
- buildDPPScanAndReduce(IRBuilder<> &B, AtomicRMWInst::BinOp ScanOp, Type *Ty,
- Value *V, Value *Identity, bool NeedResult) const;
-
void optimizeAtomic(Instruction &I, AtomicRMWInst::BinOp Op, unsigned ValIdx,
- bool ValDivergent, bool IsLDS) const;
+ bool ValDivergent, bool IsLDS,
+ unsigned DPPThreshold) const;
Value *optimizeAtomicImpl(IRBuilder<> &B, Instruction &I,
AtomicRMWInst::BinOp Op, unsigned ValIdx,
@@ -175,8 +170,14 @@ bool AMDGPUAtomicOptimizerImpl::run() {
if (ToReplace.empty())
return false;
+ unsigned DPPThreshold = 0;
+ int AttrVal = F.getFnAttributeAsParsedInteger(
+ "amdgpu-atomic-optimizer-dpp-lds-threshold", 0);
+ if (AttrVal > 0)
+ DPPThreshold = static_cast<unsigned>(AttrVal);
+
for (auto &[I, Op, ValIdx, ValDivergent, IsLDS] : ToReplace)
- optimizeAtomic(*I, Op, ValIdx, ValDivergent, IsLDS);
+ optimizeAtomic(*I, Op, ValIdx, ValDivergent, IsLDS, DPPThreshold);
ToReplace.clear();
return true;
}
@@ -621,47 +622,6 @@ std::pair<Value *, Value *> AMDGPUAtomicOptimizerImpl::buildScanIteratively(
return {OldValue, NewAccumulator};
}
-// Use the builder to create an mbcnt intrinsic call to count the number of
-// active lanes below the current lane.
-Value *AMDGPUAtomicOptimizerImpl::buildMbcnt(IRBuilder<> &B,
- Value *Ballot) const {
- Type *Int32Ty = B.getInt32Ty();
- if (ST.isWave32()) {
- return B.CreateIntrinsic(Intrinsic::amdgcn_mbcnt_lo,
- {Ballot, B.getInt32(0)});
- }
- Value *const ExtractLo = B.CreateTrunc(Ballot, Int32Ty);
- Value *const ExtractHi = B.CreateTrunc(B.CreateLShr(Ballot, 32), Int32Ty);
- Value *Mbcnt =
- B.CreateIntrinsic(Intrinsic::amdgcn_mbcnt_lo, {ExtractLo, B.getInt32(0)});
- return B.CreateIntrinsic(Intrinsic::amdgcn_mbcnt_hi, {ExtractHi, Mbcnt});
-}
-
-// Use the builder to create a DPP-based scan or reduction across the
-// wavefront on divergent values. Sets inactive lanes to the identity value,
-// then either builds a full scan (when the result is needed) or just a
-// reduction. Returns {ReducedValue, ExclusiveScan}; ExclusiveScan is nullptr
-// when !NeedResult && ST.hasPermLaneX16().
-std::pair<Value *, Value *> AMDGPUAtomicOptimizerImpl::buildDPPScanAndReduce(
- IRBuilder<> &B, AtomicRMWInst::BinOp ScanOp, Type *Ty, Value *V,
- Value *Identity, bool NeedResult) const {
- Value *NewV =
- B.CreateIntrinsic(Intrinsic::amdgcn_set_inactive, Ty, {V, Identity});
- Value *ExclScan = nullptr;
- if (!NeedResult && ST.hasPermLaneX16()) {
- NewV = buildReduction(B, ScanOp, NewV, Identity);
- } else {
- NewV = buildScan(B, ScanOp, NewV, Identity);
- if (NeedResult)
- ExclScan = buildShiftRight(B, NewV, Identity);
- Value *const LastLaneIdx = B.getInt32(ST.getWavefrontSize() - 1);
- NewV =
- B.CreateIntrinsic(Ty, Intrinsic::amdgcn_readlane, {NewV, LastLaneIdx});
- }
- NewV = B.CreateIntrinsic(Intrinsic::amdgcn_strict_wwm, Ty, NewV);
- return {NewV, ExclScan};
-}
-
static Constant *getIdentityValueForAtomicOp(Type *const Ty,
AtomicRMWInst::BinOp Op) {
LLVMContext &C = Ty->getContext();
@@ -704,20 +664,12 @@ static Value *buildMul(IRBuilder<> &B, Value *LHS, Value *RHS) {
void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
AtomicRMWInst::BinOp Op,
unsigned ValIdx,
- bool ValDivergent,
- bool IsLDS) const {
- // Read the per-function threshold attribute early so we can bail out
- // before any CFG modifications when the threshold disables optimization.
- unsigned Threshold = 0;
- int AttrVal = I.getFunction()->getFnAttributeAsParsedInteger(
- "amdgpu-atomic-optimizer-dpp-lds-threshold", 0);
- if (AttrVal > 0)
- Threshold = static_cast<unsigned>(AttrVal);
-
+ bool ValDivergent, bool IsLDS,
+ unsigned DPPThreshold) const {
// If the threshold is >= wavefront size, DPP is never profitable — skip
// the atomic optimization entirely for this instruction.
if (IsLDS && ValDivergent && ScanImpl == ScanOptions::DPP &&
- !AtomicRMWInst::isFPOperation(Op) && Threshold >= ST.getWavefrontSize())
+ DPPThreshold >= ST.getWavefrontSize())
return;
// Start building just before the instruction.
@@ -741,8 +693,8 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
// Record I's original position as the entry block.
PixelEntryBB = I.getParent();
- Value *Cond = B.CreateIntrinsic(Intrinsic::amdgcn_ps_live, {});
- Instruction *NonHelperTerminator =
+ Value *const Cond = B.CreateIntrinsic(Intrinsic::amdgcn_ps_live, {});
+ Instruction *const NonHelperTerminator =
SplitBlockAndInsertIfThen(Cond, &I, false, nullptr, &DTU, nullptr);
// Record I's new position as the exit block.
@@ -754,8 +706,8 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
Value *Result = nullptr;
if (IsLDS && ValDivergent && ScanImpl == ScanOptions::DPP &&
- !AtomicRMWInst::isFPOperation(Op) && Threshold > 0) {
- Result = optimizeAtomicWithDynamicThreshold(B, I, Op, ValIdx, Threshold);
+ DPPThreshold > 0) {
+ Result = optimizeAtomicWithDynamicThreshold(B, I, Op, ValIdx, DPPThreshold);
} else {
Result = optimizeAtomicImpl(B, I, Op, ValIdx, ValDivergent);
}
@@ -788,7 +740,7 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
AtomicRMWInst::BinOp Op,
unsigned ValIdx,
bool ValDivergent) const {
- Type *Ty = I.getType();
+ Type *const Ty = I.getType();
Type *Int32Ty = B.getInt32Ty();
bool isAtomicFloatingPointTy = Ty->isFloatingPointTy();
[[maybe_unused]] const unsigned TyBitWidth = DL.getTypeSizeInBits(Ty);
@@ -799,15 +751,25 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
// We need to know how many lanes are active within the wavefront, and we do
// this by doing a ballot of active lanes.
- Type *WaveTy = B.getIntNTy(ST.getWavefrontSize());
- CallInst *Ballot =
+ Type *const WaveTy = B.getIntNTy(ST.getWavefrontSize());
+ CallInst *const Ballot =
B.CreateIntrinsic(Intrinsic::amdgcn_ballot, WaveTy, B.getTrue());
// We need to know how many lanes are active within the wavefront that are
// below us. If we counted each lane linearly starting from 0, a lane is
// below us only if its associated index was less than ours. We do this by
// using the mbcnt intrinsic.
- Value *Mbcnt = buildMbcnt(B, Ballot);
+ Value *Mbcnt;
+ if (ST.isWave32()) {
+ Mbcnt =
+ B.CreateIntrinsic(Intrinsic::amdgcn_mbcnt_lo, {Ballot, B.getInt32(0)});
+ } else {
+ Value *const ExtractLo = B.CreateTrunc(Ballot, Int32Ty);
+ Value *const ExtractHi = B.CreateTrunc(B.CreateLShr(Ballot, 32), Int32Ty);
+ Mbcnt = B.CreateIntrinsic(Intrinsic::amdgcn_mbcnt_lo,
+ {ExtractLo, B.getInt32(0)});
+ Mbcnt = B.CreateIntrinsic(Intrinsic::amdgcn_mbcnt_hi, {ExtractHi, Mbcnt});
+ }
Function *F = I.getFunction();
LLVMContext &C = F->getContext();
@@ -833,8 +795,28 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
// using DPP.
if (ValDivergent) {
if (ScanImpl == ScanOptions::DPP) {
- std::tie(NewV, ExclScan) =
- buildDPPScanAndReduce(B, ScanOp, Ty, V, Identity, NeedResult);
+ // First we need to set all inactive invocations to the identity value, so
+ // that they can correctly contribute to the final result.
+ NewV =
+ B.CreateIntrinsic(Intrinsic::amdgcn_set_inactive, Ty, {V, Identity});
+ if (!NeedResult && ST.hasPermLaneX16()) {
+ // On GFX10 the permlanex16 instruction helps us build a reduction
+ // without too many readlanes and writelanes, which are generally bad
+ // for performance.
+ NewV = buildReduction(B, ScanOp, NewV, Identity);
+ } else {
+ NewV = buildScan(B, ScanOp, NewV, Identity);
+ if (NeedResult)
+ ExclScan = buildShiftRight(B, NewV, Identity);
+ // Read the value from the last lane, which has accumulated the values
+ // of each active lane in the wavefront. This will be our new value
+ // which we will provide to the atomic operation.
+ Value *const LastLaneIdx = B.getInt32(ST.getWavefrontSize() - 1);
+ NewV = B.CreateIntrinsic(Ty, Intrinsic::amdgcn_readlane,
+ {NewV, LastLaneIdx});
+ }
+ // Finally mark the readlanes in the WWM section.
+ NewV = B.CreateIntrinsic(Intrinsic::amdgcn_strict_wwm, Ty, NewV);
} else if (ScanImpl == ScanOptions::Iterative) {
// Alternative implementation for scan
ComputeLoop = BasicBlock::Create(C, "ComputeLoop", F);
@@ -853,16 +835,16 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
case AtomicRMWInst::Sub: {
// The new value we will be contributing to the atomic operation is the
// old value times the number of active lanes.
- Value *Ctpop = B.CreateIntCast(
+ Value *const Ctpop = B.CreateIntCast(
B.CreateUnaryIntrinsic(Intrinsic::ctpop, Ballot), Ty, false);
NewV = buildMul(B, V, Ctpop);
break;
}
case AtomicRMWInst::FAdd:
case AtomicRMWInst::FSub: {
- Value *Ctpop = B.CreateIntCast(
+ Value *const Ctpop = B.CreateIntCast(
B.CreateUnaryIntrinsic(Intrinsic::ctpop, Ballot), Int32Ty, false);
- Value *CtpopFP = B.CreateUIToFP(Ctpop, Ty);
+ Value *const CtpopFP = B.CreateUIToFP(Ctpop, Ty);
NewV = B.CreateFMul(V, CtpopFP);
break;
}
@@ -882,7 +864,7 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
case AtomicRMWInst::Xor:
// The new value we will be contributing to the atomic operation is the
// old value times the parity of the number of active lanes.
- Value *Ctpop = B.CreateIntCast(
+ Value *const Ctpop = B.CreateIntCast(
B.CreateUnaryIntrinsic(Intrinsic::ctpop, Ballot), Ty, false);
NewV = buildMul(B, V, B.CreateAnd(Ctpop, 1));
break;
@@ -892,17 +874,17 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
// We only want a single lane to enter our new control flow, and we do this
// by checking if there are any active lanes below us. Only one lane will
// have 0 active lanes below us, so that will be the only one to progress.
- Value *Cond = B.CreateICmpEQ(Mbcnt, B.getInt32(0));
+ Value *const Cond = B.CreateICmpEQ(Mbcnt, B.getInt32(0));
// Store I's original basic block before we split the block.
- BasicBlock *OriginalBB = I.getParent();
+ BasicBlock *const OriginalBB = I.getParent();
// We need to introduce some new control flow to force a single lane to be
// active. We do this by splitting I's basic block at I, and introducing the
// new block such that:
// entry --> single_lane -\
// \------------------> exit
- Instruction *SingleLaneTerminator =
+ Instruction *const SingleLaneTerminator =
SplitBlockAndInsertIfThen(Cond, &I, false, nullptr, &DTU, nullptr);
// At this point, we have split the I's block to allow one lane in wavefront
@@ -952,7 +934,7 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
// Clone the original atomic operation into single lane, replacing the
// original value with our newly created one.
- Instruction *NewI = I.clone();
+ Instruction *const NewI = I.clone();
B.Insert(NewI);
NewI->setOperand(ValIdx, NewV);
@@ -962,7 +944,7 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
if (NeedResult) {
// Create a PHI node to get our new atomic result into the exit block.
- PHINode *PHI = B.CreatePHI(Ty, 2);
+ PHINode *const PHI = B.CreatePHI(Ty, 2);
PHI->addIncoming(PoisonValue::get(Ty), Predecessor);
PHI->addIncoming(NewI, SingleLaneTerminator->getParent());
diff --git a/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll b/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
index 7540cc64bc89d..642a3fcaa2b63 100644
--- a/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
+++ b/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
@@ -5,7 +5,7 @@
; RUN: llc -mtriple=amdgcn -mcpu=gfx900 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP < %s | FileCheck -enable-var-scope -check-prefixes=GFX900 %s
;
; Test the "amdgpu-atomic-optimizer-dpp-lds-threshold" function attribute,
-; which controls dynamic DPP vs no-opt branching for integer LDS atomics.
+; which controls dynamic DPP vs no-opt branching for LDS atomics.
;
; Threshold=5: use DPP only when active lanes > 5, otherwise each lane does
; its own atomic.
@@ -632,17 +632,30 @@ entry:
ret void
}
-; Test 5: divergent float fadd -- should NOT be affected by threshold (FP excluded)
+; Test 5: divergent float fadd with result used -- dynamic threshold branch expected
define amdgpu_kernel void @fadd_f32_varying(ptr addrspace(1) %out) #0 {
; GFX1032-LABEL: fadd_f32_varying:
; GFX1032: ; %bb.0: ; %entry
; GFX1032-NEXT: v_cvt_f32_i32_e32 v0, v0
+; GFX1032-NEXT: s_bcnt1_i32_b32 s0, exec_lo
+; GFX1032-NEXT: s_cmp_lt_u32 s0, 6
+; GFX1032-NEXT: s_cbranch_scc0 .LBB4_2
+; GFX1032-NEXT: ; %bb.1:
+; GFX1032-NEXT: v_mov_b32_e32 v4, 0
+; GFX1032-NEXT: ds_add_rtn_f32 v4, v4, v0
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_gl0_inv
+; GFX1032-NEXT: s_cbranch_execz .LBB4_3
+; GFX1032-NEXT: s_branch .LBB4_6
+; GFX1032-NEXT: .LBB4_2:
+; GFX1032-NEXT: ; implicit-def: $vgpr4
+; GFX1032-NEXT: .LBB4_3:
; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-NEXT: v_bfrev_b32_e32 v1, 1
-; GFX1032-NEXT: v_cndmask_b32_e64 v2, 0x80000000, v0, s0
+; GFX1032-NEXT: v_cndmask_b32_e64 v1, 0x80000000, v0, s0
+; GFX1032-NEXT: v_bfrev_b32_e32 v2, 1
; GFX1032-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1032-NEXT: v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT: v_add_f32_e32 v1, v2, v1
+; GFX1032-NEXT: v_mov_b32_dpp v2, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-NEXT: v_add_f32_e32 v1, v1, v2
; GFX1032-NEXT: v_bfrev_b32_e32 v2, 1
; GFX1032-NEXT: v_mov_b32_dpp v3, v1 row_shr:2 row_mask:0xf bank_mask:0xf
; GFX1032-NEXT: v_add_f32_e32 v1, v1, v3
@@ -656,48 +669,62 @@ define amdgpu_kernel void @fadd_f32_varying(ptr addrspace(1) %out) #0 {
; GFX1032-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
; GFX1032-NEXT: v_bfrev_b32_e32 v3, 1
; GFX1032-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1032-NEXT: v_readlane_b32 s1, v1, 15
; GFX1032-NEXT: v_readlane_b32 s2, v1, 31
; GFX1032-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-NEXT: v_readlane_b32 s1, v1, 15
; GFX1032-NEXT: s_mov_b32 exec_lo, s0
; GFX1032-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
; GFX1032-NEXT: v_writelane_b32 v3, s1, 16
; GFX1032-NEXT: s_mov_b32 exec_lo, s0
; GFX1032-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-NEXT: s_mov_b32 s0, s2
-; GFX1032-NEXT: s_mov_b32 s2, -1
; GFX1032-NEXT: ; implicit-def: $vgpr0
-; GFX1032-NEXT: s_and_saveexec_b32 s1, vcc_lo
-; GFX1032-NEXT: s_cbranch_execz .LBB4_2
-; GFX1032-NEXT: ; %bb.1:
+; GFX1032-NEXT: s_and_saveexec_b32 s0, vcc_lo
+; GFX1032-NEXT: s_cbranch_execz .LBB4_5
+; GFX1032-NEXT: ; %bb.4:
; GFX1032-NEXT: v_mov_b32_e32 v0, 0
-; GFX1032-NEXT: v_mov_b32_e32 v4, s0
+; GFX1032-NEXT: v_mov_b32_e32 v4, s2
+; GFX1032-NEXT: s_waitcnt_vscnt null, 0x0
; GFX1032-NEXT: ds_add_rtn_f32 v0, v0, v4
; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
; GFX1032-NEXT: buffer_gl0_inv
-; GFX1032-NEXT: .LBB4_2:
+; GFX1032-NEXT: .LBB4_5:
; GFX1032-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1032-NEXT: s_or_b32 exec_lo, exec_lo, s1
-; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1032-NEXT: v_readfirstlane_b32 s3, v0
+; GFX1032-NEXT: s_or_b32 exec_lo, exec_lo, s0
+; GFX1032-NEXT: v_readfirstlane_b32 s0, v0
; GFX1032-NEXT: v_mov_b32_e32 v0, v3
-; GFX1032-NEXT: v_add_f32_e32 v0, s3, v0
-; GFX1032-NEXT: v_cndmask_b32_e64 v0, v0, s3, vcc_lo
+; GFX1032-NEXT: v_add_f32_e32 v0, s0, v0
+; GFX1032-NEXT: v_cndmask_b32_e64 v4, v0, s0, vcc_lo
+; GFX1032-NEXT: .LBB4_6:
+; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
; GFX1032-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1032-NEXT: s_mov_b32 s2, -1
; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1032-NEXT: buffer_store_dword v4, off, s[0:3], 0
; GFX1032-NEXT: s_endpgm
;
; GFX1064-LABEL: fadd_f32_varying:
; GFX1064: ; %bb.0: ; %entry
; GFX1064-NEXT: v_cvt_f32_i32_e32 v0, v0
+; GFX1064-NEXT: s_bcnt1_i32_b64 s0, exec
+; GFX1064-NEXT: s_cmp_lt_u32 s0, 6
+; GFX1064-NEXT: s_cbranch_scc0 .LBB4_2
+; GFX1064-NEXT: ; %bb.1:
+; GFX1064-NEXT: v_mov_b32_e32 v4, 0
+; GFX1064-NEXT: ds_add_rtn_f32 v4, v4, v0
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: s_cbranch_execz .LBB4_3
+; GFX1064-NEXT: s_branch .LBB4_6
+; GFX1064-NEXT: .LBB4_2:
+; GFX1064-NEXT: ; implicit-def: $vgpr4
+; GFX1064-NEXT: .LBB4_3:
; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_bfrev_b32_e32 v1, 1
-; GFX1064-NEXT: v_cndmask_b32_e64 v2, 0x80000000, v0, s[0:1]
+; GFX1064-NEXT: v_cndmask_b32_e64 v1, 0x80000000, v0, s[0:1]
+; GFX1064-NEXT: v_bfrev_b32_e32 v2, 1
; GFX1064-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1064-NEXT: v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT: v_add_f32_e32 v1, v2, v1
+; GFX1064-NEXT: v_mov_b32_dpp v2, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT: v_add_f32_e32 v1, v1, v2
; GFX1064-NEXT: v_bfrev_b32_e32 v2, 1
; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:2 row_mask:0xf bank_mask:0xf
; GFX1064-NEXT: v_add_f32_e32 v1, v1, v3
@@ -723,41 +750,55 @@ define amdgpu_kernel void @fadd_f32_varying(ptr addrspace(1) %out) #0 {
; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_readlane_b32 s2, v1, 47
-; GFX1064-NEXT: v_readlane_b32 s6, v1, 63
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 63
+; GFX1064-NEXT: v_readlane_b32 s6, v1, 47
; GFX1064-NEXT: v_writelane_b32 v3, s3, 32
; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v0, exec_hi, v0
; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_writelane_b32 v3, s2, 48
+; GFX1064-NEXT: v_writelane_b32 v3, s6, 48
; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-NEXT: s_mov_b32 s2, -1
; GFX1064-NEXT: ; implicit-def: $vgpr0
; GFX1064-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX1064-NEXT: s_cbranch_execz .LBB4_2
-; GFX1064-NEXT: ; %bb.1:
+; GFX1064-NEXT: s_cbranch_execz .LBB4_5
+; GFX1064-NEXT: ; %bb.4:
; GFX1064-NEXT: v_mov_b32_e32 v0, 0
-; GFX1064-NEXT: v_mov_b32_e32 v4, s6
+; GFX1064-NEXT: v_mov_b32_e32 v4, s2
+; GFX1064-NEXT: s_waitcnt_vscnt null, 0x0
; GFX1064-NEXT: ds_add_rtn_f32 v0, v0, v4
; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
; GFX1064-NEXT: buffer_gl0_inv
-; GFX1064-NEXT: .LBB4_2:
+; GFX1064-NEXT: .LBB4_5:
; GFX1064-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX1064-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1064-NEXT: v_readfirstlane_b32 s3, v0
+; GFX1064-NEXT: v_readfirstlane_b32 s0, v0
; GFX1064-NEXT: v_mov_b32_e32 v0, v3
-; GFX1064-NEXT: v_add_f32_e32 v0, s3, v0
-; GFX1064-NEXT: v_cndmask_b32_e64 v0, v0, s3, vcc
+; GFX1064-NEXT: v_add_f32_e32 v0, s0, v0
+; GFX1064-NEXT: v_cndmask_b32_e64 v4, v0, s0, vcc
+; GFX1064-NEXT: .LBB4_6:
+; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
; GFX1064-NEXT: s_mov_b32 s3, 0x31016000
+; GFX1064-NEXT: s_mov_b32 s2, -1
; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX1064-NEXT: buffer_store_dword v4, off, s[0:3], 0
; GFX1064-NEXT: s_endpgm
;
; GFX900-LABEL: fadd_f32_varying:
; GFX900: ; %bb.0: ; %entry
; GFX900-NEXT: v_cvt_f32_i32_e32 v0, v0
+; GFX900-NEXT: s_bcnt1_i32_b64 s0, exec
+; GFX900-NEXT: s_cmp_lt_u32 s0, 6
+; GFX900-NEXT: s_cbranch_scc0 .LBB4_2
+; GFX900-NEXT: ; %bb.1:
+; GFX900-NEXT: v_mov_b32_e32 v4, 0
+; GFX900-NEXT: ds_add_rtn_f32 v4, v4, v0
+; GFX900-NEXT: s_waitcnt lgkmcnt(0)
+; GFX900-NEXT: s_cbranch_execz .LBB4_3
+; GFX900-NEXT: s_branch .LBB4_6
+; GFX900-NEXT: .LBB4_2:
+; GFX900-NEXT: ; implicit-def: $vgpr4
+; GFX900-NEXT: .LBB4_3:
; GFX900-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
; GFX900-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v4
; GFX900-NEXT: s_or_saveexec_b64 s[0:1], -1
@@ -794,24 +835,25 @@ define amdgpu_kernel void @fadd_f32_varying(ptr addrspace(1) %out) #0 {
; GFX900-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
; GFX900-NEXT: ; implicit-def: $vgpr0
; GFX900-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX900-NEXT: s_cbranch_execz .LBB4_2
-; GFX900-NEXT: ; %bb.1:
+; GFX900-NEXT: s_cbranch_execz .LBB4_5
+; GFX900-NEXT: ; %bb.4:
; GFX900-NEXT: v_mov_b32_e32 v0, 0
; GFX900-NEXT: v_mov_b32_e32 v4, s2
; GFX900-NEXT: ds_add_rtn_f32 v0, v0, v4
; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: .LBB4_2:
+; GFX900-NEXT: .LBB4_5:
; GFX900-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX900-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX900-NEXT: v_readfirstlane_b32 s4, v0
+; GFX900-NEXT: v_readfirstlane_b32 s0, v0
; GFX900-NEXT: v_mov_b32_e32 v0, v1
-; GFX900-NEXT: v_add_f32_e32 v0, s4, v0
-; GFX900-NEXT: v_mov_b32_e32 v4, s4
+; GFX900-NEXT: v_add_f32_e32 v0, s0, v0
+; GFX900-NEXT: v_mov_b32_e32 v4, s0
+; GFX900-NEXT: v_cndmask_b32_e32 v4, v0, v4, vcc
+; GFX900-NEXT: .LBB4_6:
+; GFX900-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
; GFX900-NEXT: s_mov_b32 s3, 0xf000
; GFX900-NEXT: s_mov_b32 s2, -1
-; GFX900-NEXT: v_cndmask_b32_e32 v0, v0, v4, vcc
; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: buffer_store_dword v0, off, s[0:3], 0
+; GFX900-NEXT: buffer_store_dword v4, off, s[0:3], 0
; GFX900-NEXT: s_endpgm
entry:
%lane = call i32 @llvm.amdgcn.workitem.id.x()
>From dae600ca5c27d67999f5d578f879ae7a65a0b4e5 Mon Sep 17 00:00:00 2001
From: YaFan <YaFan.Tao at amd.com>
Date: Fri, 24 Apr 2026 16:38:44 +0800
Subject: [PATCH 12/17] Replaced the function attribute with instruction-level
metadata (!amdgpu.atomic.lds.dpp) on the atomicrmw instruction itself, since
this is a per-atomic property rather than a per-function one.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Instead of exposing a numeric threshold as a user-facing knob, the metadata uses two symbolic values:
!"none" — disables the DPP optimization entirely for this atomic (each lane issues its own ds_* independently).
!"dynamic" — inserts a runtime active-lane-count check; if lanes exceed an internal threshold (currently hardcoded to 5), the DPP scan path is taken, otherwise each lane does its own atomic.
This avoids exposing implementation details (the threshold value) to the user. The hardcoded threshold of 5 should work well for most cases — it's the crossover point where DPP scan overhead is amortized by the reduction in LDS contention. The front-end or profiling tools only need to decide whether an atomic benefits from DPP, not how much, keeping the interface minimal. The metadata is silently ignored for non-LDS atomics, uniform values, or non-DPP strategies.
---
llvm/docs/AMDGPUUsage.rst | 51 ++++-
.../Target/AMDGPU/AMDGPUAtomicOptimizer.cpp | 40 ++--
.../atomic_optimizations_dpp_lds_threshold.ll | 203 +++---------------
3 files changed, 90 insertions(+), 204 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 21fc425526d0b..11762c5b5c9c2 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -1962,6 +1962,48 @@ and
!0 = !{}
+.. _amdgpu_atomic_lds_dpp:
+
+'``amdgpu.atomic.lds.dpp``' Metadata
+-------------------------------------------------
+
+Controls the DPP-based atomic optimization strategy for this LDS
+:ref:`atomicrmw <i_atomicrmw>` instruction. When the
+``-amdgpu-atomic-optimizer-strategy=DPP`` pass is active and the value
+operand is divergent, the metadata operand selects one of the following
+modes:
+
+``!"none"``
+ Disables the DPP optimization entirely for this atomic. Every lane
+ issues its own independent atomic operation. This is useful when
+ profiling or front-end heuristics indicate that the DPP scan overhead
+ is not profitable (e.g. the atomic is rarely contended or typically
+ executed by very few lanes). As a rule of thumb, if the active lane
+ count is mostly below the internal threshold (currently 5), ``!"none"``
+ is suggested.
+
+``!"dynamic"``
+ Inserts a runtime check of the active lane count. If the number of
+ active lanes exceeds an internal threshold (currently 5), the
+ wavefront takes the DPP scan path (one atomic for the whole
+ wavefront). Otherwise, each lane independently issues its own atomic,
+ avoiding the DPP scan overhead. This is useful when the active lane
+ count sometimes exceeds the threshold and sometimes does not.
+
+Without this metadata the optimizer unconditionally applies the DPP scan
+to every eligible LDS atomic.
+
+.. code-block:: llvm
+
+ ; Skip DPP — each lane issues its own ds_add_rtn_u32.
+ %old0 = atomicrmw add ptr addrspace(3) @lds, i32 %val acq_rel, !amdgpu.atomic.lds.dpp !0
+
+ ; DPP scan is used only when more than 5 lanes are active.
+ %old1 = atomicrmw add ptr addrspace(3) @lds, i32 %val acq_rel, !amdgpu.atomic.lds.dpp !1
+
+ !0 = !{!"none"}
+ !1 = !{!"dynamic"}
+
LLVM IR Attributes
==================
@@ -2205,15 +2247,6 @@ The AMDGPU backend supports the following LLVM IR attributes.
"amdgpu-no-fwd-progress" Disable forward progress mode for wave priority
(enabled by default).
- "amdgpu-atomic-optimizer-dpp-lds-threshold"="n" Threshold that controls when the AMDGPUAtomicOptimizer uses a DPP-based
- scan for LDS atomics with divergent values. When set to a positive
- integer n, the optimizer emits a dynamic branch: if the number of
- active lanes exceeds n, the DPP scan path is taken; otherwise each
- lane independently issues its own atomic. A value of 0 (the default)
- disables the dynamic branch and always uses the DPP scan path when
- the DPP strategy is selected. Values greater than or equal to the
- wavefront size effectively disable DPP optimization for LDS atomics.
-
Relevant for GFX10+.
================================================ ==========================================================
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index f6e2bc0c37b3c..ccc23d3635f2e 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -88,8 +88,7 @@ class AMDGPUAtomicOptimizerImpl
BasicBlock *ComputeLoop, BasicBlock *ComputeEnd) const;
void optimizeAtomic(Instruction &I, AtomicRMWInst::BinOp Op, unsigned ValIdx,
- bool ValDivergent, bool IsLDS,
- unsigned DPPThreshold) const;
+ bool ValDivergent, bool IsLDS) const;
Value *optimizeAtomicImpl(IRBuilder<> &B, Instruction &I,
AtomicRMWInst::BinOp Op, unsigned ValIdx,
@@ -170,14 +169,8 @@ bool AMDGPUAtomicOptimizerImpl::run() {
if (ToReplace.empty())
return false;
- unsigned DPPThreshold = 0;
- int AttrVal = F.getFnAttributeAsParsedInteger(
- "amdgpu-atomic-optimizer-dpp-lds-threshold", 0);
- if (AttrVal > 0)
- DPPThreshold = static_cast<unsigned>(AttrVal);
-
for (auto &[I, Op, ValIdx, ValDivergent, IsLDS] : ToReplace)
- optimizeAtomic(*I, Op, ValIdx, ValDivergent, IsLDS, DPPThreshold);
+ optimizeAtomic(*I, Op, ValIdx, ValDivergent, IsLDS);
ToReplace.clear();
return true;
}
@@ -664,13 +657,21 @@ static Value *buildMul(IRBuilder<> &B, Value *LHS, Value *RHS) {
void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
AtomicRMWInst::BinOp Op,
unsigned ValIdx,
- bool ValDivergent, bool IsLDS,
- unsigned DPPThreshold) const {
- // If the threshold is >= wavefront size, DPP is never profitable — skip
- // the atomic optimization entirely for this instruction.
- if (IsLDS && ValDivergent && ScanImpl == ScanOptions::DPP &&
- DPPThreshold >= ST.getWavefrontSize())
- return;
+ bool ValDivergent,
+ bool IsLDS) const {
+ // Check !amdgpu.atomic.lds.dpp metadata for per-atomic DPP control.
+ // !{!"none"} – skip DPP optimization entirely for this atomic.
+ // !{!"dynamic"} – use a dynamic active-lane-count branch.
+ StringRef LDSDPPMode;
+ if (IsLDS && ValDivergent && ScanImpl == ScanOptions::DPP) {
+ if (MDNode *MD = I.getMetadata("amdgpu.atomic.lds.dpp")) {
+ if (MD->getNumOperands() == 1)
+ if (auto *S = dyn_cast<MDString>(MD->getOperand(0)))
+ LDSDPPMode = S->getString();
+ if (LDSDPPMode == "none")
+ return;
+ }
+ }
// Start building just before the instruction.
IRBuilder<> B(&I);
@@ -705,9 +706,10 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
}
Value *Result = nullptr;
- if (IsLDS && ValDivergent && ScanImpl == ScanOptions::DPP &&
- DPPThreshold > 0) {
- Result = optimizeAtomicWithDynamicThreshold(B, I, Op, ValIdx, DPPThreshold);
+ if (LDSDPPMode == "dynamic") {
+ constexpr unsigned DynamicThreshold = 5;
+ Result =
+ optimizeAtomicWithDynamicThreshold(B, I, Op, ValIdx, DynamicThreshold);
} else {
Result = optimizeAtomicImpl(B, I, Op, ValIdx, ValDivergent);
}
diff --git a/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll b/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
index 642a3fcaa2b63..4f45aee92ab00 100644
--- a/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
+++ b/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
@@ -4,24 +4,19 @@
; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP < %s | FileCheck -enable-var-scope -check-prefixes=GFX1064 %s
; RUN: llc -mtriple=amdgcn -mcpu=gfx900 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP < %s | FileCheck -enable-var-scope -check-prefixes=GFX900 %s
;
-; Test the "amdgpu-atomic-optimizer-dpp-lds-threshold" function attribute,
-; which controls dynamic DPP vs no-opt branching for LDS atomics.
+; Test the !amdgpu.atomic.lds.dpp instruction metadata, which controls
+; the DPP-based atomic optimization for LDS atomics.
;
-; Threshold=5: use DPP only when active lanes > 5, otherwise each lane does
-; its own atomic.
-; Threshold=32: on wave32 this disables DPP entirely (>= wavefront size).
-; Threshold=64: on wave64 this disables DPP entirely (>= wavefront size).
+; !{!"dynamic"}: use DPP only when active lanes > 5 (hardcoded),
+; otherwise each lane does its own atomic.
+; !{!"none"}: skip DPP optimization entirely for this atomic.
declare i32 @llvm.amdgcn.workitem.id.x()
@local_var32 = addrspace(3) global i32 undef, align 4
-attributes #0 = { "amdgpu-atomic-optimizer-dpp-lds-threshold"="5" }
-attributes #1 = { "amdgpu-atomic-optimizer-dpp-lds-threshold"="32" }
-attributes #2 = { "amdgpu-atomic-optimizer-dpp-lds-threshold"="64" }
-
; Test 1: divergent i32 add with result used -- dynamic threshold branch expected
-define amdgpu_kernel void @add_i32_varying(ptr addrspace(1) %out) #0 {
+define amdgpu_kernel void @add_i32_varying(ptr addrspace(1) %out) {
; GFX1032-LABEL: add_i32_varying:
; GFX1032: ; %bb.0: ; %entry
; GFX1032-NEXT: s_bcnt1_i32_b32 s0, exec_lo
@@ -204,13 +199,13 @@ define amdgpu_kernel void @add_i32_varying(ptr addrspace(1) %out) #0 {
; GFX900-NEXT: s_endpgm
entry:
%lane = call i32 @llvm.amdgcn.workitem.id.x()
- %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel
+ %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel, !amdgpu.atomic.lds.dpp !0
store i32 %old, ptr addrspace(1) %out
ret void
}
; Test 2: divergent i32 sub with result used
-define amdgpu_kernel void @sub_i32_varying(ptr addrspace(1) %out) #0 {
+define amdgpu_kernel void @sub_i32_varying(ptr addrspace(1) %out) {
; GFX1032-LABEL: sub_i32_varying:
; GFX1032: ; %bb.0: ; %entry
; GFX1032-NEXT: s_bcnt1_i32_b32 s0, exec_lo
@@ -393,7 +388,7 @@ define amdgpu_kernel void @sub_i32_varying(ptr addrspace(1) %out) #0 {
; GFX900-NEXT: s_endpgm
entry:
%lane = call i32 @llvm.amdgcn.workitem.id.x()
- %old = atomicrmw sub ptr addrspace(3) @local_var32, i32 %lane acq_rel
+ %old = atomicrmw sub ptr addrspace(3) @local_var32, i32 %lane acq_rel, !amdgpu.atomic.lds.dpp !0
store i32 %old, ptr addrspace(1) %out
ret void
}
@@ -530,12 +525,12 @@ define amdgpu_kernel void @add_i32_varying_nouse() #0 {
; GFX900-NEXT: s_endpgm
entry:
%lane = call i32 @llvm.amdgcn.workitem.id.x()
- %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel
+ %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel, !amdgpu.atomic.lds.dpp !0
ret void
}
-; Test 4: uniform i32 add -- should NOT be affected by threshold (not divergent)
-define amdgpu_kernel void @add_i32_uniform(ptr addrspace(1) %out, i32 %additive) #0 {
+; Test 4: uniform i32 add with dynamic metadata -- metadata ignored (not divergent)
+define amdgpu_kernel void @add_i32_uniform(ptr addrspace(1) %out, i32 %additive) {
; GFX1032-LABEL: add_i32_uniform:
; GFX1032: ; %bb.0: ; %entry
; GFX1032-NEXT: s_load_dword s0, s[4:5], 0x2c
@@ -627,13 +622,13 @@ define amdgpu_kernel void @add_i32_uniform(ptr addrspace(1) %out, i32 %additive)
; GFX900-NEXT: buffer_store_dword v0, off, s[0:3], 0
; GFX900-NEXT: s_endpgm
entry:
- %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %additive acq_rel
+ %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %additive acq_rel, !amdgpu.atomic.lds.dpp !0
store i32 %old, ptr addrspace(1) %out
ret void
}
; Test 5: divergent float fadd with result used -- dynamic threshold branch expected
-define amdgpu_kernel void @fadd_f32_varying(ptr addrspace(1) %out) #0 {
+define amdgpu_kernel void @fadd_f32_varying(ptr addrspace(1) %out) {
; GFX1032-LABEL: fadd_f32_varying:
; GFX1032: ; %bb.0: ; %entry
; GFX1032-NEXT: v_cvt_f32_i32_e32 v0, v0
@@ -858,13 +853,13 @@ define amdgpu_kernel void @fadd_f32_varying(ptr addrspace(1) %out) #0 {
entry:
%lane = call i32 @llvm.amdgcn.workitem.id.x()
%conv = sitofp i32 %lane to float
- %rmw = atomicrmw fadd ptr addrspace(3) @local_var32, float %conv syncscope("agent") acq_rel
+ %rmw = atomicrmw fadd ptr addrspace(3) @local_var32, float %conv syncscope("agent") acq_rel, !amdgpu.atomic.lds.dpp !0
store float %rmw, ptr addrspace(1) %out
ret void
}
-; Test 6: divergent i32 add on GLOBAL memory -- should NOT be affected by threshold
-define amdgpu_kernel void @add_i32_varying_global(ptr addrspace(1) %out, ptr addrspace(1) %inout) #0 {
+; Test 6: divergent i32 add on GLOBAL memory -- should NOT be affected by metadata
+define amdgpu_kernel void @add_i32_varying_global(ptr addrspace(1) %out, ptr addrspace(1) %inout) {
; GFX1032-LABEL: add_i32_varying_global:
; GFX1032: ; %bb.0: ; %entry
; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
@@ -1027,15 +1022,14 @@ define amdgpu_kernel void @add_i32_varying_global(ptr addrspace(1) %out, ptr add
; GFX900-NEXT: s_endpgm
entry:
%lane = call i32 @llvm.amdgcn.workitem.id.x()
- %old = atomicrmw add ptr addrspace(1) %inout, i32 %lane acq_rel
+ %old = atomicrmw add ptr addrspace(1) %inout, i32 %lane acq_rel, !amdgpu.atomic.lds.dpp !0
store i32 %old, ptr addrspace(1) %out
ret void
}
-; Test 7: threshold=32 on wave32 (>= wavefront size) should avoid the dynamic
-; threshold branch.
-define amdgpu_kernel void @add_i32_varying_thresh32(ptr addrspace(1) %out) #1 {
-; GFX1032-LABEL: add_i32_varying_thresh32:
+; Test 7: !amdgpu.atomic.lds.dpp !{!"none"} disables DPP -- plain atomic expected
+define amdgpu_kernel void @add_i32_varying_no_lds_dpp(ptr addrspace(1) %out) {
+; GFX1032-LABEL: add_i32_varying_no_lds_dpp:
; GFX1032: ; %bb.0: ; %entry
; GFX1032-NEXT: v_mov_b32_e32 v1, 0
; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
@@ -1048,153 +1042,7 @@ define amdgpu_kernel void @add_i32_varying_thresh32(ptr addrspace(1) %out) #1 {
; GFX1032-NEXT: buffer_store_dword v0, off, s[0:3], 0
; GFX1032-NEXT: s_endpgm
;
-; GFX1064-LABEL: add_i32_varying_thresh32:
-; GFX1064: ; %bb.0: ; %entry
-; GFX1064-NEXT: s_bcnt1_i32_b64 s0, exec
-; GFX1064-NEXT: s_cmp_lt_u32 s0, 33
-; GFX1064-NEXT: s_cbranch_scc0 .LBB6_2
-; GFX1064-NEXT: ; %bb.1:
-; GFX1064-NEXT: v_mov_b32_e32 v4, 0
-; GFX1064-NEXT: ds_add_rtn_u32 v4, v4, v0
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: buffer_gl0_inv
-; GFX1064-NEXT: s_cbranch_execz .LBB6_3
-; GFX1064-NEXT: s_branch .LBB6_6
-; GFX1064-NEXT: .LBB6_2:
-; GFX1064-NEXT: ; implicit-def: $vgpr4
-; GFX1064-NEXT: .LBB6_3:
-; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX1064-NEXT: v_mov_b32_e32 v3, 0
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1064-NEXT: v_mov_b32_e32 v2, s2
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT: v_readlane_b32 s2, v1, 15
-; GFX1064-NEXT: v_readlane_b32 s3, v1, 31
-; GFX1064-NEXT: v_writelane_b32 v3, s2, 16
-; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_readlane_b32 s2, v1, 63
-; GFX1064-NEXT: v_readlane_b32 s6, v1, 47
-; GFX1064-NEXT: v_writelane_b32 v3, s3, 32
-; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX1064-NEXT: v_mov_b32_e32 v0, 0
-; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_writelane_b32 v3, s6, 48
-; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX1064-NEXT: ; implicit-def: $vgpr4
-; GFX1064-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX1064-NEXT: s_cbranch_execz .LBB6_5
-; GFX1064-NEXT: ; %bb.4:
-; GFX1064-NEXT: v_mov_b32_e32 v4, s2
-; GFX1064-NEXT: s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT: ds_add_rtn_u32 v4, v0, v4
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: buffer_gl0_inv
-; GFX1064-NEXT: .LBB6_5:
-; GFX1064-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1064-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX1064-NEXT: v_readfirstlane_b32 s0, v4
-; GFX1064-NEXT: v_mov_b32_e32 v0, v3
-; GFX1064-NEXT: v_add_nc_u32_e32 v4, s0, v0
-; GFX1064-NEXT: .LBB6_6:
-; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1064-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1064-NEXT: s_mov_b32 s2, -1
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: buffer_store_dword v4, off, s[0:3], 0
-; GFX1064-NEXT: s_endpgm
-;
-; GFX900-LABEL: add_i32_varying_thresh32:
-; GFX900: ; %bb.0: ; %entry
-; GFX900-NEXT: s_bcnt1_i32_b64 s0, exec
-; GFX900-NEXT: s_cmp_lt_u32 s0, 33
-; GFX900-NEXT: s_cbranch_scc0 .LBB6_2
-; GFX900-NEXT: ; %bb.1:
-; GFX900-NEXT: v_mov_b32_e32 v3, 0
-; GFX900-NEXT: ds_add_rtn_u32 v3, v3, v0
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: s_cbranch_execz .LBB6_3
-; GFX900-NEXT: s_branch .LBB6_6
-; GFX900-NEXT: .LBB6_2:
-; GFX900-NEXT: ; implicit-def: $vgpr3
-; GFX900-NEXT: .LBB6_3:
-; GFX900-NEXT: v_mov_b32_e32 v3, 0
-; GFX900-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX900-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX900-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX900-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX900-NEXT: v_mov_b32_e32 v2, 0
-; GFX900-NEXT: s_nop 0
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX900-NEXT: v_readlane_b32 s2, v1, 63
-; GFX900-NEXT: s_nop 0
-; GFX900-NEXT: v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX900-NEXT: s_mov_b64 exec, s[0:1]
-; GFX900-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX900-NEXT: ; implicit-def: $vgpr0
-; GFX900-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX900-NEXT: s_cbranch_execz .LBB6_5
-; GFX900-NEXT: ; %bb.4:
-; GFX900-NEXT: v_mov_b32_e32 v0, s2
-; GFX900-NEXT: ds_add_rtn_u32 v0, v3, v0
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: .LBB6_5:
-; GFX900-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX900-NEXT: v_readfirstlane_b32 s0, v0
-; GFX900-NEXT: v_mov_b32_e32 v0, v2
-; GFX900-NEXT: v_add_u32_e32 v3, s0, v0
-; GFX900-NEXT: .LBB6_6:
-; GFX900-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX900-NEXT: s_mov_b32 s3, 0xf000
-; GFX900-NEXT: s_mov_b32 s2, -1
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: buffer_store_dword v3, off, s[0:3], 0
-; GFX900-NEXT: s_endpgm
-entry:
- %lane = call i32 @llvm.amdgcn.workitem.id.x()
- %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel
- store i32 %old, ptr addrspace(1) %out
- ret void
-}
-
-; Test 8: threshold=64 on wave64 (>= wavefront size) should avoid the dynamic
-; threshold branch.
-define amdgpu_kernel void @add_i32_varying_thresh64(ptr addrspace(1) %out) #2 {
-; GFX1032-LABEL: add_i32_varying_thresh64:
-; GFX1032: ; %bb.0: ; %entry
-; GFX1032-NEXT: v_mov_b32_e32 v1, 0
-; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1032-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1032-NEXT: s_mov_b32 s2, -1
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: ds_add_rtn_u32 v0, v1, v0
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: buffer_gl0_inv
-; GFX1032-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1032-NEXT: s_endpgm
-;
-; GFX1064-LABEL: add_i32_varying_thresh64:
+; GFX1064-LABEL: add_i32_varying_no_lds_dpp:
; GFX1064: ; %bb.0: ; %entry
; GFX1064-NEXT: v_mov_b32_e32 v1, 0
; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
@@ -1207,7 +1055,7 @@ define amdgpu_kernel void @add_i32_varying_thresh64(ptr addrspace(1) %out) #2 {
; GFX1064-NEXT: buffer_store_dword v0, off, s[0:3], 0
; GFX1064-NEXT: s_endpgm
;
-; GFX900-LABEL: add_i32_varying_thresh64:
+; GFX900-LABEL: add_i32_varying_no_lds_dpp:
; GFX900: ; %bb.0: ; %entry
; GFX900-NEXT: v_mov_b32_e32 v1, 0
; GFX900-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
@@ -1220,7 +1068,10 @@ define amdgpu_kernel void @add_i32_varying_thresh64(ptr addrspace(1) %out) #2 {
; GFX900-NEXT: s_endpgm
entry:
%lane = call i32 @llvm.amdgcn.workitem.id.x()
- %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel
+ %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel, !amdgpu.atomic.lds.dpp !1
store i32 %old, ptr addrspace(1) %out
ret void
}
+
+!0 = !{!"dynamic"}
+!1 = !{!"none"}
>From f3f788f23d437b6b11fd368ebb85fe82414ca914 Mon Sep 17 00:00:00 2001
From: YaFan <YaFan.Tao at amd.com>
Date: Fri, 5 Jun 2026 12:18:02 +0800
Subject: [PATCH 13/17] [AMDGPU] Replace amdgpu.atomic.lds.dpp with
amdgpu.expected.active.lanes hint
Redesign the per-atomic DPP control metadata introduced in the previous commit. Instead of a string-valued !amdgpu.atomic.lds.dpp with "none" and "dynamic" modes, use a single i32 hint, !amdgpu.expected.active.lanes, giving the number of wavefront lanes expected to be active (an absolute, wave32/wave64-independent count).
The optimizer now simply skips the DPP scan for a divergent LDS atomic when the expected active lane count is small (<= 5), and applies it otherwise. This drops the "dynamic" runtime active-lane-count branch entirely, removing optimizeAtomicWithDynamicThreshold and inlining optimizeAtomicImpl back into optimizeAtomic (NFC for the scan path).
Add Verifier enforcement of the metadata structure (exactly one i32 constant operand) with a negative test; the value itself stays an unchecked profiling hint. Update the documentation and replace the old threshold codegen test with one covering the new metadata across wave32, wave64 and gfx900.
---
llvm/docs/AMDGPUUsage.rst | 61 +-
llvm/lib/IR/Verifier.cpp | 11 +
.../Target/AMDGPU/AMDGPUAtomicOptimizer.cpp | 155 +--
...mizations_dpp_lds_expected_active_lanes.ll | 382 ++++++
.../atomic_optimizations_dpp_lds_threshold.ll | 1077 -----------------
.../Verifier/amdgpu-expected-active-lanes.ll | 30 +
6 files changed, 477 insertions(+), 1239 deletions(-)
create mode 100644 llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_expected_active_lanes.ll
delete mode 100644 llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
create mode 100644 llvm/test/Verifier/amdgpu-expected-active-lanes.ll
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index f8eba1568af92..0f0bc9c4fdeaa 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -2234,47 +2234,42 @@ and
!0 = !{}
-.. _amdgpu_atomic_lds_dpp:
+.. _amdgpu_expected_active_lanes:
-'``amdgpu.atomic.lds.dpp``' Metadata
+'``amdgpu.expected.active.lanes``' Metadata
-------------------------------------------------
-Controls the DPP-based atomic optimization strategy for this LDS
-:ref:`atomicrmw <i_atomicrmw>` instruction. When the
-``-amdgpu-atomic-optimizer-strategy=DPP`` pass is active and the value
-operand is divergent, the metadata operand selects one of the following
-modes:
-
-``!"none"``
- Disables the DPP optimization entirely for this atomic. Every lane
- issues its own independent atomic operation. This is useful when
- profiling or front-end heuristics indicate that the DPP scan overhead
- is not profitable (e.g. the atomic is rarely contended or typically
- executed by very few lanes). As a rule of thumb, if the active lane
- count is mostly below the internal threshold (currently 5), ``!"none"``
- is suggested.
-
-``!"dynamic"``
- Inserts a runtime check of the active lane count. If the number of
- active lanes exceeds an internal threshold (currently 5), the
- wavefront takes the DPP scan path (one atomic for the whole
- wavefront). Otherwise, each lane independently issues its own atomic,
- avoiding the DPP scan overhead. This is useful when the active lane
- count sometimes exceeds the threshold and sometimes does not.
-
-Without this metadata the optimizer unconditionally applies the DPP scan
-to every eligible LDS atomic.
+A profiling-derived hint describing how many lanes of the wavefront are
+expected to be active when this :ref:`atomicrmw <i_atomicrmw>` instruction
+executes. The metadata has a single ``i32`` constant operand giving the
+expected number of active lanes as an absolute count (the same value is used
+for both wave32 and wave64 targets). This structure is enforced by the
+verifier.
+
+The AMDGPU atomic optimizer uses the hint as follows. When the
+``-amdgpu-atomic-optimizer-strategy=DPP`` strategy is active, the value
+operand is divergent, and the atomic is in the LDS (``addrspace(3)``) address
+space, the optimizer normally replaces the per-lane atomics with a single
+wavefront-wide DPP scan. That scan has a fixed overhead that is only
+worthwhile when enough lanes participate. If the expected active lane count
+is small (currently ``<= 5``), the optimizer skips the DPP scan and instead
+lets each lane issue its own atomic.
+
+The metadata is ignored for non-LDS atomics and when the DPP strategy is not
+in use. Without this metadata the optimizer unconditionally applies the DPP
+scan to every eligible LDS atomic.
.. code-block:: llvm
- ; Skip DPP — each lane issues its own ds_add_rtn_u32.
- %old0 = atomicrmw add ptr addrspace(3) @lds, i32 %val acq_rel, !amdgpu.atomic.lds.dpp !0
+ ; Few expected active lanes (<= 5): DPP is skipped and each lane issues its
+ ; own ds_add_rtn_u32.
+ %old0 = atomicrmw add ptr addrspace(3) @lds, i32 %val acq_rel, !amdgpu.expected.active.lanes !0
- ; DPP scan is used only when more than 5 lanes are active.
- %old1 = atomicrmw add ptr addrspace(3) @lds, i32 %val acq_rel, !amdgpu.atomic.lds.dpp !1
+ ; Many expected active lanes (> 5): the DPP scan is applied.
+ %old1 = atomicrmw add ptr addrspace(3) @lds, i32 %val acq_rel, !amdgpu.expected.active.lanes !1
- !0 = !{!"none"}
- !1 = !{!"dynamic"}
+ !0 = !{i32 4}
+ !1 = !{i32 32}
LLVM IR Attributes
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index c9639d1420bfc..e02419a088519 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -6026,6 +6026,17 @@ void Verifier::visitInstruction(Instruction &I) {
if (MDNode *MD = I.getMetadata(LLVMContext::MD_mem_cache_hint))
visitMemCacheHintMetadata(I, MD);
+ if (MDNode *MD = I.getMetadata("amdgpu.expected.active.lanes")) {
+ Check(MD->getNumOperands() == 1,
+ "!amdgpu.expected.active.lanes must have exactly one operand", &I,
+ MD);
+ ConstantInt *CI =
+ mdconst::dyn_extract_or_null<ConstantInt>(MD->getOperand(0));
+ Check(CI && CI->getType()->isIntegerTy(32),
+ "!amdgpu.expected.active.lanes operand must be an i32 constant", &I,
+ MD);
+ }
+
if (MDNode *N = I.getDebugLoc().getAsMDNode()) {
CheckDI(isa<DILocation>(N), "invalid !dbg metadata attachment", &I, N);
visitMDNode(*N, AreDebugLocsAllowed::Yes);
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index fbfa0080341a8..0d5dda8ba5c66 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -90,15 +90,6 @@ class AMDGPUAtomicOptimizerImpl
void optimizeAtomic(Instruction &I, AtomicRMWInst::BinOp Op, unsigned ValIdx,
bool ValDivergent, bool IsLDS) const;
- Value *optimizeAtomicImpl(IRBuilder<> &B, Instruction &I,
- AtomicRMWInst::BinOp Op, unsigned ValIdx,
- bool ValDivergent) const;
-
- Value *optimizeAtomicWithDynamicThreshold(IRBuilder<> &B, Instruction &I,
- AtomicRMWInst::BinOp Op,
- unsigned ValIdx,
- unsigned Threshold) const;
-
public:
AMDGPUAtomicOptimizerImpl() = delete;
@@ -661,16 +652,19 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
unsigned ValIdx,
bool ValDivergent,
bool IsLDS) const {
- // Check !amdgpu.atomic.lds.dpp metadata for per-atomic DPP control.
- // !{!"none"} – skip DPP optimization entirely for this atomic.
- // !{!"dynamic"} – use a dynamic active-lane-count branch.
- StringRef LDSDPPMode;
+ // The DPP scan has a fixed overhead, so it only pays off with enough active
+ // lanes. The !amdgpu.expected.active.lanes metadata hints at how many lanes
+ // are expected to be active; for a divergent LDS atomic, when that count is
+ // small (<= 5), skip DPP and let each lane issue its own atomic.
+ //
+ // FIXME: The threshold was tuned empirically on gfx12. The DPP scan overhead
+ // differs across subtargets, so the break-even point may differ too; this may
+ // need to become subtarget-dependent.
if (IsLDS && ValDivergent && ScanImpl == ScanOptions::DPP) {
- if (MDNode *MD = I.getMetadata("amdgpu.atomic.lds.dpp")) {
- if (MD->getNumOperands() == 1)
- if (auto *S = dyn_cast<MDString>(MD->getOperand(0)))
- LDSDPPMode = S->getString();
- if (LDSDPPMode == "none")
+ if (MDNode *MD = I.getMetadata("amdgpu.expected.active.lanes")) {
+ auto *CI = mdconst::extract<ConstantInt>(MD->getOperand(0));
+ constexpr unsigned ActiveLanesThreshold = 5;
+ if (CI->getValue().ule(ActiveLanesThreshold))
return;
}
}
@@ -707,43 +701,6 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
B.SetInsertPoint(&I);
}
- Value *Result = nullptr;
- if (LDSDPPMode == "dynamic") {
- constexpr unsigned DynamicThreshold = 5;
- Result =
- optimizeAtomicWithDynamicThreshold(B, I, Op, ValIdx, DynamicThreshold);
- } else {
- Result = optimizeAtomicImpl(B, I, Op, ValIdx, ValDivergent);
- }
-
- // Handle pixel shader reconvergence and replace original instruction.
- if (Result) {
- if (IsPixelShader) {
- // Need a final PHI to reconverge to above the helper lane branch mask.
- B.SetInsertPoint(PixelExitBB, PixelExitBB->getFirstNonPHIIt());
-
- PHINode *PHI = B.CreatePHI(I.getType(), 2);
- PHI->addIncoming(PoisonValue::get(I.getType()), PixelEntryBB);
- PHI->addIncoming(Result, I.getParent());
- I.replaceAllUsesWith(PHI);
- } else {
- // Replace the original atomic instruction with the new one.
- I.replaceAllUsesWith(Result);
- }
- }
-
- // And delete the original.
- I.eraseFromParent();
-}
-
-// Replace a single atomic with a wavefront-wide scan/reduction followed by a
-// single-lane atomic, then reconstruct per-lane results. Returns the per-lane
-// result, or nullptr when the result is unused.
-Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
- Instruction &I,
- AtomicRMWInst::BinOp Op,
- unsigned ValIdx,
- bool ValDivergent) const {
Type *const Ty = I.getType();
Type *Int32Ty = B.getInt32Ty();
bool isAtomicFloatingPointTy = Ty->isFloatingPointTy();
@@ -1022,83 +979,23 @@ Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicImpl(IRBuilder<> &B,
// first active lane.
Result = B.CreateSelect(Cond, BroadcastI, Result);
}
- return Result;
- }
- return nullptr;
-}
-// Generate a dynamic branch based on the active lane count so that the
-// optimized scan path is only used when enough lanes are active to amortise
-// its overhead. When the active count is at or below the threshold, each
-// lane independently issues its own atomic, which is cheaper for small
-// groups.
-//
-// This function builds the outer threshold CFG and delegates the actual
-// atomic optimization to optimizeAtomicImpl.
-//
-// CFG produced (the OptimizedPath sub-CFG is built by optimizeAtomicImpl):
-//
-// EntryBB:
-// Ballot / Ctpop -> ActiveCount
-// cmp ActiveCount > Threshold
-// br -> OptBB or NoOptBB
-//
-// OptBB:
-// [optimizeAtomicImpl rewrites I here, creating its own internal CFG]
-// br -> TailBB
-//
-// NoOptBB:
-// original atomic (unoptimized, each lane issues its own)
-// br -> TailBB
-//
-// TailBB:
-// PHI merges results from both paths
-Value *AMDGPUAtomicOptimizerImpl::optimizeAtomicWithDynamicThreshold(
- IRBuilder<> &B, Instruction &I, AtomicRMWInst::BinOp Op, unsigned ValIdx,
- unsigned Threshold) const {
- Type *Ty = I.getType();
- const bool NeedResult = !I.use_empty();
+ if (IsPixelShader) {
+ // Need a final PHI to reconverge to above the helper lane branch mask.
+ B.SetInsertPoint(PixelExitBB, PixelExitBB->getFirstNonPHIIt());
- // Count active lanes and build the threshold condition.
- Type *WaveTy = B.getIntNTy(ST.getWavefrontSize());
- CallInst *Ballot =
- B.CreateIntrinsic(Intrinsic::amdgcn_ballot, WaveTy, B.getTrue());
- Value *Ctpop = B.CreateIntCast(
- B.CreateUnaryIntrinsic(Intrinsic::ctpop, Ballot), B.getInt32Ty(), false);
- Value *ThresholdCond = B.CreateICmpUGT(Ctpop, B.getInt32(Threshold));
-
- // Split at I into: EntryBB -> OptBB (then) / NoOptBB (else) -> TailBB.
- Instruction *ThenTerm = nullptr, *ElseTerm = nullptr;
- SplitBlockAndInsertIfThenElse(ThresholdCond, &I, &ThenTerm, &ElseTerm,
- nullptr, &DTU, nullptr);
- BasicBlock *NoOptBB = ElseTerm->getParent();
- BasicBlock *TailBB = I.getParent();
-
- // NoOptBB: each lane independently issues its own atomic (unoptimized).
- Instruction *NoOptNewI = I.clone();
- NoOptNewI->insertBefore(ElseTerm->getIterator());
-
- // Move I into OptBB so that optimizeAtomicImpl rewrites only the optimized
- // path. I's uses are temporarily non-dominating; they will be fixed when
- // optimizeAtomic does replaceAllUsesWith after we return.
- I.moveBefore(ThenTerm->getIterator());
- B.SetInsertPoint(&I);
+ PHINode *const PHI = B.CreatePHI(Ty, 2);
+ PHI->addIncoming(PoisonValue::get(Ty), PixelEntryBB);
+ PHI->addIncoming(Result, I.getParent());
+ I.replaceAllUsesWith(PHI);
+ } else {
+ // Replace the original atomic instruction with the new one.
+ I.replaceAllUsesWith(Result);
+ }
+ }
- // Delegate the full scan/reduction + single-lane atomic + result
- // reconstruction to optimizeAtomicImpl.
- Value *OptResult =
- optimizeAtomicImpl(B, I, Op, ValIdx, /*ValDivergent=*/true);
-
- if (!NeedResult)
- return nullptr;
-
- // After optimizeAtomicImpl, I sits in the exit block of the optimized
- // sub-CFG. Merge the optimized and no-opt results in TailBB.
- B.SetInsertPoint(TailBB, TailBB->getFirstNonPHIIt());
- PHINode *MergePHI = B.CreatePHI(Ty, 2);
- MergePHI->addIncoming(OptResult, I.getParent());
- MergePHI->addIncoming(NoOptNewI, NoOptBB);
- return MergePHI;
+ // And delete the original.
+ I.eraseFromParent();
}
INITIALIZE_PASS_BEGIN(AMDGPUAtomicOptimizer, DEBUG_TYPE,
diff --git a/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_expected_active_lanes.ll b/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_expected_active_lanes.ll
new file mode 100644
index 0000000000000..db2e2d515073f
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_expected_active_lanes.ll
@@ -0,0 +1,382 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+;
+; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32 -amdgpu-atomic-optimizer-strategy=DPP < %s | FileCheck -enable-var-scope -check-prefixes=GFX1032 %s
+; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize64 -amdgpu-atomic-optimizer-strategy=DPP < %s | FileCheck -enable-var-scope -check-prefixes=GFX1064 %s
+;
+; Test the !amdgpu.expected.active.lanes instruction metadata, a
+; profiling-derived hint giving the number of wavefront lanes expected to be
+; active when the atomic executes. When the expected active lane count is small
+; (<= 5), the DPP scan overhead is not worthwhile, so the DPP optimization is
+; skipped for divergent LDS atomics and each lane issues its own atomic.
+
+declare i32 @llvm.amdgcn.workitem.id.x()
+
+ at local_var32 = addrspace(3) global i32 undef, align 4
+
+; Few expected active lanes (<= 5) on a divergent LDS atomic: DPP is skipped,
+; so a plain per-lane ds_add is emitted with no DPP scan.
+define amdgpu_kernel void @add_i32_low_active_lanes(ptr addrspace(1) %out) {
+; GFX1032-LABEL: add_i32_low_active_lanes:
+; GFX1032: ; %bb.0: ; %entry
+; GFX1032-NEXT: v_mov_b32_e32 v1, 0
+; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: ds_add_rtn_u32 v0, v1, v0
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_gl0_inv
+; GFX1032-NEXT: global_store_dword v1, v0, s[0:1]
+; GFX1032-NEXT: s_endpgm
+;
+; GFX1064-LABEL: add_i32_low_active_lanes:
+; GFX1064: ; %bb.0: ; %entry
+; GFX1064-NEXT: v_mov_b32_e32 v1, 0
+; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: ds_add_rtn_u32 v0, v1, v0
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: global_store_dword v1, v0, s[0:1]
+; GFX1064-NEXT: s_endpgm
+entry:
+ %lane = call i32 @llvm.amdgcn.workitem.id.x()
+ %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel, !amdgpu.expected.active.lanes !0
+ store i32 %old, ptr addrspace(1) %out
+ ret void
+}
+
+; Boundary case: exactly 5 expected active lanes still skips DPP.
+define amdgpu_kernel void @add_i32_boundary_active_lanes(ptr addrspace(1) %out) {
+; GFX1032-LABEL: add_i32_boundary_active_lanes:
+; GFX1032: ; %bb.0: ; %entry
+; GFX1032-NEXT: v_mov_b32_e32 v1, 0
+; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: ds_add_rtn_u32 v0, v1, v0
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_gl0_inv
+; GFX1032-NEXT: global_store_dword v1, v0, s[0:1]
+; GFX1032-NEXT: s_endpgm
+;
+; GFX1064-LABEL: add_i32_boundary_active_lanes:
+; GFX1064: ; %bb.0: ; %entry
+; GFX1064-NEXT: v_mov_b32_e32 v1, 0
+; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: ds_add_rtn_u32 v0, v1, v0
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: global_store_dword v1, v0, s[0:1]
+; GFX1064-NEXT: s_endpgm
+entry:
+ %lane = call i32 @llvm.amdgcn.workitem.id.x()
+ %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel, !amdgpu.expected.active.lanes !1
+ store i32 %old, ptr addrspace(1) %out
+ ret void
+}
+
+; Many expected active lanes (> 5): the DPP scan optimization is applied.
+define amdgpu_kernel void @add_i32_high_active_lanes(ptr addrspace(1) %out) {
+; GFX1032-LABEL: add_i32_high_active_lanes:
+; GFX1032: ; %bb.0: ; %entry
+; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1032-NEXT: v_mov_b32_e32 v3, 0
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1032-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1032-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-NEXT: v_readlane_b32 s1, v1, 15
+; GFX1032-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1032-NEXT: v_mov_b32_e32 v0, 0
+; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-NEXT: v_writelane_b32 v3, s1, 16
+; GFX1032-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
+; GFX1032-NEXT: ; implicit-def: $vgpr4
+; GFX1032-NEXT: s_and_saveexec_b32 s0, vcc_lo
+; GFX1032-NEXT: s_cbranch_execz .LBB2_2
+; GFX1032-NEXT: ; %bb.1:
+; GFX1032-NEXT: v_mov_b32_e32 v4, s2
+; GFX1032-NEXT: ds_add_rtn_u32 v4, v0, v4
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_gl0_inv
+; GFX1032-NEXT: .LBB2_2:
+; GFX1032-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-NEXT: s_or_b32 exec_lo, exec_lo, s0
+; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1032-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1032-NEXT: v_mov_b32_e32 v4, v3
+; GFX1032-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: global_store_dword v0, v4, s[0:1]
+; GFX1032-NEXT: s_endpgm
+;
+; GFX1064-LABEL: add_i32_high_active_lanes:
+; GFX1064: ; %bb.0: ; %entry
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1064-NEXT: v_mov_b32_e32 v3, 0
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1064-NEXT: v_mov_b32_e32 v2, s2
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 15
+; GFX1064-NEXT: v_readlane_b32 s3, v1, 31
+; GFX1064-NEXT: v_writelane_b32 v3, s2, 16
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 63
+; GFX1064-NEXT: v_readlane_b32 s6, v1, 47
+; GFX1064-NEXT: v_writelane_b32 v3, s3, 32
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
+; GFX1064-NEXT: v_mov_b32_e32 v0, 0
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_writelane_b32 v3, s6, 48
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX1064-NEXT: ; implicit-def: $vgpr4
+; GFX1064-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1064-NEXT: s_cbranch_execz .LBB2_2
+; GFX1064-NEXT: ; %bb.1:
+; GFX1064-NEXT: v_mov_b32_e32 v4, s2
+; GFX1064-NEXT: ds_add_rtn_u32 v4, v0, v4
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: .LBB2_2:
+; GFX1064-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1064-NEXT: v_mov_b32_e32 v4, v3
+; GFX1064-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: global_store_dword v0, v4, s[0:1]
+; GFX1064-NEXT: s_endpgm
+entry:
+ %lane = call i32 @llvm.amdgcn.workitem.id.x()
+ %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel, !amdgpu.expected.active.lanes !2
+ store i32 %old, ptr addrspace(1) %out
+ ret void
+}
+
+; No metadata: the DPP scan optimization is applied by default.
+define amdgpu_kernel void @add_i32_no_metadata(ptr addrspace(1) %out) {
+; GFX1032-LABEL: add_i32_no_metadata:
+; GFX1032: ; %bb.0: ; %entry
+; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1032-NEXT: v_mov_b32_e32 v3, 0
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1032-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1032-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-NEXT: v_readlane_b32 s1, v1, 15
+; GFX1032-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1032-NEXT: v_mov_b32_e32 v0, 0
+; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-NEXT: v_writelane_b32 v3, s1, 16
+; GFX1032-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
+; GFX1032-NEXT: ; implicit-def: $vgpr4
+; GFX1032-NEXT: s_and_saveexec_b32 s0, vcc_lo
+; GFX1032-NEXT: s_cbranch_execz .LBB3_2
+; GFX1032-NEXT: ; %bb.1:
+; GFX1032-NEXT: v_mov_b32_e32 v4, s2
+; GFX1032-NEXT: ds_add_rtn_u32 v4, v0, v4
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: buffer_gl0_inv
+; GFX1032-NEXT: .LBB3_2:
+; GFX1032-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-NEXT: s_or_b32 exec_lo, exec_lo, s0
+; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1032-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1032-NEXT: v_mov_b32_e32 v4, v3
+; GFX1032-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: global_store_dword v0, v4, s[0:1]
+; GFX1032-NEXT: s_endpgm
+;
+; GFX1064-LABEL: add_i32_no_metadata:
+; GFX1064: ; %bb.0: ; %entry
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1064-NEXT: v_mov_b32_e32 v3, 0
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1064-NEXT: v_mov_b32_e32 v2, s2
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 15
+; GFX1064-NEXT: v_readlane_b32 s3, v1, 31
+; GFX1064-NEXT: v_writelane_b32 v3, s2, 16
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 63
+; GFX1064-NEXT: v_readlane_b32 s6, v1, 47
+; GFX1064-NEXT: v_writelane_b32 v3, s3, 32
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
+; GFX1064-NEXT: v_mov_b32_e32 v0, 0
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_writelane_b32 v3, s6, 48
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX1064-NEXT: ; implicit-def: $vgpr4
+; GFX1064-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1064-NEXT: s_cbranch_execz .LBB3_2
+; GFX1064-NEXT: ; %bb.1:
+; GFX1064-NEXT: v_mov_b32_e32 v4, s2
+; GFX1064-NEXT: ds_add_rtn_u32 v4, v0, v4
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: .LBB3_2:
+; GFX1064-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1064-NEXT: v_mov_b32_e32 v4, v3
+; GFX1064-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: global_store_dword v0, v4, s[0:1]
+; GFX1064-NEXT: s_endpgm
+entry:
+ %lane = call i32 @llvm.amdgcn.workitem.id.x()
+ %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel
+ store i32 %old, ptr addrspace(1) %out
+ ret void
+}
+
+; Non-LDS (global) atomic: the metadata only governs LDS atomics, so a small
+; active-lane hint is ignored and the DPP scan optimization is still applied.
+define amdgpu_kernel void @add_i32_global_ignores_metadata(ptr addrspace(1) %inout) {
+; GFX1032-LABEL: add_i32_global_ignores_metadata:
+; GFX1032: ; %bb.0: ; %entry
+; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1032-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1032-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1032-NEXT: s_mov_b32 exec_lo, s0
+; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1032-NEXT: s_or_saveexec_b32 s2, -1
+; GFX1032-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1032-NEXT: v_mov_b32_e32 v3, 0
+; GFX1032-NEXT: v_readlane_b32 s4, v1, 31
+; GFX1032-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-NEXT: v_readlane_b32 s3, v1, 15
+; GFX1032-NEXT: s_mov_b32 exec_lo, s2
+; GFX1032-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1032-NEXT: v_mov_b32_e32 v0, 0
+; GFX1032-NEXT: s_or_saveexec_b32 s2, -1
+; GFX1032-NEXT: v_writelane_b32 v3, s3, 16
+; GFX1032-NEXT: s_mov_b32 exec_lo, s2
+; GFX1032-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
+; GFX1032-NEXT: ; implicit-def: $vgpr4
+; GFX1032-NEXT: s_and_saveexec_b32 s2, vcc_lo
+; GFX1032-NEXT: s_cbranch_execz .LBB4_2
+; GFX1032-NEXT: ; %bb.1:
+; GFX1032-NEXT: v_mov_b32_e32 v4, s4
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: global_atomic_add v4, v0, v4, s[0:1] glc
+; GFX1032-NEXT: s_waitcnt vmcnt(0)
+; GFX1032-NEXT: buffer_gl1_inv
+; GFX1032-NEXT: buffer_gl0_inv
+; GFX1032-NEXT: .LBB4_2:
+; GFX1032-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1032-NEXT: s_or_b32 exec_lo, exec_lo, s2
+; GFX1032-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1032-NEXT: v_mov_b32_e32 v4, v3
+; GFX1032-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1032-NEXT: global_store_dword v0, v4, s[0:1]
+; GFX1032-NEXT: s_endpgm
+;
+; GFX1064-LABEL: add_i32_global_ignores_metadata:
+; GFX1064: ; %bb.0: ; %entry
+; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1064-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1064-NEXT: v_mov_b32_e32 v3, 0
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1064-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1064-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1064-NEXT: v_mov_b32_e32 v2, s2
+; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1064-NEXT: v_readlane_b32 s6, v1, 15
+; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
+; GFX1064-NEXT: s_or_saveexec_b64 s[2:3], -1
+; GFX1064-NEXT: v_readlane_b32 s4, v1, 31
+; GFX1064-NEXT: v_writelane_b32 v3, s6, 16
+; GFX1064-NEXT: s_mov_b64 exec, s[2:3]
+; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1064-NEXT: s_or_saveexec_b64 s[2:3], -1
+; GFX1064-NEXT: v_readlane_b32 s6, v1, 63
+; GFX1064-NEXT: v_readlane_b32 s5, v1, 47
+; GFX1064-NEXT: v_writelane_b32 v3, s4, 32
+; GFX1064-NEXT: s_mov_b64 exec, s[2:3]
+; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
+; GFX1064-NEXT: v_mov_b32_e32 v0, 0
+; GFX1064-NEXT: s_or_saveexec_b64 s[2:3], -1
+; GFX1064-NEXT: v_writelane_b32 v3, s5, 48
+; GFX1064-NEXT: s_mov_b64 exec, s[2:3]
+; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX1064-NEXT: ; implicit-def: $vgpr4
+; GFX1064-NEXT: s_and_saveexec_b64 s[2:3], vcc
+; GFX1064-NEXT: s_cbranch_execz .LBB4_2
+; GFX1064-NEXT: ; %bb.1:
+; GFX1064-NEXT: v_mov_b32_e32 v4, s6
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: global_atomic_add v4, v0, v4, s[0:1] glc
+; GFX1064-NEXT: s_waitcnt vmcnt(0)
+; GFX1064-NEXT: buffer_gl1_inv
+; GFX1064-NEXT: buffer_gl0_inv
+; GFX1064-NEXT: .LBB4_2:
+; GFX1064-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX1064-NEXT: s_or_b64 exec, exec, s[2:3]
+; GFX1064-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1064-NEXT: v_mov_b32_e32 v4, v3
+; GFX1064-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1064-NEXT: global_store_dword v0, v4, s[0:1]
+; GFX1064-NEXT: s_endpgm
+entry:
+ %lane = call i32 @llvm.amdgcn.workitem.id.x()
+ %old = atomicrmw add ptr addrspace(1) %inout, i32 %lane acq_rel, !amdgpu.expected.active.lanes !0
+ store i32 %old, ptr addrspace(1) %inout
+ ret void
+}
+
+!0 = !{i32 4}
+!1 = !{i32 5}
+!2 = !{i32 32}
diff --git a/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll b/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
deleted file mode 100644
index 4f45aee92ab00..0000000000000
--- a/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_threshold.ll
+++ /dev/null
@@ -1,1077 +0,0 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-;
-; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP < %s | FileCheck -enable-var-scope -check-prefixes=GFX1032 %s
-; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize64 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP < %s | FileCheck -enable-var-scope -check-prefixes=GFX1064 %s
-; RUN: llc -mtriple=amdgcn -mcpu=gfx900 -mattr=-flat-for-global -amdgpu-atomic-optimizer-strategy=DPP < %s | FileCheck -enable-var-scope -check-prefixes=GFX900 %s
-;
-; Test the !amdgpu.atomic.lds.dpp instruction metadata, which controls
-; the DPP-based atomic optimization for LDS atomics.
-;
-; !{!"dynamic"}: use DPP only when active lanes > 5 (hardcoded),
-; otherwise each lane does its own atomic.
-; !{!"none"}: skip DPP optimization entirely for this atomic.
-
-declare i32 @llvm.amdgcn.workitem.id.x()
-
- at local_var32 = addrspace(3) global i32 undef, align 4
-
-; Test 1: divergent i32 add with result used -- dynamic threshold branch expected
-define amdgpu_kernel void @add_i32_varying(ptr addrspace(1) %out) {
-; GFX1032-LABEL: add_i32_varying:
-; GFX1032: ; %bb.0: ; %entry
-; GFX1032-NEXT: s_bcnt1_i32_b32 s0, exec_lo
-; GFX1032-NEXT: s_cmp_lt_u32 s0, 6
-; GFX1032-NEXT: s_cbranch_scc0 .LBB0_2
-; GFX1032-NEXT: ; %bb.1:
-; GFX1032-NEXT: v_mov_b32_e32 v4, 0
-; GFX1032-NEXT: ds_add_rtn_u32 v4, v4, v0
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: buffer_gl0_inv
-; GFX1032-NEXT: s_cbranch_execz .LBB0_3
-; GFX1032-NEXT: s_branch .LBB0_6
-; GFX1032-NEXT: .LBB0_2:
-; GFX1032-NEXT: ; implicit-def: $vgpr4
-; GFX1032-NEXT: .LBB0_3:
-; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
-; GFX1032-NEXT: v_mov_b32_e32 v3, 0
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1032-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT: v_readlane_b32 s1, v1, 15
-; GFX1032-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX1032-NEXT: v_mov_b32_e32 v0, 0
-; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-NEXT: v_writelane_b32 v3, s1, 16
-; GFX1032-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX1032-NEXT: ; implicit-def: $vgpr4
-; GFX1032-NEXT: s_and_saveexec_b32 s0, vcc_lo
-; GFX1032-NEXT: s_cbranch_execz .LBB0_5
-; GFX1032-NEXT: ; %bb.4:
-; GFX1032-NEXT: v_mov_b32_e32 v4, s2
-; GFX1032-NEXT: s_waitcnt_vscnt null, 0x0
-; GFX1032-NEXT: ds_add_rtn_u32 v4, v0, v4
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: buffer_gl0_inv
-; GFX1032-NEXT: .LBB0_5:
-; GFX1032-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1032-NEXT: s_or_b32 exec_lo, exec_lo, s0
-; GFX1032-NEXT: v_readfirstlane_b32 s0, v4
-; GFX1032-NEXT: v_mov_b32_e32 v0, v3
-; GFX1032-NEXT: v_add_nc_u32_e32 v4, s0, v0
-; GFX1032-NEXT: .LBB0_6:
-; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1032-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1032-NEXT: s_mov_b32 s2, -1
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: buffer_store_dword v4, off, s[0:3], 0
-; GFX1032-NEXT: s_endpgm
-;
-; GFX1064-LABEL: add_i32_varying:
-; GFX1064: ; %bb.0: ; %entry
-; GFX1064-NEXT: s_bcnt1_i32_b64 s0, exec
-; GFX1064-NEXT: s_cmp_lt_u32 s0, 6
-; GFX1064-NEXT: s_cbranch_scc0 .LBB0_2
-; GFX1064-NEXT: ; %bb.1:
-; GFX1064-NEXT: v_mov_b32_e32 v4, 0
-; GFX1064-NEXT: ds_add_rtn_u32 v4, v4, v0
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: buffer_gl0_inv
-; GFX1064-NEXT: s_cbranch_execz .LBB0_3
-; GFX1064-NEXT: s_branch .LBB0_6
-; GFX1064-NEXT: .LBB0_2:
-; GFX1064-NEXT: ; implicit-def: $vgpr4
-; GFX1064-NEXT: .LBB0_3:
-; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX1064-NEXT: v_mov_b32_e32 v3, 0
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1064-NEXT: v_mov_b32_e32 v2, s2
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT: v_readlane_b32 s2, v1, 15
-; GFX1064-NEXT: v_readlane_b32 s3, v1, 31
-; GFX1064-NEXT: v_writelane_b32 v3, s2, 16
-; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_readlane_b32 s2, v1, 63
-; GFX1064-NEXT: v_readlane_b32 s6, v1, 47
-; GFX1064-NEXT: v_writelane_b32 v3, s3, 32
-; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX1064-NEXT: v_mov_b32_e32 v0, 0
-; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_writelane_b32 v3, s6, 48
-; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX1064-NEXT: ; implicit-def: $vgpr4
-; GFX1064-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX1064-NEXT: s_cbranch_execz .LBB0_5
-; GFX1064-NEXT: ; %bb.4:
-; GFX1064-NEXT: v_mov_b32_e32 v4, s2
-; GFX1064-NEXT: s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT: ds_add_rtn_u32 v4, v0, v4
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: buffer_gl0_inv
-; GFX1064-NEXT: .LBB0_5:
-; GFX1064-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1064-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX1064-NEXT: v_readfirstlane_b32 s0, v4
-; GFX1064-NEXT: v_mov_b32_e32 v0, v3
-; GFX1064-NEXT: v_add_nc_u32_e32 v4, s0, v0
-; GFX1064-NEXT: .LBB0_6:
-; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1064-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1064-NEXT: s_mov_b32 s2, -1
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: buffer_store_dword v4, off, s[0:3], 0
-; GFX1064-NEXT: s_endpgm
-;
-; GFX900-LABEL: add_i32_varying:
-; GFX900: ; %bb.0: ; %entry
-; GFX900-NEXT: s_bcnt1_i32_b64 s0, exec
-; GFX900-NEXT: s_cmp_lt_u32 s0, 6
-; GFX900-NEXT: s_cbranch_scc0 .LBB0_2
-; GFX900-NEXT: ; %bb.1:
-; GFX900-NEXT: v_mov_b32_e32 v3, 0
-; GFX900-NEXT: ds_add_rtn_u32 v3, v3, v0
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: s_cbranch_execz .LBB0_3
-; GFX900-NEXT: s_branch .LBB0_6
-; GFX900-NEXT: .LBB0_2:
-; GFX900-NEXT: ; implicit-def: $vgpr3
-; GFX900-NEXT: .LBB0_3:
-; GFX900-NEXT: v_mov_b32_e32 v3, 0
-; GFX900-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX900-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX900-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX900-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX900-NEXT: v_mov_b32_e32 v2, 0
-; GFX900-NEXT: s_nop 0
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX900-NEXT: v_readlane_b32 s2, v1, 63
-; GFX900-NEXT: s_nop 0
-; GFX900-NEXT: v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX900-NEXT: s_mov_b64 exec, s[0:1]
-; GFX900-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX900-NEXT: ; implicit-def: $vgpr0
-; GFX900-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX900-NEXT: s_cbranch_execz .LBB0_5
-; GFX900-NEXT: ; %bb.4:
-; GFX900-NEXT: v_mov_b32_e32 v0, s2
-; GFX900-NEXT: ds_add_rtn_u32 v0, v3, v0
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: .LBB0_5:
-; GFX900-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX900-NEXT: v_readfirstlane_b32 s0, v0
-; GFX900-NEXT: v_mov_b32_e32 v0, v2
-; GFX900-NEXT: v_add_u32_e32 v3, s0, v0
-; GFX900-NEXT: .LBB0_6:
-; GFX900-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX900-NEXT: s_mov_b32 s3, 0xf000
-; GFX900-NEXT: s_mov_b32 s2, -1
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: buffer_store_dword v3, off, s[0:3], 0
-; GFX900-NEXT: s_endpgm
-entry:
- %lane = call i32 @llvm.amdgcn.workitem.id.x()
- %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel, !amdgpu.atomic.lds.dpp !0
- store i32 %old, ptr addrspace(1) %out
- ret void
-}
-
-; Test 2: divergent i32 sub with result used
-define amdgpu_kernel void @sub_i32_varying(ptr addrspace(1) %out) {
-; GFX1032-LABEL: sub_i32_varying:
-; GFX1032: ; %bb.0: ; %entry
-; GFX1032-NEXT: s_bcnt1_i32_b32 s0, exec_lo
-; GFX1032-NEXT: s_cmp_lt_u32 s0, 6
-; GFX1032-NEXT: s_cbranch_scc0 .LBB1_2
-; GFX1032-NEXT: ; %bb.1:
-; GFX1032-NEXT: v_mov_b32_e32 v4, 0
-; GFX1032-NEXT: ds_sub_rtn_u32 v4, v4, v0
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: buffer_gl0_inv
-; GFX1032-NEXT: s_cbranch_execz .LBB1_3
-; GFX1032-NEXT: s_branch .LBB1_6
-; GFX1032-NEXT: .LBB1_2:
-; GFX1032-NEXT: ; implicit-def: $vgpr4
-; GFX1032-NEXT: .LBB1_3:
-; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
-; GFX1032-NEXT: v_mov_b32_e32 v3, 0
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1032-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT: v_readlane_b32 s1, v1, 15
-; GFX1032-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX1032-NEXT: v_mov_b32_e32 v0, 0
-; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-NEXT: v_writelane_b32 v3, s1, 16
-; GFX1032-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX1032-NEXT: ; implicit-def: $vgpr4
-; GFX1032-NEXT: s_and_saveexec_b32 s0, vcc_lo
-; GFX1032-NEXT: s_cbranch_execz .LBB1_5
-; GFX1032-NEXT: ; %bb.4:
-; GFX1032-NEXT: v_mov_b32_e32 v4, s2
-; GFX1032-NEXT: s_waitcnt_vscnt null, 0x0
-; GFX1032-NEXT: ds_sub_rtn_u32 v4, v0, v4
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: buffer_gl0_inv
-; GFX1032-NEXT: .LBB1_5:
-; GFX1032-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1032-NEXT: s_or_b32 exec_lo, exec_lo, s0
-; GFX1032-NEXT: v_readfirstlane_b32 s0, v4
-; GFX1032-NEXT: v_mov_b32_e32 v0, v3
-; GFX1032-NEXT: v_sub_nc_u32_e32 v4, s0, v0
-; GFX1032-NEXT: .LBB1_6:
-; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1032-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1032-NEXT: s_mov_b32 s2, -1
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: buffer_store_dword v4, off, s[0:3], 0
-; GFX1032-NEXT: s_endpgm
-;
-; GFX1064-LABEL: sub_i32_varying:
-; GFX1064: ; %bb.0: ; %entry
-; GFX1064-NEXT: s_bcnt1_i32_b64 s0, exec
-; GFX1064-NEXT: s_cmp_lt_u32 s0, 6
-; GFX1064-NEXT: s_cbranch_scc0 .LBB1_2
-; GFX1064-NEXT: ; %bb.1:
-; GFX1064-NEXT: v_mov_b32_e32 v4, 0
-; GFX1064-NEXT: ds_sub_rtn_u32 v4, v4, v0
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: buffer_gl0_inv
-; GFX1064-NEXT: s_cbranch_execz .LBB1_3
-; GFX1064-NEXT: s_branch .LBB1_6
-; GFX1064-NEXT: .LBB1_2:
-; GFX1064-NEXT: ; implicit-def: $vgpr4
-; GFX1064-NEXT: .LBB1_3:
-; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX1064-NEXT: v_mov_b32_e32 v3, 0
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1064-NEXT: v_mov_b32_e32 v2, s2
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT: v_readlane_b32 s2, v1, 15
-; GFX1064-NEXT: v_readlane_b32 s3, v1, 31
-; GFX1064-NEXT: v_writelane_b32 v3, s2, 16
-; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_readlane_b32 s2, v1, 63
-; GFX1064-NEXT: v_readlane_b32 s6, v1, 47
-; GFX1064-NEXT: v_writelane_b32 v3, s3, 32
-; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
-; GFX1064-NEXT: v_mov_b32_e32 v0, 0
-; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_writelane_b32 v3, s6, 48
-; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX1064-NEXT: ; implicit-def: $vgpr4
-; GFX1064-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX1064-NEXT: s_cbranch_execz .LBB1_5
-; GFX1064-NEXT: ; %bb.4:
-; GFX1064-NEXT: v_mov_b32_e32 v4, s2
-; GFX1064-NEXT: s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT: ds_sub_rtn_u32 v4, v0, v4
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: buffer_gl0_inv
-; GFX1064-NEXT: .LBB1_5:
-; GFX1064-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1064-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX1064-NEXT: v_readfirstlane_b32 s0, v4
-; GFX1064-NEXT: v_mov_b32_e32 v0, v3
-; GFX1064-NEXT: v_sub_nc_u32_e32 v4, s0, v0
-; GFX1064-NEXT: .LBB1_6:
-; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1064-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1064-NEXT: s_mov_b32 s2, -1
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: buffer_store_dword v4, off, s[0:3], 0
-; GFX1064-NEXT: s_endpgm
-;
-; GFX900-LABEL: sub_i32_varying:
-; GFX900: ; %bb.0: ; %entry
-; GFX900-NEXT: s_bcnt1_i32_b64 s0, exec
-; GFX900-NEXT: s_cmp_lt_u32 s0, 6
-; GFX900-NEXT: s_cbranch_scc0 .LBB1_2
-; GFX900-NEXT: ; %bb.1:
-; GFX900-NEXT: v_mov_b32_e32 v3, 0
-; GFX900-NEXT: ds_sub_rtn_u32 v3, v3, v0
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: s_cbranch_execz .LBB1_3
-; GFX900-NEXT: s_branch .LBB1_6
-; GFX900-NEXT: .LBB1_2:
-; GFX900-NEXT: ; implicit-def: $vgpr3
-; GFX900-NEXT: .LBB1_3:
-; GFX900-NEXT: v_mov_b32_e32 v3, 0
-; GFX900-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX900-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX900-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX900-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX900-NEXT: v_mov_b32_e32 v2, 0
-; GFX900-NEXT: s_nop 0
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX900-NEXT: v_readlane_b32 s2, v1, 63
-; GFX900-NEXT: s_nop 0
-; GFX900-NEXT: v_mov_b32_dpp v2, v1 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX900-NEXT: s_mov_b64 exec, s[0:1]
-; GFX900-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX900-NEXT: ; implicit-def: $vgpr0
-; GFX900-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX900-NEXT: s_cbranch_execz .LBB1_5
-; GFX900-NEXT: ; %bb.4:
-; GFX900-NEXT: v_mov_b32_e32 v0, s2
-; GFX900-NEXT: ds_sub_rtn_u32 v0, v3, v0
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: .LBB1_5:
-; GFX900-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX900-NEXT: v_readfirstlane_b32 s0, v0
-; GFX900-NEXT: v_mov_b32_e32 v0, v2
-; GFX900-NEXT: v_sub_u32_e32 v3, s0, v0
-; GFX900-NEXT: .LBB1_6:
-; GFX900-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX900-NEXT: s_mov_b32 s3, 0xf000
-; GFX900-NEXT: s_mov_b32 s2, -1
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: buffer_store_dword v3, off, s[0:3], 0
-; GFX900-NEXT: s_endpgm
-entry:
- %lane = call i32 @llvm.amdgcn.workitem.id.x()
- %old = atomicrmw sub ptr addrspace(3) @local_var32, i32 %lane acq_rel, !amdgpu.atomic.lds.dpp !0
- store i32 %old, ptr addrspace(1) %out
- ret void
-}
-
-; Test 3: divergent i32 add without result used (NeedResult=false)
-define amdgpu_kernel void @add_i32_varying_nouse() #0 {
-; GFX1032-LABEL: add_i32_varying_nouse:
-; GFX1032: ; %bb.0: ; %entry
-; GFX1032-NEXT: s_bcnt1_i32_b32 s0, exec_lo
-; GFX1032-NEXT: s_cmp_lt_u32 s0, 6
-; GFX1032-NEXT: s_mov_b32 s0, -1
-; GFX1032-NEXT: s_cbranch_scc0 .LBB2_2
-; GFX1032-NEXT: ; %bb.1:
-; GFX1032-NEXT: v_mov_b32_e32 v3, 0
-; GFX1032-NEXT: s_mov_b32 s0, 0
-; GFX1032-NEXT: ds_add_u32 v3, v0
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: buffer_gl0_inv
-; GFX1032-NEXT: .LBB2_2: ; %Flow1
-; GFX1032-NEXT: s_andn2_b32 vcc_lo, exec_lo, s0
-; GFX1032-NEXT: s_cbranch_vccnz .LBB2_5
-; GFX1032-NEXT: ; %bb.3:
-; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT: v_permlanex16_b32 v2, v1, 0, 0
-; GFX1032-NEXT: v_add_nc_u32_e32 v1, v1, v2
-; GFX1032-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX1032-NEXT: v_mov_b32_e32 v0, 0
-; GFX1032-NEXT: v_mov_b32_e32 v3, v1
-; GFX1032-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
-; GFX1032-NEXT: s_and_saveexec_b32 s0, vcc_lo
-; GFX1032-NEXT: s_cbranch_execz .LBB2_5
-; GFX1032-NEXT: ; %bb.4:
-; GFX1032-NEXT: s_waitcnt_vscnt null, 0x0
-; GFX1032-NEXT: ds_add_u32 v0, v3
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: buffer_gl0_inv
-; GFX1032-NEXT: .LBB2_5:
-; GFX1032-NEXT: s_endpgm
-;
-; GFX1064-LABEL: add_i32_varying_nouse:
-; GFX1064: ; %bb.0: ; %entry
-; GFX1064-NEXT: s_bcnt1_i32_b64 s0, exec
-; GFX1064-NEXT: s_cmp_lt_u32 s0, 6
-; GFX1064-NEXT: s_mov_b64 s[0:1], -1
-; GFX1064-NEXT: s_cbranch_scc0 .LBB2_2
-; GFX1064-NEXT: ; %bb.1:
-; GFX1064-NEXT: v_mov_b32_e32 v3, 0
-; GFX1064-NEXT: s_mov_b64 s[0:1], 0
-; GFX1064-NEXT: ds_add_u32 v3, v0
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: buffer_gl0_inv
-; GFX1064-NEXT: .LBB2_2: ; %Flow1
-; GFX1064-NEXT: s_andn2_b64 vcc, exec, s[0:1]
-; GFX1064-NEXT: s_cbranch_vccnz .LBB2_5
-; GFX1064-NEXT: ; %bb.3:
-; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_xmask:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_permlanex16_b32 v2, v1, 0, 0
-; GFX1064-NEXT: v_add_nc_u32_e32 v1, v1, v2
-; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_readlane_b32 s2, v1, 0
-; GFX1064-NEXT: v_readlane_b32 s3, v1, 32
-; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v3, exec_hi, v0
-; GFX1064-NEXT: v_mov_b32_e32 v0, 0
-; GFX1064-NEXT: s_add_i32 s0, s2, s3
-; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v3
-; GFX1064-NEXT: s_and_saveexec_b64 s[2:3], vcc
-; GFX1064-NEXT: s_cbranch_execz .LBB2_5
-; GFX1064-NEXT: ; %bb.4:
-; GFX1064-NEXT: v_mov_b32_e32 v3, s0
-; GFX1064-NEXT: s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT: ds_add_u32 v0, v3
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: buffer_gl0_inv
-; GFX1064-NEXT: .LBB2_5:
-; GFX1064-NEXT: s_endpgm
-;
-; GFX900-LABEL: add_i32_varying_nouse:
-; GFX900: ; %bb.0: ; %entry
-; GFX900-NEXT: s_bcnt1_i32_b64 s0, exec
-; GFX900-NEXT: s_cmp_lt_u32 s0, 6
-; GFX900-NEXT: s_mov_b64 s[0:1], -1
-; GFX900-NEXT: s_cbranch_scc0 .LBB2_2
-; GFX900-NEXT: ; %bb.1:
-; GFX900-NEXT: v_mov_b32_e32 v2, 0
-; GFX900-NEXT: ds_add_u32 v2, v0
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: s_mov_b64 s[0:1], 0
-; GFX900-NEXT: .LBB2_2: ; %Flow1
-; GFX900-NEXT: s_andn2_b64 vcc, exec, s[0:1]
-; GFX900-NEXT: s_cbranch_vccnz .LBB2_5
-; GFX900-NEXT: ; %bb.3:
-; GFX900-NEXT: v_mov_b32_e32 v2, 0
-; GFX900-NEXT: v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX900-NEXT: v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX900-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX900-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v1, v1, v1 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX900-NEXT: v_readlane_b32 s2, v1, 63
-; GFX900-NEXT: s_mov_b64 exec, s[0:1]
-; GFX900-NEXT: s_mov_b32 s0, s2
-; GFX900-NEXT: v_cmp_eq_u32_e32 vcc, 0, v3
-; GFX900-NEXT: s_and_saveexec_b64 s[2:3], vcc
-; GFX900-NEXT: s_cbranch_execz .LBB2_5
-; GFX900-NEXT: ; %bb.4:
-; GFX900-NEXT: v_mov_b32_e32 v0, s0
-; GFX900-NEXT: ds_add_u32 v2, v0
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: .LBB2_5:
-; GFX900-NEXT: s_endpgm
-entry:
- %lane = call i32 @llvm.amdgcn.workitem.id.x()
- %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel, !amdgpu.atomic.lds.dpp !0
- ret void
-}
-
-; Test 4: uniform i32 add with dynamic metadata -- metadata ignored (not divergent)
-define amdgpu_kernel void @add_i32_uniform(ptr addrspace(1) %out, i32 %additive) {
-; GFX1032-LABEL: add_i32_uniform:
-; GFX1032: ; %bb.0: ; %entry
-; GFX1032-NEXT: s_load_dword s0, s[4:5], 0x2c
-; GFX1032-NEXT: s_mov_b32 s2, exec_lo
-; GFX1032-NEXT: ; implicit-def: $vgpr1
-; GFX1032-NEXT: v_mbcnt_lo_u32_b32 v0, s2, 0
-; GFX1032-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-NEXT: s_and_saveexec_b32 s1, vcc_lo
-; GFX1032-NEXT: s_cbranch_execz .LBB3_2
-; GFX1032-NEXT: ; %bb.1:
-; GFX1032-NEXT: s_bcnt1_i32_b32 s2, s2
-; GFX1032-NEXT: v_mov_b32_e32 v1, 0
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: s_mul_i32 s2, s0, s2
-; GFX1032-NEXT: v_mov_b32_e32 v2, s2
-; GFX1032-NEXT: ds_add_rtn_u32 v1, v1, v2
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: buffer_gl0_inv
-; GFX1032-NEXT: .LBB3_2:
-; GFX1032-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1032-NEXT: s_or_b32 exec_lo, exec_lo, s1
-; GFX1032-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x24
-; GFX1032-NEXT: v_readfirstlane_b32 s2, v1
-; GFX1032-NEXT: s_mov_b32 s11, 0x31016000
-; GFX1032-NEXT: s_mov_b32 s10, -1
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: v_mad_u64_u32 v[0:1], s0, s0, v0, s[2:3]
-; GFX1032-NEXT: buffer_store_dword v0, off, s[8:11], 0
-; GFX1032-NEXT: s_endpgm
-;
-; GFX1064-LABEL: add_i32_uniform:
-; GFX1064: ; %bb.0: ; %entry
-; GFX1064-NEXT: s_load_dword s6, s[4:5], 0x2c
-; GFX1064-NEXT: s_mov_b64 s[2:3], exec
-; GFX1064-NEXT: ; implicit-def: $vgpr1
-; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, s2, 0
-; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v0, s3, v0
-; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX1064-NEXT: s_cbranch_execz .LBB3_2
-; GFX1064-NEXT: ; %bb.1:
-; GFX1064-NEXT: s_bcnt1_i32_b64 s2, s[2:3]
-; GFX1064-NEXT: v_mov_b32_e32 v1, 0
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: s_mul_i32 s2, s6, s2
-; GFX1064-NEXT: v_mov_b32_e32 v2, s2
-; GFX1064-NEXT: ds_add_rtn_u32 v1, v1, v2
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: buffer_gl0_inv
-; GFX1064-NEXT: .LBB3_2:
-; GFX1064-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1064-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1064-NEXT: v_readfirstlane_b32 s2, v1
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: v_mad_u64_u32 v[0:1], s[2:3], s6, v0, s[2:3]
-; GFX1064-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1064-NEXT: s_mov_b32 s2, -1
-; GFX1064-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1064-NEXT: s_endpgm
-;
-; GFX900-LABEL: add_i32_uniform:
-; GFX900: ; %bb.0: ; %entry
-; GFX900-NEXT: s_load_dword s6, s[4:5], 0x2c
-; GFX900-NEXT: s_mov_b64 s[2:3], exec
-; GFX900-NEXT: v_mbcnt_lo_u32_b32 v0, s2, 0
-; GFX900-NEXT: v_mbcnt_hi_u32_b32 v0, s3, v0
-; GFX900-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX900-NEXT: ; implicit-def: $vgpr1
-; GFX900-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX900-NEXT: s_cbranch_execz .LBB3_2
-; GFX900-NEXT: ; %bb.1:
-; GFX900-NEXT: s_bcnt1_i32_b64 s2, s[2:3]
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: s_mul_i32 s2, s6, s2
-; GFX900-NEXT: v_mov_b32_e32 v1, 0
-; GFX900-NEXT: v_mov_b32_e32 v2, s2
-; GFX900-NEXT: ds_add_rtn_u32 v1, v1, v2
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: .LBB3_2:
-; GFX900-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX900-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: v_mul_lo_u32 v0, s6, v0
-; GFX900-NEXT: v_readfirstlane_b32 s4, v1
-; GFX900-NEXT: s_mov_b32 s3, 0xf000
-; GFX900-NEXT: s_mov_b32 s2, -1
-; GFX900-NEXT: v_add_u32_e32 v0, s4, v0
-; GFX900-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX900-NEXT: s_endpgm
-entry:
- %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %additive acq_rel, !amdgpu.atomic.lds.dpp !0
- store i32 %old, ptr addrspace(1) %out
- ret void
-}
-
-; Test 5: divergent float fadd with result used -- dynamic threshold branch expected
-define amdgpu_kernel void @fadd_f32_varying(ptr addrspace(1) %out) {
-; GFX1032-LABEL: fadd_f32_varying:
-; GFX1032: ; %bb.0: ; %entry
-; GFX1032-NEXT: v_cvt_f32_i32_e32 v0, v0
-; GFX1032-NEXT: s_bcnt1_i32_b32 s0, exec_lo
-; GFX1032-NEXT: s_cmp_lt_u32 s0, 6
-; GFX1032-NEXT: s_cbranch_scc0 .LBB4_2
-; GFX1032-NEXT: ; %bb.1:
-; GFX1032-NEXT: v_mov_b32_e32 v4, 0
-; GFX1032-NEXT: ds_add_rtn_f32 v4, v4, v0
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: buffer_gl0_inv
-; GFX1032-NEXT: s_cbranch_execz .LBB4_3
-; GFX1032-NEXT: s_branch .LBB4_6
-; GFX1032-NEXT: .LBB4_2:
-; GFX1032-NEXT: ; implicit-def: $vgpr4
-; GFX1032-NEXT: .LBB4_3:
-; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-NEXT: v_cndmask_b32_e64 v1, 0x80000000, v0, s0
-; GFX1032-NEXT: v_bfrev_b32_e32 v2, 1
-; GFX1032-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1032-NEXT: v_mov_b32_dpp v2, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1032-NEXT: v_bfrev_b32_e32 v2, 1
-; GFX1032-NEXT: v_mov_b32_dpp v3, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT: v_add_f32_e32 v1, v1, v3
-; GFX1032-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1032-NEXT: v_mov_b32_dpp v2, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1032-NEXT: v_bfrev_b32_e32 v2, 1
-; GFX1032-NEXT: v_mov_b32_dpp v3, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT: v_add_f32_e32 v1, v1, v3
-; GFX1032-NEXT: v_permlanex16_b32 v3, v1, -1, -1
-; GFX1032-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1032-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1032-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1032-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT: v_readlane_b32 s1, v1, 15
-; GFX1032-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-NEXT: v_writelane_b32 v3, s1, 16
-; GFX1032-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-NEXT: ; implicit-def: $vgpr0
-; GFX1032-NEXT: s_and_saveexec_b32 s0, vcc_lo
-; GFX1032-NEXT: s_cbranch_execz .LBB4_5
-; GFX1032-NEXT: ; %bb.4:
-; GFX1032-NEXT: v_mov_b32_e32 v0, 0
-; GFX1032-NEXT: v_mov_b32_e32 v4, s2
-; GFX1032-NEXT: s_waitcnt_vscnt null, 0x0
-; GFX1032-NEXT: ds_add_rtn_f32 v0, v0, v4
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: buffer_gl0_inv
-; GFX1032-NEXT: .LBB4_5:
-; GFX1032-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1032-NEXT: s_or_b32 exec_lo, exec_lo, s0
-; GFX1032-NEXT: v_readfirstlane_b32 s0, v0
-; GFX1032-NEXT: v_mov_b32_e32 v0, v3
-; GFX1032-NEXT: v_add_f32_e32 v0, s0, v0
-; GFX1032-NEXT: v_cndmask_b32_e64 v4, v0, s0, vcc_lo
-; GFX1032-NEXT: .LBB4_6:
-; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1032-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1032-NEXT: s_mov_b32 s2, -1
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: buffer_store_dword v4, off, s[0:3], 0
-; GFX1032-NEXT: s_endpgm
-;
-; GFX1064-LABEL: fadd_f32_varying:
-; GFX1064: ; %bb.0: ; %entry
-; GFX1064-NEXT: v_cvt_f32_i32_e32 v0, v0
-; GFX1064-NEXT: s_bcnt1_i32_b64 s0, exec
-; GFX1064-NEXT: s_cmp_lt_u32 s0, 6
-; GFX1064-NEXT: s_cbranch_scc0 .LBB4_2
-; GFX1064-NEXT: ; %bb.1:
-; GFX1064-NEXT: v_mov_b32_e32 v4, 0
-; GFX1064-NEXT: ds_add_rtn_f32 v4, v4, v0
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: buffer_gl0_inv
-; GFX1064-NEXT: s_cbranch_execz .LBB4_3
-; GFX1064-NEXT: s_branch .LBB4_6
-; GFX1064-NEXT: .LBB4_2:
-; GFX1064-NEXT: ; implicit-def: $vgpr4
-; GFX1064-NEXT: .LBB4_3:
-; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_cndmask_b32_e64 v1, 0x80000000, v0, s[0:1]
-; GFX1064-NEXT: v_bfrev_b32_e32 v2, 1
-; GFX1064-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1064-NEXT: v_mov_b32_dpp v2, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1064-NEXT: v_bfrev_b32_e32 v2, 1
-; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT: v_add_f32_e32 v1, v1, v3
-; GFX1064-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1064-NEXT: v_mov_b32_dpp v2, v1 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1064-NEXT: v_bfrev_b32_e32 v2, 1
-; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT: v_add_f32_e32 v1, v1, v3
-; GFX1064-NEXT: v_permlanex16_b32 v3, v1, -1, -1
-; GFX1064-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1064-NEXT: v_bfrev_b32_e32 v2, 1
-; GFX1064-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1064-NEXT: v_mov_b32_e32 v3, s2
-; GFX1064-NEXT: v_mov_b32_dpp v2, v3 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX1064-NEXT: v_add_f32_e32 v1, v1, v2
-; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT: v_readlane_b32 s2, v1, 15
-; GFX1064-NEXT: v_readlane_b32 s3, v1, 31
-; GFX1064-NEXT: v_writelane_b32 v3, s2, 16
-; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_readlane_b32 s2, v1, 63
-; GFX1064-NEXT: v_readlane_b32 s6, v1, 47
-; GFX1064-NEXT: v_writelane_b32 v3, s3, 32
-; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_writelane_b32 v3, s6, 48
-; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-NEXT: ; implicit-def: $vgpr0
-; GFX1064-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX1064-NEXT: s_cbranch_execz .LBB4_5
-; GFX1064-NEXT: ; %bb.4:
-; GFX1064-NEXT: v_mov_b32_e32 v0, 0
-; GFX1064-NEXT: v_mov_b32_e32 v4, s2
-; GFX1064-NEXT: s_waitcnt_vscnt null, 0x0
-; GFX1064-NEXT: ds_add_rtn_f32 v0, v0, v4
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: buffer_gl0_inv
-; GFX1064-NEXT: .LBB4_5:
-; GFX1064-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1064-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX1064-NEXT: v_readfirstlane_b32 s0, v0
-; GFX1064-NEXT: v_mov_b32_e32 v0, v3
-; GFX1064-NEXT: v_add_f32_e32 v0, s0, v0
-; GFX1064-NEXT: v_cndmask_b32_e64 v4, v0, s0, vcc
-; GFX1064-NEXT: .LBB4_6:
-; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1064-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1064-NEXT: s_mov_b32 s2, -1
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: buffer_store_dword v4, off, s[0:3], 0
-; GFX1064-NEXT: s_endpgm
-;
-; GFX900-LABEL: fadd_f32_varying:
-; GFX900: ; %bb.0: ; %entry
-; GFX900-NEXT: v_cvt_f32_i32_e32 v0, v0
-; GFX900-NEXT: s_bcnt1_i32_b64 s0, exec
-; GFX900-NEXT: s_cmp_lt_u32 s0, 6
-; GFX900-NEXT: s_cbranch_scc0 .LBB4_2
-; GFX900-NEXT: ; %bb.1:
-; GFX900-NEXT: v_mov_b32_e32 v4, 0
-; GFX900-NEXT: ds_add_rtn_f32 v4, v4, v0
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: s_cbranch_execz .LBB4_3
-; GFX900-NEXT: s_branch .LBB4_6
-; GFX900-NEXT: .LBB4_2:
-; GFX900-NEXT: ; implicit-def: $vgpr4
-; GFX900-NEXT: .LBB4_3:
-; GFX900-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
-; GFX900-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v4
-; GFX900-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX900-NEXT: v_bfrev_b32_e32 v1, 1
-; GFX900-NEXT: v_cndmask_b32_e64 v2, v1, v0, s[0:1]
-; GFX900-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_mov_b32_dpp v3, v2 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX900-NEXT: v_add_f32_e32 v2, v2, v3
-; GFX900-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_mov_b32_dpp v3, v2 row_shr:2 row_mask:0xf bank_mask:0xf
-; GFX900-NEXT: v_add_f32_e32 v2, v2, v3
-; GFX900-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_mov_b32_dpp v3, v2 row_shr:4 row_mask:0xf bank_mask:0xf
-; GFX900-NEXT: v_add_f32_e32 v2, v2, v3
-; GFX900-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_mov_b32_dpp v3, v2 row_shr:8 row_mask:0xf bank_mask:0xf
-; GFX900-NEXT: v_add_f32_e32 v2, v2, v3
-; GFX900-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_mov_b32_dpp v3, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX900-NEXT: v_add_f32_e32 v2, v2, v3
-; GFX900-NEXT: v_bfrev_b32_e32 v3, 1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_mov_b32_dpp v3, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX900-NEXT: v_add_f32_e32 v2, v2, v3
-; GFX900-NEXT: v_readlane_b32 s2, v2, 63
-; GFX900-NEXT: s_nop 0
-; GFX900-NEXT: v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX900-NEXT: s_mov_b64 exec, s[0:1]
-; GFX900-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
-; GFX900-NEXT: ; implicit-def: $vgpr0
-; GFX900-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX900-NEXT: s_cbranch_execz .LBB4_5
-; GFX900-NEXT: ; %bb.4:
-; GFX900-NEXT: v_mov_b32_e32 v0, 0
-; GFX900-NEXT: v_mov_b32_e32 v4, s2
-; GFX900-NEXT: ds_add_rtn_f32 v0, v0, v4
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: .LBB4_5:
-; GFX900-NEXT: s_or_b64 exec, exec, s[0:1]
-; GFX900-NEXT: v_readfirstlane_b32 s0, v0
-; GFX900-NEXT: v_mov_b32_e32 v0, v1
-; GFX900-NEXT: v_add_f32_e32 v0, s0, v0
-; GFX900-NEXT: v_mov_b32_e32 v4, s0
-; GFX900-NEXT: v_cndmask_b32_e32 v4, v0, v4, vcc
-; GFX900-NEXT: .LBB4_6:
-; GFX900-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX900-NEXT: s_mov_b32 s3, 0xf000
-; GFX900-NEXT: s_mov_b32 s2, -1
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: buffer_store_dword v4, off, s[0:3], 0
-; GFX900-NEXT: s_endpgm
-entry:
- %lane = call i32 @llvm.amdgcn.workitem.id.x()
- %conv = sitofp i32 %lane to float
- %rmw = atomicrmw fadd ptr addrspace(3) @local_var32, float %conv syncscope("agent") acq_rel, !amdgpu.atomic.lds.dpp !0
- store float %rmw, ptr addrspace(1) %out
- ret void
-}
-
-; Test 6: divergent i32 add on GLOBAL memory -- should NOT be affected by metadata
-define amdgpu_kernel void @add_i32_varying_global(ptr addrspace(1) %out, ptr addrspace(1) %inout) {
-; GFX1032-LABEL: add_i32_varying_global:
-; GFX1032: ; %bb.0: ; %entry
-; GFX1032-NEXT: s_or_saveexec_b32 s0, -1
-; GFX1032-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1032-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1032-NEXT: s_mov_b32 exec_lo, s0
-; GFX1032-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
-; GFX1032-NEXT: s_or_saveexec_b32 s4, -1
-; GFX1032-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1032-NEXT: v_mov_b32_e32 v3, 0
-; GFX1032-NEXT: v_readlane_b32 s6, v1, 31
-; GFX1032-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1032-NEXT: v_readlane_b32 s5, v1, 15
-; GFX1032-NEXT: s_mov_b32 exec_lo, s4
-; GFX1032-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1032-NEXT: s_or_saveexec_b32 s4, -1
-; GFX1032-NEXT: v_writelane_b32 v3, s5, 16
-; GFX1032-NEXT: s_mov_b32 exec_lo, s4
-; GFX1032-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-NEXT: s_mov_b32 s4, s6
-; GFX1032-NEXT: s_mov_b32 s6, -1
-; GFX1032-NEXT: ; implicit-def: $vgpr0
-; GFX1032-NEXT: s_and_saveexec_b32 s8, vcc_lo
-; GFX1032-NEXT: s_cbranch_execz .LBB5_2
-; GFX1032-NEXT: ; %bb.1:
-; GFX1032-NEXT: v_mov_b32_e32 v0, s4
-; GFX1032-NEXT: s_mov_b32 s7, 0x31016000
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: s_mov_b32 s4, s2
-; GFX1032-NEXT: s_mov_b32 s5, s3
-; GFX1032-NEXT: buffer_atomic_add v0, off, s[4:7], 0 glc
-; GFX1032-NEXT: s_waitcnt vmcnt(0)
-; GFX1032-NEXT: buffer_gl1_inv
-; GFX1032-NEXT: buffer_gl0_inv
-; GFX1032-NEXT: .LBB5_2:
-; GFX1032-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1032-NEXT: s_or_b32 exec_lo, exec_lo, s8
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: v_readfirstlane_b32 s2, v0
-; GFX1032-NEXT: v_mov_b32_e32 v0, v3
-; GFX1032-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1032-NEXT: v_add_nc_u32_e32 v0, s2, v0
-; GFX1032-NEXT: s_mov_b32 s2, s6
-; GFX1032-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1032-NEXT: s_endpgm
-;
-; GFX1064-LABEL: add_i32_varying_global:
-; GFX1064: ; %bb.0: ; %entry
-; GFX1064-NEXT: s_or_saveexec_b64 s[0:1], -1
-; GFX1064-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
-; GFX1064-NEXT: v_mov_b32_e32 v3, 0
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX1064-NEXT: v_permlanex16_b32 v2, v1, -1, -1
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
-; GFX1064-NEXT: v_readlane_b32 s2, v1, 31
-; GFX1064-NEXT: v_mov_b32_e32 v2, s2
-; GFX1064-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT: v_readlane_b32 s6, v1, 15
-; GFX1064-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
-; GFX1064-NEXT: s_mov_b64 exec, s[0:1]
-; GFX1064-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
-; GFX1064-NEXT: s_or_saveexec_b64 s[4:5], -1
-; GFX1064-NEXT: v_readlane_b32 s7, v1, 31
-; GFX1064-NEXT: v_writelane_b32 v3, s6, 16
-; GFX1064-NEXT: s_mov_b64 exec, s[4:5]
-; GFX1064-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
-; GFX1064-NEXT: s_or_saveexec_b64 s[4:5], -1
-; GFX1064-NEXT: v_readlane_b32 s8, v1, 47
-; GFX1064-NEXT: v_readlane_b32 s9, v1, 63
-; GFX1064-NEXT: v_writelane_b32 v3, s7, 32
-; GFX1064-NEXT: s_mov_b64 exec, s[4:5]
-; GFX1064-NEXT: v_mbcnt_hi_u32_b32 v0, exec_hi, v0
-; GFX1064-NEXT: s_or_saveexec_b64 s[6:7], -1
-; GFX1064-NEXT: s_mov_b32 s4, s9
-; GFX1064-NEXT: v_writelane_b32 v3, s8, 48
-; GFX1064-NEXT: s_mov_b64 exec, s[6:7]
-; GFX1064-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
-; GFX1064-NEXT: s_mov_b32 s6, -1
-; GFX1064-NEXT: ; implicit-def: $vgpr0
-; GFX1064-NEXT: s_and_saveexec_b64 s[8:9], vcc
-; GFX1064-NEXT: s_cbranch_execz .LBB5_2
-; GFX1064-NEXT: ; %bb.1:
-; GFX1064-NEXT: v_mov_b32_e32 v0, s4
-; GFX1064-NEXT: s_mov_b32 s7, 0x31016000
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: s_mov_b32 s4, s2
-; GFX1064-NEXT: s_mov_b32 s5, s3
-; GFX1064-NEXT: buffer_atomic_add v0, off, s[4:7], 0 glc
-; GFX1064-NEXT: s_waitcnt vmcnt(0)
-; GFX1064-NEXT: buffer_gl1_inv
-; GFX1064-NEXT: buffer_gl0_inv
-; GFX1064-NEXT: .LBB5_2:
-; GFX1064-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX1064-NEXT: s_or_b64 exec, exec, s[8:9]
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: v_readfirstlane_b32 s2, v0
-; GFX1064-NEXT: v_mov_b32_e32 v0, v3
-; GFX1064-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1064-NEXT: v_add_nc_u32_e32 v0, s2, v0
-; GFX1064-NEXT: s_mov_b32 s2, s6
-; GFX1064-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1064-NEXT: s_endpgm
-;
-; GFX900-LABEL: add_i32_varying_global:
-; GFX900: ; %bb.0: ; %entry
-; GFX900-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
-; GFX900-NEXT: s_or_saveexec_b64 s[4:5], -1
-; GFX900-NEXT: v_mov_b32_e32 v1, 0
-; GFX900-NEXT: s_mov_b64 exec, s[4:5]
-; GFX900-NEXT: v_mbcnt_lo_u32_b32 v3, exec_lo, 0
-; GFX900-NEXT: v_mbcnt_hi_u32_b32 v3, exec_hi, v3
-; GFX900-NEXT: s_or_saveexec_b64 s[4:5], -1
-; GFX900-NEXT: v_cndmask_b32_e64 v2, 0, v0, s[4:5]
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v2, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v2, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v2, v2, v2 row_bcast:15 row_mask:0xa bank_mask:0xf
-; GFX900-NEXT: s_nop 1
-; GFX900-NEXT: v_add_u32_dpp v2, v2, v2 row_bcast:31 row_mask:0xc bank_mask:0xf
-; GFX900-NEXT: v_readlane_b32 s6, v2, 63
-; GFX900-NEXT: s_nop 0
-; GFX900-NEXT: v_mov_b32_dpp v1, v2 wave_shr:1 row_mask:0xf bank_mask:0xf
-; GFX900-NEXT: s_mov_b64 exec, s[4:5]
-; GFX900-NEXT: v_cmp_eq_u32_e32 vcc, 0, v3
-; GFX900-NEXT: ; implicit-def: $vgpr0
-; GFX900-NEXT: s_and_saveexec_b64 s[4:5], vcc
-; GFX900-NEXT: s_cbranch_execz .LBB5_2
-; GFX900-NEXT: ; %bb.1:
-; GFX900-NEXT: s_mov_b32 s11, 0xf000
-; GFX900-NEXT: s_mov_b32 s10, -1
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: s_mov_b32 s8, s2
-; GFX900-NEXT: s_mov_b32 s9, s3
-; GFX900-NEXT: v_mov_b32_e32 v0, s6
-; GFX900-NEXT: buffer_atomic_add v0, off, s[8:11], 0 glc
-; GFX900-NEXT: s_waitcnt vmcnt(0)
-; GFX900-NEXT: buffer_wbinvl1_vol
-; GFX900-NEXT: .LBB5_2:
-; GFX900-NEXT: s_or_b64 exec, exec, s[4:5]
-; GFX900-NEXT: v_readfirstlane_b32 s4, v0
-; GFX900-NEXT: v_mov_b32_e32 v0, v1
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: s_mov_b32 s3, 0xf000
-; GFX900-NEXT: s_mov_b32 s2, -1
-; GFX900-NEXT: v_add_u32_e32 v0, s4, v0
-; GFX900-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX900-NEXT: s_endpgm
-entry:
- %lane = call i32 @llvm.amdgcn.workitem.id.x()
- %old = atomicrmw add ptr addrspace(1) %inout, i32 %lane acq_rel, !amdgpu.atomic.lds.dpp !0
- store i32 %old, ptr addrspace(1) %out
- ret void
-}
-
-; Test 7: !amdgpu.atomic.lds.dpp !{!"none"} disables DPP -- plain atomic expected
-define amdgpu_kernel void @add_i32_varying_no_lds_dpp(ptr addrspace(1) %out) {
-; GFX1032-LABEL: add_i32_varying_no_lds_dpp:
-; GFX1032: ; %bb.0: ; %entry
-; GFX1032-NEXT: v_mov_b32_e32 v1, 0
-; GFX1032-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1032-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1032-NEXT: s_mov_b32 s2, -1
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: ds_add_rtn_u32 v0, v1, v0
-; GFX1032-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT: buffer_gl0_inv
-; GFX1032-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1032-NEXT: s_endpgm
-;
-; GFX1064-LABEL: add_i32_varying_no_lds_dpp:
-; GFX1064: ; %bb.0: ; %entry
-; GFX1064-NEXT: v_mov_b32_e32 v1, 0
-; GFX1064-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX1064-NEXT: s_mov_b32 s3, 0x31016000
-; GFX1064-NEXT: s_mov_b32 s2, -1
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: ds_add_rtn_u32 v0, v1, v0
-; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT: buffer_gl0_inv
-; GFX1064-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX1064-NEXT: s_endpgm
-;
-; GFX900-LABEL: add_i32_varying_no_lds_dpp:
-; GFX900: ; %bb.0: ; %entry
-; GFX900-NEXT: v_mov_b32_e32 v1, 0
-; GFX900-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: ds_add_rtn_u32 v0, v1, v0
-; GFX900-NEXT: s_waitcnt lgkmcnt(0)
-; GFX900-NEXT: s_mov_b32 s3, 0xf000
-; GFX900-NEXT: s_mov_b32 s2, -1
-; GFX900-NEXT: buffer_store_dword v0, off, s[0:3], 0
-; GFX900-NEXT: s_endpgm
-entry:
- %lane = call i32 @llvm.amdgcn.workitem.id.x()
- %old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel, !amdgpu.atomic.lds.dpp !1
- store i32 %old, ptr addrspace(1) %out
- ret void
-}
-
-!0 = !{!"dynamic"}
-!1 = !{!"none"}
diff --git a/llvm/test/Verifier/amdgpu-expected-active-lanes.ll b/llvm/test/Verifier/amdgpu-expected-active-lanes.ll
new file mode 100644
index 0000000000000..573ab0bc7eac0
--- /dev/null
+++ b/llvm/test/Verifier/amdgpu-expected-active-lanes.ll
@@ -0,0 +1,30 @@
+; RUN: not llvm-as -disable-output %s 2>&1 | FileCheck %s
+
+; CHECK-DAG: !amdgpu.expected.active.lanes must have exactly one operand
+define i32 @no_operands(ptr %p, i32 %v) {
+ %old = atomicrmw add ptr %p, i32 %v seq_cst, !amdgpu.expected.active.lanes !0
+ ret i32 %old
+}
+
+; CHECK-DAG: !amdgpu.expected.active.lanes must have exactly one operand
+define i32 @too_many_operands(ptr %p, i32 %v) {
+ %old = atomicrmw add ptr %p, i32 %v seq_cst, !amdgpu.expected.active.lanes !1
+ ret i32 %old
+}
+
+; CHECK-DAG: !amdgpu.expected.active.lanes operand must be an i32 constant
+define i32 @operand_not_a_constant(ptr %p, i32 %v) {
+ %old = atomicrmw add ptr %p, i32 %v seq_cst, !amdgpu.expected.active.lanes !2
+ ret i32 %old
+}
+
+; CHECK-DAG: !amdgpu.expected.active.lanes operand must be an i32 constant
+define i32 @operand_wrong_width(ptr %p, i32 %v) {
+ %old = atomicrmw add ptr %p, i32 %v seq_cst, !amdgpu.expected.active.lanes !3
+ ret i32 %old
+}
+
+!0 = !{}
+!1 = !{i32 4, i32 5}
+!2 = !{!"not a constant"}
+!3 = !{i64 4}
>From ec992dea6bf6e73808841eb5ddbd23ef7d1edb8b Mon Sep 17 00:00:00 2001
From: YaFan <YaFan.Tao at amd.com>
Date: Fri, 5 Jun 2026 14:58:00 +0800
Subject: [PATCH 14/17] Add comments that this threshold also works for gfx11
---
llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index 0d5dda8ba5c66..ad7e7ad2de806 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -657,9 +657,9 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
// are expected to be active; for a divergent LDS atomic, when that count is
// small (<= 5), skip DPP and let each lane issue its own atomic.
//
- // FIXME: The threshold was tuned empirically on gfx12. The DPP scan overhead
- // differs across subtargets, so the break-even point may differ too; this may
- // need to become subtarget-dependent.
+ // FIXME: The threshold was tuned empirically on gfx11 and gfx12. The DPP scan
+ // overhead differs across subtargets, so the break-even point may differ too;
+ // this may need to become subtarget-dependent.
if (IsLDS && ValDivergent && ScanImpl == ScanOptions::DPP) {
if (MDNode *MD = I.getMetadata("amdgpu.expected.active.lanes")) {
auto *CI = mdconst::extract<ConstantInt>(MD->getOperand(0));
>From 6700cbe90751da84f618da9db5e4a3fd6f82bbd5 Mon Sep 17 00:00:00 2001
From: YaFan <YaFan.Tao at amd.com>
Date: Mon, 22 Jun 2026 11:14:18 +0800
Subject: [PATCH 15/17] Change comments and add tests for gfx11 and gfx12
---
llvm/docs/AMDGPUUsage.rst | 9 +-
.../Target/AMDGPU/AMDGPUAtomicOptimizer.cpp | 8 +-
...mizations_dpp_lds_expected_active_lanes.ll | 777 +++++++++++++++++-
3 files changed, 778 insertions(+), 16 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 0f0bc9c4fdeaa..5f8bdbfdd988c 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -2240,11 +2240,10 @@ and
-------------------------------------------------
A profiling-derived hint describing how many lanes of the wavefront are
-expected to be active when this :ref:`atomicrmw <i_atomicrmw>` instruction
-executes. The metadata has a single ``i32`` constant operand giving the
-expected number of active lanes as an absolute count (the same value is used
-for both wave32 and wave64 targets). This structure is enforced by the
-verifier.
+expected to be active. The metadata has a single ``i32`` constant operand
+giving the expected number of active lanes as an absolute count (the same
+value is used for both wave32 and wave64 targets). This structure is
+enforced by the verifier.
The AMDGPU atomic optimizer uses the hint as follows. When the
``-amdgpu-atomic-optimizer-strategy=DPP`` strategy is active, the value
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index ad7e7ad2de806..cbfd78bcadbf1 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -652,11 +652,9 @@ void AMDGPUAtomicOptimizerImpl::optimizeAtomic(Instruction &I,
unsigned ValIdx,
bool ValDivergent,
bool IsLDS) const {
- // The DPP scan has a fixed overhead, so it only pays off with enough active
- // lanes. The !amdgpu.expected.active.lanes metadata hints at how many lanes
- // are expected to be active; for a divergent LDS atomic, when that count is
- // small (<= 5), skip DPP and let each lane issue its own atomic.
- //
+ // Don't generate a DPP scan if !amdgpu.expected.active.lane hint indicates
+ // insufficient lanes to offset fixed overhead.
+
// FIXME: The threshold was tuned empirically on gfx11 and gfx12. The DPP scan
// overhead differs across subtargets, so the break-even point may differ too;
// this may need to become subtarget-dependent.
diff --git a/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_expected_active_lanes.ll b/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_expected_active_lanes.ll
index db2e2d515073f..2d9c55b827fc7 100644
--- a/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_expected_active_lanes.ll
+++ b/llvm/test/CodeGen/AMDGPU/atomic_optimizations_dpp_lds_expected_active_lanes.ll
@@ -2,12 +2,10 @@
;
; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32 -amdgpu-atomic-optimizer-strategy=DPP < %s | FileCheck -enable-var-scope -check-prefixes=GFX1032 %s
; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize64 -amdgpu-atomic-optimizer-strategy=DPP < %s | FileCheck -enable-var-scope -check-prefixes=GFX1064 %s
-;
-; Test the !amdgpu.expected.active.lanes instruction metadata, a
-; profiling-derived hint giving the number of wavefront lanes expected to be
-; active when the atomic executes. When the expected active lane count is small
-; (<= 5), the DPP scan overhead is not worthwhile, so the DPP optimization is
-; skipped for divergent LDS atomics and each lane issues its own atomic.
+; RUN: llc -mtriple=amdgcn -mcpu=gfx1100 -mattr=+wavefrontsize32 -amdgpu-atomic-optimizer-strategy=DPP < %s | FileCheck -enable-var-scope -check-prefixes=GFX1132 %s
+; RUN: llc -mtriple=amdgcn -mcpu=gfx1100 -mattr=+wavefrontsize64 -amdgpu-atomic-optimizer-strategy=DPP < %s | FileCheck -enable-var-scope -check-prefixes=GFX1164 %s
+; RUN: llc -mtriple=amdgcn -mcpu=gfx1200 -mattr=+wavefrontsize32 -amdgpu-atomic-optimizer-strategy=DPP < %s | FileCheck -enable-var-scope -check-prefixes=GFX1232 %s
+; RUN: llc -mtriple=amdgcn -mcpu=gfx1200 -mattr=+wavefrontsize64 -amdgpu-atomic-optimizer-strategy=DPP < %s | FileCheck -enable-var-scope -check-prefixes=GFX1264 %s
declare i32 @llvm.amdgcn.workitem.id.x()
@@ -37,6 +35,52 @@ define amdgpu_kernel void @add_i32_low_active_lanes(ptr addrspace(1) %out) {
; GFX1064-NEXT: buffer_gl0_inv
; GFX1064-NEXT: global_store_dword v1, v0, s[0:1]
; GFX1064-NEXT: s_endpgm
+;
+; GFX1132-LABEL: add_i32_low_active_lanes:
+; GFX1132: ; %bb.0: ; %entry
+; GFX1132-NEXT: v_dual_mov_b32 v1, 0 :: v_dual_and_b32 v0, 0x3ff, v0
+; GFX1132-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1132-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1132-NEXT: ds_add_rtn_u32 v0, v1, v0
+; GFX1132-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1132-NEXT: buffer_gl0_inv
+; GFX1132-NEXT: global_store_b32 v1, v0, s[0:1]
+; GFX1132-NEXT: s_endpgm
+;
+; GFX1164-LABEL: add_i32_low_active_lanes:
+; GFX1164: ; %bb.0: ; %entry
+; GFX1164-NEXT: v_and_b32_e32 v0, 0x3ff, v0
+; GFX1164-NEXT: v_mov_b32_e32 v1, 0
+; GFX1164-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1164-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1164-NEXT: ds_add_rtn_u32 v0, v1, v0
+; GFX1164-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1164-NEXT: buffer_gl0_inv
+; GFX1164-NEXT: global_store_b32 v1, v0, s[0:1]
+; GFX1164-NEXT: s_endpgm
+;
+; GFX1232-LABEL: add_i32_low_active_lanes:
+; GFX1232: ; %bb.0: ; %entry
+; GFX1232-NEXT: v_dual_mov_b32 v1, 0 :: v_dual_and_b32 v0, 0x3ff, v0
+; GFX1232-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1232-NEXT: ds_add_rtn_u32 v0, v1, v0
+; GFX1232-NEXT: s_wait_dscnt 0x0
+; GFX1232-NEXT: global_inv scope:SCOPE_SE
+; GFX1232-NEXT: s_wait_kmcnt 0x0
+; GFX1232-NEXT: global_store_b32 v1, v0, s[0:1]
+; GFX1232-NEXT: s_endpgm
+;
+; GFX1264-LABEL: add_i32_low_active_lanes:
+; GFX1264: ; %bb.0: ; %entry
+; GFX1264-NEXT: v_and_b32_e32 v0, 0x3ff, v0
+; GFX1264-NEXT: v_mov_b32_e32 v1, 0
+; GFX1264-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1264-NEXT: ds_add_rtn_u32 v0, v1, v0
+; GFX1264-NEXT: s_wait_dscnt 0x0
+; GFX1264-NEXT: global_inv scope:SCOPE_SE
+; GFX1264-NEXT: s_wait_kmcnt 0x0
+; GFX1264-NEXT: global_store_b32 v1, v0, s[0:1]
+; GFX1264-NEXT: s_endpgm
entry:
%lane = call i32 @llvm.amdgcn.workitem.id.x()
%old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel, !amdgpu.expected.active.lanes !0
@@ -67,6 +111,52 @@ define amdgpu_kernel void @add_i32_boundary_active_lanes(ptr addrspace(1) %out)
; GFX1064-NEXT: buffer_gl0_inv
; GFX1064-NEXT: global_store_dword v1, v0, s[0:1]
; GFX1064-NEXT: s_endpgm
+;
+; GFX1132-LABEL: add_i32_boundary_active_lanes:
+; GFX1132: ; %bb.0: ; %entry
+; GFX1132-NEXT: v_dual_mov_b32 v1, 0 :: v_dual_and_b32 v0, 0x3ff, v0
+; GFX1132-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1132-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1132-NEXT: ds_add_rtn_u32 v0, v1, v0
+; GFX1132-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1132-NEXT: buffer_gl0_inv
+; GFX1132-NEXT: global_store_b32 v1, v0, s[0:1]
+; GFX1132-NEXT: s_endpgm
+;
+; GFX1164-LABEL: add_i32_boundary_active_lanes:
+; GFX1164: ; %bb.0: ; %entry
+; GFX1164-NEXT: v_and_b32_e32 v0, 0x3ff, v0
+; GFX1164-NEXT: v_mov_b32_e32 v1, 0
+; GFX1164-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1164-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1164-NEXT: ds_add_rtn_u32 v0, v1, v0
+; GFX1164-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1164-NEXT: buffer_gl0_inv
+; GFX1164-NEXT: global_store_b32 v1, v0, s[0:1]
+; GFX1164-NEXT: s_endpgm
+;
+; GFX1232-LABEL: add_i32_boundary_active_lanes:
+; GFX1232: ; %bb.0: ; %entry
+; GFX1232-NEXT: v_dual_mov_b32 v1, 0 :: v_dual_and_b32 v0, 0x3ff, v0
+; GFX1232-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1232-NEXT: ds_add_rtn_u32 v0, v1, v0
+; GFX1232-NEXT: s_wait_dscnt 0x0
+; GFX1232-NEXT: global_inv scope:SCOPE_SE
+; GFX1232-NEXT: s_wait_kmcnt 0x0
+; GFX1232-NEXT: global_store_b32 v1, v0, s[0:1]
+; GFX1232-NEXT: s_endpgm
+;
+; GFX1264-LABEL: add_i32_boundary_active_lanes:
+; GFX1264: ; %bb.0: ; %entry
+; GFX1264-NEXT: v_and_b32_e32 v0, 0x3ff, v0
+; GFX1264-NEXT: v_mov_b32_e32 v1, 0
+; GFX1264-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1264-NEXT: ds_add_rtn_u32 v0, v1, v0
+; GFX1264-NEXT: s_wait_dscnt 0x0
+; GFX1264-NEXT: global_inv scope:SCOPE_SE
+; GFX1264-NEXT: s_wait_kmcnt 0x0
+; GFX1264-NEXT: global_store_b32 v1, v0, s[0:1]
+; GFX1264-NEXT: s_endpgm
entry:
%lane = call i32 @llvm.amdgcn.workitem.id.x()
%old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel, !amdgpu.expected.active.lanes !1
@@ -165,6 +255,225 @@ define amdgpu_kernel void @add_i32_high_active_lanes(ptr addrspace(1) %out) {
; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
; GFX1064-NEXT: global_store_dword v0, v4, s[0:1]
; GFX1064-NEXT: s_endpgm
+;
+; GFX1132-LABEL: add_i32_high_active_lanes:
+; GFX1132: ; %bb.0: ; %entry
+; GFX1132-NEXT: v_and_b32_e32 v0, 0x3ff, v0
+; GFX1132-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_1) | instid1(SALU_CYCLE_1)
+; GFX1132-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1132-NEXT: v_mov_b32_e32 v3, 0
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1132-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1132-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1132-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1132-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1132-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1132-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_3) | instid1(SALU_CYCLE_1)
+; GFX1132-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1132-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1132-NEXT: v_readlane_b32 s1, v1, 15
+; GFX1132-NEXT: s_mov_b32 exec_lo, s0
+; GFX1132-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1132-NEXT: v_mov_b32_e32 v0, 0
+; GFX1132-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1132-NEXT: v_writelane_b32 v3, s1, 16
+; GFX1132-NEXT: s_mov_b32 exec_lo, s0
+; GFX1132-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
+; GFX1132-NEXT: ; implicit-def: $vgpr4
+; GFX1132-NEXT: s_and_saveexec_b32 s0, vcc_lo
+; GFX1132-NEXT: s_cbranch_execz .LBB2_2
+; GFX1132-NEXT: ; %bb.1:
+; GFX1132-NEXT: v_mov_b32_e32 v4, s2
+; GFX1132-NEXT: ds_add_rtn_u32 v4, v0, v4
+; GFX1132-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1132-NEXT: buffer_gl0_inv
+; GFX1132-NEXT: .LBB2_2:
+; GFX1132-NEXT: s_or_b32 exec_lo, exec_lo, s0
+; GFX1132-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1132-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1132-NEXT: v_mov_b32_e32 v4, v3
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1132-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1132-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1132-NEXT: global_store_b32 v0, v4, s[0:1]
+; GFX1132-NEXT: s_endpgm
+;
+; GFX1164-LABEL: add_i32_high_active_lanes:
+; GFX1164: ; %bb.0: ; %entry
+; GFX1164-NEXT: v_and_b32_e32 v0, 0x3ff, v0
+; GFX1164-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1164-NEXT: v_mov_b32_e32 v3, 0
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1164-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1164-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1164-NEXT: v_mov_b32_e32 v2, s2
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1164-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1164-NEXT: v_readlane_b32 s2, v1, 15
+; GFX1164-NEXT: v_readlane_b32 s3, v1, 31
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT: v_writelane_b32 v3, s2, 16
+; GFX1164-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1164-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1164-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1164-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX1164-NEXT: v_readlane_b32 s2, v1, 63
+; GFX1164-NEXT: v_readlane_b32 s6, v1, 47
+; GFX1164-NEXT: v_writelane_b32 v3, s3, 32
+; GFX1164-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1164-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1164-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
+; GFX1164-NEXT: v_mov_b32_e32 v0, 0
+; GFX1164-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1164-NEXT: v_writelane_b32 v3, s6, 48
+; GFX1164-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1164-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX1164-NEXT: ; implicit-def: $vgpr4
+; GFX1164-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1164-NEXT: s_cbranch_execz .LBB2_2
+; GFX1164-NEXT: ; %bb.1:
+; GFX1164-NEXT: v_mov_b32_e32 v4, s2
+; GFX1164-NEXT: ds_add_rtn_u32 v4, v0, v4
+; GFX1164-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1164-NEXT: buffer_gl0_inv
+; GFX1164-NEXT: .LBB2_2:
+; GFX1164-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1164-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1164-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1164-NEXT: v_mov_b32_e32 v4, v3
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1164-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1164-NEXT: global_store_b32 v0, v4, s[0:1]
+; GFX1164-NEXT: s_endpgm
+;
+; GFX1232-LABEL: add_i32_high_active_lanes:
+; GFX1232: ; %bb.0: ; %entry
+; GFX1232-NEXT: v_and_b32_e32 v0, 0x3ff, v0
+; GFX1232-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_1) | instid1(SALU_CYCLE_1)
+; GFX1232-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1232-NEXT: v_mov_b32_e32 v3, 0
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1232-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1232-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1232-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1232-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1232-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1232-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_3) | instid1(SALU_CYCLE_1)
+; GFX1232-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1232-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1232-NEXT: v_readlane_b32 s1, v1, 15
+; GFX1232-NEXT: s_mov_b32 exec_lo, s0
+; GFX1232-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1232-NEXT: v_mov_b32_e32 v0, 0
+; GFX1232-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1232-NEXT: v_writelane_b32 v3, s1, 16
+; GFX1232-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX1232-NEXT: s_mov_b32 exec_lo, s0
+; GFX1232-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
+; GFX1232-NEXT: ; implicit-def: $vgpr4
+; GFX1232-NEXT: s_and_saveexec_b32 s0, vcc_lo
+; GFX1232-NEXT: s_cbranch_execz .LBB2_2
+; GFX1232-NEXT: ; %bb.1:
+; GFX1232-NEXT: v_mov_b32_e32 v4, s2
+; GFX1232-NEXT: ds_add_rtn_u32 v4, v0, v4
+; GFX1232-NEXT: s_wait_dscnt 0x0
+; GFX1232-NEXT: global_inv scope:SCOPE_SE
+; GFX1232-NEXT: .LBB2_2:
+; GFX1232-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX1232-NEXT: s_or_b32 exec_lo, exec_lo, s0
+; GFX1232-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1232-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1232-NEXT: v_mov_b32_e32 v4, v3
+; GFX1232-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1232-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1232-NEXT: s_wait_kmcnt 0x0
+; GFX1232-NEXT: global_store_b32 v0, v4, s[0:1]
+; GFX1232-NEXT: s_endpgm
+;
+; GFX1264-LABEL: add_i32_high_active_lanes:
+; GFX1264: ; %bb.0: ; %entry
+; GFX1264-NEXT: v_and_b32_e32 v0, 0x3ff, v0
+; GFX1264-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1) | instid1(SALU_CYCLE_1)
+; GFX1264-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1264-NEXT: v_mov_b32_e32 v3, 0
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1264-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1264-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1264-NEXT: v_mov_b32_e32 v2, s2
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1264-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1264-NEXT: v_readlane_b32 s2, v1, 15
+; GFX1264-NEXT: v_readlane_b32 s3, v1, 31
+; GFX1264-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
+; GFX1264-NEXT: v_writelane_b32 v3, s2, 16
+; GFX1264-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1264-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1264-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1264-NEXT: v_readlane_b32 s2, v1, 63
+; GFX1264-NEXT: v_readlane_b32 s6, v1, 47
+; GFX1264-NEXT: v_writelane_b32 v3, s3, 32
+; GFX1264-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX1264-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1264-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1264-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
+; GFX1264-NEXT: v_mov_b32_e32 v0, 0
+; GFX1264-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1264-NEXT: v_writelane_b32 v3, s6, 48
+; GFX1264-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX1264-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1264-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX1264-NEXT: ; implicit-def: $vgpr4
+; GFX1264-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1264-NEXT: s_cbranch_execz .LBB2_2
+; GFX1264-NEXT: ; %bb.1:
+; GFX1264-NEXT: v_mov_b32_e32 v4, s2
+; GFX1264-NEXT: ds_add_rtn_u32 v4, v0, v4
+; GFX1264-NEXT: s_wait_dscnt 0x0
+; GFX1264-NEXT: global_inv scope:SCOPE_SE
+; GFX1264-NEXT: .LBB2_2:
+; GFX1264-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX1264-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1264-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1264-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1264-NEXT: v_mov_b32_e32 v4, v3
+; GFX1264-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1264-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1264-NEXT: s_wait_kmcnt 0x0
+; GFX1264-NEXT: global_store_b32 v0, v4, s[0:1]
+; GFX1264-NEXT: s_endpgm
entry:
%lane = call i32 @llvm.amdgcn.workitem.id.x()
%old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel, !amdgpu.expected.active.lanes !2
@@ -263,6 +572,225 @@ define amdgpu_kernel void @add_i32_no_metadata(ptr addrspace(1) %out) {
; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
; GFX1064-NEXT: global_store_dword v0, v4, s[0:1]
; GFX1064-NEXT: s_endpgm
+;
+; GFX1132-LABEL: add_i32_no_metadata:
+; GFX1132: ; %bb.0: ; %entry
+; GFX1132-NEXT: v_and_b32_e32 v0, 0x3ff, v0
+; GFX1132-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_1) | instid1(SALU_CYCLE_1)
+; GFX1132-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1132-NEXT: v_mov_b32_e32 v3, 0
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1132-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1132-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1132-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1132-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1132-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1132-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_3) | instid1(SALU_CYCLE_1)
+; GFX1132-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1132-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1132-NEXT: v_readlane_b32 s1, v1, 15
+; GFX1132-NEXT: s_mov_b32 exec_lo, s0
+; GFX1132-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1132-NEXT: v_mov_b32_e32 v0, 0
+; GFX1132-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1132-NEXT: v_writelane_b32 v3, s1, 16
+; GFX1132-NEXT: s_mov_b32 exec_lo, s0
+; GFX1132-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
+; GFX1132-NEXT: ; implicit-def: $vgpr4
+; GFX1132-NEXT: s_and_saveexec_b32 s0, vcc_lo
+; GFX1132-NEXT: s_cbranch_execz .LBB3_2
+; GFX1132-NEXT: ; %bb.1:
+; GFX1132-NEXT: v_mov_b32_e32 v4, s2
+; GFX1132-NEXT: ds_add_rtn_u32 v4, v0, v4
+; GFX1132-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1132-NEXT: buffer_gl0_inv
+; GFX1132-NEXT: .LBB3_2:
+; GFX1132-NEXT: s_or_b32 exec_lo, exec_lo, s0
+; GFX1132-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1132-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1132-NEXT: v_mov_b32_e32 v4, v3
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1132-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1132-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1132-NEXT: global_store_b32 v0, v4, s[0:1]
+; GFX1132-NEXT: s_endpgm
+;
+; GFX1164-LABEL: add_i32_no_metadata:
+; GFX1164: ; %bb.0: ; %entry
+; GFX1164-NEXT: v_and_b32_e32 v0, 0x3ff, v0
+; GFX1164-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1164-NEXT: v_mov_b32_e32 v3, 0
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1164-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1164-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1164-NEXT: v_mov_b32_e32 v2, s2
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1164-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1164-NEXT: v_readlane_b32 s2, v1, 15
+; GFX1164-NEXT: v_readlane_b32 s3, v1, 31
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT: v_writelane_b32 v3, s2, 16
+; GFX1164-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1164-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1164-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1164-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX1164-NEXT: v_readlane_b32 s2, v1, 63
+; GFX1164-NEXT: v_readlane_b32 s6, v1, 47
+; GFX1164-NEXT: v_writelane_b32 v3, s3, 32
+; GFX1164-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1164-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1164-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
+; GFX1164-NEXT: v_mov_b32_e32 v0, 0
+; GFX1164-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1164-NEXT: v_writelane_b32 v3, s6, 48
+; GFX1164-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1164-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX1164-NEXT: ; implicit-def: $vgpr4
+; GFX1164-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1164-NEXT: s_cbranch_execz .LBB3_2
+; GFX1164-NEXT: ; %bb.1:
+; GFX1164-NEXT: v_mov_b32_e32 v4, s2
+; GFX1164-NEXT: ds_add_rtn_u32 v4, v0, v4
+; GFX1164-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1164-NEXT: buffer_gl0_inv
+; GFX1164-NEXT: .LBB3_2:
+; GFX1164-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1164-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1164-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1164-NEXT: v_mov_b32_e32 v4, v3
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1164-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1164-NEXT: global_store_b32 v0, v4, s[0:1]
+; GFX1164-NEXT: s_endpgm
+;
+; GFX1232-LABEL: add_i32_no_metadata:
+; GFX1232: ; %bb.0: ; %entry
+; GFX1232-NEXT: v_and_b32_e32 v0, 0x3ff, v0
+; GFX1232-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_1) | instid1(SALU_CYCLE_1)
+; GFX1232-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1232-NEXT: v_mov_b32_e32 v3, 0
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1232-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1232-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1232-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1232-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1232-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1232-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_3) | instid1(SALU_CYCLE_1)
+; GFX1232-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1232-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1232-NEXT: v_readlane_b32 s1, v1, 15
+; GFX1232-NEXT: s_mov_b32 exec_lo, s0
+; GFX1232-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1232-NEXT: v_mov_b32_e32 v0, 0
+; GFX1232-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1232-NEXT: v_writelane_b32 v3, s1, 16
+; GFX1232-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX1232-NEXT: s_mov_b32 exec_lo, s0
+; GFX1232-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
+; GFX1232-NEXT: ; implicit-def: $vgpr4
+; GFX1232-NEXT: s_and_saveexec_b32 s0, vcc_lo
+; GFX1232-NEXT: s_cbranch_execz .LBB3_2
+; GFX1232-NEXT: ; %bb.1:
+; GFX1232-NEXT: v_mov_b32_e32 v4, s2
+; GFX1232-NEXT: ds_add_rtn_u32 v4, v0, v4
+; GFX1232-NEXT: s_wait_dscnt 0x0
+; GFX1232-NEXT: global_inv scope:SCOPE_SE
+; GFX1232-NEXT: .LBB3_2:
+; GFX1232-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX1232-NEXT: s_or_b32 exec_lo, exec_lo, s0
+; GFX1232-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1232-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1232-NEXT: v_mov_b32_e32 v4, v3
+; GFX1232-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1232-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1232-NEXT: s_wait_kmcnt 0x0
+; GFX1232-NEXT: global_store_b32 v0, v4, s[0:1]
+; GFX1232-NEXT: s_endpgm
+;
+; GFX1264-LABEL: add_i32_no_metadata:
+; GFX1264: ; %bb.0: ; %entry
+; GFX1264-NEXT: v_and_b32_e32 v0, 0x3ff, v0
+; GFX1264-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1) | instid1(SALU_CYCLE_1)
+; GFX1264-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1264-NEXT: v_mov_b32_e32 v3, 0
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1264-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1264-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1264-NEXT: v_mov_b32_e32 v2, s2
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1264-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1264-NEXT: v_readlane_b32 s2, v1, 15
+; GFX1264-NEXT: v_readlane_b32 s3, v1, 31
+; GFX1264-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
+; GFX1264-NEXT: v_writelane_b32 v3, s2, 16
+; GFX1264-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1264-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1264-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1264-NEXT: v_readlane_b32 s2, v1, 63
+; GFX1264-NEXT: v_readlane_b32 s6, v1, 47
+; GFX1264-NEXT: v_writelane_b32 v3, s3, 32
+; GFX1264-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX1264-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1264-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1264-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
+; GFX1264-NEXT: v_mov_b32_e32 v0, 0
+; GFX1264-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1264-NEXT: v_writelane_b32 v3, s6, 48
+; GFX1264-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX1264-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1264-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX1264-NEXT: ; implicit-def: $vgpr4
+; GFX1264-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX1264-NEXT: s_cbranch_execz .LBB3_2
+; GFX1264-NEXT: ; %bb.1:
+; GFX1264-NEXT: v_mov_b32_e32 v4, s2
+; GFX1264-NEXT: ds_add_rtn_u32 v4, v0, v4
+; GFX1264-NEXT: s_wait_dscnt 0x0
+; GFX1264-NEXT: global_inv scope:SCOPE_SE
+; GFX1264-NEXT: .LBB3_2:
+; GFX1264-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX1264-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1264-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1264-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1264-NEXT: v_mov_b32_e32 v4, v3
+; GFX1264-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1264-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1264-NEXT: s_wait_kmcnt 0x0
+; GFX1264-NEXT: global_store_b32 v0, v4, s[0:1]
+; GFX1264-NEXT: s_endpgm
entry:
%lane = call i32 @llvm.amdgcn.workitem.id.x()
%old = atomicrmw add ptr addrspace(3) @local_var32, i32 %lane acq_rel
@@ -370,6 +898,243 @@ define amdgpu_kernel void @add_i32_global_ignores_metadata(ptr addrspace(1) %ino
; GFX1064-NEXT: s_waitcnt lgkmcnt(0)
; GFX1064-NEXT: global_store_dword v0, v4, s[0:1]
; GFX1064-NEXT: s_endpgm
+;
+; GFX1132-LABEL: add_i32_global_ignores_metadata:
+; GFX1132: ; %bb.0: ; %entry
+; GFX1132-NEXT: v_and_b32_e32 v0, 0x3ff, v0
+; GFX1132-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_1) | instid1(SALU_CYCLE_1)
+; GFX1132-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1132-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1132-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1132-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1132-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_3) | instid1(VALU_DEP_1)
+; GFX1132-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1132-NEXT: s_mov_b32 exec_lo, s0
+; GFX1132-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1132-NEXT: s_or_saveexec_b32 s2, -1
+; GFX1132-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1132-NEXT: v_mov_b32_e32 v3, 0
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1132-NEXT: v_readlane_b32 s4, v1, 31
+; GFX1132-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1132-NEXT: v_readlane_b32 s3, v1, 15
+; GFX1132-NEXT: s_mov_b32 exec_lo, s2
+; GFX1132-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1132-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1132-NEXT: v_mov_b32_e32 v0, 0
+; GFX1132-NEXT: s_or_saveexec_b32 s2, -1
+; GFX1132-NEXT: v_writelane_b32 v3, s3, 16
+; GFX1132-NEXT: s_mov_b32 exec_lo, s2
+; GFX1132-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
+; GFX1132-NEXT: ; implicit-def: $vgpr4
+; GFX1132-NEXT: s_and_saveexec_b32 s2, vcc_lo
+; GFX1132-NEXT: s_cbranch_execz .LBB4_2
+; GFX1132-NEXT: ; %bb.1:
+; GFX1132-NEXT: v_mov_b32_e32 v4, s4
+; GFX1132-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1132-NEXT: global_atomic_add_u32 v4, v0, v4, s[0:1] glc
+; GFX1132-NEXT: s_waitcnt vmcnt(0)
+; GFX1132-NEXT: buffer_gl1_inv
+; GFX1132-NEXT: buffer_gl0_inv
+; GFX1132-NEXT: .LBB4_2:
+; GFX1132-NEXT: s_or_b32 exec_lo, exec_lo, s2
+; GFX1132-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1132-NEXT: v_mov_b32_e32 v4, v3
+; GFX1132-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1132-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1132-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1132-NEXT: global_store_b32 v0, v4, s[0:1]
+; GFX1132-NEXT: s_endpgm
+;
+; GFX1164-LABEL: add_i32_global_ignores_metadata:
+; GFX1164: ; %bb.0: ; %entry
+; GFX1164-NEXT: v_and_b32_e32 v0, 0x3ff, v0
+; GFX1164-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1) | instid1(SALU_CYCLE_1)
+; GFX1164-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1164-NEXT: v_mov_b32_e32 v3, 0
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1164-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1164-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1164-NEXT: v_mov_b32_e32 v2, s2
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1164-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1164-NEXT: v_readlane_b32 s6, v1, 15
+; GFX1164-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1164-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1164-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1164-NEXT: s_or_saveexec_b64 s[2:3], -1
+; GFX1164-NEXT: v_readlane_b32 s4, v1, 31
+; GFX1164-NEXT: v_writelane_b32 v3, s6, 16
+; GFX1164-NEXT: s_mov_b64 exec, s[2:3]
+; GFX1164-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1164-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1164-NEXT: s_or_saveexec_b64 s[2:3], -1
+; GFX1164-NEXT: v_readlane_b32 s6, v1, 63
+; GFX1164-NEXT: v_readlane_b32 s5, v1, 47
+; GFX1164-NEXT: v_writelane_b32 v3, s4, 32
+; GFX1164-NEXT: s_mov_b64 exec, s[2:3]
+; GFX1164-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1164-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
+; GFX1164-NEXT: v_mov_b32_e32 v0, 0
+; GFX1164-NEXT: s_or_saveexec_b64 s[2:3], -1
+; GFX1164-NEXT: v_writelane_b32 v3, s5, 48
+; GFX1164-NEXT: s_mov_b64 exec, s[2:3]
+; GFX1164-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX1164-NEXT: ; implicit-def: $vgpr4
+; GFX1164-NEXT: s_and_saveexec_b64 s[2:3], vcc
+; GFX1164-NEXT: s_cbranch_execz .LBB4_2
+; GFX1164-NEXT: ; %bb.1:
+; GFX1164-NEXT: v_mov_b32_e32 v4, s6
+; GFX1164-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1164-NEXT: global_atomic_add_u32 v4, v0, v4, s[0:1] glc
+; GFX1164-NEXT: s_waitcnt vmcnt(0)
+; GFX1164-NEXT: buffer_gl1_inv
+; GFX1164-NEXT: buffer_gl0_inv
+; GFX1164-NEXT: .LBB4_2:
+; GFX1164-NEXT: s_or_b64 exec, exec, s[2:3]
+; GFX1164-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1164-NEXT: v_mov_b32_e32 v4, v3
+; GFX1164-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1164-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1164-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1164-NEXT: global_store_b32 v0, v4, s[0:1]
+; GFX1164-NEXT: s_endpgm
+;
+; GFX1232-LABEL: add_i32_global_ignores_metadata:
+; GFX1232: ; %bb.0: ; %entry
+; GFX1232-NEXT: v_and_b32_e32 v0, 0x3ff, v0
+; GFX1232-NEXT: s_or_saveexec_b32 s0, -1
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_1) | instid1(SALU_CYCLE_1)
+; GFX1232-NEXT: v_cndmask_b32_e64 v1, 0, v0, s0
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1232-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1232-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1232-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1232-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_3) | instid1(VALU_DEP_1)
+; GFX1232-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1232-NEXT: s_mov_b32 exec_lo, s0
+; GFX1232-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1232-NEXT: s_or_saveexec_b32 s2, -1
+; GFX1232-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1232-NEXT: v_mov_b32_e32 v3, 0
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1232-NEXT: v_readlane_b32 s4, v1, 31
+; GFX1232-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1232-NEXT: v_readlane_b32 s3, v1, 15
+; GFX1232-NEXT: s_mov_b32 exec_lo, s2
+; GFX1232-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1232-NEXT: v_mbcnt_lo_u32_b32 v4, exec_lo, 0
+; GFX1232-NEXT: v_mov_b32_e32 v0, 0
+; GFX1232-NEXT: s_or_saveexec_b32 s2, -1
+; GFX1232-NEXT: v_writelane_b32 v3, s3, 16
+; GFX1232-NEXT: s_mov_b32 exec_lo, s2
+; GFX1232-NEXT: v_cmp_eq_u32_e32 vcc_lo, 0, v4
+; GFX1232-NEXT: ; implicit-def: $vgpr4
+; GFX1232-NEXT: s_and_saveexec_b32 s2, vcc_lo
+; GFX1232-NEXT: s_cbranch_execz .LBB4_2
+; GFX1232-NEXT: ; %bb.1:
+; GFX1232-NEXT: v_mov_b32_e32 v4, s4
+; GFX1232-NEXT: global_wb scope:SCOPE_SYS
+; GFX1232-NEXT: s_wait_storecnt 0x0
+; GFX1232-NEXT: s_wait_kmcnt 0x0
+; GFX1232-NEXT: global_atomic_add_u32 v4, v0, v4, s[0:1] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1232-NEXT: s_wait_loadcnt 0x0
+; GFX1232-NEXT: global_inv scope:SCOPE_SYS
+; GFX1232-NEXT: .LBB4_2:
+; GFX1232-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX1232-NEXT: s_or_b32 exec_lo, exec_lo, s2
+; GFX1232-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1232-NEXT: v_mov_b32_e32 v4, v3
+; GFX1232-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX1232-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1232-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1232-NEXT: s_wait_kmcnt 0x0
+; GFX1232-NEXT: global_store_b32 v0, v4, s[0:1]
+; GFX1232-NEXT: s_endpgm
+;
+; GFX1264-LABEL: add_i32_global_ignores_metadata:
+; GFX1264: ; %bb.0: ; %entry
+; GFX1264-NEXT: v_and_b32_e32 v0, 0x3ff, v0
+; GFX1264-NEXT: s_or_saveexec_b64 s[0:1], -1
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1) | instid1(SALU_CYCLE_1)
+; GFX1264-NEXT: v_cndmask_b32_e64 v1, 0, v0, s[0:1]
+; GFX1264-NEXT: v_mov_b32_e32 v3, 0
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v1, v1 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:1
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1264-NEXT: v_permlanex16_b32 v2, v1, -1, -1
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1264-NEXT: v_readlane_b32 s2, v1, 31
+; GFX1264-NEXT: v_mov_b32_e32 v2, s2
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1264-NEXT: v_add_nc_u32_dpp v1, v2, v1 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
+; GFX1264-NEXT: v_readlane_b32 s6, v1, 15
+; GFX1264-NEXT: v_mov_b32_dpp v3, v1 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1264-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1264-NEXT: s_load_b64 s[0:1], s[4:5], 0x24
+; GFX1264-NEXT: s_or_saveexec_b64 s[2:3], -1
+; GFX1264-NEXT: v_readlane_b32 s4, v1, 31
+; GFX1264-NEXT: v_writelane_b32 v3, s6, 16
+; GFX1264-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX1264-NEXT: s_mov_b64 exec, s[2:3]
+; GFX1264-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1264-NEXT: v_mbcnt_lo_u32_b32 v0, exec_lo, 0
+; GFX1264-NEXT: s_or_saveexec_b64 s[2:3], -1
+; GFX1264-NEXT: v_readlane_b32 s6, v1, 63
+; GFX1264-NEXT: v_readlane_b32 s5, v1, 47
+; GFX1264-NEXT: v_writelane_b32 v3, s4, 32
+; GFX1264-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX1264-NEXT: s_mov_b64 exec, s[2:3]
+; GFX1264-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1264-NEXT: v_mbcnt_hi_u32_b32 v4, exec_hi, v0
+; GFX1264-NEXT: v_mov_b32_e32 v0, 0
+; GFX1264-NEXT: s_or_saveexec_b64 s[2:3], -1
+; GFX1264-NEXT: v_writelane_b32 v3, s5, 48
+; GFX1264-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX1264-NEXT: s_mov_b64 exec, s[2:3]
+; GFX1264-NEXT: v_cmp_eq_u32_e32 vcc, 0, v4
+; GFX1264-NEXT: ; implicit-def: $vgpr4
+; GFX1264-NEXT: s_and_saveexec_b64 s[2:3], vcc
+; GFX1264-NEXT: s_cbranch_execz .LBB4_2
+; GFX1264-NEXT: ; %bb.1:
+; GFX1264-NEXT: v_mov_b32_e32 v4, s6
+; GFX1264-NEXT: global_wb scope:SCOPE_SYS
+; GFX1264-NEXT: s_wait_storecnt 0x0
+; GFX1264-NEXT: s_wait_kmcnt 0x0
+; GFX1264-NEXT: global_atomic_add_u32 v4, v0, v4, s[0:1] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1264-NEXT: s_wait_loadcnt 0x0
+; GFX1264-NEXT: global_inv scope:SCOPE_SYS
+; GFX1264-NEXT: .LBB4_2:
+; GFX1264-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX1264-NEXT: s_or_b64 exec, exec, s[2:3]
+; GFX1264-NEXT: v_readfirstlane_b32 s2, v4
+; GFX1264-NEXT: v_mov_b32_e32 v4, v3
+; GFX1264-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX1264-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1264-NEXT: v_add_nc_u32_e32 v4, s2, v4
+; GFX1264-NEXT: s_wait_kmcnt 0x0
+; GFX1264-NEXT: global_store_b32 v0, v4, s[0:1]
+; GFX1264-NEXT: s_endpgm
entry:
%lane = call i32 @llvm.amdgcn.workitem.id.x()
%old = atomicrmw add ptr addrspace(1) %inout, i32 %lane acq_rel, !amdgpu.expected.active.lanes !0
>From 126deef19f17566120b4ae4db372e99f9cee4ea8 Mon Sep 17 00:00:00 2001
From: YaFan <YaFan.Tao at amd.com>
Date: Mon, 22 Jun 2026 11:17:30 +0800
Subject: [PATCH 16/17] Change commetns
---
llvm/docs/AMDGPUUsage.rst | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 8693698cab3d1..e0292973dc247 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -2261,8 +2261,8 @@ operand is divergent, and the atomic is in the LDS (``addrspace(3)``) address
space, the optimizer normally replaces the per-lane atomics with a single
wavefront-wide DPP scan. That scan has a fixed overhead that is only
worthwhile when enough lanes participate. If the expected active lane count
-is small (currently ``<= 5``), the optimizer skips the DPP scan and instead
-lets each lane issue its own atomic.
+is small, the optimizer skips the DPP scan and instead lets each lane issue
+its own atomic.
The metadata is ignored for non-LDS atomics and when the DPP strategy is not
in use. Without this metadata the optimizer unconditionally applies the DPP
>From 2708342d5b73fd5e4c0da4770ed2d519c0aee744 Mon Sep 17 00:00:00 2001
From: YaFan <YaFan.Tao at amd.com>
Date: Tue, 30 Jun 2026 17:06:57 +0800
Subject: [PATCH 17/17] Fold metadata into the example lines
---
llvm/docs/AMDGPUUsage.rst | 7 ++-----
1 file changed, 2 insertions(+), 5 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index e0292973dc247..460ea287f49ed 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -2272,13 +2272,10 @@ scan to every eligible LDS atomic.
; Few expected active lanes (<= 5): DPP is skipped and each lane issues its
; own ds_add_rtn_u32.
- %old0 = atomicrmw add ptr addrspace(3) @lds, i32 %val acq_rel, !amdgpu.expected.active.lanes !0
+ %old0 = atomicrmw add ptr addrspace(3) @lds, i32 %val acq_rel, !amdgpu.expected.active.lanes !{i32 4}
; Many expected active lanes (> 5): the DPP scan is applied.
- %old1 = atomicrmw add ptr addrspace(3) @lds, i32 %val acq_rel, !amdgpu.expected.active.lanes !1
-
- !0 = !{i32 4}
- !1 = !{i32 32}
+ %old1 = atomicrmw add ptr addrspace(3) @lds, i32 %val acq_rel, !amdgpu.expected.active.lanes !{i32 32}
LLVM IR Attributes
More information about the llvm-commits
mailing list