[llvm] [AMDGPU] Add waterfall intrinsics (PR #192409)
via llvm-commits
llvm-commits at lists.llvm.org
Wed Jun 3 05:08:21 PDT 2026
https://github.com/gretay-amd updated https://github.com/llvm/llvm-project/pull/192409
>From 375bafbf1487b4972989a8eb8eb6321bf92e6f8b Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Thu, 2 Apr 2026 11:18:22 +0100
Subject: [PATCH 01/35] Add waterfall intrinsics
Co-authored-by: David Stuttard <David.Stuttard at amd.com>
Co-authored-by: Carl Ritson <Carl.Ritson at amd.com>
Co-authored-by: Jay Foad <Jay.Foad at amd.com>
Co-authored-by: Piotr Sobczak <piotr.sobczak at amd.com>
Co-authored-by: Joseph Nash <Joseph.Nash at amd.com>
Co-authored-by: Sebastian Neubauer <Sebastian.Neubauer at amd.com>
Assisted-by: Claude
---
llvm/docs/AMDGPUUsage.rst | 52 +
llvm/include/llvm/IR/IntrinsicsAMDGPU.td | 67 +
llvm/lib/Target/AMDGPU/AMDGPU.h | 4 +
.../Target/AMDGPU/AMDGPUAtomicOptimizer.cpp | 89 +-
.../Target/AMDGPU/AMDGPUCodeGenPrepare.cpp | 69 +-
.../AMDGPU/AMDGPUInstCombineIntrinsic.cpp | 26 +
llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def | 1 +
.../Target/AMDGPU/AMDGPURegBankCombiner.cpp | 54 +
.../Target/AMDGPU/AMDGPURegisterBankInfo.cpp | 50 +
.../Target/AMDGPU/AMDGPUSearchableTables.td | 4 +
.../lib/Target/AMDGPU/AMDGPUTargetMachine.cpp | 4 +
llvm/lib/Target/AMDGPU/CMakeLists.txt | 1 +
llvm/lib/Target/AMDGPU/SIISelLowering.cpp | 21 +
llvm/lib/Target/AMDGPU/SIInsertWaterfall.cpp | 896 ++
llvm/lib/Target/AMDGPU/SIInsertWaterfall.h | 22 +
llvm/lib/Target/AMDGPU/SIInstructions.td | 107 +
.../AMDGPU/MIR/llvm.amdgcn.waterfall.mir | 51 +
.../AMDGPU/llvm.amdgcn.waterfall.ll | 29 +
.../AMDGPU/amdgcn.waterfall.atomic.opt.ll | 406 +
llvm/test/CodeGen/AMDGPU/llc-pipeline-npm.ll | 3 +
llvm/test/CodeGen/AMDGPU/llc-pipeline.ll | 16 +
.../CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll | 12656 ++++++++++++++++
.../AMDGPU/si-insert-waterfall-licm.ll | 43 +
.../CodeGen/AMDGPU/si-insert-waterfall.mir | 54 +
.../InstCombine/AMDGPU/waterfall.ll | 73 +
25 files changed, 14793 insertions(+), 5 deletions(-)
create mode 100644 llvm/lib/Target/AMDGPU/SIInsertWaterfall.cpp
create mode 100644 llvm/lib/Target/AMDGPU/SIInsertWaterfall.h
create mode 100644 llvm/test/Analysis/UniformityAnalysis/AMDGPU/MIR/llvm.amdgcn.waterfall.mir
create mode 100644 llvm/test/Analysis/UniformityAnalysis/AMDGPU/llvm.amdgcn.waterfall.ll
create mode 100644 llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
create mode 100644 llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
create mode 100644 llvm/test/CodeGen/AMDGPU/si-insert-waterfall-licm.ll
create mode 100644 llvm/test/CodeGen/AMDGPU/si-insert-waterfall.mir
create mode 100644 llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 61841716792ef..2e8b535a66486 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -1918,6 +1918,58 @@ The AMDGPU backend implements the following LLVM IR intrinsics.
This intrinsic has 1 operand:
* Local pointer to the LDS barrier data.
+ llvm.amdgcn.waterfall.begin Marks the beginning of a waterfall region of code.
+
+ The compiler generates a waterfall loop around the region.
+ A waterfall loop handles the case where an operation that requires
+ a uniform operand (e.g., held in an SGPR) is applied to a divergent operand
+ (held in a VGPR, with values varying per lane).
+ Each iteration of the waterfall loop activates a subset of lanes that
+ share the same value of the VGPR (the value in the first active lane).
+ The operation is then executed using that value as the uniform operand.
+ If the VGPR is already uniform, the waterfall loop executes only once.
+ The worst case for a waterfall loop is one iteration per lane (all lanes
+ have different values of the VGPR), but it is not common in practice.
+
+ The intrinsic takes a previous token
+ (``i32``; use a null/zero value if this is the first ``waterfall.begin`` in
+ a waterfall group) and a VGPR.
+
+ The intrinsic returns a new token that must be threaded through the
+ corresponding ``waterfall.readfirstlane`` and ``waterfall.end`` or
+ ``waterfall.last_use`` intrinsics, forming a waterfall group of intrinsics
+ that together define a waterfall region.
+
+ All intrinsics in a waterfall group must reside in the same basic block.
+
+ Multiple ``waterfall.begin`` intrinsics can be chained by
+ passing the token of the preceding ``waterfall.begin`` as the first argument.
+ This allows a front-end to create one waterfall loop for
+ multiple non-uniform values.
+ Later compiler passes may remove values determined as uniform.
+ The final token is used for other waterfall intrinsics in the same group.
+
+ llvm.amdgcn.waterfall.readfirstlane Reads the first active lane's value of the VGPR and returns it as
+ an SGPR for use within a waterfall region.
+
+ Takes the ``i32`` token from the final ``waterfall.begin`` in the waterfall
+ group and the VGPR. Returns the uniform (SGPR) result.
+
+ If the VGPR is determined to be uniform at compile time, this intrinsic
+ is optimized away (the input VGPR value is used directly).
+
+ llvm.amdgcn.waterfall.end Marks the end of a waterfall region.
+ Takes the ``i32`` token from the final ``waterfall.begin``
+
+ llvm.amdgcn.waterfall.last_use Variant of ``waterfall.end`` for values whose last use is in a
+ non-defining operation such as a store. Marks that the use of the value
+ constitutes the end of the waterfall region.
+
+ llvm.amdgcn.waterfall.last_use_vgpr Variant of ``waterfall.last_use`` for values that remain in a VGPR.
+
+ llvm.amdgcn.waterfall.loop_end Inserted later by the compiler to be used with ```waterfall.last_use*``
+ to mark the loop-end point for special
+ handling such as SCC clobber tracking.
============================================== ==========================================================
diff --git a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
index 2fd5285dfc330..c49535305e651 100644
--- a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
+++ b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
@@ -2720,6 +2720,73 @@ def int_amdgcn_cs_chain:
],
[IntrConvergent, IntrNoReturn, ImmArg<ArgIndex<4>>]>;
+// Waterfall intrinsics are used to mark a region as requiring waterfall loops to
+// activate and deactivate lanes in a loop (and sometimes transform VGPR values
+// into SGPR values).
+// Best case (uniform index in VGPR) the loop will execute once, worst case (all
+// values in the index are different) the loop will execute wave-size times.
+// The waterfall.begin intrinsic returns a new token that must be threaded through the
+// corresponding waterfall.readfirstlane and waterfall.end or
+// waterfall.last_use intrinsics, forming a waterfall group of intrinsics that together
+// define a waterfall region.
+// All intrinsics in a waterfall group must reside within the same basic block.
+// Where the intrinsic specifies "llvm_any_ty", the intrinsic will accept any of
+// the following types:
+// i16, v2i16, v4i16, i32, v2i32, v4i32, v8i32, f16, v2f16, v4f16, f32, v2f32,
+// v4f32, v8f32
+// TODO: extend the support to allow spanning multiple blocks
+def int_amdgcn_waterfall_begin :
+ Intrinsic<[llvm_i32_ty], // Returns sgpr token
+ [llvm_i32_ty, // Previous begin token / null if first
+ llvm_any_ty], // VGPR index for waterfall
+ [IntrConvergent]>;
+
+// Often the index from begin will be used as the readfirstlane value from
+// waterfall code - however, this isn't always the case. The readfirstlane style
+// intrinsic is required to be used, but often is a no-op if the index is the
+// used directly.
+def int_amdgcn_waterfall_readfirstlane :
+ Intrinsic<[llvm_any_ty], // Returns any value
+ [llvm_i32_ty, // Token from final begin
+ llvm_any_ty], // VGPR value to be used in readfirstlane
+ [IntrConvergent]>;
+
+// Effectively a no-op, but this intrinsic is used to indicate the end of the
+// region for waterfall by tagging a value as the end of the section.
+// Waterfall will happen for all values dependent on the begin until the end
+// intrinsic is reached,
+def int_amdgcn_waterfall_end :
+ Intrinsic<[llvm_any_ty], // Returns any value
+ [llvm_i32_ty, // Token from final begin
+ LLVMMatchType<0>], // Same as return value
+ [IntrConvergent]>;
+
+// Effectively a no-op, no code results directly from this intrinsic. This is
+// used to indicate that the use of the defined value constitutes the end of the
+// waterfall propagation. This is roughly equivalent to the end intrinsic, but
+// can be used to tag values that are used in non-defining operations such as a
+// store.
+def int_amdgcn_waterfall_last_use :
+ Intrinsic<[llvm_any_ty], // Returns any value
+ [llvm_i32_ty, // Token from begin
+ LLVMMatchType<0>], // Same as return value
+ [IntrConvergent]>;
+
+// Special variant where the last-use uses a VGPR that needs to be uniform.
+def int_amdgcn_waterfall_last_use_vgpr :
+ Intrinsic<[llvm_any_ty], // Returns any value
+ [llvm_i32_ty, // Token from begin
+ LLVMMatchType<0>], // Same as return value
+ []>;
+
+// Special waterfall loop end intrinsic to be used with last_use
+// This is inserted late, before codegen, mainly to mark the loop end
+// for special handling such as scc clobbers.
+def int_amdgcn_waterfall_loop_end :
+ Intrinsic<[],
+ [llvm_i32_ty], // Token from begin
+ [IntrConvergent]>;
+
// Run a function with all the lanes enabled. Only direct calls are allowed. The
// first argument is the callee, which must have the `amdgpu_gfx_whole_wave`
// calling convention and must not be variadic. The remaining arguments to the
diff --git a/llvm/lib/Target/AMDGPU/AMDGPU.h b/llvm/lib/Target/AMDGPU/AMDGPU.h
index c6dd1dbb62449..fc46d0f82345a 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPU.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPU.h
@@ -53,6 +53,7 @@ FunctionPass *createSIFixSGPRCopiesLegacyPass();
FunctionPass *createLowerWWMCopiesPass();
FunctionPass *createSIMemoryLegalizerPass();
FunctionPass *createSIInsertWaitcntsPass();
+FunctionPass *createSIInsertWaterfallPass();
FunctionPass *createSIPreAllocateWWMRegsLegacyPass();
FunctionPass *createSIFormMemoryClausesLegacyPass();
@@ -244,6 +245,9 @@ extern char &SILateBranchLoweringPassID;
void initializeSIOptimizeExecMaskingLegacyPass(PassRegistry &);
extern char &SIOptimizeExecMaskingLegacyID;
+void initializeSIInsertWaterfallPass(PassRegistry &);
+extern char &SIInsertWaterfallID;
+
void initializeSIPreAllocateWWMRegsLegacyPass(PassRegistry &);
extern char &SIPreAllocateWWMRegsLegacyID;
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index a4029fb79b49a..5579202350b5d 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -28,6 +28,7 @@
#include "llvm/CodeGen/TargetPassConfig.h"
#include "llvm/IR/IRBuilder.h"
#include "llvm/IR/InstVisitor.h"
+#include "llvm/IR/IntrinsicInst.h"
#include "llvm/IR/IntrinsicsAMDGPU.h"
#include "llvm/InitializePasses.h"
#include "llvm/Target/TargetMachine.h"
@@ -89,6 +90,8 @@ class AMDGPUAtomicOptimizerImpl
void optimizeAtomic(Instruction &I, AtomicRMWInst::BinOp Op, unsigned ValIdx,
bool ValDivergent) const;
+ void processBB(BasicBlock &BB);
+
public:
AMDGPUAtomicOptimizerImpl() = delete;
@@ -157,7 +160,8 @@ bool AMDGPUAtomicOptimizerImpl::run() {
if (ST.isSingleLaneExecution(F))
return false;
- visit(F);
+ for (auto &BB : F)
+ processBB(BB);
if (ToReplace.empty())
return false;
@@ -181,6 +185,89 @@ static bool isLegalCrossLaneType(Type *Ty) {
}
}
+static Instruction *findLastInWaterfall(Instruction &Begin) {
+ // Given Begin as the first begin for a waterfall group, look through the
+ // instructions tagged as part of the waterfall and return the last use.
+ // If the group is malformed, then return nullptr.
+
+ Instruction *FinalBegin = &Begin;
+
+ // Drill through any begin intrinsics
+ do {
+ if (FinalBegin->hasOneUse()) {
+ User *U = *FinalBegin->user_begin();
+ IntrinsicInst *Intrin = dyn_cast<IntrinsicInst>(U);
+ if (Intrin &&
+ Intrin->getIntrinsicID() == Intrinsic::amdgcn_waterfall_begin) {
+ FinalBegin = Intrin;
+ continue;
+ }
+ }
+ } while (false);
+
+ Instruction *Last = FinalBegin;
+
+ for (auto Use : FinalBegin->users()) {
+ if (auto *Intrin = dyn_cast<IntrinsicInst>(Use)) {
+ switch (Intrin->getIntrinsicID()) {
+ default: {
+ // Unexpected intrinsic
+ return nullptr;
+ }
+ case Intrinsic::amdgcn_waterfall_begin:
+ // Badly formed WF group - should already have discovered the last begin
+ // intrinsic before entering this loop.
+ return nullptr;
+ case Intrinsic::amdgcn_waterfall_end:
+ if (Last->comesBefore(Intrin))
+ Last = Intrin;
+ break;
+ case Intrinsic::amdgcn_waterfall_last_use:
+ case Intrinsic::amdgcn_waterfall_last_use_vgpr: {
+ // Find the actual last use
+ for (auto &LastUse : Intrin->uses()) {
+ auto *LUI = static_cast<Instruction *>(LastUse.getUser());
+ if (LUI->getParent() != Intrin->getParent()) {
+ // LUI has to be in same BB as waterfall intrinsics.
+ return nullptr;
+ }
+ if (Last->comesBefore(LUI))
+ Last = LUI;
+ }
+ break;
+ }
+ case Intrinsic::amdgcn_waterfall_readfirstlane:
+ // Always in the middle of a group - so doesn't have any effect on group
+ // detection
+ break;
+ }
+ }
+ }
+
+ return Last;
+}
+
+void AMDGPUAtomicOptimizerImpl::processBB(BasicBlock &BB) {
+ // Visit all instructions in a BB unless they're inside a waterfall loop
+ for (auto I = BB.begin(); I != BB.end(); ++I) {
+ if (auto *Intrin = dyn_cast<IntrinsicInst>(&*I)) {
+ if (Intrin->getIntrinsicID() == Intrinsic::amdgcn_waterfall_begin) {
+ auto *Last = findLastInWaterfall(*I);
+ if (!Last) {
+ // Malformed waterfall group - assume that all instructions in this
+ // BB are inside a waterfall group
+ return;
+ }
+ I = Last->getIterator();
+ continue;
+ }
+ }
+ // Safe to perform transformations on this instruction
+ // which is not inside a waterfall group
+ visit(*I);
+ }
+}
+
void AMDGPUAtomicOptimizerImpl::visitAtomicRMWInst(AtomicRMWInst &I) {
// Early exit for unhandled address space atomic instructions.
switch (I.getPointerAddressSpace()) {
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp b/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp
index 1bcfc1da3b84e..52e517dffd43e 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp
@@ -18,6 +18,7 @@
#include "llvm/ADT/SetVector.h"
#include "llvm/Analysis/AssumptionCache.h"
#include "llvm/Analysis/ConstantFolding.h"
+#include "llvm/Analysis/LoopInfo.h"
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/Analysis/UniformityAnalysis.h"
@@ -102,6 +103,7 @@ class AMDGPUCodeGenPrepareImpl
const AMDGPUTargetMachine &TM;
const TargetLibraryInfo *TLI;
const UniformityInfo &UA;
+ const LoopInfo &LI;
const DataLayout &DL;
SimplifyQuery SQ;
const bool HasFP32DenormalFlush;
@@ -112,11 +114,14 @@ class AMDGPUCodeGenPrepareImpl
DenseMap<const PHINode *, bool> BreakPhiNodesCache;
+ IntrinsicInst *CurrentWaterfall = nullptr;
+
AMDGPUCodeGenPrepareImpl(Function &F, const AMDGPUTargetMachine &TM,
const TargetLibraryInfo *TLI, AssumptionCache *AC,
- const DominatorTree *DT, const UniformityInfo &UA)
+ const DominatorTree *DT, const UniformityInfo &UA,
+ const LoopInfo &LI)
: F(F), ST(TM.getSubtarget<GCNSubtarget>(F)), TM(TM), TLI(TLI), UA(UA),
- DL(F.getDataLayout()), SQ(DL, TLI, DT, AC),
+ LI(LI), DL(F.getDataLayout()), SQ(DL, TLI, DT, AC),
HasFP32DenormalFlush(SIModeRegisterDefaults(F, ST).FP32Denormals ==
DenormalMode::getPreserveSign()) {}
@@ -259,6 +264,7 @@ class AMDGPUCodeGenPrepareImpl
bool visitAddrSpaceCastInst(AddrSpaceCastInst &I);
bool visitIntrinsicInst(IntrinsicInst &I);
+ bool visitWaterfallLastUseIntrinsic(IntrinsicInst &I);
bool visitFMinLike(IntrinsicInst &I);
bool visitSqrt(IntrinsicInst &I);
bool visitLog(FPMathOperator &Log, Intrinsic::ID IID);
@@ -266,6 +272,8 @@ class AMDGPUCodeGenPrepareImpl
bool visitMbcntHi(IntrinsicInst &I) const;
bool visitVectorReduceAdd(IntrinsicInst &I);
bool visitSaturatingAdd(IntrinsicInst &I);
+ bool visitGetElementPtrInst(GetElementPtrInst &I);
+
bool run();
};
@@ -275,6 +283,7 @@ class AMDGPUCodeGenPrepare : public FunctionPass {
AMDGPUCodeGenPrepare() : FunctionPass(ID) {}
void getAnalysisUsage(AnalysisUsage &AU) const override {
AU.addRequired<AssumptionCacheTracker>();
+ AU.addRequired<LoopInfoWrapperPass>();
AU.addRequired<UniformityInfoWrapperPass>();
AU.addRequired<TargetLibraryInfoWrapperPass>();
@@ -2075,11 +2084,60 @@ bool AMDGPUCodeGenPrepareImpl::visitIntrinsicInst(IntrinsicInst &I) {
case Intrinsic::uadd_sat:
case Intrinsic::sadd_sat:
return visitSaturatingAdd(I);
+ case Intrinsic::amdgcn_waterfall_begin:
+ CurrentWaterfall = nullptr;
+ return false;
+ case Intrinsic::amdgcn_waterfall_end:
+ CurrentWaterfall = &I;
+ return false;
+ case Intrinsic::amdgcn_waterfall_last_use:
+ return visitWaterfallLastUseIntrinsic(I);
default:
return false;
}
}
+bool AMDGPUCodeGenPrepareImpl::visitGetElementPtrInst(GetElementPtrInst &I) {
+ if (!CurrentWaterfall || UA.isUniform(&I))
+ return false;
+ if (I.getParent() != CurrentWaterfall->getParent())
+ return false;
+
+ // Divergent GEP within a waterfall region will introduce a nested waterfall.
+ // This will lead to bad or broken code gen.
+ // Pointer likely became non-uniform due to sinking into a divergent loop.
+ // Make sure LICM can run on the relevant loops and hope it can
+ // hoist/canonicalize the pointer.
+ LLVM_DEBUG(dbgs() << "Divergent GEP found in waterfall = " << I << "\n");
+ Value *PtrVal = I.getPointerOperand()->stripPointerCasts();
+ Instruction *PtrInst = dyn_cast<Instruction>(PtrVal);
+ if (!PtrInst)
+ return false;
+
+ Loop *CurrentLoop = LI.getLoopFor(PtrInst->getParent());
+ while (CurrentLoop) {
+ MDNode *LoopID = CurrentLoop->getLoopID();
+ MDNode *NewLoopID = makePostTransformationMetadata(
+ I.getContext(), LoopID, {"llvm.licm.disable"}, {});
+ CurrentLoop->setLoopID(NewLoopID);
+ CurrentLoop = CurrentLoop->getParentLoop();
+ }
+
+ return true;
+}
+
+bool AMDGPUCodeGenPrepareImpl::visitWaterfallLastUseIntrinsic(
+ IntrinsicInst &I) {
+ auto *Token = I.getOperand(0);
+ for (auto *U : I.users()) {
+ auto *UI = cast<Instruction>(U);
+ BasicBlock::iterator InsertPt = std::next(UI->getIterator());
+ IRBuilder<> Builder(I.getParent(), InsertPt);
+ Builder.CreateIntrinsic(Intrinsic::amdgcn_waterfall_loop_end, {}, {Token});
+ }
+ return true;
+}
+
/// Match the core sequence in the fract pattern (x - floor(x), which doesn't
/// need to consider edge case handling.
Value *AMDGPUCodeGenPrepareImpl::matchFractPatImpl(Value &FractSrc,
@@ -2283,7 +2341,8 @@ bool AMDGPUCodeGenPrepare::runOnFunction(Function &F) {
const DominatorTree *DT = DTWP ? &DTWP->getDomTree() : nullptr;
const UniformityInfo &UA =
getAnalysis<UniformityInfoWrapperPass>().getUniformityInfo();
- return AMDGPUCodeGenPrepareImpl(F, TM, TLI, AC, DT, UA).run();
+ const LoopInfo &LI = getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
+ return AMDGPUCodeGenPrepareImpl(F, TM, TLI, AC, DT, UA, LI).run();
}
PreservedAnalyses AMDGPUCodeGenPreparePass::run(Function &F,
@@ -2293,7 +2352,8 @@ PreservedAnalyses AMDGPUCodeGenPreparePass::run(Function &F,
AssumptionCache *AC = &FAM.getResult<AssumptionAnalysis>(F);
const DominatorTree *DT = FAM.getCachedResult<DominatorTreeAnalysis>(F);
const UniformityInfo &UA = FAM.getResult<UniformityInfoAnalysis>(F);
- AMDGPUCodeGenPrepareImpl Impl(F, ATM, TLI, AC, DT, UA);
+ const LoopInfo &LI = FAM.getResult<LoopAnalysis>(F);
+ AMDGPUCodeGenPrepareImpl Impl(F, ATM, TLI, AC, DT, UA, LI);
if (!Impl.run())
return PreservedAnalyses::all();
PreservedAnalyses PA = PreservedAnalyses::none();
@@ -2305,6 +2365,7 @@ PreservedAnalyses AMDGPUCodeGenPreparePass::run(Function &F,
INITIALIZE_PASS_BEGIN(AMDGPUCodeGenPrepare, DEBUG_TYPE,
"AMDGPU IR optimizations", false, false)
INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
+INITIALIZE_PASS_DEPENDENCY(LoopInfoWrapperPass)
INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
INITIALIZE_PASS_DEPENDENCY(UniformityInfoWrapperPass)
INITIALIZE_PASS_END(AMDGPUCodeGenPrepare, DEBUG_TYPE, "AMDGPU IR optimizations",
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp
index b288f30d8c291..7d3428d794e29 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp
@@ -143,6 +143,9 @@ static std::optional<Instruction *> modifyIntrinsicCall(
NewCall->copyMetadata(OldIntr);
if (isa<FPMathOperator>(NewCall))
NewCall->copyFastMathFlags(&OldIntr);
+ // Copy attributes
+ AttributeList OldAttrList = OldIntr.getAttributes();
+ NewCall->setAttributes(OldAttrList);
// Erase and replace uses
if (!InstToReplace.getType()->isVoidTy())
@@ -2082,6 +2085,22 @@ GCNTTIImpl::instCombineIntrinsic(InstCombiner &IC, IntrinsicInst &II) const {
return IC.replaceInstUsesWith(II, ConstantInt::getFalse(II.getType()));
break;
}
+ case Intrinsic::amdgcn_waterfall_begin: {
+ Value *Index = II.getArgOperand(1);
+ // If there is a previous waterfall begin with the same index, we can remove
+ // this one.
+ IntrinsicInst *PrevII;
+ for (Value *Token = II.getArgOperand(0);
+ (PrevII = dyn_cast<IntrinsicInst>(Token)) &&
+ PrevII->getIntrinsicID() == Intrinsic::amdgcn_waterfall_begin;
+ Token = PrevII->getArgOperand(0)) {
+ if (Index == PrevII->getArgOperand(1)) {
+ IC.replaceInstUsesWith(II, II.getArgOperand(0));
+ return IC.eraseInstFromFunction(II);
+ }
+ }
+ break;
+ }
case Intrinsic::amdgcn_make_buffer_rsrc: {
Value *Src = II.getArgOperand(0);
if (isa<PoisonValue>(Src))
@@ -2379,6 +2398,8 @@ static Value *simplifyAMDGCNMemoryIntrinsicDemanded(InstCombiner &IC,
IC.Builder.CreateIntrinsic(II.getIntrinsicID(), OverloadTys, Args);
NewCall->takeName(&II);
NewCall->copyMetadata(II);
+ AttributeList OldAttrList = II.getAttributes();
+ NewCall->setAttributes(OldAttrList);
if (IsLoad) {
if (NewNumElts == 1) {
@@ -2492,6 +2513,11 @@ std::optional<Value *> GCNTTIImpl::simplifyDemandedVectorEltsIntrinsic(
case Intrinsic::amdgcn_struct_tbuffer_load:
case Intrinsic::amdgcn_struct_ptr_tbuffer_load:
return simplifyAMDGCNMemoryIntrinsicDemanded(IC, II, DemandedElts);
+ case Intrinsic::amdgcn_waterfall_end:
+ // Propagate demanded elements through to the value being returned by the
+ // waterfall loop.
+ SimplifyAndSetOp(&II, 1, DemandedElts, UndefElts);
+ break;
default: {
if (getAMDGPUImageDMaskIntrinsic(II.getIntrinsicID())) {
return simplifyAMDGCNMemoryIntrinsicDemanded(IC, II, DemandedElts, 0);
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def b/llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def
index 2a6560b309e62..ae151adccf605 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def
+++ b/llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def
@@ -137,6 +137,7 @@ MACHINE_FUNCTION_PASS("si-form-memory-clauses", SIFormMemoryClausesPass())
MACHINE_FUNCTION_PASS("si-i1-copies", SILowerI1CopiesPass())
MACHINE_FUNCTION_PASS("si-insert-hard-clauses", SIInsertHardClausesPass())
MACHINE_FUNCTION_PASS("si-insert-waitcnts", SIInsertWaitcntsPass())
+MACHINE_FUNCTION_PASS("si-insert-waterfall", SIInsertWaterfallPass())
MACHINE_FUNCTION_PASS("si-late-branch-lowering", SILateBranchLoweringPass())
MACHINE_FUNCTION_PASS("si-load-store-opt", SILoadStoreOptimizerPass())
MACHINE_FUNCTION_PASS("si-lower-control-flow", SILowerControlFlowPass())
diff --git a/llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp b/llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp
index 35c0d4046f41e..eeff9c53ed998 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp
@@ -20,11 +20,14 @@
#include "llvm/CodeGen/GlobalISel/Combiner.h"
#include "llvm/CodeGen/GlobalISel/CombinerHelper.h"
#include "llvm/CodeGen/GlobalISel/CombinerInfo.h"
+#include "llvm/CodeGen/GlobalISel/GIMatchTableExecutor.h"
#include "llvm/CodeGen/GlobalISel/GIMatchTableExecutorImpl.h"
#include "llvm/CodeGen/GlobalISel/GISelValueTracking.h"
+#include "llvm/CodeGen/GlobalISel/GenericMachineInstrs.h"
#include "llvm/CodeGen/GlobalISel/MIPatternMatch.h"
#include "llvm/CodeGen/MachineDominators.h"
#include "llvm/CodeGen/TargetPassConfig.h"
+#include "llvm/IR/IntrinsicsAMDGPU.h"
#include "llvm/Target/TargetMachine.h"
#define GET_GICOMBINER_DEPS
@@ -89,6 +92,19 @@ class AMDGPURegBankCombinerImpl : public Combiner {
void applyCanonicalizeZextShiftAmt(MachineInstr &MI, MachineInstr &Ext) const;
+ struct RmUniformWFMatchInfo {
+ Register Dst, WFReplaceReg;
+ };
+
+ // Combine support to remove amdgcn_waterfall_readfirstlane or
+ // amdgcn_waterfall_begin intrinsics if the index is determined to be
+ // uniform. The SIInsertWaterfall pass can handle their removal and in some
+ // cases remove the waterfall altogether
+ bool matchRmUniformWF(MachineInstr &MI,
+ RmUniformWFMatchInfo &MatchInfo) const;
+ void applyRmUniformWF(MachineInstr &MI,
+ RmUniformWFMatchInfo &MatchInfo) const;
+
bool combineD16Load(MachineInstr &MI) const;
bool applyD16Load(unsigned D16Opc, MachineInstr &DstMI,
MachineInstr *SmallLoad, Register ToOverwriteD16) const;
@@ -396,6 +412,44 @@ void AMDGPURegBankCombinerImpl::applyCanonicalizeZextShiftAmt(
MI.eraseFromParent();
}
+bool AMDGPURegBankCombinerImpl::matchRmUniformWF(
+ MachineInstr &MI, RmUniformWFMatchInfo &MatchInfo) const {
+ auto IntrID = cast<GIntrinsic>(MI).getIntrinsicID();
+ Register Dst, WFReplaceReg;
+
+ switch (IntrID) {
+ case Intrinsic::amdgcn_waterfall_readfirstlane: {
+ Register IdxReg = MI.getOperand(3).getReg();
+ Register IdxSrcReg = getSrcRegIgnoringCopies(IdxReg, MRI);
+ if (!isVgprRegBank(IdxSrcReg)) {
+ WFReplaceReg = IdxSrcReg;
+ break;
+ }
+ return false;
+ }
+ case Intrinsic::amdgcn_waterfall_begin: {
+ Register IdxReg = MI.getOperand(3).getReg();
+ if (!isVgprRegBank(getSrcRegIgnoringCopies(IdxReg, MRI))) {
+ WFReplaceReg = MI.getOperand(2).getReg();
+ break;
+ }
+ return false;
+ }
+ default:
+ return false;
+ }
+
+ Dst = MI.getOperand(0).getReg();
+ MatchInfo = {Dst, WFReplaceReg};
+ return true;
+}
+
+void AMDGPURegBankCombinerImpl::applyRmUniformWF(
+ MachineInstr &MI, RmUniformWFMatchInfo &MatchInfo) const {
+ MI.eraseFromParent();
+ Helper.replaceRegWith(MRI, MatchInfo.Dst, MatchInfo.WFReplaceReg);
+}
+
bool AMDGPURegBankCombinerImpl::combineD16Load(MachineInstr &MI) const {
Register Dst;
MachineInstr *Load, *SextLoad;
diff --git a/llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp b/llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp
index a24df782cf28a..e680e658835bb 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp
@@ -5520,6 +5520,56 @@ AMDGPURegisterBankInfo::getInstrMapping(const MachineInstr &MI) const {
OpdsMapping[1] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, Size);
break;
}
+ case Intrinsic::amdgcn_waterfall_begin: {
+ unsigned SizeDst = getSizeInBits(MI.getOperand(0).getReg(), MRI, *TRI);
+ unsigned SizeSrc1 = getSizeInBits(MI.getOperand(2).getReg(), MRI, *TRI);
+ unsigned SizeSrc2 = getSizeInBits(MI.getOperand(3).getReg(), MRI, *TRI);
+ OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, SizeDst);
+ OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, SizeSrc1);
+ OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, SizeSrc2);
+ break;
+ }
+ case Intrinsic::amdgcn_waterfall_readfirstlane: {
+ unsigned SizeDst = getSizeInBits(MI.getOperand(0).getReg(), MRI, *TRI);
+ unsigned SizeSrc1 = getSizeInBits(MI.getOperand(2).getReg(), MRI, *TRI);
+ unsigned SizeSrc2 = getSizeInBits(MI.getOperand(3).getReg(), MRI, *TRI);
+ OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, SizeDst);
+ OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, SizeSrc1);
+ OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, SizeSrc2);
+ break;
+ }
+ case Intrinsic::amdgcn_waterfall_end: {
+ unsigned SizeDst = getSizeInBits(MI.getOperand(0).getReg(), MRI, *TRI);
+ unsigned SizeSrc1 = getSizeInBits(MI.getOperand(2).getReg(), MRI, *TRI);
+ unsigned SizeSrc2 = getSizeInBits(MI.getOperand(3).getReg(), MRI, *TRI);
+ OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, SizeDst);
+ OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, SizeSrc1);
+ OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, SizeSrc2);
+ break;
+ }
+ case Intrinsic::amdgcn_waterfall_last_use: {
+ unsigned SizeDst = getSizeInBits(MI.getOperand(0).getReg(), MRI, *TRI);
+ unsigned SizeSrc1 = getSizeInBits(MI.getOperand(2).getReg(), MRI, *TRI);
+ unsigned SizeSrc2 = getSizeInBits(MI.getOperand(3).getReg(), MRI, *TRI);
+ OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, SizeDst);
+ OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, SizeSrc1);
+ OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, SizeSrc2);
+ break;
+ }
+ case Intrinsic::amdgcn_waterfall_last_use_vgpr: {
+ unsigned SizeDst = getSizeInBits(MI.getOperand(0).getReg(), MRI, *TRI);
+ unsigned SizeSrc1 = getSizeInBits(MI.getOperand(2).getReg(), MRI, *TRI);
+ unsigned SizeSrc2 = getSizeInBits(MI.getOperand(3).getReg(), MRI, *TRI);
+ OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, SizeDst);
+ OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, SizeSrc1);
+ OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, SizeSrc2);
+ break;
+ }
+ case Intrinsic::amdgcn_waterfall_loop_end: {
+ unsigned SizeSrc1 = getSizeInBits(MI.getOperand(1).getReg(), MRI, *TRI);
+ OpdsMapping[1] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, SizeSrc1);
+ break;
+ }
case Intrinsic::amdgcn_ds_gws_init:
case Intrinsic::amdgcn_ds_gws_barrier:
case Intrinsic::amdgcn_ds_gws_sema_br: {
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUSearchableTables.td b/llvm/lib/Target/AMDGPU/AMDGPUSearchableTables.td
index 0968d7030578b..b83ea7b96afad 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUSearchableTables.td
+++ b/llvm/lib/Target/AMDGPU/AMDGPUSearchableTables.td
@@ -337,6 +337,7 @@ def : SourceOfDivergence<int_amdgcn_permlane16_swap>;
def : SourceOfDivergence<int_amdgcn_permlane32_swap>;
def : SourceOfDivergence<int_amdgcn_set_inactive>;
def : SourceOfDivergence<int_amdgcn_set_inactive_chain_arg>;
+def : SourceOfDivergence<int_amdgcn_waterfall_end>;
foreach intr = AMDGPUMFMAIntrinsics908 in
def : SourceOfDivergence<intr>;
@@ -402,6 +403,9 @@ def : AlwaysUniform<int_amdgcn_icmp>;
def : AlwaysUniform<int_amdgcn_fcmp>;
def : AlwaysUniform<int_amdgcn_ballot>;
def : AlwaysUniform<int_amdgcn_if_break>;
+def : AlwaysUniform<int_amdgcn_waterfall_readfirstlane>;
+def : AlwaysUniform<int_amdgcn_waterfall_begin>;
+def : AlwaysUniform<int_amdgcn_waterfall_last_use>;
def : AlwaysUniform<int_amdgcn_cluster_workgroup_id_x>;
def : AlwaysUniform<int_amdgcn_cluster_workgroup_id_y>;
def : AlwaysUniform<int_amdgcn_cluster_workgroup_id_z>;
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
index b572e47ce0f21..21eee8419b518 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
@@ -53,6 +53,7 @@
#include "SIFixVGPRCopies.h"
#include "SIFoldOperands.h"
#include "SIFormMemoryClauses.h"
+#include "SIInsertWaterfall.h"
#include "SILoadStoreOptimizer.h"
#include "SILowerControlFlow.h"
#include "SILowerSGPRSpills.h"
@@ -675,6 +676,7 @@ extern "C" LLVM_ABI LLVM_EXTERNAL_VISIBILITY void LLVMInitializeAMDGPUTarget() {
initializeSIFixSGPRCopiesLegacyPass(*PR);
initializeSIFixVGPRCopiesLegacyPass(*PR);
initializeSIFoldOperandsLegacyPass(*PR);
+ initializeSIInsertWaterfallPass(*PR);
initializeSIPeepholeSDWALegacyPass(*PR);
initializeSIShrinkInstructionsLegacyPass(*PR);
initializeSIOptimizeExecMaskingPreRALegacyPass(*PR);
@@ -1736,6 +1738,7 @@ void GCNPassConfig::addFastRegAlloc() {
}
void GCNPassConfig::addPreRegAlloc() {
+ addPass(createSIInsertWaterfallPass());
if (getOptLevel() != CodeGenOptLevel::None)
addPass(&AMDGPUPrepareAGPRAllocLegacyID);
}
@@ -2550,6 +2553,7 @@ Error AMDGPUCodeGenPassBuilder::addOptimizedRegAlloc(
}
void AMDGPUCodeGenPassBuilder::addPreRegAlloc(PassManagerWrapper &PMW) const {
+ addMachineFunctionPass(SIInsertWaterfallPass(), PMW);
if (getOptLevel() != CodeGenOptLevel::None)
addMachineFunctionPass(AMDGPUPrepareAGPRAllocPass(), PMW);
}
diff --git a/llvm/lib/Target/AMDGPU/CMakeLists.txt b/llvm/lib/Target/AMDGPU/CMakeLists.txt
index ae8f1c0fad5ba..632af3aadaf93 100644
--- a/llvm/lib/Target/AMDGPU/CMakeLists.txt
+++ b/llvm/lib/Target/AMDGPU/CMakeLists.txt
@@ -165,6 +165,7 @@ add_llvm_target(AMDGPUCodeGen
SIFrameLowering.cpp
SIInsertHardClauses.cpp
SIInsertWaitcnts.cpp
+ SIInsertWaterfall.cpp
SIInstrInfo.cpp
SIISelLowering.cpp
SILateBranchLowering.cpp
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index 86f2479490c29..278c6183bf555 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -11793,6 +11793,27 @@ SDValue SITargetLowering::LowerINTRINSIC_W_CHAIN(SDValue Op,
DAG.setNodeMemRefs(NewNode, {MemRef});
return SDValue(NewNode, 0);
}
+ case Intrinsic::amdgcn_waterfall_readfirstlane: {
+ if (!Op->getOperand(3)->isDivergent()) {
+ // If waterfall_readfirstlane is uniform, it can be removed
+ SDValue InChain = Op.getOperand(0);
+ SDValue OutChain = Op.getValue(1);
+ DAG.ReplaceAllUsesOfValueWith(OutChain, InChain);
+ DAG.ReplaceAllUsesOfValueWith(Op.getValue(0), Op.getOperand(3));
+ return SDValue();
+ }
+ return Op;
+ }
+ case Intrinsic::amdgcn_waterfall_begin: {
+ // If the index in a waterfall.begin is uniform, it can be removed
+ if (!Op->getOperand(3)->isDivergent()) {
+ SDValue InChain = Op.getOperand(0);
+ SDValue OutChain = Op.getValue(1);
+ DAG.ReplaceAllUsesOfValueWith(OutChain, InChain);
+ DAG.ReplaceAllUsesOfValueWith(Op.getValue(0), Op.getOperand(2));
+ }
+ return Op;
+ }
case Intrinsic::amdgcn_global_atomic_fmin_num:
case Intrinsic::amdgcn_global_atomic_fmax_num:
case Intrinsic::amdgcn_flat_atomic_fmin_num:
diff --git a/llvm/lib/Target/AMDGPU/SIInsertWaterfall.cpp b/llvm/lib/Target/AMDGPU/SIInsertWaterfall.cpp
new file mode 100644
index 0000000000000..f0f0662c1b28b
--- /dev/null
+++ b/llvm/lib/Target/AMDGPU/SIInsertWaterfall.cpp
@@ -0,0 +1,896 @@
+//===- SIInsertWaterfall.cpp - insert waterall loops at intrinsic markers -===//
+//
+// The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+/// \file
+/// Replace 3 intrinsics used to mark waterfall regions with actual waterfall
+/// loops. This is done at MachineIR level rather than LLVM-IR due to the use of
+/// exec mask in this operation.
+///
+//===----------------------------------------------------------------------===//
+
+#include "SIInsertWaterfall.h"
+#include "AMDGPU.h"
+#include "GCNSubtarget.h"
+#include "MCTargetDesc/AMDGPUMCTargetDesc.h"
+#include "SIInstrInfo.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "si-insert-waterfall"
+
+namespace {
+
+static unsigned getWFBeginSize(const unsigned Opcode) {
+ switch (Opcode) {
+ case AMDGPU::SI_WATERFALL_BEGIN_V1:
+ return 1;
+ case AMDGPU::SI_WATERFALL_BEGIN_V2:
+ return 2;
+ case AMDGPU::SI_WATERFALL_BEGIN_V4:
+ return 4;
+ case AMDGPU::SI_WATERFALL_BEGIN_V8:
+ return 8;
+ default:
+ break;
+ }
+
+ return 0; // Not SI_WATERFALL_BEGIN_*
+}
+
+static unsigned getWFRFLSize(const unsigned Opcode) {
+ switch (Opcode) {
+ case AMDGPU::SI_WATERFALL_READFIRSTLANE_V1:
+ return 1;
+ case AMDGPU::SI_WATERFALL_READFIRSTLANE_V2:
+ return 2;
+ case AMDGPU::SI_WATERFALL_READFIRSTLANE_V4:
+ return 4;
+ case AMDGPU::SI_WATERFALL_READFIRSTLANE_V8:
+ return 8;
+ default:
+ break;
+ }
+
+ return 0; // Not SI_WATERFALL_READFIRSTLANE_*
+}
+
+static unsigned getWFEndSize(const unsigned Opcode) {
+ switch (Opcode) {
+ case AMDGPU::SI_WATERFALL_END_V1:
+ return 1;
+ case AMDGPU::SI_WATERFALL_END_V2:
+ return 2;
+ case AMDGPU::SI_WATERFALL_END_V4:
+ return 4;
+ case AMDGPU::SI_WATERFALL_END_V8:
+ return 8;
+ default:
+ break;
+ }
+
+ return 0; // Not SI_WATERFALL_END_*
+}
+
+static unsigned getWFLastUseSize(const unsigned Opcode) {
+ switch (Opcode) {
+ case AMDGPU::SI_WATERFALL_LAST_USE_V1:
+ case AMDGPU::SI_WATERFALL_LAST_USE_V1_V:
+ return 1;
+ case AMDGPU::SI_WATERFALL_LAST_USE_V2:
+ case AMDGPU::SI_WATERFALL_LAST_USE_V2_V:
+ return 2;
+ case AMDGPU::SI_WATERFALL_LAST_USE_V4:
+ case AMDGPU::SI_WATERFALL_LAST_USE_V4_V:
+ return 4;
+ case AMDGPU::SI_WATERFALL_LAST_USE_V8:
+ case AMDGPU::SI_WATERFALL_LAST_USE_V8_V:
+ return 8;
+ default:
+ break;
+ }
+
+ return 0; // Not SI_WATERFALL_LAST_USE_*
+}
+
+static bool isWFLastUseVGPR(const unsigned Opcode) {
+ switch (Opcode) {
+ case AMDGPU::SI_WATERFALL_LAST_USE_V1_V:
+ case AMDGPU::SI_WATERFALL_LAST_USE_V2_V:
+ case AMDGPU::SI_WATERFALL_LAST_USE_V4_V:
+ case AMDGPU::SI_WATERFALL_LAST_USE_V8_V:
+ return true;
+ default:
+ break;
+ }
+
+ return false;
+}
+
+static bool isWFLoopEnd(const unsigned Opcode) {
+ return Opcode == AMDGPU::SI_WATERFALL_LOOP_END;
+}
+
+static void readFirstLaneReg(MachineBasicBlock &MBB, MachineRegisterInfo *MRI,
+ const SIRegisterInfo *RI, const SIInstrInfo *TII,
+ MachineBasicBlock::iterator &I, const DebugLoc &DL,
+ Register RFLReg, Register RFLSrcReg,
+ const MachineOperand &RFLSrcOp) {
+ auto RFLRegRC = MRI->getRegClass(RFLReg);
+ uint32_t RegSize = RI->getRegSizeInBits(*RFLRegRC) / 32;
+ assert(RI->hasVGPRs(MRI->getRegClass(RFLSrcReg)) &&
+ "unexpected uniform operand for readfirstlane");
+
+ if (RegSize == 1) {
+ MRI->constrainRegClass(RFLReg, &AMDGPU::SReg_32_XM0RegClass);
+ BuildMI(MBB, I, DL, TII->get(AMDGPU::V_READFIRSTLANE_B32), RFLReg)
+ .addReg(RFLSrcReg, getUndefRegState(RFLSrcOp.isUndef()),
+ RFLSrcOp.getSubReg());
+ } else {
+ SmallVector<Register, 8> TRegs;
+ for (unsigned i = 0; i < RegSize; ++i) {
+ Register TReg = MRI->createVirtualRegister(&AMDGPU::SReg_32_XM0RegClass);
+ BuildMI(MBB, I, DL, TII->get(AMDGPU::V_READFIRSTLANE_B32), TReg)
+ .addReg(RFLSrcReg, {}, RI->getSubRegFromChannel(i));
+ TRegs.push_back(TReg);
+ }
+ MachineInstrBuilder MIB =
+ BuildMI(MBB, I, DL, TII->get(AMDGPU::REG_SEQUENCE), RFLReg);
+ for (unsigned i = 0; i < RegSize; ++i) {
+ MIB.addReg(TRegs[i]);
+ MIB.addImm(RI->getSubRegFromChannel(i));
+ }
+ }
+}
+
+// Check if operand is uniform by checking:
+// 1. Trivially detectable as operand in SGPR
+// 2. Direct def is from an SGPR->VGPR copy (which may happen if assumed
+// non-uniform value
+// turns out to be uniform)
+static Register getUniformOperandReplacementReg(MachineRegisterInfo *MRI,
+ const SIRegisterInfo *RI,
+ Register Reg) {
+ auto RegRC = MRI->getRegClass(Reg);
+ if (!RI->hasVGPRs(RegRC)) {
+ return Reg;
+ }
+
+ // Check for operand def being a copy from SGPR
+ MachineInstr *DefMI = MRI->getVRegDef(Reg);
+ if (DefMI->isFullCopy()) {
+ auto const &DefSrcOp = DefMI->getOperand(1);
+ if (DefSrcOp.isReg() && DefSrcOp.getReg().isVirtual()) {
+ Register ReplaceReg = DefSrcOp.getReg();
+ if (!RI->hasVGPRs(MRI->getRegClass(ReplaceReg)))
+ return ReplaceReg;
+ }
+ }
+ return AMDGPU::NoRegister;
+}
+
+static void compareIdxUsingCmpx(
+ MachineBasicBlock &MBB, MachineRegisterInfo *MRI, const SIRegisterInfo *RI,
+ const SIInstrInfo *TII, MachineBasicBlock::iterator &I, const DebugLoc &DL,
+ Register CurrentIdxReg, const MachineOperand &IndexOp, bool IsWave32) {
+
+ unsigned CmpxEqU32Opc = (IsWave32 ? AMDGPU::V_CMPX_EQ_U32_nosdst_e32
+ : AMDGPU::V_CMPX_EQ_U32_nosdst_e64);
+
+ Register IndexReg = IndexOp.getReg();
+ const TargetRegisterClass *IndexRC =
+ RI->getRegClassForOperandReg(*MRI, IndexOp);
+
+ // Iterate over the index in dword chunks
+ uint32_t RegSize = RI->getRegSizeInBits(*IndexRC) / 32;
+ if (RegSize == 1) {
+ BuildMI(MBB, I, DL, TII->get(CmpxEqU32Opc))
+ .addReg(CurrentIdxReg)
+ .addReg(IndexReg, {}, IndexOp.getSubReg());
+
+ } else {
+ for (unsigned Idx = 0; Idx < RegSize; ++Idx) {
+ BuildMI(MBB, I, DL, TII->get(CmpxEqU32Opc))
+ .addReg(CurrentIdxReg, {}, RI->getSubRegFromChannel(Idx))
+ .addReg(IndexReg, {}, RI->getSubRegFromChannel(Idx));
+ }
+ }
+}
+
+static Register compareIdx(MachineBasicBlock &MBB, MachineRegisterInfo *MRI,
+ const SIRegisterInfo *RI, const SIInstrInfo *TII,
+ MachineBasicBlock::iterator &I, const DebugLoc &DL,
+ Register CurrentIdxReg,
+ const MachineOperand &IndexOp, Register CondReg,
+ bool IsWave32) {
+ // Iterate over the index in dword chunks and'ing the result with the
+ // CondReg
+ // Optionally CondReg is passed in from a previous compareIdx call
+ Register IndexReg = IndexOp.getReg();
+ auto IndexRC = RI->getRegClassForOperandReg(*MRI, IndexOp);
+ unsigned AndOpc = IsWave32 ? AMDGPU::S_AND_B32 : AMDGPU::S_AND_B64;
+ const auto *BoolXExecRC = TII->getRegisterInfo().getWaveMaskRegClass();
+
+ uint32_t RegSize = RI->getRegSizeInBits(*IndexRC) / 32;
+
+ if (RegSize == 1) {
+ Register TReg = MRI->createVirtualRegister(BoolXExecRC);
+ BuildMI(MBB, I, DL, TII->get(AMDGPU::V_CMP_EQ_U32_e64), TReg)
+ .addReg(CurrentIdxReg)
+ .addReg(IndexReg, {}, IndexOp.getSubReg());
+
+ if (CondReg != AMDGPU::NoRegister) {
+ Register TReg2 = MRI->createVirtualRegister(BoolXExecRC);
+ BuildMI(MBB, I, DL, TII->get(AndOpc), TReg2).addReg(CondReg).addReg(TReg);
+ CondReg = TReg2;
+ } else {
+ CondReg = TReg;
+ }
+ } else {
+ unsigned StartCount;
+ Register TReg;
+ if (CondReg != AMDGPU::NoRegister) {
+ TReg = CondReg;
+ StartCount = 0;
+ } else {
+ TReg = MRI->createVirtualRegister(BoolXExecRC);
+ BuildMI(MBB, I, DL, TII->get(AMDGPU::V_CMP_EQ_U32_e64), TReg)
+ .addReg(CurrentIdxReg, {}, AMDGPU::sub0)
+ .addReg(IndexReg, {}, AMDGPU::sub0);
+ StartCount = 1;
+ }
+
+ for (unsigned i = StartCount; i < RegSize; ++i) {
+ Register TReg2 = MRI->createVirtualRegister(BoolXExecRC);
+ BuildMI(MBB, I, DL, TII->get(AMDGPU::V_CMP_EQ_U32_e64), TReg2)
+ .addReg(CurrentIdxReg, {}, RI->getSubRegFromChannel(i))
+ .addReg(IndexReg, {}, RI->getSubRegFromChannel(i));
+ Register TReg3 = MRI->createVirtualRegister(BoolXExecRC);
+ BuildMI(MBB, I, DL, TII->get(AndOpc), TReg3).addReg(TReg).addReg(TReg2);
+ TReg = TReg3;
+ }
+ CondReg = TReg;
+ }
+ return CondReg;
+}
+
+// Replace all registers From with To.
+// Also handles From and To MachineOperands having sub registers.
+// Note: MRI->replaceRegWith doesn't handle sub registers since it is
+// register based and subreg is carried on the operand.
+static void replaceRegIncSubReg(const MachineRegisterInfo *MRI,
+ const TargetRegisterInfo *TRI,
+ const MachineOperand *From,
+ const MachineOperand *To) {
+ for (auto &O : make_early_inc_range(MRI->reg_operands(From->getReg())))
+ O.substVirtReg(To->getReg(), To->getSubReg(), *TRI);
+}
+
+class SIInsertWaterfall : public MachineFunctionPass {
+private:
+ struct WaterfallWorkitem {
+ const SIInstrInfo *TII;
+ const MachineRegisterInfo *MRI;
+ Register TokReg; // This is always the token from the last begin intrinsic
+ MachineInstr *Final;
+ bool hasVGPRLastUse;
+
+ std::vector<MachineInstr *> BeginList;
+ std::vector<MachineInstr *> RFLList;
+ std::vector<MachineInstr *> EndList;
+ std::vector<MachineInstr *> LastUseList;
+ std::vector<MachineInstr *> LoopEndList;
+
+ // List of corresponding init, newdst and phi registers used in loop for
+ // end pseudos
+ std::vector<std::pair<MachineOperand *, MachineOperand *>> EndRegs;
+ std::vector<Register> RFLRegs;
+
+ WaterfallWorkitem() = default;
+ WaterfallWorkitem(MachineInstr *_Begin, const SIInstrInfo *_TII,
+ MachineRegisterInfo *_MRI)
+ : TII(_TII), MRI(_MRI), Final(nullptr), hasVGPRLastUse(false) {
+
+ auto TokMO = TII->getNamedOperand(*_Begin, AMDGPU::OpName::tok_ret);
+
+ assert(tokIsStart(TII->getNamedOperand(*_Begin, AMDGPU::OpName::tok)) &&
+ "first begin does not have an undefined input token as expected");
+ assert(TokMO &&
+ "Unable to extract tok operand from SI_WATERFALL_BEGIN pseudo op");
+
+ BeginList.push_back(_Begin);
+ TokReg = TokMO->getReg();
+ }
+
+ WaterfallWorkitem(const SIInstrInfo *_TII, MachineRegisterInfo *_MRI)
+ : TII(_TII), MRI(_MRI), TokReg(AMDGPU::NoRegister), Final(nullptr) {}
+
+ void processCandidate(MachineInstr *Cand) {
+ unsigned Opcode = Cand->getOpcode();
+ // Trivially end any waterfall intrinsic instructions
+ if (getWFBeginSize(Opcode) || getWFRFLSize(Opcode) ||
+ getWFEndSize(Opcode) || getWFLastUseSize(Opcode) ||
+ isWFLoopEnd(Opcode)) {
+ // TODO: A new waterfall clause shouldn't overlap with any uses
+ // tagged by a last_use intrinsic
+ return;
+ }
+
+ // Iterate over the LastUseList to determine if this instruction has a
+ // later use of a tagged last_use
+ for (auto Use : LastUseList) {
+ auto UseMO = TII->getNamedOperand(*Use, AMDGPU::OpName::dst);
+ Register UseReg = UseMO->getReg();
+
+ if (Cand->findRegisterUseOperand(UseReg, /*TRI=*/nullptr))
+ Final = Cand;
+ }
+ }
+
+ MachineInstr *getDefInstr(const MachineOperand *MO) const {
+ if (MO->isReg() && MRI->hasOneDef(MO->getReg())) {
+ return (*MRI->def_begin(MO->getReg())).getParent();
+ }
+ return nullptr;
+ }
+
+ bool tokIsStart(const MachineOperand *MO) const {
+ MachineInstr *defInstr = getDefInstr(MO);
+ if (defInstr && defInstr->getOpcode() == AMDGPU::S_MOV_B32) {
+ auto CopySrcOp = TII->getNamedOperand(*defInstr, AMDGPU::OpName::src0);
+ if (CopySrcOp && CopySrcOp->isImm()) {
+ if (CopySrcOp->getImm() == 0)
+ return true;
+ }
+ }
+ return false;
+ }
+
+ bool addCandidate(MachineInstr *Cand) {
+ unsigned Opcode = Cand->getOpcode();
+
+ assert((getWFBeginSize(Opcode) || getWFRFLSize(Opcode) ||
+ getWFEndSize(Opcode) || getWFLastUseSize(Opcode) ||
+ isWFLoopEnd(Opcode)) &&
+ "expected a waterfall instruction in addCandidate");
+
+ auto CandTokMO = TII->getNamedOperand(*Cand, AMDGPU::OpName::tok);
+ // There are a couple of scenarios at this point:
+ // 1. Standard - there's already been a begin that's been processed and
+ // set up the WaterfallWorkItem. In which case the token is
+ // valid and needs to be checked to ensure well- formed
+ // waterfall groups.
+ // 2. Begins removed - begins were uniform - and all of them have been
+ // removed. Need to process the rest of the instructions
+ // in the group, and verify that they have a undefined
+ // token
+ if (TokReg == AMDGPU::NoRegister) {
+ // All begins have been removed - continue to process the rest of the
+ // grouping ready for them to be removed in the next stage
+ assert(!getWFBeginSize(Opcode) &&
+ "unexpected begin instruction for addCandidate");
+ assert(tokIsStart(CandTokMO) &&
+ "waterfall group with no begin doesn't have undef tok input");
+
+ TokReg = CandTokMO->getReg();
+ }
+ if (CandTokMO->getReg() == TokReg) {
+ if (getWFBeginSize(Opcode)) {
+ auto TokRetMO = TII->getNamedOperand(*Cand, AMDGPU::OpName::tok_ret);
+ assert(TokRetMO && "Unable to extract tok_ret operand from "
+ "SI_WATERFALL_BEGIN pseudo op");
+ BeginList.push_back(Cand);
+ TokReg = TokRetMO->getReg();
+ return true;
+ } else if (getWFRFLSize(Opcode)) {
+ RFLList.push_back(Cand);
+ return true;
+ } else if (getWFEndSize(Opcode)) {
+ EndList.push_back(Cand);
+ Final = Cand;
+ return true;
+ } else if (getWFLastUseSize(Opcode)) {
+ LastUseList.push_back(Cand);
+ if (isWFLastUseVGPR(Opcode))
+ hasVGPRLastUse = true;
+ return true;
+ } else if (isWFLoopEnd(Opcode)) {
+ LoopEndList.push_back(Cand);
+ return true;
+ } else {
+ report_fatal_error("Unknown opcode, expected waterfall intrinsic");
+ }
+ }
+ LLVM_DEBUG(dbgs() << "malformed waterfall instruction group");
+ return false;
+ }
+
+ void eraseFromParent() {
+ for (auto BeginMI : BeginList)
+ BeginMI->eraseFromParent();
+ for (auto RFLMI : RFLList)
+ RFLMI->eraseFromParent();
+ for (auto EndMI : EndList)
+ EndMI->eraseFromParent();
+ for (auto LUMI : LastUseList)
+ LUMI->eraseFromParent();
+ for (auto LEMI : LoopEndList)
+ LEMI->eraseFromParent();
+ }
+ };
+
+ std::vector<WaterfallWorkitem> Worklist;
+
+ const GCNSubtarget *ST;
+ const SIInstrInfo *TII;
+ MachineRegisterInfo *MRI;
+ const SIRegisterInfo *RI;
+
+public:
+ static char ID;
+
+ SIInsertWaterfall() : MachineFunctionPass(ID) {
+ initializeSIInsertWaterfallPass(*PassRegistry::getPassRegistry());
+ }
+
+ void getAnalysisUsage(AnalysisUsage &AU) const override {
+ MachineFunctionPass::getAnalysisUsage(AU);
+ }
+
+ bool removeRedundantWaterfall(WaterfallWorkitem &Item);
+ bool processWaterfall(MachineBasicBlock &MBB);
+
+ Register getToken(MachineInstr *MI);
+
+ bool runOnMachineFunction(MachineFunction &MF) override;
+};
+
+} // End anonymous namespace.
+
+INITIALIZE_PASS(SIInsertWaterfall, DEBUG_TYPE, "SI Insert waterfalls", false,
+ false)
+
+char SIInsertWaterfall::ID = 0;
+
+char &llvm::SIInsertWaterfallID = SIInsertWaterfall::ID;
+
+FunctionPass *llvm::createSIInsertWaterfallPass() {
+ return new SIInsertWaterfall;
+}
+
+PreservedAnalyses
+SIInsertWaterfallPass::run(MachineFunction &MF,
+ MachineFunctionAnalysisManager &MFAM) {
+ SIInsertWaterfall Impl;
+ if (!Impl.runOnMachineFunction(MF))
+ return PreservedAnalyses::all();
+ return PreservedAnalyses::none();
+}
+
+bool SIInsertWaterfall::removeRedundantWaterfall(WaterfallWorkitem &Item) {
+ // In some cases, the waterfall is actually redundant
+ // If all the readfirstlane intrinsics are actually for uniform values and
+ // the token used in the begin/end isn't used in anything else the waterfall
+ // can be removed.
+ // Alternatively, prior passes may have removed the readfirstlane intrinsics
+ // altogether, in this case the begin/end intrinsics are now redundant and can
+ // also be removed.
+ // The readfirstlane intrinsics are replaced with the uniform source value,
+ // the loop is removed and the defs in the end intrinsics are just replaced
+ // with the input operands
+ // We can also have cases where the begins are all removed (all the indices
+ // were actually uniform).
+
+ // First step is to identify any readfirstlane intrinsics that are actually
+ // uniform - unless there are no begin instructions, in which case we always
+ // remove
+ bool LoopRemoved = false;
+ unsigned Removed = 0;
+ std::vector<MachineInstr *> NewRFLList;
+ std::vector<MachineInstr *> ToRemoveRFLList;
+
+ for (auto RFLMI : Item.RFLList) {
+ auto RFLSrcOp = TII->getNamedOperand(*RFLMI, AMDGPU::OpName::src);
+ auto RFLDstOp = TII->getNamedOperand(*RFLMI, AMDGPU::OpName::dst);
+ Register RFLSrcReg = RFLSrcOp->getReg();
+ Register RFLDstReg = RFLDstOp->getReg();
+
+ Register ReplaceReg = getUniformOperandReplacementReg(MRI, RI, RFLSrcReg);
+ if (ReplaceReg != AMDGPU::NoRegister) {
+ MRI->replaceRegWith(RFLDstReg, ReplaceReg);
+ Removed++;
+ ToRemoveRFLList.push_back(RFLMI);
+ } else if (RFLDstOp->isDead()) {
+ Removed++;
+ ToRemoveRFLList.push_back(RFLMI);
+ } else {
+ NewRFLList.push_back(RFLMI);
+ }
+ }
+
+ // Note: this test also returns true when there are NO RFL intrinsics, the
+ // case where a prior pass has removed all of them and the loop is now
+ // redundant
+ // Also check for the special case where there are no RFL intrinsics, but
+ // there are some last.use with VGPR uses.
+ if (!Item.BeginList.size() ||
+ (Removed == Item.RFLList.size() && !Item.hasVGPRLastUse)) {
+ // Removed all of the RFLs
+ // We can remove the waterfall loop entirely
+
+ // Protocol is to replace all dst operands for the waterfall_end intrinsics
+ // with their src operands. Replace all last_use dst operands with their src
+ // operands We don't need to check that the loop index isn't used anywhere
+ // as the protocol for waterfall intrinsics is to only use the begin index
+ // via a readfirstlane intrinsic anyway (which should also be removed) Any
+ // problems due to errors in this pass around loop removal will be picked up
+ // later by e.g. use before def errors
+ LLVM_DEBUG(
+ dbgs()
+ << "detected case for waterfall loop removal - already all uniform\n");
+ for (auto EndMI : Item.EndList) {
+ auto EndDstOp = TII->getNamedOperand(*EndMI, AMDGPU::OpName::dst);
+ auto EndSrcOp = TII->getNamedOperand(*EndMI, AMDGPU::OpName::src);
+ replaceRegIncSubReg(MRI, RI, EndDstOp, EndSrcOp);
+ }
+ for (auto LUMI : Item.LastUseList) {
+ auto LUDstOp = TII->getNamedOperand(*LUMI, AMDGPU::OpName::dst);
+ auto LUSrcOp = TII->getNamedOperand(*LUMI, AMDGPU::OpName::src);
+ replaceRegIncSubReg(MRI, RI, LUDstOp, LUSrcOp);
+ }
+ // If all the begins were removed, we have to replace the RFL with actual
+ // RFL, these will show up in the NewRLFList
+ for (auto RFLMI : NewRFLList) {
+ auto DstReg = TII->getNamedOperand(*RFLMI, AMDGPU::OpName::dst)->getReg();
+ auto SrcOp = TII->getNamedOperand(*RFLMI, AMDGPU::OpName::src);
+ Register SrcReg = SrcOp->getReg();
+
+ MachineBasicBlock::iterator RFLInsert(RFLMI);
+ readFirstLaneReg(*RFLMI->getParent(), MRI, RI, TII, RFLInsert,
+ RFLMI->getDebugLoc(), DstReg, SrcReg, *SrcOp);
+ }
+
+ Item.eraseFromParent();
+
+ LoopRemoved = true;
+
+ } else if (Removed) {
+ LLVM_DEBUG(dbgs() << "Removed " << Removed
+ << " waterfall rfl intrinsics due to being uniform - "
+ "updating remaining rfl list\n");
+ // TODO: there's an opportunity to pull the DAG involving the (removed) rfls
+ // out of the waterfall loop
+ Item.RFLList.clear();
+ std::copy(NewRFLList.begin(), NewRFLList.end(),
+ std::back_inserter(Item.RFLList));
+ for (auto RFLMI : ToRemoveRFLList)
+ RFLMI->eraseFromParent();
+ }
+
+ return LoopRemoved;
+}
+
+bool SIInsertWaterfall::processWaterfall(MachineBasicBlock &MBB) {
+ bool Changed = false;
+ MachineFunction &MF = *MBB.getParent();
+ MachineBasicBlock *CurrMBB = &MBB;
+
+ // Firstly we check that there are at least 3 related waterfall instructions
+ // for this begin
+ // SI_WATERFALL_BEGIN [ SI_WATERFALL_BEGIN ]*
+ // [ SI_WATERFALL_READFIRSTLANE ]+ [ SI_WATERFALL_END ]+ If there are multiple
+ // waterfall loops they must also be disjoint
+
+ for (WaterfallWorkitem &Item : Worklist) {
+ LLVM_DEBUG(
+ if (Item.BeginList.size()) {
+ dbgs() << "Processing " << *Item.BeginList[0] << "\n";
+
+ for (auto RUse = MRI->use_begin(Item.TokReg), RSE = MRI->use_end();
+ RUse != RSE; ++RUse) {
+ MachineInstr *RUseMI = RUse->getParent();
+ assert((CurrMBB->getNumber() == RUseMI->getParent()->getNumber()) &&
+ "Linked WATERFALL pseudo ops found in different BBs");
+ }
+ } else { dbgs() << "Processing redundant waterfall\n"; });
+
+ if (removeRedundantWaterfall(Item)) {
+ Changed = true;
+ continue;
+ }
+
+ assert((Item.RFLList.size() || Item.hasVGPRLastUse) &&
+ (Item.EndList.size() || Item.LastUseList.size()) &&
+ "SI_WATERFALL* pseudo instruction group must have at least 1 of "
+ "each type");
+
+ // Insert the waterfall loop code around the identified region of
+ // instructions
+ // Loop starts at the last SI_WATERFALL_BEGIN
+ // SI_WATERFALL_READFIRSTLANE is replaced with appropriate readfirstlane
+ // instructions OR is removed
+ // if the readfirstlane is using the same index as the SI_WATERFALL_BEGIN
+ // Loop is ended after the last SI_WATERFALL_END and these instructions are
+ // removed with the src replacing all dst uses
+ typedef struct {
+ const MachineOperand *Index;
+ const TargetRegisterClass *IndexRC;
+ const TargetRegisterClass *IndexSRC;
+ Register CurrentIdxReg;
+ } IdxInfo;
+
+ std::vector<IdxInfo> IndexList;
+#ifndef NDEBUG
+ bool IsUniform = true;
+#endif
+ for (auto BeginMI : Item.BeginList) {
+ IdxInfo CurrIdx;
+ CurrIdx.Index = TII->getNamedOperand(*(BeginMI), AMDGPU::OpName::idx);
+ CurrIdx.IndexRC = RI->getRegClassForOperandReg(*MRI, *CurrIdx.Index);
+ CurrIdx.IndexSRC = RI->getEquivalentSGPRClass(CurrIdx.IndexRC);
+ if (CurrIdx.IndexSRC == &AMDGPU::SGPR_32RegClass)
+ CurrIdx.IndexSRC = &AMDGPU::SReg_32_XM0RegClass;
+
+ IndexList.push_back(CurrIdx);
+
+ LLVM_DEBUG(if (RI->hasVGPRs(CurrIdx.IndexRC)) IsUniform = false;);
+ }
+
+ LLVM_DEBUG(if (IsUniform) {
+ // Waterfall loop index is uniform! Loop can be removed
+ // TODO:: Implement loop removal
+ dbgs() << "Uniform loop detected - waterfall loop is redundant\n";
+ });
+
+ MachineBasicBlock::iterator I(Item.BeginList.back());
+ const DebugLoc &DL = Item.BeginList[0]->getDebugLoc();
+
+ // Initialize the register we accumulate the result into, which is the
+ // target of any SI_WATERFALL_END instruction
+ for (auto EndMI : Item.EndList)
+ Item.EndRegs.emplace_back(
+ TII->getNamedOperand(*EndMI, AMDGPU::OpName::dst),
+ TII->getNamedOperand(*EndMI, AMDGPU::OpName::src));
+ for (auto LUMI : Item.LastUseList) {
+ auto LUSrc = TII->getNamedOperand(*LUMI, AMDGPU::OpName::src);
+ auto LUDst = TII->getNamedOperand(*LUMI, AMDGPU::OpName::dst);
+ replaceRegIncSubReg(MRI, RI, LUDst, LUSrc);
+ }
+
+ // EXEC mask handling
+ Register Exec = ST->isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
+ unsigned SaveExecOpc = ST->isWave32() ? AMDGPU::S_AND_SAVEEXEC_B32
+ : AMDGPU::S_AND_SAVEEXEC_B64;
+ unsigned XorTermOpc =
+ ST->isWave32() ? AMDGPU::S_XOR_B32_term : AMDGPU::S_XOR_B64_term;
+ unsigned MovOpc = ST->isWave32() ? AMDGPU::S_MOV_B32 : AMDGPU::S_MOV_B64;
+ unsigned AndNotWRExecOpc = ST->isWave32() ? AMDGPU::S_ANDN2_WREXEC_B32
+ : AMDGPU::S_ANDN2_WREXEC_B64;
+
+ // Emit [v_cmpx_eq] and [s_andn2_wwrexec] when these instructions are
+ // available.
+ // TODO: Accurately detect the availability of [s_andn2_wrexec] instruction
+ // in the target. For now, use the same condition as for the detection
+ // [v_cmpx_eq].
+ auto UseNewExecInstructions = ST->hasNoSdstCMPX();
+
+ const auto *BoolXExecRC = TII->getRegisterInfo().getWaveMaskRegClass();
+
+ MachineBasicBlock &LoopHeaderBB = *MF.CreateMachineBasicBlock();
+ MachineBasicBlock &LoopBB = *MF.CreateMachineBasicBlock();
+ MachineBasicBlock &RemainderBB = *MF.CreateMachineBasicBlock();
+ MachineFunction::iterator MBBI(*CurrMBB);
+ ++MBBI;
+
+ MF.insert(MBBI, &LoopHeaderBB);
+ MF.insert(MBBI, &LoopBB);
+ MF.insert(MBBI, &RemainderBB);
+
+ LoopHeaderBB.addSuccessor(&LoopBB);
+ LoopBB.addSuccessor(&LoopBB);
+ LoopBB.addSuccessor(&RemainderBB);
+
+ Register SaveExec = MRI->createVirtualRegister(BoolXExecRC);
+ Register TmpExec = MRI->createVirtualRegister(BoolXExecRC);
+
+ // Put TmpExec and SaveExec in the loop header.
+ MachineBasicBlock::iterator LH = LoopHeaderBB.begin();
+
+ if (UseNewExecInstructions) {
+ // Initialize TmpExec with the current EXEC mask.
+ // Represents remaining threads to process.
+ BuildMI(LoopHeaderBB, LH, DL, TII->get(MovOpc), TmpExec).addReg(Exec);
+ } else {
+ BuildMI(LoopHeaderBB, LH, DL, TII->get(TargetOpcode::IMPLICIT_DEF),
+ TmpExec);
+ }
+
+ // Save the EXEC mask
+ BuildMI(LoopHeaderBB, LH, DL, TII->get(MovOpc), SaveExec).addReg(Exec);
+
+ // Move all instructions from the SI_WATERFALL_BEGIN to the last
+ // SI_WATERFALL_END or last use tagged from SI_WATERFALL_LAST_USE
+ // into the new LoopBB
+ MachineBasicBlock::iterator SpliceE(Item.Final);
+ ++SpliceE;
+ LoopBB.splice(LoopBB.begin(), CurrMBB, I, SpliceE);
+
+ // Iterate over the instructions inserted into the loop
+ // Need to unset any kill flag on any uses as now this is a loop that is no
+ // longer valid
+ for (MachineInstr &MI : LoopBB)
+ MI.clearKillInfo();
+
+ RemainderBB.transferSuccessorsAndUpdatePHIs(CurrMBB);
+ RemainderBB.splice(RemainderBB.begin(), CurrMBB, SpliceE, CurrMBB->end());
+ MachineBasicBlock::iterator E(Item.Final);
+ ++E;
+
+ CurrMBB->addSuccessor(&LoopHeaderBB);
+
+ MachineBasicBlock::iterator J = LoopBB.begin();
+
+ Register PhiExec = MRI->createVirtualRegister(BoolXExecRC);
+ Register NewExec = MRI->createVirtualRegister(BoolXExecRC);
+
+ for (auto &CurrIdx : IndexList)
+ CurrIdx.CurrentIdxReg = MRI->createVirtualRegister(CurrIdx.IndexSRC);
+
+ BuildMI(LoopBB, J, DL, TII->get(TargetOpcode::PHI), PhiExec)
+ .addReg(TmpExec)
+ .addMBB(&LoopHeaderBB)
+ .addReg(NewExec)
+ .addMBB(&LoopBB);
+
+ // Get the next index to use from the first enabled lane
+ for (auto &CurrIdx : IndexList)
+ readFirstLaneReg(LoopBB, MRI, RI, TII, J, DL, CurrIdx.CurrentIdxReg,
+ CurrIdx.Index->getReg(), *CurrIdx.Index);
+
+ // Also process the readlane pseudo ops - if readfirstlane is using the
+ // index then just replace with the CurrentIdxReg instead
+ for (auto RFLMI : Item.RFLList) {
+ auto RFLSrcOp = TII->getNamedOperand(*RFLMI, AMDGPU::OpName::src);
+ auto RFLDstOp = TII->getNamedOperand(*RFLMI, AMDGPU::OpName::dst);
+ Register RFLSrcReg = RFLSrcOp->getReg();
+ Register RFLDstReg = RFLDstOp->getReg();
+
+ bool MatchedIdx = false;
+ for (auto &CurrIdx : IndexList) {
+ if (RFLSrcReg == CurrIdx.Index->getReg()) {
+ // Use the CurrentIdxReg for this
+ Item.RFLRegs.push_back(CurrIdx.CurrentIdxReg);
+ MRI->replaceRegWith(RFLDstReg, CurrIdx.CurrentIdxReg);
+ MatchedIdx = true;
+ break;
+ }
+ }
+ if (!MatchedIdx) {
+ Item.RFLRegs.push_back(RFLDstReg);
+ // Insert function to expand to required size here
+ MachineBasicBlock::iterator RFLInsert(RFLMI);
+ readFirstLaneReg(LoopBB, MRI, RI, TII, RFLInsert, DL, RFLDstReg,
+ RFLSrcReg, *RFLSrcOp);
+ }
+ }
+
+ // Compare the just read idx value to all possible idx values, and update
+ // EXEC
+ if (UseNewExecInstructions) {
+ for (auto &CurrIdx : IndexList)
+ compareIdxUsingCmpx(LoopBB, MRI, RI, TII, J, DL, CurrIdx.CurrentIdxReg,
+ *CurrIdx.Index, ST->isWave32());
+ MRI->setSimpleHint(NewExec, PhiExec);
+ } else {
+ Register CondReg = AMDGPU::NoRegister;
+ for (auto &CurrIdx : IndexList)
+ CondReg = compareIdx(LoopBB, MRI, RI, TII, J, DL, CurrIdx.CurrentIdxReg,
+ *CurrIdx.Index, CondReg, ST->isWave32());
+
+ // Update EXEC, save the original EXEC value to VCC
+ BuildMI(LoopBB, J, DL, TII->get(SaveExecOpc), NewExec)
+ .addReg(CondReg, RegState::Kill);
+
+ MRI->setSimpleHint(NewExec, CondReg);
+ }
+ // TODO: Conditional branch here to loop header as potential optimization?
+
+ // Copy the just read value into the destination
+ // Handle cases where sub registers are involved
+ for (auto EndReg : Item.EndRegs) {
+ MachineBasicBlock::iterator EndInsert(Item.Final);
+ BuildMI(LoopBB, EndInsert, DL, TII->get(AMDGPU::COPY))
+ .addReg(EndReg.first->getReg(), RegState::Define,
+ EndReg.first->getSubReg())
+ .addReg(EndReg.second->getReg(), {}, EndReg.second->getSubReg());
+ }
+
+ // Update EXEC, switch all done bits to 0 and all todo bits to 1.
+ if (UseNewExecInstructions) {
+ BuildMI(LoopBB, E, DL, TII->get(AndNotWRExecOpc), NewExec)
+ .addReg(PhiExec);
+ } else {
+ BuildMI(LoopBB, E, DL, TII->get(XorTermOpc), Exec)
+ .addReg(Exec)
+ .addReg(NewExec);
+ }
+
+ // Loop back if there are still variants to cover
+ BuildMI(LoopBB, E, DL, TII->get(AMDGPU::SI_WATERFALL_LOOP)).addMBB(&LoopBB);
+
+ MachineBasicBlock::iterator First = RemainderBB.begin();
+ BuildMI(RemainderBB, First, DL, TII->get(MovOpc), Exec).addReg(SaveExec);
+
+ Item.eraseFromParent();
+
+ // To process subsequent waterfall groups, update CurrMBB to the RemainderBB
+ CurrMBB = &RemainderBB;
+
+ Changed = true;
+ }
+ return Changed;
+}
+
+Register SIInsertWaterfall::getToken(MachineInstr *MI) {
+ auto CandTokMO = TII->getNamedOperand(*MI, AMDGPU::OpName::tok);
+ return CandTokMO->isReg() ? CandTokMO->getReg() : AMDGPU::NoRegister;
+}
+
+bool SIInsertWaterfall::runOnMachineFunction(MachineFunction &MF) {
+ bool Changed = false;
+
+ ST = &MF.getSubtarget<GCNSubtarget>();
+ TII = ST->getInstrInfo();
+ MRI = &MF.getRegInfo();
+ RI = ST->getRegisterInfo();
+
+ for (MachineBasicBlock &MBB : MF) {
+ Worklist.clear();
+ bool StartNew = true;
+
+ for (MachineInstr &MI : MBB) {
+ unsigned Opcode = MI.getOpcode();
+
+ if (getWFBeginSize(Opcode)) {
+ if (StartNew) {
+ Worklist.push_back(WaterfallWorkitem(&MI, TII, MRI));
+ StartNew = false;
+ } else {
+ if (!Worklist.back().addCandidate(&MI)) {
+ llvm_unreachable("Incorrect SI_WATERFALL_* groups");
+ }
+ }
+ } else if (getWFRFLSize(Opcode) || getWFEndSize(Opcode) ||
+ getWFLastUseSize(Opcode) || isWFLoopEnd(Opcode)) {
+ // On to the body of the group intrinsics,
+
+ // Tag StartNew as true if we encounter another begin
+ StartNew = true;
+
+ if (!Worklist.size() || getToken(&MI) != Worklist.back().TokReg) {
+ // There's no associated begin for these body intrinsics
+ // That means it's either an error - or all the begin intrinsics
+ // were removed due to being uniform
+ // Set up a WorkItem so we can process this correctly
+ Worklist.push_back(WaterfallWorkitem(TII, MRI));
+ }
+
+ if (!Worklist.back().addCandidate(&MI)) {
+ llvm_unreachable("Overlapping SI_WATERFALL_* groups");
+ }
+ } else {
+ if (Worklist.size())
+ Worklist.back().processCandidate(&MI);
+ }
+ }
+ Changed |= processWaterfall(MBB);
+ }
+
+ return Changed;
+}
diff --git a/llvm/lib/Target/AMDGPU/SIInsertWaterfall.h b/llvm/lib/Target/AMDGPU/SIInsertWaterfall.h
new file mode 100644
index 0000000000000..406df5a8b3b67
--- /dev/null
+++ b/llvm/lib/Target/AMDGPU/SIInsertWaterfall.h
@@ -0,0 +1,22 @@
+//===- SIInsertWaterfall.h --------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIB_TARGET_AMDGPU_SIINSERTWATERFALL_H
+#define LLVM_LIB_TARGET_AMDGPU_SIINSERTWATERFALL_H
+
+#include "llvm/CodeGen/MachinePassManager.h"
+
+namespace llvm {
+class SIInsertWaterfallPass : public PassInfoMixin<SIInsertWaterfallPass> {
+public:
+ PreservedAnalyses run(MachineFunction &MF,
+ MachineFunctionAnalysisManager &MFAM);
+};
+} // namespace llvm
+
+#endif // LLVM_LIB_TARGET_AMDGPU_SIINSERTWATERFALL_H
diff --git a/llvm/lib/Target/AMDGPU/SIInstructions.td b/llvm/lib/Target/AMDGPU/SIInstructions.td
index 148f15014b823..1c2e435f88ca9 100644
--- a/llvm/lib/Target/AMDGPU/SIInstructions.td
+++ b/llvm/lib/Target/AMDGPU/SIInstructions.td
@@ -1395,6 +1395,60 @@ def : GCNPat <
(COPY $src) // Return the SGPRs representing i1 src
>;
+// Waterfall pseudo ops - these are effectively specialized control flow pseudo
+// instructions
+// They are tagged as defining SCC since the loop that gets inserted will usually clobber it
+let UseNamedOperandTable = 1, Defs = [SCC] in {
+
+class SI_WATERFALL_BEGIN<RegisterClass idx_rc> : SPseudoInstSI <
+ (outs SReg_32:$tok_ret),
+ (ins SReg_32:$tok, idx_rc:$idx)>;
+
+class SI_WATERFALL_READFIRSTLANE<RegisterClass dst_rc, RegisterClass src_rc> : SPseudoInstSI <
+ (outs dst_rc:$dst),
+ (ins SReg_32:$tok, src_rc:$src)>;
+
+class SI_WATERFALL_END<RegisterClass rc> : SPseudoInstSI <
+ (outs rc:$dst),
+ (ins SReg_32:$tok, rc:$src)>;
+
+class SI_WATERFALL_LAST_USE<RegisterClass rc> : SPseudoInstSI <
+ (outs rc:$dst),
+ (ins SReg_32:$tok, rc:$src)>;
+
+} // End UseNamedOperandTable = 1
+
+def SI_WATERFALL_BEGIN_V1 : SI_WATERFALL_BEGIN<VGPR_32>;
+def SI_WATERFALL_BEGIN_V2 : SI_WATERFALL_BEGIN<VReg_64>;
+def SI_WATERFALL_BEGIN_V4 : SI_WATERFALL_BEGIN<VReg_128>;
+def SI_WATERFALL_BEGIN_V8 : SI_WATERFALL_BEGIN<VReg_256>;
+
+def SI_WATERFALL_READFIRSTLANE_V1 : SI_WATERFALL_READFIRSTLANE<SReg_32_XM0, VGPR_32>;
+def SI_WATERFALL_READFIRSTLANE_V2 : SI_WATERFALL_READFIRSTLANE<SReg_64, VReg_64>;
+def SI_WATERFALL_READFIRSTLANE_V4 : SI_WATERFALL_READFIRSTLANE<SGPR_128, VReg_128>;
+def SI_WATERFALL_READFIRSTLANE_V8 : SI_WATERFALL_READFIRSTLANE<SReg_256, VReg_256>;
+
+def SI_WATERFALL_END_V1 : SI_WATERFALL_END<VGPR_32>;
+def SI_WATERFALL_END_V2 : SI_WATERFALL_END<VReg_64>;
+def SI_WATERFALL_END_V4 : SI_WATERFALL_END<VReg_128>;
+def SI_WATERFALL_END_V8 : SI_WATERFALL_END<VReg_256>;
+
+def SI_WATERFALL_LAST_USE_V1 : SI_WATERFALL_LAST_USE<SGPR_32>;
+def SI_WATERFALL_LAST_USE_V1_V : SI_WATERFALL_LAST_USE<VGPR_32>;
+def SI_WATERFALL_LAST_USE_V2 : SI_WATERFALL_LAST_USE<SReg_64>;
+def SI_WATERFALL_LAST_USE_V2_V : SI_WATERFALL_LAST_USE<VReg_64>;
+def SI_WATERFALL_LAST_USE_V4 : SI_WATERFALL_LAST_USE<SGPR_128>;
+def SI_WATERFALL_LAST_USE_V4_V : SI_WATERFALL_LAST_USE<SReg_128>;
+def SI_WATERFALL_LAST_USE_V8 : SI_WATERFALL_LAST_USE<SReg_256>;
+def SI_WATERFALL_LAST_USE_V8_V : SI_WATERFALL_LAST_USE<VReg_256>;
+
+let Defs = [SCC],
+ UseNamedOperandTable = 1 in
+def SI_WATERFALL_LOOP_END : SPseudoInstSI <
+ (outs),
+ (ins SReg_32:$tok)
+>;
+
//===----------------------------------------------------------------------===//
// VOP1 Patterns
//===----------------------------------------------------------------------===//
@@ -2705,6 +2759,59 @@ def : GCNPat <
(V_RCP_IFLAG_F32_e32 (V_CVT_F32_U32_e32 $src0))))
>;
+multiclass SI_WATERFALL_Pattern<ValueType dvt, ValueType svt, string VecSize> {
+ def : GCNPat<(dvt(int_amdgcn_waterfall_readfirstlane i32:$tok, svt:$src)),
+ (!cast<Instruction>("SI_WATERFALL_READFIRSTLANE_"#VecSize)
+ i32:$tok,
+ svt:$src)>;
+}
+
+multiclass SI_WATERFALL_S_Pattern<list<ValueType> vts, string VecSize> {
+ foreach vt = vts in {
+ def : GCNPat<(i32(int_amdgcn_waterfall_begin i32:$tok, vt:$idx)),
+ (!cast<Instruction>("SI_WATERFALL_BEGIN_"#VecSize) i32:$tok,
+ vt:$idx)>;
+
+ def : GCNPat<(vt(int_amdgcn_waterfall_readfirstlane i32:$tok, vt:$src)),
+ (!cast<Instruction>("SI_WATERFALL_READFIRSTLANE_"#VecSize)
+ i32:$tok,
+ vt:$src)>;
+
+ def : GCNPat<(vt(int_amdgcn_waterfall_end i32:$tok, vt:$src)),
+ (!cast<Instruction>("SI_WATERFALL_END_"#VecSize) i32:$tok,
+ vt:$src)>;
+
+ def : GCNPat<(vt(int_amdgcn_waterfall_last_use i32:$tok, vt:$src)),
+ (!cast<Instruction>("SI_WATERFALL_LAST_USE_"#VecSize) i32:$tok,
+ vt:$src)>;
+
+ def : GCNPat<(vt(int_amdgcn_waterfall_last_use_vgpr i32:$tok, vt:$src)),
+ (!cast<Instruction>("SI_WATERFALL_LAST_USE_"#VecSize#"_V")
+ i32:$tok,
+ vt:$src)>;
+ }
+}
+
+defm : SI_WATERFALL_S_Pattern<Reg16Types.types, "V1">;
+defm : SI_WATERFALL_S_Pattern<Reg32Types.types, "V1">;
+defm : SI_WATERFALL_S_Pattern<Reg64Types.types, "V2">;
+defm : SI_WATERFALL_S_Pattern<Reg128Types.types, "V4">;
+defm : SI_WATERFALL_S_Pattern<Reg256Types.types, "V8">;
+
+defm : SI_WATERFALL_Pattern <i16, f16, "V1">;
+defm : SI_WATERFALL_Pattern <v2i16, v2f16, "V1">;
+defm : SI_WATERFALL_Pattern <v4i16, v4f16, "V2">;
+
+defm : SI_WATERFALL_Pattern <i32, f32, "V1">;
+defm : SI_WATERFALL_Pattern <v2i32, v2f32, "V2">;
+defm : SI_WATERFALL_Pattern <v4i32, v4f32, "V4">;
+defm : SI_WATERFALL_Pattern <v8i32, v8f32, "V8">;
+
+def : GCNPat<
+ (int_amdgcn_waterfall_loop_end i32:$tok),
+ (SI_WATERFALL_LOOP_END i32:$tok)
+>;
+
//===----------------------------------------------------------------------===//
// VOP3 Patterns
//===----------------------------------------------------------------------===//
diff --git a/llvm/test/Analysis/UniformityAnalysis/AMDGPU/MIR/llvm.amdgcn.waterfall.mir b/llvm/test/Analysis/UniformityAnalysis/AMDGPU/MIR/llvm.amdgcn.waterfall.mir
new file mode 100644
index 0000000000000..930d926b76b79
--- /dev/null
+++ b/llvm/test/Analysis/UniformityAnalysis/AMDGPU/MIR/llvm.amdgcn.waterfall.mir
@@ -0,0 +1,51 @@
+# RUN: llc -mtriple=amdgcn-- -mcpu=gfx1010 -run-pass=print-machine-uniformity -o - %s 2>&1 | FileCheck %s
+
+# CHECK-LABEL: MachineUniformityInfo for function: @test_waterfall_readlane
+# CHECK-NOT: DIVERGENT: %0: %0:_(s32) = COPY $sgpr0
+# CHECK-NOT: DIVERGENT: %1: %1:_(s32) = COPY $sgpr1
+# CHECK-NOT: DIVERGENT: %2: %2:_(p1) = G_MERGE_VALUES %0:_(s32), %1:_(s32)
+# CHECK-NOT: DIVERGENT: %3: %3:_(s32) = COPY $sgpr2
+# CHECK-NOT: DIVERGENT: %4: %4:_(s32) = COPY $sgpr3
+# CHECK-NOT: DIVERGENT: %5: %5:_(p1) = G_MERGE_VALUES %3:_(s32), %4:_(s32)
+# CHECK: DIVERGENT: %6: %6:_(s32) = COPY $vgpr0
+# CHECK: DIVERGENT: %7: %7:_(s64) = G_SEXT %6:_(s32)
+# CHECK-NOT: DIVERGENT: %8: %8:_(s32) = G_CONSTANT i32 3
+# CHECK: DIVERGENT: %9: %9:_(s64) = G_SHL %7:_, %8:_(s32)
+# CHECK: DIVERGENT: %10: %10:_(p1) = G_PTR_ADD %5:_, %9:_(s64)
+# CHECK: DIVERGENT: %11: %11:_(<2 x s32>) = G_LOAD %10:_(p1) :: (load (<2 x s32>), addrspace 1)
+# CHECK-NOT: DIVERGENT: %12: %12:_(s32) = G_CONSTANT i32 0
+# CHECK: DIVERGENT: %13: %13:_(s32), %14:_(s32) = G_UNMERGE_VALUES %11:_(<2 x s32>)
+# CHECK: DIVERGENT: %14: %13:_(s32), %14:_(s32) = G_UNMERGE_VALUES %11:_(<2 x s32>)
+# CHECK-NOT: DIVERGENT: %15: %15:_(s32) = G_INTRINSIC_CONVERGENT_W_SIDE_EFFECTS intrinsic(@llvm.amdgcn.waterfall.begin), %12:_(s32), %14:_(s32)
+# CHECK-NOT: DIVERGENT: %16: %16:_(s32) = G_INTRINSIC_CONVERGENT_W_SIDE_EFFECTS intrinsic(@llvm.amdgcn.waterfall.readfirstlane), %15:_(s32), %14:_(s32)
+# CHECK-NOT: DIVERGENT: %17: %17:_(s32) = G_INTRINSIC_CONVERGENT intrinsic(@llvm.amdgcn.readlane), %13:_(s32), %16:_(s32)
+# CHECK: DIVERGENT: %18: %18:_(s32) = G_INTRINSIC_CONVERGENT_W_SIDE_EFFECTS intrinsic(@llvm.amdgcn.waterfall.end), %15:_(s32), %17:_(s32)
+
+---
+name: test_waterfall_readlane
+tracksRegLiveness: true
+body: |
+ bb.0:
+ liveins: $sgpr0, $sgpr1, $sgpr2, $sgpr3, $vgpr0
+
+ %3:_(s32) = COPY $sgpr0
+ %4:_(s32) = COPY $sgpr1
+ %0:_(p1) = G_MERGE_VALUES %3(s32), %4(s32)
+ %5:_(s32) = COPY $sgpr2
+ %6:_(s32) = COPY $sgpr3
+ %1:_(p1) = G_MERGE_VALUES %5(s32), %6(s32)
+ %2:_(s32) = COPY $vgpr0
+ %8:_(s64) = G_SEXT %2(s32)
+ %25:_(s32) = G_CONSTANT i32 3
+ %10:_(s64) = G_SHL %8, %25(s32)
+ %11:_(p1) = G_PTR_ADD %1, %10(s64)
+ %13:_(<2 x s32>) = G_LOAD %11(p1) :: (load (<2 x s32>), addrspace 1)
+ %15:_(s32) = G_CONSTANT i32 0
+ %23:_(s32), %24:_(s32) = G_UNMERGE_VALUES %13(<2 x s32>)
+ %18:_(s32) = G_INTRINSIC_CONVERGENT_W_SIDE_EFFECTS intrinsic(@llvm.amdgcn.waterfall.begin), %15(s32), %24(s32)
+ %19:_(s32) = G_INTRINSIC_CONVERGENT_W_SIDE_EFFECTS intrinsic(@llvm.amdgcn.waterfall.readfirstlane), %18(s32), %24(s32)
+ %20:_(s32) = G_INTRINSIC_CONVERGENT intrinsic(@llvm.amdgcn.readlane), %23(s32), %19(s32)
+ %21:_(s32) = G_INTRINSIC_CONVERGENT_W_SIDE_EFFECTS intrinsic(@llvm.amdgcn.waterfall.end), %18(s32), %20(s32)
+ G_STORE %21(s32), %0(p1) :: (store (s32), addrspace 1)
+ S_ENDPGM 0
+...
diff --git a/llvm/test/Analysis/UniformityAnalysis/AMDGPU/llvm.amdgcn.waterfall.ll b/llvm/test/Analysis/UniformityAnalysis/AMDGPU/llvm.amdgcn.waterfall.ll
new file mode 100644
index 0000000000000..7b934e85b68ab
--- /dev/null
+++ b/llvm/test/Analysis/UniformityAnalysis/AMDGPU/llvm.amdgcn.waterfall.ll
@@ -0,0 +1,29 @@
+; RUN: opt -mtriple amdgcn-- -passes='print<uniformity>' -disable-output %s 2>&1 | FileCheck %s
+
+; CHECK-LABEL: UniformityInfo for function 'test_waterfall_readlane':
+; CHECK: DIVERGENT: %gep.in = getelementptr <2 x i32>, ptr addrspace(1) %in, i32 %tid
+; CHECK: DIVERGENT: %args = load <2 x i32>, ptr addrspace(1) %gep.in, align 8
+; CHECK: DIVERGENT: %value = extractelement <2 x i32> %args, i32 0
+; CHECK: DIVERGENT: %lane = extractelement <2 x i32> %args, i32 1
+; CHECK-NOT: DIVERGENT: %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %lane)
+; CHECK-NOT: DIVERGENT: %readlane = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %lane)
+; CHECK-NOT: DIVERGENT: %readlane1 = call i32 @llvm.amdgcn.readlane(i32 %value, i32 %readlane)
+; CHECK: DIVERGENT: %readlane2 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %wf_token, i32 %readlane1)
+; CHECK: DIVERGENT: store i32 %readlane2, ptr addrspace(1) %out, align 4
+define amdgpu_ps void @test_waterfall_readlane(i32 addrspace(1)* inreg %out, <2 x i32> addrspace(1)* inreg %in, i32 %tid) #1 {
+ %gep.in = getelementptr <2 x i32>, <2 x i32> addrspace(1)* %in, i32 %tid
+ %args = load <2 x i32>, <2 x i32> addrspace(1)* %gep.in
+ %value = extractelement <2 x i32> %args, i32 0
+ %lane = extractelement <2 x i32> %args, i32 1
+ %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %lane)
+ %readlane = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %lane)
+ %readlane1 = call i32 @llvm.amdgcn.readlane(i32 %value, i32 %readlane)
+ %readlane2 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %wf_token, i32 %readlane1)
+ store i32 %readlane2, i32 addrspace(1)* %out, align 4
+ ret void
+}
+
+declare i32 @llvm.amdgcn.waterfall.begin.i32(i32, i32)
+declare i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32, i32)
+declare i32 @llvm.amdgcn.readlane(i32, i32)
+declare i32 @llvm.amdgcn.waterfall.end.i32(i32, i32)
diff --git a/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll b/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
new file mode 100644
index 0000000000000..2f63293aaa09c
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
@@ -0,0 +1,406 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX10 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX11 %s
+
+define dllexport amdgpu_cs void @atomic_add_in_wf(ptr addrspace(1) %arg, i32 inreg %arg1, ptr addrspace(4) inreg noundef %arg2) #0 {
+; GFX10-LABEL: atomic_add_in_wf:
+; GFX10: ; %bb.0: ; %bb
+; GFX10-NEXT: s_ashr_i32 s3, s0, 31
+; GFX10-NEXT: v_add_co_u32 v0, vcc, v0, s0
+; GFX10-NEXT: v_add_co_ci_u32_e32 v1, vcc, s3, v1, vcc
+; GFX10-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-NEXT: global_load_dword v0, v[0:1], off offset:4
+; GFX10-NEXT: .LBB0_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-NEXT: s_waitcnt vmcnt(0)
+; GFX10-NEXT: v_readfirstlane_b32 s0, v0
+; GFX10-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-NEXT: v_cmpx_eq_u32_e64 s0, v0
+; GFX10-NEXT: s_ashr_i32 s3, s0, 31
+; GFX10-NEXT: s_add_u32 s6, s1, s0
+; GFX10-NEXT: s_addc_u32 s7, s2, s3
+; GFX10-NEXT: v_mov_b32_e32 v0, 1
+; GFX10-NEXT: s_load_dwordx4 s[8:11], s[6:7], 0x0
+; GFX10-NEXT: v_mov_b32_e32 v1, 0
+; GFX10-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-NEXT: buffer_atomic_add v0, v1, s[8:11], 0 idxen glc
+; GFX10-NEXT: s_andn2_wrexec_b64 s[4:5], s[4:5]
+; GFX10-NEXT: ; implicit-def: $vgpr0
+; GFX10-NEXT: s_cbranch_execnz .LBB0_1
+; GFX10-NEXT: ; %bb.2:
+; GFX10-NEXT: s_endpgm
+;
+; GFX11-LABEL: atomic_add_in_wf:
+; GFX11: ; %bb.0: ; %bb
+; GFX11-NEXT: v_add_co_u32 v0, vcc, v0, s0
+; GFX11-NEXT: s_ashr_i32 s3, s0, 31
+; GFX11-NEXT: s_mov_b64 s[4:5], exec
+; GFX11-NEXT: s_mov_b64 s[6:7], exec
+; GFX11-NEXT: v_add_co_ci_u32_e64 v1, null, s3, v1, vcc
+; GFX11-NEXT: global_load_b32 v0, v[0:1], off offset:4
+; GFX11-NEXT: .LBB0_1: ; =>This Inner Loop Header: Depth=1
+; GFX11-NEXT: s_waitcnt vmcnt(0)
+; GFX11-NEXT: v_readfirstlane_b32 s0, v0
+; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT: v_cmpx_eq_u32_e64 s0, v0
+; GFX11-NEXT: s_ashr_i32 s3, s0, 31
+; GFX11-NEXT: s_add_u32 s6, s1, s0
+; GFX11-NEXT: s_addc_u32 s7, s2, s3
+; GFX11-NEXT: v_mov_b32_e32 v0, 1
+; GFX11-NEXT: s_load_b128 s[8:11], s[6:7], 0x0
+; GFX11-NEXT: v_mov_b32_e32 v1, 0
+; GFX11-NEXT: s_waitcnt lgkmcnt(0)
+; GFX11-NEXT: buffer_atomic_add_u32 v0, v1, s[8:11], 0 idxen glc
+; GFX11-NEXT: s_and_not1_wrexec_b64 s[4:5], s[4:5]
+; GFX11-NEXT: ; implicit-def: $vgpr0
+; GFX11-NEXT: s_cbranch_execnz .LBB0_1
+; GFX11-NEXT: ; %bb.2:
+; GFX11-NEXT: s_endpgm
+bb:
+ %getelementptr = getelementptr i8, ptr addrspace(1) %arg, i32 %arg1
+ %load = load <2 x i32>, ptr addrspace(1) %getelementptr, align 8
+ %extractelement = extractelement <2 x i32> %load, i32 0
+ %extractelement3 = extractelement <2 x i32> %load, i32 1
+ %call = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %extractelement3)
+ %call4 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %call, i32 %extractelement3)
+ %sext = sext i32 %call4 to i64
+ %getelementptr5 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext
+ %load6 = load <4 x i32>, ptr addrspace(4) %getelementptr5, align 4
+ %call7 = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %load6, i32 0, i32 0, i32 0, i32 0)
+ %call8 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %call, i32 %call7)
+ ret void
+}
+
+define dllexport amdgpu_cs void @atomic_add_before(ptr addrspace(1) %arg, i32 inreg %arg1, ptr addrspace(4) inreg noundef %arg2, i32 inreg %arg3) #0 {
+; GFX10-LABEL: atomic_add_before:
+; GFX10: ; %bb.0: ; %bb
+; GFX10-NEXT: s_ashr_i32 s6, s0, 31
+; GFX10-NEXT: v_add_co_u32 v0, vcc, v0, s0
+; GFX10-NEXT: v_add_co_ci_u32_e32 v1, vcc, s6, v1, vcc
+; GFX10-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-NEXT: global_load_dwordx2 v[0:1], v[0:1], off
+; GFX10-NEXT: s_waitcnt vmcnt(0)
+; GFX10-NEXT: v_mbcnt_lo_u32_b32 v0, s4, 0
+; GFX10-NEXT: v_mbcnt_hi_u32_b32 v0, s5, v0
+; GFX10-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX10-NEXT: s_and_saveexec_b64 s[6:7], vcc
+; GFX10-NEXT: s_cbranch_execz .LBB1_2
+; GFX10-NEXT: ; %bb.1:
+; GFX10-NEXT: s_ashr_i32 s0, s3, 31
+; GFX10-NEXT: s_add_u32 s12, s1, s3
+; GFX10-NEXT: s_addc_u32 s13, s2, s0
+; GFX10-NEXT: s_bcnt1_i32_b64 s0, s[4:5]
+; GFX10-NEXT: s_load_dwordx4 s[8:11], s[12:13], 0x0
+; GFX10-NEXT: v_mov_b32_e32 v0, s0
+; GFX10-NEXT: v_mov_b32_e32 v2, 0
+; GFX10-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-NEXT: buffer_atomic_add v0, v2, s[8:11], 0 idxen
+; GFX10-NEXT: .LBB1_2:
+; GFX10-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-NEXT: s_or_b64 exec, exec, s[6:7]
+; GFX10-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-NEXT: .LBB1_3: ; =>This Inner Loop Header: Depth=1
+; GFX10-NEXT: v_readfirstlane_b32 s0, v1
+; GFX10-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-NEXT: v_cmpx_eq_u32_e64 s0, v1
+; GFX10-NEXT: s_ashr_i32 s3, s0, 31
+; GFX10-NEXT: s_add_u32 s6, s1, s0
+; GFX10-NEXT: s_addc_u32 s7, s2, s3
+; GFX10-NEXT: s_waitcnt vmcnt(0)
+; GFX10-NEXT: v_mov_b32_e32 v0, 1
+; GFX10-NEXT: s_load_dwordx4 s[8:11], s[6:7], 0x0
+; GFX10-NEXT: v_mov_b32_e32 v1, 0
+; GFX10-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-NEXT: buffer_atomic_add v0, v1, s[8:11], 0 idxen glc
+; GFX10-NEXT: s_andn2_wrexec_b64 s[4:5], s[4:5]
+; GFX10-NEXT: ; implicit-def: $vgpr0_vgpr1
+; GFX10-NEXT: s_cbranch_execnz .LBB1_3
+; GFX10-NEXT: ; %bb.4:
+; GFX10-NEXT: s_endpgm
+;
+; GFX11-LABEL: atomic_add_before:
+; GFX11: ; %bb.0: ; %bb
+; GFX11-NEXT: v_add_co_u32 v0, vcc, v0, s0
+; GFX11-NEXT: s_ashr_i32 s6, s0, 31
+; GFX11-NEXT: s_mov_b64 s[4:5], exec
+; GFX11-NEXT: v_add_co_ci_u32_e64 v1, null, s6, v1, vcc
+; GFX11-NEXT: s_mov_b64 s[6:7], exec
+; GFX11-NEXT: global_load_b64 v[0:1], v[0:1], off
+; GFX11-NEXT: s_waitcnt vmcnt(0)
+; GFX11-NEXT: v_mbcnt_lo_u32_b32 v0, s4, 0
+; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX11-NEXT: v_mbcnt_hi_u32_b32 v0, s5, v0
+; GFX11-NEXT: v_cmpx_eq_u32_e32 0, v0
+; GFX11-NEXT: s_cbranch_execz .LBB1_2
+; GFX11-NEXT: ; %bb.1:
+; GFX11-NEXT: s_ashr_i32 s0, s3, 31
+; GFX11-NEXT: s_add_u32 s8, s1, s3
+; GFX11-NEXT: s_addc_u32 s9, s2, s0
+; GFX11-NEXT: s_bcnt1_i32_b64 s0, s[4:5]
+; GFX11-NEXT: s_load_b128 s[8:11], s[8:9], 0x0
+; GFX11-NEXT: v_mov_b32_e32 v0, s0
+; GFX11-NEXT: v_mov_b32_e32 v2, 0
+; GFX11-NEXT: s_waitcnt lgkmcnt(0)
+; GFX11-NEXT: buffer_atomic_add_u32 v0, v2, s[8:11], 0 idxen
+; GFX11-NEXT: .LBB1_2:
+; GFX11-NEXT: s_or_b64 exec, exec, s[6:7]
+; GFX11-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX11-NEXT: s_mov_b64 s[4:5], exec
+; GFX11-NEXT: s_mov_b64 s[6:7], exec
+; GFX11-NEXT: .LBB1_3: ; =>This Inner Loop Header: Depth=1
+; GFX11-NEXT: v_readfirstlane_b32 s0, v1
+; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT: v_cmpx_eq_u32_e64 s0, v1
+; GFX11-NEXT: s_ashr_i32 s3, s0, 31
+; GFX11-NEXT: s_add_u32 s6, s1, s0
+; GFX11-NEXT: s_addc_u32 s7, s2, s3
+; GFX11-NEXT: s_waitcnt vmcnt(0)
+; GFX11-NEXT: v_mov_b32_e32 v0, 1
+; GFX11-NEXT: s_load_b128 s[8:11], s[6:7], 0x0
+; GFX11-NEXT: v_mov_b32_e32 v1, 0
+; GFX11-NEXT: s_waitcnt lgkmcnt(0)
+; GFX11-NEXT: buffer_atomic_add_u32 v0, v1, s[8:11], 0 idxen glc
+; GFX11-NEXT: s_and_not1_wrexec_b64 s[4:5], s[4:5]
+; GFX11-NEXT: ; implicit-def: $vgpr0_vgpr1
+; GFX11-NEXT: s_cbranch_execnz .LBB1_3
+; GFX11-NEXT: ; %bb.4:
+; GFX11-NEXT: s_endpgm
+bb:
+ %getelementptr = getelementptr i8, ptr addrspace(1) %arg, i32 %arg1
+ %load = load <2 x i32>, ptr addrspace(1) %getelementptr, align 8
+ %extractelement = extractelement <2 x i32> %load, i32 0
+ %extractelement4 = extractelement <2 x i32> %load, i32 1
+ %sext = sext i32 %arg3 to i64
+ %getelementptr5 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext
+ %load6 = load <4 x i32>, ptr addrspace(4) %getelementptr5, align 4
+ %call = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %load6, i32 0, i32 0, i32 0, i32 0)
+ %call7 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %extractelement4)
+ %call8 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %call7, i32 %extractelement4)
+ %sext9 = sext i32 %call8 to i64
+ %getelementptr10 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext9
+ %load11 = load <4 x i32>, ptr addrspace(4) %getelementptr10, align 4
+ %call12 = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %load11, i32 0, i32 0, i32 0, i32 0)
+ %call13 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %call7, i32 %call12)
+ ret void
+}
+
+define dllexport amdgpu_cs void @atomic_add_after(ptr addrspace(1) %arg, i32 inreg %arg1, ptr addrspace(4) inreg noundef %arg2, i32 inreg %arg3) #0 {
+; GFX10-LABEL: atomic_add_after:
+; GFX10: ; %bb.0: ; %bb
+; GFX10-NEXT: s_ashr_i32 s4, s0, 31
+; GFX10-NEXT: v_add_co_u32 v0, vcc, v0, s0
+; GFX10-NEXT: v_add_co_ci_u32_e32 v1, vcc, s4, v1, vcc
+; GFX10-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-NEXT: global_load_dword v1, v[0:1], off offset:4
+; GFX10-NEXT: .LBB2_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-NEXT: s_waitcnt vmcnt(0)
+; GFX10-NEXT: v_readfirstlane_b32 s0, v1
+; GFX10-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-NEXT: v_cmpx_eq_u32_e64 s0, v1
+; GFX10-NEXT: s_ashr_i32 s8, s0, 31
+; GFX10-NEXT: s_add_u32 s12, s1, s0
+; GFX10-NEXT: s_addc_u32 s13, s2, s8
+; GFX10-NEXT: v_mov_b32_e32 v1, 1
+; GFX10-NEXT: s_load_dwordx4 s[8:11], s[12:13], 0x0
+; GFX10-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-NEXT: buffer_atomic_add v1, v0, s[8:11], 0 idxen glc
+; GFX10-NEXT: s_andn2_wrexec_b64 s[6:7], s[6:7]
+; GFX10-NEXT: ; implicit-def: $vgpr1
+; GFX10-NEXT: s_cbranch_execnz .LBB2_1
+; GFX10-NEXT: ; %bb.2:
+; GFX10-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-NEXT: s_mov_b64 exec, s[4:5]
+; GFX10-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-NEXT: s_waitcnt vmcnt(0)
+; GFX10-NEXT: v_mbcnt_lo_u32_b32 v1, s4, 0
+; GFX10-NEXT: v_mbcnt_hi_u32_b32 v1, s5, v1
+; GFX10-NEXT: v_cmp_eq_u32_e32 vcc, 0, v1
+; GFX10-NEXT: s_and_saveexec_b64 s[6:7], vcc
+; GFX10-NEXT: s_cbranch_execz .LBB2_4
+; GFX10-NEXT: ; %bb.3:
+; GFX10-NEXT: s_ashr_i32 s0, s3, 31
+; GFX10-NEXT: s_add_u32 s6, s1, s3
+; GFX10-NEXT: s_addc_u32 s7, s2, s0
+; GFX10-NEXT: s_bcnt1_i32_b64 s4, s[4:5]
+; GFX10-NEXT: s_load_dwordx4 s[0:3], s[6:7], 0x0
+; GFX10-NEXT: v_mov_b32_e32 v1, s4
+; GFX10-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-NEXT: buffer_atomic_add v1, v0, s[0:3], 0 idxen
+; GFX10-NEXT: .LBB2_4:
+; GFX10-NEXT: s_endpgm
+;
+; GFX11-LABEL: atomic_add_after:
+; GFX11: ; %bb.0: ; %bb
+; GFX11-NEXT: v_add_co_u32 v0, vcc, v0, s0
+; GFX11-NEXT: s_ashr_i32 s4, s0, 31
+; GFX11-NEXT: s_mov_b64 s[6:7], exec
+; GFX11-NEXT: v_add_co_ci_u32_e64 v1, null, s4, v1, vcc
+; GFX11-NEXT: s_mov_b64 s[4:5], exec
+; GFX11-NEXT: global_load_b32 v1, v[0:1], off offset:4
+; GFX11-NEXT: .LBB2_1: ; =>This Inner Loop Header: Depth=1
+; GFX11-NEXT: s_waitcnt vmcnt(0)
+; GFX11-NEXT: v_readfirstlane_b32 s0, v1
+; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT: v_cmpx_eq_u32_e64 s0, v1
+; GFX11-NEXT: s_ashr_i32 s9, s0, 31
+; GFX11-NEXT: s_add_u32 s8, s1, s0
+; GFX11-NEXT: s_addc_u32 s9, s2, s9
+; GFX11-NEXT: v_mov_b32_e32 v1, 1
+; GFX11-NEXT: s_load_b128 s[8:11], s[8:9], 0x0
+; GFX11-NEXT: v_mov_b32_e32 v0, 0
+; GFX11-NEXT: s_waitcnt lgkmcnt(0)
+; GFX11-NEXT: buffer_atomic_add_u32 v1, v0, s[8:11], 0 idxen glc
+; GFX11-NEXT: s_and_not1_wrexec_b64 s[6:7], s[6:7]
+; GFX11-NEXT: ; implicit-def: $vgpr1
+; GFX11-NEXT: s_cbranch_execnz .LBB2_1
+; GFX11-NEXT: ; %bb.2:
+; GFX11-NEXT: s_mov_b64 exec, s[4:5]
+; GFX11-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_1)
+; GFX11-NEXT: s_mov_b64 s[4:5], exec
+; GFX11-NEXT: s_mov_b64 s[6:7], exec
+; GFX11-NEXT: s_waitcnt vmcnt(0)
+; GFX11-NEXT: v_mbcnt_lo_u32_b32 v1, s4, 0
+; GFX11-NEXT: v_mbcnt_hi_u32_b32 v1, s5, v1
+; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT: v_cmpx_eq_u32_e32 0, v1
+; GFX11-NEXT: s_cbranch_execz .LBB2_4
+; GFX11-NEXT: ; %bb.3:
+; GFX11-NEXT: s_ashr_i32 s6, s3, 31
+; GFX11-NEXT: s_add_u32 s0, s1, s3
+; GFX11-NEXT: s_addc_u32 s1, s2, s6
+; GFX11-NEXT: s_bcnt1_i32_b64 s4, s[4:5]
+; GFX11-NEXT: s_load_b128 s[0:3], s[0:1], 0x0
+; GFX11-NEXT: v_mov_b32_e32 v1, s4
+; GFX11-NEXT: s_waitcnt lgkmcnt(0)
+; GFX11-NEXT: buffer_atomic_add_u32 v1, v0, s[0:3], 0 idxen
+; GFX11-NEXT: .LBB2_4:
+; GFX11-NEXT: s_endpgm
+bb:
+ %getelementptr = getelementptr i8, ptr addrspace(1) %arg, i32 %arg1
+ %load = load <2 x i32>, ptr addrspace(1) %getelementptr, align 8
+ %extractelement = extractelement <2 x i32> %load, i32 0
+ %extractelement4 = extractelement <2 x i32> %load, i32 1
+ %call7 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %extractelement4)
+ %call8 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %call7, i32 %extractelement4)
+ %sext9 = sext i32 %call8 to i64
+ %getelementptr10 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext9
+ %load11 = load <4 x i32>, ptr addrspace(4) %getelementptr10, align 4
+ %call12 = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %load11, i32 0, i32 0, i32 0, i32 0)
+ %call13 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %call7, i32 %call12)
+ %sext = sext i32 %arg3 to i64
+ %getelementptr5 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext
+ %load6 = load <4 x i32>, ptr addrspace(4) %getelementptr5, align 4
+ %call = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %load6, i32 0, i32 0, i32 0, i32 0)
+ ret void
+}
+
+define dllexport amdgpu_cs void @atomic_add_in_wf_partial(ptr addrspace(1) %arg, i32 inreg %arg1, ptr addrspace(4) inreg noundef %arg2, i32 inreg %arg3) #0 {
+; GFX10-LABEL: atomic_add_in_wf_partial:
+; GFX10: ; %bb.0: ; %bb
+; GFX10-NEXT: s_ashr_i32 s10, s0, 31
+; GFX10-NEXT: s_ashr_i32 s4, s3, 31
+; GFX10-NEXT: s_add_u32 s8, s1, s3
+; GFX10-NEXT: s_addc_u32 s9, s2, s4
+; GFX10-NEXT: v_add_co_u32 v0, vcc, v0, s0
+; GFX10-NEXT: s_load_dwordx4 s[4:7], s[8:9], 0x0
+; GFX10-NEXT: v_add_co_ci_u32_e32 v1, vcc, s10, v1, vcc
+; GFX10-NEXT: v_mov_b32_e32 v2, 1
+; GFX10-NEXT: v_mov_b32_e32 v3, 0
+; GFX10-NEXT: global_load_dwordx2 v[0:1], v[0:1], off
+; GFX10-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-NEXT: buffer_atomic_add v2, v3, s[4:7], 0 idxen
+; GFX10-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-NEXT: .LBB3_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-NEXT: s_waitcnt vmcnt(0)
+; GFX10-NEXT: v_readfirstlane_b32 s0, v0
+; GFX10-NEXT: v_readfirstlane_b32 s3, v1
+; GFX10-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-NEXT: v_cmpx_eq_u32_e64 s0, v0
+; GFX10-NEXT: v_cmpx_eq_u32_e64 s3, v1
+; GFX10-NEXT: s_ashr_i32 s3, s0, 31
+; GFX10-NEXT: s_add_u32 s6, s1, s0
+; GFX10-NEXT: s_addc_u32 s7, s2, s3
+; GFX10-NEXT: s_load_dwordx4 s[8:11], s[6:7], 0x0
+; GFX10-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-NEXT: buffer_atomic_add v2, v3, s[8:11], 0 idxen glc
+; GFX10-NEXT: s_andn2_wrexec_b64 s[4:5], s[4:5]
+; GFX10-NEXT: ; implicit-def: $vgpr0_vgpr1
+; GFX10-NEXT: ; implicit-def: $vgpr2
+; GFX10-NEXT: ; implicit-def: $vgpr3
+; GFX10-NEXT: s_cbranch_execnz .LBB3_1
+; GFX10-NEXT: ; %bb.2:
+; GFX10-NEXT: s_endpgm
+;
+; GFX11-LABEL: atomic_add_in_wf_partial:
+; GFX11: ; %bb.0: ; %bb
+; GFX11-NEXT: s_ashr_i32 s8, s0, 31
+; GFX11-NEXT: s_ashr_i32 s5, s3, 31
+; GFX11-NEXT: s_add_u32 s4, s1, s3
+; GFX11-NEXT: s_addc_u32 s5, s2, s5
+; GFX11-NEXT: v_add_co_u32 v0, vcc, v0, s0
+; GFX11-NEXT: s_load_b128 s[4:7], s[4:5], 0x0
+; GFX11-NEXT: v_mov_b32_e32 v2, 1
+; GFX11-NEXT: v_mov_b32_e32 v3, 0
+; GFX11-NEXT: v_add_co_ci_u32_e64 v1, null, s8, v1, vcc
+; GFX11-NEXT: global_load_b64 v[0:1], v[0:1], off
+; GFX11-NEXT: s_waitcnt lgkmcnt(0)
+; GFX11-NEXT: buffer_atomic_add_u32 v2, v3, s[4:7], 0 idxen
+; GFX11-NEXT: s_mov_b64 s[4:5], exec
+; GFX11-NEXT: s_mov_b64 s[6:7], exec
+; GFX11-NEXT: .LBB3_1: ; =>This Inner Loop Header: Depth=1
+; GFX11-NEXT: s_waitcnt vmcnt(0)
+; GFX11-NEXT: v_readfirstlane_b32 s0, v0
+; GFX11-NEXT: v_readfirstlane_b32 s3, v1
+; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT: v_cmpx_eq_u32_e64 s0, v0
+; GFX11-NEXT: v_cmpx_eq_u32_e64 s3, v1
+; GFX11-NEXT: s_ashr_i32 s3, s0, 31
+; GFX11-NEXT: s_add_u32 s6, s1, s0
+; GFX11-NEXT: s_addc_u32 s7, s2, s3
+; GFX11-NEXT: s_load_b128 s[8:11], s[6:7], 0x0
+; GFX11-NEXT: s_waitcnt lgkmcnt(0)
+; GFX11-NEXT: buffer_atomic_add_u32 v2, v3, s[8:11], 0 idxen glc
+; GFX11-NEXT: s_and_not1_wrexec_b64 s[4:5], s[4:5]
+; GFX11-NEXT: ; implicit-def: $vgpr0_vgpr1
+; GFX11-NEXT: ; implicit-def: $vgpr2
+; GFX11-NEXT: ; implicit-def: $vgpr3
+; GFX11-NEXT: s_cbranch_execnz .LBB3_1
+; GFX11-NEXT: ; %bb.2:
+; GFX11-NEXT: s_endpgm
+bb:
+ %getelementptr = getelementptr i8, ptr addrspace(1) %arg, i32 %arg1
+ %load = load <2 x i32>, ptr addrspace(1) %getelementptr, align 8
+ %extractelement = extractelement <2 x i32> %load, i32 0
+ %extractelement4 = extractelement <2 x i32> %load, i32 1
+ %token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %extractelement)
+ %sext = sext i32 %arg3 to i64
+ %getelementptr5 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext
+ %load6 = load <4 x i32>, ptr addrspace(4) %getelementptr5, align 4
+ %call = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %load6, i32 0, i32 0, i32 0, i32 0)
+ %token2 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %token, i32 %extractelement4)
+ %call8 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %token2, i32 %extractelement4)
+ %sext9 = sext i32 %call8 to i64
+ %getelementptr10 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext9
+ %load11 = load <4 x i32>, ptr addrspace(4) %getelementptr10, align 4
+ %call12 = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %load11, i32 0, i32 0, i32 0, i32 0)
+ %call13 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %token2, i32 %call12)
+ ret void
+}
+
+
+
+; Function Attrs: nocallback nofree nounwind willreturn
+declare i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32, <4 x i32>, i32, i32, i32, i32 immarg) #3
+; Function Attrs: convergent nounwind
+declare i32 @llvm.amdgcn.waterfall.begin.i32(i32, i32) #4
+; Function Attrs: convergent nounwind
+declare i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32, i32) #4
+; Function Attrs: convergent nounwind
+declare i32 @llvm.amdgcn.waterfall.end.i32(i32, i32) #4
+
+attributes #0 = { nounwind readnone convergent }
diff --git a/llvm/test/CodeGen/AMDGPU/llc-pipeline-npm.ll b/llvm/test/CodeGen/AMDGPU/llc-pipeline-npm.ll
index c49b2b927bd31..0cbd3dae53ba8 100644
--- a/llvm/test/CodeGen/AMDGPU/llc-pipeline-npm.ll
+++ b/llvm/test/CodeGen/AMDGPU/llc-pipeline-npm.ll
@@ -57,6 +57,7 @@
; GCN-O0-NEXT: localstackalloc)))
; GCN-O0-NEXT: require<reg-usage>
; GCN-O0-NEXT: cgscc(function(machine-function(reg-usage-propagation
+; GCN-O0-NEXT: si-insert-waterfall
; GCN-O0-NEXT: phi-node-elimination
; GCN-O0-NEXT: si-lower-control-flow
; GCN-O0-NEXT: two-address-instruction
@@ -186,6 +187,7 @@
; GCN-O2-NEXT: si-shrink-instructions)))
; GCN-O2-NEXT: require<reg-usage>
; GCN-O2-NEXT: cgscc(function(machine-function(reg-usage-propagation
+; GCN-O2-NEXT: si-insert-waterfall
; GCN-O2-NEXT: amdgpu-prepare-agpr-alloc
; GCN-O2-NEXT: detect-dead-lanes
; GCN-O2-NEXT: dead-mi-elimination
@@ -355,6 +357,7 @@
; GCN-O3-NEXT: si-shrink-instructions)))
; GCN-O3-NEXT: require<reg-usage>
; GCN-O3-NEXT: cgscc(function(machine-function(reg-usage-propagation
+; GCN-O3-NEXT: si-insert-waterfall
; GCN-O3-NEXT: amdgpu-prepare-agpr-alloc
; GCN-O3-NEXT: detect-dead-lanes
; GCN-O3-NEXT: dead-mi-elimination
diff --git a/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll b/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll
index 070c873798647..e5ae5fdaff6ab 100644
--- a/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll
+++ b/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll
@@ -109,6 +109,7 @@
; GCN-O0-NEXT: Finalize ISel and expand pseudo-instructions
; GCN-O0-NEXT: Local Stack Slot Allocation
; GCN-O0-NEXT: Register Usage Information Propagation
+; GCN-O0-NEXT: SI Insert waterfalls
; GCN-O0-NEXT: Eliminate PHI nodes for register allocation
; GCN-O0-NEXT: SI Lower control flow pseudo instructions
; GCN-O0-NEXT: Two-Address instruction pass
@@ -343,6 +344,7 @@
; GCN-O1-NEXT: Remove dead machine instructions
; GCN-O1-NEXT: SI Shrink Instructions
; GCN-O1-NEXT: Register Usage Information Propagation
+; GCN-O1-NEXT: SI Insert waterfalls
; GCN-O1-NEXT: AMDGPU Prepare AGPR Alloc
; GCN-O1-NEXT: Detect Dead Lanes
; GCN-O1-NEXT: Remove dead machine instructions
@@ -351,6 +353,7 @@
; GCN-O1-NEXT: Remove unreachable machine basic blocks
; GCN-O1-NEXT: Live Variable Analysis
; GCN-O1-NEXT: MachineDominator Tree Construction
+; GCN-O1-NEXT: Machine Natural Loop Construction
; GCN-O1-NEXT: SI Optimize VGPR LiveRange
; GCN-O1-NEXT: Eliminate PHI nodes for register allocation
; GCN-O1-NEXT: SI Lower control flow pseudo instructions
@@ -361,6 +364,7 @@
; GCN-O1-NEXT: Register Coalescer
; GCN-O1-NEXT: Rename Disconnected Subregister Components
; GCN-O1-NEXT: Rewrite Partial Register Uses
+; GCN-O1-NEXT: Machine Block Frequency Analysis
; GCN-O1-NEXT: Machine Instruction Scheduler
; GCN-O1-NEXT: SI Whole Quad Mode
; GCN-O1-NEXT: SI optimize exec mask operations pre-RA
@@ -661,6 +665,7 @@
; GCN-O1-OPTS-NEXT: Remove dead machine instructions
; GCN-O1-OPTS-NEXT: SI Shrink Instructions
; GCN-O1-OPTS-NEXT: Register Usage Information Propagation
+; GCN-O1-OPTS-NEXT: SI Insert waterfalls
; GCN-O1-OPTS-NEXT: AMDGPU Prepare AGPR Alloc
; GCN-O1-OPTS-NEXT: Detect Dead Lanes
; GCN-O1-OPTS-NEXT: Remove dead machine instructions
@@ -668,6 +673,8 @@
; GCN-O1-OPTS-NEXT: Process Implicit Definitions
; GCN-O1-OPTS-NEXT: Remove unreachable machine basic blocks
; GCN-O1-OPTS-NEXT: Live Variable Analysis
+; GCN-O1-OPTS-NEXT: MachineDominator Tree Construction
+; GCN-O1-OPTS-NEXT: Machine Natural Loop Construction
; GCN-O1-OPTS-NEXT: SI Optimize VGPR LiveRange
; GCN-O1-OPTS-NEXT: Eliminate PHI nodes for register allocation
; GCN-O1-OPTS-NEXT: SI Lower control flow pseudo instructions
@@ -678,6 +685,7 @@
; GCN-O1-OPTS-NEXT: Register Coalescer
; GCN-O1-OPTS-NEXT: Rename Disconnected Subregister Components
; GCN-O1-OPTS-NEXT: Rewrite Partial Register Uses
+; GCN-O1-OPTS-NEXT: Machine Block Frequency Analysis
; GCN-O1-OPTS-NEXT: Machine Instruction Scheduler
; GCN-O1-OPTS-NEXT: AMDGPU Pre-RA optimizations
; GCN-O1-OPTS-NEXT: SI Whole Quad Mode
@@ -983,6 +991,7 @@
; GCN-O2-NEXT: Remove dead machine instructions
; GCN-O2-NEXT: SI Shrink Instructions
; GCN-O2-NEXT: Register Usage Information Propagation
+; GCN-O2-NEXT: SI Insert waterfalls
; GCN-O2-NEXT: AMDGPU Prepare AGPR Alloc
; GCN-O2-NEXT: Detect Dead Lanes
; GCN-O2-NEXT: Remove dead machine instructions
@@ -990,6 +999,8 @@
; GCN-O2-NEXT: Process Implicit Definitions
; GCN-O2-NEXT: Remove unreachable machine basic blocks
; GCN-O2-NEXT: Live Variable Analysis
+; GCN-O2-NEXT: MachineDominator Tree Construction
+; GCN-O2-NEXT: Machine Natural Loop Construction
; GCN-O2-NEXT: SI Optimize VGPR LiveRange
; GCN-O2-NEXT: Eliminate PHI nodes for register allocation
; GCN-O2-NEXT: SI Lower control flow pseudo instructions
@@ -1000,6 +1011,7 @@
; GCN-O2-NEXT: Register Coalescer
; GCN-O2-NEXT: Rename Disconnected Subregister Components
; GCN-O2-NEXT: Rewrite Partial Register Uses
+; GCN-O2-NEXT: Machine Block Frequency Analysis
; GCN-O2-NEXT: Machine Instruction Scheduler
; GCN-O2-NEXT: AMDGPU Pre-RA optimizations
; GCN-O2-NEXT: SI Whole Quad Mode
@@ -1319,6 +1331,7 @@
; GCN-O3-NEXT: Remove dead machine instructions
; GCN-O3-NEXT: SI Shrink Instructions
; GCN-O3-NEXT: Register Usage Information Propagation
+; GCN-O3-NEXT: SI Insert waterfalls
; GCN-O3-NEXT: AMDGPU Prepare AGPR Alloc
; GCN-O3-NEXT: Detect Dead Lanes
; GCN-O3-NEXT: Remove dead machine instructions
@@ -1326,6 +1339,8 @@
; GCN-O3-NEXT: Process Implicit Definitions
; GCN-O3-NEXT: Remove unreachable machine basic blocks
; GCN-O3-NEXT: Live Variable Analysis
+; GCN-O3-NEXT: MachineDominator Tree Construction
+; GCN-O3-NEXT: Machine Natural Loop Construction
; GCN-O3-NEXT: SI Optimize VGPR LiveRange
; GCN-O3-NEXT: Eliminate PHI nodes for register allocation
; GCN-O3-NEXT: SI Lower control flow pseudo instructions
@@ -1336,6 +1351,7 @@
; GCN-O3-NEXT: Register Coalescer
; GCN-O3-NEXT: Rename Disconnected Subregister Components
; GCN-O3-NEXT: Rewrite Partial Register Uses
+; GCN-O3-NEXT: Machine Block Frequency Analysis
; GCN-O3-NEXT: Machine Instruction Scheduler
; GCN-O3-NEXT: AMDGPU Pre-RA optimizations
; GCN-O3-NEXT: SI Whole Quad Mode
diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
new file mode 100644
index 0000000000000..4ceb4245c77e9
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
@@ -0,0 +1,12656 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -global-isel=0 -march=amdgcn -mcpu=fiji -verify-machineinstrs < %s | FileCheck -check-prefixes=PRE-GFX10,VI,VI-SDAG %s
+; RUN: llc -global-isel=1 -march=amdgcn -mcpu=fiji -verify-machineinstrs < %s | FileCheck -check-prefixes=PRE-GFX10,VI,VI-GISEL %s
+; RUN: llc -global-isel=0 -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=PRE-GFX10,GFX9,GFX9-SDAG %s
+; RUN: llc -global-isel=1 -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=PRE-GFX10,GFX9,GFX9-GISEL %s
+; RUN: llc -global-isel=0 -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX10GFX11-SDAG,GFX10,GFX10-32,GFX10-32-SDAG %s
+; RUN: llc -global-isel=1 -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX10,GFX10-32,GFX10-32-GISEL %s
+; RUN: llc -global-isel=0 -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX10GFX11-SDAG,GFX10,GFX10-64,GFX10-64-SDAG %s
+; RUN: llc -global-isel=1 -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX10,GFX10-64,GFX10-64-GISEL %s
+; RUN: llc -global-isel=0 -march=amdgcn -mcpu=gfx1150 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX10GFX11-SDAG,GFX1150,GFX1150-SDAG %s
+; RUN: llc -global-isel=1 -march=amdgcn -mcpu=gfx1150 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX1150,GFX1150-GISEL %s
+; RUN: llc -global-isel=0 -march=amdgcn -mcpu=gfx1200 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX12,GFX12-SDAG %s
+; RUN: llc -global-isel=1 -march=amdgcn -mcpu=gfx1200 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX12,GFX12-GISEL %s
+
+ at Lds = addrspace(3) global [16384 x i32] undef
+
+define amdgpu_ps void @test_waterfall_readlane(i32 addrspace(1)* inreg %out, <2 x i32> addrspace(1)* inreg %in, i32 %tid) #1 {
+; VI-SDAG-LABEL: test_waterfall_readlane:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; VI-SDAG-NEXT: v_lshlrev_b64 v[0:1], 3, v[0:1]
+; VI-SDAG-NEXT: v_mov_b32_e32 v2, s3
+; VI-SDAG-NEXT: v_add_u32_e32 v0, vcc, s2, v0
+; VI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, v2, v1, vcc
+; VI-SDAG-NEXT: flat_load_dwordx2 v[2:3], v[0:1]
+; VI-SDAG-NEXT: v_mov_b32_e32 v0, s0
+; VI-SDAG-NEXT: v_mov_b32_e32 v1, s1
+; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; VI-SDAG-NEXT: .LBB0_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s4, v3
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s4, v3
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-SDAG-NEXT: s_nop 1
+; VI-SDAG-NEXT: v_readlane_b32 s4, v2, s4
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, s4
+; VI-SDAG-NEXT: ; implicit-def: $vgpr2_vgpr3
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB0_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; VI-SDAG-NEXT: flat_store_dword v[0:1], v4
+; VI-SDAG-NEXT: s_endpgm
+;
+; VI-GISEL-LABEL: test_waterfall_readlane:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; VI-GISEL-NEXT: v_lshlrev_b64 v[0:1], 3, v[0:1]
+; VI-GISEL-NEXT: v_mov_b32_e32 v2, s2
+; VI-GISEL-NEXT: v_mov_b32_e32 v3, s3
+; VI-GISEL-NEXT: v_add_u32_e32 v0, vcc, v2, v0
+; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, v3, v1, vcc
+; VI-GISEL-NEXT: flat_load_dwordx2 v[0:1], v[0:1]
+; VI-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; VI-GISEL-NEXT: .LBB0_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s6, v1
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s6, v1
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
+; VI-GISEL-NEXT: s_nop 1
+; VI-GISEL-NEXT: v_readlane_b32 s6, v0, s6
+; VI-GISEL-NEXT: v_mov_b32_e32 v2, s6
+; VI-GISEL-NEXT: ; implicit-def: $vgpr0_vgpr1
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[4:5]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB0_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; VI-GISEL-NEXT: flat_store_dword v[0:1], v2
+; VI-GISEL-NEXT: s_endpgm
+;
+; GFX9-SDAG-LABEL: test_waterfall_readlane:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX9-SDAG-NEXT: v_lshlrev_b64 v[0:1], 3, v[0:1]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v2, s3
+; GFX9-SDAG-NEXT: v_add_co_u32_e32 v0, vcc, s2, v0
+; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v1, vcc, v2, v1, vcc
+; GFX9-SDAG-NEXT: global_load_dwordx2 v[0:1], v[0:1], off
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v2, 0
+; GFX9-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX9-SDAG-NEXT: .LBB0_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s6, v1
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[4:5], s6, v1
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
+; GFX9-SDAG-NEXT: s_nop 1
+; GFX9-SDAG-NEXT: v_readlane_b32 s6, v0, s6
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v3, s6
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr0_vgpr1
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[4:5]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB0_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-SDAG-NEXT: global_store_dword v2, v3, s[0:1]
+; GFX9-SDAG-NEXT: s_endpgm
+;
+; GFX9-GISEL-LABEL: test_waterfall_readlane:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX9-GISEL-NEXT: v_lshlrev_b64 v[0:1], 3, v[0:1]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v2, s2
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v3, s3
+; GFX9-GISEL-NEXT: v_add_co_u32_e32 v0, vcc, v2, v0
+; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v1, vcc, v3, v1, vcc
+; GFX9-GISEL-NEXT: global_load_dwordx2 v[0:1], v[0:1], off
+; GFX9-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX9-GISEL-NEXT: .LBB0_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s6, v1
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s6, v1
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
+; GFX9-GISEL-NEXT: s_nop 1
+; GFX9-GISEL-NEXT: v_readlane_b32 s6, v0, s6
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v2, s6
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr0_vgpr1
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[4:5]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB0_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX9-GISEL-NEXT: global_store_dword v0, v2, s[0:1]
+; GFX9-GISEL-NEXT: s_endpgm
+;
+; GFX10-32-SDAG-LABEL: test_waterfall_readlane:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v2, 0
+; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[0:1], 3, v[0:1]
+; GFX10-32-SDAG-NEXT: v_add_co_u32 v0, vcc_lo, s2, v0
+; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, s3, v1, vcc_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s3, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s2, exec_lo
+; GFX10-32-SDAG-NEXT: global_load_dwordx2 v[0:1], v[0:1], off
+; GFX10-32-SDAG-NEXT: .LBB0_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s4, v1
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s4, v1
+; GFX10-32-SDAG-NEXT: v_readlane_b32 s4, v0, s4
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v3, s4
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s3, s3
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr0_vgpr1
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB0_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s2
+; GFX10-32-SDAG-NEXT: global_store_dword v2, v3, s[0:1]
+; GFX10-32-SDAG-NEXT: s_endpgm
+;
+; GFX10-32-GISEL-LABEL: test_waterfall_readlane:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v2, s2
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v3, s3
+; GFX10-32-GISEL-NEXT: s_mov_b32 s3, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s2, exec_lo
+; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[0:1], 3, v[0:1]
+; GFX10-32-GISEL-NEXT: v_add_co_u32 v0, vcc_lo, v2, v0
+; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
+; GFX10-32-GISEL-NEXT: global_load_dwordx2 v[0:1], v[0:1], off
+; GFX10-32-GISEL-NEXT: .LBB0_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s4, v1
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s4, v1
+; GFX10-32-GISEL-NEXT: v_readlane_b32 s4, v0, s4
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v2, s4
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s3, s3
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr0_vgpr1
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB0_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s2
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-GISEL-NEXT: global_store_dword v0, v2, s[0:1]
+; GFX10-32-GISEL-NEXT: s_endpgm
+;
+; GFX10-64-SDAG-LABEL: test_waterfall_readlane:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v2, 0
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[0:1], 3, v[0:1]
+; GFX10-64-SDAG-NEXT: v_add_co_u32 v0, vcc, s2, v0
+; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc, s3, v1, vcc
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-SDAG-NEXT: global_load_dwordx2 v[0:1], v[0:1], off
+; GFX10-64-SDAG-NEXT: .LBB0_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s6, v1
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v1
+; GFX10-64-SDAG-NEXT: v_readlane_b32 s6, v0, s6
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v3, s6
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[4:5], s[4:5]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr0_vgpr1
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB0_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[2:3]
+; GFX10-64-SDAG-NEXT: global_store_dword v2, v3, s[0:1]
+; GFX10-64-SDAG-NEXT: s_endpgm
+;
+; GFX10-64-GISEL-LABEL: test_waterfall_readlane:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v2, s2
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v3, s3
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[0:1], 3, v[0:1]
+; GFX10-64-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc, v3, v1, vcc
+; GFX10-64-GISEL-NEXT: global_load_dwordx2 v[0:1], v[0:1], off
+; GFX10-64-GISEL-NEXT: .LBB0_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s6, v1
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s6, v1
+; GFX10-64-GISEL-NEXT: v_readlane_b32 s6, v0, s6
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v2, s6
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[4:5], s[4:5]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr0_vgpr1
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB0_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-GISEL-NEXT: global_store_dword v0, v2, s[0:1]
+; GFX10-64-GISEL-NEXT: s_endpgm
+;
+; GFX1150-SDAG-LABEL: test_waterfall_readlane:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v2, 0
+; GFX1150-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[0:1], 3, v[0:1]
+; GFX1150-SDAG-NEXT: v_add_co_u32 v0, vcc, s2, v0
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s3, v1, vcc
+; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-SDAG-NEXT: global_load_b64 v[0:1], v[0:1], off
+; GFX1150-SDAG-NEXT: .LBB0_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s6, v1
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v1
+; GFX1150-SDAG-NEXT: v_readlane_b32 s6, v0, s6
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v3, s6
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[4:5], s[4:5]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr0_vgpr1
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB0_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[2:3]
+; GFX1150-SDAG-NEXT: global_store_b32 v2, v3, s[0:1]
+; GFX1150-SDAG-NEXT: s_endpgm
+;
+; GFX1150-GISEL-LABEL: test_waterfall_readlane:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v2, s2
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v3, s3
+; GFX1150-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[0:1], 3, v[0:1]
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX1150-GISEL-NEXT: global_load_b64 v[0:1], v[0:1], off
+; GFX1150-GISEL-NEXT: .LBB0_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s6, v1
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s6, v1
+; GFX1150-GISEL-NEXT: v_readlane_b32 s6, v0, s6
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v2, s6
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[4:5], s[4:5]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr0_vgpr1
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB0_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-GISEL-NEXT: global_store_b32 v0, v2, s[0:1]
+; GFX1150-GISEL-NEXT: s_endpgm
+;
+; GFX12-SDAG-LABEL: test_waterfall_readlane:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v2, 0
+; GFX12-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[0:1], 3, v[0:1]
+; GFX12-SDAG-NEXT: v_add_co_u32 v0, vcc, s2, v0
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s3, v1, vcc
+; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-SDAG-NEXT: global_load_b64 v[0:1], v[0:1], off
+; GFX12-SDAG-NEXT: .LBB0_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x0
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s6, v1
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v1
+; GFX12-SDAG-NEXT: v_readlane_b32 s6, v0, s6
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v3, s6
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[4:5], s[4:5]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr0_vgpr1
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB0_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[2:3]
+; GFX12-SDAG-NEXT: global_store_b32 v2, v3, s[0:1]
+; GFX12-SDAG-NEXT: s_endpgm
+;
+; GFX12-GISEL-LABEL: test_waterfall_readlane:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v2, s2
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v3, s3
+; GFX12-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[0:1], 3, v[0:1]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX12-GISEL-NEXT: global_load_b64 v[0:1], v[0:1], off
+; GFX12-GISEL-NEXT: .LBB0_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s6, v1
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s6, v1
+; GFX12-GISEL-NEXT: v_readlane_b32 s6, v0, s6
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v2, s6
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[4:5], s[4:5]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr0_vgpr1
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB0_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-GISEL-NEXT: global_store_b32 v0, v2, s[0:1]
+; GFX12-GISEL-NEXT: s_endpgm
+ %gep.in = getelementptr <2 x i32>, <2 x i32> addrspace(1)* %in, i32 %tid
+ %args = load <2 x i32>, <2 x i32> addrspace(1)* %gep.in
+ %value = extractelement <2 x i32> %args, i32 0
+ %lane = extractelement <2 x i32> %args, i32 1
+ %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %lane)
+ %readlane = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %lane)
+ %readlane1 = call i32 @llvm.amdgcn.readlane(i32 %value, i32 %readlane)
+ %readlane2 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %wf_token, i32 %readlane1)
+ ; This store instruction should be outside the waterfall loop and the value
+ ; being stored generated incrementally in the loop itself
+ store i32 %readlane2, i32 addrspace(1)* %out, align 4
+
+ ret void
+}
+
+define amdgpu_ps <4 x float> @test_waterfall_non_uniform_img(<8 x i32> addrspace(4)* inreg %in, i32 %index, float %s, <4 x i32> inreg %samp) #1 {
+; VI-SDAG-LABEL: test_waterfall_non_uniform_img:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; VI-SDAG-NEXT: s_wqm_b64 exec, exec
+; VI-SDAG-NEXT: s_mov_b32 s7, s5
+; VI-SDAG-NEXT: s_mov_b32 s6, s4
+; VI-SDAG-NEXT: s_mov_b32 s5, s3
+; VI-SDAG-NEXT: s_mov_b32 s4, s2
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; VI-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; VI-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; VI-SDAG-NEXT: .LBB1_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s10, v5
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[12:13], s10, v5
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[20:21], s[12:13]
+; VI-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; VI-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; VI-SDAG-NEXT: s_add_u32 s10, s0, s10
+; VI-SDAG-NEXT: s_addc_u32 s11, s1, s11
+; VI-SDAG-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; VI-SDAG-NEXT: ; implicit-def: $vgpr5
+; VI-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; VI-SDAG-NEXT: image_sample v[0:3], v4, s[12:19], s[4:7] dmask:0xf
+; VI-SDAG-NEXT: ; implicit-def: $vgpr4
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[20:21]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB1_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[2:3]
+; VI-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: ; return to shader part epilog
+;
+; VI-GISEL-LABEL: test_waterfall_non_uniform_img:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; VI-GISEL-NEXT: s_wqm_b64 exec, exec
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; VI-GISEL-NEXT: s_mov_b32 s8, s2
+; VI-GISEL-NEXT: s_mov_b32 s9, s3
+; VI-GISEL-NEXT: s_mov_b32 s10, s4
+; VI-GISEL-NEXT: s_mov_b32 s11, s5
+; VI-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; VI-GISEL-NEXT: .LBB1_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s4, v5
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[12:13], s4, v5
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[20:21], s[12:13]
+; VI-GISEL-NEXT: s_ashr_i32 s5, s4, 31
+; VI-GISEL-NEXT: s_lshl_b64 s[4:5], s[4:5], 5
+; VI-GISEL-NEXT: s_add_u32 s4, s0, s4
+; VI-GISEL-NEXT: s_addc_u32 s5, s1, s5
+; VI-GISEL-NEXT: s_load_dwordx8 s[12:19], s[4:5], 0x0
+; VI-GISEL-NEXT: ; implicit-def: $vgpr5
+; VI-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; VI-GISEL-NEXT: image_sample v[0:3], v4, s[12:19], s[8:11] dmask:0xf
+; VI-GISEL-NEXT: ; implicit-def: $vgpr4
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[20:21]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB1_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; VI-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX9-SDAG-LABEL: test_waterfall_non_uniform_img:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX9-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX9-SDAG-NEXT: s_mov_b32 s7, s5
+; GFX9-SDAG-NEXT: s_mov_b32 s6, s4
+; GFX9-SDAG-NEXT: s_mov_b32 s5, s3
+; GFX9-SDAG-NEXT: s_mov_b32 s4, s2
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX9-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX9-SDAG-NEXT: .LBB1_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s10, v5
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[12:13], s10, v5
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[20:21], s[12:13]
+; GFX9-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; GFX9-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX9-SDAG-NEXT: s_add_u32 s10, s0, s10
+; GFX9-SDAG-NEXT: s_addc_u32 s11, s1, s11
+; GFX9-SDAG-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX9-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-SDAG-NEXT: image_sample v[0:3], v4, s[12:19], s[4:7] dmask:0xf
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[20:21]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB1_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX9-GISEL-LABEL: test_waterfall_non_uniform_img:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX9-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; GFX9-GISEL-NEXT: s_mov_b32 s8, s2
+; GFX9-GISEL-NEXT: s_mov_b32 s9, s3
+; GFX9-GISEL-NEXT: s_mov_b32 s10, s4
+; GFX9-GISEL-NEXT: s_mov_b32 s11, s5
+; GFX9-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX9-GISEL-NEXT: .LBB1_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s4, v5
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[12:13], s4, v5
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[20:21], s[12:13]
+; GFX9-GISEL-NEXT: s_ashr_i32 s5, s4, 31
+; GFX9-GISEL-NEXT: s_lshl_b64 s[4:5], s[4:5], 5
+; GFX9-GISEL-NEXT: s_add_u32 s4, s0, s4
+; GFX9-GISEL-NEXT: s_addc_u32 s5, s1, s5
+; GFX9-GISEL-NEXT: s_load_dwordx8 s[12:19], s[4:5], 0x0
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX9-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-GISEL-NEXT: image_sample v[0:3], v4, s[12:19], s[8:11] dmask:0xf
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[20:21]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB1_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-32-SDAG-LABEL: test_waterfall_non_uniform_img:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: s_mov_b32 s8, exec_lo
+; GFX10-32-SDAG-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s7, s5
+; GFX10-32-SDAG-NEXT: s_mov_b32 s6, s4
+; GFX10-32-SDAG-NEXT: s_mov_b32 s5, s3
+; GFX10-32-SDAG-NEXT: s_mov_b32 s4, s2
+; GFX10-32-SDAG-NEXT: s_mov_b32 s3, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s2, exec_lo
+; GFX10-32-SDAG-NEXT: .LBB1_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s10, v5
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s10, v5
+; GFX10-32-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; GFX10-32-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX10-32-SDAG-NEXT: s_add_u32 s10, s0, s10
+; GFX10-32-SDAG-NEXT: s_addc_u32 s11, s1, s11
+; GFX10-32-SDAG-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; GFX10-32-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-SDAG-NEXT: image_sample v[0:3], v4, s[12:19], s[4:7] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s3, s3
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB1_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s2
+; GFX10-32-SDAG-NEXT: s_and_b32 exec_lo, exec_lo, s8
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-32-GISEL-LABEL: test_waterfall_non_uniform_img:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: s_mov_b32 s6, exec_lo
+; GFX10-32-GISEL-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; GFX10-32-GISEL-NEXT: s_mov_b32 s8, s2
+; GFX10-32-GISEL-NEXT: s_mov_b32 s9, s3
+; GFX10-32-GISEL-NEXT: s_mov_b32 s10, s4
+; GFX10-32-GISEL-NEXT: s_mov_b32 s11, s5
+; GFX10-32-GISEL-NEXT: s_mov_b32 s3, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s2, exec_lo
+; GFX10-32-GISEL-NEXT: .LBB1_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s4, v5
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s4, v5
+; GFX10-32-GISEL-NEXT: s_ashr_i32 s5, s4, 31
+; GFX10-32-GISEL-NEXT: s_lshl_b64 s[4:5], s[4:5], 5
+; GFX10-32-GISEL-NEXT: s_add_u32 s4, s0, s4
+; GFX10-32-GISEL-NEXT: s_addc_u32 s5, s1, s5
+; GFX10-32-GISEL-NEXT: s_load_dwordx8 s[12:19], s[4:5], 0x0
+; GFX10-32-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-GISEL-NEXT: image_sample v[0:3], v4, s[12:19], s[8:11] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s3, s3
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB1_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s2
+; GFX10-32-GISEL-NEXT: s_and_b32 exec_lo, exec_lo, s6
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-64-SDAG-LABEL: test_waterfall_non_uniform_img:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX10-64-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX10-64-SDAG-NEXT: s_mov_b32 s7, s5
+; GFX10-64-SDAG-NEXT: s_mov_b32 s6, s4
+; GFX10-64-SDAG-NEXT: s_mov_b32 s5, s3
+; GFX10-64-SDAG-NEXT: s_mov_b32 s4, s2
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-SDAG-NEXT: .LBB1_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX10-64-SDAG-NEXT: s_ashr_i32 s13, s12, 31
+; GFX10-64-SDAG-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; GFX10-64-SDAG-NEXT: s_add_u32 s12, s0, s12
+; GFX10-64-SDAG-NEXT: s_addc_u32 s13, s1, s13
+; GFX10-64-SDAG-NEXT: s_load_dwordx8 s[12:19], s[12:13], 0x0
+; GFX10-64-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-SDAG-NEXT: image_sample v[0:3], v4, s[12:19], s[4:7] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[10:11], s[10:11]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB1_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[2:3]
+; GFX10-64-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-64-GISEL-LABEL: test_waterfall_non_uniform_img:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; GFX10-64-GISEL-NEXT: s_mov_b32 s8, s2
+; GFX10-64-GISEL-NEXT: s_mov_b32 s9, s3
+; GFX10-64-GISEL-NEXT: s_mov_b32 s10, s4
+; GFX10-64-GISEL-NEXT: s_mov_b32 s11, s5
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-GISEL-NEXT: .LBB1_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s12, v5
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX10-64-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; GFX10-64-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; GFX10-64-GISEL-NEXT: s_add_u32 s12, s0, s12
+; GFX10-64-GISEL-NEXT: s_addc_u32 s13, s1, s13
+; GFX10-64-GISEL-NEXT: s_load_dwordx8 s[12:19], s[12:13], 0x0
+; GFX10-64-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-GISEL-NEXT: image_sample v[0:3], v4, s[12:19], s[8:11] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[4:5], s[4:5]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB1_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX10-64-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX1150-SDAG-LABEL: test_waterfall_non_uniform_img:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX1150-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX1150-SDAG-NEXT: s_mov_b32 s7, s5
+; GFX1150-SDAG-NEXT: s_mov_b32 s6, s4
+; GFX1150-SDAG-NEXT: s_mov_b32 s5, s3
+; GFX1150-SDAG-NEXT: s_mov_b32 s4, s2
+; GFX1150-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-SDAG-NEXT: .LBB1_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX1150-SDAG-NEXT: s_ashr_i32 s13, s12, 31
+; GFX1150-SDAG-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_add_u32 s12, s0, s12
+; GFX1150-SDAG-NEXT: s_addc_u32 s13, s1, s13
+; GFX1150-SDAG-NEXT: s_load_b256 s[12:19], s[12:13], 0x0
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX1150-SDAG-NEXT: image_sample v[0:3], v4, s[12:19], s[4:7] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB1_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[2:3]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX1150-GISEL-LABEL: test_waterfall_non_uniform_img:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; GFX1150-GISEL-NEXT: s_mov_b32 s8, s2
+; GFX1150-GISEL-NEXT: s_mov_b32 s9, s3
+; GFX1150-GISEL-NEXT: s_mov_b32 s10, s4
+; GFX1150-GISEL-NEXT: s_mov_b32 s11, s5
+; GFX1150-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-GISEL-NEXT: .LBB1_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s12, v5
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX1150-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; GFX1150-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_add_u32 s12, s0, s12
+; GFX1150-GISEL-NEXT: s_addc_u32 s13, s1, s13
+; GFX1150-GISEL-NEXT: s_load_b256 s[12:19], s[12:13], 0x0
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX1150-GISEL-NEXT: image_sample v[0:3], v4, s[12:19], s[8:11] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[4:5], s[4:5]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB1_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX12-SDAG-LABEL: test_waterfall_non_uniform_img:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX12-SDAG-NEXT: s_mov_b32 s7, s5
+; GFX12-SDAG-NEXT: s_mov_b32 s6, s4
+; GFX12-SDAG-NEXT: s_mov_b32 s5, s3
+; GFX12-SDAG-NEXT: s_mov_b32 s4, s2
+; GFX12-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-SDAG-NEXT: .LBB1_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX12-SDAG-NEXT: s_ashr_i32 s13, s12, 31
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_add_nc_u64 s[12:13], s[0:1], s[12:13]
+; GFX12-SDAG-NEXT: s_load_b256 s[12:19], s[12:13], 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: image_sample v[0:3], v4, s[12:19], s[4:7] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB1_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[2:3]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX12-GISEL-LABEL: test_waterfall_non_uniform_img:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; GFX12-GISEL-NEXT: s_mov_b32 s8, s2
+; GFX12-GISEL-NEXT: s_mov_b32 s9, s3
+; GFX12-GISEL-NEXT: s_mov_b32 s10, s4
+; GFX12-GISEL-NEXT: s_mov_b32 s11, s5
+; GFX12-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-GISEL-NEXT: .LBB1_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s12, v5
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX12-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_add_co_u32 s12, s0, s12
+; GFX12-GISEL-NEXT: s_add_co_ci_u32 s13, s1, s13
+; GFX12-GISEL-NEXT: s_load_b256 s[12:19], s[12:13], 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: image_sample v[0:3], v4, s[12:19], s[8:11] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[4:5], s[4:5]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB1_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: ; return to shader part epilog
+ %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %index)
+ %ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
+ %r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token, <4 x float> %r)
+
+ ret <4 x float> %r1
+}
+
+define amdgpu_ps <4 x float> @test_waterfall_non_uniform_img_single_read(<8 x i32> addrspace(4)* inreg %in, i32 %index, float %s, <4 x i32> inreg %samp) #1 {
+; VI-SDAG-LABEL: test_waterfall_non_uniform_img_single_read:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; VI-SDAG-NEXT: s_wqm_b64 exec, exec
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; VI-SDAG-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; VI-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; VI-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[4:5]
+; VI-SDAG-NEXT: v_mov_b32_e32 v2, s1
+; VI-SDAG-NEXT: v_add_u32_e32 v0, vcc, s0, v0
+; VI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, v2, v1, vcc
+; VI-SDAG-NEXT: v_add_u32_e32 v2, vcc, 16, v0
+; VI-SDAG-NEXT: v_addc_u32_e32 v3, vcc, 0, v1, vcc
+; VI-SDAG-NEXT: flat_load_dwordx4 v[7:10], v[0:1]
+; VI-SDAG-NEXT: flat_load_dwordx4 v[11:14], v[2:3]
+; VI-SDAG-NEXT: s_mov_b32 s7, s5
+; VI-SDAG-NEXT: s_mov_b32 s6, s4
+; VI-SDAG-NEXT: s_mov_b32 s5, s3
+; VI-SDAG-NEXT: s_mov_b32 s4, s2
+; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: .LBB2_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v4
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-SDAG-NEXT: v_readfirstlane_b32 s12, v7
+; VI-SDAG-NEXT: v_readfirstlane_b32 s13, v8
+; VI-SDAG-NEXT: v_readfirstlane_b32 s14, v9
+; VI-SDAG-NEXT: v_readfirstlane_b32 s15, v10
+; VI-SDAG-NEXT: v_readfirstlane_b32 s16, v11
+; VI-SDAG-NEXT: v_readfirstlane_b32 s17, v12
+; VI-SDAG-NEXT: v_readfirstlane_b32 s18, v13
+; VI-SDAG-NEXT: v_readfirstlane_b32 s19, v14
+; VI-SDAG-NEXT: ; implicit-def: $vgpr4
+; VI-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; VI-SDAG-NEXT: s_nop 4
+; VI-SDAG-NEXT: image_sample v[0:3], v6, s[12:19], s[4:7] dmask:0xf
+; VI-SDAG-NEXT: ; implicit-def: $vgpr6
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB2_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; VI-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: ; return to shader part epilog
+;
+; VI-GISEL-LABEL: test_waterfall_non_uniform_img_single_read:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; VI-GISEL-NEXT: s_wqm_b64 exec, exec
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; VI-GISEL-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; VI-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; VI-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[4:5]
+; VI-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; VI-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; VI-GISEL-NEXT: v_add_u32_e32 v0, vcc, v2, v0
+; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, v3, v1, vcc
+; VI-GISEL-NEXT: v_add_u32_e32 v2, vcc, 16, v0
+; VI-GISEL-NEXT: v_addc_u32_e32 v3, vcc, 0, v1, vcc
+; VI-GISEL-NEXT: flat_load_dwordx4 v[7:10], v[0:1]
+; VI-GISEL-NEXT: flat_load_dwordx4 v[11:14], v[2:3]
+; VI-GISEL-NEXT: s_mov_b32 s8, s2
+; VI-GISEL-NEXT: s_mov_b32 s9, s3
+; VI-GISEL-NEXT: s_mov_b32 s10, s4
+; VI-GISEL-NEXT: s_mov_b32 s11, s5
+; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: .LBB2_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v4
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v7
+; VI-GISEL-NEXT: v_readfirstlane_b32 s13, v8
+; VI-GISEL-NEXT: v_readfirstlane_b32 s14, v9
+; VI-GISEL-NEXT: v_readfirstlane_b32 s15, v10
+; VI-GISEL-NEXT: v_readfirstlane_b32 s16, v11
+; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v12
+; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v13
+; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v14
+; VI-GISEL-NEXT: ; implicit-def: $vgpr4
+; VI-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; VI-GISEL-NEXT: s_nop 4
+; VI-GISEL-NEXT: image_sample v[0:3], v6, s[12:19], s[8:11] dmask:0xf
+; VI-GISEL-NEXT: ; implicit-def: $vgpr6
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB2_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; VI-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX9-SDAG-LABEL: test_waterfall_non_uniform_img_single_read:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX9-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX9-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[4:5]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v2, s1
+; GFX9-SDAG-NEXT: v_add_co_u32_e32 v0, vcc, s0, v0
+; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v1, vcc, v2, v1, vcc
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[11:14], v[0:1], off offset:16
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[7:10], v[0:1], off
+; GFX9-SDAG-NEXT: s_mov_b32 s7, s5
+; GFX9-SDAG-NEXT: s_mov_b32 s6, s4
+; GFX9-SDAG-NEXT: s_mov_b32 s5, s3
+; GFX9-SDAG-NEXT: s_mov_b32 s4, s2
+; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: .LBB2_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v4
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s12, v7
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s13, v8
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s14, v9
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s15, v10
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s16, v11
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s17, v12
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s18, v13
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s19, v14
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX9-SDAG-NEXT: s_nop 4
+; GFX9-SDAG-NEXT: image_sample v[0:3], v6, s[12:19], s[4:7] dmask:0xf
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB2_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX9-GISEL-LABEL: test_waterfall_non_uniform_img_single_read:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX9-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX9-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[4:5]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX9-GISEL-NEXT: v_add_co_u32_e32 v0, vcc, v2, v0
+; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v1, vcc, v3, v1, vcc
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[7:10], v[0:1], off
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[11:14], v[0:1], off offset:16
+; GFX9-GISEL-NEXT: s_mov_b32 s8, s2
+; GFX9-GISEL-NEXT: s_mov_b32 s9, s3
+; GFX9-GISEL-NEXT: s_mov_b32 s10, s4
+; GFX9-GISEL-NEXT: s_mov_b32 s11, s5
+; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: .LBB2_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v4
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v7
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s13, v8
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s14, v9
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s15, v10
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s16, v11
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v12
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v13
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v14
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX9-GISEL-NEXT: s_nop 4
+; GFX9-GISEL-NEXT: image_sample v[0:3], v6, s[12:19], s[8:11] dmask:0xf
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB2_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-32-SDAG-LABEL: test_waterfall_non_uniform_img_single_read:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: s_mov_b32 s8, exec_lo
+; GFX10-32-SDAG-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-32-SDAG-NEXT: s_mov_b32 s7, s5
+; GFX10-32-SDAG-NEXT: s_mov_b32 s6, s4
+; GFX10-32-SDAG-NEXT: s_mov_b32 s5, s3
+; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; GFX10-32-SDAG-NEXT: s_mov_b32 s4, s2
+; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[4:5]
+; GFX10-32-SDAG-NEXT: v_add_co_u32 v0, vcc_lo, s0, v0
+; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, s1, v1, vcc_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-SDAG-NEXT: s_clause 0x1
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[11:14], v[0:1], off offset:16
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[7:10], v[0:1], off
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: .LBB2_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s2, v4
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s2, v4
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s12, v7
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s13, v8
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s14, v9
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s15, v10
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s16, v11
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s17, v12
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s18, v13
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s19, v14
+; GFX10-32-SDAG-NEXT: image_sample v[0:3], v6, s[12:19], s[4:7] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB2_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-SDAG-NEXT: s_and_b32 exec_lo, exec_lo, s8
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-32-GISEL-LABEL: test_waterfall_non_uniform_img_single_read:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: s_mov_b32 s6, exec_lo
+; GFX10-32-GISEL-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX10-32-GISEL-NEXT: s_mov_b32 s8, s2
+; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; GFX10-32-GISEL-NEXT: s_mov_b32 s9, s3
+; GFX10-32-GISEL-NEXT: s_mov_b32 s10, s4
+; GFX10-32-GISEL-NEXT: s_mov_b32 s11, s5
+; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[4:5]
+; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-GISEL-NEXT: v_add_co_u32 v0, vcc_lo, v2, v0
+; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
+; GFX10-32-GISEL-NEXT: s_clause 0x1
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[7:10], v[0:1], off
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[11:14], v[0:1], off offset:16
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: .LBB2_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v4
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s2, v4
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s12, v7
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s13, v8
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s14, v9
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s15, v10
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s16, v11
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s17, v12
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s18, v13
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s19, v14
+; GFX10-32-GISEL-NEXT: image_sample v[0:3], v6, s[12:19], s[8:11] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB2_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-GISEL-NEXT: s_and_b32 exec_lo, exec_lo, s6
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-64-SDAG-LABEL: test_waterfall_non_uniform_img_single_read:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX10-64-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-64-SDAG-NEXT: s_mov_b32 s7, s5
+; GFX10-64-SDAG-NEXT: s_mov_b32 s6, s4
+; GFX10-64-SDAG-NEXT: s_mov_b32 s5, s3
+; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; GFX10-64-SDAG-NEXT: s_mov_b32 s4, s2
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[4:5]
+; GFX10-64-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc, s1, v1, vcc
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-SDAG-NEXT: s_clause 0x1
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[11:14], v[0:1], off offset:16
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[7:10], v[0:1], off
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: .LBB2_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s10, v4
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v4
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s12, v7
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s13, v8
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s14, v9
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s15, v10
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s16, v11
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s17, v12
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s18, v13
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s19, v14
+; GFX10-64-SDAG-NEXT: image_sample v[0:3], v6, s[12:19], s[4:7] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB2_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-64-GISEL-LABEL: test_waterfall_non_uniform_img_single_read:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX10-64-GISEL-NEXT: s_mov_b32 s8, s2
+; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; GFX10-64-GISEL-NEXT: s_mov_b32 s9, s3
+; GFX10-64-GISEL-NEXT: s_mov_b32 s10, s4
+; GFX10-64-GISEL-NEXT: s_mov_b32 s11, s5
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[4:5]
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc, v3, v1, vcc
+; GFX10-64-GISEL-NEXT: s_clause 0x1
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[7:10], v[0:1], off
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[11:14], v[0:1], off offset:16
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: .LBB2_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s4, v4
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v4
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s12, v7
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s13, v8
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s14, v9
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s15, v10
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s16, v11
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s17, v12
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s18, v13
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s19, v14
+; GFX10-64-GISEL-NEXT: image_sample v[0:3], v6, s[12:19], s[8:11] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB2_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX1150-SDAG-LABEL: test_waterfall_non_uniform_img_single_read:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX1150-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX1150-SDAG-NEXT: s_mov_b32 s7, s5
+; GFX1150-SDAG-NEXT: s_mov_b32 s6, s4
+; GFX1150-SDAG-NEXT: s_mov_b32 s5, s3
+; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; GFX1150-SDAG-NEXT: s_mov_b32 s4, s2
+; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[4:5]
+; GFX1150-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s1, v1, vcc
+; GFX1150-SDAG-NEXT: s_clause 0x1
+; GFX1150-SDAG-NEXT: global_load_b128 v[11:14], v[0:1], off offset:16
+; GFX1150-SDAG-NEXT: global_load_b128 v[7:10], v[0:1], off
+; GFX1150-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: .LBB2_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s10, v4
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v4
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s12, v7
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s13, v8
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s14, v9
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s15, v10
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s16, v11
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s17, v12
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s18, v13
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s19, v14
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: image_sample v[0:3], v6, s[12:19], s[4:7] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB2_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX1150-GISEL-LABEL: test_waterfall_non_uniform_img_single_read:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX1150-GISEL-NEXT: s_mov_b32 s8, s2
+; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; GFX1150-GISEL-NEXT: s_mov_b32 s9, s3
+; GFX1150-GISEL-NEXT: s_mov_b32 s10, s4
+; GFX1150-GISEL-NEXT: s_mov_b32 s11, s5
+; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[4:5]
+; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX1150-GISEL-NEXT: s_clause 0x1
+; GFX1150-GISEL-NEXT: global_load_b128 v[7:10], v[0:1], off
+; GFX1150-GISEL-NEXT: global_load_b128 v[11:14], v[0:1], off offset:16
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: .LBB2_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s4, v4
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v4
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s12, v7
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s13, v8
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s14, v9
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s15, v10
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s16, v11
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s17, v12
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s18, v13
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s19, v14
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: image_sample v[0:3], v6, s[12:19], s[8:11] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB2_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX12-SDAG-LABEL: test_waterfall_non_uniform_img_single_read:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX12-SDAG-NEXT: s_mov_b32 s7, s5
+; GFX12-SDAG-NEXT: s_mov_b32 s6, s4
+; GFX12-SDAG-NEXT: s_mov_b32 s5, s3
+; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; GFX12-SDAG-NEXT: s_mov_b32 s4, s2
+; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[0:1], 5, v[4:5]
+; GFX12-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s1, v1, vcc
+; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-SDAG-NEXT: s_clause 0x1
+; GFX12-SDAG-NEXT: global_load_b128 v[11:14], v[0:1], off offset:16
+; GFX12-SDAG-NEXT: global_load_b128 v[7:10], v[0:1], off
+; GFX12-SDAG-NEXT: .LBB2_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s10, v4
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v4
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x0
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s12, v7
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s13, v8
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s14, v9
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s15, v10
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s16, v11
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s17, v12
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s18, v13
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s19, v14
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: image_sample v[0:3], v6, s[12:19], s[4:7] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB2_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX12-GISEL-LABEL: test_waterfall_non_uniform_img_single_read:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX12-GISEL-NEXT: s_mov_b32 s8, s2
+; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; GFX12-GISEL-NEXT: s_mov_b32 s9, s3
+; GFX12-GISEL-NEXT: s_mov_b32 s10, s4
+; GFX12-GISEL-NEXT: s_mov_b32 s11, s5
+; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[0:1], 5, v[4:5]
+; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX12-GISEL-NEXT: s_clause 0x1
+; GFX12-GISEL-NEXT: global_load_b128 v[7:10], v[0:1], off
+; GFX12-GISEL-NEXT: global_load_b128 v[11:14], v[0:1], off offset:16
+; GFX12-GISEL-NEXT: .LBB2_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s4, v4
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v4
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s12, v7
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s13, v8
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s14, v9
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s15, v10
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s16, v11
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s17, v12
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s18, v13
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s19, v14
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: image_sample v[0:3], v6, s[12:19], s[8:11] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB2_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: ; return to shader part epilog
+ %ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %index
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
+ %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %wf_token, <8 x i32> %rsrc)
+ %r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %s_rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token, <4 x float> %r)
+
+ ret <4 x float> %r1
+}
+
+define amdgpu_ps void @test_multiple_groups(i32 addrspace(1)* inreg %out1, i32 addrspace(1)* inreg %out2, i32 %idx1, i32 %idx2, i32 %val) #1 {
+; VI-SDAG-LABEL: test_multiple_groups:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: v_mov_b32_e32 v3, s2
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, s3
+; VI-SDAG-NEXT: v_mov_b32_e32 v5, s0
+; VI-SDAG-NEXT: v_mov_b32_e32 v6, s1
+; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; VI-SDAG-NEXT: .LBB3_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s4, v0
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s4, v0
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-SDAG-NEXT: s_nop 1
+; VI-SDAG-NEXT: v_readlane_b32 s4, v2, s4
+; VI-SDAG-NEXT: v_mov_b32_e32 v7, s4
+; VI-SDAG-NEXT: ; implicit-def: $vgpr0
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB3_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; VI-SDAG-NEXT: flat_store_dword v[5:6], v7
+; VI-SDAG-NEXT: .LBB3_3: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s4, v1
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s4, v1
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-SDAG-NEXT: s_nop 1
+; VI-SDAG-NEXT: v_readlane_b32 s4, v2, s4
+; VI-SDAG-NEXT: v_mov_b32_e32 v0, s4
+; VI-SDAG-NEXT: ; implicit-def: $vgpr1
+; VI-SDAG-NEXT: ; implicit-def: $vgpr2
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB3_3
+; VI-SDAG-NEXT: ; %bb.4:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; VI-SDAG-NEXT: flat_store_dword v[3:4], v0
+; VI-SDAG-NEXT: s_endpgm
+;
+; VI-GISEL-LABEL: test_multiple_groups:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; VI-GISEL-NEXT: .LBB3_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v0
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[6:7], s8, v0
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[6:7], s[6:7]
+; VI-GISEL-NEXT: s_nop 1
+; VI-GISEL-NEXT: v_readlane_b32 s8, v2, s8
+; VI-GISEL-NEXT: v_mov_b32_e32 v3, s8
+; VI-GISEL-NEXT: ; implicit-def: $vgpr0
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[6:7]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB3_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[4:5]
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, s1
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, s0
+; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; VI-GISEL-NEXT: flat_store_dword v[4:5], v3
+; VI-GISEL-NEXT: .LBB3_3: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s6, v1
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s6, v1
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
+; VI-GISEL-NEXT: s_nop 1
+; VI-GISEL-NEXT: v_readlane_b32 s6, v2, s6
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s6
+; VI-GISEL-NEXT: ; implicit-def: $vgpr1
+; VI-GISEL-NEXT: ; implicit-def: $vgpr2
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[4:5]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB3_3
+; VI-GISEL-NEXT: ; %bb.4:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s2
+; VI-GISEL-NEXT: v_mov_b32_e32 v2, s3
+; VI-GISEL-NEXT: flat_store_dword v[1:2], v0
+; VI-GISEL-NEXT: s_endpgm
+;
+; GFX9-SDAG-LABEL: test_multiple_groups:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v3, 0
+; GFX9-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-SDAG-NEXT: .LBB3_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v0
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[6:7], s8, v0
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[6:7], s[6:7]
+; GFX9-SDAG-NEXT: s_nop 1
+; GFX9-SDAG-NEXT: v_readlane_b32 s8, v2, s8
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v4, s8
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[6:7]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB3_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[4:5]
+; GFX9-SDAG-NEXT: global_store_dword v3, v4, s[0:1]
+; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX9-SDAG-NEXT: .LBB3_3: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s6, v1
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[4:5], s6, v1
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
+; GFX9-SDAG-NEXT: s_nop 1
+; GFX9-SDAG-NEXT: v_readlane_b32 s6, v2, s6
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v0, s6
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr1
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr2
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[4:5]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB3_3
+; GFX9-SDAG-NEXT: ; %bb.4:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX9-SDAG-NEXT: global_store_dword v3, v0, s[2:3]
+; GFX9-SDAG-NEXT: s_endpgm
+;
+; GFX9-GISEL-LABEL: test_multiple_groups:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-GISEL-NEXT: .LBB3_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v0
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[6:7], s8, v0
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[6:7], s[6:7]
+; GFX9-GISEL-NEXT: s_nop 1
+; GFX9-GISEL-NEXT: v_readlane_b32 s8, v2, s8
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v3, s8
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[6:7]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB3_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[4:5]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX9-GISEL-NEXT: global_store_dword v0, v3, s[0:1]
+; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX9-GISEL-NEXT: .LBB3_3: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s6, v1
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s6, v1
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
+; GFX9-GISEL-NEXT: s_nop 1
+; GFX9-GISEL-NEXT: v_readlane_b32 s6, v2, s6
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v3, s6
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr1
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr2
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[4:5]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB3_3
+; GFX9-GISEL-NEXT: ; %bb.4:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX9-GISEL-NEXT: global_store_dword v0, v3, s[2:3]
+; GFX9-GISEL-NEXT: s_endpgm
+;
+; GFX10-32-SDAG-LABEL: test_multiple_groups:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v3, 0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s5, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-SDAG-NEXT: .LBB3_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s6, v0
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s6, v0
+; GFX10-32-SDAG-NEXT: v_readlane_b32 s6, v2, s6
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v4, s6
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s5, s5
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB3_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s4
+; GFX10-32-SDAG-NEXT: global_store_dword v3, v4, s[0:1]
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-SDAG-NEXT: .LBB3_3: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s4, v1
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s4, v1
+; GFX10-32-SDAG-NEXT: v_readlane_b32 s4, v2, s4
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v0, s4
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr1
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr2
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB3_3
+; GFX10-32-SDAG-NEXT: ; %bb.4:
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-SDAG-NEXT: global_store_dword v3, v0, s[2:3]
+; GFX10-32-SDAG-NEXT: s_endpgm
+;
+; GFX10-32-GISEL-LABEL: test_multiple_groups:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: s_mov_b32 s5, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-GISEL-NEXT: .LBB3_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s6, v0
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s6, v0
+; GFX10-32-GISEL-NEXT: v_readlane_b32 s6, v2, s6
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v3, s6
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s5, s5
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB3_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s4
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-GISEL-NEXT: global_store_dword v0, v3, s[0:1]
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-GISEL-NEXT: .LBB3_3: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s4, v1
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s4, v1
+; GFX10-32-GISEL-NEXT: v_readlane_b32 s4, v2, s4
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v3, s4
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr1
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr2
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB3_3
+; GFX10-32-GISEL-NEXT: ; %bb.4:
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-GISEL-NEXT: global_store_dword v0, v3, s[2:3]
+; GFX10-32-GISEL-NEXT: s_endpgm
+;
+; GFX10-64-SDAG-LABEL: test_multiple_groups:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v3, 0
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-SDAG-NEXT: .LBB3_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s8, v0
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s8, v0
+; GFX10-64-SDAG-NEXT: v_readlane_b32 s8, v2, s8
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v4, s8
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[6:7], s[6:7]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB3_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[4:5]
+; GFX10-64-SDAG-NEXT: global_store_dword v3, v4, s[0:1]
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-SDAG-NEXT: .LBB3_3: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s6, v1
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v1
+; GFX10-64-SDAG-NEXT: v_readlane_b32 s6, v2, s6
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v0, s6
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[4:5], s[4:5]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr1
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr2
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB3_3
+; GFX10-64-SDAG-NEXT: ; %bb.4:
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-SDAG-NEXT: global_store_dword v3, v0, s[2:3]
+; GFX10-64-SDAG-NEXT: s_endpgm
+;
+; GFX10-64-GISEL-LABEL: test_multiple_groups:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-GISEL-NEXT: .LBB3_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s8, v0
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s8, v0
+; GFX10-64-GISEL-NEXT: v_readlane_b32 s8, v2, s8
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v3, s8
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[6:7], s[6:7]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB3_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[4:5]
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-GISEL-NEXT: global_store_dword v0, v3, s[0:1]
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-GISEL-NEXT: .LBB3_3: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s6, v1
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s6, v1
+; GFX10-64-GISEL-NEXT: v_readlane_b32 s6, v2, s6
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v3, s6
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[4:5], s[4:5]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr1
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr2
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB3_3
+; GFX10-64-GISEL-NEXT: ; %bb.4:
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-GISEL-NEXT: global_store_dword v0, v3, s[2:3]
+; GFX10-64-GISEL-NEXT: s_endpgm
+;
+; GFX1150-SDAG-LABEL: test_multiple_groups:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v3, 0
+; GFX1150-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-SDAG-NEXT: .LBB3_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s8, v0
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s8, v0
+; GFX1150-SDAG-NEXT: v_readlane_b32 s8, v2, s8
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v4, s8
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[6:7], s[6:7]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB3_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[4:5]
+; GFX1150-SDAG-NEXT: global_store_b32 v3, v4, s[0:1]
+; GFX1150-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-SDAG-NEXT: .LBB3_3: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s6, v1
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v1
+; GFX1150-SDAG-NEXT: v_readlane_b32 s6, v2, s6
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v0, s6
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[4:5], s[4:5]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr1
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr2
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB3_3
+; GFX1150-SDAG-NEXT: ; %bb.4:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-SDAG-NEXT: global_store_b32 v3, v0, s[2:3]
+; GFX1150-SDAG-NEXT: s_endpgm
+;
+; GFX1150-GISEL-LABEL: test_multiple_groups:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-GISEL-NEXT: .LBB3_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s8, v0
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s8, v0
+; GFX1150-GISEL-NEXT: v_readlane_b32 s8, v2, s8
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v3, s8
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[6:7], s[6:7]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB3_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[4:5]
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-GISEL-NEXT: global_store_b32 v0, v3, s[0:1]
+; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-GISEL-NEXT: .LBB3_3: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s6, v1
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s6, v1
+; GFX1150-GISEL-NEXT: v_readlane_b32 s6, v2, s6
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v3, s6
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[4:5], s[4:5]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr1
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr2
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB3_3
+; GFX1150-GISEL-NEXT: ; %bb.4:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-GISEL-NEXT: global_store_b32 v0, v3, s[2:3]
+; GFX1150-GISEL-NEXT: s_endpgm
+;
+; GFX12-SDAG-LABEL: test_multiple_groups:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v3, 0
+; GFX12-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-SDAG-NEXT: .LBB3_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s8, v0
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s8, v0
+; GFX12-SDAG-NEXT: v_readlane_b32 s8, v2, s8
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v4, s8
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[6:7], s[6:7]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB3_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[4:5]
+; GFX12-SDAG-NEXT: global_store_b32 v3, v4, s[0:1]
+; GFX12-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-SDAG-NEXT: .LBB3_3: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s6, v1
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v1
+; GFX12-SDAG-NEXT: v_readlane_b32 s6, v2, s6
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v0, s6
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[4:5], s[4:5]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr1
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr2
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB3_3
+; GFX12-SDAG-NEXT: ; %bb.4:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-SDAG-NEXT: global_store_b32 v3, v0, s[2:3]
+; GFX12-SDAG-NEXT: s_endpgm
+;
+; GFX12-GISEL-LABEL: test_multiple_groups:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-GISEL-NEXT: .LBB3_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s8, v0
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s8, v0
+; GFX12-GISEL-NEXT: v_readlane_b32 s8, v2, s8
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v3, s8
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[6:7], s[6:7]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB3_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[4:5]
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-GISEL-NEXT: global_store_b32 v0, v3, s[0:1]
+; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-GISEL-NEXT: .LBB3_3: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s6, v1
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s6, v1
+; GFX12-GISEL-NEXT: v_readlane_b32 s6, v2, s6
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v3, s6
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[4:5], s[4:5]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr1
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr2
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB3_3
+; GFX12-GISEL-NEXT: ; %bb.4:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-GISEL-NEXT: global_store_b32 v0, v3, s[2:3]
+; GFX12-GISEL-NEXT: s_endpgm
+ %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
+ %readlane1 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %idx1)
+ %readlane1.1 = call i32 @llvm.amdgcn.readlane(i32 %val, i32 %readlane1)
+ %readlane1.2 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %wf_token, i32 %readlane1.1)
+ ; This store instruction should be outside the waterfall loop and the value
+ ; being stored generated incrementally in the loop itself
+ store i32 %readlane1.2, i32 addrspace(1)* %out1, align 4
+
+ %wf_token2 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx2)
+ %readlane2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token2, i32 %idx2)
+ %readlane2.1 = call i32 @llvm.amdgcn.readlane(i32 %val, i32 %readlane2)
+ %readlane2.2 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %wf_token2, i32 %readlane2.1)
+ store i32 %readlane2.2, i32 addrspace(1)* %out2, align 4
+
+ ret void
+}
+
+
+define amdgpu_ps <4 x float> @test_waterfall_non_uniform_img_multi_rl(<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %samp_in, i32 %index, float %s, i32 %val) #1 {
+; VI-SDAG-LABEL: test_waterfall_non_uniform_img_multi_rl:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; VI-SDAG-NEXT: s_wqm_b64 exec, exec
+; VI-SDAG-NEXT: v_mov_b32_e32 v5, v2
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; VI-SDAG-NEXT: v_mov_b32_e32 v6, v0
+; VI-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; VI-SDAG-NEXT: .LBB4_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v6
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[10:11], s8, v6
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[20:21], s[10:11]
+; VI-SDAG-NEXT: s_ashr_i32 s9, s8, 31
+; VI-SDAG-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; VI-SDAG-NEXT: v_readfirstlane_b32 s10, v5
+; VI-SDAG-NEXT: s_add_u32 s8, s0, s8
+; VI-SDAG-NEXT: s_addc_u32 s9, s1, s9
+; VI-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; VI-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 4
+; VI-SDAG-NEXT: s_add_u32 s16, s2, s10
+; VI-SDAG-NEXT: s_addc_u32 s17, s3, s11
+; VI-SDAG-NEXT: s_load_dwordx8 s[8:15], s[8:9], 0x0
+; VI-SDAG-NEXT: s_load_dwordx4 s[16:19], s[16:17], 0x0
+; VI-SDAG-NEXT: ; implicit-def: $vgpr6
+; VI-SDAG-NEXT: ; implicit-def: $vgpr5
+; VI-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; VI-SDAG-NEXT: image_sample v[0:3], v4, s[8:15], s[16:19] dmask:0xf
+; VI-SDAG-NEXT: ; implicit-def: $vgpr4
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[20:21]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB4_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[6:7]
+; VI-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: ; return to shader part epilog
+;
+; VI-GISEL-LABEL: test_waterfall_non_uniform_img_multi_rl:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; VI-GISEL-NEXT: s_wqm_b64 exec, exec
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; VI-GISEL-NEXT: v_mov_b32_e32 v6, v2
+; VI-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; VI-GISEL-NEXT: .LBB4_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s8, v5
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[20:21], s[10:11]
+; VI-GISEL-NEXT: s_ashr_i32 s9, s8, 31
+; VI-GISEL-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v6
+; VI-GISEL-NEXT: s_add_u32 s8, s0, s8
+; VI-GISEL-NEXT: s_addc_u32 s9, s1, s9
+; VI-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; VI-GISEL-NEXT: s_lshl_b64 s[10:11], s[10:11], 4
+; VI-GISEL-NEXT: s_add_u32 s16, s2, s10
+; VI-GISEL-NEXT: s_addc_u32 s17, s3, s11
+; VI-GISEL-NEXT: s_load_dwordx8 s[8:15], s[8:9], 0x0
+; VI-GISEL-NEXT: s_load_dwordx4 s[16:19], s[16:17], 0x0
+; VI-GISEL-NEXT: ; implicit-def: $vgpr5
+; VI-GISEL-NEXT: ; implicit-def: $vgpr6
+; VI-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; VI-GISEL-NEXT: image_sample v[0:3], v4, s[8:15], s[16:19] dmask:0xf
+; VI-GISEL-NEXT: ; implicit-def: $vgpr4
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[20:21]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB4_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[6:7]
+; VI-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX9-SDAG-LABEL: test_waterfall_non_uniform_img_multi_rl:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v5, v2
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v6, v0
+; GFX9-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX9-SDAG-NEXT: .LBB4_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v6
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[10:11], s8, v6
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[20:21], s[10:11]
+; GFX9-SDAG-NEXT: s_ashr_i32 s9, s8, 31
+; GFX9-SDAG-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s10, v5
+; GFX9-SDAG-NEXT: s_add_u32 s22, s0, s8
+; GFX9-SDAG-NEXT: s_addc_u32 s23, s1, s9
+; GFX9-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; GFX9-SDAG-NEXT: s_lshl_b64 s[8:9], s[10:11], 4
+; GFX9-SDAG-NEXT: s_add_u32 s24, s2, s8
+; GFX9-SDAG-NEXT: s_addc_u32 s25, s3, s9
+; GFX9-SDAG-NEXT: s_load_dwordx8 s[8:15], s[22:23], 0x0
+; GFX9-SDAG-NEXT: s_load_dwordx4 s[16:19], s[24:25], 0x0
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX9-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-SDAG-NEXT: image_sample v[0:3], v4, s[8:15], s[16:19] dmask:0xf
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[20:21]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB4_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[6:7]
+; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX9-GISEL-LABEL: test_waterfall_non_uniform_img_multi_rl:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v6, v2
+; GFX9-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX9-GISEL-NEXT: .LBB4_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s8, v5
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[20:21], s[10:11]
+; GFX9-GISEL-NEXT: s_ashr_i32 s9, s8, 31
+; GFX9-GISEL-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v6
+; GFX9-GISEL-NEXT: s_add_u32 s22, s0, s8
+; GFX9-GISEL-NEXT: s_addc_u32 s23, s1, s9
+; GFX9-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; GFX9-GISEL-NEXT: s_lshl_b64 s[8:9], s[10:11], 4
+; GFX9-GISEL-NEXT: s_add_u32 s24, s2, s8
+; GFX9-GISEL-NEXT: s_addc_u32 s25, s3, s9
+; GFX9-GISEL-NEXT: s_load_dwordx8 s[8:15], s[22:23], 0x0
+; GFX9-GISEL-NEXT: s_load_dwordx4 s[16:19], s[24:25], 0x0
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX9-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-GISEL-NEXT: image_sample v[0:3], v4, s[8:15], s[16:19] dmask:0xf
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[20:21]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB4_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[6:7]
+; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-32-SDAG-LABEL: test_waterfall_non_uniform_img_multi_rl:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-SDAG-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v5, v2
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v6, v0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s6, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s5, exec_lo
+; GFX10-32-SDAG-NEXT: .LBB4_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s8, v6
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s8, v6
+; GFX10-32-SDAG-NEXT: s_ashr_i32 s9, s8, 31
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s10, v5
+; GFX10-32-SDAG-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; GFX10-32-SDAG-NEXT: s_add_u32 s20, s0, s8
+; GFX10-32-SDAG-NEXT: s_addc_u32 s21, s1, s9
+; GFX10-32-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; GFX10-32-SDAG-NEXT: s_lshl_b64 s[8:9], s[10:11], 4
+; GFX10-32-SDAG-NEXT: s_add_u32 s22, s2, s8
+; GFX10-32-SDAG-NEXT: s_addc_u32 s23, s3, s9
+; GFX10-32-SDAG-NEXT: s_load_dwordx8 s[8:15], s[20:21], 0x0
+; GFX10-32-SDAG-NEXT: s_load_dwordx4 s[16:19], s[22:23], 0x0
+; GFX10-32-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-SDAG-NEXT: image_sample v[0:3], v4, s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s6, s6
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB4_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s5
+; GFX10-32-SDAG-NEXT: s_and_b32 exec_lo, exec_lo, s4
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-32-GISEL-LABEL: test_waterfall_non_uniform_img_multi_rl:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-GISEL-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v6, v2
+; GFX10-32-GISEL-NEXT: s_mov_b32 s6, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s5, exec_lo
+; GFX10-32-GISEL-NEXT: .LBB4_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s8, v5
+; GFX10-32-GISEL-NEXT: s_ashr_i32 s9, s8, 31
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s10, v6
+; GFX10-32-GISEL-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; GFX10-32-GISEL-NEXT: s_add_u32 s20, s0, s8
+; GFX10-32-GISEL-NEXT: s_addc_u32 s21, s1, s9
+; GFX10-32-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; GFX10-32-GISEL-NEXT: s_lshl_b64 s[8:9], s[10:11], 4
+; GFX10-32-GISEL-NEXT: s_add_u32 s22, s2, s8
+; GFX10-32-GISEL-NEXT: s_addc_u32 s23, s3, s9
+; GFX10-32-GISEL-NEXT: s_load_dwordx8 s[8:15], s[20:21], 0x0
+; GFX10-32-GISEL-NEXT: s_load_dwordx4 s[16:19], s[22:23], 0x0
+; GFX10-32-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-GISEL-NEXT: image_sample v[0:3], v4, s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s6, s6
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB4_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s5
+; GFX10-32-GISEL-NEXT: s_and_b32 exec_lo, exec_lo, s4
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-64-SDAG-LABEL: test_waterfall_non_uniform_img_multi_rl:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v5, v2
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v6, v0
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-SDAG-NEXT: .LBB4_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s10, v6
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v6
+; GFX10-64-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; GFX10-64-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX10-64-SDAG-NEXT: s_add_u32 s10, s0, s10
+; GFX10-64-SDAG-NEXT: s_addc_u32 s11, s1, s11
+; GFX10-64-SDAG-NEXT: s_ashr_i32 s13, s12, 31
+; GFX10-64-SDAG-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX10-64-SDAG-NEXT: s_add_u32 s24, s2, s12
+; GFX10-64-SDAG-NEXT: s_addc_u32 s25, s3, s13
+; GFX10-64-SDAG-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; GFX10-64-SDAG-NEXT: s_load_dwordx4 s[20:23], s[24:25], 0x0
+; GFX10-64-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-SDAG-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[8:9], s[8:9]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB4_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[6:7]
+; GFX10-64-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-64-GISEL-LABEL: test_waterfall_non_uniform_img_multi_rl:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v6, v2
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-GISEL-NEXT: .LBB4_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s10, v5
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s10, v5
+; GFX10-64-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s12, v6
+; GFX10-64-GISEL-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX10-64-GISEL-NEXT: s_add_u32 s10, s0, s10
+; GFX10-64-GISEL-NEXT: s_addc_u32 s11, s1, s11
+; GFX10-64-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; GFX10-64-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX10-64-GISEL-NEXT: s_add_u32 s24, s2, s12
+; GFX10-64-GISEL-NEXT: s_addc_u32 s25, s3, s13
+; GFX10-64-GISEL-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; GFX10-64-GISEL-NEXT: s_load_dwordx4 s[20:23], s[24:25], 0x0
+; GFX10-64-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-GISEL-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[8:9], s[8:9]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB4_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[6:7]
+; GFX10-64-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX1150-SDAG-LABEL: test_waterfall_non_uniform_img_multi_rl:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v5, v2
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v6, v0
+; GFX1150-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX1150-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-SDAG-NEXT: .LBB4_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s10, v6
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v6
+; GFX1150-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; GFX1150-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_2) | instid1(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_add_u32 s10, s0, s10
+; GFX1150-SDAG-NEXT: s_addc_u32 s11, s1, s11
+; GFX1150-SDAG-NEXT: s_ashr_i32 s13, s12, 31
+; GFX1150-SDAG-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_add_u32 s20, s2, s12
+; GFX1150-SDAG-NEXT: s_addc_u32 s21, s3, s13
+; GFX1150-SDAG-NEXT: s_load_b256 s[12:19], s[10:11], 0x0
+; GFX1150-SDAG-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX1150-SDAG-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB4_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[6:7]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX1150-GISEL-LABEL: test_waterfall_non_uniform_img_multi_rl:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v6, v2
+; GFX1150-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-GISEL-NEXT: .LBB4_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s10, v5
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_3) | instid1(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s10, v5
+; GFX1150-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s12, v6
+; GFX1150-GISEL-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX1150-GISEL-NEXT: s_add_u32 s10, s0, s10
+; GFX1150-GISEL-NEXT: s_addc_u32 s11, s1, s11
+; GFX1150-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX1150-GISEL-NEXT: s_add_u32 s20, s2, s12
+; GFX1150-GISEL-NEXT: s_addc_u32 s21, s3, s13
+; GFX1150-GISEL-NEXT: s_load_b256 s[12:19], s[10:11], 0x0
+; GFX1150-GISEL-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX1150-GISEL-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB4_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[6:7]
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX12-SDAG-LABEL: test_waterfall_non_uniform_img_multi_rl:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v5, v2
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v6, v0
+; GFX12-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-SDAG-NEXT: .LBB4_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s10, v6
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v6
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; GFX12-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_add_nc_u64 s[10:11], s[0:1], s[10:11]
+; GFX12-SDAG-NEXT: s_ashr_i32 s13, s12, 31
+; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX12-SDAG-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX12-SDAG-NEXT: s_add_nc_u64 s[20:21], s[2:3], s[12:13]
+; GFX12-SDAG-NEXT: s_load_b256 s[12:19], s[10:11], 0x0
+; GFX12-SDAG-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB4_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[6:7]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX12-GISEL-LABEL: test_waterfall_non_uniform_img_multi_rl:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v6, v2
+; GFX12-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-GISEL-NEXT: .LBB4_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s10, v5
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s10, v5
+; GFX12-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s12, v6
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_add_co_u32 s10, s0, s10
+; GFX12-GISEL-NEXT: s_add_co_ci_u32 s11, s1, s11
+; GFX12-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX12-GISEL-NEXT: s_add_co_u32 s20, s2, s12
+; GFX12-GISEL-NEXT: s_add_co_ci_u32 s21, s3, s13
+; GFX12-GISEL-NEXT: s_load_b256 s[12:19], s[10:11], 0x0
+; GFX12-GISEL-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB4_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[6:7]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: ; return to shader part epilog
+ %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %index)
+ %s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %val)
+ %ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
+ %ptr2 = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %samp_in, i32 %s_idx2
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
+ %samp = load <4 x i32>, <4 x i32> addrspace(4) * %ptr2, align 32
+ %r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token, <4 x float> %r)
+
+ ret <4 x float> %r1
+}
+
+define amdgpu_ps <4 x float> @test_waterfall_non_uni_img_2_idx(<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %samp_in, i32 %index1, i32 %index2, float %s) #1 {
+; VI-SDAG-LABEL: test_waterfall_non_uni_img_2_idx:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; VI-SDAG-NEXT: s_wqm_b64 exec, exec
+; VI-SDAG-NEXT: v_mov_b32_e32 v6, v2
+; VI-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; VI-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; VI-SDAG-NEXT: .LBB5_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s12, v4
+; VI-SDAG-NEXT: v_readfirstlane_b32 s13, v5
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[8:9], s12, v4
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[10:11], s13, v5
+; VI-SDAG-NEXT: s_and_b64 s[8:9], s[8:9], s[10:11]
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[20:21], s[8:9]
+; VI-SDAG-NEXT: s_ashr_i32 s9, s12, 31
+; VI-SDAG-NEXT: s_mov_b32 s8, s12
+; VI-SDAG-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; VI-SDAG-NEXT: s_add_u32 s8, s0, s8
+; VI-SDAG-NEXT: s_addc_u32 s9, s1, s9
+; VI-SDAG-NEXT: s_ashr_i32 s11, s13, 31
+; VI-SDAG-NEXT: s_mov_b32 s10, s13
+; VI-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 4
+; VI-SDAG-NEXT: s_add_u32 s16, s2, s10
+; VI-SDAG-NEXT: s_addc_u32 s17, s3, s11
+; VI-SDAG-NEXT: s_load_dwordx8 s[8:15], s[8:9], 0x0
+; VI-SDAG-NEXT: s_load_dwordx4 s[16:19], s[16:17], 0x0
+; VI-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5
+; VI-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; VI-SDAG-NEXT: image_sample v[0:3], v6, s[8:15], s[16:19] dmask:0xf
+; VI-SDAG-NEXT: ; implicit-def: $vgpr6
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[20:21]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB5_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[6:7]
+; VI-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: ; return to shader part epilog
+;
+; VI-GISEL-LABEL: test_waterfall_non_uni_img_2_idx:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; VI-GISEL-NEXT: s_wqm_b64 exec, exec
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; VI-GISEL-NEXT: v_mov_b32_e32 v6, v2
+; VI-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; VI-GISEL-NEXT: .LBB5_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v4
+; VI-GISEL-NEXT: v_readfirstlane_b32 s13, v5
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s12, v4
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s13, v5
+; VI-GISEL-NEXT: s_and_b64 s[8:9], s[8:9], s[10:11]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[20:21], s[8:9]
+; VI-GISEL-NEXT: s_ashr_i32 s9, s12, 31
+; VI-GISEL-NEXT: s_mov_b32 s8, s12
+; VI-GISEL-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; VI-GISEL-NEXT: s_add_u32 s8, s0, s8
+; VI-GISEL-NEXT: s_addc_u32 s9, s1, s9
+; VI-GISEL-NEXT: s_ashr_i32 s11, s13, 31
+; VI-GISEL-NEXT: s_mov_b32 s10, s13
+; VI-GISEL-NEXT: s_lshl_b64 s[10:11], s[10:11], 4
+; VI-GISEL-NEXT: s_add_u32 s16, s2, s10
+; VI-GISEL-NEXT: s_addc_u32 s17, s3, s11
+; VI-GISEL-NEXT: s_load_dwordx8 s[8:15], s[8:9], 0x0
+; VI-GISEL-NEXT: s_load_dwordx4 s[16:19], s[16:17], 0x0
+; VI-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5
+; VI-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; VI-GISEL-NEXT: image_sample v[0:3], v6, s[8:15], s[16:19] dmask:0xf
+; VI-GISEL-NEXT: ; implicit-def: $vgpr6
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[20:21]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB5_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[6:7]
+; VI-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX9-SDAG-LABEL: test_waterfall_non_uni_img_2_idx:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v6, v2
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX9-SDAG-NEXT: .LBB5_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s12, v4
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s13, v5
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[8:9], s12, v4
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[10:11], s13, v5
+; GFX9-SDAG-NEXT: s_and_b64 s[8:9], s[8:9], s[10:11]
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[20:21], s[8:9]
+; GFX9-SDAG-NEXT: s_ashr_i32 s9, s12, 31
+; GFX9-SDAG-NEXT: s_mov_b32 s8, s12
+; GFX9-SDAG-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; GFX9-SDAG-NEXT: s_add_u32 s22, s0, s8
+; GFX9-SDAG-NEXT: s_addc_u32 s23, s1, s9
+; GFX9-SDAG-NEXT: s_ashr_i32 s9, s13, 31
+; GFX9-SDAG-NEXT: s_mov_b32 s8, s13
+; GFX9-SDAG-NEXT: s_lshl_b64 s[8:9], s[8:9], 4
+; GFX9-SDAG-NEXT: s_add_u32 s24, s2, s8
+; GFX9-SDAG-NEXT: s_addc_u32 s25, s3, s9
+; GFX9-SDAG-NEXT: s_load_dwordx8 s[8:15], s[22:23], 0x0
+; GFX9-SDAG-NEXT: s_load_dwordx4 s[16:19], s[24:25], 0x0
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5
+; GFX9-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-SDAG-NEXT: image_sample v[0:3], v6, s[8:15], s[16:19] dmask:0xf
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[20:21]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB5_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[6:7]
+; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX9-GISEL-LABEL: test_waterfall_non_uni_img_2_idx:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v6, v2
+; GFX9-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX9-GISEL-NEXT: .LBB5_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v4
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s13, v5
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s12, v4
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s13, v5
+; GFX9-GISEL-NEXT: s_and_b64 s[8:9], s[8:9], s[10:11]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[20:21], s[8:9]
+; GFX9-GISEL-NEXT: s_ashr_i32 s9, s12, 31
+; GFX9-GISEL-NEXT: s_mov_b32 s8, s12
+; GFX9-GISEL-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; GFX9-GISEL-NEXT: s_add_u32 s22, s0, s8
+; GFX9-GISEL-NEXT: s_addc_u32 s23, s1, s9
+; GFX9-GISEL-NEXT: s_ashr_i32 s9, s13, 31
+; GFX9-GISEL-NEXT: s_mov_b32 s8, s13
+; GFX9-GISEL-NEXT: s_lshl_b64 s[8:9], s[8:9], 4
+; GFX9-GISEL-NEXT: s_add_u32 s24, s2, s8
+; GFX9-GISEL-NEXT: s_addc_u32 s25, s3, s9
+; GFX9-GISEL-NEXT: s_load_dwordx8 s[8:15], s[22:23], 0x0
+; GFX9-GISEL-NEXT: s_load_dwordx4 s[16:19], s[24:25], 0x0
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5
+; GFX9-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-GISEL-NEXT: image_sample v[0:3], v6, s[8:15], s[16:19] dmask:0xf
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[20:21]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB5_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[6:7]
+; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-32-SDAG-LABEL: test_waterfall_non_uni_img_2_idx:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-SDAG-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v6, v2
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s6, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s5, exec_lo
+; GFX10-32-SDAG-NEXT: .LBB5_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s7, v4
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s10, v5
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s7, v4
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s10, v5
+; GFX10-32-SDAG-NEXT: s_ashr_i32 s9, s7, 31
+; GFX10-32-SDAG-NEXT: s_mov_b32 s8, s7
+; GFX10-32-SDAG-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; GFX10-32-SDAG-NEXT: s_add_u32 s20, s0, s8
+; GFX10-32-SDAG-NEXT: s_addc_u32 s21, s1, s9
+; GFX10-32-SDAG-NEXT: s_ashr_i32 s9, s10, 31
+; GFX10-32-SDAG-NEXT: s_mov_b32 s8, s10
+; GFX10-32-SDAG-NEXT: s_lshl_b64 s[8:9], s[8:9], 4
+; GFX10-32-SDAG-NEXT: s_add_u32 s22, s2, s8
+; GFX10-32-SDAG-NEXT: s_addc_u32 s23, s3, s9
+; GFX10-32-SDAG-NEXT: s_load_dwordx8 s[8:15], s[20:21], 0x0
+; GFX10-32-SDAG-NEXT: s_load_dwordx4 s[16:19], s[22:23], 0x0
+; GFX10-32-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-SDAG-NEXT: image_sample v[0:3], v6, s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s6, s6
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB5_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s5
+; GFX10-32-SDAG-NEXT: s_and_b32 exec_lo, exec_lo, s4
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-32-GISEL-LABEL: test_waterfall_non_uni_img_2_idx:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-GISEL-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v6, v2
+; GFX10-32-GISEL-NEXT: s_mov_b32 s6, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s5, exec_lo
+; GFX10-32-GISEL-NEXT: .LBB5_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s7, v4
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s10, v5
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s7, v4
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s10, v5
+; GFX10-32-GISEL-NEXT: s_ashr_i32 s9, s7, 31
+; GFX10-32-GISEL-NEXT: s_mov_b32 s8, s7
+; GFX10-32-GISEL-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; GFX10-32-GISEL-NEXT: s_add_u32 s20, s0, s8
+; GFX10-32-GISEL-NEXT: s_addc_u32 s21, s1, s9
+; GFX10-32-GISEL-NEXT: s_ashr_i32 s9, s10, 31
+; GFX10-32-GISEL-NEXT: s_mov_b32 s8, s10
+; GFX10-32-GISEL-NEXT: s_lshl_b64 s[8:9], s[8:9], 4
+; GFX10-32-GISEL-NEXT: s_add_u32 s22, s2, s8
+; GFX10-32-GISEL-NEXT: s_addc_u32 s23, s3, s9
+; GFX10-32-GISEL-NEXT: s_load_dwordx8 s[8:15], s[20:21], 0x0
+; GFX10-32-GISEL-NEXT: s_load_dwordx4 s[16:19], s[22:23], 0x0
+; GFX10-32-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-GISEL-NEXT: image_sample v[0:3], v6, s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s6, s6
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB5_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s5
+; GFX10-32-GISEL-NEXT: s_and_b32 exec_lo, exec_lo, s4
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-64-SDAG-LABEL: test_waterfall_non_uni_img_2_idx:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v6, v2
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-SDAG-NEXT: .LBB5_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s10, v4
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v4
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX10-64-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; GFX10-64-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX10-64-SDAG-NEXT: s_add_u32 s10, s0, s10
+; GFX10-64-SDAG-NEXT: s_addc_u32 s11, s1, s11
+; GFX10-64-SDAG-NEXT: s_ashr_i32 s13, s12, 31
+; GFX10-64-SDAG-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX10-64-SDAG-NEXT: s_add_u32 s24, s2, s12
+; GFX10-64-SDAG-NEXT: s_addc_u32 s25, s3, s13
+; GFX10-64-SDAG-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; GFX10-64-SDAG-NEXT: s_load_dwordx4 s[20:23], s[24:25], 0x0
+; GFX10-64-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-SDAG-NEXT: image_sample v[0:3], v6, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[8:9], s[8:9]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB5_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[6:7]
+; GFX10-64-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-64-GISEL-LABEL: test_waterfall_non_uni_img_2_idx:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v6, v2
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-GISEL-NEXT: .LBB5_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s10, v4
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s12, v5
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s10, v4
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX10-64-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; GFX10-64-GISEL-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX10-64-GISEL-NEXT: s_add_u32 s10, s0, s10
+; GFX10-64-GISEL-NEXT: s_addc_u32 s11, s1, s11
+; GFX10-64-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; GFX10-64-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX10-64-GISEL-NEXT: s_add_u32 s24, s2, s12
+; GFX10-64-GISEL-NEXT: s_addc_u32 s25, s3, s13
+; GFX10-64-GISEL-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; GFX10-64-GISEL-NEXT: s_load_dwordx4 s[20:23], s[24:25], 0x0
+; GFX10-64-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-GISEL-NEXT: image_sample v[0:3], v6, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[8:9], s[8:9]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB5_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[6:7]
+; GFX10-64-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX1150-SDAG-LABEL: test_waterfall_non_uni_img_2_idx:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v6, v2
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX1150-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX1150-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-SDAG-NEXT: .LBB5_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s10, v4
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v4
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX1150-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; GFX1150-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_2) | instid1(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_add_u32 s10, s0, s10
+; GFX1150-SDAG-NEXT: s_addc_u32 s11, s1, s11
+; GFX1150-SDAG-NEXT: s_ashr_i32 s13, s12, 31
+; GFX1150-SDAG-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_add_u32 s20, s2, s12
+; GFX1150-SDAG-NEXT: s_addc_u32 s21, s3, s13
+; GFX1150-SDAG-NEXT: s_load_b256 s[12:19], s[10:11], 0x0
+; GFX1150-SDAG-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX1150-SDAG-NEXT: image_sample v[0:3], v6, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB5_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[6:7]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX1150-GISEL-LABEL: test_waterfall_non_uni_img_2_idx:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v6, v2
+; GFX1150-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-GISEL-NEXT: .LBB5_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s10, v4
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s12, v5
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s10, v4
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX1150-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX1150-GISEL-NEXT: s_add_u32 s10, s0, s10
+; GFX1150-GISEL-NEXT: s_addc_u32 s11, s1, s11
+; GFX1150-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX1150-GISEL-NEXT: s_add_u32 s20, s2, s12
+; GFX1150-GISEL-NEXT: s_addc_u32 s21, s3, s13
+; GFX1150-GISEL-NEXT: s_load_b256 s[12:19], s[10:11], 0x0
+; GFX1150-GISEL-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX1150-GISEL-NEXT: image_sample v[0:3], v6, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB5_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[6:7]
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX12-SDAG-LABEL: test_waterfall_non_uni_img_2_idx:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v6, v2
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-SDAG-NEXT: .LBB5_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s10, v4
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v4
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX12-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; GFX12-SDAG-NEXT: s_ashr_i32 s13, s12, 31
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX12-SDAG-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_add_nc_u64 s[10:11], s[0:1], s[10:11]
+; GFX12-SDAG-NEXT: s_add_nc_u64 s[20:21], s[2:3], s[12:13]
+; GFX12-SDAG-NEXT: s_load_b256 s[12:19], s[10:11], 0x0
+; GFX12-SDAG-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: image_sample v[0:3], v6, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB5_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[6:7]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX12-GISEL-LABEL: test_waterfall_non_uni_img_2_idx:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v6, v2
+; GFX12-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-GISEL-NEXT: .LBB5_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s10, v4
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s12, v5
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s10, v4
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX12-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_add_co_u32 s10, s0, s10
+; GFX12-GISEL-NEXT: s_add_co_ci_u32 s11, s1, s11
+; GFX12-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_add_co_u32 s20, s2, s12
+; GFX12-GISEL-NEXT: s_add_co_ci_u32 s21, s3, s13
+; GFX12-GISEL-NEXT: s_load_b256 s[12:19], s[10:11], 0x0
+; GFX12-GISEL-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: image_sample v[0:3], v6, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB5_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[6:7]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: ; return to shader part epilog
+ %t_idx = insertelement <2 x i32> undef, i32 %index1, i32 0
+ %combined_idx = insertelement <2 x i32> %t_idx, i32 %index2, i32 1
+ %wf_token = call i32 @llvm.amdgcn.waterfall.begin.v2i32(i32 0, <2 x i32> %combined_idx)
+ %s_c_idx = call <2 x i32> @llvm.amdgcn.waterfall.readfirstlane.v2i32.v2i32(i32 %wf_token, <2 x i32> %combined_idx)
+ %s_idx1 = extractelement <2 x i32> %s_c_idx, i32 0
+ %s_idx2 = extractelement <2 x i32> %s_c_idx, i32 1
+ %ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx1
+ %ptr2 = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %samp_in, i32 %s_idx2
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
+ %samp = load <4 x i32>, <4 x i32> addrspace(4) * %ptr2, align 32
+ %r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token, <4 x float> %r)
+
+ ret <4 x float> %r1
+}
+
+define amdgpu_ps void @test_waterfall_non_uniform_img_single_store(<8 x i32> addrspace(4)* inreg %in, i32 %index, i32 %s, <4 x float> %data) #1 {
+; VI-SDAG-LABEL: test_waterfall_non_uniform_img_single_store:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; VI-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; VI-SDAG-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
+; VI-SDAG-NEXT: v_mov_b32_e32 v1, s1
+; VI-SDAG-NEXT: v_add_u32_e32 v7, vcc, s0, v7
+; VI-SDAG-NEXT: v_addc_u32_e32 v8, vcc, v1, v8, vcc
+; VI-SDAG-NEXT: v_add_u32_e32 v11, vcc, 16, v7
+; VI-SDAG-NEXT: v_addc_u32_e32 v12, vcc, 0, v8, vcc
+; VI-SDAG-NEXT: flat_load_dwordx4 v[7:10], v[7:8]
+; VI-SDAG-NEXT: flat_load_dwordx4 v[11:14], v[11:12]
+; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s0, v0
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[0:1], s0, v0
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[8:9], s[0:1]
+; VI-SDAG-NEXT: v_readfirstlane_b32 s0, v7
+; VI-SDAG-NEXT: v_readfirstlane_b32 s1, v8
+; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v9
+; VI-SDAG-NEXT: v_readfirstlane_b32 s3, v10
+; VI-SDAG-NEXT: v_readfirstlane_b32 s4, v11
+; VI-SDAG-NEXT: v_readfirstlane_b32 s5, v12
+; VI-SDAG-NEXT: v_readfirstlane_b32 s6, v13
+; VI-SDAG-NEXT: v_readfirstlane_b32 s7, v14
+; VI-SDAG-NEXT: ; implicit-def: $vgpr0
+; VI-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; VI-SDAG-NEXT: s_nop 4
+; VI-SDAG-NEXT: image_store v[2:5], v6, s[0:7] dmask:0xf unorm
+; VI-SDAG-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
+; VI-SDAG-NEXT: ; implicit-def: $vgpr6
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[8:9]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB6_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_endpgm
+;
+; VI-GISEL-LABEL: test_waterfall_non_uniform_img_single_store:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; VI-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; VI-GISEL-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
+; VI-GISEL-NEXT: v_mov_b32_e32 v10, s1
+; VI-GISEL-NEXT: v_mov_b32_e32 v9, s0
+; VI-GISEL-NEXT: v_add_u32_e32 v7, vcc, v9, v7
+; VI-GISEL-NEXT: v_addc_u32_e32 v8, vcc, v10, v8, vcc
+; VI-GISEL-NEXT: v_add_u32_e32 v11, vcc, 16, v7
+; VI-GISEL-NEXT: v_addc_u32_e32 v12, vcc, 0, v8, vcc
+; VI-GISEL-NEXT: flat_load_dwordx4 v[7:10], v[7:8]
+; VI-GISEL-NEXT: flat_load_dwordx4 v[11:14], v[11:12]
+; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s0, v0
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[0:1], s0, v0
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[8:9], s[0:1]
+; VI-GISEL-NEXT: v_readfirstlane_b32 s0, v7
+; VI-GISEL-NEXT: v_readfirstlane_b32 s1, v8
+; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v9
+; VI-GISEL-NEXT: v_readfirstlane_b32 s3, v10
+; VI-GISEL-NEXT: v_readfirstlane_b32 s4, v11
+; VI-GISEL-NEXT: v_readfirstlane_b32 s5, v12
+; VI-GISEL-NEXT: v_readfirstlane_b32 s6, v13
+; VI-GISEL-NEXT: v_readfirstlane_b32 s7, v14
+; VI-GISEL-NEXT: ; implicit-def: $vgpr0
+; VI-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; VI-GISEL-NEXT: s_nop 4
+; VI-GISEL-NEXT: image_store v[2:5], v6, s[0:7] dmask:0xf unorm
+; VI-GISEL-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
+; VI-GISEL-NEXT: ; implicit-def: $vgpr6
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[8:9]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB6_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_endpgm
+;
+; GFX9-SDAG-LABEL: test_waterfall_non_uniform_img_single_store:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX9-SDAG-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v1, s1
+; GFX9-SDAG-NEXT: v_add_co_u32_e32 v15, vcc, s0, v7
+; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v16, vcc, v1, v8, vcc
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[11:14], v[15:16], off offset:16
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[7:10], v[15:16], off
+; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s0, v0
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[0:1], s0, v0
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[8:9], s[0:1]
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s0, v7
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s1, v8
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v9
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s3, v10
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s4, v11
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s5, v12
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s6, v13
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s7, v14
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX9-SDAG-NEXT: s_nop 4
+; GFX9-SDAG-NEXT: image_store v[2:5], v6, s[0:7] dmask:0xf unorm
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[8:9]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB6_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_endpgm
+;
+; GFX9-GISEL-LABEL: test_waterfall_non_uniform_img_single_store:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX9-GISEL-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v10, s1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, s0
+; GFX9-GISEL-NEXT: v_add_co_u32_e32 v15, vcc, v9, v7
+; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v16, vcc, v10, v8, vcc
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[7:10], v[15:16], off
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[11:14], v[15:16], off offset:16
+; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s0, v0
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[0:1], s0, v0
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[8:9], s[0:1]
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s0, v7
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s1, v8
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v9
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s3, v10
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s4, v11
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s5, v12
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s6, v13
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s7, v14
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX9-GISEL-NEXT: s_nop 4
+; GFX9-GISEL-NEXT: image_store v[2:5], v6, s[0:7] dmask:0xf unorm
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[8:9]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB6_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_endpgm
+;
+; GFX10-32-SDAG-LABEL: test_waterfall_non_uniform_img_single_store:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
+; GFX10-32-SDAG-NEXT: v_add_co_u32 v15, vcc_lo, s0, v7
+; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v16, vcc_lo, s1, v8, vcc_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-SDAG-NEXT: s_clause 0x1
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[11:14], v[15:16], off offset:16
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[7:10], v[15:16], off
+; GFX10-32-SDAG-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s1, v0
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s1, v0
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s4, v7
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s5, v8
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s6, v9
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s7, v10
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s8, v11
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s9, v12
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s10, v13
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s11, v14
+; GFX10-32-SDAG-NEXT: image_store v[2:5], v6, s[4:11] dmask:0xf dim:SQ_RSRC_IMG_1D unorm
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s0, s0
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB6_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_endpgm
+;
+; GFX10-32-GISEL-LABEL: test_waterfall_non_uniform_img_single_store:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v10, s1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v9, s0
+; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
+; GFX10-32-GISEL-NEXT: v_add_co_u32 v15, vcc_lo, v9, v7
+; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v16, vcc_lo, v10, v8, vcc_lo
+; GFX10-32-GISEL-NEXT: s_clause 0x1
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[7:10], v[15:16], off
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[11:14], v[15:16], off offset:16
+; GFX10-32-GISEL-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s1, v0
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s1, v0
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s4, v7
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s5, v8
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s6, v9
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s7, v10
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s8, v11
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s9, v12
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s10, v13
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s11, v14
+; GFX10-32-GISEL-NEXT: image_store v[2:5], v6, s[4:11] dmask:0xf dim:SQ_RSRC_IMG_1D unorm
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s0, s0
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB6_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_endpgm
+;
+; GFX10-64-SDAG-LABEL: test_waterfall_non_uniform_img_single_store:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
+; GFX10-64-SDAG-NEXT: v_add_co_u32 v15, vcc, s0, v7
+; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v16, vcc, s1, v8, vcc
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-SDAG-NEXT: s_clause 0x1
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[11:14], v[15:16], off offset:16
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[7:10], v[15:16], off
+; GFX10-64-SDAG-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s2, v0
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s2, v0
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s4, v7
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s5, v8
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s6, v9
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s7, v10
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s8, v11
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s9, v12
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s10, v13
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s11, v14
+; GFX10-64-SDAG-NEXT: image_store v[2:5], v6, s[4:11] dmask:0xf dim:SQ_RSRC_IMG_1D unorm
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[0:1], s[0:1]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB6_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_endpgm
+;
+; GFX10-64-GISEL-LABEL: test_waterfall_non_uniform_img_single_store:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v10, s1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v9, s0
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
+; GFX10-64-GISEL-NEXT: v_add_co_u32 v15, vcc, v9, v7
+; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v16, vcc, v10, v8, vcc
+; GFX10-64-GISEL-NEXT: s_clause 0x1
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[7:10], v[15:16], off
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[11:14], v[15:16], off offset:16
+; GFX10-64-GISEL-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s2, v0
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s2, v0
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s4, v7
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s5, v8
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s6, v9
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s7, v10
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s8, v11
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s9, v12
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s10, v13
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s11, v14
+; GFX10-64-GISEL-NEXT: image_store v[2:5], v6, s[4:11] dmask:0xf dim:SQ_RSRC_IMG_1D unorm
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[0:1], s[0:1]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB6_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_endpgm
+;
+; GFX1150-SDAG-LABEL: test_waterfall_non_uniform_img_single_store:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
+; GFX1150-SDAG-NEXT: v_add_co_u32 v7, vcc, s0, v7
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v8, null, s1, v8, vcc
+; GFX1150-SDAG-NEXT: s_clause 0x1
+; GFX1150-SDAG-NEXT: global_load_b128 v[11:14], v[7:8], off offset:16
+; GFX1150-SDAG-NEXT: global_load_b128 v[7:10], v[7:8], off
+; GFX1150-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-SDAG-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s2, v0
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s2, v0
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s4, v7
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s5, v8
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s6, v9
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s7, v10
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s8, v11
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s9, v12
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s10, v13
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s11, v14
+; GFX1150-SDAG-NEXT: image_store v[2:5], v6, s[4:11] dmask:0xf dim:SQ_RSRC_IMG_1D unorm
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[0:1], s[0:1]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB6_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_endpgm
+;
+; GFX1150-GISEL-LABEL: test_waterfall_non_uniform_img_single_store:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v10, s1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v9, s0
+; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-GISEL-NEXT: v_add_co_u32 v11, vcc, v9, v7
+; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v12, null, v10, v8, vcc
+; GFX1150-GISEL-NEXT: s_clause 0x1
+; GFX1150-GISEL-NEXT: global_load_b128 v[7:10], v[11:12], off
+; GFX1150-GISEL-NEXT: global_load_b128 v[11:14], v[11:12], off offset:16
+; GFX1150-GISEL-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s2, v0
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s2, v0
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s4, v7
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s5, v8
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s6, v9
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s7, v10
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s8, v11
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s9, v12
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s10, v13
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s11, v14
+; GFX1150-GISEL-NEXT: image_store v[2:5], v6, s[4:11] dmask:0xf dim:SQ_RSRC_IMG_1D unorm
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[0:1], s[0:1]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB6_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_endpgm
+;
+; GFX12-SDAG-LABEL: test_waterfall_non_uniform_img_single_store:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[7:8], 5, v[0:1]
+; GFX12-SDAG-NEXT: v_add_co_u32 v7, vcc, s0, v7
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v8, null, s1, v8, vcc
+; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-SDAG-NEXT: s_clause 0x1
+; GFX12-SDAG-NEXT: global_load_b128 v[11:14], v[7:8], off offset:16
+; GFX12-SDAG-NEXT: global_load_b128 v[7:10], v[7:8], off
+; GFX12-SDAG-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s2, v0
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s2, v0
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x0
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s4, v7
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s5, v8
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s6, v9
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s7, v10
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s8, v11
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s9, v12
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s10, v13
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s11, v14
+; GFX12-SDAG-NEXT: image_store v[2:5], v6, s[4:11] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[0:1], s[0:1]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB6_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_endpgm
+;
+; GFX12-GISEL-LABEL: test_waterfall_non_uniform_img_single_store:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v10, s1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v9, s0
+; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[7:8], 5, v[0:1]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_add_co_u32 v11, vcc, v9, v7
+; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v12, null, v10, v8, vcc
+; GFX12-GISEL-NEXT: s_clause 0x1
+; GFX12-GISEL-NEXT: global_load_b128 v[7:10], v[11:12], off
+; GFX12-GISEL-NEXT: global_load_b128 v[11:14], v[11:12], off offset:16
+; GFX12-GISEL-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s2, v0
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s2, v0
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s4, v7
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s5, v8
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s6, v9
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s7, v10
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s8, v11
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s9, v12
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s10, v13
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s11, v14
+; GFX12-GISEL-NEXT: image_store v[2:5], v6, s[4:11] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[0:1], s[0:1]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr6
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB6_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_endpgm
+ %ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %index
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
+ %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %wf_token, <8 x i32> %rsrc)
+ %s_rsrc_use = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(i32 %wf_token, <8 x i32> %s_rsrc)
+ call void @llvm.amdgcn.image.store.1d.v4f32.i32(<4 x float> %data, i32 15, i32 %s, <8 x i32> %s_rsrc_use, i32 0, i32 0)
+
+ ret void
+}
+
+define amdgpu_ps void @test_remove_waterfall_last_use(<8 x i32> addrspace(4)* inreg %in, i32 %index, i32 %s, <4 x float> %data) #1 {
+; PRE-GFX10-LABEL: test_remove_waterfall_last_use:
+; PRE-GFX10: ; %bb.0:
+; PRE-GFX10-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x0
+; PRE-GFX10-NEXT: s_waitcnt lgkmcnt(0)
+; PRE-GFX10-NEXT: image_store v[2:5], v1, s[0:7] dmask:0xf unorm
+; PRE-GFX10-NEXT: s_endpgm
+;
+; GFX10-LABEL: test_remove_waterfall_last_use:
+; GFX10: ; %bb.0:
+; GFX10-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x0
+; GFX10-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-NEXT: image_store v[2:5], v1, s[0:7] dmask:0xf dim:SQ_RSRC_IMG_1D unorm
+; GFX10-NEXT: s_endpgm
+;
+; GFX1150-LABEL: test_remove_waterfall_last_use:
+; GFX1150: ; %bb.0:
+; GFX1150-NEXT: s_load_b256 s[0:7], s[0:1], 0x0
+; GFX1150-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1150-NEXT: image_store v[2:5], v1, s[0:7] dmask:0xf dim:SQ_RSRC_IMG_1D unorm
+; GFX1150-NEXT: s_endpgm
+;
+; GFX12-LABEL: test_remove_waterfall_last_use:
+; GFX12: ; %bb.0:
+; GFX12-NEXT: s_load_b256 s[0:7], s[0:1], 0x0
+; GFX12-NEXT: s_wait_kmcnt 0x0
+; GFX12-NEXT: image_store v[2:5], v1, s[0:7] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-NEXT: s_endpgm
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %in, align 32
+ %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %wf_token, <8 x i32> %rsrc)
+ %s_rsrc_use = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(i32 %wf_token, <8 x i32> %s_rsrc)
+ call void @llvm.amdgcn.image.store.1d.v4f32.i32(<4 x float> %data, i32 15, i32 %s, <8 x i32> %s_rsrc_use, i32 0, i32 0)
+
+ ret void
+}
+
+define amdgpu_ps <4 x float> @test_remove_waterfall_multi_rl(<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %samp_in, i32 %index, float %s, i32 inreg %val1, i32 inreg %val2) #1 {
+; VI-LABEL: test_remove_waterfall_multi_rl:
+; VI: ; %bb.0:
+; VI-NEXT: s_mov_b64 s[12:13], exec
+; VI-NEXT: s_mov_b32 s6, s5
+; VI-NEXT: s_wqm_b64 exec, exec
+; VI-NEXT: s_ashr_i32 s5, s4, 31
+; VI-NEXT: s_lshl_b64 s[4:5], s[4:5], 5
+; VI-NEXT: s_add_u32 s0, s0, s4
+; VI-NEXT: s_addc_u32 s1, s1, s5
+; VI-NEXT: s_ashr_i32 s7, s6, 31
+; VI-NEXT: s_lshl_b64 s[4:5], s[6:7], 4
+; VI-NEXT: s_add_u32 s8, s2, s4
+; VI-NEXT: s_addc_u32 s9, s3, s5
+; VI-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x0
+; VI-NEXT: s_load_dwordx4 s[8:11], s[8:9], 0x0
+; VI-NEXT: s_and_b64 exec, exec, s[12:13]
+; VI-NEXT: s_waitcnt lgkmcnt(0)
+; VI-NEXT: image_sample v[0:3], v1, s[0:7], s[8:11] dmask:0xf
+; VI-NEXT: s_waitcnt vmcnt(0)
+; VI-NEXT: ; return to shader part epilog
+;
+; GFX9-LABEL: test_remove_waterfall_multi_rl:
+; GFX9: ; %bb.0:
+; GFX9-NEXT: s_mov_b64 s[12:13], exec
+; GFX9-NEXT: s_mov_b32 s6, s5
+; GFX9-NEXT: s_wqm_b64 exec, exec
+; GFX9-NEXT: s_ashr_i32 s5, s4, 31
+; GFX9-NEXT: s_lshl_b64 s[4:5], s[4:5], 5
+; GFX9-NEXT: s_add_u32 s14, s0, s4
+; GFX9-NEXT: s_addc_u32 s15, s1, s5
+; GFX9-NEXT: s_ashr_i32 s7, s6, 31
+; GFX9-NEXT: s_lshl_b64 s[0:1], s[6:7], 4
+; GFX9-NEXT: s_add_u32 s16, s2, s0
+; GFX9-NEXT: s_addc_u32 s17, s3, s1
+; GFX9-NEXT: s_load_dwordx8 s[0:7], s[14:15], 0x0
+; GFX9-NEXT: s_load_dwordx4 s[8:11], s[16:17], 0x0
+; GFX9-NEXT: s_and_b64 exec, exec, s[12:13]
+; GFX9-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-NEXT: image_sample v[0:3], v1, s[0:7], s[8:11] dmask:0xf
+; GFX9-NEXT: s_waitcnt vmcnt(0)
+; GFX9-NEXT: ; return to shader part epilog
+;
+; GFX10-32-LABEL: test_remove_waterfall_multi_rl:
+; GFX10-32: ; %bb.0:
+; GFX10-32-NEXT: s_mov_b32 s16, exec_lo
+; GFX10-32-NEXT: s_mov_b32 s6, s5
+; GFX10-32-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-NEXT: s_ashr_i32 s5, s4, 31
+; GFX10-32-NEXT: s_lshl_b64 s[4:5], s[4:5], 5
+; GFX10-32-NEXT: s_add_u32 s12, s0, s4
+; GFX10-32-NEXT: s_addc_u32 s13, s1, s5
+; GFX10-32-NEXT: s_ashr_i32 s7, s6, 31
+; GFX10-32-NEXT: s_lshl_b64 s[0:1], s[6:7], 4
+; GFX10-32-NEXT: s_add_u32 s14, s2, s0
+; GFX10-32-NEXT: s_addc_u32 s15, s3, s1
+; GFX10-32-NEXT: s_load_dwordx8 s[0:7], s[12:13], 0x0
+; GFX10-32-NEXT: s_load_dwordx4 s[8:11], s[14:15], 0x0
+; GFX10-32-NEXT: s_and_b32 exec_lo, exec_lo, s16
+; GFX10-32-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-NEXT: image_sample v[0:3], v1, s[0:7], s[8:11] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-32-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-NEXT: ; return to shader part epilog
+;
+; GFX10-64-LABEL: test_remove_waterfall_multi_rl:
+; GFX10-64: ; %bb.0:
+; GFX10-64-NEXT: s_mov_b64 s[12:13], exec
+; GFX10-64-NEXT: s_mov_b32 s6, s5
+; GFX10-64-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-NEXT: s_ashr_i32 s5, s4, 31
+; GFX10-64-NEXT: s_lshl_b64 s[4:5], s[4:5], 5
+; GFX10-64-NEXT: s_add_u32 s14, s0, s4
+; GFX10-64-NEXT: s_addc_u32 s15, s1, s5
+; GFX10-64-NEXT: s_ashr_i32 s7, s6, 31
+; GFX10-64-NEXT: s_lshl_b64 s[0:1], s[6:7], 4
+; GFX10-64-NEXT: s_add_u32 s16, s2, s0
+; GFX10-64-NEXT: s_addc_u32 s17, s3, s1
+; GFX10-64-NEXT: s_load_dwordx8 s[0:7], s[14:15], 0x0
+; GFX10-64-NEXT: s_load_dwordx4 s[8:11], s[16:17], 0x0
+; GFX10-64-NEXT: s_and_b64 exec, exec, s[12:13]
+; GFX10-64-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-NEXT: image_sample v[0:3], v1, s[0:7], s[8:11] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-64-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-NEXT: ; return to shader part epilog
+;
+; GFX1150-LABEL: test_remove_waterfall_multi_rl:
+; GFX1150: ; %bb.0:
+; GFX1150-NEXT: s_mov_b64 s[12:13], exec
+; GFX1150-NEXT: s_mov_b32 s6, s5
+; GFX1150-NEXT: s_wqm_b64 exec, exec
+; GFX1150-NEXT: s_ashr_i32 s5, s4, 31
+; GFX1150-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1150-NEXT: s_lshl_b64 s[4:5], s[4:5], 5
+; GFX1150-NEXT: s_add_u32 s0, s0, s4
+; GFX1150-NEXT: s_addc_u32 s1, s1, s5
+; GFX1150-NEXT: s_ashr_i32 s7, s6, 31
+; GFX1150-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1150-NEXT: s_lshl_b64 s[4:5], s[6:7], 4
+; GFX1150-NEXT: s_add_u32 s8, s2, s4
+; GFX1150-NEXT: s_addc_u32 s9, s3, s5
+; GFX1150-NEXT: s_load_b256 s[0:7], s[0:1], 0x0
+; GFX1150-NEXT: s_load_b128 s[8:11], s[8:9], 0x0
+; GFX1150-NEXT: s_and_b64 exec, exec, s[12:13]
+; GFX1150-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1150-NEXT: image_sample v[0:3], v1, s[0:7], s[8:11] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX1150-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-NEXT: ; return to shader part epilog
+;
+; GFX12-SDAG-LABEL: test_remove_waterfall_multi_rl:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_mov_b64 s[12:13], exec
+; GFX12-SDAG-NEXT: s_mov_b32 s6, s5
+; GFX12-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX12-SDAG-NEXT: s_ashr_i32 s5, s4, 31
+; GFX12-SDAG-NEXT: s_ashr_i32 s7, s6, 31
+; GFX12-SDAG-NEXT: s_lshl_b64 s[4:5], s[4:5], 5
+; GFX12-SDAG-NEXT: s_lshl_b64 s[6:7], s[6:7], 4
+; GFX12-SDAG-NEXT: s_add_nc_u64 s[0:1], s[0:1], s[4:5]
+; GFX12-SDAG-NEXT: s_add_nc_u64 s[8:9], s[2:3], s[6:7]
+; GFX12-SDAG-NEXT: s_load_b256 s[0:7], s[0:1], 0x0
+; GFX12-SDAG-NEXT: s_load_b128 s[8:11], s[8:9], 0x0
+; GFX12-SDAG-NEXT: s_and_b64 exec, exec, s[12:13]
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: image_sample v[0:3], v1, s[0:7], s[8:11] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX12-GISEL-LABEL: test_remove_waterfall_multi_rl:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_mov_b64 s[12:13], exec
+; GFX12-GISEL-NEXT: s_mov_b32 s6, s5
+; GFX12-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX12-GISEL-NEXT: s_ashr_i32 s5, s4, 31
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_lshl_b64 s[4:5], s[4:5], 5
+; GFX12-GISEL-NEXT: s_add_co_u32 s0, s0, s4
+; GFX12-GISEL-NEXT: s_add_co_ci_u32 s1, s1, s5
+; GFX12-GISEL-NEXT: s_ashr_i32 s7, s6, 31
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_lshl_b64 s[4:5], s[6:7], 4
+; GFX12-GISEL-NEXT: s_add_co_u32 s8, s2, s4
+; GFX12-GISEL-NEXT: s_add_co_ci_u32 s9, s3, s5
+; GFX12-GISEL-NEXT: s_load_b256 s[0:7], s[0:1], 0x0
+; GFX12-GISEL-NEXT: s_load_b128 s[8:11], s[8:9], 0x0
+; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[12:13]
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: image_sample v[0:3], v1, s[0:7], s[8:11] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: ; return to shader part epilog
+ %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %val1)
+ %s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %val2)
+ %ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
+ %ptr2 = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %samp_in, i32 %s_idx2
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
+ %samp = load <4 x i32>, <4 x i32> addrspace(4) * %ptr2, align 32
+ %r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token, <4 x float> %r)
+
+ ret <4 x float> %r1
+}
+
+define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %samp_in, i32 %index, float %s, i32 inreg %val) #1 {
+; VI-SDAG-LABEL: test_keep_waterfall_multi_rl:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; VI-SDAG-NEXT: s_wqm_b64 exec, exec
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; VI-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; VI-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; VI-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s10, v5
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[12:13], s10, v5
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[24:25], s[12:13]
+; VI-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; VI-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; VI-SDAG-NEXT: s_add_u32 s10, s0, s10
+; VI-SDAG-NEXT: s_addc_u32 s11, s1, s11
+; VI-SDAG-NEXT: s_ashr_i32 s5, s4, 31
+; VI-SDAG-NEXT: s_lshl_b64 s[12:13], s[4:5], 4
+; VI-SDAG-NEXT: s_add_u32 s20, s2, s12
+; VI-SDAG-NEXT: s_addc_u32 s21, s3, s13
+; VI-SDAG-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; VI-SDAG-NEXT: s_load_dwordx4 s[20:23], s[20:21], 0x0
+; VI-SDAG-NEXT: ; implicit-def: $vgpr5
+; VI-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; VI-SDAG-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf
+; VI-SDAG-NEXT: ; implicit-def: $vgpr4
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[24:25]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB9_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[8:9]
+; VI-SDAG-NEXT: s_and_b64 exec, exec, s[6:7]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: ; return to shader part epilog
+;
+; VI-GISEL-LABEL: test_keep_waterfall_multi_rl:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; VI-GISEL-NEXT: s_wqm_b64 exec, exec
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; VI-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; VI-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v5
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[12:13], s10, v5
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[24:25], s[12:13]
+; VI-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; VI-GISEL-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; VI-GISEL-NEXT: s_add_u32 s10, s0, s10
+; VI-GISEL-NEXT: s_addc_u32 s11, s1, s11
+; VI-GISEL-NEXT: s_ashr_i32 s5, s4, 31
+; VI-GISEL-NEXT: s_lshl_b64 s[12:13], s[4:5], 4
+; VI-GISEL-NEXT: s_add_u32 s20, s2, s12
+; VI-GISEL-NEXT: s_addc_u32 s21, s3, s13
+; VI-GISEL-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; VI-GISEL-NEXT: s_load_dwordx4 s[20:23], s[20:21], 0x0
+; VI-GISEL-NEXT: ; implicit-def: $vgpr5
+; VI-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; VI-GISEL-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf
+; VI-GISEL-NEXT: ; implicit-def: $vgpr4
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[24:25]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB9_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[8:9]
+; VI-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX9-SDAG-LABEL: test_keep_waterfall_multi_rl:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX9-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX9-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX9-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s10, v5
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[12:13], s10, v5
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[24:25], s[12:13]
+; GFX9-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; GFX9-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX9-SDAG-NEXT: s_add_u32 s10, s0, s10
+; GFX9-SDAG-NEXT: s_addc_u32 s11, s1, s11
+; GFX9-SDAG-NEXT: s_ashr_i32 s5, s4, 31
+; GFX9-SDAG-NEXT: s_lshl_b64 s[12:13], s[4:5], 4
+; GFX9-SDAG-NEXT: s_add_u32 s26, s2, s12
+; GFX9-SDAG-NEXT: s_addc_u32 s27, s3, s13
+; GFX9-SDAG-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; GFX9-SDAG-NEXT: s_load_dwordx4 s[20:23], s[26:27], 0x0
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX9-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-SDAG-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[24:25]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB9_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[8:9]
+; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX9-GISEL-LABEL: test_keep_waterfall_multi_rl:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX9-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; GFX9-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX9-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v5
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[12:13], s10, v5
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[24:25], s[12:13]
+; GFX9-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; GFX9-GISEL-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX9-GISEL-NEXT: s_add_u32 s10, s0, s10
+; GFX9-GISEL-NEXT: s_addc_u32 s11, s1, s11
+; GFX9-GISEL-NEXT: s_ashr_i32 s5, s4, 31
+; GFX9-GISEL-NEXT: s_lshl_b64 s[12:13], s[4:5], 4
+; GFX9-GISEL-NEXT: s_add_u32 s26, s2, s12
+; GFX9-GISEL-NEXT: s_addc_u32 s27, s3, s13
+; GFX9-GISEL-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; GFX9-GISEL-NEXT: s_load_dwordx4 s[20:23], s[26:27], 0x0
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX9-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-GISEL-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[24:25]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB9_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[8:9]
+; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-32-SDAG-LABEL: test_keep_waterfall_multi_rl:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: s_mov_b32 s6, exec_lo
+; GFX10-32-SDAG-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s8, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s7, exec_lo
+; GFX10-32-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s10, v5
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s10, v5
+; GFX10-32-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; GFX10-32-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX10-32-SDAG-NEXT: s_add_u32 s10, s0, s10
+; GFX10-32-SDAG-NEXT: s_addc_u32 s11, s1, s11
+; GFX10-32-SDAG-NEXT: s_ashr_i32 s5, s4, 31
+; GFX10-32-SDAG-NEXT: s_lshl_b64 s[12:13], s[4:5], 4
+; GFX10-32-SDAG-NEXT: s_add_u32 s24, s2, s12
+; GFX10-32-SDAG-NEXT: s_addc_u32 s25, s3, s13
+; GFX10-32-SDAG-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; GFX10-32-SDAG-NEXT: s_load_dwordx4 s[20:23], s[24:25], 0x0
+; GFX10-32-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-SDAG-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s8, s8
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB9_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s7
+; GFX10-32-SDAG-NEXT: s_and_b32 exec_lo, exec_lo, s6
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-32-GISEL-LABEL: test_keep_waterfall_multi_rl:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: s_mov_b32 s6, exec_lo
+; GFX10-32-GISEL-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; GFX10-32-GISEL-NEXT: s_mov_b32 s8, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s7, exec_lo
+; GFX10-32-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s10, v5
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s10, v5
+; GFX10-32-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; GFX10-32-GISEL-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX10-32-GISEL-NEXT: s_add_u32 s10, s0, s10
+; GFX10-32-GISEL-NEXT: s_addc_u32 s11, s1, s11
+; GFX10-32-GISEL-NEXT: s_ashr_i32 s5, s4, 31
+; GFX10-32-GISEL-NEXT: s_lshl_b64 s[12:13], s[4:5], 4
+; GFX10-32-GISEL-NEXT: s_add_u32 s24, s2, s12
+; GFX10-32-GISEL-NEXT: s_addc_u32 s25, s3, s13
+; GFX10-32-GISEL-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; GFX10-32-GISEL-NEXT: s_load_dwordx4 s[20:23], s[24:25], 0x0
+; GFX10-32-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-GISEL-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s8, s8
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB9_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s7
+; GFX10-32-GISEL-NEXT: s_and_b32 exec_lo, exec_lo, s6
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-64-SDAG-LABEL: test_keep_waterfall_multi_rl:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX10-64-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX10-64-SDAG-NEXT: s_ashr_i32 s13, s12, 31
+; GFX10-64-SDAG-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; GFX10-64-SDAG-NEXT: s_add_u32 s24, s0, s12
+; GFX10-64-SDAG-NEXT: s_addc_u32 s25, s1, s13
+; GFX10-64-SDAG-NEXT: s_ashr_i32 s5, s4, 31
+; GFX10-64-SDAG-NEXT: s_lshl_b64 s[12:13], s[4:5], 4
+; GFX10-64-SDAG-NEXT: s_add_u32 s26, s2, s12
+; GFX10-64-SDAG-NEXT: s_addc_u32 s27, s3, s13
+; GFX10-64-SDAG-NEXT: s_load_dwordx8 s[12:19], s[24:25], 0x0
+; GFX10-64-SDAG-NEXT: s_load_dwordx4 s[20:23], s[26:27], 0x0
+; GFX10-64-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-SDAG-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[10:11], s[10:11]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB9_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[8:9]
+; GFX10-64-SDAG-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-64-GISEL-LABEL: test_keep_waterfall_multi_rl:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[10:11], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX10-64-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s12, v5
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX10-64-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; GFX10-64-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; GFX10-64-GISEL-NEXT: s_add_u32 s24, s0, s12
+; GFX10-64-GISEL-NEXT: s_addc_u32 s25, s1, s13
+; GFX10-64-GISEL-NEXT: s_ashr_i32 s5, s4, 31
+; GFX10-64-GISEL-NEXT: s_lshl_b64 s[12:13], s[4:5], 4
+; GFX10-64-GISEL-NEXT: s_add_u32 s26, s2, s12
+; GFX10-64-GISEL-NEXT: s_addc_u32 s27, s3, s13
+; GFX10-64-GISEL-NEXT: s_load_dwordx8 s[12:19], s[24:25], 0x0
+; GFX10-64-GISEL-NEXT: s_load_dwordx4 s[20:23], s[26:27], 0x0
+; GFX10-64-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-GISEL-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[10:11], s[10:11]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB9_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[8:9]
+; GFX10-64-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX1150-SDAG-LABEL: test_keep_waterfall_multi_rl:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX1150-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; GFX1150-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX1150-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX1150-SDAG-NEXT: s_ashr_i32 s13, s12, 31
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; GFX1150-SDAG-NEXT: s_add_u32 s12, s0, s12
+; GFX1150-SDAG-NEXT: s_addc_u32 s13, s1, s13
+; GFX1150-SDAG-NEXT: s_ashr_i32 s5, s4, 31
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_lshl_b64 s[14:15], s[4:5], 4
+; GFX1150-SDAG-NEXT: s_add_u32 s20, s2, s14
+; GFX1150-SDAG-NEXT: s_addc_u32 s21, s3, s15
+; GFX1150-SDAG-NEXT: s_load_b256 s[12:19], s[12:13], 0x0
+; GFX1150-SDAG-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX1150-SDAG-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB9_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[8:9]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX1150-GISEL-LABEL: test_keep_waterfall_multi_rl:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; GFX1150-GISEL-NEXT: s_mov_b64 s[10:11], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX1150-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s12, v5
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX1150-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; GFX1150-GISEL-NEXT: s_add_u32 s12, s0, s12
+; GFX1150-GISEL-NEXT: s_addc_u32 s13, s1, s13
+; GFX1150-GISEL-NEXT: s_ashr_i32 s5, s4, 31
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_lshl_b64 s[14:15], s[4:5], 4
+; GFX1150-GISEL-NEXT: s_add_u32 s20, s2, s14
+; GFX1150-GISEL-NEXT: s_addc_u32 s21, s3, s15
+; GFX1150-GISEL-NEXT: s_load_b256 s[12:19], s[12:13], 0x0
+; GFX1150-GISEL-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX1150-GISEL-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB9_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[8:9]
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX12-SDAG-LABEL: test_keep_waterfall_multi_rl:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v4, v1
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX12-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; GFX12-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX12-SDAG-NEXT: s_ashr_i32 s13, s12, 31
+; GFX12-SDAG-NEXT: s_ashr_i32 s5, s4, 31
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; GFX12-SDAG-NEXT: s_lshl_b64 s[14:15], s[4:5], 4
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_add_nc_u64 s[12:13], s[0:1], s[12:13]
+; GFX12-SDAG-NEXT: s_add_nc_u64 s[20:21], s[2:3], s[14:15]
+; GFX12-SDAG-NEXT: s_load_b256 s[12:19], s[12:13], 0x0
+; GFX12-SDAG-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB9_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[8:9]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-SDAG-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX12-GISEL-LABEL: test_keep_waterfall_multi_rl:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, v1
+; GFX12-GISEL-NEXT: s_mov_b64 s[10:11], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s12, v5
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s12, v5
+; GFX12-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_add_co_u32 s12, s0, s12
+; GFX12-GISEL-NEXT: s_add_co_ci_u32 s13, s1, s13
+; GFX12-GISEL-NEXT: s_ashr_i32 s5, s4, 31
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_lshl_b64 s[14:15], s[4:5], 4
+; GFX12-GISEL-NEXT: s_add_co_u32 s20, s2, s14
+; GFX12-GISEL-NEXT: s_add_co_ci_u32 s21, s3, s15
+; GFX12-GISEL-NEXT: s_load_b256 s[12:19], s[12:13], 0x0
+; GFX12-GISEL-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB9_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[8:9]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: ; return to shader part epilog
+ %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %index)
+ %s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %val)
+ %ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
+ %ptr2 = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %samp_in, i32 %s_idx2
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
+ %samp = load <4 x i32>, <4 x i32> addrspace(4) * %ptr2, align 32
+ %r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token, <4 x float> %r)
+
+ ret <4 x float> %r1
+}
+
+define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %samp_in, i32 %index, float %s, i32 inreg %val) #1 {
+; VI-LABEL: test_waterfall_sample_with_kill:
+; VI: ; %bb.0:
+; VI-NEXT: s_mov_b64 s[6:7], exec
+; VI-NEXT: s_wqm_b64 exec, exec
+; VI-NEXT: s_mov_b64 s[8:9], exec
+; VI-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
+; VI-NEXT: v_readfirstlane_b32 s10, v0
+; VI-NEXT: v_cmp_eq_u32_e64 s[12:13], s10, v0
+; VI-NEXT: s_and_saveexec_b64 s[24:25], s[12:13]
+; VI-NEXT: s_ashr_i32 s11, s10, 31
+; VI-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; VI-NEXT: s_add_u32 s10, s0, s10
+; VI-NEXT: s_addc_u32 s11, s1, s11
+; VI-NEXT: s_ashr_i32 s5, s4, 31
+; VI-NEXT: s_lshl_b64 s[12:13], s[4:5], 4
+; VI-NEXT: s_add_u32 s20, s2, s12
+; VI-NEXT: s_addc_u32 s21, s3, s13
+; VI-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; VI-NEXT: s_load_dwordx4 s[20:23], s[20:21], 0x0
+; VI-NEXT: ; implicit-def: $vgpr0
+; VI-NEXT: s_waitcnt lgkmcnt(0)
+; VI-NEXT: image_sample v[2:5], v1, s[12:19], s[20:23] dmask:0xf
+; VI-NEXT: ; implicit-def: $vgpr1
+; VI-NEXT: s_xor_b64 exec, exec, s[24:25]
+; VI-NEXT: s_cbranch_execnz .LBB10_1
+; VI-NEXT: ; %bb.2:
+; VI-NEXT: s_mov_b64 exec, s[8:9]
+; VI-NEXT: s_and_b64 exec, exec, s[6:7]
+; VI-NEXT: s_waitcnt vmcnt(0)
+; VI-NEXT: v_cmp_gt_f32_e32 vcc, 0, v2
+; VI-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; VI-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
+; VI-NEXT: s_cbranch_execz .LBB10_5
+; VI-NEXT: ; %bb.3: ; %.kill
+; VI-NEXT: s_andn2_b64 s[6:7], s[6:7], exec
+; VI-NEXT: s_cbranch_scc0 .LBB10_6
+; VI-NEXT: ; %bb.4: ; %.kill
+; VI-NEXT: s_mov_b64 exec, 0
+; VI-NEXT: .LBB10_5: ; %.exit
+; VI-NEXT: s_or_b64 exec, exec, s[0:1]
+; VI-NEXT: v_mov_b32_e32 v0, 0
+; VI-NEXT: exp mrt0, v0, off, off, off done vm
+; VI-NEXT: s_endpgm
+; VI-NEXT: .LBB10_6:
+; VI-NEXT: s_mov_b64 exec, 0
+; VI-NEXT: exp null, off, off, off, off done vm
+; VI-NEXT: s_endpgm
+;
+; GFX9-LABEL: test_waterfall_sample_with_kill:
+; GFX9: ; %bb.0:
+; GFX9-NEXT: s_mov_b64 s[6:7], exec
+; GFX9-NEXT: s_wqm_b64 exec, exec
+; GFX9-NEXT: s_mov_b64 s[8:9], exec
+; GFX9-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT: v_readfirstlane_b32 s10, v0
+; GFX9-NEXT: v_cmp_eq_u32_e64 s[12:13], s10, v0
+; GFX9-NEXT: s_and_saveexec_b64 s[24:25], s[12:13]
+; GFX9-NEXT: s_ashr_i32 s11, s10, 31
+; GFX9-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX9-NEXT: s_add_u32 s10, s0, s10
+; GFX9-NEXT: s_addc_u32 s11, s1, s11
+; GFX9-NEXT: s_ashr_i32 s5, s4, 31
+; GFX9-NEXT: s_lshl_b64 s[12:13], s[4:5], 4
+; GFX9-NEXT: s_add_u32 s26, s2, s12
+; GFX9-NEXT: s_addc_u32 s27, s3, s13
+; GFX9-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; GFX9-NEXT: s_load_dwordx4 s[20:23], s[26:27], 0x0
+; GFX9-NEXT: ; implicit-def: $vgpr0
+; GFX9-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-NEXT: image_sample v[2:5], v1, s[12:19], s[20:23] dmask:0xf
+; GFX9-NEXT: ; implicit-def: $vgpr1
+; GFX9-NEXT: s_xor_b64 exec, exec, s[24:25]
+; GFX9-NEXT: s_cbranch_execnz .LBB10_1
+; GFX9-NEXT: ; %bb.2:
+; GFX9-NEXT: s_mov_b64 exec, s[8:9]
+; GFX9-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX9-NEXT: s_waitcnt vmcnt(0)
+; GFX9-NEXT: v_cmp_gt_f32_e32 vcc, 0, v2
+; GFX9-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX9-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
+; GFX9-NEXT: s_cbranch_execz .LBB10_5
+; GFX9-NEXT: ; %bb.3: ; %.kill
+; GFX9-NEXT: s_andn2_b64 s[6:7], s[6:7], exec
+; GFX9-NEXT: s_cbranch_scc0 .LBB10_6
+; GFX9-NEXT: ; %bb.4: ; %.kill
+; GFX9-NEXT: s_mov_b64 exec, 0
+; GFX9-NEXT: .LBB10_5: ; %.exit
+; GFX9-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX9-NEXT: v_mov_b32_e32 v0, 0
+; GFX9-NEXT: exp mrt0, v0, off, off, off done vm
+; GFX9-NEXT: s_endpgm
+; GFX9-NEXT: .LBB10_6:
+; GFX9-NEXT: s_mov_b64 exec, 0
+; GFX9-NEXT: exp null, off, off, off, off done vm
+; GFX9-NEXT: s_endpgm
+;
+; GFX10-32-LABEL: test_waterfall_sample_with_kill:
+; GFX10-32: ; %bb.0:
+; GFX10-32-NEXT: s_mov_b32 s6, exec_lo
+; GFX10-32-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-NEXT: s_mov_b32 s8, exec_lo
+; GFX10-32-NEXT: s_mov_b32 s7, exec_lo
+; GFX10-32-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-NEXT: v_readfirstlane_b32 s10, v0
+; GFX10-32-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-NEXT: v_cmpx_eq_u32_e32 s10, v0
+; GFX10-32-NEXT: s_ashr_i32 s11, s10, 31
+; GFX10-32-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX10-32-NEXT: s_add_u32 s10, s0, s10
+; GFX10-32-NEXT: s_addc_u32 s11, s1, s11
+; GFX10-32-NEXT: s_ashr_i32 s5, s4, 31
+; GFX10-32-NEXT: s_lshl_b64 s[12:13], s[4:5], 4
+; GFX10-32-NEXT: s_add_u32 s24, s2, s12
+; GFX10-32-NEXT: s_addc_u32 s25, s3, s13
+; GFX10-32-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; GFX10-32-NEXT: s_load_dwordx4 s[20:23], s[24:25], 0x0
+; GFX10-32-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-NEXT: image_sample v[2:5], v1, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-32-NEXT: s_andn2_wrexec_b32 s8, s8
+; GFX10-32-NEXT: ; implicit-def: $vgpr0
+; GFX10-32-NEXT: ; implicit-def: $vgpr1
+; GFX10-32-NEXT: s_cbranch_execnz .LBB10_1
+; GFX10-32-NEXT: ; %bb.2:
+; GFX10-32-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-NEXT: s_mov_b32 exec_lo, s7
+; GFX10-32-NEXT: s_and_b32 exec_lo, exec_lo, s6
+; GFX10-32-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-NEXT: v_cmp_gt_f32_e32 vcc_lo, 0, v2
+; GFX10-32-NEXT: s_and_saveexec_b32 s0, vcc_lo
+; GFX10-32-NEXT: s_xor_b32 s0, exec_lo, s0
+; GFX10-32-NEXT: s_cbranch_execz .LBB10_5
+; GFX10-32-NEXT: ; %bb.3: ; %.kill
+; GFX10-32-NEXT: s_andn2_b32 s6, s6, exec_lo
+; GFX10-32-NEXT: s_cbranch_scc0 .LBB10_6
+; GFX10-32-NEXT: ; %bb.4: ; %.kill
+; GFX10-32-NEXT: s_mov_b32 exec_lo, 0
+; GFX10-32-NEXT: .LBB10_5: ; %.exit
+; GFX10-32-NEXT: s_or_b32 exec_lo, exec_lo, s0
+; GFX10-32-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-NEXT: exp mrt0, v0, off, off, off done vm
+; GFX10-32-NEXT: s_endpgm
+; GFX10-32-NEXT: .LBB10_6:
+; GFX10-32-NEXT: s_mov_b32 exec_lo, 0
+; GFX10-32-NEXT: exp null, off, off, off, off done vm
+; GFX10-32-NEXT: s_endpgm
+;
+; GFX10-64-LABEL: test_waterfall_sample_with_kill:
+; GFX10-64: ; %bb.0:
+; GFX10-64-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-NEXT: s_mov_b64 s[10:11], exec
+; GFX10-64-NEXT: s_mov_b64 s[8:9], exec
+; GFX10-64-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-NEXT: v_readfirstlane_b32 s12, v0
+; GFX10-64-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-NEXT: v_cmpx_eq_u32_e64 s12, v0
+; GFX10-64-NEXT: s_ashr_i32 s13, s12, 31
+; GFX10-64-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; GFX10-64-NEXT: s_add_u32 s24, s0, s12
+; GFX10-64-NEXT: s_addc_u32 s25, s1, s13
+; GFX10-64-NEXT: s_ashr_i32 s5, s4, 31
+; GFX10-64-NEXT: s_lshl_b64 s[12:13], s[4:5], 4
+; GFX10-64-NEXT: s_add_u32 s26, s2, s12
+; GFX10-64-NEXT: s_addc_u32 s27, s3, s13
+; GFX10-64-NEXT: s_load_dwordx8 s[12:19], s[24:25], 0x0
+; GFX10-64-NEXT: s_load_dwordx4 s[20:23], s[26:27], 0x0
+; GFX10-64-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-NEXT: image_sample v[2:5], v1, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-64-NEXT: s_andn2_wrexec_b64 s[10:11], s[10:11]
+; GFX10-64-NEXT: ; implicit-def: $vgpr0
+; GFX10-64-NEXT: ; implicit-def: $vgpr1
+; GFX10-64-NEXT: s_cbranch_execnz .LBB10_1
+; GFX10-64-NEXT: ; %bb.2:
+; GFX10-64-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-NEXT: s_mov_b64 exec, s[8:9]
+; GFX10-64-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX10-64-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-NEXT: v_cmp_gt_f32_e32 vcc, 0, v2
+; GFX10-64-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX10-64-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
+; GFX10-64-NEXT: s_cbranch_execz .LBB10_5
+; GFX10-64-NEXT: ; %bb.3: ; %.kill
+; GFX10-64-NEXT: s_andn2_b64 s[6:7], s[6:7], exec
+; GFX10-64-NEXT: s_cbranch_scc0 .LBB10_6
+; GFX10-64-NEXT: ; %bb.4: ; %.kill
+; GFX10-64-NEXT: s_mov_b64 exec, 0
+; GFX10-64-NEXT: .LBB10_5: ; %.exit
+; GFX10-64-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX10-64-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-NEXT: exp mrt0, v0, off, off, off done vm
+; GFX10-64-NEXT: s_endpgm
+; GFX10-64-NEXT: .LBB10_6:
+; GFX10-64-NEXT: s_mov_b64 exec, 0
+; GFX10-64-NEXT: exp null, off, off, off, off done vm
+; GFX10-64-NEXT: s_endpgm
+;
+; GFX1150-LABEL: test_waterfall_sample_with_kill:
+; GFX1150: ; %bb.0:
+; GFX1150-NEXT: s_setprio 2
+; GFX1150-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-NEXT: s_wqm_b64 exec, exec
+; GFX1150-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-NEXT: s_mov_b64 s[10:11], exec
+; GFX1150-NEXT: s_mov_b64 s[8:9], exec
+; GFX1150-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-NEXT: v_readfirstlane_b32 s12, v0
+; GFX1150-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
+; GFX1150-NEXT: v_cmpx_eq_u32_e64 s12, v0
+; GFX1150-NEXT: s_ashr_i32 s13, s12, 31
+; GFX1150-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; GFX1150-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_2) | instid1(SALU_CYCLE_1)
+; GFX1150-NEXT: s_add_u32 s12, s0, s12
+; GFX1150-NEXT: s_addc_u32 s13, s1, s13
+; GFX1150-NEXT: s_ashr_i32 s5, s4, 31
+; GFX1150-NEXT: s_lshl_b64 s[14:15], s[4:5], 4
+; GFX1150-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-NEXT: s_add_u32 s20, s2, s14
+; GFX1150-NEXT: s_addc_u32 s21, s3, s15
+; GFX1150-NEXT: s_load_b256 s[12:19], s[12:13], 0x0
+; GFX1150-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX1150-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
+; GFX1150-NEXT: image_sample v[2:5], v1, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX1150-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
+; GFX1150-NEXT: ; implicit-def: $vgpr0
+; GFX1150-NEXT: ; implicit-def: $vgpr1
+; GFX1150-NEXT: s_cbranch_execnz .LBB10_1
+; GFX1150-NEXT: ; %bb.2:
+; GFX1150-NEXT: s_mov_b64 exec, s[8:9]
+; GFX1150-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX1150-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX1150-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-NEXT: v_cmpx_gt_f32_e32 0, v2
+; GFX1150-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
+; GFX1150-NEXT: s_cbranch_execz .LBB10_5
+; GFX1150-NEXT: ; %bb.3: ; %.kill
+; GFX1150-NEXT: s_and_not1_b64 s[6:7], s[6:7], exec
+; GFX1150-NEXT: s_cbranch_scc0 .LBB10_6
+; GFX1150-NEXT: ; %bb.4: ; %.kill
+; GFX1150-NEXT: s_mov_b64 exec, 0
+; GFX1150-NEXT: .LBB10_5: ; %.exit
+; GFX1150-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX1150-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-NEXT: exp mrt0, v0, off, off, off done
+; GFX1150-NEXT: s_setprio 0
+; GFX1150-NEXT: s_nop 0
+; GFX1150-NEXT: s_nop 0
+; GFX1150-NEXT: s_endpgm
+; GFX1150-NEXT: .LBB10_6:
+; GFX1150-NEXT: s_mov_b64 exec, 0
+; GFX1150-NEXT: exp mrt0, off, off, off, off done
+; GFX1150-NEXT: s_setprio 0
+; GFX1150-NEXT: s_nop 0
+; GFX1150-NEXT: s_nop 0
+; GFX1150-NEXT: s_endpgm
+;
+; GFX12-SDAG-LABEL: test_waterfall_sample_with_kill:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; GFX12-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-SDAG-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s12, v0
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s12, v0
+; GFX12-SDAG-NEXT: s_ashr_i32 s13, s12, 31
+; GFX12-SDAG-NEXT: s_ashr_i32 s5, s4, 31
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; GFX12-SDAG-NEXT: s_lshl_b64 s[14:15], s[4:5], 4
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_add_nc_u64 s[12:13], s[0:1], s[12:13]
+; GFX12-SDAG-NEXT: s_add_nc_u64 s[20:21], s[2:3], s[14:15]
+; GFX12-SDAG-NEXT: s_load_b256 s[12:19], s[12:13], 0x0
+; GFX12-SDAG-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: image_sample v[2:5], v1, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr1
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB10_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[8:9]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX12-SDAG-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: v_cmpx_gt_f32_e32 0, v2
+; GFX12-SDAG-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
+; GFX12-SDAG-NEXT: s_cbranch_execz .LBB10_5
+; GFX12-SDAG-NEXT: ; %bb.3: ; %.kill
+; GFX12-SDAG-NEXT: s_and_not1_b64 s[6:7], s[6:7], exec
+; GFX12-SDAG-NEXT: s_cbranch_scc0 .LBB10_6
+; GFX12-SDAG-NEXT: ; %bb.4: ; %.kill
+; GFX12-SDAG-NEXT: s_mov_b64 exec, 0
+; GFX12-SDAG-NEXT: .LBB10_5: ; %.exit
+; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-SDAG-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-SDAG-NEXT: export mrt0, v0, off, off, off done
+; GFX12-SDAG-NEXT: s_endpgm
+; GFX12-SDAG-NEXT: .LBB10_6:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, 0
+; GFX12-SDAG-NEXT: export mrt0, off, off, off, off done
+; GFX12-SDAG-NEXT: s_endpgm
+;
+; GFX12-GISEL-LABEL: test_waterfall_sample_with_kill:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_mov_b64 s[10:11], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-GISEL-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s12, v0
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s12, v0
+; GFX12-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_add_co_u32 s12, s0, s12
+; GFX12-GISEL-NEXT: s_add_co_ci_u32 s13, s1, s13
+; GFX12-GISEL-NEXT: s_ashr_i32 s5, s4, 31
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_lshl_b64 s[14:15], s[4:5], 4
+; GFX12-GISEL-NEXT: s_add_co_u32 s20, s2, s14
+; GFX12-GISEL-NEXT: s_add_co_ci_u32 s21, s3, s15
+; GFX12-GISEL-NEXT: s_load_b256 s[12:19], s[12:13], 0x0
+; GFX12-GISEL-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: image_sample v[2:5], v1, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr1
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB10_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[8:9]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: v_cmpx_gt_f32_e32 0, v2
+; GFX12-GISEL-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
+; GFX12-GISEL-NEXT: s_cbranch_execz .LBB10_5
+; GFX12-GISEL-NEXT: ; %bb.3: ; %.kill
+; GFX12-GISEL-NEXT: s_and_not1_b64 s[6:7], s[6:7], exec
+; GFX12-GISEL-NEXT: s_cbranch_scc0 .LBB10_6
+; GFX12-GISEL-NEXT: ; %bb.4: ; %.kill
+; GFX12-GISEL-NEXT: s_mov_b64 exec, 0
+; GFX12-GISEL-NEXT: .LBB10_5: ; %.exit
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_or_b64 exec, exec, s[0:1]
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-GISEL-NEXT: export mrt0, v0, off, off, off done
+; GFX12-GISEL-NEXT: s_endpgm
+; GFX12-GISEL-NEXT: .LBB10_6:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, 0
+; GFX12-GISEL-NEXT: export mrt0, off, off, off, off done
+; GFX12-GISEL-NEXT: s_endpgm
+ %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %index)
+ %s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %val)
+ %ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
+ %ptr2 = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %samp_in, i32 %s_idx2
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
+ %samp = load <4 x i32>, <4 x i32> addrspace(4) * %ptr2, align 32
+ %r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token, <4 x float> %r)
+ %r2 = extractelement <4 x float> %r1, i32 0
+ %cond = fcmp olt float %r2, 0.000000e+00
+ br i1 %cond, label %.kill, label %.exit
+
+.kill:
+ call void @llvm.amdgcn.kill(i1 false)
+ br label %.exit
+
+.exit:
+ call void @llvm.amdgcn.exp.f32(i32 immarg 0, i32 immarg 1, float 0.000000e+00, float undef, float undef, float undef, i1 immarg true, i1 immarg true)
+ ret void
+}
+
+
+define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %s_in,
+; VI-SDAG-LABEL: test_waterfall_multi_begin:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; VI-SDAG-NEXT: s_wqm_b64 exec, exec
+; VI-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; VI-SDAG-NEXT: v_ashrrev_i32_e32 v8, 31, v7
+; VI-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; VI-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; VI-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[7:8]
+; VI-SDAG-NEXT: v_mov_b32_e32 v2, s1
+; VI-SDAG-NEXT: v_add_u32_e32 v0, vcc, s0, v0
+; VI-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; VI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, v2, v1, vcc
+; VI-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[3:4]
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, s3
+; VI-SDAG-NEXT: v_add_u32_e32 v2, vcc, s2, v2
+; VI-SDAG-NEXT: v_addc_u32_e32 v3, vcc, v4, v3, vcc
+; VI-SDAG-NEXT: flat_load_dwordx4 v[7:10], v[0:1]
+; VI-SDAG-NEXT: v_add_u32_e32 v0, vcc, 16, v0
+; VI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; VI-SDAG-NEXT: flat_load_dwordx4 v[11:14], v[0:1]
+; VI-SDAG-NEXT: flat_load_dwordx4 v[15:18], v[2:3]
+; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; VI-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v5
+; VI-SDAG-NEXT: v_readfirstlane_b32 s6, v6
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v5
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[6:7], s6, v6
+; VI-SDAG-NEXT: s_and_b64 s[2:3], s[2:3], s[6:7]
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; VI-SDAG-NEXT: s_waitcnt vmcnt(2)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v7
+; VI-SDAG-NEXT: v_readfirstlane_b32 s9, v8
+; VI-SDAG-NEXT: v_readfirstlane_b32 s10, v9
+; VI-SDAG-NEXT: v_readfirstlane_b32 s11, v10
+; VI-SDAG-NEXT: s_waitcnt vmcnt(1)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s12, v11
+; VI-SDAG-NEXT: v_readfirstlane_b32 s13, v12
+; VI-SDAG-NEXT: v_readfirstlane_b32 s14, v13
+; VI-SDAG-NEXT: v_readfirstlane_b32 s15, v14
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s16, v15
+; VI-SDAG-NEXT: v_readfirstlane_b32 s17, v16
+; VI-SDAG-NEXT: v_readfirstlane_b32 s18, v17
+; VI-SDAG-NEXT: v_readfirstlane_b32 s19, v18
+; VI-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; VI-SDAG-NEXT: ; implicit-def: $vgpr5
+; VI-SDAG-NEXT: ; implicit-def: $vgpr6
+; VI-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; VI-SDAG-NEXT: ; implicit-def: $vgpr15_vgpr16_vgpr17_vgpr18
+; VI-SDAG-NEXT: s_nop 3
+; VI-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB11_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; VI-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: ; return to shader part epilog
+;
+; VI-GISEL-LABEL: test_waterfall_multi_begin:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; VI-GISEL-NEXT: s_wqm_b64 exec, exec
+; VI-GISEL-NEXT: v_mov_b32_e32 v6, v3
+; VI-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; VI-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[2:3]
+; VI-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; VI-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; VI-GISEL-NEXT: v_add_u32_e32 v0, vcc, v2, v0
+; VI-GISEL-NEXT: v_ashrrev_i32_e32 v7, 31, v6
+; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, v3, v1, vcc
+; VI-GISEL-NEXT: v_lshlrev_b64 v[2:3], 4, v[6:7]
+; VI-GISEL-NEXT: v_mov_b32_e32 v7, s3
+; VI-GISEL-NEXT: v_mov_b32_e32 v6, s2
+; VI-GISEL-NEXT: v_add_u32_e32 v2, vcc, v6, v2
+; VI-GISEL-NEXT: v_addc_u32_e32 v3, vcc, v7, v3, vcc
+; VI-GISEL-NEXT: flat_load_dwordx4 v[6:9], v[0:1]
+; VI-GISEL-NEXT: v_add_u32_e32 v0, vcc, 16, v0
+; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; VI-GISEL-NEXT: flat_load_dwordx4 v[10:13], v[0:1]
+; VI-GISEL-NEXT: flat_load_dwordx4 v[14:17], v[2:3]
+; VI-GISEL-NEXT: s_mov_b32 s0, 0
+; VI-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; VI-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s1, v4
+; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[6:7], s1, v4
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v5
+; VI-GISEL-NEXT: s_and_b64 s[6:7], s[6:7], s[8:9]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[6:7], s[6:7]
+; VI-GISEL-NEXT: s_mov_b32 s1, s0
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; VI-GISEL-NEXT: s_waitcnt vmcnt(2)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v6
+; VI-GISEL-NEXT: v_readfirstlane_b32 s9, v7
+; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v8
+; VI-GISEL-NEXT: v_readfirstlane_b32 s11, v9
+; VI-GISEL-NEXT: s_waitcnt vmcnt(1)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v10
+; VI-GISEL-NEXT: v_readfirstlane_b32 s13, v11
+; VI-GISEL-NEXT: v_readfirstlane_b32 s14, v12
+; VI-GISEL-NEXT: v_readfirstlane_b32 s15, v13
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s16, v14
+; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v15
+; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v16
+; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v17
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; VI-GISEL-NEXT: ; implicit-def: $vgpr4
+; VI-GISEL-NEXT: ; implicit-def: $vgpr5
+; VI-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13
+; VI-GISEL-NEXT: ; implicit-def: $vgpr14_vgpr15_vgpr16_vgpr17
+; VI-GISEL-NEXT: s_nop 3
+; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[6:7]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB11_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; VI-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX9-SDAG-LABEL: test_waterfall_multi_begin:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v8, 31, v7
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX9-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[7:8]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v2, s1
+; GFX9-SDAG-NEXT: v_add_co_u32_e32 v0, vcc, s0, v0
+; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v1, vcc, v2, v1, vcc
+; GFX9-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[3:4]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v4, s3
+; GFX9-SDAG-NEXT: v_add_co_u32_e32 v2, vcc, s2, v2
+; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v3, vcc, v4, v3, vcc
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[11:14], v[0:1], off offset:16
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[7:10], v[0:1], off
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[15:18], v[2:3], off
+; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX9-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v5
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s6, v6
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v5
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[6:7], s6, v6
+; GFX9-SDAG-NEXT: s_and_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v7
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s9, v8
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s10, v9
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s11, v10
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s12, v11
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s13, v12
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s14, v13
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s15, v14
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s16, v15
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s17, v16
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s18, v17
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s19, v18
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr15_vgpr16_vgpr17_vgpr18
+; GFX9-SDAG-NEXT: s_nop 3
+; GFX9-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB11_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX9-GISEL-LABEL: test_waterfall_multi_begin:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v6, v3
+; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; GFX9-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[2:3]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX9-GISEL-NEXT: v_add_co_u32_e32 v0, vcc, v2, v0
+; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v7, 31, v6
+; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v1, vcc, v3, v1, vcc
+; GFX9-GISEL-NEXT: v_lshlrev_b64 v[2:3], 4, v[6:7]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v7, s3
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v6, s2
+; GFX9-GISEL-NEXT: v_add_co_u32_e32 v2, vcc, v6, v2
+; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v3, vcc, v7, v3, vcc
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[6:9], v[0:1], off
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[10:13], v[0:1], off offset:16
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[14:17], v[2:3], off
+; GFX9-GISEL-NEXT: s_mov_b32 s0, 0
+; GFX9-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX9-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s1, v4
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[6:7], s1, v4
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v5
+; GFX9-GISEL-NEXT: s_and_b64 s[6:7], s[6:7], s[8:9]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[6:7], s[6:7]
+; GFX9-GISEL-NEXT: s_mov_b32 s1, s0
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(2)
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v6
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s9, v7
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v8
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s11, v9
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v10
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s13, v11
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s14, v12
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s15, v13
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s16, v14
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v15
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v16
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v17
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr14_vgpr15_vgpr16_vgpr17
+; GFX9-GISEL-NEXT: s_nop 3
+; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[6:7]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB11_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-32-SDAG-LABEL: test_waterfall_multi_begin:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-SDAG-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v8, 31, v7
+; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[3:4]
+; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[7:8]
+; GFX10-32-SDAG-NEXT: v_add_co_u32 v0, vcc_lo, s0, v0
+; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, s1, v1, vcc_lo
+; GFX10-32-SDAG-NEXT: v_add_co_u32 v2, vcc_lo, s2, v2
+; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v3, vcc_lo, s3, v3, vcc_lo
+; GFX10-32-SDAG-NEXT: s_clause 0x1
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[11:14], v[0:1], off offset:16
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[7:10], v[0:1], off
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[15:18], v[2:3], off
+; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s2, v5
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s3, v6
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s2, v5
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s3, v6
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s8, v7
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s9, v8
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s10, v9
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s11, v10
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s12, v11
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s13, v12
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s14, v13
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s15, v14
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s16, v15
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s17, v16
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s18, v17
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s19, v18
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr15_vgpr16_vgpr17_vgpr18
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB11_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-SDAG-NEXT: s_and_b32 exec_lo, exec_lo, s4
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-32-GISEL-LABEL: test_waterfall_multi_begin:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-GISEL-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v6, v3
+; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v9, s3
+; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v7, 31, v6
+; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[2:3]
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v8, s2
+; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[6:7], 4, v[6:7]
+; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-GISEL-NEXT: v_add_co_u32 v0, vcc_lo, v2, v0
+; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
+; GFX10-32-GISEL-NEXT: v_add_co_u32 v2, vcc_lo, v8, v6
+; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v3, vcc_lo, v9, v7, vcc_lo
+; GFX10-32-GISEL-NEXT: s_clause 0x1
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[6:9], v[0:1], off
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[10:13], v[0:1], off offset:16
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[14:17], v[2:3], off
+; GFX10-32-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v4
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s3, v5
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s2, v4
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s3, v5
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(2)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s8, v6
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s9, v7
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s10, v8
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s11, v9
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s12, v10
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s13, v11
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s14, v12
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s15, v13
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s16, v14
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s17, v15
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s18, v16
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s19, v17
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr14_vgpr15_vgpr16_vgpr17
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB11_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-GISEL-NEXT: s_and_b32 exec_lo, exec_lo, s4
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-64-SDAG-LABEL: test_waterfall_multi_begin:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v8, 31, v7
+; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[3:4]
+; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[7:8]
+; GFX10-64-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc, s1, v1, vcc
+; GFX10-64-SDAG-NEXT: v_add_co_u32 v2, vcc, s2, v2
+; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v3, vcc, s3, v3, vcc
+; GFX10-64-SDAG-NEXT: s_clause 0x1
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[11:14], v[0:1], off offset:16
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[7:10], v[0:1], off
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[15:18], v[2:3], off
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s6, v5
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s7, v6
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v5
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s7, v6
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s8, v7
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s9, v8
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s10, v9
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s11, v10
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s12, v11
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s13, v12
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s14, v13
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s15, v14
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s16, v15
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s17, v16
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s18, v17
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s19, v18
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr15_vgpr16_vgpr17_vgpr18
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB11_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-64-GISEL-LABEL: test_waterfall_multi_begin:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v6, v3
+; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v9, s3
+; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v7, 31, v6
+; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[2:3]
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v8, s2
+; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[6:7], 4, v[6:7]
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc, v3, v1, vcc
+; GFX10-64-GISEL-NEXT: v_add_co_u32 v2, vcc, v8, v6
+; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v3, vcc, v9, v7, vcc
+; GFX10-64-GISEL-NEXT: s_clause 0x1
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[6:9], v[0:1], off
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[10:13], v[0:1], off offset:16
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[14:17], v[2:3], off
+; GFX10-64-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s6, v4
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s7, v5
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s6, v4
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s7, v5
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(2)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s8, v6
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s9, v7
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s10, v8
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s11, v9
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s12, v10
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s13, v11
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s14, v12
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s15, v13
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s16, v14
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s17, v15
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s18, v16
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s19, v17
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr14_vgpr15_vgpr16_vgpr17
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB11_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX1150-SDAG-LABEL: test_waterfall_multi_begin:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_4) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v8, 31, v7
+; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[3:4]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[7:8]
+; GFX1150-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_4)
+; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s1, v1, vcc
+; GFX1150-SDAG-NEXT: v_add_co_u32 v2, vcc, s2, v2
+; GFX1150-SDAG-NEXT: s_waitcnt_depctr depctr_va_vcc(0)
+; GFX1150-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v3, null, s3, v3, vcc
+; GFX1150-SDAG-NEXT: s_clause 0x1
+; GFX1150-SDAG-NEXT: global_load_b128 v[11:14], v[0:1], off offset:16
+; GFX1150-SDAG-NEXT: global_load_b128 v[7:10], v[0:1], off
+; GFX1150-SDAG-NEXT: global_load_b128 v[15:18], v[2:3], off
+; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s6, v5
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s7, v6
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v5
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s7, v6
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s8, v7
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s9, v8
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s10, v9
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s11, v10
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s12, v11
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s13, v12
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s14, v13
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s15, v14
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s16, v15
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s17, v16
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s18, v17
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s19, v18
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr15_vgpr16_vgpr17_vgpr18
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB11_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX1150-GISEL-LABEL: test_waterfall_multi_begin:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v6, v3
+; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v9, s3
+; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v7, 31, v6
+; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[2:3]
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v8, s2
+; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[6:7], 4, v[6:7]
+; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_3)
+; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX1150-GISEL-NEXT: v_add_co_u32 v2, vcc, v8, v6
+; GFX1150-GISEL-NEXT: s_waitcnt_depctr depctr_va_vcc(0)
+; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v3, null, v9, v7, vcc
+; GFX1150-GISEL-NEXT: s_clause 0x1
+; GFX1150-GISEL-NEXT: global_load_b128 v[6:9], v[0:1], off
+; GFX1150-GISEL-NEXT: global_load_b128 v[10:13], v[0:1], off offset:16
+; GFX1150-GISEL-NEXT: global_load_b128 v[14:17], v[2:3], off
+; GFX1150-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s6, v4
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s7, v5
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s6, v4
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s7, v5
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(2)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s8, v6
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s9, v7
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s10, v8
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s11, v9
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s12, v10
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s13, v11
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s14, v12
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s15, v13
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s16, v14
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s17, v15
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s18, v16
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s19, v17
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr14_vgpr15_vgpr16_vgpr17
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB11_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX12-SDAG-LABEL: test_waterfall_multi_begin:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_4) | instskip(NEXT) | instid1(VALU_DEP_4)
+; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v8, 31, v7
+; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[2:3], 4, v[3:4]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[0:1], 5, v[7:8]
+; GFX12-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_4)
+; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s1, v1, vcc
+; GFX12-SDAG-NEXT: v_add_co_u32 v2, vcc, s2, v2
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_vcc(0)
+; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v3, null, s3, v3, vcc
+; GFX12-SDAG-NEXT: s_clause 0x1
+; GFX12-SDAG-NEXT: global_load_b128 v[11:14], v[0:1], off offset:16
+; GFX12-SDAG-NEXT: global_load_b128 v[7:10], v[0:1], off
+; GFX12-SDAG-NEXT: global_load_b128 v[15:18], v[2:3], off
+; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s6, v5
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s7, v6
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v5
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s7, v6
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s8, v7
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s9, v8
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s10, v9
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s11, v10
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s12, v11
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s13, v12
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s14, v13
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s15, v14
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x0
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s16, v15
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s17, v16
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s18, v17
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s19, v18
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr15_vgpr16_vgpr17_vgpr18
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB11_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX12-GISEL-LABEL: test_waterfall_multi_begin:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v6, v3
+; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v9, s3
+; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v7, 31, v6
+; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[0:1], 5, v[2:3]
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v8, s2
+; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[6:7], 4, v[6:7]
+; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_3)
+; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX12-GISEL-NEXT: v_add_co_u32 v2, vcc, v8, v6
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_vcc(0)
+; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v3, null, v9, v7, vcc
+; GFX12-GISEL-NEXT: s_clause 0x1
+; GFX12-GISEL-NEXT: global_load_b128 v[6:9], v[0:1], off
+; GFX12-GISEL-NEXT: global_load_b128 v[10:13], v[0:1], off offset:16
+; GFX12-GISEL-NEXT: global_load_b128 v[14:17], v[2:3], off
+; GFX12-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s6, v4
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s7, v5
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s6, v4
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s7, v5
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x2
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s8, v6
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s9, v7
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s10, v8
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s11, v9
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s12, v10
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s13, v11
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s14, v12
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s15, v13
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s16, v14
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s17, v15
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s18, v16
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s19, v17
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr14_vgpr15_vgpr16_vgpr17
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB11_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: ; return to shader part epilog
+ i32 %idx1, i32 %idx2, i32 %s_idx, i32 %s_idx2) #1 {
+ %rptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
+ %sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
+ %srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
+ %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
+ %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok, i32 %idx2)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok1, <8 x i32> %rsrc)
+ %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok1, <4 x i32> %srsrc)
+ %r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
+
+ ret <4 x float> %r1
+}
+
+define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %s_in,
+; VI-SDAG-LABEL: test_waterfall_full_idx_multi_begin:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; VI-SDAG-NEXT: s_wqm_b64 exec, exec
+; VI-SDAG-NEXT: v_mov_b32_e32 v2, v0
+; VI-SDAG-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; VI-SDAG-NEXT: v_lshlrev_b64 v[2:3], 5, v[2:3]
+; VI-SDAG-NEXT: v_mov_b32_e32 v0, s1
+; VI-SDAG-NEXT: v_add_u32_e32 v8, vcc, s0, v2
+; VI-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; VI-SDAG-NEXT: v_addc_u32_e32 v9, vcc, v0, v3, vcc
+; VI-SDAG-NEXT: v_lshlrev_b64 v[0:1], 4, v[1:2]
+; VI-SDAG-NEXT: v_mov_b32_e32 v2, s3
+; VI-SDAG-NEXT: v_add_u32_e32 v0, vcc, s2, v0
+; VI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, v2, v1, vcc
+; VI-SDAG-NEXT: v_add_u32_e32 v2, vcc, 16, v8
+; VI-SDAG-NEXT: v_addc_u32_e32 v3, vcc, 0, v9, vcc
+; VI-SDAG-NEXT: flat_load_dwordx4 v[4:7], v[8:9]
+; VI-SDAG-NEXT: flat_load_dwordx4 v[8:11], v[2:3]
+; VI-SDAG-NEXT: flat_load_dwordx4 v[12:15], v[0:1]
+; VI-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; VI-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: s_waitcnt vmcnt(2)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s0, v4
+; VI-SDAG-NEXT: v_readfirstlane_b32 s1, v5
+; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v6
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[16:17], s0, v4
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s1, v5
+; VI-SDAG-NEXT: v_readfirstlane_b32 s3, v7
+; VI-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s2, v6
+; VI-SDAG-NEXT: s_waitcnt vmcnt(1)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s4, v8
+; VI-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s3, v7
+; VI-SDAG-NEXT: v_readfirstlane_b32 s5, v9
+; VI-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s4, v8
+; VI-SDAG-NEXT: v_readfirstlane_b32 s6, v10
+; VI-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s5, v9
+; VI-SDAG-NEXT: v_readfirstlane_b32 s7, v11
+; VI-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s6, v10
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s12, v12
+; VI-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s7, v11
+; VI-SDAG-NEXT: v_readfirstlane_b32 s13, v13
+; VI-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s12, v12
+; VI-SDAG-NEXT: v_readfirstlane_b32 s14, v14
+; VI-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s13, v13
+; VI-SDAG-NEXT: v_readfirstlane_b32 s15, v15
+; VI-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s14, v14
+; VI-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s15, v15
+; VI-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[16:17], s[16:17]
+; VI-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; VI-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; VI-SDAG-NEXT: image_sample v[0:3], v[0:1], s[0:7], s[12:15] dmask:0xf
+; VI-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
+; VI-SDAG-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[16:17]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB12_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[10:11]
+; VI-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: ; return to shader part epilog
+;
+; VI-GISEL-LABEL: test_waterfall_full_idx_multi_begin:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; VI-GISEL-NEXT: s_wqm_b64 exec, exec
+; VI-GISEL-NEXT: v_mov_b32_e32 v2, v1
+; VI-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; VI-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[0:1]
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, s1
+; VI-GISEL-NEXT: v_mov_b32_e32 v3, s0
+; VI-GISEL-NEXT: v_add_u32_e32 v0, vcc, v3, v0
+; VI-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, v4, v1, vcc
+; VI-GISEL-NEXT: v_lshlrev_b64 v[2:3], 4, v[2:3]
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, s3
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, s2
+; VI-GISEL-NEXT: v_add_u32_e32 v2, vcc, v4, v2
+; VI-GISEL-NEXT: v_addc_u32_e32 v3, vcc, v5, v3, vcc
+; VI-GISEL-NEXT: flat_load_dwordx4 v[4:7], v[0:1]
+; VI-GISEL-NEXT: v_add_u32_e32 v0, vcc, 16, v0
+; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; VI-GISEL-NEXT: flat_load_dwordx4 v[8:11], v[0:1]
+; VI-GISEL-NEXT: flat_load_dwordx4 v[12:15], v[2:3]
+; VI-GISEL-NEXT: s_mov_b32 s10, 0
+; VI-GISEL-NEXT: s_mov_b64 s[12:13], exec
+; VI-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: s_waitcnt vmcnt(2)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s0, v4
+; VI-GISEL-NEXT: v_readfirstlane_b32 s1, v5
+; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v6
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[14:15], s0, v4
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s1, v5
+; VI-GISEL-NEXT: v_readfirstlane_b32 s3, v7
+; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s2, v6
+; VI-GISEL-NEXT: s_waitcnt vmcnt(1)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s4, v8
+; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s3, v7
+; VI-GISEL-NEXT: v_readfirstlane_b32 s5, v9
+; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s4, v8
+; VI-GISEL-NEXT: v_readfirstlane_b32 s6, v10
+; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s5, v9
+; VI-GISEL-NEXT: v_readfirstlane_b32 s7, v11
+; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s6, v10
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s16, v12
+; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s7, v11
+; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v13
+; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s16, v12
+; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v14
+; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s17, v13
+; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v15
+; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s18, v14
+; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s19, v15
+; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[14:15], s[14:15]
+; VI-GISEL-NEXT: s_mov_b32 s11, s10
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s10
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s11
+; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[0:7], s[16:19] dmask:0xf
+; VI-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
+; VI-GISEL-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[14:15]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB12_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[12:13]
+; VI-GISEL-NEXT: s_and_b64 exec, exec, s[8:9]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX9-SDAG-LABEL: test_waterfall_full_idx_multi_begin:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX9-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v2, v0
+; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX9-SDAG-NEXT: v_lshlrev_b64 v[2:3], 5, v[2:3]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v0, s1
+; GFX9-SDAG-NEXT: v_add_co_u32_e32 v16, vcc, s0, v2
+; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v17, vcc, v0, v3, vcc
+; GFX9-SDAG-NEXT: v_lshlrev_b64 v[0:1], 4, v[1:2]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v2, s3
+; GFX9-SDAG-NEXT: v_add_co_u32_e32 v0, vcc, s2, v0
+; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v1, vcc, v2, v1, vcc
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[8:11], v[16:17], off offset:16
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[4:7], v[16:17], off
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[12:15], v[0:1], off
+; GFX9-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; GFX9-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s0, v4
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s1, v5
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v6
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[16:17], s0, v4
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s1, v5
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s3, v7
+; GFX9-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s2, v6
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s4, v8
+; GFX9-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s3, v7
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s5, v9
+; GFX9-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s4, v8
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s6, v10
+; GFX9-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s5, v9
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s7, v11
+; GFX9-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s6, v10
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s12, v12
+; GFX9-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s7, v11
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s13, v13
+; GFX9-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s12, v12
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s14, v14
+; GFX9-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s13, v13
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s15, v15
+; GFX9-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s14, v14
+; GFX9-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[18:19], s15, v15
+; GFX9-SDAG-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[16:17], s[16:17]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; GFX9-SDAG-NEXT: image_sample v[0:3], v[0:1], s[0:7], s[12:15] dmask:0xf
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[16:17]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB12_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[10:11]
+; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX9-GISEL-LABEL: test_waterfall_full_idx_multi_begin:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX9-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v2, v1
+; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX9-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[0:1]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, s1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v3, s0
+; GFX9-GISEL-NEXT: v_add_co_u32_e32 v0, vcc, v3, v0
+; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v1, vcc, v4, v1, vcc
+; GFX9-GISEL-NEXT: v_lshlrev_b64 v[2:3], 4, v[2:3]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v5, s3
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, s2
+; GFX9-GISEL-NEXT: v_add_co_u32_e32 v2, vcc, v4, v2
+; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v3, vcc, v5, v3, vcc
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[4:7], v[0:1], off
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[8:11], v[0:1], off offset:16
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[12:15], v[2:3], off
+; GFX9-GISEL-NEXT: s_mov_b32 s10, 0
+; GFX9-GISEL-NEXT: s_mov_b64 s[12:13], exec
+; GFX9-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(2)
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s0, v4
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s1, v5
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v6
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[14:15], s0, v4
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s1, v5
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s3, v7
+; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s2, v6
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s4, v8
+; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s3, v7
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s5, v9
+; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s4, v8
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s6, v10
+; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s5, v9
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s7, v11
+; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s6, v10
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s16, v12
+; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s7, v11
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v13
+; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s16, v12
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v14
+; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s17, v13
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v15
+; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s18, v14
+; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s19, v15
+; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[14:15], s[14:15]
+; GFX9-GISEL-NEXT: s_mov_b32 s11, s10
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s10
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s11
+; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[0:7], s[16:19] dmask:0xf
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[14:15]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB12_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[12:13]
+; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[8:9]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-32-SDAG-LABEL: test_waterfall_full_idx_multi_begin:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-SDAG-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v3, v0
+; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[0:1], 4, v[1:2]
+; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[3:4], 5, v[3:4]
+; GFX10-32-SDAG-NEXT: v_add_co_u32 v2, vcc_lo, s0, v3
+; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v3, vcc_lo, s1, v4, vcc_lo
+; GFX10-32-SDAG-NEXT: v_add_co_u32 v0, vcc_lo, s2, v0
+; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, s3, v1, vcc_lo
+; GFX10-32-SDAG-NEXT: s_clause 0x1
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[8:11], v[2:3], off offset:16
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[4:7], v[2:3], off
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[12:15], v[0:1], off
+; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s8, v4
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s9, v5
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s10, v6
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s11, v7
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s12, v8
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s13, v9
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s14, v10
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s15, v11
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s16, v12
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s17, v13
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s18, v14
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s19, v15
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s8, v4
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s9, v5
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s10, v6
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s11, v7
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s12, v8
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s13, v9
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s14, v10
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s15, v11
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s16, v12
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s17, v13
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s18, v14
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s19, v15
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB12_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-SDAG-NEXT: s_and_b32 exec_lo, exec_lo, s4
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-32-GISEL-LABEL: test_waterfall_full_idx_multi_begin:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-GISEL-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v2, v1
+; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v5, s1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v4, s0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v7, s3
+; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[0:1]
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v6, s2
+; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[2:3], 4, v[2:3]
+; GFX10-32-GISEL-NEXT: v_add_co_u32 v0, vcc_lo, v4, v0
+; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, v5, v1, vcc_lo
+; GFX10-32-GISEL-NEXT: v_add_co_u32 v2, vcc_lo, v6, v2
+; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v3, vcc_lo, v7, v3, vcc_lo
+; GFX10-32-GISEL-NEXT: s_clause 0x1
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[4:7], v[0:1], off
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[8:11], v[0:1], off offset:16
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[12:15], v[2:3], off
+; GFX10-32-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(2)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s8, v4
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s9, v5
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s10, v6
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s11, v7
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s12, v8
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s13, v9
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s14, v10
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s15, v11
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s16, v12
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s17, v13
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s18, v14
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s19, v15
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s8, v4
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s9, v5
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s10, v6
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s11, v7
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s12, v8
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s13, v9
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s14, v10
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s15, v11
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s16, v12
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s17, v13
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s18, v14
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s19, v15
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB12_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-GISEL-NEXT: s_and_b32 exec_lo, exec_lo, s4
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-64-SDAG-LABEL: test_waterfall_full_idx_multi_begin:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v3, v0
+; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[0:1], 4, v[1:2]
+; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[3:4], 5, v[3:4]
+; GFX10-64-SDAG-NEXT: v_add_co_u32 v2, vcc, s0, v3
+; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v3, vcc, s1, v4, vcc
+; GFX10-64-SDAG-NEXT: v_add_co_u32 v0, vcc, s2, v0
+; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc, s3, v1, vcc
+; GFX10-64-SDAG-NEXT: s_clause 0x1
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[8:11], v[2:3], off offset:16
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[4:7], v[2:3], off
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[12:15], v[0:1], off
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s8, v4
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s9, v5
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s10, v6
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s11, v7
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s12, v8
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s13, v9
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s14, v10
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s15, v11
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s16, v12
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s17, v13
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s18, v14
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s19, v15
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s8, v4
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s9, v5
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v6
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s11, v7
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s12, v8
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s13, v9
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s14, v10
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s15, v11
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s16, v12
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s17, v13
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s18, v14
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s19, v15
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB12_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-64-GISEL-LABEL: test_waterfall_full_idx_multi_begin:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v2, v1
+; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v5, s1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v4, s0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v7, s3
+; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[0:1]
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v6, s2
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[2:3], 4, v[2:3]
+; GFX10-64-GISEL-NEXT: v_add_co_u32 v0, vcc, v4, v0
+; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc, v5, v1, vcc
+; GFX10-64-GISEL-NEXT: v_add_co_u32 v2, vcc, v6, v2
+; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v3, vcc, v7, v3, vcc
+; GFX10-64-GISEL-NEXT: s_clause 0x1
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[4:7], v[0:1], off
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[8:11], v[0:1], off offset:16
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[12:15], v[2:3], off
+; GFX10-64-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(2)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s8, v4
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s9, v5
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s10, v6
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s11, v7
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s12, v8
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s13, v9
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s14, v10
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s15, v11
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s16, v12
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s17, v13
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s18, v14
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s19, v15
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s8, v4
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s9, v5
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s10, v6
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s11, v7
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s12, v8
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s13, v9
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s14, v10
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s15, v11
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s16, v12
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s17, v13
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s18, v14
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s19, v15
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB12_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX1150-SDAG-LABEL: test_waterfall_full_idx_multi_begin:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v3, v0
+; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[0:1], 4, v[1:2]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[3:4], 5, v[3:4]
+; GFX1150-SDAG-NEXT: v_add_co_u32 v2, vcc, s0, v3
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_4)
+; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v3, null, s1, v4, vcc
+; GFX1150-SDAG-NEXT: v_add_co_u32 v0, vcc, s2, v0
+; GFX1150-SDAG-NEXT: s_waitcnt_depctr depctr_va_vcc(0)
+; GFX1150-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s3, v1, vcc
+; GFX1150-SDAG-NEXT: s_clause 0x1
+; GFX1150-SDAG-NEXT: global_load_b128 v[8:11], v[2:3], off offset:16
+; GFX1150-SDAG-NEXT: global_load_b128 v[4:7], v[2:3], off
+; GFX1150-SDAG-NEXT: global_load_b128 v[12:15], v[0:1], off
+; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-SDAG-NEXT: s_set_inst_prefetch_distance 0x1
+; GFX1150-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s8, v4
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s9, v5
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s10, v6
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s11, v7
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s12, v8
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s13, v9
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s14, v10
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s15, v11
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s16, v12
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s17, v13
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s18, v14
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s19, v15
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s8, v4
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s9, v5
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v6
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s11, v7
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s12, v8
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s13, v9
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s14, v10
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s15, v11
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s16, v12
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s17, v13
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s18, v14
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s19, v15
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB12_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_set_inst_prefetch_distance 0x2
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX1150-GISEL-LABEL: test_waterfall_full_idx_multi_begin:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v2, v1
+; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v5, s1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, s0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v7, s3
+; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[0:1]
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v6, s2
+; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[2:3], 4, v[2:3]
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_3) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-GISEL-NEXT: v_add_co_u32 v0, vcc, v4, v0
+; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v5, v1, vcc
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_3) | instskip(SKIP_1) | instid1(VALU_DEP_4)
+; GFX1150-GISEL-NEXT: v_add_co_u32 v2, vcc, v6, v2
+; GFX1150-GISEL-NEXT: s_waitcnt_depctr depctr_va_vcc(0)
+; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v3, null, v7, v3, vcc
+; GFX1150-GISEL-NEXT: s_clause 0x1
+; GFX1150-GISEL-NEXT: global_load_b128 v[4:7], v[0:1], off
+; GFX1150-GISEL-NEXT: global_load_b128 v[8:11], v[0:1], off offset:16
+; GFX1150-GISEL-NEXT: global_load_b128 v[12:15], v[2:3], off
+; GFX1150-GISEL-NEXT: s_set_inst_prefetch_distance 0x1
+; GFX1150-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(2)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s8, v4
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s9, v5
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s10, v6
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s11, v7
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s12, v8
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s13, v9
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s14, v10
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s15, v11
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s16, v12
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s17, v13
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s18, v14
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s19, v15
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s8, v4
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s9, v5
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s10, v6
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s11, v7
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s12, v8
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s13, v9
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s14, v10
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s15, v11
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s16, v12
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s17, v13
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s18, v14
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s19, v15
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB12_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_set_inst_prefetch_distance 0x2
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX12-SDAG-LABEL: test_waterfall_full_idx_multi_begin:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v3, v0
+; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[0:1], 4, v[1:2]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[3:4], 5, v[3:4]
+; GFX12-SDAG-NEXT: v_add_co_u32 v2, vcc, s0, v3
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_4)
+; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v3, null, s1, v4, vcc
+; GFX12-SDAG-NEXT: v_add_co_u32 v0, vcc, s2, v0
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_vcc(0)
+; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s3, v1, vcc
+; GFX12-SDAG-NEXT: s_clause 0x1
+; GFX12-SDAG-NEXT: global_load_b128 v[8:11], v[2:3], off offset:16
+; GFX12-SDAG-NEXT: global_load_b128 v[4:7], v[2:3], off
+; GFX12-SDAG-NEXT: global_load_b128 v[12:15], v[0:1], off
+; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s8, v4
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s9, v5
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s10, v6
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s11, v7
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s12, v8
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s13, v9
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s14, v10
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s15, v11
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x0
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s16, v12
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s17, v13
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s18, v14
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s19, v15
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s8, v4
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s9, v5
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v6
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s11, v7
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s12, v8
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s13, v9
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s14, v10
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s15, v11
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s16, v12
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s17, v13
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s18, v14
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s19, v15
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB12_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX12-GISEL-LABEL: test_waterfall_full_idx_multi_begin:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v2, v1
+; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v5, s1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, s0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v7, s3
+; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[0:1], 5, v[0:1]
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v6, s2
+; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[2:3], 4, v[2:3]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_3) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_add_co_u32 v0, vcc, v4, v0
+; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v5, v1, vcc
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_3) | instskip(SKIP_1) | instid1(VALU_DEP_4)
+; GFX12-GISEL-NEXT: v_add_co_u32 v2, vcc, v6, v2
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_vcc(0)
+; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v3, null, v7, v3, vcc
+; GFX12-GISEL-NEXT: s_clause 0x1
+; GFX12-GISEL-NEXT: global_load_b128 v[4:7], v[0:1], off
+; GFX12-GISEL-NEXT: global_load_b128 v[8:11], v[0:1], off offset:16
+; GFX12-GISEL-NEXT: global_load_b128 v[12:15], v[2:3], off
+; GFX12-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x2
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s8, v4
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s9, v5
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s10, v6
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s11, v7
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s12, v8
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s13, v9
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s14, v10
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s15, v11
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s16, v12
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s17, v13
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s18, v14
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s19, v15
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s8, v4
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s9, v5
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s10, v6
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s11, v7
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s12, v8
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s13, v9
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s14, v10
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s15, v11
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s16, v12
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s17, v13
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s18, v14
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s19, v15
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB12_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: ; return to shader part epilog
+ i32 %s_idx, i32 %s_idx2) #1 {
+ %rptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
+ %sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
+ %srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
+ %tok = call i32 @llvm.amdgcn.waterfall.begin.v8i32(i32 0, <8 x i32> %rsrc)
+ %tok1 = call i32 @llvm.amdgcn.waterfall.begin.v4i32(i32 %tok, <4 x i32> %srsrc)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok1, <8 x i32> %rsrc)
+ %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok1, <4 x i32> %srsrc)
+ %r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
+
+ ret <4 x float> %r1
+}
+
+define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32> addrspace(4)* inreg %in,
+; VI-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_1:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; VI-SDAG-NEXT: s_wqm_b64 exec, exec
+; VI-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; VI-SDAG-NEXT: v_lshlrev_b64 v[0:1], 4, v[1:2]
+; VI-SDAG-NEXT: v_mov_b32_e32 v2, s3
+; VI-SDAG-NEXT: v_add_u32_e32 v0, vcc, s2, v0
+; VI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, v2, v1, vcc
+; VI-SDAG-NEXT: flat_load_dwordx4 v[5:8], v[0:1]
+; VI-SDAG-NEXT: s_mov_b32 s4, s5
+; VI-SDAG-NEXT: s_ashr_i32 s5, s5, 31
+; VI-SDAG-NEXT: s_lshl_b64 s[2:3], s[4:5], 5
+; VI-SDAG-NEXT: s_add_u32 s0, s0, s2
+; VI-SDAG-NEXT: s_addc_u32 s1, s1, s3
+; VI-SDAG-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x0
+; VI-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; VI-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s12, v4
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[12:13], s12, v4
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[16:17], s[12:13]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; VI-SDAG-NEXT: v_readfirstlane_b32 s13, v6
+; VI-SDAG-NEXT: v_readfirstlane_b32 s14, v7
+; VI-SDAG-NEXT: v_readfirstlane_b32 s15, v8
+; VI-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; VI-SDAG-NEXT: ; implicit-def: $vgpr4
+; VI-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; VI-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; VI-SDAG-NEXT: s_nop 2
+; VI-SDAG-NEXT: image_sample v[0:3], v[0:1], s[0:7], s[12:15] dmask:0xf
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[16:17]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB13_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[10:11]
+; VI-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: ; return to shader part epilog
+;
+; VI-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_1:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; VI-GISEL-NEXT: s_wqm_b64 exec, exec
+; VI-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; VI-GISEL-NEXT: v_lshlrev_b64 v[0:1], 4, v[1:2]
+; VI-GISEL-NEXT: v_mov_b32_e32 v2, s2
+; VI-GISEL-NEXT: v_mov_b32_e32 v3, s3
+; VI-GISEL-NEXT: v_add_u32_e32 v0, vcc, v2, v0
+; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, v3, v1, vcc
+; VI-GISEL-NEXT: flat_load_dwordx4 v[5:8], v[0:1]
+; VI-GISEL-NEXT: s_mov_b32 s8, s5
+; VI-GISEL-NEXT: s_ashr_i32 s9, s5, 31
+; VI-GISEL-NEXT: s_lshl_b64 s[2:3], s[8:9], 5
+; VI-GISEL-NEXT: s_add_u32 s0, s0, s2
+; VI-GISEL-NEXT: s_addc_u32 s1, s1, s3
+; VI-GISEL-NEXT: s_load_dwordx8 s[8:15], s[0:1], 0x0
+; VI-GISEL-NEXT: s_mov_b32 s0, 0
+; VI-GISEL-NEXT: v_mov_b32_e32 v9, s4
+; VI-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; VI-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s1, v9
+; VI-GISEL-NEXT: v_readfirstlane_b32 s16, v4
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s1, v9
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[16:17], s16, v4
+; VI-GISEL-NEXT: s_and_b64 s[4:5], s[4:5], s[16:17]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
+; VI-GISEL-NEXT: s_mov_b32 s1, s0
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s16, v5
+; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v6
+; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v7
+; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v8
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; VI-GISEL-NEXT: ; implicit-def: $vgpr9
+; VI-GISEL-NEXT: ; implicit-def: $vgpr4
+; VI-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; VI-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; VI-GISEL-NEXT: s_nop 2
+; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[4:5]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB13_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; VI-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX9-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_1:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX9-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-SDAG-NEXT: v_lshlrev_b64 v[0:1], 4, v[1:2]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v2, s3
+; GFX9-SDAG-NEXT: v_add_co_u32_e32 v0, vcc, s2, v0
+; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v1, vcc, v2, v1, vcc
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX9-SDAG-NEXT: s_mov_b32 s4, s5
+; GFX9-SDAG-NEXT: s_ashr_i32 s5, s5, 31
+; GFX9-SDAG-NEXT: s_lshl_b64 s[2:3], s[4:5], 5
+; GFX9-SDAG-NEXT: s_add_u32 s10, s0, s2
+; GFX9-SDAG-NEXT: s_addc_u32 s11, s1, s3
+; GFX9-SDAG-NEXT: s_load_dwordx8 s[0:7], s[10:11], 0x0
+; GFX9-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; GFX9-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s12, v4
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[12:13], s12, v4
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[16:17], s[12:13]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s13, v6
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s14, v7
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s15, v8
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; GFX9-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-SDAG-NEXT: s_nop 2
+; GFX9-SDAG-NEXT: image_sample v[0:3], v[0:1], s[0:7], s[12:15] dmask:0xf
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[16:17]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB13_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[10:11]
+; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX9-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_1:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX9-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-GISEL-NEXT: v_lshlrev_b64 v[0:1], 4, v[1:2]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v2, s2
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v3, s3
+; GFX9-GISEL-NEXT: v_add_co_u32_e32 v0, vcc, v2, v0
+; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v1, vcc, v3, v1, vcc
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX9-GISEL-NEXT: s_mov_b32 s8, s5
+; GFX9-GISEL-NEXT: s_ashr_i32 s9, s5, 31
+; GFX9-GISEL-NEXT: s_lshl_b64 s[2:3], s[8:9], 5
+; GFX9-GISEL-NEXT: s_add_u32 s0, s0, s2
+; GFX9-GISEL-NEXT: s_addc_u32 s1, s1, s3
+; GFX9-GISEL-NEXT: s_load_dwordx8 s[8:15], s[0:1], 0x0
+; GFX9-GISEL-NEXT: s_mov_b32 s0, 0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, s4
+; GFX9-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX9-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s1, v9
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s16, v4
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s1, v9
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[16:17], s16, v4
+; GFX9-GISEL-NEXT: s_and_b64 s[4:5], s[4:5], s[16:17]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
+; GFX9-GISEL-NEXT: s_mov_b32 s1, s0
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s16, v5
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v6
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v7
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v8
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; GFX9-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-GISEL-NEXT: s_nop 2
+; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[4:5]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB13_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-32-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_1:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: s_mov_b32 s8, exec_lo
+; GFX10-32-SDAG-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s4, s5
+; GFX10-32-SDAG-NEXT: s_ashr_i32 s5, s5, 31
+; GFX10-32-SDAG-NEXT: s_mov_b32 s9, exec_lo
+; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[0:1], 4, v[1:2]
+; GFX10-32-SDAG-NEXT: v_add_co_u32 v0, vcc_lo, s2, v0
+; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, s3, v1, vcc_lo
+; GFX10-32-SDAG-NEXT: s_lshl_b64 s[2:3], s[4:5], 5
+; GFX10-32-SDAG-NEXT: s_add_u32 s10, s0, s2
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX10-32-SDAG-NEXT: s_addc_u32 s11, s1, s3
+; GFX10-32-SDAG-NEXT: s_load_dwordx8 s[0:7], s[10:11], 0x0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s10, exec_lo
+; GFX10-32-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s11, v4
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s11, v4
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s13, v6
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s14, v7
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s15, v8
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[0:7], s[12:15] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s10, s10
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB13_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s9
+; GFX10-32-SDAG-NEXT: s_and_b32 exec_lo, exec_lo, s8
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-32-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_1:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: s_mov_b32 s6, exec_lo
+; GFX10-32-GISEL-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-32-GISEL-NEXT: s_mov_b32 s8, s5
+; GFX10-32-GISEL-NEXT: s_ashr_i32 s9, s5, 31
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v9, s4
+; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[0:1], 4, v[1:2]
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v2, s2
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v3, s3
+; GFX10-32-GISEL-NEXT: s_lshl_b64 s[2:3], s[8:9], 5
+; GFX10-32-GISEL-NEXT: s_add_u32 s0, s0, s2
+; GFX10-32-GISEL-NEXT: v_add_co_u32 v0, vcc_lo, v2, v0
+; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
+; GFX10-32-GISEL-NEXT: s_addc_u32 s1, s1, s3
+; GFX10-32-GISEL-NEXT: s_load_dwordx8 s[8:15], s[0:1], 0x0
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v9
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s3, v4
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s2, v9
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s3, v4
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s16, v5
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s17, v6
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s18, v7
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s19, v8
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB13_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-GISEL-NEXT: s_and_b32 exec_lo, exec_lo, s6
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-64-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_1:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX10-64-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-64-SDAG-NEXT: s_mov_b32 s4, s5
+; GFX10-64-SDAG-NEXT: s_ashr_i32 s5, s5, 31
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[12:13], exec
+; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[0:1], 4, v[1:2]
+; GFX10-64-SDAG-NEXT: v_add_co_u32 v0, vcc, s2, v0
+; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc, s3, v1, vcc
+; GFX10-64-SDAG-NEXT: s_lshl_b64 s[2:3], s[4:5], 5
+; GFX10-64-SDAG-NEXT: s_add_u32 s10, s0, s2
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX10-64-SDAG-NEXT: s_addc_u32 s11, s1, s3
+; GFX10-64-SDAG-NEXT: s_load_dwordx8 s[0:7], s[10:11], 0x0
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; GFX10-64-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s14, v4
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s14, v4
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s16, v5
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s17, v6
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s18, v7
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s19, v8
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[0:7], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[12:13], s[12:13]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB13_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[10:11]
+; GFX10-64-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-64-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_1:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-64-GISEL-NEXT: s_mov_b32 s8, s5
+; GFX10-64-GISEL-NEXT: s_ashr_i32 s9, s5, 31
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v9, s4
+; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[0:1], 4, v[1:2]
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v2, s2
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v3, s3
+; GFX10-64-GISEL-NEXT: s_lshl_b64 s[2:3], s[8:9], 5
+; GFX10-64-GISEL-NEXT: s_add_u32 s0, s0, s2
+; GFX10-64-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc, v3, v1, vcc
+; GFX10-64-GISEL-NEXT: s_addc_u32 s1, s1, s3
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-GISEL-NEXT: s_load_dwordx8 s[8:15], s[0:1], 0x0
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s4, v9
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s5, v4
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v9
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s5, v4
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s16, v5
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s17, v6
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s18, v7
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s19, v8
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB13_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX1150-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_1:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX1150-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX1150-SDAG-NEXT: s_mov_b32 s4, s5
+; GFX1150-SDAG-NEXT: s_ashr_i32 s5, s5, 31
+; GFX1150-SDAG-NEXT: s_mov_b64 s[12:13], exec
+; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[0:1], 4, v[1:2]
+; GFX1150-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_add_co_u32 v0, vcc, s2, v0
+; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s3, v1, vcc
+; GFX1150-SDAG-NEXT: s_lshl_b64 s[2:3], s[4:5], 5
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_add_u32 s0, s0, s2
+; GFX1150-SDAG-NEXT: global_load_b128 v[5:8], v[0:1], off
+; GFX1150-SDAG-NEXT: s_addc_u32 s1, s1, s3
+; GFX1150-SDAG-NEXT: s_load_b256 s[0:7], s[0:1], 0x0
+; GFX1150-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s14, v4
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s14, v4
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s16, v5
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s17, v6
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s18, v7
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s19, v8
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1150-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[0:7], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[12:13], s[12:13]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB13_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[10:11]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX1150-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_1:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX1150-GISEL-NEXT: s_mov_b32 s8, s5
+; GFX1150-GISEL-NEXT: s_ashr_i32 s9, s5, 31
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v9, s4
+; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[0:1], 4, v[1:2]
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v2, s2
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v3, s3
+; GFX1150-GISEL-NEXT: s_lshl_b64 s[2:3], s[8:9], 5
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-GISEL-NEXT: s_add_u32 s0, s0, s2
+; GFX1150-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX1150-GISEL-NEXT: s_addc_u32 s1, s1, s3
+; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-GISEL-NEXT: s_load_b256 s[8:15], s[0:1], 0x0
+; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX1150-GISEL-NEXT: global_load_b128 v[5:8], v[0:1], off
+; GFX1150-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s4, v9
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s5, v4
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v9
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s5, v4
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s16, v5
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s17, v6
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s18, v7
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s19, v8
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1150-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB13_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX12-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_1:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-SDAG-NEXT: s_mov_b32 s4, s5
+; GFX12-SDAG-NEXT: s_ashr_i32 s5, s5, 31
+; GFX12-SDAG-NEXT: s_mov_b64 s[12:13], exec
+; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[0:1], 4, v[1:2]
+; GFX12-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_add_co_u32 v0, vcc, s2, v0
+; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s3, v1, vcc
+; GFX12-SDAG-NEXT: s_lshl_b64 s[2:3], s[4:5], 5
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_add_nc_u64 s[0:1], s[0:1], s[2:3]
+; GFX12-SDAG-NEXT: global_load_b128 v[5:8], v[0:1], off
+; GFX12-SDAG-NEXT: s_load_b256 s[0:7], s[0:1], 0x0
+; GFX12-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s14, v4
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s14, v4
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x0
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s16, v5
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s17, v6
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s18, v7
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s19, v8
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[0:7], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[12:13], s[12:13]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB13_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[10:11]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX12-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_1:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-GISEL-NEXT: s_mov_b32 s8, s5
+; GFX12-GISEL-NEXT: s_ashr_i32 s9, s5, 31
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v9, s4
+; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[0:1], 4, v[1:2]
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v2, s2
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v3, s3
+; GFX12-GISEL-NEXT: s_lshl_b64 s[2:3], s[8:9], 5
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_add_co_u32 s0, s0, s2
+; GFX12-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX12-GISEL-NEXT: s_add_co_ci_u32 s1, s1, s3
+; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-GISEL-NEXT: s_load_b256 s[8:15], s[0:1], 0x0
+; GFX12-GISEL-NEXT: global_load_b128 v[5:8], v[0:1], off
+; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s4, v9
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s5, v4
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v9
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s5, v4
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s16, v5
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s17, v6
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s18, v7
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s19, v8
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB13_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: ; return to shader part epilog
+ <4 x i32> addrspace(4)* inreg %s_in, i32 inreg %idx1, i32 %idx2, i32 inreg %s_idx, i32 %s_idx2) #1 {
+ %rptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
+ %sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
+ %srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
+ %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
+ %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok, i32 %idx2)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok1, <8 x i32> %rsrc)
+ %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok1, <4 x i32> %srsrc)
+ %r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
+
+ ret <4 x float> %r1
+}
+
+define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32> addrspace(4)* inreg %in,
+; VI-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_2:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; VI-SDAG-NEXT: s_mov_b32 s4, s5
+; VI-SDAG-NEXT: s_wqm_b64 exec, exec
+; VI-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; VI-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; VI-SDAG-NEXT: v_mov_b32_e32 v2, s1
+; VI-SDAG-NEXT: v_add_u32_e32 v0, vcc, s0, v0
+; VI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, v2, v1, vcc
+; VI-SDAG-NEXT: flat_load_dwordx4 v[5:8], v[0:1]
+; VI-SDAG-NEXT: v_add_u32_e32 v0, vcc, 16, v0
+; VI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; VI-SDAG-NEXT: flat_load_dwordx4 v[9:12], v[0:1]
+; VI-SDAG-NEXT: s_ashr_i32 s5, s5, 31
+; VI-SDAG-NEXT: s_lshl_b64 s[0:1], s[4:5], 4
+; VI-SDAG-NEXT: s_add_u32 s0, s2, s0
+; VI-SDAG-NEXT: s_addc_u32 s1, s3, s1
+; VI-SDAG-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x0
+; VI-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; VI-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v4
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v4
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[16:17], s[8:9]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; VI-SDAG-NEXT: s_waitcnt vmcnt(1)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v5
+; VI-SDAG-NEXT: v_readfirstlane_b32 s9, v6
+; VI-SDAG-NEXT: v_readfirstlane_b32 s10, v7
+; VI-SDAG-NEXT: v_readfirstlane_b32 s11, v8
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s12, v9
+; VI-SDAG-NEXT: v_readfirstlane_b32 s13, v10
+; VI-SDAG-NEXT: v_readfirstlane_b32 s14, v11
+; VI-SDAG-NEXT: v_readfirstlane_b32 s15, v12
+; VI-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; VI-SDAG-NEXT: ; implicit-def: $vgpr4
+; VI-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; VI-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; VI-SDAG-NEXT: s_nop 2
+; VI-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[0:3] dmask:0xf
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[16:17]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB14_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[4:5]
+; VI-SDAG-NEXT: s_and_b64 exec, exec, s[6:7]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: ; return to shader part epilog
+;
+; VI-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_2:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; VI-GISEL-NEXT: s_wqm_b64 exec, exec
+; VI-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; VI-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; VI-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; VI-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; VI-GISEL-NEXT: v_add_u32_e32 v0, vcc, v2, v0
+; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, v3, v1, vcc
+; VI-GISEL-NEXT: flat_load_dwordx4 v[5:8], v[0:1]
+; VI-GISEL-NEXT: v_add_u32_e32 v0, vcc, 16, v0
+; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; VI-GISEL-NEXT: flat_load_dwordx4 v[9:12], v[0:1]
+; VI-GISEL-NEXT: s_mov_b32 s8, s5
+; VI-GISEL-NEXT: s_ashr_i32 s9, s5, 31
+; VI-GISEL-NEXT: s_lshl_b64 s[0:1], s[8:9], 4
+; VI-GISEL-NEXT: s_add_u32 s0, s2, s0
+; VI-GISEL-NEXT: s_addc_u32 s1, s3, s1
+; VI-GISEL-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x0
+; VI-GISEL-NEXT: s_mov_b32 s8, 0
+; VI-GISEL-NEXT: v_mov_b32_e32 v13, s4
+; VI-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; VI-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s9, v4
+; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v13
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s9, v4
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[12:13], s12, v13
+; VI-GISEL-NEXT: s_and_b64 s[10:11], s[10:11], s[12:13]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[10:11], s[10:11]
+; VI-GISEL-NEXT: s_mov_b32 s9, s8
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s8
+; VI-GISEL-NEXT: s_waitcnt vmcnt(1)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v5
+; VI-GISEL-NEXT: v_readfirstlane_b32 s13, v6
+; VI-GISEL-NEXT: v_readfirstlane_b32 s14, v7
+; VI-GISEL-NEXT: v_readfirstlane_b32 s15, v8
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s16, v9
+; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v10
+; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v11
+; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v12
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s9
+; VI-GISEL-NEXT: ; implicit-def: $vgpr4
+; VI-GISEL-NEXT: ; implicit-def: $vgpr13
+; VI-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; VI-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; VI-GISEL-NEXT: s_nop 2
+; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[12:19], s[0:3] dmask:0xf
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[10:11]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB14_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[4:5]
+; VI-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX9-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_2:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX9-SDAG-NEXT: s_mov_b32 s4, s5
+; GFX9-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v2, s1
+; GFX9-SDAG-NEXT: v_add_co_u32_e32 v0, vcc, s0, v0
+; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v1, vcc, v2, v1, vcc
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX9-SDAG-NEXT: s_ashr_i32 s5, s5, 31
+; GFX9-SDAG-NEXT: s_lshl_b64 s[0:1], s[4:5], 4
+; GFX9-SDAG-NEXT: s_add_u32 s0, s2, s0
+; GFX9-SDAG-NEXT: s_addc_u32 s1, s3, s1
+; GFX9-SDAG-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x0
+; GFX9-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v4
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v4
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[16:17], s[8:9]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v5
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s9, v6
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s10, v7
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s11, v8
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s12, v9
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s13, v10
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s14, v11
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s15, v12
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX9-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-SDAG-NEXT: s_nop 2
+; GFX9-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[0:3] dmask:0xf
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[16:17]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB14_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[4:5]
+; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX9-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_2:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX9-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX9-GISEL-NEXT: v_add_co_u32_e32 v0, vcc, v2, v0
+; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v1, vcc, v3, v1, vcc
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX9-GISEL-NEXT: s_mov_b32 s8, s5
+; GFX9-GISEL-NEXT: s_ashr_i32 s9, s5, 31
+; GFX9-GISEL-NEXT: s_lshl_b64 s[0:1], s[8:9], 4
+; GFX9-GISEL-NEXT: s_add_u32 s0, s2, s0
+; GFX9-GISEL-NEXT: s_addc_u32 s1, s3, s1
+; GFX9-GISEL-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x0
+; GFX9-GISEL-NEXT: s_mov_b32 s8, 0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v13, s4
+; GFX9-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s9, v4
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v13
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s9, v4
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[12:13], s12, v13
+; GFX9-GISEL-NEXT: s_and_b64 s[10:11], s[10:11], s[12:13]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[10:11], s[10:11]
+; GFX9-GISEL-NEXT: s_mov_b32 s9, s8
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s8
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v5
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s13, v6
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s14, v7
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s15, v8
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s16, v9
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v10
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v11
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v12
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s9
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr13
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX9-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-GISEL-NEXT: s_nop 2
+; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[12:19], s[0:3] dmask:0xf
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[10:11]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB14_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[4:5]
+; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-32-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_2:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s6, s5
+; GFX10-32-SDAG-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-32-SDAG-NEXT: s_ashr_i32 s7, s5, 31
+; GFX10-32-SDAG-NEXT: s_mov_b32 s5, exec_lo
+; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; GFX10-32-SDAG-NEXT: v_add_co_u32 v0, vcc_lo, s0, v0
+; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, s1, v1, vcc_lo
+; GFX10-32-SDAG-NEXT: s_lshl_b64 s[0:1], s[6:7], 4
+; GFX10-32-SDAG-NEXT: s_mov_b32 s6, exec_lo
+; GFX10-32-SDAG-NEXT: s_add_u32 s0, s2, s0
+; GFX10-32-SDAG-NEXT: s_clause 0x1
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX10-32-SDAG-NEXT: s_addc_u32 s1, s3, s1
+; GFX10-32-SDAG-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x0
+; GFX10-32-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s7, v4
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s7, v4
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s8, v5
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s9, v6
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s10, v7
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s11, v8
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s12, v9
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s13, v10
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s14, v11
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s15, v12
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[0:3] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s6, s6
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB14_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s5
+; GFX10-32-SDAG-NEXT: s_and_b32 exec_lo, exec_lo, s4
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-32-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_2:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: s_mov_b32 s6, exec_lo
+; GFX10-32-GISEL-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-32-GISEL-NEXT: s_mov_b32 s8, s5
+; GFX10-32-GISEL-NEXT: s_ashr_i32 s9, s5, 31
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v13, s4
+; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX10-32-GISEL-NEXT: s_lshl_b64 s[0:1], s[8:9], 4
+; GFX10-32-GISEL-NEXT: s_mov_b32 s5, exec_lo
+; GFX10-32-GISEL-NEXT: s_add_u32 s0, s2, s0
+; GFX10-32-GISEL-NEXT: s_addc_u32 s1, s3, s1
+; GFX10-32-GISEL-NEXT: v_add_co_u32 v0, vcc_lo, v2, v0
+; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
+; GFX10-32-GISEL-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x0
+; GFX10-32-GISEL-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-GISEL-NEXT: s_clause 0x1
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX10-32-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s7, v4
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s8, v13
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s7, v4
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s8, v13
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s9, v6
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s10, v7
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s11, v8
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s12, v9
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s13, v10
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s14, v11
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s15, v12
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[0:3] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s5, s5
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr13
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB14_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s4
+; GFX10-32-GISEL-NEXT: s_and_b32 exec_lo, exec_lo, s6
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-64-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_2:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-SDAG-NEXT: s_mov_b32 s4, s5
+; GFX10-64-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-64-SDAG-NEXT: s_ashr_i32 s5, s5, 31
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; GFX10-64-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc, s1, v1, vcc
+; GFX10-64-SDAG-NEXT: s_lshl_b64 s[0:1], s[4:5], 4
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-SDAG-NEXT: s_add_u32 s0, s2, s0
+; GFX10-64-SDAG-NEXT: s_clause 0x1
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX10-64-SDAG-NEXT: s_addc_u32 s1, s3, s1
+; GFX10-64-SDAG-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x0
+; GFX10-64-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s10, v4
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v4
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s13, v6
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s14, v7
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s15, v8
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s16, v9
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s17, v10
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s18, v11
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s19, v12
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[12:19], s[0:3] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[8:9], s[8:9]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB14_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[4:5]
+; GFX10-64-SDAG-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-64-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_2:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-64-GISEL-NEXT: s_mov_b32 s8, s5
+; GFX10-64-GISEL-NEXT: s_ashr_i32 s9, s5, 31
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v13, s4
+; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX10-64-GISEL-NEXT: s_lshl_b64 s[0:1], s[8:9], 4
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX10-64-GISEL-NEXT: s_add_u32 s0, s2, s0
+; GFX10-64-GISEL-NEXT: s_addc_u32 s1, s3, s1
+; GFX10-64-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc, v3, v1, vcc
+; GFX10-64-GISEL-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x0
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-GISEL-NEXT: s_clause 0x1
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX10-64-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s10, v4
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s11, v13
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s10, v4
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s11, v13
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s12, v5
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s13, v6
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s14, v7
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s15, v8
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s16, v9
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s17, v10
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s18, v11
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s19, v12
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[12:19], s[0:3] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[8:9], s[8:9]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr13
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB14_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[4:5]
+; GFX10-64-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX1150-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_2:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-SDAG-NEXT: s_mov_b32 s4, s5
+; GFX1150-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX1150-SDAG-NEXT: s_ashr_i32 s5, s5, 31
+; GFX1150-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; GFX1150-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s1, v1, vcc
+; GFX1150-SDAG-NEXT: s_clause 0x1
+; GFX1150-SDAG-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
+; GFX1150-SDAG-NEXT: global_load_b128 v[5:8], v[0:1], off
+; GFX1150-SDAG-NEXT: s_lshl_b64 s[0:1], s[4:5], 4
+; GFX1150-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-SDAG-NEXT: s_add_u32 s0, s2, s0
+; GFX1150-SDAG-NEXT: s_addc_u32 s1, s3, s1
+; GFX1150-SDAG-NEXT: s_load_b128 s[0:3], s[0:1], 0x0
+; GFX1150-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s10, v4
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v4
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s13, v6
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s14, v7
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s15, v8
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s16, v9
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s17, v10
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s18, v11
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s19, v12
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1150-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[12:19], s[0:3] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB14_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[4:5]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX1150-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_2:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX1150-GISEL-NEXT: s_mov_b32 s8, s5
+; GFX1150-GISEL-NEXT: s_ashr_i32 s9, s5, 31
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v13, s4
+; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX1150-GISEL-NEXT: s_lshl_b64 s[0:1], s[8:9], 4
+; GFX1150-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX1150-GISEL-NEXT: s_add_u32 s0, s2, s0
+; GFX1150-GISEL-NEXT: s_addc_u32 s1, s3, s1
+; GFX1150-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX1150-GISEL-NEXT: s_load_b128 s[0:3], s[0:1], 0x0
+; GFX1150-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX1150-GISEL-NEXT: s_clause 0x1
+; GFX1150-GISEL-NEXT: global_load_b128 v[5:8], v[0:1], off
+; GFX1150-GISEL-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
+; GFX1150-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s10, v4
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s11, v13
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s10, v4
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s11, v13
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s12, v5
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s13, v6
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s14, v7
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s15, v8
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s16, v9
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s17, v10
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s18, v11
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s19, v12
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1150-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[12:19], s[0:3] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr13
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB14_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[4:5]
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX12-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_2:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-SDAG-NEXT: s_mov_b32 s4, s5
+; GFX12-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-SDAG-NEXT: s_ashr_i32 s5, s5, 31
+; GFX12-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[0:1], 5, v[1:2]
+; GFX12-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s1, v1, vcc
+; GFX12-SDAG-NEXT: s_lshl_b64 s[0:1], s[4:5], 4
+; GFX12-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_add_nc_u64 s[0:1], s[2:3], s[0:1]
+; GFX12-SDAG-NEXT: s_clause 0x1
+; GFX12-SDAG-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
+; GFX12-SDAG-NEXT: global_load_b128 v[5:8], v[0:1], off
+; GFX12-SDAG-NEXT: s_load_b128 s[0:3], s[0:1], 0x0
+; GFX12-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s10, v4
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v4
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x0
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s12, v5
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s13, v6
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s14, v7
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s15, v8
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s16, v9
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s17, v10
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s18, v11
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s19, v12
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[12:19], s[0:3] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB14_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[4:5]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-SDAG-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX12-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_2:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-GISEL-NEXT: s_mov_b32 s8, s5
+; GFX12-GISEL-NEXT: s_ashr_i32 s9, s5, 31
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v13, s4
+; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[0:1], 5, v[1:2]
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX12-GISEL-NEXT: s_lshl_b64 s[0:1], s[8:9], 4
+; GFX12-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_add_co_u32 s0, s2, s0
+; GFX12-GISEL-NEXT: s_add_co_ci_u32 s1, s3, s1
+; GFX12-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX12-GISEL-NEXT: s_load_b128 s[0:3], s[0:1], 0x0
+; GFX12-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-GISEL-NEXT: s_clause 0x1
+; GFX12-GISEL-NEXT: global_load_b128 v[5:8], v[0:1], off
+; GFX12-GISEL-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
+; GFX12-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s10, v4
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s11, v13
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s10, v4
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s11, v13
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s12, v5
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s13, v6
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s14, v7
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s15, v8
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s16, v9
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s17, v10
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s18, v11
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s19, v12
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[12:19], s[0:3] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr13
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB14_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[4:5]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: ; return to shader part epilog
+ <4 x i32> addrspace(4)* inreg %s_in, i32 %idx1, i32 inreg %idx2, i32 %s_idx, i32 inreg %s_idx2) #1 {
+ %rptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
+ %sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
+ %srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
+ %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
+ %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok, i32 %idx2)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok1, <8 x i32> %rsrc)
+ %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok1, <4 x i32> %srsrc)
+ %r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
+
+ ret <4 x float> %r1
+}
+
+define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32> addrspace(4)* inreg %in,
+; VI-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_3:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; VI-SDAG-NEXT: s_wqm_b64 exec, exec
+; VI-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; VI-SDAG-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; VI-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[5:6]
+; VI-SDAG-NEXT: v_mov_b32_e32 v3, s1
+; VI-SDAG-NEXT: v_add_u32_e32 v0, vcc, s0, v0
+; VI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, v3, v1, vcc
+; VI-SDAG-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; VI-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[2:3]
+; VI-SDAG-NEXT: v_mov_b32_e32 v5, s3
+; VI-SDAG-NEXT: v_add_u32_e32 v2, vcc, s2, v2
+; VI-SDAG-NEXT: v_addc_u32_e32 v3, vcc, v5, v3, vcc
+; VI-SDAG-NEXT: flat_load_dwordx4 v[5:8], v[0:1]
+; VI-SDAG-NEXT: v_add_u32_e32 v0, vcc, 16, v0
+; VI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; VI-SDAG-NEXT: flat_load_dwordx4 v[9:12], v[0:1]
+; VI-SDAG-NEXT: flat_load_dwordx4 v[13:16], v[2:3]
+; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; VI-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v4
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; VI-SDAG-NEXT: s_waitcnt vmcnt(2)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v5
+; VI-SDAG-NEXT: v_readfirstlane_b32 s9, v6
+; VI-SDAG-NEXT: v_readfirstlane_b32 s10, v7
+; VI-SDAG-NEXT: v_readfirstlane_b32 s11, v8
+; VI-SDAG-NEXT: s_waitcnt vmcnt(1)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s12, v9
+; VI-SDAG-NEXT: v_readfirstlane_b32 s13, v10
+; VI-SDAG-NEXT: v_readfirstlane_b32 s14, v11
+; VI-SDAG-NEXT: v_readfirstlane_b32 s15, v12
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s16, v13
+; VI-SDAG-NEXT: v_readfirstlane_b32 s17, v14
+; VI-SDAG-NEXT: v_readfirstlane_b32 s18, v15
+; VI-SDAG-NEXT: v_readfirstlane_b32 s19, v16
+; VI-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; VI-SDAG-NEXT: ; implicit-def: $vgpr4
+; VI-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; VI-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; VI-SDAG-NEXT: s_nop 3
+; VI-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB15_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; VI-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: ; return to shader part epilog
+;
+; VI-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_3:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; VI-GISEL-NEXT: s_wqm_b64 exec, exec
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, v2
+; VI-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; VI-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; VI-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; VI-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; VI-GISEL-NEXT: v_add_u32_e32 v0, vcc, v2, v0
+; VI-GISEL-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, v3, v1, vcc
+; VI-GISEL-NEXT: v_lshlrev_b64 v[2:3], 4, v[5:6]
+; VI-GISEL-NEXT: v_mov_b32_e32 v6, s3
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, s2
+; VI-GISEL-NEXT: v_add_u32_e32 v2, vcc, v5, v2
+; VI-GISEL-NEXT: v_addc_u32_e32 v3, vcc, v6, v3, vcc
+; VI-GISEL-NEXT: flat_load_dwordx4 v[5:8], v[0:1]
+; VI-GISEL-NEXT: v_add_u32_e32 v0, vcc, 16, v0
+; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; VI-GISEL-NEXT: flat_load_dwordx4 v[9:12], v[0:1]
+; VI-GISEL-NEXT: flat_load_dwordx4 v[13:16], v[2:3]
+; VI-GISEL-NEXT: s_mov_b32 s0, 0
+; VI-GISEL-NEXT: v_mov_b32_e32 v17, s4
+; VI-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; VI-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s1, v17
+; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v4
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s1, v17
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v4
+; VI-GISEL-NEXT: s_and_b64 s[4:5], s[4:5], s[8:9]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
+; VI-GISEL-NEXT: s_mov_b32 s1, s0
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; VI-GISEL-NEXT: s_waitcnt vmcnt(2)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; VI-GISEL-NEXT: v_readfirstlane_b32 s9, v6
+; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v7
+; VI-GISEL-NEXT: v_readfirstlane_b32 s11, v8
+; VI-GISEL-NEXT: s_waitcnt vmcnt(1)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v9
+; VI-GISEL-NEXT: v_readfirstlane_b32 s13, v10
+; VI-GISEL-NEXT: v_readfirstlane_b32 s14, v11
+; VI-GISEL-NEXT: v_readfirstlane_b32 s15, v12
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s16, v13
+; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v14
+; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v15
+; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v16
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; VI-GISEL-NEXT: ; implicit-def: $vgpr17
+; VI-GISEL-NEXT: ; implicit-def: $vgpr4
+; VI-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; VI-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; VI-GISEL-NEXT: s_nop 3
+; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[4:5]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB15_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; VI-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX9-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_3:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[5:6]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v3, s1
+; GFX9-SDAG-NEXT: v_add_co_u32_e32 v0, vcc, s0, v0
+; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v1, vcc, v3, v1, vcc
+; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX9-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[2:3]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v5, s3
+; GFX9-SDAG-NEXT: v_add_co_u32_e32 v2, vcc, s2, v2
+; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v3, vcc, v5, v3, vcc
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
+; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX9-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v4
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v5
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s9, v6
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s10, v7
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s11, v8
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s12, v9
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s13, v10
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s14, v11
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s15, v12
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s16, v13
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s17, v14
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s18, v15
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s19, v16
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX9-SDAG-NEXT: s_nop 3
+; GFX9-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB15_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX9-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_3:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX9-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v5, v2
+; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX9-GISEL-NEXT: v_add_co_u32_e32 v0, vcc, v2, v0
+; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v1, vcc, v3, v1, vcc
+; GFX9-GISEL-NEXT: v_lshlrev_b64 v[2:3], 4, v[5:6]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v6, s3
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v5, s2
+; GFX9-GISEL-NEXT: v_add_co_u32_e32 v2, vcc, v5, v2
+; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v3, vcc, v6, v3, vcc
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
+; GFX9-GISEL-NEXT: s_mov_b32 s0, 0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v17, s4
+; GFX9-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX9-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s1, v17
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v4
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s1, v17
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v4
+; GFX9-GISEL-NEXT: s_and_b64 s[4:5], s[4:5], s[8:9]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
+; GFX9-GISEL-NEXT: s_mov_b32 s1, s0
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(2)
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s9, v6
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v7
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s11, v8
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v9
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s13, v10
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s14, v11
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s15, v12
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s16, v13
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v14
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v15
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v16
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr17
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX9-GISEL-NEXT: s_nop 3
+; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[4:5]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB15_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-32-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_3:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-SDAG-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[2:3]
+; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[5:6]
+; GFX10-32-SDAG-NEXT: v_add_co_u32 v0, vcc_lo, s0, v0
+; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, s1, v1, vcc_lo
+; GFX10-32-SDAG-NEXT: v_add_co_u32 v2, vcc_lo, s2, v2
+; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v3, vcc_lo, s3, v3, vcc_lo
+; GFX10-32-SDAG-NEXT: s_clause 0x1
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
+; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s2, v4
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s2, v4
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s8, v5
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s9, v6
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s10, v7
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s11, v8
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s12, v9
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s13, v10
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s14, v11
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s15, v12
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s16, v13
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s17, v14
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s18, v15
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s19, v16
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB15_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-SDAG-NEXT: s_and_b32 exec_lo, exec_lo, s4
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-32-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_3:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: s_mov_b32 s5, exec_lo
+; GFX10-32-GISEL-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v5, v2
+; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v8, s3
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v7, s2
+; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v17, s4
+; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[5:6], 4, v[5:6]
+; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-GISEL-NEXT: v_add_co_u32 v0, vcc_lo, v2, v0
+; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
+; GFX10-32-GISEL-NEXT: v_add_co_u32 v2, vcc_lo, v7, v5
+; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v3, vcc_lo, v8, v6, vcc_lo
+; GFX10-32-GISEL-NEXT: s_clause 0x1
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
+; GFX10-32-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v17
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s3, v4
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s2, v17
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s3, v4
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(2)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s9, v6
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s10, v7
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s11, v8
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s12, v9
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s13, v10
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s14, v11
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s15, v12
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s16, v13
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s17, v14
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s18, v15
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s19, v16
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr17
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB15_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-GISEL-NEXT: s_and_b32 exec_lo, exec_lo, s5
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-64-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_3:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[2:3]
+; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[5:6]
+; GFX10-64-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc, s1, v1, vcc
+; GFX10-64-SDAG-NEXT: v_add_co_u32 v2, vcc, s2, v2
+; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v3, vcc, s3, v3, vcc
+; GFX10-64-SDAG-NEXT: s_clause 0x1
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s6, v4
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v4
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s8, v5
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s9, v6
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s10, v7
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s11, v8
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s12, v9
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s13, v10
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s14, v11
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s15, v12
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s16, v13
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s17, v14
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s18, v15
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s19, v16
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB15_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-64-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_3:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v5, v2
+; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v8, s3
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v7, s2
+; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v17, s4
+; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[5:6], 4, v[5:6]
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc, v3, v1, vcc
+; GFX10-64-GISEL-NEXT: v_add_co_u32 v2, vcc, v7, v5
+; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v3, vcc, v8, v6, vcc
+; GFX10-64-GISEL-NEXT: s_clause 0x1
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
+; GFX10-64-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s4, v17
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s5, v4
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v17
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s5, v4
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(2)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s9, v6
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s10, v7
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s11, v8
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s12, v9
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s13, v10
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s14, v11
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s15, v12
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s16, v13
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s17, v14
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s18, v15
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s19, v16
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr17
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB15_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX1150-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_3:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_3) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[2:3]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[5:6]
+; GFX1150-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_4)
+; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s1, v1, vcc
+; GFX1150-SDAG-NEXT: v_add_co_u32 v2, vcc, s2, v2
+; GFX1150-SDAG-NEXT: s_waitcnt_depctr depctr_va_vcc(0)
+; GFX1150-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v3, null, s3, v3, vcc
+; GFX1150-SDAG-NEXT: s_clause 0x1
+; GFX1150-SDAG-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
+; GFX1150-SDAG-NEXT: global_load_b128 v[5:8], v[0:1], off
+; GFX1150-SDAG-NEXT: global_load_b128 v[13:16], v[2:3], off
+; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s6, v4
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v4
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s8, v5
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s9, v6
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s10, v7
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s11, v8
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s12, v9
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s13, v10
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s14, v11
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s15, v12
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s16, v13
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s17, v14
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s18, v15
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s19, v16
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB15_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX1150-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_3:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v5, v2
+; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v8, s3
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v7, s2
+; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v17, s4
+; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[5:6], 4, v[5:6]
+; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_3)
+; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX1150-GISEL-NEXT: v_add_co_u32 v2, vcc, v7, v5
+; GFX1150-GISEL-NEXT: s_waitcnt_depctr depctr_va_vcc(0)
+; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v3, null, v8, v6, vcc
+; GFX1150-GISEL-NEXT: s_clause 0x1
+; GFX1150-GISEL-NEXT: global_load_b128 v[5:8], v[0:1], off
+; GFX1150-GISEL-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
+; GFX1150-GISEL-NEXT: global_load_b128 v[13:16], v[2:3], off
+; GFX1150-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s4, v17
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s5, v4
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v17
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s5, v4
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(2)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s9, v6
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s10, v7
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s11, v8
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s12, v9
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s13, v10
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s14, v11
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s15, v12
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s16, v13
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s17, v14
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s18, v15
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s19, v16
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr17
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB15_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX12-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_3:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_3) | instskip(NEXT) | instid1(VALU_DEP_3)
+; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[2:3], 4, v[2:3]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[0:1], 5, v[5:6]
+; GFX12-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_4)
+; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s1, v1, vcc
+; GFX12-SDAG-NEXT: v_add_co_u32 v2, vcc, s2, v2
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_vcc(0)
+; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v3, null, s3, v3, vcc
+; GFX12-SDAG-NEXT: s_clause 0x1
+; GFX12-SDAG-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
+; GFX12-SDAG-NEXT: global_load_b128 v[5:8], v[0:1], off
+; GFX12-SDAG-NEXT: global_load_b128 v[13:16], v[2:3], off
+; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s6, v4
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v4
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s8, v5
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s9, v6
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s10, v7
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s11, v8
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s12, v9
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s13, v10
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s14, v11
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s15, v12
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x0
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s16, v13
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s17, v14
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s18, v15
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s19, v16
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB15_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX12-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_3:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v5, v2
+; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v8, s3
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v7, s2
+; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[0:1], 5, v[1:2]
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v17, s4
+; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[5:6], 4, v[5:6]
+; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_3)
+; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX12-GISEL-NEXT: v_add_co_u32 v2, vcc, v7, v5
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_vcc(0)
+; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v3, null, v8, v6, vcc
+; GFX12-GISEL-NEXT: s_clause 0x1
+; GFX12-GISEL-NEXT: global_load_b128 v[5:8], v[0:1], off
+; GFX12-GISEL-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
+; GFX12-GISEL-NEXT: global_load_b128 v[13:16], v[2:3], off
+; GFX12-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s4, v17
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s5, v4
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v17
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s5, v4
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x2
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s9, v6
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s10, v7
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s11, v8
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s12, v9
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s13, v10
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s14, v11
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s15, v12
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s16, v13
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s17, v14
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s18, v15
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s19, v16
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr17
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB15_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: ; return to shader part epilog
+ <4 x i32> addrspace(4)* inreg %s_in, i32 inreg %idx1, i32 %idx2, i32 %s_idx, i32 %s_idx2) #1 {
+ %rptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
+ %sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
+ %srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
+ %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
+ %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok, i32 %idx2)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok1, <8 x i32> %rsrc)
+ %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok1, <4 x i32> %srsrc)
+ %r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
+
+ ret <4 x float> %r1
+}
+
+define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32> addrspace(4)* inreg %in,
+; VI-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_4:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; VI-SDAG-NEXT: s_wqm_b64 exec, exec
+; VI-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; VI-SDAG-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; VI-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[5:6]
+; VI-SDAG-NEXT: v_mov_b32_e32 v3, s1
+; VI-SDAG-NEXT: v_add_u32_e32 v0, vcc, s0, v0
+; VI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, v3, v1, vcc
+; VI-SDAG-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; VI-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[2:3]
+; VI-SDAG-NEXT: v_mov_b32_e32 v5, s3
+; VI-SDAG-NEXT: v_add_u32_e32 v2, vcc, s2, v2
+; VI-SDAG-NEXT: v_addc_u32_e32 v3, vcc, v5, v3, vcc
+; VI-SDAG-NEXT: flat_load_dwordx4 v[5:8], v[0:1]
+; VI-SDAG-NEXT: v_add_u32_e32 v0, vcc, 16, v0
+; VI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; VI-SDAG-NEXT: flat_load_dwordx4 v[9:12], v[0:1]
+; VI-SDAG-NEXT: flat_load_dwordx4 v[13:16], v[2:3]
+; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; VI-SDAG-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v4
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; VI-SDAG-NEXT: s_waitcnt vmcnt(2)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v5
+; VI-SDAG-NEXT: v_readfirstlane_b32 s9, v6
+; VI-SDAG-NEXT: v_readfirstlane_b32 s10, v7
+; VI-SDAG-NEXT: v_readfirstlane_b32 s11, v8
+; VI-SDAG-NEXT: s_waitcnt vmcnt(1)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s12, v9
+; VI-SDAG-NEXT: v_readfirstlane_b32 s13, v10
+; VI-SDAG-NEXT: v_readfirstlane_b32 s14, v11
+; VI-SDAG-NEXT: v_readfirstlane_b32 s15, v12
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s16, v13
+; VI-SDAG-NEXT: v_readfirstlane_b32 s17, v14
+; VI-SDAG-NEXT: v_readfirstlane_b32 s18, v15
+; VI-SDAG-NEXT: v_readfirstlane_b32 s19, v16
+; VI-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; VI-SDAG-NEXT: ; implicit-def: $vgpr4
+; VI-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; VI-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; VI-SDAG-NEXT: s_nop 3
+; VI-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB16_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; VI-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: ; return to shader part epilog
+;
+; VI-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_4:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; VI-GISEL-NEXT: s_wqm_b64 exec, exec
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, v2
+; VI-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; VI-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; VI-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; VI-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; VI-GISEL-NEXT: v_add_u32_e32 v0, vcc, v2, v0
+; VI-GISEL-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, v3, v1, vcc
+; VI-GISEL-NEXT: v_lshlrev_b64 v[2:3], 4, v[5:6]
+; VI-GISEL-NEXT: v_mov_b32_e32 v6, s3
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, s2
+; VI-GISEL-NEXT: v_add_u32_e32 v2, vcc, v5, v2
+; VI-GISEL-NEXT: v_addc_u32_e32 v3, vcc, v6, v3, vcc
+; VI-GISEL-NEXT: flat_load_dwordx4 v[5:8], v[0:1]
+; VI-GISEL-NEXT: v_add_u32_e32 v0, vcc, 16, v0
+; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; VI-GISEL-NEXT: flat_load_dwordx4 v[9:12], v[0:1]
+; VI-GISEL-NEXT: flat_load_dwordx4 v[13:16], v[2:3]
+; VI-GISEL-NEXT: s_mov_b32 s0, 0
+; VI-GISEL-NEXT: v_mov_b32_e32 v17, s4
+; VI-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; VI-GISEL-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s1, v4
+; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v17
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s1, v4
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v17
+; VI-GISEL-NEXT: s_and_b64 s[4:5], s[4:5], s[8:9]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
+; VI-GISEL-NEXT: s_mov_b32 s1, s0
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; VI-GISEL-NEXT: s_waitcnt vmcnt(2)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; VI-GISEL-NEXT: v_readfirstlane_b32 s9, v6
+; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v7
+; VI-GISEL-NEXT: v_readfirstlane_b32 s11, v8
+; VI-GISEL-NEXT: s_waitcnt vmcnt(1)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v9
+; VI-GISEL-NEXT: v_readfirstlane_b32 s13, v10
+; VI-GISEL-NEXT: v_readfirstlane_b32 s14, v11
+; VI-GISEL-NEXT: v_readfirstlane_b32 s15, v12
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s16, v13
+; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v14
+; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v15
+; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v16
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; VI-GISEL-NEXT: ; implicit-def: $vgpr4
+; VI-GISEL-NEXT: ; implicit-def: $vgpr17
+; VI-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; VI-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; VI-GISEL-NEXT: s_nop 3
+; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[4:5]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB16_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; VI-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX9-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_4:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[5:6]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v3, s1
+; GFX9-SDAG-NEXT: v_add_co_u32_e32 v0, vcc, s0, v0
+; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v1, vcc, v3, v1, vcc
+; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX9-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[2:3]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v5, s3
+; GFX9-SDAG-NEXT: v_add_co_u32_e32 v2, vcc, s2, v2
+; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v3, vcc, v5, v3, vcc
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
+; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX9-SDAG-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v4
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v5
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s9, v6
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s10, v7
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s11, v8
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s12, v9
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s13, v10
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s14, v11
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s15, v12
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s16, v13
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s17, v14
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s18, v15
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s19, v16
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX9-SDAG-NEXT: s_nop 3
+; GFX9-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB16_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX9-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_4:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX9-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v5, v2
+; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX9-GISEL-NEXT: v_add_co_u32_e32 v0, vcc, v2, v0
+; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v1, vcc, v3, v1, vcc
+; GFX9-GISEL-NEXT: v_lshlrev_b64 v[2:3], 4, v[5:6]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v6, s3
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v5, s2
+; GFX9-GISEL-NEXT: v_add_co_u32_e32 v2, vcc, v5, v2
+; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v3, vcc, v6, v3, vcc
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
+; GFX9-GISEL-NEXT: s_mov_b32 s0, 0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v17, s4
+; GFX9-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX9-GISEL-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s1, v4
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v17
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s1, v4
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v17
+; GFX9-GISEL-NEXT: s_and_b64 s[4:5], s[4:5], s[8:9]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
+; GFX9-GISEL-NEXT: s_mov_b32 s1, s0
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(2)
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s9, v6
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v7
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s11, v8
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v9
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s13, v10
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s14, v11
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s15, v12
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s16, v13
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v14
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v15
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v16
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr17
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX9-GISEL-NEXT: s_nop 3
+; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[4:5]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB16_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-32-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_4:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-SDAG-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[2:3]
+; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[5:6]
+; GFX10-32-SDAG-NEXT: v_add_co_u32 v0, vcc_lo, s0, v0
+; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, s1, v1, vcc_lo
+; GFX10-32-SDAG-NEXT: v_add_co_u32 v2, vcc_lo, s2, v2
+; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v3, vcc_lo, s3, v3, vcc_lo
+; GFX10-32-SDAG-NEXT: s_clause 0x1
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
+; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-SDAG-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s2, v4
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s2, v4
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s8, v5
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s9, v6
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s10, v7
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s11, v8
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s12, v9
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s13, v10
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s14, v11
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s15, v12
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s16, v13
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s17, v14
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s18, v15
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s19, v16
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB16_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-SDAG-NEXT: s_and_b32 exec_lo, exec_lo, s4
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-32-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_4:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: s_mov_b32 s5, exec_lo
+; GFX10-32-GISEL-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v5, v2
+; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v8, s3
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v7, s2
+; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v17, s4
+; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[5:6], 4, v[5:6]
+; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-GISEL-NEXT: v_add_co_u32 v0, vcc_lo, v2, v0
+; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
+; GFX10-32-GISEL-NEXT: v_add_co_u32 v2, vcc_lo, v7, v5
+; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v3, vcc_lo, v8, v6, vcc_lo
+; GFX10-32-GISEL-NEXT: s_clause 0x1
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
+; GFX10-32-GISEL-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v4
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s3, v17
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s2, v4
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s3, v17
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(2)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s9, v6
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s10, v7
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s11, v8
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s12, v9
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s13, v10
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s14, v11
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s15, v12
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s16, v13
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s17, v14
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s18, v15
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s19, v16
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr17
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB16_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-GISEL-NEXT: s_and_b32 exec_lo, exec_lo, s5
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-64-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_4:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[2:3]
+; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[5:6]
+; GFX10-64-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc, s1, v1, vcc
+; GFX10-64-SDAG-NEXT: v_add_co_u32 v2, vcc, s2, v2
+; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v3, vcc, s3, v3, vcc
+; GFX10-64-SDAG-NEXT: s_clause 0x1
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-SDAG-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s6, v4
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v4
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s8, v5
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s9, v6
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s10, v7
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s11, v8
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s12, v9
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s13, v10
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s14, v11
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s15, v12
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s16, v13
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s17, v14
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s18, v15
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s19, v16
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB16_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-64-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_4:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v5, v2
+; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v8, s3
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v7, s2
+; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v17, s4
+; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[5:6], 4, v[5:6]
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc, v3, v1, vcc
+; GFX10-64-GISEL-NEXT: v_add_co_u32 v2, vcc, v7, v5
+; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v3, vcc, v8, v6, vcc
+; GFX10-64-GISEL-NEXT: s_clause 0x1
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
+; GFX10-64-GISEL-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s4, v4
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s5, v17
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v4
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s5, v17
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(2)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s9, v6
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s10, v7
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s11, v8
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s12, v9
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s13, v10
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s14, v11
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s15, v12
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s16, v13
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s17, v14
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s18, v15
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s19, v16
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr17
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB16_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX1150-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_4:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_3) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[2:3]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[5:6]
+; GFX1150-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_4)
+; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s1, v1, vcc
+; GFX1150-SDAG-NEXT: v_add_co_u32 v2, vcc, s2, v2
+; GFX1150-SDAG-NEXT: s_waitcnt_depctr depctr_va_vcc(0)
+; GFX1150-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v3, null, s3, v3, vcc
+; GFX1150-SDAG-NEXT: s_clause 0x1
+; GFX1150-SDAG-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
+; GFX1150-SDAG-NEXT: global_load_b128 v[5:8], v[0:1], off
+; GFX1150-SDAG-NEXT: global_load_b128 v[13:16], v[2:3], off
+; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-SDAG-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s6, v4
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v4
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s8, v5
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s9, v6
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s10, v7
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s11, v8
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s12, v9
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s13, v10
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s14, v11
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s15, v12
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s16, v13
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s17, v14
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s18, v15
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s19, v16
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB16_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX1150-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_4:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v5, v2
+; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v8, s3
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v7, s2
+; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[1:2]
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v17, s4
+; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[5:6], 4, v[5:6]
+; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_3)
+; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX1150-GISEL-NEXT: v_add_co_u32 v2, vcc, v7, v5
+; GFX1150-GISEL-NEXT: s_waitcnt_depctr depctr_va_vcc(0)
+; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v3, null, v8, v6, vcc
+; GFX1150-GISEL-NEXT: s_clause 0x1
+; GFX1150-GISEL-NEXT: global_load_b128 v[5:8], v[0:1], off
+; GFX1150-GISEL-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
+; GFX1150-GISEL-NEXT: global_load_b128 v[13:16], v[2:3], off
+; GFX1150-GISEL-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s4, v4
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s5, v17
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v4
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s5, v17
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(2)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s9, v6
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s10, v7
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s11, v8
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s12, v9
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s13, v10
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s14, v11
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s15, v12
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s16, v13
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s17, v14
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s18, v15
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s19, v16
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr17
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB16_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX12-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_4:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v5, v1
+; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_3) | instskip(NEXT) | instid1(VALU_DEP_3)
+; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[2:3], 4, v[2:3]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[0:1], 5, v[5:6]
+; GFX12-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_4)
+; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s1, v1, vcc
+; GFX12-SDAG-NEXT: v_add_co_u32 v2, vcc, s2, v2
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_vcc(0)
+; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v3, null, s3, v3, vcc
+; GFX12-SDAG-NEXT: s_clause 0x1
+; GFX12-SDAG-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
+; GFX12-SDAG-NEXT: global_load_b128 v[5:8], v[0:1], off
+; GFX12-SDAG-NEXT: global_load_b128 v[13:16], v[2:3], off
+; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-SDAG-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s6, v4
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v4
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s8, v5
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s9, v6
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s10, v7
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s11, v8
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s12, v9
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s13, v10
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s14, v11
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s15, v12
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x0
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s16, v13
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s17, v14
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s18, v15
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s19, v16
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr4
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB16_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX12-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_4:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v5, v2
+; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v2, 31, v1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v8, s3
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v7, s2
+; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[0:1], 5, v[1:2]
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v17, s4
+; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[5:6], 4, v[5:6]
+; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_3)
+; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX12-GISEL-NEXT: v_add_co_u32 v2, vcc, v7, v5
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_vcc(0)
+; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v3, null, v8, v6, vcc
+; GFX12-GISEL-NEXT: s_clause 0x1
+; GFX12-GISEL-NEXT: global_load_b128 v[5:8], v[0:1], off
+; GFX12-GISEL-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
+; GFX12-GISEL-NEXT: global_load_b128 v[13:16], v[2:3], off
+; GFX12-GISEL-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s4, v4
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s5, v17
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v4
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s5, v17
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x2
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s9, v6
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s10, v7
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s11, v8
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s12, v9
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s13, v10
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s14, v11
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s15, v12
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s16, v13
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s17, v14
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s18, v15
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s19, v16
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr17
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB16_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: ; return to shader part epilog
+ <4 x i32> addrspace(4)* inreg %s_in, i32 %idx1, i32 inreg %idx2, i32 %s_idx, i32 %s_idx2) #1 {
+ %rptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
+ %sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
+ %srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
+ %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
+ %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok, i32 %idx2)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok1, <8 x i32> %rsrc)
+ %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok1, <4 x i32> %srsrc)
+ %r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
+
+ ret <4 x float> %r1
+}
+
+define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_idx_1_2loops(
+; VI-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_1_2loops:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; VI-SDAG-NEXT: s_wqm_b64 exec, exec
+; VI-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; VI-SDAG-NEXT: v_ashrrev_i32_e32 v8, 31, v7
+; VI-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; VI-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; VI-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[7:8]
+; VI-SDAG-NEXT: v_mov_b32_e32 v2, s1
+; VI-SDAG-NEXT: v_add_u32_e32 v0, vcc, s0, v0
+; VI-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; VI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, v2, v1, vcc
+; VI-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[3:4]
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, s3
+; VI-SDAG-NEXT: v_add_u32_e32 v2, vcc, s2, v2
+; VI-SDAG-NEXT: v_addc_u32_e32 v3, vcc, v4, v3, vcc
+; VI-SDAG-NEXT: flat_load_dwordx4 v[11:14], v[0:1]
+; VI-SDAG-NEXT: v_add_u32_e32 v0, vcc, 16, v0
+; VI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; VI-SDAG-NEXT: flat_load_dwordx4 v[15:18], v[0:1]
+; VI-SDAG-NEXT: flat_load_dwordx4 v[7:10], v[2:3]
+; VI-SDAG-NEXT: s_mov_b32 s0, 0
+; VI-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s1, v5
+; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v6
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[6:7], s1, v5
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v6
+; VI-SDAG-NEXT: s_and_b64 s[6:7], s[6:7], s[8:9]
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[6:7], s[6:7]
+; VI-SDAG-NEXT: s_mov_b32 s1, s0
+; VI-SDAG-NEXT: v_mov_b32_e32 v20, s1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v11
+; VI-SDAG-NEXT: v_readfirstlane_b32 s9, v12
+; VI-SDAG-NEXT: v_readfirstlane_b32 s10, v13
+; VI-SDAG-NEXT: v_readfirstlane_b32 s11, v14
+; VI-SDAG-NEXT: v_readfirstlane_b32 s12, v15
+; VI-SDAG-NEXT: v_readfirstlane_b32 s13, v16
+; VI-SDAG-NEXT: v_readfirstlane_b32 s14, v17
+; VI-SDAG-NEXT: v_readfirstlane_b32 s15, v18
+; VI-SDAG-NEXT: v_readfirstlane_b32 s16, v7
+; VI-SDAG-NEXT: v_readfirstlane_b32 s17, v8
+; VI-SDAG-NEXT: v_readfirstlane_b32 s18, v9
+; VI-SDAG-NEXT: v_readfirstlane_b32 s19, v10
+; VI-SDAG-NEXT: v_mov_b32_e32 v19, s0
+; VI-SDAG-NEXT: ; implicit-def: $vgpr5
+; VI-SDAG-NEXT: ; implicit-def: $vgpr6
+; VI-SDAG-NEXT: s_nop 3
+; VI-SDAG-NEXT: image_sample v[0:3], v[19:20], s[8:15], s[16:19] dmask:0xf
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[6:7]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB17_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[2:3]
+; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v11
+; VI-SDAG-NEXT: v_readfirstlane_b32 s9, v12
+; VI-SDAG-NEXT: v_readfirstlane_b32 s10, v13
+; VI-SDAG-NEXT: v_readfirstlane_b32 s11, v14
+; VI-SDAG-NEXT: v_readfirstlane_b32 s12, v15
+; VI-SDAG-NEXT: v_readfirstlane_b32 s13, v16
+; VI-SDAG-NEXT: v_readfirstlane_b32 s14, v17
+; VI-SDAG-NEXT: v_readfirstlane_b32 s15, v18
+; VI-SDAG-NEXT: v_readfirstlane_b32 s0, v7
+; VI-SDAG-NEXT: v_readfirstlane_b32 s1, v8
+; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v9
+; VI-SDAG-NEXT: v_readfirstlane_b32 s3, v10
+; VI-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; VI-SDAG-NEXT: s_nop 3
+; VI-SDAG-NEXT: image_sample v[4:7], v[19:20], s[8:15], s[0:3] dmask:0xf
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: ; return to shader part epilog
+;
+; VI-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_1_2loops:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; VI-GISEL-NEXT: s_wqm_b64 exec, exec
+; VI-GISEL-NEXT: v_mov_b32_e32 v6, v3
+; VI-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; VI-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[2:3]
+; VI-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; VI-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; VI-GISEL-NEXT: v_add_u32_e32 v0, vcc, v2, v0
+; VI-GISEL-NEXT: v_ashrrev_i32_e32 v7, 31, v6
+; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, v3, v1, vcc
+; VI-GISEL-NEXT: v_lshlrev_b64 v[2:3], 4, v[6:7]
+; VI-GISEL-NEXT: v_mov_b32_e32 v7, s3
+; VI-GISEL-NEXT: v_mov_b32_e32 v6, s2
+; VI-GISEL-NEXT: v_add_u32_e32 v2, vcc, v6, v2
+; VI-GISEL-NEXT: v_addc_u32_e32 v3, vcc, v7, v3, vcc
+; VI-GISEL-NEXT: flat_load_dwordx4 v[8:11], v[0:1]
+; VI-GISEL-NEXT: v_add_u32_e32 v0, vcc, 16, v0
+; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; VI-GISEL-NEXT: flat_load_dwordx4 v[12:15], v[0:1]
+; VI-GISEL-NEXT: flat_load_dwordx4 v[16:19], v[2:3]
+; VI-GISEL-NEXT: s_mov_b32 s0, 0
+; VI-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s1, v4
+; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v5
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s1, v4
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s10, v5
+; VI-GISEL-NEXT: s_and_b64 s[8:9], s[8:9], s[10:11]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[20:21], s[8:9]
+; VI-GISEL-NEXT: s_mov_b32 s1, s0
+; VI-GISEL-NEXT: v_mov_b32_e32 v21, s1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v8
+; VI-GISEL-NEXT: v_readfirstlane_b32 s9, v9
+; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v10
+; VI-GISEL-NEXT: v_readfirstlane_b32 s11, v11
+; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v12
+; VI-GISEL-NEXT: v_readfirstlane_b32 s13, v13
+; VI-GISEL-NEXT: v_readfirstlane_b32 s14, v14
+; VI-GISEL-NEXT: v_readfirstlane_b32 s15, v15
+; VI-GISEL-NEXT: v_readfirstlane_b32 s16, v16
+; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v17
+; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v18
+; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v19
+; VI-GISEL-NEXT: v_mov_b32_e32 v20, s0
+; VI-GISEL-NEXT: ; implicit-def: $vgpr4
+; VI-GISEL-NEXT: ; implicit-def: $vgpr5
+; VI-GISEL-NEXT: s_nop 3
+; VI-GISEL-NEXT: image_sample v[0:3], v[20:21], s[8:15], s[16:19] dmask:0xf
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[20:21]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB17_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; VI-GISEL-NEXT: v_mov_b32_e32 v22, s4
+; VI-GISEL-NEXT: v_mov_b32_e32 v23, s5
+; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; VI-GISEL-NEXT: .LBB17_3: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v22
+; VI-GISEL-NEXT: v_readfirstlane_b32 s4, v23
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v22
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s4, v23
+; VI-GISEL-NEXT: s_and_b64 s[2:3], s[2:3], s[4:5]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v8
+; VI-GISEL-NEXT: v_readfirstlane_b32 s9, v9
+; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v10
+; VI-GISEL-NEXT: v_readfirstlane_b32 s11, v11
+; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v12
+; VI-GISEL-NEXT: v_readfirstlane_b32 s13, v13
+; VI-GISEL-NEXT: v_readfirstlane_b32 s14, v14
+; VI-GISEL-NEXT: v_readfirstlane_b32 s15, v15
+; VI-GISEL-NEXT: v_readfirstlane_b32 s16, v16
+; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v17
+; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v18
+; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v19
+; VI-GISEL-NEXT: ; implicit-def: $vgpr22
+; VI-GISEL-NEXT: ; implicit-def: $vgpr23
+; VI-GISEL-NEXT: ; implicit-def: $vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15
+; VI-GISEL-NEXT: ; implicit-def: $vgpr16_vgpr17_vgpr18_vgpr19
+; VI-GISEL-NEXT: s_nop 4
+; VI-GISEL-NEXT: image_sample v[4:7], v[20:21], s[8:15], s[16:19] dmask:0xf
+; VI-GISEL-NEXT: ; implicit-def: $vgpr20_vgpr21
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB17_3
+; VI-GISEL-NEXT: ; %bb.4:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; VI-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX9-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_1_2loops:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v8, 31, v7
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX9-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[7:8]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v2, s1
+; GFX9-SDAG-NEXT: v_add_co_u32_e32 v0, vcc, s0, v0
+; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v1, vcc, v2, v1, vcc
+; GFX9-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[3:4]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v4, s3
+; GFX9-SDAG-NEXT: v_add_co_u32_e32 v2, vcc, s2, v2
+; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v3, vcc, v4, v3, vcc
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[11:14], v[0:1], off offset:16
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[15:18], v[0:1], off
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[7:10], v[2:3], off
+; GFX9-SDAG-NEXT: s_mov_b32 s0, 0
+; GFX9-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s1, v5
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v6
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[6:7], s1, v5
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v6
+; GFX9-SDAG-NEXT: s_and_b64 s[6:7], s[6:7], s[8:9]
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[6:7], s[6:7]
+; GFX9-SDAG-NEXT: s_mov_b32 s1, s0
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v20, s1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v15
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s9, v16
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s10, v17
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s11, v18
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s12, v11
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s13, v12
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s14, v13
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s15, v14
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s16, v7
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s17, v8
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s18, v9
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s19, v10
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v19, s0
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX9-SDAG-NEXT: s_nop 3
+; GFX9-SDAG-NEXT: image_sample v[0:3], v[19:20], s[8:15], s[16:19] dmask:0xf
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[6:7]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB17_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v15
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s9, v16
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s10, v17
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s11, v18
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s12, v11
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s13, v12
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s14, v13
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s15, v14
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s0, v7
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s1, v8
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v9
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s3, v10
+; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX9-SDAG-NEXT: s_nop 3
+; GFX9-SDAG-NEXT: image_sample v[4:7], v[19:20], s[8:15], s[0:3] dmask:0xf
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX9-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_1_2loops:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX9-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v6, v3
+; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; GFX9-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[2:3]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX9-GISEL-NEXT: v_add_co_u32_e32 v0, vcc, v2, v0
+; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v7, 31, v6
+; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v1, vcc, v3, v1, vcc
+; GFX9-GISEL-NEXT: v_lshlrev_b64 v[2:3], 4, v[6:7]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v7, s3
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v6, s2
+; GFX9-GISEL-NEXT: v_add_co_u32_e32 v2, vcc, v6, v2
+; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v3, vcc, v7, v3, vcc
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[8:11], v[0:1], off
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[12:15], v[0:1], off offset:16
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[16:19], v[2:3], off
+; GFX9-GISEL-NEXT: s_mov_b32 s0, 0
+; GFX9-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s1, v4
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v5
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s1, v4
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s10, v5
+; GFX9-GISEL-NEXT: s_and_b64 s[8:9], s[8:9], s[10:11]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[20:21], s[8:9]
+; GFX9-GISEL-NEXT: s_mov_b32 s1, s0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v21, s1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v8
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s9, v9
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v10
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s11, v11
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v12
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s13, v13
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s14, v14
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s15, v15
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s16, v16
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v17
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v18
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v19
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v20, s0
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX9-GISEL-NEXT: s_nop 3
+; GFX9-GISEL-NEXT: image_sample v[0:3], v[20:21], s[8:15], s[16:19] dmask:0xf
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[20:21]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB17_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v22, s4
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v23, s5
+; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX9-GISEL-NEXT: .LBB17_3: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v22
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s4, v23
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v22
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s4, v23
+; GFX9-GISEL-NEXT: s_and_b64 s[2:3], s[2:3], s[4:5]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v8
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s9, v9
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v10
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s11, v11
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v12
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s13, v13
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s14, v14
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s15, v15
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s16, v16
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v17
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v18
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v19
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr22
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr23
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr16_vgpr17_vgpr18_vgpr19
+; GFX9-GISEL-NEXT: s_nop 4
+; GFX9-GISEL-NEXT: image_sample v[4:7], v[20:21], s[8:15], s[16:19] dmask:0xf
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr20_vgpr21
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB17_3
+; GFX9-GISEL-NEXT: ; %bb.4:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-32-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_1_2loops:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-SDAG-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v8, 31, v7
+; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[3:4]
+; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[7:8]
+; GFX10-32-SDAG-NEXT: v_add_co_u32 v0, vcc_lo, s0, v0
+; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, s1, v1, vcc_lo
+; GFX10-32-SDAG-NEXT: v_add_co_u32 v2, vcc_lo, s2, v2
+; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v3, vcc_lo, s3, v3, vcc_lo
+; GFX10-32-SDAG-NEXT: s_clause 0x1
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[11:14], v[0:1], off offset:16
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[15:18], v[0:1], off
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[7:10], v[2:3], off
+; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s2, v5
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s3, v6
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s2, v5
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s3, v6
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s8, v15
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s9, v16
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s10, v17
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s11, v18
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s12, v11
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s13, v12
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s14, v13
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s15, v14
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s16, v7
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s17, v8
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s18, v9
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s19, v10
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v4, 0
+; GFX10-32-SDAG-NEXT: image_sample v[0:3], [v4, v4], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB17_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s8, v15
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s9, v16
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s10, v17
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s11, v18
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s12, v11
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s13, v12
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s14, v13
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s15, v14
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s0, v7
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s1, v8
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s2, v9
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s3, v10
+; GFX10-32-SDAG-NEXT: s_and_b32 exec_lo, exec_lo, s4
+; GFX10-32-SDAG-NEXT: image_sample v[4:7], [v4, v4], s[8:15], s[0:3] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-32-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_1_2loops:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: s_mov_b32 s6, exec_lo
+; GFX10-32-GISEL-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v6, v3
+; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v9, s3
+; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v7, 31, v6
+; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[2:3]
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v8, s2
+; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[6:7], 4, v[6:7]
+; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-GISEL-NEXT: v_add_co_u32 v0, vcc_lo, v2, v0
+; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
+; GFX10-32-GISEL-NEXT: v_add_co_u32 v2, vcc_lo, v8, v6
+; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v3, vcc_lo, v9, v7, vcc_lo
+; GFX10-32-GISEL-NEXT: s_clause 0x1
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[8:11], v[0:1], off
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[12:15], v[0:1], off offset:16
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[16:19], v[2:3], off
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v4
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s3, v5
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s2, v4
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s3, v5
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s8, v8
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s9, v9
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s10, v10
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s11, v11
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s12, v12
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s13, v13
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s14, v14
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s15, v15
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s16, v16
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s17, v17
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s18, v18
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s19, v19
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v20, 0
+; GFX10-32-GISEL-NEXT: image_sample v[0:3], [v20, v20], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB17_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v21, s4
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v22, s5
+; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-GISEL-NEXT: .LBB17_3: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v21
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s3, v22
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s2, v21
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s3, v22
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s8, v8
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s9, v9
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s10, v10
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s11, v11
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s12, v12
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s13, v13
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s14, v14
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s15, v15
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s16, v16
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s17, v17
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s18, v18
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s19, v19
+; GFX10-32-GISEL-NEXT: image_sample v[4:7], [v20, v20], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr21
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr22
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr16_vgpr17_vgpr18_vgpr19
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr20
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB17_3
+; GFX10-32-GISEL-NEXT: ; %bb.4:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-GISEL-NEXT: s_and_b32 exec_lo, exec_lo, s6
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-64-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_1_2loops:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v8, 31, v7
+; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[3:4]
+; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[7:8]
+; GFX10-64-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc, s1, v1, vcc
+; GFX10-64-SDAG-NEXT: v_add_co_u32 v2, vcc, s2, v2
+; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v3, vcc, s3, v3, vcc
+; GFX10-64-SDAG-NEXT: s_clause 0x1
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[11:14], v[0:1], off offset:16
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[15:18], v[0:1], off
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[7:10], v[2:3], off
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s6, v5
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s7, v6
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v5
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s7, v6
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s8, v15
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s9, v16
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s10, v17
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s11, v18
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s12, v11
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s13, v12
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s14, v13
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s15, v14
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s16, v7
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s17, v8
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s18, v9
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s19, v10
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v4, 0
+; GFX10-64-SDAG-NEXT: image_sample v[0:3], [v4, v4], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB17_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s8, v15
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s9, v16
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s10, v17
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s11, v18
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s12, v11
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s13, v12
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s14, v13
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s15, v14
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s0, v7
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s1, v8
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s2, v9
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s3, v10
+; GFX10-64-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX10-64-SDAG-NEXT: image_sample v[4:7], [v4, v4], s[8:15], s[0:3] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-64-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_1_2loops:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v6, v3
+; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v9, s3
+; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v7, 31, v6
+; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[2:3]
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v8, s2
+; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[6:7], 4, v[6:7]
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc, v3, v1, vcc
+; GFX10-64-GISEL-NEXT: v_add_co_u32 v2, vcc, v8, v6
+; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v3, vcc, v9, v7, vcc
+; GFX10-64-GISEL-NEXT: s_clause 0x1
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[8:11], v[0:1], off
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[12:15], v[0:1], off offset:16
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[16:19], v[2:3], off
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s8, v4
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s9, v5
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s8, v4
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s9, v5
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s8, v8
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s9, v9
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s10, v10
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s11, v11
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s12, v12
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s13, v13
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s14, v14
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s15, v15
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s16, v16
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s17, v17
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s18, v18
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s19, v19
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v20, 0
+; GFX10-64-GISEL-NEXT: image_sample v[0:3], [v20, v20], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB17_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v21, s4
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v22, s5
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-GISEL-NEXT: .LBB17_3: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s4, v21
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s5, v22
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v21
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s5, v22
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s8, v8
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s9, v9
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s10, v10
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s11, v11
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s12, v12
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s13, v13
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s14, v14
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s15, v15
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s16, v16
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s17, v17
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s18, v18
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s19, v19
+; GFX10-64-GISEL-NEXT: image_sample v[4:7], [v20, v20], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr21
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr22
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr16_vgpr17_vgpr18_vgpr19
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr20
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB17_3
+; GFX10-64-GISEL-NEXT: ; %bb.4:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX1150-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_1_2loops:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_4) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v8, 31, v7
+; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[3:4]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[7:8]
+; GFX1150-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_4)
+; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s1, v1, vcc
+; GFX1150-SDAG-NEXT: v_add_co_u32 v2, vcc, s2, v2
+; GFX1150-SDAG-NEXT: s_waitcnt_depctr depctr_va_vcc(0)
+; GFX1150-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v3, null, s3, v3, vcc
+; GFX1150-SDAG-NEXT: s_clause 0x1
+; GFX1150-SDAG-NEXT: global_load_b128 v[11:14], v[0:1], off offset:16
+; GFX1150-SDAG-NEXT: global_load_b128 v[15:18], v[0:1], off
+; GFX1150-SDAG-NEXT: global_load_b128 v[7:10], v[2:3], off
+; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s6, v5
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s7, v6
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v5
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s7, v6
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s8, v15
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s9, v16
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s10, v17
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s11, v18
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s12, v11
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s13, v12
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s14, v13
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s15, v14
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s16, v7
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s17, v8
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s18, v9
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s19, v10
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v4, 0
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: image_sample v[0:3], [v4, v4], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB17_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s8, v15
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s9, v16
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s10, v17
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s11, v18
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s12, v11
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s13, v12
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s14, v13
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s15, v14
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s0, v7
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s1, v8
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s2, v9
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s3, v10
+; GFX1150-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX1150-SDAG-NEXT: image_sample v[4:7], [v4, v4], s[8:15], s[0:3] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX1150-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_1_2loops:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v6, v3
+; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v9, s3
+; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v7, 31, v6
+; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[2:3]
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v8, s2
+; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[6:7], 4, v[6:7]
+; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_3)
+; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX1150-GISEL-NEXT: v_add_co_u32 v2, vcc, v8, v6
+; GFX1150-GISEL-NEXT: s_waitcnt_depctr depctr_va_vcc(0)
+; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v3, null, v9, v7, vcc
+; GFX1150-GISEL-NEXT: s_clause 0x1
+; GFX1150-GISEL-NEXT: global_load_b128 v[8:11], v[0:1], off
+; GFX1150-GISEL-NEXT: global_load_b128 v[12:15], v[0:1], off offset:16
+; GFX1150-GISEL-NEXT: global_load_b128 v[16:19], v[2:3], off
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s8, v4
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s9, v5
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s8, v4
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s9, v5
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s8, v8
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s9, v9
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s10, v10
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s11, v11
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s12, v12
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s13, v13
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s14, v14
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s15, v15
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s16, v16
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s17, v17
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s18, v18
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s19, v19
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v20, 0
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: image_sample v[0:3], [v20, v20], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB17_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v21, s4
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v22, s5
+; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-GISEL-NEXT: .LBB17_3: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s4, v21
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s5, v22
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v21
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s5, v22
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s8, v8
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s9, v9
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s10, v10
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s11, v11
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s12, v12
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s13, v13
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s14, v14
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s15, v15
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s16, v16
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s17, v17
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s18, v18
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s19, v19
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: image_sample v[4:7], [v20, v20], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr21
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr22
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr16_vgpr17_vgpr18_vgpr19
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr20
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB17_3
+; GFX1150-GISEL-NEXT: ; %bb.4:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX12-SDAG-LABEL: test_waterfall_multi_begin_uniform_idx_1_2loops:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v5, v0
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_4) | instskip(NEXT) | instid1(VALU_DEP_4)
+; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v8, 31, v7
+; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[2:3], 4, v[3:4]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[0:1], 5, v[7:8]
+; GFX12-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_4)
+; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s1, v1, vcc
+; GFX12-SDAG-NEXT: v_add_co_u32 v2, vcc, s2, v2
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_vcc(0)
+; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v3, null, s3, v3, vcc
+; GFX12-SDAG-NEXT: s_clause 0x1
+; GFX12-SDAG-NEXT: global_load_b128 v[11:14], v[0:1], off offset:16
+; GFX12-SDAG-NEXT: global_load_b128 v[15:18], v[0:1], off
+; GFX12-SDAG-NEXT: global_load_b128 v[7:10], v[2:3], off
+; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s6, v5
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s7, v6
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v5
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s7, v6
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s8, v15
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s9, v16
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s10, v17
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s11, v18
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s12, v11
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s13, v12
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s14, v13
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s15, v14
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x0
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s16, v7
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s17, v8
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s18, v9
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s19, v10
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v4, 0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: image_sample v[0:3], [v4, v4], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr6
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB17_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s8, v15
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s9, v16
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s10, v17
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s11, v18
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s12, v11
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s13, v12
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s14, v13
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s15, v14
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s0, v7
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s1, v8
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s2, v9
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s3, v10
+; GFX12-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX12-SDAG-NEXT: image_sample v[4:7], [v4, v4], s[8:15], s[0:3] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX12-GISEL-LABEL: test_waterfall_multi_begin_uniform_idx_1_2loops:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v6, v3
+; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v5, v1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v9, s3
+; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v7, 31, v6
+; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[0:1], 5, v[2:3]
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v8, s2
+; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[6:7], 4, v[6:7]
+; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_3)
+; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX12-GISEL-NEXT: v_add_co_u32 v2, vcc, v8, v6
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_vcc(0)
+; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v3, null, v9, v7, vcc
+; GFX12-GISEL-NEXT: s_clause 0x1
+; GFX12-GISEL-NEXT: global_load_b128 v[8:11], v[0:1], off
+; GFX12-GISEL-NEXT: global_load_b128 v[12:15], v[0:1], off offset:16
+; GFX12-GISEL-NEXT: global_load_b128 v[16:19], v[2:3], off
+; GFX12-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s8, v4
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s9, v5
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s8, v4
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s9, v5
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x2
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s8, v8
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s9, v9
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s10, v10
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s11, v11
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s12, v12
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s13, v13
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s14, v14
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s15, v15
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s16, v16
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s17, v17
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s18, v18
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s19, v19
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v20, 0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: image_sample v[0:3], [v20, v20], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB17_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v21, s4
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v22, s5
+; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-GISEL-NEXT: .LBB17_3: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s4, v21
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s5, v22
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v21
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s5, v22
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s8, v8
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s9, v9
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s10, v10
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s11, v11
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s12, v12
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s13, v13
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s14, v14
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s15, v15
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s16, v16
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s17, v17
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s18, v18
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s19, v19
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: image_sample v[4:7], [v20, v20], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr21
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr22
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr16_vgpr17_vgpr18_vgpr19
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr20
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB17_3
+; GFX12-GISEL-NEXT: ; %bb.4:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: ; return to shader part epilog
+ <8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %s_in,
+ i32 %idx1, i32 %idx2, i32 inreg %idx3, i32 inreg%idx4,
+ i32 %s_idx, i32 %s_idx2) #1 {
+ %rptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
+ %sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
+ %srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
+ %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
+ %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok, i32 %idx2)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok1, <8 x i32> %rsrc)
+ %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok1, <4 x i32> %srsrc)
+ %r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
+
+ %tok2 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx3)
+ %tok3 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok2, i32 %idx4)
+ %s_rsrc1 = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok3, <8 x i32> %rsrc)
+ %s_srsrc1 = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok3, <4 x i32> %srsrc)
+ %r2 = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc1, <4 x i32> %s_srsrc1, i1 false, i32 0, i32 0)
+ %r3 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok3, <4 x float> %r2)
+
+ %insert = insertvalue { <4 x float>, <4 x float> } undef, <4 x float> %r1, 0
+ %insert1 = insertvalue { <4 x float>, <4 x float> } %insert, <4 x float> %r3, 1
+ ret {<4 x float> , <4 x float>} %insert1
+}
+
+define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fptr) {
+; PRE-GFX10-LABEL: test_indirect_call_vgpr_ptr_arg_and_reuse:
+; PRE-GFX10: ; %bb.0:
+; PRE-GFX10-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; PRE-GFX10-NEXT: s_mov_b32 s34, s33
+; PRE-GFX10-NEXT: s_mov_b32 s33, s32
+; PRE-GFX10-NEXT: s_or_saveexec_b64 s[36:37], -1
+; PRE-GFX10-NEXT: buffer_store_dword v40, off, s[0:3], s33 ; 4-byte Folded Spill
+; PRE-GFX10-NEXT: s_mov_b64 exec, s[36:37]
+; PRE-GFX10-NEXT: v_writelane_b32 v40, s34, 8
+; PRE-GFX10-NEXT: v_writelane_b32 v40, s4, 0
+; PRE-GFX10-NEXT: v_writelane_b32 v40, s5, 1
+; PRE-GFX10-NEXT: v_writelane_b32 v40, s6, 2
+; PRE-GFX10-NEXT: v_writelane_b32 v40, s7, 3
+; PRE-GFX10-NEXT: v_writelane_b32 v40, s8, 4
+; PRE-GFX10-NEXT: v_writelane_b32 v40, s9, 5
+; PRE-GFX10-NEXT: v_writelane_b32 v40, s30, 6
+; PRE-GFX10-NEXT: s_mov_b32 s5, 0
+; PRE-GFX10-NEXT: s_mov_b64 s[6:7], exec
+; PRE-GFX10-NEXT: s_addk_i32 s32, 0x400
+; PRE-GFX10-NEXT: v_writelane_b32 v40, s31, 7
+; PRE-GFX10-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
+; PRE-GFX10-NEXT: v_readfirstlane_b32 s4, v1
+; PRE-GFX10-NEXT: v_cmp_eq_u32_e64 s[34:35], s4, v1
+; PRE-GFX10-NEXT: s_and_saveexec_b64 s[8:9], s[34:35]
+; PRE-GFX10-NEXT: s_swappc_b64 s[30:31], s[4:5]
+; PRE-GFX10-NEXT: v_mov_b32_e32 v2, v0
+; PRE-GFX10-NEXT: ; implicit-def: $vgpr1
+; PRE-GFX10-NEXT: ; implicit-def: $vgpr0
+; PRE-GFX10-NEXT: s_xor_b64 exec, exec, s[8:9]
+; PRE-GFX10-NEXT: s_cbranch_execnz .LBB18_1
+; PRE-GFX10-NEXT: ; %bb.2:
+; PRE-GFX10-NEXT: s_mov_b64 exec, s[6:7]
+; PRE-GFX10-NEXT: v_mov_b32_e32 v0, v2
+; PRE-GFX10-NEXT: v_readlane_b32 s31, v40, 7
+; PRE-GFX10-NEXT: v_readlane_b32 s30, v40, 6
+; PRE-GFX10-NEXT: v_readlane_b32 s9, v40, 5
+; PRE-GFX10-NEXT: v_readlane_b32 s8, v40, 4
+; PRE-GFX10-NEXT: v_readlane_b32 s7, v40, 3
+; PRE-GFX10-NEXT: v_readlane_b32 s6, v40, 2
+; PRE-GFX10-NEXT: v_readlane_b32 s5, v40, 1
+; PRE-GFX10-NEXT: v_readlane_b32 s4, v40, 0
+; PRE-GFX10-NEXT: s_mov_b32 s32, s33
+; PRE-GFX10-NEXT: v_readlane_b32 s34, v40, 8
+; PRE-GFX10-NEXT: s_or_saveexec_b64 s[36:37], -1
+; PRE-GFX10-NEXT: buffer_load_dword v40, off, s[0:3], s33 ; 4-byte Folded Reload
+; PRE-GFX10-NEXT: s_mov_b64 exec, s[36:37]
+; PRE-GFX10-NEXT: s_mov_b32 s33, s34
+; PRE-GFX10-NEXT: s_waitcnt vmcnt(0)
+; PRE-GFX10-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX10-32-LABEL: test_indirect_call_vgpr_ptr_arg_and_reuse:
+; GFX10-32: ; %bb.0:
+; GFX10-32-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10-32-NEXT: s_mov_b32 s34, s33
+; GFX10-32-NEXT: s_mov_b32 s33, s32
+; GFX10-32-NEXT: s_or_saveexec_b32 s35, -1
+; GFX10-32-NEXT: buffer_store_dword v40, off, s[0:3], s33 ; 4-byte Folded Spill
+; GFX10-32-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-NEXT: s_mov_b32 exec_lo, s35
+; GFX10-32-NEXT: v_writelane_b32 v40, s34, 6
+; GFX10-32-NEXT: s_addk_i32 s32, 0x200
+; GFX10-32-NEXT: v_writelane_b32 v40, s4, 0
+; GFX10-32-NEXT: v_writelane_b32 v40, s5, 1
+; GFX10-32-NEXT: s_mov_b32 s5, 0
+; GFX10-32-NEXT: v_writelane_b32 v40, s6, 2
+; GFX10-32-NEXT: s_mov_b32 s6, exec_lo
+; GFX10-32-NEXT: v_writelane_b32 v40, s7, 3
+; GFX10-32-NEXT: s_mov_b32 s7, exec_lo
+; GFX10-32-NEXT: v_writelane_b32 v40, s30, 4
+; GFX10-32-NEXT: v_writelane_b32 v40, s31, 5
+; GFX10-32-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-NEXT: v_readfirstlane_b32 s4, v1
+; GFX10-32-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-NEXT: v_cmpx_eq_u32_e32 s4, v1
+; GFX10-32-NEXT: s_swappc_b64 s[30:31], s[4:5]
+; GFX10-32-NEXT: v_mov_b32_e32 v2, v0
+; GFX10-32-NEXT: s_andn2_wrexec_b32 s7, s7
+; GFX10-32-NEXT: ; implicit-def: $vgpr1
+; GFX10-32-NEXT: ; implicit-def: $vgpr0
+; GFX10-32-NEXT: s_cbranch_execnz .LBB18_1
+; GFX10-32-NEXT: ; %bb.2:
+; GFX10-32-NEXT: s_mov_b32 exec_lo, s6
+; GFX10-32-NEXT: v_mov_b32_e32 v0, v2
+; GFX10-32-NEXT: v_readlane_b32 s31, v40, 5
+; GFX10-32-NEXT: v_readlane_b32 s30, v40, 4
+; GFX10-32-NEXT: v_readlane_b32 s7, v40, 3
+; GFX10-32-NEXT: v_readlane_b32 s6, v40, 2
+; GFX10-32-NEXT: v_readlane_b32 s5, v40, 1
+; GFX10-32-NEXT: v_readlane_b32 s4, v40, 0
+; GFX10-32-NEXT: s_mov_b32 s32, s33
+; GFX10-32-NEXT: v_readlane_b32 s34, v40, 6
+; GFX10-32-NEXT: s_or_saveexec_b32 s35, -1
+; GFX10-32-NEXT: buffer_load_dword v40, off, s[0:3], s33 ; 4-byte Folded Reload
+; GFX10-32-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-NEXT: s_mov_b32 exec_lo, s35
+; GFX10-32-NEXT: s_mov_b32 s33, s34
+; GFX10-32-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX10-64-LABEL: test_indirect_call_vgpr_ptr_arg_and_reuse:
+; GFX10-64: ; %bb.0:
+; GFX10-64-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10-64-NEXT: s_mov_b32 s34, s33
+; GFX10-64-NEXT: s_mov_b32 s33, s32
+; GFX10-64-NEXT: s_or_saveexec_b64 s[36:37], -1
+; GFX10-64-NEXT: buffer_store_dword v40, off, s[0:3], s33 ; 4-byte Folded Spill
+; GFX10-64-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-NEXT: s_mov_b64 exec, s[36:37]
+; GFX10-64-NEXT: v_writelane_b32 v40, s34, 8
+; GFX10-64-NEXT: s_addk_i32 s32, 0x400
+; GFX10-64-NEXT: v_writelane_b32 v40, s4, 0
+; GFX10-64-NEXT: v_writelane_b32 v40, s5, 1
+; GFX10-64-NEXT: s_mov_b32 s5, 0
+; GFX10-64-NEXT: v_writelane_b32 v40, s6, 2
+; GFX10-64-NEXT: v_writelane_b32 v40, s7, 3
+; GFX10-64-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-NEXT: v_writelane_b32 v40, s8, 4
+; GFX10-64-NEXT: v_writelane_b32 v40, s9, 5
+; GFX10-64-NEXT: s_mov_b64 s[8:9], exec
+; GFX10-64-NEXT: v_writelane_b32 v40, s30, 6
+; GFX10-64-NEXT: v_writelane_b32 v40, s31, 7
+; GFX10-64-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-NEXT: v_readfirstlane_b32 s4, v1
+; GFX10-64-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-NEXT: v_cmpx_eq_u32_e64 s4, v1
+; GFX10-64-NEXT: s_swappc_b64 s[30:31], s[4:5]
+; GFX10-64-NEXT: v_mov_b32_e32 v2, v0
+; GFX10-64-NEXT: s_andn2_wrexec_b64 s[8:9], s[8:9]
+; GFX10-64-NEXT: ; implicit-def: $vgpr1
+; GFX10-64-NEXT: ; implicit-def: $vgpr0
+; GFX10-64-NEXT: s_cbranch_execnz .LBB18_1
+; GFX10-64-NEXT: ; %bb.2:
+; GFX10-64-NEXT: s_mov_b64 exec, s[6:7]
+; GFX10-64-NEXT: v_mov_b32_e32 v0, v2
+; GFX10-64-NEXT: v_readlane_b32 s31, v40, 7
+; GFX10-64-NEXT: v_readlane_b32 s30, v40, 6
+; GFX10-64-NEXT: v_readlane_b32 s9, v40, 5
+; GFX10-64-NEXT: v_readlane_b32 s8, v40, 4
+; GFX10-64-NEXT: v_readlane_b32 s7, v40, 3
+; GFX10-64-NEXT: v_readlane_b32 s6, v40, 2
+; GFX10-64-NEXT: v_readlane_b32 s5, v40, 1
+; GFX10-64-NEXT: v_readlane_b32 s4, v40, 0
+; GFX10-64-NEXT: s_mov_b32 s32, s33
+; GFX10-64-NEXT: v_readlane_b32 s34, v40, 8
+; GFX10-64-NEXT: s_or_saveexec_b64 s[36:37], -1
+; GFX10-64-NEXT: buffer_load_dword v40, off, s[0:3], s33 ; 4-byte Folded Reload
+; GFX10-64-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-NEXT: s_mov_b64 exec, s[36:37]
+; GFX10-64-NEXT: s_mov_b32 s33, s34
+; GFX10-64-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX1150-LABEL: test_indirect_call_vgpr_ptr_arg_and_reuse:
+; GFX1150: ; %bb.0:
+; GFX1150-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX1150-NEXT: s_mov_b32 s0, s33
+; GFX1150-NEXT: s_mov_b32 s33, s32
+; GFX1150-NEXT: s_or_saveexec_b64 s[2:3], -1
+; GFX1150-NEXT: scratch_store_b32 off, v40, s33 ; 4-byte Folded Spill
+; GFX1150-NEXT: s_mov_b64 exec, s[2:3]
+; GFX1150-NEXT: v_writelane_b32 v40, s0, 8
+; GFX1150-NEXT: s_add_i32 s32, s32, 16
+; GFX1150-NEXT: v_writelane_b32 v40, s4, 0
+; GFX1150-NEXT: v_writelane_b32 v40, s5, 1
+; GFX1150-NEXT: s_mov_b32 s5, 0
+; GFX1150-NEXT: v_writelane_b32 v40, s6, 2
+; GFX1150-NEXT: v_writelane_b32 v40, s7, 3
+; GFX1150-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-NEXT: v_writelane_b32 v40, s8, 4
+; GFX1150-NEXT: v_writelane_b32 v40, s9, 5
+; GFX1150-NEXT: s_mov_b64 s[8:9], exec
+; GFX1150-NEXT: v_writelane_b32 v40, s30, 6
+; GFX1150-NEXT: v_writelane_b32 v40, s31, 7
+; GFX1150-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-NEXT: v_readfirstlane_b32 s4, v1
+; GFX1150-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-NEXT: v_cmpx_eq_u32_e64 s4, v1
+; GFX1150-NEXT: s_swappc_b64 s[30:31], s[4:5]
+; GFX1150-NEXT: v_mov_b32_e32 v2, v0
+; GFX1150-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX1150-NEXT: ; implicit-def: $vgpr1
+; GFX1150-NEXT: ; implicit-def: $vgpr0
+; GFX1150-NEXT: s_cbranch_execnz .LBB18_1
+; GFX1150-NEXT: ; %bb.2:
+; GFX1150-NEXT: s_mov_b64 exec, s[6:7]
+; GFX1150-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-NEXT: v_mov_b32_e32 v0, v2
+; GFX1150-NEXT: v_readlane_b32 s31, v40, 7
+; GFX1150-NEXT: v_readlane_b32 s30, v40, 6
+; GFX1150-NEXT: v_readlane_b32 s9, v40, 5
+; GFX1150-NEXT: v_readlane_b32 s8, v40, 4
+; GFX1150-NEXT: v_readlane_b32 s7, v40, 3
+; GFX1150-NEXT: v_readlane_b32 s6, v40, 2
+; GFX1150-NEXT: v_readlane_b32 s5, v40, 1
+; GFX1150-NEXT: v_readlane_b32 s4, v40, 0
+; GFX1150-NEXT: s_mov_b32 s32, s33
+; GFX1150-NEXT: v_readlane_b32 s0, v40, 8
+; GFX1150-NEXT: s_or_saveexec_b64 s[2:3], -1
+; GFX1150-NEXT: scratch_load_b32 v40, off, s33 ; 4-byte Folded Reload
+; GFX1150-NEXT: s_mov_b64 exec, s[2:3]
+; GFX1150-NEXT: s_mov_b32 s33, s0
+; GFX1150-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-NEXT: s_setpc_b64 s[30:31]
+;
+; GFX12-LABEL: test_indirect_call_vgpr_ptr_arg_and_reuse:
+; GFX12: ; %bb.0:
+; GFX12-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-NEXT: s_wait_expcnt 0x0
+; GFX12-NEXT: s_wait_samplecnt 0x0
+; GFX12-NEXT: s_wait_bvhcnt 0x0
+; GFX12-NEXT: s_wait_kmcnt 0x0
+; GFX12-NEXT: s_mov_b32 s0, s33
+; GFX12-NEXT: s_mov_b32 s33, s32
+; GFX12-NEXT: s_or_saveexec_b64 s[2:3], -1
+; GFX12-NEXT: scratch_store_b32 off, v40, s33 ; 4-byte Folded Spill
+; GFX12-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-NEXT: s_mov_b64 exec, s[2:3]
+; GFX12-NEXT: v_writelane_b32 v40, s0, 8
+; GFX12-NEXT: s_add_co_i32 s32, s32, 16
+; GFX12-NEXT: v_writelane_b32 v40, s4, 0
+; GFX12-NEXT: v_writelane_b32 v40, s5, 1
+; GFX12-NEXT: s_mov_b32 s5, 0
+; GFX12-NEXT: v_writelane_b32 v40, s6, 2
+; GFX12-NEXT: v_writelane_b32 v40, s7, 3
+; GFX12-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-NEXT: v_writelane_b32 v40, s8, 4
+; GFX12-NEXT: v_writelane_b32 v40, s9, 5
+; GFX12-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-NEXT: v_writelane_b32 v40, s30, 6
+; GFX12-NEXT: v_writelane_b32 v40, s31, 7
+; GFX12-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-NEXT: v_readfirstlane_b32 s4, v1
+; GFX12-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-NEXT: v_cmpx_eq_u32_e64 s4, v1
+; GFX12-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-NEXT: s_swappc_b64 s[30:31], s[4:5]
+; GFX12-NEXT: v_mov_b32_e32 v2, v0
+; GFX12-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX12-NEXT: ; implicit-def: $vgpr1
+; GFX12-NEXT: ; implicit-def: $vgpr0
+; GFX12-NEXT: s_cbranch_execnz .LBB18_1
+; GFX12-NEXT: ; %bb.2:
+; GFX12-NEXT: s_mov_b64 exec, s[6:7]
+; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-NEXT: v_mov_b32_e32 v0, v2
+; GFX12-NEXT: v_readlane_b32 s31, v40, 7
+; GFX12-NEXT: v_readlane_b32 s30, v40, 6
+; GFX12-NEXT: v_readlane_b32 s9, v40, 5
+; GFX12-NEXT: v_readlane_b32 s8, v40, 4
+; GFX12-NEXT: v_readlane_b32 s7, v40, 3
+; GFX12-NEXT: v_readlane_b32 s6, v40, 2
+; GFX12-NEXT: v_readlane_b32 s5, v40, 1
+; GFX12-NEXT: v_readlane_b32 s4, v40, 0
+; GFX12-NEXT: s_mov_b32 s32, s33
+; GFX12-NEXT: v_readlane_b32 s0, v40, 8
+; GFX12-NEXT: s_or_saveexec_b64 s[2:3], -1
+; GFX12-NEXT: scratch_load_b32 v40, off, s33 ; 4-byte Folded Reload
+; GFX12-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-NEXT: s_mov_b64 exec, s[2:3]
+; GFX12-NEXT: s_mov_b32 s33, s0
+; GFX12-NEXT: s_wait_loadcnt 0x0
+; GFX12-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-NEXT: s_setpc_b64 s[30:31]
+ %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %fptr)
+ %s_fptr = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %tok, i32 %fptr)
+ %ext = zext i32 %s_fptr to i64
+ %f = inttoptr i64 %ext to i32(i32)*
+ %callres = call amdgpu_gfx i32 %f(i32 %i)
+ %r3 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %tok, i32 %callres)
+ ret i32 %r3
+}
+
+define amdgpu_cs void @ds_write_8(i8 %value, i32 %index) #1 {
+; VI-LABEL: ds_write_8:
+; VI: ; %bb.0: ; %.entry
+; VI-NEXT: v_lshlrev_b32_e32 v1, 2, v1
+; VI-NEXT: v_add_u32_e32 v1, vcc, Lds at abs32@lo, v1
+; VI-NEXT: s_mov_b32 m0, -1
+; VI-NEXT: s_mov_b64 s[0:1], exec
+; VI-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
+; VI-NEXT: v_readfirstlane_b32 s0, v1
+; VI-NEXT: v_cmp_eq_u32_e64 s[0:1], s0, v1
+; VI-NEXT: s_and_saveexec_b64 s[0:1], s[0:1]
+; VI-NEXT: ds_write_b8 v1, v0
+; VI-NEXT: ; implicit-def: $vgpr1
+; VI-NEXT: ; implicit-def: $vgpr0
+; VI-NEXT: s_xor_b64 exec, exec, s[0:1]
+; VI-NEXT: s_cbranch_execnz .LBB19_1
+; VI-NEXT: ; %bb.2:
+; VI-NEXT: s_endpgm
+;
+; GFX9-SDAG-LABEL: ds_write_8:
+; GFX9-SDAG: ; %bb.0: ; %.entry
+; GFX9-SDAG-NEXT: s_mov_b32 s0, Lds at abs32@lo
+; GFX9-SDAG-NEXT: v_lshl_add_u32 v1, v1, 2, s0
+; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX9-SDAG-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s0, v1
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[0:1], s0, v1
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[0:1], s[0:1]
+; GFX9-SDAG-NEXT: ds_write_b8 v1, v0
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr1
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[0:1]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB19_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_endpgm
+;
+; GFX9-GISEL-LABEL: ds_write_8:
+; GFX9-GISEL: ; %bb.0: ; %.entry
+; GFX9-GISEL-NEXT: v_lshlrev_b32_e32 v1, 2, v1
+; GFX9-GISEL-NEXT: v_add_u32_e32 v1, Lds at abs32@lo, v1
+; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX9-GISEL-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s0, v1
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[0:1], s0, v1
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[0:1], s[0:1]
+; GFX9-GISEL-NEXT: ds_write_b8 v1, v0
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr1
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[0:1]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB19_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_endpgm
+;
+; GFX10-32-SDAG-LABEL: ds_write_8:
+; GFX10-32-SDAG: ; %bb.0: ; %.entry
+; GFX10-32-SDAG-NEXT: v_lshl_add_u32 v1, v1, 2, Lds at abs32@lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-SDAG-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s1, v1
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s1, v1
+; GFX10-32-SDAG-NEXT: ds_write_b8 v1, v0
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s0, s0
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr1
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB19_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_endpgm
+;
+; GFX10-32-GISEL-LABEL: ds_write_8:
+; GFX10-32-GISEL: ; %bb.0: ; %.entry
+; GFX10-32-GISEL-NEXT: v_lshlrev_b32_e32 v1, 2, v1
+; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-GISEL-NEXT: v_add_nc_u32_e32 v1, Lds at abs32@lo, v1
+; GFX10-32-GISEL-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s1, v1
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s1, v1
+; GFX10-32-GISEL-NEXT: ds_write_b8 v1, v0
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s0, s0
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr1
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB19_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_endpgm
+;
+; GFX10-64-SDAG-LABEL: ds_write_8:
+; GFX10-64-SDAG: ; %bb.0: ; %.entry
+; GFX10-64-SDAG-NEXT: v_lshl_add_u32 v1, v1, 2, Lds at abs32@lo
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-SDAG-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s2, v1
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s2, v1
+; GFX10-64-SDAG-NEXT: ds_write_b8 v1, v0
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[0:1], s[0:1]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr1
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB19_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_endpgm
+;
+; GFX10-64-GISEL-LABEL: ds_write_8:
+; GFX10-64-GISEL: ; %bb.0: ; %.entry
+; GFX10-64-GISEL-NEXT: v_lshlrev_b32_e32 v1, 2, v1
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-GISEL-NEXT: v_add_nc_u32_e32 v1, Lds at abs32@lo, v1
+; GFX10-64-GISEL-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s2, v1
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s2, v1
+; GFX10-64-GISEL-NEXT: ds_write_b8 v1, v0
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[0:1], s[0:1]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr1
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB19_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_endpgm
+;
+; GFX1150-SDAG-LABEL: ds_write_8:
+; GFX1150-SDAG: ; %bb.0: ; %.entry
+; GFX1150-SDAG-NEXT: v_lshl_add_u32 v1, v1, 2, Lds at abs32@lo
+; GFX1150-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-SDAG-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s2, v1
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s2, v1
+; GFX1150-SDAG-NEXT: ds_store_b8 v1, v0
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[0:1], s[0:1]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr1
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB19_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_endpgm
+;
+; GFX1150-GISEL-LABEL: ds_write_8:
+; GFX1150-GISEL: ; %bb.0: ; %.entry
+; GFX1150-GISEL-NEXT: v_lshlrev_b32_e32 v1, 2, v1
+; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-GISEL-NEXT: v_add_nc_u32_e32 v1, Lds at abs32@lo, v1
+; GFX1150-GISEL-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s2, v1
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s2, v1
+; GFX1150-GISEL-NEXT: ds_store_b8 v1, v0
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[0:1], s[0:1]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr1
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB19_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_endpgm
+;
+; GFX12-SDAG-LABEL: ds_write_8:
+; GFX12-SDAG: ; %bb.0: ; %.entry
+; GFX12-SDAG-NEXT: v_lshl_add_u32 v1, v1, 2, Lds at abs32@lo
+; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-SDAG-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s2, v1
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s2, v1
+; GFX12-SDAG-NEXT: ds_store_b8 v1, v0
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[0:1], s[0:1]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr1
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr0
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB19_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_endpgm
+;
+; GFX12-GISEL-LABEL: ds_write_8:
+; GFX12-GISEL: ; %bb.0: ; %.entry
+; GFX12-GISEL-NEXT: v_lshlrev_b32_e32 v1, 2, v1
+; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_add_nc_u32_e32 v1, Lds at abs32@lo, v1
+; GFX12-GISEL-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s2, v1
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s2, v1
+; GFX12-GISEL-NEXT: ds_store_b8 v1, v0
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[0:1], s[0:1]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr1
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr0
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB19_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_endpgm
+.entry:
+ %gep = getelementptr [16384 x i32], ptr addrspace(3) @Lds, i32 0, i32 %index
+ %0 = call i32 @llvm.amdgcn.waterfall.begin.p3(i32 0, ptr addrspace(3) %gep)
+ %1 = call ptr addrspace(3) @llvm.amdgcn.waterfall.last.use.vgpr.p3(i32 %0, ptr addrspace(3) %gep)
+ store i8 %value, ptr addrspace(3) %1, align 1
+ ret void
+}
+
+define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
+; VI-SDAG-LABEL: test_waterfall_multi_end_1loop:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; VI-SDAG-NEXT: s_wqm_b64 exec, exec
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, v2
+; VI-SDAG-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; VI-SDAG-NEXT: v_mov_b32_e32 v8, v1
+; VI-SDAG-NEXT: v_mov_b32_e32 v9, v0
+; VI-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[4:5]
+; VI-SDAG-NEXT: v_mov_b32_e32 v2, s1
+; VI-SDAG-NEXT: v_add_u32_e32 v0, vcc, s0, v0
+; VI-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; VI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, v2, v1, vcc
+; VI-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[3:4]
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, s3
+; VI-SDAG-NEXT: v_add_u32_e32 v2, vcc, s2, v2
+; VI-SDAG-NEXT: v_addc_u32_e32 v3, vcc, v4, v3, vcc
+; VI-SDAG-NEXT: flat_load_dwordx4 v[10:13], v[0:1]
+; VI-SDAG-NEXT: v_add_u32_e32 v0, vcc, 16, v0
+; VI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; VI-SDAG-NEXT: flat_load_dwordx4 v[14:17], v[0:1]
+; VI-SDAG-NEXT: flat_load_dwordx4 v[18:21], v[2:3]
+; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; VI-SDAG-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v9
+; VI-SDAG-NEXT: v_readfirstlane_b32 s6, v8
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v9
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[6:7], s6, v8
+; VI-SDAG-NEXT: s_and_b64 s[2:3], s[2:3], s[6:7]
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(1)
+; VI-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, 0.5
+; VI-SDAG-NEXT: s_waitcnt vmcnt(2)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v10
+; VI-SDAG-NEXT: v_readfirstlane_b32 s9, v11
+; VI-SDAG-NEXT: v_readfirstlane_b32 s10, v12
+; VI-SDAG-NEXT: v_readfirstlane_b32 s11, v13
+; VI-SDAG-NEXT: s_waitcnt vmcnt(1)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s12, v14
+; VI-SDAG-NEXT: v_readfirstlane_b32 s13, v15
+; VI-SDAG-NEXT: v_readfirstlane_b32 s14, v16
+; VI-SDAG-NEXT: v_readfirstlane_b32 s15, v17
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: v_readfirstlane_b32 s16, v18
+; VI-SDAG-NEXT: v_readfirstlane_b32 s17, v19
+; VI-SDAG-NEXT: v_readfirstlane_b32 s18, v20
+; VI-SDAG-NEXT: v_readfirstlane_b32 s19, v21
+; VI-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; VI-SDAG-NEXT: v_mov_b32_e32 v5, v4
+; VI-SDAG-NEXT: s_nop 2
+; VI-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; VI-SDAG-NEXT: image_sample v[4:7], v[4:5], s[8:15], s[16:19] dmask:0xf
+; VI-SDAG-NEXT: ; implicit-def: $vgpr9
+; VI-SDAG-NEXT: ; implicit-def: $vgpr8
+; VI-SDAG-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
+; VI-SDAG-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB20_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; VI-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: ; return to shader part epilog
+;
+; VI-GISEL-LABEL: test_waterfall_multi_end_1loop:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; VI-GISEL-NEXT: s_wqm_b64 exec, exec
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, v3
+; VI-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; VI-GISEL-NEXT: v_mov_b32_e32 v8, v0
+; VI-GISEL-NEXT: v_mov_b32_e32 v9, v1
+; VI-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[2:3]
+; VI-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; VI-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; VI-GISEL-NEXT: v_add_u32_e32 v0, vcc, v2, v0
+; VI-GISEL-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, v3, v1, vcc
+; VI-GISEL-NEXT: v_lshlrev_b64 v[2:3], 4, v[4:5]
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, s3
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, s2
+; VI-GISEL-NEXT: v_add_u32_e32 v2, vcc, v4, v2
+; VI-GISEL-NEXT: v_addc_u32_e32 v3, vcc, v5, v3, vcc
+; VI-GISEL-NEXT: flat_load_dwordx4 v[10:13], v[0:1]
+; VI-GISEL-NEXT: v_add_u32_e32 v0, vcc, 16, v0
+; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; VI-GISEL-NEXT: flat_load_dwordx4 v[14:17], v[0:1]
+; VI-GISEL-NEXT: flat_load_dwordx4 v[18:21], v[2:3]
+; VI-GISEL-NEXT: s_mov_b32 s0, 0
+; VI-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; VI-GISEL-NEXT: s_mov_b32 s20, 0.5
+; VI-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s1, v8
+; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v9
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[6:7], s1, v8
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v9
+; VI-GISEL-NEXT: s_and_b64 s[6:7], s[6:7], s[8:9]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[6:7], s[6:7]
+; VI-GISEL-NEXT: s_mov_b32 s1, s0
+; VI-GISEL-NEXT: s_waitcnt vmcnt(1)
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; VI-GISEL-NEXT: s_mov_b32 s21, s20
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, s20
+; VI-GISEL-NEXT: s_waitcnt vmcnt(2)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v10
+; VI-GISEL-NEXT: v_readfirstlane_b32 s9, v11
+; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v12
+; VI-GISEL-NEXT: v_readfirstlane_b32 s11, v13
+; VI-GISEL-NEXT: s_waitcnt vmcnt(1)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v14
+; VI-GISEL-NEXT: v_readfirstlane_b32 s13, v15
+; VI-GISEL-NEXT: v_readfirstlane_b32 s14, v16
+; VI-GISEL-NEXT: v_readfirstlane_b32 s15, v17
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: v_readfirstlane_b32 s16, v18
+; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v19
+; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v20
+; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v21
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, s21
+; VI-GISEL-NEXT: s_nop 2
+; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; VI-GISEL-NEXT: image_sample v[4:7], v[4:5], s[8:15], s[16:19] dmask:0xf
+; VI-GISEL-NEXT: ; implicit-def: $vgpr8
+; VI-GISEL-NEXT: ; implicit-def: $vgpr9
+; VI-GISEL-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
+; VI-GISEL-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[6:7]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB20_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; VI-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX9-SDAG-LABEL: test_waterfall_multi_end_1loop:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v4, v2
+; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v8, v1
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v9, v0
+; GFX9-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[4:5]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v2, s1
+; GFX9-SDAG-NEXT: v_add_co_u32_e32 v0, vcc, s0, v0
+; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v1, vcc, v2, v1, vcc
+; GFX9-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[3:4]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v4, s3
+; GFX9-SDAG-NEXT: v_add_co_u32_e32 v2, vcc, s2, v2
+; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v3, vcc, v4, v3, vcc
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[14:17], v[0:1], off offset:16
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[10:13], v[0:1], off
+; GFX9-SDAG-NEXT: global_load_dwordx4 v[18:21], v[2:3], off
+; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v9
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s6, v8
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v9
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[6:7], s6, v8
+; GFX9-SDAG-NEXT: s_and_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v8, 0
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v10
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s9, v11
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s10, v12
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s11, v13
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s12, v14
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s13, v15
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s14, v16
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s15, v17
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s16, v18
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s17, v19
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s18, v20
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s19, v21
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v9, v8
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v10, 0.5
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v11, v10
+; GFX9-SDAG-NEXT: s_nop 1
+; GFX9-SDAG-NEXT: image_sample v[0:3], v[8:9], s[8:15], s[16:19] dmask:0xf
+; GFX9-SDAG-NEXT: image_sample v[4:7], v[10:11], s[8:15], s[16:19] dmask:0xf
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr9
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr8
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB20_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX9-GISEL-LABEL: test_waterfall_multi_end_1loop:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, v3
+; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v8, v0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, v1
+; GFX9-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[2:3]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX9-GISEL-NEXT: v_add_co_u32_e32 v0, vcc, v2, v0
+; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v1, vcc, v3, v1, vcc
+; GFX9-GISEL-NEXT: v_lshlrev_b64 v[2:3], 4, v[4:5]
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v5, s3
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, s2
+; GFX9-GISEL-NEXT: v_add_co_u32_e32 v2, vcc, v4, v2
+; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v3, vcc, v5, v3, vcc
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[10:13], v[0:1], off
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[14:17], v[0:1], off offset:16
+; GFX9-GISEL-NEXT: global_load_dwordx4 v[18:21], v[2:3], off
+; GFX9-GISEL-NEXT: s_mov_b32 s0, 0
+; GFX9-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX9-GISEL-NEXT: s_mov_b32 s20, 0.5
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s1, v8
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v9
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[6:7], s1, v8
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v9
+; GFX9-GISEL-NEXT: s_and_b64 s[6:7], s[6:7], s[8:9]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[6:7], s[6:7]
+; GFX9-GISEL-NEXT: s_mov_b32 s1, s0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, s1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v10
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s9, v11
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v12
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s11, v13
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v14
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s13, v15
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s14, v16
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s15, v17
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s16, v18
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v19
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v20
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v21
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v8, s0
+; GFX9-GISEL-NEXT: s_mov_b32 s21, s20
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v10, s20
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v11, s21
+; GFX9-GISEL-NEXT: s_nop 0
+; GFX9-GISEL-NEXT: image_sample v[0:3], v[8:9], s[8:15], s[16:19] dmask:0xf
+; GFX9-GISEL-NEXT: image_sample v[4:7], v[10:11], s[8:15], s[16:19] dmask:0xf
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr8
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[6:7]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB20_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-32-SDAG-LABEL: test_waterfall_multi_end_1loop:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-SDAG-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v5, v2
+; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v8, v1
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v9, v0
+; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[3:4]
+; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[5:6]
+; GFX10-32-SDAG-NEXT: v_add_co_u32 v0, vcc_lo, s0, v0
+; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, s1, v1, vcc_lo
+; GFX10-32-SDAG-NEXT: v_add_co_u32 v2, vcc_lo, s2, v2
+; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v3, vcc_lo, s3, v3, vcc_lo
+; GFX10-32-SDAG-NEXT: s_clause 0x1
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[14:17], v[0:1], off offset:16
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[10:13], v[0:1], off
+; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[18:21], v[2:3], off
+; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s2, v9
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s3, v8
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s2, v9
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s3, v8
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s8, v10
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s9, v11
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s10, v12
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s11, v13
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s12, v14
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s13, v15
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s14, v16
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s15, v17
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s16, v18
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s17, v19
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s18, v20
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s19, v21
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v8, 0
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v9, 0.5
+; GFX10-32-SDAG-NEXT: image_sample v[0:3], [v8, v8], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-SDAG-NEXT: image_sample v[4:7], [v9, v9], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr9
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr8
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB20_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-SDAG-NEXT: s_and_b32 exec_lo, exec_lo, s4
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-32-GISEL-LABEL: test_waterfall_multi_end_1loop:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-GISEL-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v4, v3
+; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v8, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v9, v1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v7, s3
+; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[2:3]
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v6, s2
+; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[4:5], 4, v[4:5]
+; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-GISEL-NEXT: v_add_co_u32 v0, vcc_lo, v2, v0
+; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, v3, v1, vcc_lo
+; GFX10-32-GISEL-NEXT: v_add_co_u32 v2, vcc_lo, v6, v4
+; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v3, vcc_lo, v7, v5, vcc_lo
+; GFX10-32-GISEL-NEXT: s_clause 0x1
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[10:13], v[0:1], off
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[14:17], v[0:1], off offset:16
+; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[18:21], v[2:3], off
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v8
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s3, v9
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s2, v8
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s3, v9
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s8, v10
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s9, v11
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s10, v12
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s11, v13
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s12, v14
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s13, v15
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s14, v16
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s15, v17
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s16, v18
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s17, v19
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s18, v20
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s19, v21
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v8, 0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v9, 0.5
+; GFX10-32-GISEL-NEXT: image_sample v[0:3], [v8, v8], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-GISEL-NEXT: image_sample v[4:7], [v9, v9], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr8
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB20_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-GISEL-NEXT: s_and_b32 exec_lo, exec_lo, s4
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-64-SDAG-LABEL: test_waterfall_multi_end_1loop:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v5, v2
+; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v8, v1
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v9, v0
+; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[3:4]
+; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[5:6]
+; GFX10-64-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v1, vcc, s1, v1, vcc
+; GFX10-64-SDAG-NEXT: v_add_co_u32 v2, vcc, s2, v2
+; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v3, vcc, s3, v3, vcc
+; GFX10-64-SDAG-NEXT: s_clause 0x1
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[14:17], v[0:1], off offset:16
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[10:13], v[0:1], off
+; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[18:21], v[2:3], off
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s6, v9
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s7, v8
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v9
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s7, v8
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s8, v10
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s9, v11
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s10, v12
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s11, v13
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s12, v14
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s13, v15
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s14, v16
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s15, v17
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s16, v18
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s17, v19
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s18, v20
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s19, v21
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v8, 0
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v9, 0.5
+; GFX10-64-SDAG-NEXT: image_sample v[0:3], [v8, v8], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-SDAG-NEXT: image_sample v[4:7], [v9, v9], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr9
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr8
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB20_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-64-GISEL-LABEL: test_waterfall_multi_end_1loop:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v4, v3
+; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v8, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v9, v1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v7, s3
+; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[2:3]
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v6, s2
+; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[4:5], 4, v[4:5]
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v1, vcc, v3, v1, vcc
+; GFX10-64-GISEL-NEXT: v_add_co_u32 v2, vcc, v6, v4
+; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v3, vcc, v7, v5, vcc
+; GFX10-64-GISEL-NEXT: s_clause 0x1
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[10:13], v[0:1], off
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[14:17], v[0:1], off offset:16
+; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[18:21], v[2:3], off
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s6, v8
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s7, v9
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s6, v8
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s7, v9
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s8, v10
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s9, v11
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s10, v12
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s11, v13
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s12, v14
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s13, v15
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s14, v16
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s15, v17
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s16, v18
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s17, v19
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s18, v20
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s19, v21
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v8, 0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v9, 0.5
+; GFX10-64-GISEL-NEXT: image_sample v[0:3], [v8, v8], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-GISEL-NEXT: image_sample v[4:7], [v9, v9], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr8
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB20_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX1150-SDAG-LABEL: test_waterfall_multi_end_1loop:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v5, v2
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v8, v1
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v9, v0
+; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_4) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[2:3], 4, v[3:4]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[0:1], 5, v[5:6]
+; GFX1150-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_4)
+; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s1, v1, vcc
+; GFX1150-SDAG-NEXT: v_add_co_u32 v2, vcc, s2, v2
+; GFX1150-SDAG-NEXT: s_waitcnt_depctr depctr_va_vcc(0)
+; GFX1150-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v3, null, s3, v3, vcc
+; GFX1150-SDAG-NEXT: s_clause 0x1
+; GFX1150-SDAG-NEXT: global_load_b128 v[14:17], v[0:1], off offset:16
+; GFX1150-SDAG-NEXT: global_load_b128 v[10:13], v[0:1], off
+; GFX1150-SDAG-NEXT: global_load_b128 v[18:21], v[2:3], off
+; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-SDAG-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s6, v9
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s7, v8
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v9
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s7, v8
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s8, v10
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s9, v11
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s10, v12
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s11, v13
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s12, v14
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s13, v15
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s14, v16
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s15, v17
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s16, v18
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s17, v19
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s18, v20
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s19, v21
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v4, 0.5
+; GFX1150-SDAG-NEXT: s_clause 0x1
+; GFX1150-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-SDAG-NEXT: image_sample v[4:7], [v4, v4], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr9
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr8
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB20_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX1150-GISEL-LABEL: test_waterfall_multi_end_1loop:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, v3
+; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v8, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v9, v1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v7, s3
+; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[0:1], 5, v[2:3]
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v6, s2
+; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[4:5], 4, v[4:5]
+; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_3)
+; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX1150-GISEL-NEXT: v_add_co_u32 v2, vcc, v6, v4
+; GFX1150-GISEL-NEXT: s_waitcnt_depctr depctr_va_vcc(0)
+; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v3, null, v7, v5, vcc
+; GFX1150-GISEL-NEXT: s_clause 0x1
+; GFX1150-GISEL-NEXT: global_load_b128 v[10:13], v[0:1], off
+; GFX1150-GISEL-NEXT: global_load_b128 v[14:17], v[0:1], off offset:16
+; GFX1150-GISEL-NEXT: global_load_b128 v[18:21], v[2:3], off
+; GFX1150-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s6, v8
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s7, v9
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s6, v8
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s7, v9
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(2)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s8, v10
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s9, v11
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s10, v12
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s11, v13
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s12, v14
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s13, v15
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s14, v16
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s15, v17
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s16, v18
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s17, v19
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s18, v20
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s19, v21
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, 0.5
+; GFX1150-GISEL-NEXT: s_clause 0x1
+; GFX1150-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-GISEL-NEXT: image_sample v[4:7], [v4, v4], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr8
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB20_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX12-SDAG-LABEL: test_waterfall_multi_end_1loop:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v5, v2
+; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v4, 31, v3
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v8, v1
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v9, v0
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_4) | instskip(NEXT) | instid1(VALU_DEP_4)
+; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v6, 31, v5
+; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[2:3], 4, v[3:4]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[0:1], 5, v[5:6]
+; GFX12-SDAG-NEXT: v_add_co_u32 v0, vcc, s0, v0
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_4)
+; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v1, null, s1, v1, vcc
+; GFX12-SDAG-NEXT: v_add_co_u32 v2, vcc, s2, v2
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_vcc(0)
+; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v3, null, s3, v3, vcc
+; GFX12-SDAG-NEXT: s_clause 0x1
+; GFX12-SDAG-NEXT: global_load_b128 v[14:17], v[0:1], off offset:16
+; GFX12-SDAG-NEXT: global_load_b128 v[10:13], v[0:1], off
+; GFX12-SDAG-NEXT: global_load_b128 v[18:21], v[2:3], off
+; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-SDAG-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s6, v9
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s7, v8
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v9
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s7, v8
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s8, v10
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s9, v11
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s10, v12
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s11, v13
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s12, v14
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s13, v15
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s14, v16
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s15, v17
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x0
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s16, v18
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s17, v19
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s18, v20
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s19, v21
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x1
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v4, 0.5
+; GFX12-SDAG-NEXT: s_clause 0x1
+; GFX12-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-SDAG-NEXT: image_sample v[4:7], [v4, v4], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr9
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr8
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB20_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX12-GISEL-LABEL: test_waterfall_multi_end_1loop:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, v3
+; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v3, 31, v2
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v8, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v9, v1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v7, s3
+; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v5, 31, v4
+; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[0:1], 5, v[2:3]
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v3, s1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v2, s0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v6, s2
+; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[4:5], 4, v[4:5]
+; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-GISEL-NEXT: v_add_co_u32 v0, vcc, v2, v0
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_3)
+; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
+; GFX12-GISEL-NEXT: v_add_co_u32 v2, vcc, v6, v4
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_vcc(0)
+; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v3, null, v7, v5, vcc
+; GFX12-GISEL-NEXT: s_clause 0x1
+; GFX12-GISEL-NEXT: global_load_b128 v[10:13], v[0:1], off
+; GFX12-GISEL-NEXT: global_load_b128 v[14:17], v[0:1], off offset:16
+; GFX12-GISEL-NEXT: global_load_b128 v[18:21], v[2:3], off
+; GFX12-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s6, v8
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s7, v9
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s6, v8
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s7, v9
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x2
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s8, v10
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s9, v11
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s10, v12
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s11, v13
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s12, v14
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s13, v15
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s14, v16
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s15, v17
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s16, v18
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s17, v19
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s18, v20
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s19, v21
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, 0.5
+; GFX12-GISEL-NEXT: s_clause 0x1
+; GFX12-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-GISEL-NEXT: image_sample v[4:7], [v4, v4], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr8
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB20_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: ; return to shader part epilog
+ <8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %s_in,
+ i32 %idx1, i32 %idx2, i32 %s_idx, i32 %s_idx2) #1 {
+ %rptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
+ %sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
+ %srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
+ %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
+ %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok, i32 %idx2)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok1, <8 x i32> %rsrc)
+ %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok1, <4 x i32> %srsrc)
+ %r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
+ %r2 = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.500000e+00, float 0.500000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
+ %r3 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r2)
+
+ %insert = insertvalue { <4 x float>, <4 x float> } undef, <4 x float> %r1, 0
+ %insert1 = insertvalue { <4 x float>, <4 x float> } %insert, <4 x float> %r3, 1
+ ret {<4 x float> , <4 x float>} %insert1
+}
+
+define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_load_inside(
+; VI-SDAG-LABEL: test_waterfall_multi_end_1loop_rsrc_load_inside:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; VI-SDAG-NEXT: s_wqm_b64 exec, exec
+; VI-SDAG-NEXT: v_mov_b32_e32 v8, v3
+; VI-SDAG-NEXT: v_mov_b32_e32 v9, v2
+; VI-SDAG-NEXT: v_mov_b32_e32 v10, v1
+; VI-SDAG-NEXT: v_mov_b32_e32 v11, v0
+; VI-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; VI-SDAG-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v11
+; VI-SDAG-NEXT: v_readfirstlane_b32 s10, v10
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v11
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[10:11], s10, v10
+; VI-SDAG-NEXT: s_and_b64 s[8:9], s[8:9], s[10:11]
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[20:21], s[8:9]
+; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v9
+; VI-SDAG-NEXT: v_readfirstlane_b32 s10, v8
+; VI-SDAG-NEXT: s_ashr_i32 s9, s8, 31
+; VI-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; VI-SDAG-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; VI-SDAG-NEXT: s_add_u32 s8, s0, s8
+; VI-SDAG-NEXT: s_addc_u32 s9, s1, s9
+; VI-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 4
+; VI-SDAG-NEXT: s_add_u32 s16, s2, s10
+; VI-SDAG-NEXT: s_addc_u32 s17, s3, s11
+; VI-SDAG-NEXT: s_load_dwordx8 s[8:15], s[8:9], 0x0
+; VI-SDAG-NEXT: s_load_dwordx4 s[16:19], s[16:17], 0x0
+; VI-SDAG-NEXT: s_waitcnt vmcnt(1)
+; VI-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, 0.5
+; VI-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; VI-SDAG-NEXT: v_mov_b32_e32 v5, v4
+; VI-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; VI-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; VI-SDAG-NEXT: image_sample v[4:7], v[4:5], s[8:15], s[16:19] dmask:0xf
+; VI-SDAG-NEXT: ; implicit-def: $vgpr11
+; VI-SDAG-NEXT: ; implicit-def: $vgpr10
+; VI-SDAG-NEXT: ; implicit-def: $vgpr9
+; VI-SDAG-NEXT: ; implicit-def: $vgpr8
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[20:21]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB21_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[6:7]
+; VI-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: ; return to shader part epilog
+;
+; VI-GISEL-LABEL: test_waterfall_multi_end_1loop_rsrc_load_inside:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; VI-GISEL-NEXT: s_wqm_b64 exec, exec
+; VI-GISEL-NEXT: s_mov_b32 s6, 0
+; VI-GISEL-NEXT: v_mov_b32_e32 v8, v0
+; VI-GISEL-NEXT: v_mov_b32_e32 v11, v1
+; VI-GISEL-NEXT: v_mov_b32_e32 v9, v2
+; VI-GISEL-NEXT: v_mov_b32_e32 v10, v3
+; VI-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; VI-GISEL-NEXT: s_mov_b32 s24, 0.5
+; VI-GISEL-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s7, v8
+; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v11
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s7, v8
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[12:13], s12, v11
+; VI-GISEL-NEXT: s_and_b64 s[10:11], s[10:11], s[12:13]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[10:11], s[10:11]
+; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v9
+; VI-GISEL-NEXT: v_readfirstlane_b32 s14, v10
+; VI-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; VI-GISEL-NEXT: s_ashr_i32 s15, s14, 31
+; VI-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; VI-GISEL-NEXT: s_add_u32 s12, s0, s12
+; VI-GISEL-NEXT: s_addc_u32 s13, s1, s13
+; VI-GISEL-NEXT: s_lshl_b64 s[14:15], s[14:15], 4
+; VI-GISEL-NEXT: s_add_u32 s20, s2, s14
+; VI-GISEL-NEXT: s_addc_u32 s21, s3, s15
+; VI-GISEL-NEXT: s_load_dwordx8 s[12:19], s[12:13], 0x0
+; VI-GISEL-NEXT: s_load_dwordx4 s[20:23], s[20:21], 0x0
+; VI-GISEL-NEXT: s_mov_b32 s7, s6
+; VI-GISEL-NEXT: s_waitcnt vmcnt(1)
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s6
+; VI-GISEL-NEXT: s_mov_b32 s25, s24
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, s24
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s7
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, s25
+; VI-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[12:19], s[20:23] dmask:0xf
+; VI-GISEL-NEXT: image_sample v[4:7], v[4:5], s[12:19], s[20:23] dmask:0xf
+; VI-GISEL-NEXT: ; implicit-def: $vgpr8
+; VI-GISEL-NEXT: ; implicit-def: $vgpr11
+; VI-GISEL-NEXT: ; implicit-def: $vgpr9
+; VI-GISEL-NEXT: ; implicit-def: $vgpr10
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[10:11]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB21_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[8:9]
+; VI-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX9-SDAG-LABEL: test_waterfall_multi_end_1loop_rsrc_load_inside:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v8, v3
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v9, v2
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v10, v1
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v11, v0
+; GFX9-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX9-SDAG-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v11
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s10, v10
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v11
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[10:11], s10, v10
+; GFX9-SDAG-NEXT: s_and_b64 s[8:9], s[8:9], s[10:11]
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[20:21], s[8:9]
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v9
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s10, v8
+; GFX9-SDAG-NEXT: s_ashr_i32 s9, s8, 31
+; GFX9-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; GFX9-SDAG-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; GFX9-SDAG-NEXT: s_add_u32 s22, s0, s8
+; GFX9-SDAG-NEXT: s_addc_u32 s23, s1, s9
+; GFX9-SDAG-NEXT: s_lshl_b64 s[8:9], s[10:11], 4
+; GFX9-SDAG-NEXT: s_add_u32 s24, s2, s8
+; GFX9-SDAG-NEXT: s_addc_u32 s25, s3, s9
+; GFX9-SDAG-NEXT: s_load_dwordx8 s[8:15], s[22:23], 0x0
+; GFX9-SDAG-NEXT: s_load_dwordx4 s[16:19], s[24:25], 0x0
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v8, 0
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v9, v8
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v10, 0.5
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v11, v10
+; GFX9-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-SDAG-NEXT: image_sample v[0:3], v[8:9], s[8:15], s[16:19] dmask:0xf
+; GFX9-SDAG-NEXT: image_sample v[4:7], v[10:11], s[8:15], s[16:19] dmask:0xf
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr11
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr10
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr9
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr8
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[20:21]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB21_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[6:7]
+; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX9-GISEL-LABEL: test_waterfall_multi_end_1loop_rsrc_load_inside:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX9-GISEL-NEXT: s_mov_b32 s6, 0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v8, v0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v11, v1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, v2
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v10, v3
+; GFX9-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX9-GISEL-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s7, v8
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v11
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s7, v8
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[12:13], s12, v11
+; GFX9-GISEL-NEXT: s_and_b64 s[10:11], s[10:11], s[12:13]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[10:11], s[10:11]
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v9
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s14, v10
+; GFX9-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; GFX9-GISEL-NEXT: s_ashr_i32 s15, s14, 31
+; GFX9-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
+; GFX9-GISEL-NEXT: s_add_u32 s24, s0, s12
+; GFX9-GISEL-NEXT: s_addc_u32 s25, s1, s13
+; GFX9-GISEL-NEXT: s_lshl_b64 s[12:13], s[14:15], 4
+; GFX9-GISEL-NEXT: s_add_u32 s26, s2, s12
+; GFX9-GISEL-NEXT: s_addc_u32 s27, s3, s13
+; GFX9-GISEL-NEXT: s_load_dwordx8 s[12:19], s[24:25], 0x0
+; GFX9-GISEL-NEXT: s_load_dwordx4 s[20:23], s[26:27], 0x0
+; GFX9-GISEL-NEXT: s_mov_b32 s7, s6
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, s7
+; GFX9-GISEL-NEXT: s_mov_b32 s24, 0.5
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v8, s6
+; GFX9-GISEL-NEXT: s_mov_b32 s25, s24
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v10, s24
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v11, s25
+; GFX9-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-GISEL-NEXT: image_sample v[0:3], v[8:9], s[12:19], s[20:23] dmask:0xf
+; GFX9-GISEL-NEXT: image_sample v[4:7], v[10:11], s[12:19], s[20:23] dmask:0xf
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr8
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr11
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr10
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[10:11]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB21_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[8:9]
+; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-32-SDAG-LABEL: test_waterfall_multi_end_1loop_rsrc_load_inside:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-SDAG-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v8, v3
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v9, v2
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v10, v1
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v11, v0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s6, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s5, exec_lo
+; GFX10-32-SDAG-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s7, v11
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s8, v10
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s7, v11
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s8, v10
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s8, v9
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s10, v8
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v8, 0
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v9, 0.5
+; GFX10-32-SDAG-NEXT: s_ashr_i32 s9, s8, 31
+; GFX10-32-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; GFX10-32-SDAG-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; GFX10-32-SDAG-NEXT: s_add_u32 s20, s0, s8
+; GFX10-32-SDAG-NEXT: s_addc_u32 s21, s1, s9
+; GFX10-32-SDAG-NEXT: s_lshl_b64 s[8:9], s[10:11], 4
+; GFX10-32-SDAG-NEXT: s_add_u32 s22, s2, s8
+; GFX10-32-SDAG-NEXT: s_addc_u32 s23, s3, s9
+; GFX10-32-SDAG-NEXT: s_load_dwordx8 s[8:15], s[20:21], 0x0
+; GFX10-32-SDAG-NEXT: s_load_dwordx4 s[16:19], s[22:23], 0x0
+; GFX10-32-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-SDAG-NEXT: image_sample v[0:3], [v8, v8], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-SDAG-NEXT: image_sample v[4:7], [v9, v9], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s6, s6
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr11
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr10
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr9
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr8
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB21_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s5
+; GFX10-32-SDAG-NEXT: s_and_b32 exec_lo, exec_lo, s4
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-32-GISEL-LABEL: test_waterfall_multi_end_1loop_rsrc_load_inside:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v8, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v9, v1
+; GFX10-32-GISEL-NEXT: s_mov_b32 s4, exec_lo
+; GFX10-32-GISEL-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v10, v2
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v11, v3
+; GFX10-32-GISEL-NEXT: s_mov_b32 s6, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s5, exec_lo
+; GFX10-32-GISEL-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s7, v8
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s8, v9
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s7, v8
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s8, v9
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s8, v10
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s10, v11
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v8, 0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v9, 0.5
+; GFX10-32-GISEL-NEXT: s_ashr_i32 s9, s8, 31
+; GFX10-32-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; GFX10-32-GISEL-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; GFX10-32-GISEL-NEXT: s_add_u32 s20, s0, s8
+; GFX10-32-GISEL-NEXT: s_addc_u32 s21, s1, s9
+; GFX10-32-GISEL-NEXT: s_lshl_b64 s[8:9], s[10:11], 4
+; GFX10-32-GISEL-NEXT: s_add_u32 s22, s2, s8
+; GFX10-32-GISEL-NEXT: s_addc_u32 s23, s3, s9
+; GFX10-32-GISEL-NEXT: s_load_dwordx8 s[8:15], s[20:21], 0x0
+; GFX10-32-GISEL-NEXT: s_load_dwordx4 s[16:19], s[22:23], 0x0
+; GFX10-32-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-GISEL-NEXT: image_sample v[0:3], [v8, v8], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-GISEL-NEXT: image_sample v[4:7], [v9, v9], s[8:15], s[16:19] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s6, s6
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr8
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr10
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr11
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB21_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s5
+; GFX10-32-GISEL-NEXT: s_and_b32 exec_lo, exec_lo, s4
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-64-SDAG-LABEL: test_waterfall_multi_end_1loop_rsrc_load_inside:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v8, v3
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v9, v2
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v10, v1
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v11, v0
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-SDAG-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s10, v11
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s11, v10
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v11
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s11, v10
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s10, v9
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s12, v8
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v8, 0
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v9, 0.5
+; GFX10-64-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; GFX10-64-SDAG-NEXT: s_ashr_i32 s13, s12, 31
+; GFX10-64-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX10-64-SDAG-NEXT: s_add_u32 s10, s0, s10
+; GFX10-64-SDAG-NEXT: s_addc_u32 s11, s1, s11
+; GFX10-64-SDAG-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX10-64-SDAG-NEXT: s_add_u32 s24, s2, s12
+; GFX10-64-SDAG-NEXT: s_addc_u32 s25, s3, s13
+; GFX10-64-SDAG-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; GFX10-64-SDAG-NEXT: s_load_dwordx4 s[20:23], s[24:25], 0x0
+; GFX10-64-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-SDAG-NEXT: image_sample v[0:3], [v8, v8], s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-SDAG-NEXT: image_sample v[4:7], [v9, v9], s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[8:9], s[8:9]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr11
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr10
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr9
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr8
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB21_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[6:7]
+; GFX10-64-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-64-GISEL-LABEL: test_waterfall_multi_end_1loop_rsrc_load_inside:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v8, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v9, v1
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-64-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v10, v2
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v11, v3
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-GISEL-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s10, v8
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s11, v9
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s10, v8
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s11, v9
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s10, v10
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s12, v11
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v8, 0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v9, 0.5
+; GFX10-64-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; GFX10-64-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; GFX10-64-GISEL-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX10-64-GISEL-NEXT: s_add_u32 s10, s0, s10
+; GFX10-64-GISEL-NEXT: s_addc_u32 s11, s1, s11
+; GFX10-64-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX10-64-GISEL-NEXT: s_add_u32 s24, s2, s12
+; GFX10-64-GISEL-NEXT: s_addc_u32 s25, s3, s13
+; GFX10-64-GISEL-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
+; GFX10-64-GISEL-NEXT: s_load_dwordx4 s[20:23], s[24:25], 0x0
+; GFX10-64-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-GISEL-NEXT: image_sample v[0:3], [v8, v8], s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-GISEL-NEXT: image_sample v[4:7], [v9, v9], s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[8:9], s[8:9]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr8
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr10
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr11
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB21_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[6:7]
+; GFX10-64-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX1150-SDAG-LABEL: test_waterfall_multi_end_1loop_rsrc_load_inside:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v8, v3
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v9, v2
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v10, v1
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v11, v0
+; GFX1150-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX1150-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-SDAG-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s10, v11
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s11, v10
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v11
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2)
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s11, v10
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s10, v9
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s12, v8
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(1)
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v4, 0.5
+; GFX1150-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; GFX1150-SDAG-NEXT: s_ashr_i32 s13, s12, 31
+; GFX1150-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_2) | instid1(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_add_u32 s10, s0, s10
+; GFX1150-SDAG-NEXT: s_addc_u32 s11, s1, s11
+; GFX1150-SDAG-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX1150-SDAG-NEXT: s_add_u32 s20, s2, s12
+; GFX1150-SDAG-NEXT: s_addc_u32 s21, s3, s13
+; GFX1150-SDAG-NEXT: s_load_b256 s[12:19], s[10:11], 0x0
+; GFX1150-SDAG-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX1150-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1150-SDAG-NEXT: s_clause 0x1
+; GFX1150-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-SDAG-NEXT: image_sample v[4:7], [v4, v4], s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr11
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr10
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr9
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr8
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB21_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[6:7]
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX1150-GISEL-LABEL: test_waterfall_multi_end_1loop_rsrc_load_inside:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v8, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v9, v1
+; GFX1150-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX1150-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v10, v2
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v11, v3
+; GFX1150-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-GISEL-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s10, v8
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s11, v9
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s10, v8
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s11, v9
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s10, v10
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s12, v11
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(1)
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, 0.5
+; GFX1150-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; GFX1150-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; GFX1150-GISEL-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_2) | instid1(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_add_u32 s10, s0, s10
+; GFX1150-GISEL-NEXT: s_addc_u32 s11, s1, s11
+; GFX1150-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX1150-GISEL-NEXT: s_add_u32 s20, s2, s12
+; GFX1150-GISEL-NEXT: s_addc_u32 s21, s3, s13
+; GFX1150-GISEL-NEXT: s_load_b256 s[12:19], s[10:11], 0x0
+; GFX1150-GISEL-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX1150-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1150-GISEL-NEXT: s_clause 0x1
+; GFX1150-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-GISEL-NEXT: image_sample v[4:7], [v4, v4], s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr8
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr10
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr11
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB21_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[6:7]
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX12-SDAG-LABEL: test_waterfall_multi_end_1loop_rsrc_load_inside:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-SDAG-NEXT: s_wqm_b64 exec, exec
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v8, v3
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v9, v2
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v10, v1
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v11, v0
+; GFX12-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-SDAG-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-SDAG-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s10, v11
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s11, v10
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v11
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s11, v10
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s10, v9
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s12, v8
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x1
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v4, 0.5
+; GFX12-SDAG-NEXT: s_ashr_i32 s11, s10, 31
+; GFX12-SDAG-NEXT: s_ashr_i32 s13, s12, 31
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX12-SDAG-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_add_nc_u64 s[10:11], s[0:1], s[10:11]
+; GFX12-SDAG-NEXT: s_add_nc_u64 s[20:21], s[2:3], s[12:13]
+; GFX12-SDAG-NEXT: s_load_b256 s[12:19], s[10:11], 0x0
+; GFX12-SDAG-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_clause 0x1
+; GFX12-SDAG-NEXT: image_sample v[0:3], [v0, v0], s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-SDAG-NEXT: image_sample v[4:7], [v4, v4], s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr11
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr10
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr9
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr8
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB21_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[6:7]
+; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX12-GISEL-LABEL: test_waterfall_multi_end_1loop_rsrc_load_inside:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v8, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v9, v1
+; GFX12-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX12-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v10, v2
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v11, v3
+; GFX12-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-GISEL-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s10, v8
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s11, v9
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s10, v8
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s11, v9
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s10, v10
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s12, v11
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, 0.5
+; GFX12-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; GFX12-GISEL-NEXT: s_ashr_i32 s13, s12, 31
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_lshl_b64 s[10:11], s[10:11], 5
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_add_co_u32 s10, s0, s10
+; GFX12-GISEL-NEXT: s_add_co_ci_u32 s11, s1, s11
+; GFX12-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 4
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_add_co_u32 s20, s2, s12
+; GFX12-GISEL-NEXT: s_add_co_ci_u32 s21, s3, s13
+; GFX12-GISEL-NEXT: s_load_b256 s[12:19], s[10:11], 0x0
+; GFX12-GISEL-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_clause 0x1
+; GFX12-GISEL-NEXT: image_sample v[0:3], [v0, v0], s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-GISEL-NEXT: image_sample v[4:7], [v4, v4], s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_2D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr8
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr10
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr11
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB21_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[6:7]
+; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: ; return to shader part epilog
+ <8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %s_in,
+ i32 %idx1, i32 %idx2, i32 %s_idx, i32 %s_idx2) #1 {
+ %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
+ %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok, i32 %idx2)
+ %widx0 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %tok1, i32 %s_idx)
+ %widx1 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %tok1, i32 %s_idx2)
+ %widx0a = sext i32 %widx0 to i64
+ %widx1a = sext i32 %widx1 to i64
+ %rptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i64 %widx0a
+ %sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i64 %widx1a
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 32
+ %srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
+
+ %r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %rsrc, <4 x i32> %srsrc, i1 false, i32 0, i32 0)
+ %r2 = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.500000e+00, float 0.500000e+00, <8 x i32> %rsrc, <4 x i32> %srsrc, i1 false, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
+ %r3 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r2)
+
+ %insert = insertvalue { <4 x float>, <4 x float> } undef, <4 x float> %r1, 0
+ %insert1 = insertvalue { <4 x float>, <4 x float> } %insert, <4 x float> %r3, 1
+ ret {<4 x float> , <4 x float>} %insert1
+}
+
+define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
+; VI-SDAG-LABEL: test_waterfall_multi_end_struct:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: v_mov_b32_e32 v9, v4
+; VI-SDAG-NEXT: v_mov_b32_e32 v8, v3
+; VI-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; VI-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; VI-SDAG-NEXT: v_mov_b32_e32 v10, v0
+; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; VI-SDAG-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v10
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v10
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; VI-SDAG-NEXT: v_readfirstlane_b32 s4, v6
+; VI-SDAG-NEXT: v_readfirstlane_b32 s5, v7
+; VI-SDAG-NEXT: v_readfirstlane_b32 s6, v8
+; VI-SDAG-NEXT: v_readfirstlane_b32 s7, v9
+; VI-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; VI-SDAG-NEXT: v_mov_b32_e32 v2, v0
+; VI-SDAG-NEXT: v_mov_b32_e32 v3, v0
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; VI-SDAG-NEXT: ; implicit-def: $vgpr10
+; VI-SDAG-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
+; VI-SDAG-NEXT: s_nop 0
+; VI-SDAG-NEXT: buffer_load_format_xyzw v[0:4], v5, s[4:7], 0 idxen tfe
+; VI-SDAG-NEXT: ; implicit-def: $vgpr5
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB22_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: ; return to shader part epilog
+;
+; VI-GISEL-LABEL: test_waterfall_multi_end_struct:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: v_mov_b32_e32 v10, v0
+; VI-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; VI-GISEL-NEXT: v_mov_b32_e32 v7, v2
+; VI-GISEL-NEXT: v_mov_b32_e32 v8, v3
+; VI-GISEL-NEXT: v_mov_b32_e32 v9, v4
+; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; VI-GISEL-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v10
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v10
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; VI-GISEL-NEXT: v_readfirstlane_b32 s4, v6
+; VI-GISEL-NEXT: v_readfirstlane_b32 s5, v7
+; VI-GISEL-NEXT: v_readfirstlane_b32 s6, v8
+; VI-GISEL-NEXT: v_readfirstlane_b32 s7, v9
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, v0
+; VI-GISEL-NEXT: v_mov_b32_e32 v2, v0
+; VI-GISEL-NEXT: v_mov_b32_e32 v3, v0
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; VI-GISEL-NEXT: ; implicit-def: $vgpr10
+; VI-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
+; VI-GISEL-NEXT: s_nop 0
+; VI-GISEL-NEXT: buffer_load_format_xyzw v[0:4], v5, s[4:7], 0 idxen tfe
+; VI-GISEL-NEXT: ; implicit-def: $vgpr5
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB22_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX9-SDAG-LABEL: test_waterfall_multi_end_struct:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v9, v4
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v8, v3
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v10, v0
+; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX9-SDAG-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v10
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v10
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s4, v6
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s5, v7
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s6, v8
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s7, v9
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v2, v0
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v3, v0
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr10
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
+; GFX9-SDAG-NEXT: s_nop 0
+; GFX9-SDAG-NEXT: buffer_load_format_xyzw v[0:4], v5, s[4:7], 0 idxen tfe
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB22_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX9-GISEL-LABEL: test_waterfall_multi_end_struct:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v10, v0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v7, v2
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v8, v3
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, v4
+; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX9-GISEL-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v10
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v10
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s4, v6
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s5, v7
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s6, v8
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s7, v9
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, v0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v2, v0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v3, v0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr10
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
+; GFX9-GISEL-NEXT: s_nop 0
+; GFX9-GISEL-NEXT: buffer_load_format_xyzw v[0:4], v5, s[4:7], 0 idxen tfe
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB22_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-32-SDAG-LABEL: test_waterfall_multi_end_struct:
+; GFX10-32-SDAG: ; %bb.0:
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v9, v4
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v8, v3
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v10, v0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-SDAG-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s2, v10
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s2, v10
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s4, v6
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s5, v7
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s6, v8
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s7, v9
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v2, v0
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v3, v0
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-32-SDAG-NEXT: buffer_load_format_xyzw v[0:4], v5, s[4:7], 0 idxen tfe
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr10
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB22_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-32-GISEL-LABEL: test_waterfall_multi_end_struct:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v10, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v7, v2
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v8, v3
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v9, v4
+; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-GISEL-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v10
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s2, v10
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s4, v6
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s5, v7
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s6, v8
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s7, v9
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v1, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v2, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v3, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-32-GISEL-NEXT: buffer_load_format_xyzw v[0:4], v5, s[4:7], 0 idxen tfe
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr10
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB22_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-64-SDAG-LABEL: test_waterfall_multi_end_struct:
+; GFX10-64-SDAG: ; %bb.0:
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v9, v4
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v8, v3
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v10, v0
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-SDAG-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s4, v10
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s4, v10
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s4, v6
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s5, v7
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s6, v8
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s7, v9
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v2, v0
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v3, v0
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-64-SDAG-NEXT: buffer_load_format_xyzw v[0:4], v5, s[4:7], 0 idxen tfe
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr10
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB22_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-64-GISEL-LABEL: test_waterfall_multi_end_struct:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v10, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v7, v2
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v8, v3
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v9, v4
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-GISEL-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s4, v10
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v10
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s4, v6
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s5, v7
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s6, v8
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s7, v9
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v1, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v2, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v3, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-64-GISEL-NEXT: buffer_load_format_xyzw v[0:4], v5, s[4:7], 0 idxen tfe
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr10
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB22_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX1150-SDAG-LABEL: test_waterfall_multi_end_struct:
+; GFX1150-SDAG: ; %bb.0:
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v9, v4
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v8, v3
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v10, v0
+; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-SDAG-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s4, v10
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s4, v10
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s4, v6
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s5, v7
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s6, v8
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s7, v9
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v2, v0
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v3, v0
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX1150-SDAG-NEXT: buffer_load_format_xyzw v[0:4], v5, s[4:7], 0 idxen tfe
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr10
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB22_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX1150-GISEL-LABEL: test_waterfall_multi_end_struct:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v10, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v7, v2
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v8, v3
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v9, v4
+; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-GISEL-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s4, v10
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v10
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s4, v6
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s5, v7
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s6, v8
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s7, v9
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v1, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v2, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v3, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX1150-GISEL-NEXT: buffer_load_format_xyzw v[0:4], v5, s[4:7], 0 idxen tfe
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr10
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB22_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX12-SDAG-LABEL: test_waterfall_multi_end_struct:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v9, v4
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v8, v3
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v7, v2
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v6, v1
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v10, v0
+; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-SDAG-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s4, v10
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s4, v10
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s4, v6
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s5, v7
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s6, v8
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s7, v9
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v2, v0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v3, v0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-SDAG-NEXT: buffer_load_format_xyzw v[0:4], v5, s[4:7], null idxen tfe
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr10
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr5
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB22_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x0
+; GFX12-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX12-GISEL-LABEL: test_waterfall_multi_end_struct:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v10, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v7, v2
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v8, v3
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v9, v4
+; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-GISEL-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s4, v10
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v10
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s4, v6
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s5, v7
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s6, v8
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s7, v9
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v1, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v2, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v3, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-GISEL-NEXT: buffer_load_format_xyzw v[0:4], v5, s[4:7], null idxen tfe
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr10
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB22_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
+; GFX12-GISEL-NEXT: ; return to shader part epilog
+ i32 %idx, <4 x i32> %s_idx, i32 %v_inp) #1 {
+ %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx)
+ %widx0 = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok, <4 x i32> %s_idx)
+ %val = call { <4 x float>, i32 } @llvm.amdgcn.struct.buffer.load.format.sl_v4f32i32s(<4 x i32> %widx0, i32 %v_inp, i32 0, i32 0, i32 0)
+ %payl = extractvalue { <4 x float>, i32 } %val, 0
+ %tfe = extractvalue { <4 x float>, i32 } %val, 1
+ %r0 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok, <4 x float> %payl)
+ %r1 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %tok, i32 %tfe)
+ %r1.float = bitcast i32 %r1 to float
+
+ %insert = insertvalue { <4 x float>, float } undef, <4 x float> %r0, 0
+ %insert1 = insertvalue { <4 x float>, float } %insert, float %r1.float, 1
+ ret {<4 x float> , float} %insert1
+}
+
+define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
+; VI-SDAG-LABEL: test_waterfall_multi_end_struct_uniform:
+; VI-SDAG: ; %bb.0:
+; VI-SDAG-NEXT: v_readfirstlane_b32 s0, v0
+; VI-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; VI-SDAG-NEXT: v_mov_b32_e32 v5, v4
+; VI-SDAG-NEXT: v_readfirstlane_b32 s1, v1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v2
+; VI-SDAG-NEXT: v_readfirstlane_b32 s3, v3
+; VI-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; VI-SDAG-NEXT: v_mov_b32_e32 v2, v0
+; VI-SDAG-NEXT: v_mov_b32_e32 v3, v0
+; VI-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; VI-SDAG-NEXT: s_nop 0
+; VI-SDAG-NEXT: buffer_load_format_xyzw v[0:4], v5, s[0:3], 0 idxen tfe
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
+; VI-SDAG-NEXT: ; return to shader part epilog
+;
+; VI-GISEL-LABEL: test_waterfall_multi_end_struct_uniform:
+; VI-GISEL: ; %bb.0:
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; VI-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; VI-GISEL-NEXT: v_mov_b32_e32 v7, v2
+; VI-GISEL-NEXT: v_mov_b32_e32 v8, v3
+; VI-GISEL-NEXT: v_mov_b32_e32 v9, v4
+; VI-GISEL-NEXT: v_mov_b32_e32 v10, s0
+; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; VI-GISEL-NEXT: .LBB23_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v10
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v10
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; VI-GISEL-NEXT: v_readfirstlane_b32 s4, v5
+; VI-GISEL-NEXT: v_readfirstlane_b32 s5, v6
+; VI-GISEL-NEXT: v_readfirstlane_b32 s6, v7
+; VI-GISEL-NEXT: v_readfirstlane_b32 s7, v8
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, v0
+; VI-GISEL-NEXT: v_mov_b32_e32 v2, v0
+; VI-GISEL-NEXT: v_mov_b32_e32 v3, v0
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; VI-GISEL-NEXT: ; implicit-def: $vgpr10
+; VI-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; VI-GISEL-NEXT: s_nop 0
+; VI-GISEL-NEXT: buffer_load_format_xyzw v[0:4], v9, s[4:7], 0 idxen tfe
+; VI-GISEL-NEXT: ; implicit-def: $vgpr9
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB23_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
+; VI-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX9-SDAG-LABEL: test_waterfall_multi_end_struct_uniform:
+; GFX9-SDAG: ; %bb.0:
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s0, v0
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v5, v4
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s1, v1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v2
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s3, v3
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v2, v0
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v3, v0
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-SDAG-NEXT: s_nop 0
+; GFX9-SDAG-NEXT: buffer_load_format_xyzw v[0:4], v5, s[0:3], 0 idxen tfe
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX9-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX9-GISEL-LABEL: test_waterfall_multi_end_struct_uniform:
+; GFX9-GISEL: ; %bb.0:
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v7, v2
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v8, v3
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, v4
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v10, s0
+; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX9-GISEL-NEXT: .LBB23_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v10
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v10
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s4, v5
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s5, v6
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s6, v7
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s7, v8
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, v0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v2, v0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v3, v0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr10
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; GFX9-GISEL-NEXT: s_nop 0
+; GFX9-GISEL-NEXT: buffer_load_format_xyzw v[0:4], v9, s[4:7], 0 idxen tfe
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB23_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX9-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10GFX11-SDAG-LABEL: test_waterfall_multi_end_struct_uniform:
+; GFX10GFX11-SDAG: ; %bb.0:
+; GFX10GFX11-SDAG-NEXT: v_mov_b32_e32 v5, v4
+; GFX10GFX11-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX10GFX11-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX10GFX11-SDAG-NEXT: v_readfirstlane_b32 s1, v1
+; GFX10GFX11-SDAG-NEXT: v_readfirstlane_b32 s2, v2
+; GFX10GFX11-SDAG-NEXT: v_readfirstlane_b32 s3, v3
+; GFX10GFX11-SDAG-NEXT: v_readfirstlane_b32 s0, v4
+; GFX10GFX11-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; GFX10GFX11-SDAG-NEXT: v_mov_b32_e32 v2, v0
+; GFX10GFX11-SDAG-NEXT: v_mov_b32_e32 v3, v0
+; GFX10GFX11-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX10GFX11-SDAG-NEXT: buffer_load_format_xyzw v[0:4], v5, s[0:3], 0 idxen tfe
+; GFX10GFX11-SDAG-NEXT: s_waitcnt vmcnt(0)
+; GFX10GFX11-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX10-32-GISEL-LABEL: test_waterfall_multi_end_struct_uniform:
+; GFX10-32-GISEL: ; %bb.0:
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v7, v2
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v8, v3
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v9, v4
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v10, s0
+; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
+; GFX10-32-GISEL-NEXT: .LBB23_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v10
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s2, v10
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s4, v5
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s5, v6
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s6, v7
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s7, v8
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v1, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v2, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v3, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-32-GISEL-NEXT: buffer_load_format_xyzw v[0:4], v9, s[4:7], 0 idxen tfe
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s1, s1
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr10
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB23_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-32-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX10-64-GISEL-LABEL: test_waterfall_multi_end_struct_uniform:
+; GFX10-64-GISEL: ; %bb.0:
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v7, v2
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v8, v3
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v9, v4
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v10, s0
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX10-64-GISEL-NEXT: .LBB23_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s4, v10
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v10
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s4, v5
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s5, v6
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s6, v7
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s7, v8
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v1, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v2, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v3, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX10-64-GISEL-NEXT: buffer_load_format_xyzw v[0:4], v9, s[4:7], 0 idxen tfe
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr10
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB23_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX10-64-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX1150-GISEL-LABEL: test_waterfall_multi_end_struct_uniform:
+; GFX1150-GISEL: ; %bb.0:
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v7, v2
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v8, v3
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v9, v4
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v10, s0
+; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-GISEL-NEXT: .LBB23_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s4, v10
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v10
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s4, v5
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s5, v6
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s6, v7
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s7, v8
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v1, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v2, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v3, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX1150-GISEL-NEXT: buffer_load_format_xyzw v[0:4], v9, s[4:7], 0 idxen tfe
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr10
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB23_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
+; GFX1150-GISEL-NEXT: ; return to shader part epilog
+;
+; GFX12-SDAG-LABEL: test_waterfall_multi_end_struct_uniform:
+; GFX12-SDAG: ; %bb.0:
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v5, v4
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s1, v1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s2, v2
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s3, v3
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s0, v4
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v2, v0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v3, v0
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-SDAG-NEXT: buffer_load_format_xyzw v[0:4], v5, s[0:3], null idxen tfe
+; GFX12-SDAG-NEXT: s_wait_loadcnt 0x0
+; GFX12-SDAG-NEXT: ; return to shader part epilog
+;
+; GFX12-GISEL-LABEL: test_waterfall_multi_end_struct_uniform:
+; GFX12-GISEL: ; %bb.0:
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v5, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v6, v1
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v7, v2
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v8, v3
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v9, v4
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v10, s0
+; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-GISEL-NEXT: .LBB23_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s4, v10
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v10
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s4, v5
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s5, v6
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s6, v7
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s7, v8
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v1, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v2, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v3, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, v0
+; GFX12-GISEL-NEXT: buffer_load_format_xyzw v[0:4], v9, s[4:7], null idxen tfe
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr10
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr9
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB23_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
+; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
+; GFX12-GISEL-NEXT: ; return to shader part epilog
+ i32 inreg %idx, <4 x i32> %s_idx, i32 %v_inp) #1 {
+ %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx)
+ %widx0 = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok, <4 x i32> %s_idx)
+ %val = call { <4 x float>, i32 } @llvm.amdgcn.struct.buffer.load.format.sl_v4f32i32s(<4 x i32> %widx0, i32 %v_inp, i32 0, i32 0, i32 0)
+ %payl = extractvalue { <4 x float>, i32 } %val, 0
+ %tfe = extractvalue { <4 x float>, i32 } %val, 1
+ %r0 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok, <4 x float> %payl)
+ %r1 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %tok, i32 %tfe)
+ %r1.float = bitcast i32 %r1 to float
+
+ %insert = insertvalue { <4 x float>, float } undef, <4 x float> %r0, 0
+ %insert1 = insertvalue { <4 x float>, float } %insert, float %r1.float, 1
+ ret {<4 x float> , float} %insert1
+}
+
+%struct.wobble = type { float, float }
+
+define amdgpu_cs_chain void @wf_scc_check(i32 %arg, %struct.wobble %arg1, float %arg2) {
+; VI-SDAG-LABEL: wf_scc_check:
+; VI-SDAG: ; %bb.0: ; %bb
+; VI-SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; VI-SDAG-NEXT: s_mov_b32 s0, 0
+; VI-SDAG-NEXT: s_mov_b32 s1, s0
+; VI-SDAG-NEXT: s_mov_b32 s2, s0
+; VI-SDAG-NEXT: s_mov_b32 s3, s0
+; VI-SDAG-NEXT: s_buffer_load_dword s1, s[0:3], 0x0
+; VI-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; VI-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; VI-SDAG-NEXT: s_cmp_eq_u32 s1, 0
+; VI-SDAG-NEXT: s_cselect_b64 s[8:9], -1, 0
+; VI-SDAG-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s1, v8
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s1, v8
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[12:13], s[2:3]
+; VI-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; VI-SDAG-NEXT: s_mov_b32 s1, s0
+; VI-SDAG-NEXT: s_mov_b32 s2, s0
+; VI-SDAG-NEXT: s_mov_b32 s3, s0
+; VI-SDAG-NEXT: s_mov_b32 s4, s0
+; VI-SDAG-NEXT: s_mov_b32 s5, s0
+; VI-SDAG-NEXT: s_mov_b32 s6, s0
+; VI-SDAG-NEXT: s_mov_b32 s7, s0
+; VI-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; VI-SDAG-NEXT: image_store v0, v[0:1], s[0:7] unorm
+; VI-SDAG-NEXT: ; implicit-def: $vgpr8
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[12:13]
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB24_1
+; VI-SDAG-NEXT: ; %bb.2:
+; VI-SDAG-NEXT: s_mov_b64 exec, s[10:11]
+; VI-SDAG-NEXT: v_rcp_f32_e32 v0, v11
+; VI-SDAG-NEXT: v_cndmask_b32_e64 v1, 0, 1.0, s[8:9]
+; VI-SDAG-NEXT: v_mul_f32_e32 v0, v1, v0
+; VI-SDAG-NEXT: v_cmp_le_f32_e32 vcc, 0, v0
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; VI-SDAG-NEXT: s_endpgm
+;
+; VI-GISEL-LABEL: wf_scc_check:
+; VI-GISEL: ; %bb.0: ; %bb
+; VI-GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; VI-GISEL-NEXT: s_mov_b32 s8, 0
+; VI-GISEL-NEXT: s_mov_b32 s9, s8
+; VI-GISEL-NEXT: s_mov_b32 s10, s8
+; VI-GISEL-NEXT: s_mov_b32 s11, s8
+; VI-GISEL-NEXT: s_buffer_load_dword s9, s[8:11], 0x0
+; VI-GISEL-NEXT: s_mov_b32 s0, 0
+; VI-GISEL-NEXT: s_mov_b32 s1, s8
+; VI-GISEL-NEXT: s_mov_b32 s2, s8
+; VI-GISEL-NEXT: s_mov_b32 s3, s8
+; VI-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; VI-GISEL-NEXT: s_cmp_eq_u32 s9, 0
+; VI-GISEL-NEXT: s_mov_b32 s4, s8
+; VI-GISEL-NEXT: s_mov_b32 s5, s8
+; VI-GISEL-NEXT: s_mov_b32 s6, s8
+; VI-GISEL-NEXT: s_mov_b32 s7, s8
+; VI-GISEL-NEXT: s_cselect_b32 s12, 1, 0
+; VI-GISEL-NEXT: s_mov_b64 s[10:11], exec
+; VI-GISEL-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s9, v8
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[14:15], s9, v8
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[14:15], s[14:15]
+; VI-GISEL-NEXT: s_mov_b32 s9, s8
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s8
+; VI-GISEL-NEXT: v_mov_b32_e32 v2, 0
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s9
+; VI-GISEL-NEXT: image_store v2, v[0:1], s[0:7] unorm
+; VI-GISEL-NEXT: ; implicit-def: $vgpr8
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[14:15]
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB24_1
+; VI-GISEL-NEXT: ; %bb.2:
+; VI-GISEL-NEXT: s_mov_b64 exec, s[10:11]
+; VI-GISEL-NEXT: v_rcp_f32_e32 v0, v11
+; VI-GISEL-NEXT: s_cmp_lg_u32 s12, 0
+; VI-GISEL-NEXT: s_cselect_b32 s0, 1.0, 0
+; VI-GISEL-NEXT: v_mul_f32_e32 v0, s0, v0
+; VI-GISEL-NEXT: v_cmp_le_f32_e32 vcc, 0, v0
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; VI-GISEL-NEXT: s_endpgm
+;
+; GFX9-SDAG-LABEL: wf_scc_check:
+; GFX9-SDAG: ; %bb.0: ; %bb
+; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-SDAG-NEXT: s_mov_b32 s0, 0
+; GFX9-SDAG-NEXT: s_mov_b32 s1, s0
+; GFX9-SDAG-NEXT: s_mov_b32 s2, s0
+; GFX9-SDAG-NEXT: s_mov_b32 s3, s0
+; GFX9-SDAG-NEXT: s_buffer_load_dword s1, s[0:3], 0x0
+; GFX9-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; GFX9-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-SDAG-NEXT: s_cmp_eq_u32 s1, 0
+; GFX9-SDAG-NEXT: s_cselect_b64 s[8:9], -1, 0
+; GFX9-SDAG-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s1, v8
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s1, v8
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[12:13], s[2:3]
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX9-SDAG-NEXT: s_mov_b32 s1, s0
+; GFX9-SDAG-NEXT: s_mov_b32 s2, s0
+; GFX9-SDAG-NEXT: s_mov_b32 s3, s0
+; GFX9-SDAG-NEXT: s_mov_b32 s4, s0
+; GFX9-SDAG-NEXT: s_mov_b32 s5, s0
+; GFX9-SDAG-NEXT: s_mov_b32 s6, s0
+; GFX9-SDAG-NEXT: s_mov_b32 s7, s0
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v1, v0
+; GFX9-SDAG-NEXT: image_store v0, v[0:1], s[0:7] unorm
+; GFX9-SDAG-NEXT: ; implicit-def: $vgpr8
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[12:13]
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB24_1
+; GFX9-SDAG-NEXT: ; %bb.2:
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[10:11]
+; GFX9-SDAG-NEXT: v_rcp_f32_e32 v0, v11
+; GFX9-SDAG-NEXT: v_cndmask_b32_e64 v1, 0, 1.0, s[8:9]
+; GFX9-SDAG-NEXT: v_mul_f32_e32 v0, v1, v0
+; GFX9-SDAG-NEXT: v_cmp_le_f32_e32 vcc, 0, v0
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX9-SDAG-NEXT: s_endpgm
+;
+; GFX9-GISEL-LABEL: wf_scc_check:
+; GFX9-GISEL: ; %bb.0: ; %bb
+; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-GISEL-NEXT: s_mov_b32 s8, 0
+; GFX9-GISEL-NEXT: s_mov_b32 s9, s8
+; GFX9-GISEL-NEXT: s_mov_b32 s10, s8
+; GFX9-GISEL-NEXT: s_mov_b32 s11, s8
+; GFX9-GISEL-NEXT: s_buffer_load_dword s9, s[8:11], 0x0
+; GFX9-GISEL-NEXT: s_mov_b32 s0, 0
+; GFX9-GISEL-NEXT: s_mov_b32 s1, s8
+; GFX9-GISEL-NEXT: s_mov_b32 s2, s8
+; GFX9-GISEL-NEXT: s_mov_b32 s3, s8
+; GFX9-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX9-GISEL-NEXT: s_cmp_eq_u32 s9, 0
+; GFX9-GISEL-NEXT: s_mov_b32 s4, s8
+; GFX9-GISEL-NEXT: s_mov_b32 s5, s8
+; GFX9-GISEL-NEXT: s_mov_b32 s6, s8
+; GFX9-GISEL-NEXT: s_mov_b32 s7, s8
+; GFX9-GISEL-NEXT: s_cselect_b32 s12, 1, 0
+; GFX9-GISEL-NEXT: s_mov_b64 s[10:11], exec
+; GFX9-GISEL-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s9, v8
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[14:15], s9, v8
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[14:15], s[14:15]
+; GFX9-GISEL-NEXT: s_mov_b32 s9, s8
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s8
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v2, 0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s9
+; GFX9-GISEL-NEXT: image_store v2, v[0:1], s[0:7] unorm
+; GFX9-GISEL-NEXT: ; implicit-def: $vgpr8
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[14:15]
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB24_1
+; GFX9-GISEL-NEXT: ; %bb.2:
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[10:11]
+; GFX9-GISEL-NEXT: v_rcp_f32_e32 v0, v11
+; GFX9-GISEL-NEXT: s_cmp_lg_u32 s12, 0
+; GFX9-GISEL-NEXT: s_cselect_b32 s0, 1.0, 0
+; GFX9-GISEL-NEXT: v_mul_f32_e32 v0, s0, v0
+; GFX9-GISEL-NEXT: v_cmp_le_f32_e32 vcc, 0, v0
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX9-GISEL-NEXT: s_endpgm
+;
+; GFX10-32-SDAG-LABEL: wf_scc_check:
+; GFX10-32-SDAG: ; %bb.0: ; %bb
+; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 s0, 0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s10, exec_lo
+; GFX10-32-SDAG-NEXT: s_mov_b32 s1, s0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s2, s0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s3, s0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s9, exec_lo
+; GFX10-32-SDAG-NEXT: s_buffer_load_dword s1, s[0:3], 0x0
+; GFX10-32-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-SDAG-NEXT: s_cmp_eq_u32 s1, 0
+; GFX10-32-SDAG-NEXT: s_cselect_b32 s8, -1, 0
+; GFX10-32-SDAG-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s1, v8
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s1, v8
+; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s1, s0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s2, s0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s3, s0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s4, s0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s5, s0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s6, s0
+; GFX10-32-SDAG-NEXT: s_mov_b32 s7, s0
+; GFX10-32-SDAG-NEXT: image_store v0, [v0, v0], s[0:7] dim:SQ_RSRC_IMG_2D unorm
+; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s10, s10
+; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr8
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB24_1
+; GFX10-32-SDAG-NEXT: ; %bb.2:
+; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s9
+; GFX10-32-SDAG-NEXT: v_rcp_f32_e32 v0, v11
+; GFX10-32-SDAG-NEXT: v_cndmask_b32_e64 v1, 0, 1.0, s8
+; GFX10-32-SDAG-NEXT: v_mul_f32_e32 v0, v1, v0
+; GFX10-32-SDAG-NEXT: v_cmp_le_f32_e32 vcc_lo, 0, v0
+; GFX10-32-SDAG-NEXT: s_and_saveexec_b32 s0, vcc_lo
+; GFX10-32-SDAG-NEXT: s_endpgm
+;
+; GFX10-32-GISEL-LABEL: wf_scc_check:
+; GFX10-32-GISEL: ; %bb.0: ; %bb
+; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 s12, 0
+; GFX10-32-GISEL-NEXT: s_mov_b32 s10, exec_lo
+; GFX10-32-GISEL-NEXT: s_mov_b32 s13, s12
+; GFX10-32-GISEL-NEXT: s_mov_b32 s14, s12
+; GFX10-32-GISEL-NEXT: s_mov_b32 s15, s12
+; GFX10-32-GISEL-NEXT: s_mov_b32 s0, s12
+; GFX10-32-GISEL-NEXT: s_buffer_load_dword s7, s[12:15], 0x0
+; GFX10-32-GISEL-NEXT: s_mov_b32 s1, s12
+; GFX10-32-GISEL-NEXT: s_mov_b32 s2, s12
+; GFX10-32-GISEL-NEXT: s_mov_b32 s3, s12
+; GFX10-32-GISEL-NEXT: s_mov_b32 s4, s12
+; GFX10-32-GISEL-NEXT: s_mov_b32 s5, s12
+; GFX10-32-GISEL-NEXT: s_mov_b32 s6, s12
+; GFX10-32-GISEL-NEXT: s_mov_b32 s9, exec_lo
+; GFX10-32-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-32-GISEL-NEXT: s_cmp_eq_u32 s7, 0
+; GFX10-32-GISEL-NEXT: s_mov_b32 s7, s12
+; GFX10-32-GISEL-NEXT: s_cselect_b32 s8, 1, 0
+; GFX10-32-GISEL-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s11, v8
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s11, v8
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-32-GISEL-NEXT: image_store v0, [v0, v0], s[0:7] dim:SQ_RSRC_IMG_2D unorm
+; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s10, s10
+; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr8
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB24_1
+; GFX10-32-GISEL-NEXT: ; %bb.2:
+; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s9
+; GFX10-32-GISEL-NEXT: v_rcp_f32_e32 v0, v11
+; GFX10-32-GISEL-NEXT: s_cmp_lg_u32 s8, 0
+; GFX10-32-GISEL-NEXT: s_cselect_b32 s0, 1.0, 0
+; GFX10-32-GISEL-NEXT: v_mul_f32_e32 v0, s0, v0
+; GFX10-32-GISEL-NEXT: v_cmp_le_f32_e32 vcc_lo, 0, v0
+; GFX10-32-GISEL-NEXT: s_and_saveexec_b32 s0, vcc_lo
+; GFX10-32-GISEL-NEXT: s_endpgm
+;
+; GFX10-64-SDAG-LABEL: wf_scc_check:
+; GFX10-64-SDAG: ; %bb.0: ; %bb
+; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10-64-SDAG-NEXT: s_mov_b32 s0, 0
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[12:13], exec
+; GFX10-64-SDAG-NEXT: s_mov_b32 s1, s0
+; GFX10-64-SDAG-NEXT: s_mov_b32 s2, s0
+; GFX10-64-SDAG-NEXT: s_mov_b32 s3, s0
+; GFX10-64-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; GFX10-64-SDAG-NEXT: s_buffer_load_dword s1, s[0:3], 0x0
+; GFX10-64-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-SDAG-NEXT: s_cmp_eq_u32 s1, 0
+; GFX10-64-SDAG-NEXT: s_cselect_b64 s[8:9], -1, 0
+; GFX10-64-SDAG-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s1, v8
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s1, v8
+; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-SDAG-NEXT: s_mov_b32 s1, s0
+; GFX10-64-SDAG-NEXT: s_mov_b32 s2, s0
+; GFX10-64-SDAG-NEXT: s_mov_b32 s3, s0
+; GFX10-64-SDAG-NEXT: s_mov_b32 s4, s0
+; GFX10-64-SDAG-NEXT: s_mov_b32 s5, s0
+; GFX10-64-SDAG-NEXT: s_mov_b32 s6, s0
+; GFX10-64-SDAG-NEXT: s_mov_b32 s7, s0
+; GFX10-64-SDAG-NEXT: image_store v0, [v0, v0], s[0:7] dim:SQ_RSRC_IMG_2D unorm
+; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[12:13], s[12:13]
+; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr8
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB24_1
+; GFX10-64-SDAG-NEXT: ; %bb.2:
+; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[10:11]
+; GFX10-64-SDAG-NEXT: v_rcp_f32_e32 v0, v11
+; GFX10-64-SDAG-NEXT: v_cndmask_b32_e64 v1, 0, 1.0, s[8:9]
+; GFX10-64-SDAG-NEXT: v_mul_f32_e32 v0, v1, v0
+; GFX10-64-SDAG-NEXT: v_cmp_le_f32_e32 vcc, 0, v0
+; GFX10-64-SDAG-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX10-64-SDAG-NEXT: s_endpgm
+;
+; GFX10-64-GISEL-LABEL: wf_scc_check:
+; GFX10-64-GISEL: ; %bb.0: ; %bb
+; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX10-64-GISEL-NEXT: s_mov_b32 s8, 0
+; GFX10-64-GISEL-NEXT: s_mov_b32 s9, s8
+; GFX10-64-GISEL-NEXT: s_mov_b32 s10, s8
+; GFX10-64-GISEL-NEXT: s_mov_b32 s11, s8
+; GFX10-64-GISEL-NEXT: s_mov_b32 s0, s8
+; GFX10-64-GISEL-NEXT: s_buffer_load_dword s7, s[8:11], 0x0
+; GFX10-64-GISEL-NEXT: s_mov_b32 s1, s8
+; GFX10-64-GISEL-NEXT: s_mov_b32 s2, s8
+; GFX10-64-GISEL-NEXT: s_mov_b32 s3, s8
+; GFX10-64-GISEL-NEXT: s_mov_b32 s4, s8
+; GFX10-64-GISEL-NEXT: s_mov_b32 s5, s8
+; GFX10-64-GISEL-NEXT: s_mov_b32 s6, s8
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[10:11], exec
+; GFX10-64-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-64-GISEL-NEXT: s_cmp_eq_u32 s7, 0
+; GFX10-64-GISEL-NEXT: s_mov_b32 s7, s8
+; GFX10-64-GISEL-NEXT: s_cselect_b32 s12, 1, 0
+; GFX10-64-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX10-64-GISEL-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s13, v8
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s13, v8
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX10-64-GISEL-NEXT: image_store v0, [v0, v0], s[0:7] dim:SQ_RSRC_IMG_2D unorm
+; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[10:11], s[10:11]
+; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr8
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB24_1
+; GFX10-64-GISEL-NEXT: ; %bb.2:
+; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[8:9]
+; GFX10-64-GISEL-NEXT: v_rcp_f32_e32 v0, v11
+; GFX10-64-GISEL-NEXT: s_cmp_lg_u32 s12, 0
+; GFX10-64-GISEL-NEXT: s_cselect_b32 s0, 1.0, 0
+; GFX10-64-GISEL-NEXT: v_mul_f32_e32 v0, s0, v0
+; GFX10-64-GISEL-NEXT: v_cmp_le_f32_e32 vcc, 0, v0
+; GFX10-64-GISEL-NEXT: s_and_saveexec_b64 s[0:1], vcc
+; GFX10-64-GISEL-NEXT: s_endpgm
+;
+; GFX1150-SDAG-LABEL: wf_scc_check:
+; GFX1150-SDAG: ; %bb.0: ; %bb
+; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX1150-SDAG-NEXT: s_mov_b32 s0, 0
+; GFX1150-SDAG-NEXT: s_mov_b64 s[12:13], exec
+; GFX1150-SDAG-NEXT: s_mov_b32 s1, s0
+; GFX1150-SDAG-NEXT: s_mov_b32 s2, s0
+; GFX1150-SDAG-NEXT: s_mov_b32 s3, s0
+; GFX1150-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; GFX1150-SDAG-NEXT: s_buffer_load_b32 s1, s[0:3], 0x0
+; GFX1150-SDAG-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1150-SDAG-NEXT: s_cmp_eq_u32 s1, 0
+; GFX1150-SDAG-NEXT: s_cselect_b64 s[8:9], -1, 0
+; GFX1150-SDAG-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s1, v8
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s1, v8
+; GFX1150-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-SDAG-NEXT: s_mov_b32 s1, s0
+; GFX1150-SDAG-NEXT: s_mov_b32 s2, s0
+; GFX1150-SDAG-NEXT: s_mov_b32 s3, s0
+; GFX1150-SDAG-NEXT: s_mov_b32 s4, s0
+; GFX1150-SDAG-NEXT: s_mov_b32 s5, s0
+; GFX1150-SDAG-NEXT: s_mov_b32 s6, s0
+; GFX1150-SDAG-NEXT: s_mov_b32 s7, s0
+; GFX1150-SDAG-NEXT: image_store v0, [v0, v0], s[0:7] dim:SQ_RSRC_IMG_2D unorm
+; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[12:13], s[12:13]
+; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr8
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB24_1
+; GFX1150-SDAG-NEXT: ; %bb.2:
+; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[10:11]
+; GFX1150-SDAG-NEXT: v_rcp_f32_e32 v0, v11
+; GFX1150-SDAG-NEXT: v_cndmask_b32_e64 v1, 0, 1.0, s[8:9]
+; GFX1150-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(TRANS32_DEP_1) | instid1(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_mul_f32_e32 v0, v1, v0
+; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-SDAG-NEXT: v_cmpx_le_f32_e32 0, v0
+; GFX1150-SDAG-NEXT: s_endpgm
+;
+; GFX1150-GISEL-LABEL: wf_scc_check:
+; GFX1150-GISEL: ; %bb.0: ; %bb
+; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX1150-GISEL-NEXT: s_mov_b32 s8, 0
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: s_mov_b32 s9, s8
+; GFX1150-GISEL-NEXT: s_mov_b32 s10, s8
+; GFX1150-GISEL-NEXT: s_mov_b32 s11, s8
+; GFX1150-GISEL-NEXT: s_mov_b32 s0, s8
+; GFX1150-GISEL-NEXT: s_buffer_load_b32 s7, s[8:11], 0x0
+; GFX1150-GISEL-NEXT: s_mov_b32 s1, s8
+; GFX1150-GISEL-NEXT: s_mov_b32 s2, s8
+; GFX1150-GISEL-NEXT: s_mov_b32 s3, s8
+; GFX1150-GISEL-NEXT: s_mov_b32 s4, s8
+; GFX1150-GISEL-NEXT: s_mov_b32 s5, s8
+; GFX1150-GISEL-NEXT: s_mov_b32 s6, s8
+; GFX1150-GISEL-NEXT: s_mov_b64 s[10:11], exec
+; GFX1150-GISEL-NEXT: s_waitcnt lgkmcnt(0)
+; GFX1150-GISEL-NEXT: s_cmp_eq_u32 s7, 0
+; GFX1150-GISEL-NEXT: s_mov_b32 s7, s8
+; GFX1150-GISEL-NEXT: s_cselect_b32 s12, 1, 0
+; GFX1150-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX1150-GISEL-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s13, v8
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s13, v8
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX1150-GISEL-NEXT: image_store v0, [v0, v0], s[0:7] dim:SQ_RSRC_IMG_2D unorm
+; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
+; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr8
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB24_1
+; GFX1150-GISEL-NEXT: ; %bb.2:
+; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[8:9]
+; GFX1150-GISEL-NEXT: v_rcp_f32_e32 v0, v11
+; GFX1150-GISEL-NEXT: s_cmp_lg_u32 s12, 0
+; GFX1150-GISEL-NEXT: s_cselect_b32 s0, 1.0, 0
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(TRANS32_DEP_1) | instid1(SALU_CYCLE_1)
+; GFX1150-GISEL-NEXT: v_mul_f32_e32 v0, s0, v0
+; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-GISEL-NEXT: v_cmpx_le_f32_e32 0, v0
+; GFX1150-GISEL-NEXT: s_endpgm
+;
+; GFX12-SDAG-LABEL: wf_scc_check:
+; GFX12-SDAG: ; %bb.0: ; %bb
+; GFX12-SDAG-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-SDAG-NEXT: s_wait_expcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
+; GFX12-SDAG-NEXT: s_wait_bvhcnt 0x0
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_mov_b32 s0, 0
+; GFX12-SDAG-NEXT: s_mov_b64 s[10:11], exec
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_mov_b32 s1, s0
+; GFX12-SDAG-NEXT: s_mov_b32 s2, s0
+; GFX12-SDAG-NEXT: s_mov_b32 s3, s0
+; GFX12-SDAG-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-SDAG-NEXT: s_buffer_load_b32 s12, s[0:3], 0x0
+; GFX12-SDAG-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: v_readfirstlane_b32 s1, v8
+; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s1, v8
+; GFX12-SDAG-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-SDAG-NEXT: s_mov_b32 s1, s0
+; GFX12-SDAG-NEXT: s_mov_b32 s2, s0
+; GFX12-SDAG-NEXT: s_mov_b32 s3, s0
+; GFX12-SDAG-NEXT: s_mov_b32 s4, s0
+; GFX12-SDAG-NEXT: s_mov_b32 s5, s0
+; GFX12-SDAG-NEXT: s_mov_b32 s6, s0
+; GFX12-SDAG-NEXT: s_mov_b32 s7, s0
+; GFX12-SDAG-NEXT: image_store v0, [v0, v0], s[0:7] dim:SQ_RSRC_IMG_2D
+; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
+; GFX12-SDAG-NEXT: ; implicit-def: $vgpr8
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB24_1
+; GFX12-SDAG-NEXT: ; %bb.2:
+; GFX12-SDAG-NEXT: s_mov_b64 exec, s[8:9]
+; GFX12-SDAG-NEXT: v_rcp_f32_e32 v0, v11
+; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
+; GFX12-SDAG-NEXT: s_cmp_eq_u32 s12, 0
+; GFX12-SDAG-NEXT: s_cselect_b32 s0, 1.0, 0
+; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-SDAG-NEXT: s_delay_alu instid0(TRANS32_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX12-SDAG-NEXT: v_mul_f32_e32 v0, s0, v0
+; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-SDAG-NEXT: v_cmpx_le_f32_e32 0, v0
+; GFX12-SDAG-NEXT: s_endpgm
+;
+; GFX12-GISEL-LABEL: wf_scc_check:
+; GFX12-GISEL: ; %bb.0: ; %bb
+; GFX12-GISEL-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX12-GISEL-NEXT: s_wait_expcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
+; GFX12-GISEL-NEXT: s_wait_bvhcnt 0x0
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_mov_b32 s8, 0
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_mov_b32 s9, s8
+; GFX12-GISEL-NEXT: s_mov_b32 s10, s8
+; GFX12-GISEL-NEXT: s_mov_b32 s11, s8
+; GFX12-GISEL-NEXT: s_mov_b32 s0, s8
+; GFX12-GISEL-NEXT: s_buffer_load_b32 s7, s[8:11], 0x0
+; GFX12-GISEL-NEXT: s_mov_b32 s1, s8
+; GFX12-GISEL-NEXT: s_mov_b32 s2, s8
+; GFX12-GISEL-NEXT: s_mov_b32 s3, s8
+; GFX12-GISEL-NEXT: s_mov_b32 s4, s8
+; GFX12-GISEL-NEXT: s_mov_b32 s5, s8
+; GFX12-GISEL-NEXT: s_mov_b32 s6, s8
+; GFX12-GISEL-NEXT: s_mov_b64 s[10:11], exec
+; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
+; GFX12-GISEL-NEXT: s_cmp_eq_u32 s7, 0
+; GFX12-GISEL-NEXT: s_mov_b32 s7, s8
+; GFX12-GISEL-NEXT: s_cselect_b32 s12, 1, 0
+; GFX12-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX12-GISEL-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: v_readfirstlane_b32 s13, v8
+; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s13, v8
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v0, 0
+; GFX12-GISEL-NEXT: image_store v0, [v0, v0], s[0:7] dim:SQ_RSRC_IMG_2D
+; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
+; GFX12-GISEL-NEXT: ; implicit-def: $vgpr8
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB24_1
+; GFX12-GISEL-NEXT: ; %bb.2:
+; GFX12-GISEL-NEXT: s_mov_b64 exec, s[8:9]
+; GFX12-GISEL-NEXT: v_rcp_f32_e32 v0, v11
+; GFX12-GISEL-NEXT: s_cmp_lg_u32 s12, 0
+; GFX12-GISEL-NEXT: s_cselect_b32 s0, 1.0, 0
+; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-GISEL-NEXT: s_delay_alu instid0(TRANS32_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX12-GISEL-NEXT: v_mul_f32_e32 v0, s0, v0
+; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX12-GISEL-NEXT: v_cmpx_le_f32_e32 0, v0
+; GFX12-GISEL-NEXT: s_endpgm
+bb:
+ %call = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> zeroinitializer, i32 0, i32 0)
+ %icmp = icmp eq i32 %call, 0
+ %call3 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %arg)
+ %call4 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %call3, i32 %arg)
+ %call5 = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(i32 %call3, <8 x i32> zeroinitializer)
+ call void @llvm.amdgcn.image.store.2d.f32.i32.v8i32(float 0.000000e+00, i32 0, i32 0, i32 0, <8 x i32> %call5, i32 0, i32 0)
+ %select = select i1 %icmp, float 1.000000e+00, float 0.000000e+00
+ %fdiv = fdiv afn float %select, %arg2
+ %fcmp = fcmp ult float %fdiv, 0.000000e+00
+ br i1 %fcmp, label %bb6, label %bb7
+
+bb6: ; preds = %bb7, %bb
+ ret void
+
+bb7: ; preds = %bb
+ %extractvalue = extractvalue %struct.wobble %arg1, 0
+ %fsub = fsub float 0.000000e+00, %extractvalue
+ %fcmp8 = fcmp olt float %fsub, 0.000000e+00
+ %or = or i1 false, %fcmp8
+ br label %bb6
+}
+
+declare i32 @llvm.amdgcn.waterfall.begin.i32(i32, i32) #6
+declare i32 @llvm.amdgcn.waterfall.begin.v2i32(i32, <2 x i32>) #6
+declare i32 @llvm.amdgcn.waterfall.begin.v4i32(i32, <4 x i32>) #6
+declare i32 @llvm.amdgcn.waterfall.begin.v8i32(i32, <8 x i32>) #1
+declare i32 @llvm.amdgcn.waterfall.begin.p3(i32 , ptr addrspace(3))
+declare i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32, i32) #6
+declare <2 x i32> @llvm.amdgcn.waterfall.readfirstlane.v2i32.v2i32(i32, <2 x i32>) #6
+declare <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32, <8 x i32>) #6
+declare <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32, <4 x i32>) #1
+declare i16 @llvm.amdgcn.waterfall.end.i16(i32, i16) #6
+declare i32 @llvm.amdgcn.waterfall.end.i32(i32, i32) #6
+declare <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32, <4 x float>) #6
+declare <8 x i32> @llvm.amdgcn.waterfall.end.v8i32(i32, <8 x i32>) #6
+declare <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(i32, <8 x i32>) #6
+declare ptr addrspace(3) @llvm.amdgcn.waterfall.last.use.vgpr.p3(i32, ptr addrspace(3))
+declare i32 @llvm.amdgcn.readlane(i32, i32) #0
+declare <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32, float, <8 x i32>, <4 x i32>, i1, i32, i32)
+declare <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32, float, float, <8 x i32>, <4 x i32>, i1, i32, i32)
+declare void @llvm.amdgcn.image.store.1d.v4f32.i32(<4 x float>, i32, i32, <8 x i32>, i32, i32)
+declare i64 @llvm.amdgcn.s.getpc() #3
+declare float @llvm.amdgcn.buffer.load.ushort(<4 x i32>, i32, i32, i1, i1) #4
+declare float @llvm.amdgcn.buffer.load.f32(<4 x i32>, i32, i32, i1, i1) #4
+declare <4 x float> @llvm.amdgcn.buffer.load.v4f32(<4 x i32>, i32, i32, i1, i1) #4
+declare i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32>, i32, i32) #2
+declare { <4 x float>, i32 } @llvm.amdgcn.struct.buffer.load.format.sl_v4f32i32s(<4 x i32>, i32, i32, i32, i32) #4
+declare <4 x i32> @llvm.amdgcn.s.buffer.load.v4i32(<4 x i32>, i32, i1) #2
+declare void @llvm.amdgcn.buffer.store.short(float, <4 x i32>, i32, i32, i1, i1) #5
+declare void @llvm.amdgcn.buffer.store.f32(float, <4 x i32>, i32, i32, i1, i1) #5
+declare void @llvm.amdgcn.buffer.store.v4f32(<4 x float>, <4 x i32>, i32, i32, i1, i1) #5
+declare void @llvm.amdgcn.kill(i1) #1
+declare void @llvm.amdgcn.exp.f32(i32 immarg, i32 immarg, float, float, float, float, i1 immarg, i1 immarg) #7
+
+attributes #0 = { nounwind readnone convergent }
+attributes #1 = { nounwind }
+attributes #2 = { nounwind readnone }
+attributes #3 = { nounwind readnone speculatable }
+attributes #4 = { nounwind readonly }
+attributes #5 = { nounwind writeonly }
+attributes #6 = { nounwind }
+attributes #7 = { inaccessiblememonly nounwind writeonly }
diff --git a/llvm/test/CodeGen/AMDGPU/si-insert-waterfall-licm.ll b/llvm/test/CodeGen/AMDGPU/si-insert-waterfall-licm.ll
new file mode 100644
index 0000000000000..3c9aa3f80a32c
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/si-insert-waterfall-licm.ll
@@ -0,0 +1,43 @@
+; RUN: opt -mtriple=amdgcn -mcpu=gfx1100 -amdgpu-codegenprepare -S -o - < %s | FileCheck %s
+
+; Check that llvm.licm.disable is removed for loop forming divergent pointer.
+; CHECK-NOT: !{!"llvm.licm.disable"}
+
+define amdgpu_ps <4 x float> @needs_licm(i32 inreg %userdata, i32 %index, float %s, <4 x i32> inreg %samp, i32 inreg %a, i32 inreg %b) {
+entry:
+ %pc.0 = call i64 @llvm.amdgcn.s.getpc()
+ %pc.1 = and i64 %pc.0, -4294967296
+ %lo = call i32 @llvm.amdgcn.mbcnt.lo(i32 -1, i32 0)
+ %hi = call i32 @llvm.amdgcn.mbcnt.hi(i32 -1, i32 %lo)
+ %cc = icmp uge i32 %hi, 16
+ br i1 %cc, label %alt.entry, label %preheader
+
+alt.entry:
+ br label %preheader
+
+preheader:
+ %limit = phi i32 [ %a, %entry ], [ %b, %alt.entry ]
+ br label %loop.body
+
+loop.body:
+ %i.0 = phi i32 [ 0, %preheader ], [ %inc, %loop.body ]
+ %inc = add nsw i32 %i.0, 1
+ %cmp = icmp slt i32 %inc, %limit
+ %ptr.0 = zext i32 %userdata to i64
+ %ptr.1 = or disjoint i64 %pc.1, %ptr.0
+ %loop.ptr = inttoptr i64 %ptr.1 to ptr addrspace(4)
+ br i1 %cmp, label %loop.body, label %loop.exit, !llvm.loop !1
+
+loop.exit:
+ %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %index)
+ %ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %loop.ptr, i32 %s_idx
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %ptr, align 32
+ %r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token, <4 x float> %r)
+
+ ret <4 x float> %r1
+}
+
+!1 = !{!1, !2}
+!2 = !{!"llvm.licm.disable"}
diff --git a/llvm/test/CodeGen/AMDGPU/si-insert-waterfall.mir b/llvm/test/CodeGen/AMDGPU/si-insert-waterfall.mir
new file mode 100644
index 0000000000000..30dedf0a89f18
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/si-insert-waterfall.mir
@@ -0,0 +1,54 @@
+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple=amdgcn--amdpal -march=amdgcn -mcpu=gfx1030 -run-pass=si-insert-waterfall -o - %s | FileCheck %s
+
+# The test demonstrates a waterfall loop expansion in si-insert-waterfall pass.
+---
+name: waterfall
+tracksRegLiveness: true
+body: |
+ bb.0:
+ liveins: $sgpr0, $vgpr0, $vgpr1
+ ; CHECK-LABEL: name: waterfall
+ ; CHECK: successors: %bb.1(0x80000000)
+ ; CHECK-NEXT: liveins: $sgpr0, $vgpr0, $vgpr1
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[COPY:%[0-9]+]]:sgpr_32 = COPY $sgpr0
+ ; CHECK-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr0
+ ; CHECK-NEXT: [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr1
+ ; CHECK-NEXT: [[REG_SEQUENCE:%[0-9]+]]:sgpr_128 = REG_SEQUENCE [[COPY]], %subreg.sub0, [[COPY]], %subreg.sub1, [[COPY]], %subreg.sub2, [[COPY]], %subreg.sub3
+ ; CHECK-NEXT: [[S_MOV_B32_:%[0-9]+]]:sreg_32_xm0 = S_MOV_B32 0
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: .1:
+ ; CHECK-NEXT: successors: %bb.2(0x80000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[S_MOV_B32_1:%[0-9]+]]:sreg_32_xm0_xexec = S_MOV_B32 $exec_lo
+ ; CHECK-NEXT: [[S_MOV_B32_2:%[0-9]+]]:sreg_32_xm0_xexec = S_MOV_B32 $exec_lo
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: .2:
+ ; CHECK-NEXT: successors: %bb.2(0x40000000), %bb.3(0x40000000)
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[PHI:%[0-9]+]]:sreg_32_xm0_xexec = PHI [[S_MOV_B32_1]], %bb.1, %14, %bb.2
+ ; CHECK-NEXT: [[V_READFIRSTLANE_B32_:%[0-9]+]]:sreg_32_xm0 = V_READFIRSTLANE_B32 [[COPY2]], implicit $exec
+ ; CHECK-NEXT: V_CMPX_EQ_U32_nosdst_e32 [[V_READFIRSTLANE_B32_]], [[COPY2]], implicit-def $exec, implicit $exec
+ ; CHECK-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:sreg_64 = REG_SEQUENCE [[V_READFIRSTLANE_B32_]], %subreg.sub0, [[V_READFIRSTLANE_B32_]], %subreg.sub1
+ ; CHECK-NEXT: [[S_LOAD_DWORDX8_IMM:%[0-9]+]]:sgpr_256 = S_LOAD_DWORDX8_IMM [[REG_SEQUENCE1]], 0, 0
+ ; CHECK-NEXT: [[IMAGE_SAMPLE_V4_V1_gfx10_:%[0-9]+]]:vreg_128 = IMAGE_SAMPLE_V4_V1_gfx10 [[COPY1]], [[S_LOAD_DWORDX8_IMM]], [[REG_SEQUENCE]], 15, 0, 0, 0, 0, 0, 0, 0, 0, implicit $exec :: (dereferenceable load (s128), addrspace 8)
+ ; CHECK-NEXT: [[COPY3:%[0-9]+]]:vreg_128 = COPY [[IMAGE_SAMPLE_V4_V1_gfx10_]]
+ ; CHECK-NEXT: [[S_ANDN2_WREXEC_B32_:%[0-9]+]]:sreg_32_xm0_xexec = S_ANDN2_WREXEC_B32 [[PHI]], implicit-def $exec, implicit-def $scc, implicit $exec
+ ; CHECK-NEXT: SI_WATERFALL_LOOP %bb.2, implicit $exec
+ ; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: .3:
+ ; CHECK-NEXT: $exec_lo = S_MOV_B32 [[S_MOV_B32_2]]
+ %0:sgpr_32 = COPY $sgpr0
+ %1:vgpr_32 = COPY $vgpr0
+ %2:vgpr_32 = COPY $vgpr1
+ %3:sgpr_128 = REG_SEQUENCE %0:sgpr_32, %subreg.sub0, %0:sgpr_32, %subreg.sub1, %0:sgpr_32, %subreg.sub2, %0:sgpr_32, %subreg.sub3
+ %4:sreg_32_xm0 = S_MOV_B32 0
+ %5:sreg_32 = SI_WATERFALL_BEGIN_V1 killed %4:sreg_32_xm0, %2:vgpr_32, implicit-def $scc
+ %6:sreg_32_xm0 = SI_WATERFALL_READFIRSTLANE_V1 %5:sreg_32, %2:vgpr_32, implicit-def $scc
+ %7:sreg_64 = REG_SEQUENCE %6:sreg_32_xm0, %subreg.sub0, %6:sreg_32_xm0, %subreg.sub1
+ %8:sgpr_256 = S_LOAD_DWORDX8_IMM killed %7:sreg_64, 0, 0
+ %9:vreg_128 = IMAGE_SAMPLE_V4_V1_gfx10 %1:vgpr_32, killed %8:sgpr_256, killed %3:sgpr_128, 15, 0, 0, 0, 0, 0, 0, 0, 0, implicit $exec :: (dereferenceable load (s128), addrspace 8)
+ %10:vreg_128 = SI_WATERFALL_END_V4 %5:sreg_32, killed %9:vreg_128, implicit-def $scc
+...
+
diff --git a/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll b/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
new file mode 100644
index 0000000000000..df57d55e2f674
--- /dev/null
+++ b/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
@@ -0,0 +1,73 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -S -passes=instcombine -mtriple=amdgcn-amd-amdhsa %s | FileCheck %s
+
+define amdgpu_ps <4 x float> @test_waterfall_same_index2(<8 x i32> addrspace(4)* inreg %in, i32 %index, float %s, <4 x i32> inreg %samp) {
+; CHECK-LABEL: @test_waterfall_same_index2(
+; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 [[INDEX:%.*]])
+; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 [[WF_TOKEN1]], i32 [[INDEX]])
+; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
+; CHECK-NEXT: [[PTR:%.*]] = getelementptr <8 x i32>, ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
+; CHECK-NEXT: [[RSRC:%.*]] = load <8 x i32>, ptr addrspace(4) [[PTR]], align 32
+; CHECK-NEXT: [[R:%.*]] = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32.v8i32.v4i32(i32 15, float [[S:%.*]], <8 x i32> [[RSRC]], <4 x i32> [[SAMP:%.*]], i1 false, i32 0, i32 0)
+; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 [[WF_TOKEN1]], <4 x float> [[R]])
+; CHECK-NEXT: ret <4 x float> [[R1]]
+;
+ %wf_token1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
+ %wf_token2 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %wf_token1, i32 %index)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token2, i32 %index)
+ %ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
+ %r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token2, <4 x float> %r)
+ ret <4 x float> %r1
+}
+
+define amdgpu_ps <4 x float> @test_waterfall_same_index3(<8 x i32> addrspace(4)* inreg %in, i32 %index, float %s, <4 x i32> inreg %samp) {
+; CHECK-LABEL: @test_waterfall_same_index3(
+; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 [[INDEX:%.*]])
+; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 [[WF_TOKEN1]], i32 [[INDEX]])
+; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
+; CHECK-NEXT: [[PTR:%.*]] = getelementptr <8 x i32>, ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
+; CHECK-NEXT: [[RSRC:%.*]] = load <8 x i32>, ptr addrspace(4) [[PTR]], align 32
+; CHECK-NEXT: [[R:%.*]] = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32.v8i32.v4i32(i32 15, float [[S:%.*]], <8 x i32> [[RSRC]], <4 x i32> [[SAMP:%.*]], i1 false, i32 0, i32 0)
+; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 [[WF_TOKEN1]], <4 x float> [[R]])
+; CHECK-NEXT: ret <4 x float> [[R1]]
+;
+ %wf_token1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
+ %wf_token2 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %wf_token1, i32 %index)
+ %wf_token3 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %wf_token2, i32 %index)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token3, i32 %index)
+ %ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
+ %r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token3, <4 x float> %r)
+ ret <4 x float> %r1
+}
+
+define amdgpu_ps <4 x float> @test_waterfall_same_index_aba(<8 x i32> addrspace(4)* inreg %in, i32 %index1, i32 %index2, float %s, <4 x i32> inreg %samp) {
+; CHECK-LABEL: @test_waterfall_same_index_aba(
+; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 [[INDEX1:%.*]])
+; CHECK-NEXT: [[WF_TOKEN2:%.*]] = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 [[WF_TOKEN1]], i32 [[INDEX2:%.*]])
+; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 [[WF_TOKEN2]], i32 [[INDEX2]])
+; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
+; CHECK-NEXT: [[PTR:%.*]] = getelementptr <8 x i32>, ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
+; CHECK-NEXT: [[RSRC:%.*]] = load <8 x i32>, ptr addrspace(4) [[PTR]], align 32
+; CHECK-NEXT: [[R:%.*]] = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32.v8i32.v4i32(i32 15, float [[S:%.*]], <8 x i32> [[RSRC]], <4 x i32> [[SAMP:%.*]], i1 false, i32 0, i32 0)
+; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 [[WF_TOKEN2]], <4 x float> [[R]])
+; CHECK-NEXT: ret <4 x float> [[R1]]
+;
+ %wf_token1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index1)
+ %wf_token2 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %wf_token1, i32 %index2)
+ %wf_token3 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %wf_token2, i32 %index1)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token3, i32 %index2)
+ %ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
+ %rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
+ %r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token3, <4 x float> %r)
+ ret <4 x float> %r1
+}
+
+declare i32 @llvm.amdgcn.waterfall.begin.i32(i32, i32)
+declare i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32, i32)
+declare <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32, <4 x float>)
+declare <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32, float, <8 x i32>, <4 x i32>, i1, i32, i32)
>From 3e0c5374b2fb408efe7c7cb06f8670059a94fe65 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Thu, 16 Apr 2026 14:05:14 +0100
Subject: [PATCH 02/35] [AMDGPU] Rename SIInsertWaterfall pass to
AMDGPUInsertWaterfall
Co-Authored-By: Claude Sonnet 4.6 <noreply at anthropic.com>
---
llvm/lib/Target/AMDGPU/AMDGPU.h | 6 +--
...aterfall.cpp => AMDGPUInsertWaterfall.cpp} | 38 +++++++++----------
...ertWaterfall.h => AMDGPUInsertWaterfall.h} | 11 +++---
llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def | 2 +-
.../Target/AMDGPU/AMDGPURegBankCombiner.cpp | 2 +-
.../lib/Target/AMDGPU/AMDGPUTargetMachine.cpp | 8 ++--
llvm/lib/Target/AMDGPU/CMakeLists.txt | 2 +-
7 files changed, 35 insertions(+), 34 deletions(-)
rename llvm/lib/Target/AMDGPU/{SIInsertWaterfall.cpp => AMDGPUInsertWaterfall.cpp} (96%)
rename llvm/lib/Target/AMDGPU/{SIInsertWaterfall.h => AMDGPUInsertWaterfall.h} (61%)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPU.h b/llvm/lib/Target/AMDGPU/AMDGPU.h
index fc46d0f82345a..a87c5f6ca7d25 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPU.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPU.h
@@ -53,7 +53,7 @@ FunctionPass *createSIFixSGPRCopiesLegacyPass();
FunctionPass *createLowerWWMCopiesPass();
FunctionPass *createSIMemoryLegalizerPass();
FunctionPass *createSIInsertWaitcntsPass();
-FunctionPass *createSIInsertWaterfallPass();
+FunctionPass *createAMDGPUInsertWaterfallPass();
FunctionPass *createSIPreAllocateWWMRegsLegacyPass();
FunctionPass *createSIFormMemoryClausesLegacyPass();
@@ -245,8 +245,8 @@ extern char &SILateBranchLoweringPassID;
void initializeSIOptimizeExecMaskingLegacyPass(PassRegistry &);
extern char &SIOptimizeExecMaskingLegacyID;
-void initializeSIInsertWaterfallPass(PassRegistry &);
-extern char &SIInsertWaterfallID;
+void initializeAMDGPUInsertWaterfallPass(PassRegistry &);
+extern char &AMDGPUInsertWaterfallID;
void initializeSIPreAllocateWWMRegsLegacyPass(PassRegistry &);
extern char &SIPreAllocateWWMRegsLegacyID;
diff --git a/llvm/lib/Target/AMDGPU/SIInsertWaterfall.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
similarity index 96%
rename from llvm/lib/Target/AMDGPU/SIInsertWaterfall.cpp
rename to llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
index f0f0662c1b28b..a7e1d97c36b7f 100644
--- a/llvm/lib/Target/AMDGPU/SIInsertWaterfall.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
@@ -1,4 +1,4 @@
-//===- SIInsertWaterfall.cpp - insert waterall loops at intrinsic markers -===//
+//===- AMDGPUInsertWaterfall.cpp - insert waterfall loops at intrinsic markers -===//
//
// The LLVM Compiler Infrastructure
//
@@ -14,7 +14,7 @@
///
//===----------------------------------------------------------------------===//
-#include "SIInsertWaterfall.h"
+#include "AMDGPUInsertWaterfall.h"
#include "AMDGPU.h"
#include "GCNSubtarget.h"
#include "MCTargetDesc/AMDGPUMCTargetDesc.h"
@@ -23,7 +23,7 @@
using namespace llvm;
-#define DEBUG_TYPE "si-insert-waterfall"
+#define DEBUG_TYPE "amdgpu-insert-waterfall"
namespace {
@@ -272,7 +272,7 @@ static void replaceRegIncSubReg(const MachineRegisterInfo *MRI,
O.substVirtReg(To->getReg(), To->getSubReg(), *TRI);
}
-class SIInsertWaterfall : public MachineFunctionPass {
+class AMDGPUInsertWaterfall : public MachineFunctionPass {
private:
struct WaterfallWorkitem {
const SIInstrInfo *TII;
@@ -435,8 +435,8 @@ class SIInsertWaterfall : public MachineFunctionPass {
public:
static char ID;
- SIInsertWaterfall() : MachineFunctionPass(ID) {
- initializeSIInsertWaterfallPass(*PassRegistry::getPassRegistry());
+ AMDGPUInsertWaterfall() : MachineFunctionPass(ID) {
+ initializeAMDGPUInsertWaterfallPass(*PassRegistry::getPassRegistry());
}
void getAnalysisUsage(AnalysisUsage &AU) const override {
@@ -453,27 +453,27 @@ class SIInsertWaterfall : public MachineFunctionPass {
} // End anonymous namespace.
-INITIALIZE_PASS(SIInsertWaterfall, DEBUG_TYPE, "SI Insert waterfalls", false,
- false)
+INITIALIZE_PASS(AMDGPUInsertWaterfall, DEBUG_TYPE, "AMDGPU Insert waterfalls",
+ false, false)
-char SIInsertWaterfall::ID = 0;
+char AMDGPUInsertWaterfall::ID = 0;
-char &llvm::SIInsertWaterfallID = SIInsertWaterfall::ID;
+char &llvm::AMDGPUInsertWaterfallID = AMDGPUInsertWaterfall::ID;
-FunctionPass *llvm::createSIInsertWaterfallPass() {
- return new SIInsertWaterfall;
+FunctionPass *llvm::createAMDGPUInsertWaterfallPass() {
+ return new AMDGPUInsertWaterfall;
}
PreservedAnalyses
-SIInsertWaterfallPass::run(MachineFunction &MF,
- MachineFunctionAnalysisManager &MFAM) {
- SIInsertWaterfall Impl;
+AMDGPUInsertWaterfallPass::run(MachineFunction &MF,
+ MachineFunctionAnalysisManager &MFAM) {
+ AMDGPUInsertWaterfall Impl;
if (!Impl.runOnMachineFunction(MF))
return PreservedAnalyses::all();
return PreservedAnalyses::none();
}
-bool SIInsertWaterfall::removeRedundantWaterfall(WaterfallWorkitem &Item) {
+bool AMDGPUInsertWaterfall::removeRedundantWaterfall(WaterfallWorkitem &Item) {
// In some cases, the waterfall is actually redundant
// If all the readfirstlane intrinsics are actually for uniform values and
// the token used in the begin/end isn't used in anything else the waterfall
@@ -576,7 +576,7 @@ bool SIInsertWaterfall::removeRedundantWaterfall(WaterfallWorkitem &Item) {
return LoopRemoved;
}
-bool SIInsertWaterfall::processWaterfall(MachineBasicBlock &MBB) {
+bool AMDGPUInsertWaterfall::processWaterfall(MachineBasicBlock &MBB) {
bool Changed = false;
MachineFunction &MF = *MBB.getParent();
MachineBasicBlock *CurrMBB = &MBB;
@@ -837,12 +837,12 @@ bool SIInsertWaterfall::processWaterfall(MachineBasicBlock &MBB) {
return Changed;
}
-Register SIInsertWaterfall::getToken(MachineInstr *MI) {
+Register AMDGPUInsertWaterfall::getToken(MachineInstr *MI) {
auto CandTokMO = TII->getNamedOperand(*MI, AMDGPU::OpName::tok);
return CandTokMO->isReg() ? CandTokMO->getReg() : AMDGPU::NoRegister;
}
-bool SIInsertWaterfall::runOnMachineFunction(MachineFunction &MF) {
+bool AMDGPUInsertWaterfall::runOnMachineFunction(MachineFunction &MF) {
bool Changed = false;
ST = &MF.getSubtarget<GCNSubtarget>();
diff --git a/llvm/lib/Target/AMDGPU/SIInsertWaterfall.h b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.h
similarity index 61%
rename from llvm/lib/Target/AMDGPU/SIInsertWaterfall.h
rename to llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.h
index 406df5a8b3b67..1c337b34ab2c2 100644
--- a/llvm/lib/Target/AMDGPU/SIInsertWaterfall.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.h
@@ -1,4 +1,4 @@
-//===- SIInsertWaterfall.h --------------------------------------*- C++ -*-===//
+//===- AMDGPUInsertWaterfall.h ----------------------------------*- C++ -*-===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
@@ -6,17 +6,18 @@
//
//===----------------------------------------------------------------------===//
-#ifndef LLVM_LIB_TARGET_AMDGPU_SIINSERTWATERFALL_H
-#define LLVM_LIB_TARGET_AMDGPU_SIINSERTWATERFALL_H
+#ifndef LLVM_LIB_TARGET_AMDGPU_AMDGPUINSERTWATERFALL_H
+#define LLVM_LIB_TARGET_AMDGPU_AMDGPUINSERTWATERFALL_H
#include "llvm/CodeGen/MachinePassManager.h"
namespace llvm {
-class SIInsertWaterfallPass : public PassInfoMixin<SIInsertWaterfallPass> {
+class AMDGPUInsertWaterfallPass
+ : public PassInfoMixin<AMDGPUInsertWaterfallPass> {
public:
PreservedAnalyses run(MachineFunction &MF,
MachineFunctionAnalysisManager &MFAM);
};
} // namespace llvm
-#endif // LLVM_LIB_TARGET_AMDGPU_SIINSERTWATERFALL_H
+#endif // LLVM_LIB_TARGET_AMDGPU_AMDGPUINSERTWATERFALL_H
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def b/llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def
index ae151adccf605..9c17fb46fca47 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def
+++ b/llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def
@@ -137,7 +137,7 @@ MACHINE_FUNCTION_PASS("si-form-memory-clauses", SIFormMemoryClausesPass())
MACHINE_FUNCTION_PASS("si-i1-copies", SILowerI1CopiesPass())
MACHINE_FUNCTION_PASS("si-insert-hard-clauses", SIInsertHardClausesPass())
MACHINE_FUNCTION_PASS("si-insert-waitcnts", SIInsertWaitcntsPass())
-MACHINE_FUNCTION_PASS("si-insert-waterfall", SIInsertWaterfallPass())
+MACHINE_FUNCTION_PASS("si-insert-waterfall", AMDGPUInsertWaterfallPass())
MACHINE_FUNCTION_PASS("si-late-branch-lowering", SILateBranchLoweringPass())
MACHINE_FUNCTION_PASS("si-load-store-opt", SILoadStoreOptimizerPass())
MACHINE_FUNCTION_PASS("si-lower-control-flow", SILowerControlFlowPass())
diff --git a/llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp b/llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp
index eeff9c53ed998..c612f90250992 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp
@@ -98,7 +98,7 @@ class AMDGPURegBankCombinerImpl : public Combiner {
// Combine support to remove amdgcn_waterfall_readfirstlane or
// amdgcn_waterfall_begin intrinsics if the index is determined to be
- // uniform. The SIInsertWaterfall pass can handle their removal and in some
+ // uniform. The AMDGPUInsertWaterfall pass can handle their removal and in some
// cases remove the waterfall altogether
bool matchRmUniformWF(MachineInstr &MI,
RmUniformWFMatchInfo &MatchInfo) const;
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
index 21eee8419b518..f12d9a9e33921 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
@@ -53,7 +53,7 @@
#include "SIFixVGPRCopies.h"
#include "SIFoldOperands.h"
#include "SIFormMemoryClauses.h"
-#include "SIInsertWaterfall.h"
+#include "AMDGPUInsertWaterfall.h"
#include "SILoadStoreOptimizer.h"
#include "SILowerControlFlow.h"
#include "SILowerSGPRSpills.h"
@@ -676,7 +676,7 @@ extern "C" LLVM_ABI LLVM_EXTERNAL_VISIBILITY void LLVMInitializeAMDGPUTarget() {
initializeSIFixSGPRCopiesLegacyPass(*PR);
initializeSIFixVGPRCopiesLegacyPass(*PR);
initializeSIFoldOperandsLegacyPass(*PR);
- initializeSIInsertWaterfallPass(*PR);
+ initializeAMDGPUInsertWaterfallPass(*PR);
initializeSIPeepholeSDWALegacyPass(*PR);
initializeSIShrinkInstructionsLegacyPass(*PR);
initializeSIOptimizeExecMaskingPreRALegacyPass(*PR);
@@ -1738,7 +1738,7 @@ void GCNPassConfig::addFastRegAlloc() {
}
void GCNPassConfig::addPreRegAlloc() {
- addPass(createSIInsertWaterfallPass());
+ addPass(createAMDGPUInsertWaterfallPass());
if (getOptLevel() != CodeGenOptLevel::None)
addPass(&AMDGPUPrepareAGPRAllocLegacyID);
}
@@ -2553,7 +2553,7 @@ Error AMDGPUCodeGenPassBuilder::addOptimizedRegAlloc(
}
void AMDGPUCodeGenPassBuilder::addPreRegAlloc(PassManagerWrapper &PMW) const {
- addMachineFunctionPass(SIInsertWaterfallPass(), PMW);
+ addMachineFunctionPass(AMDGPUInsertWaterfallPass(), PMW);
if (getOptLevel() != CodeGenOptLevel::None)
addMachineFunctionPass(AMDGPUPrepareAGPRAllocPass(), PMW);
}
diff --git a/llvm/lib/Target/AMDGPU/CMakeLists.txt b/llvm/lib/Target/AMDGPU/CMakeLists.txt
index 632af3aadaf93..1f0b65a4cf660 100644
--- a/llvm/lib/Target/AMDGPU/CMakeLists.txt
+++ b/llvm/lib/Target/AMDGPU/CMakeLists.txt
@@ -62,6 +62,7 @@ add_llvm_target(AMDGPUCodeGen
AMDGPUHazardLatency.cpp
AMDGPUHSAMetadataStreamer.cpp
AMDGPUInsertDelayAlu.cpp
+ AMDGPUInsertWaterfall.cpp
AMDGPUInstCombineIntrinsic.cpp
AMDGPUUniformIntrinsicCombine.cpp
AMDGPUInstrInfo.cpp
@@ -165,7 +166,6 @@ add_llvm_target(AMDGPUCodeGen
SIFrameLowering.cpp
SIInsertHardClauses.cpp
SIInsertWaitcnts.cpp
- SIInsertWaterfall.cpp
SIInstrInfo.cpp
SIISelLowering.cpp
SILateBranchLowering.cpp
>From a4d4ada2875c30ef6dd4b7cf865b00a040ab1560 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Thu, 16 Apr 2026 14:38:00 +0100
Subject: [PATCH 03/35] Fix test
llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
---
llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll b/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
index df57d55e2f674..367e65fb68045 100644
--- a/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
+++ b/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
@@ -6,7 +6,7 @@ define amdgpu_ps <4 x float> @test_waterfall_same_index2(<8 x i32> addrspace(4)*
; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 [[INDEX:%.*]])
; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 [[WF_TOKEN1]], i32 [[INDEX]])
; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
-; CHECK-NEXT: [[PTR:%.*]] = getelementptr <8 x i32>, ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
+; CHECK-NEXT: [[PTR:%.*]] = getelementptr [32 x i8], ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
; CHECK-NEXT: [[RSRC:%.*]] = load <8 x i32>, ptr addrspace(4) [[PTR]], align 32
; CHECK-NEXT: [[R:%.*]] = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32.v8i32.v4i32(i32 15, float [[S:%.*]], <8 x i32> [[RSRC]], <4 x i32> [[SAMP:%.*]], i1 false, i32 0, i32 0)
; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 [[WF_TOKEN1]], <4 x float> [[R]])
@@ -27,7 +27,7 @@ define amdgpu_ps <4 x float> @test_waterfall_same_index3(<8 x i32> addrspace(4)*
; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 [[INDEX:%.*]])
; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 [[WF_TOKEN1]], i32 [[INDEX]])
; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
-; CHECK-NEXT: [[PTR:%.*]] = getelementptr <8 x i32>, ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
+; CHECK-NEXT: [[PTR:%.*]] = getelementptr [32 x i8], ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
; CHECK-NEXT: [[RSRC:%.*]] = load <8 x i32>, ptr addrspace(4) [[PTR]], align 32
; CHECK-NEXT: [[R:%.*]] = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32.v8i32.v4i32(i32 15, float [[S:%.*]], <8 x i32> [[RSRC]], <4 x i32> [[SAMP:%.*]], i1 false, i32 0, i32 0)
; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 [[WF_TOKEN1]], <4 x float> [[R]])
@@ -50,7 +50,7 @@ define amdgpu_ps <4 x float> @test_waterfall_same_index_aba(<8 x i32> addrspace(
; CHECK-NEXT: [[WF_TOKEN2:%.*]] = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 [[WF_TOKEN1]], i32 [[INDEX2:%.*]])
; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 [[WF_TOKEN2]], i32 [[INDEX2]])
; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
-; CHECK-NEXT: [[PTR:%.*]] = getelementptr <8 x i32>, ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
+; CHECK-NEXT: [[PTR:%.*]] = getelementptr [32 x i8], ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
; CHECK-NEXT: [[RSRC:%.*]] = load <8 x i32>, ptr addrspace(4) [[PTR]], align 32
; CHECK-NEXT: [[R:%.*]] = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32.v8i32.v4i32(i32 15, float [[S:%.*]], <8 x i32> [[RSRC]], <4 x i32> [[SAMP:%.*]], i1 false, i32 0, i32 0)
; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 [[WF_TOKEN2]], <4 x float> [[R]])
>From 59baf814ff9d42d506633fd5423478e9e6348f19 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Fri, 17 Apr 2026 08:41:33 +0100
Subject: [PATCH 04/35] Replace "undef" with "poison" or "dummy" arguments in a
new test.
---
.../CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll | 58 +++++++++----------
1 file changed, 29 insertions(+), 29 deletions(-)
diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
index 4ceb4245c77e9..755a730edab65 100644
--- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
+++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
@@ -12,7 +12,7 @@
; RUN: llc -global-isel=0 -march=amdgcn -mcpu=gfx1200 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX12,GFX12-SDAG %s
; RUN: llc -global-isel=1 -march=amdgcn -mcpu=gfx1200 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX12,GFX12-GISEL %s
- at Lds = addrspace(3) global [16384 x i32] undef
+ at Lds = addrspace(3) global [16384 x i32] poison
define amdgpu_ps void @test_waterfall_readlane(i32 addrspace(1)* inreg %out, <2 x i32> addrspace(1)* inreg %in, i32 %tid) #1 {
; VI-SDAG-LABEL: test_waterfall_readlane:
@@ -2265,7 +2265,7 @@ define amdgpu_ps <4 x float> @test_waterfall_non_uniform_img_multi_rl(<8 x i32>
ret <4 x float> %r1
}
-define amdgpu_ps <4 x float> @test_waterfall_non_uni_img_2_idx(<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %samp_in, i32 %index1, i32 %index2, float %s) #1 {
+define amdgpu_ps <4 x float> @test_waterfall_non_uni_img_2_idx(<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %samp_in, i32 %index1, i32 %index2, float %s, <2 x i32> %dummy) #1 {
; VI-SDAG-LABEL: test_waterfall_non_uni_img_2_idx:
; VI-SDAG: ; %bb.0:
; VI-SDAG-NEXT: s_mov_b64 s[4:5], exec
@@ -2743,7 +2743,7 @@ define amdgpu_ps <4 x float> @test_waterfall_non_uni_img_2_idx(<8 x i32> addrspa
; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
- %t_idx = insertelement <2 x i32> undef, i32 %index1, i32 0
+ %t_idx = insertelement <2 x i32> %dummy, i32 %index1, i32 0
%combined_idx = insertelement <2 x i32> %t_idx, i32 %index2, i32 1
%wf_token = call i32 @llvm.amdgcn.waterfall.begin.v2i32(i32 0, <2 x i32> %combined_idx)
%s_c_idx = call <2 x i32> @llvm.amdgcn.waterfall.readfirstlane.v2i32.v2i32(i32 %wf_token, <2 x i32> %combined_idx)
@@ -3850,7 +3850,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
ret <4 x float> %r1
}
-define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %samp_in, i32 %index, float %s, i32 inreg %val) #1 {
+define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %samp_in, i32 %index, float %s, i32 inreg %val, float %dummy_f0, float %dummy_f1, float %dummy_f2) #1 {
; VI-LABEL: test_waterfall_sample_with_kill:
; VI: ; %bb.0:
; VI-NEXT: s_mov_b64 s[6:7], exec
@@ -3872,7 +3872,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; VI-NEXT: s_load_dwordx4 s[20:23], s[20:21], 0x0
; VI-NEXT: ; implicit-def: $vgpr0
; VI-NEXT: s_waitcnt lgkmcnt(0)
-; VI-NEXT: image_sample v[2:5], v1, s[12:19], s[20:23] dmask:0xf
+; VI-NEXT: image_sample v[5:8], v1, s[12:19], s[20:23] dmask:0xf
; VI-NEXT: ; implicit-def: $vgpr1
; VI-NEXT: s_xor_b64 exec, exec, s[24:25]
; VI-NEXT: s_cbranch_execnz .LBB10_1
@@ -3880,7 +3880,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; VI-NEXT: s_mov_b64 exec, s[8:9]
; VI-NEXT: s_and_b64 exec, exec, s[6:7]
; VI-NEXT: s_waitcnt vmcnt(0)
-; VI-NEXT: v_cmp_gt_f32_e32 vcc, 0, v2
+; VI-NEXT: v_cmp_gt_f32_e32 vcc, 0, v5
; VI-NEXT: s_and_saveexec_b64 s[0:1], vcc
; VI-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
; VI-NEXT: s_cbranch_execz .LBB10_5
@@ -3920,7 +3920,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX9-NEXT: s_load_dwordx4 s[20:23], s[26:27], 0x0
; GFX9-NEXT: ; implicit-def: $vgpr0
; GFX9-NEXT: s_waitcnt lgkmcnt(0)
-; GFX9-NEXT: image_sample v[2:5], v1, s[12:19], s[20:23] dmask:0xf
+; GFX9-NEXT: image_sample v[5:8], v1, s[12:19], s[20:23] dmask:0xf
; GFX9-NEXT: ; implicit-def: $vgpr1
; GFX9-NEXT: s_xor_b64 exec, exec, s[24:25]
; GFX9-NEXT: s_cbranch_execnz .LBB10_1
@@ -3928,7 +3928,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX9-NEXT: s_mov_b64 exec, s[8:9]
; GFX9-NEXT: s_and_b64 exec, exec, s[6:7]
; GFX9-NEXT: s_waitcnt vmcnt(0)
-; GFX9-NEXT: v_cmp_gt_f32_e32 vcc, 0, v2
+; GFX9-NEXT: v_cmp_gt_f32_e32 vcc, 0, v5
; GFX9-NEXT: s_and_saveexec_b64 s[0:1], vcc
; GFX9-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
; GFX9-NEXT: s_cbranch_execz .LBB10_5
@@ -3968,7 +3968,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX10-32-NEXT: s_load_dwordx8 s[12:19], s[10:11], 0x0
; GFX10-32-NEXT: s_load_dwordx4 s[20:23], s[24:25], 0x0
; GFX10-32-NEXT: s_waitcnt lgkmcnt(0)
-; GFX10-32-NEXT: image_sample v[2:5], v1, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-32-NEXT: image_sample v[5:8], v1, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
; GFX10-32-NEXT: s_andn2_wrexec_b32 s8, s8
; GFX10-32-NEXT: ; implicit-def: $vgpr0
; GFX10-32-NEXT: ; implicit-def: $vgpr1
@@ -3978,7 +3978,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX10-32-NEXT: s_mov_b32 exec_lo, s7
; GFX10-32-NEXT: s_and_b32 exec_lo, exec_lo, s6
; GFX10-32-NEXT: s_waitcnt vmcnt(0)
-; GFX10-32-NEXT: v_cmp_gt_f32_e32 vcc_lo, 0, v2
+; GFX10-32-NEXT: v_cmp_gt_f32_e32 vcc_lo, 0, v5
; GFX10-32-NEXT: s_and_saveexec_b32 s0, vcc_lo
; GFX10-32-NEXT: s_xor_b32 s0, exec_lo, s0
; GFX10-32-NEXT: s_cbranch_execz .LBB10_5
@@ -4018,7 +4018,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX10-64-NEXT: s_load_dwordx8 s[12:19], s[24:25], 0x0
; GFX10-64-NEXT: s_load_dwordx4 s[20:23], s[26:27], 0x0
; GFX10-64-NEXT: s_waitcnt lgkmcnt(0)
-; GFX10-64-NEXT: image_sample v[2:5], v1, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX10-64-NEXT: image_sample v[5:8], v1, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
; GFX10-64-NEXT: s_andn2_wrexec_b64 s[10:11], s[10:11]
; GFX10-64-NEXT: ; implicit-def: $vgpr0
; GFX10-64-NEXT: ; implicit-def: $vgpr1
@@ -4028,7 +4028,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX10-64-NEXT: s_mov_b64 exec, s[8:9]
; GFX10-64-NEXT: s_and_b64 exec, exec, s[6:7]
; GFX10-64-NEXT: s_waitcnt vmcnt(0)
-; GFX10-64-NEXT: v_cmp_gt_f32_e32 vcc, 0, v2
+; GFX10-64-NEXT: v_cmp_gt_f32_e32 vcc, 0, v5
; GFX10-64-NEXT: s_and_saveexec_b64 s[0:1], vcc
; GFX10-64-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
; GFX10-64-NEXT: s_cbranch_execz .LBB10_5
@@ -4072,7 +4072,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX1150-NEXT: s_load_b256 s[12:19], s[12:13], 0x0
; GFX1150-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
; GFX1150-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
-; GFX1150-NEXT: image_sample v[2:5], v1, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX1150-NEXT: image_sample v[5:8], v1, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
; GFX1150-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
; GFX1150-NEXT: ; implicit-def: $vgpr0
; GFX1150-NEXT: ; implicit-def: $vgpr1
@@ -4083,7 +4083,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX1150-NEXT: s_and_b64 exec, exec, s[6:7]
; GFX1150-NEXT: s_mov_b64 s[0:1], exec
; GFX1150-NEXT: s_waitcnt vmcnt(0)
-; GFX1150-NEXT: v_cmpx_gt_f32_e32 0, v2
+; GFX1150-NEXT: v_cmpx_gt_f32_e32 0, v5
; GFX1150-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
; GFX1150-NEXT: s_cbranch_execz .LBB10_5
; GFX1150-NEXT: ; %bb.3: ; %.kill
@@ -4132,7 +4132,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX12-SDAG-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
-; GFX12-SDAG-NEXT: image_sample v[2:5], v1, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-SDAG-NEXT: image_sample v[5:8], v1, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr0
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr1
@@ -4143,7 +4143,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX12-SDAG-NEXT: s_and_b64 exec, exec, s[6:7]
; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
-; GFX12-SDAG-NEXT: v_cmpx_gt_f32_e32 0, v2
+; GFX12-SDAG-NEXT: v_cmpx_gt_f32_e32 0, v5
; GFX12-SDAG-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
; GFX12-SDAG-NEXT: s_cbranch_execz .LBB10_5
; GFX12-SDAG-NEXT: ; %bb.3: ; %.kill
@@ -4189,7 +4189,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX12-GISEL-NEXT: s_load_b128 s[20:23], s[20:21], 0x0
; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
-; GFX12-GISEL-NEXT: image_sample v[2:5], v1, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
+; GFX12-GISEL-NEXT: image_sample v[5:8], v1, s[12:19], s[20:23] dmask:0xf dim:SQ_RSRC_IMG_1D
; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr0
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr1
@@ -4200,7 +4200,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
-; GFX12-GISEL-NEXT: v_cmpx_gt_f32_e32 0, v2
+; GFX12-GISEL-NEXT: v_cmpx_gt_f32_e32 0, v5
; GFX12-GISEL-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
; GFX12-GISEL-NEXT: s_cbranch_execz .LBB10_5
; GFX12-GISEL-NEXT: ; %bb.3: ; %.kill
@@ -4236,7 +4236,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
br label %.exit
.exit:
- call void @llvm.amdgcn.exp.f32(i32 immarg 0, i32 immarg 1, float 0.000000e+00, float undef, float undef, float undef, i1 immarg true, i1 immarg true)
+ call void @llvm.amdgcn.exp.f32(i32 immarg 0, i32 immarg 1, float 0.000000e+00, float %dummy_f0, float %dummy_f1, float %dummy_f2, i1 immarg true, i1 immarg true)
ret void
}
@@ -9468,7 +9468,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX12-GISEL-NEXT: ; return to shader part epilog
<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %s_in,
i32 %idx1, i32 %idx2, i32 inreg %idx3, i32 inreg%idx4,
- i32 %s_idx, i32 %s_idx2) #1 {
+ i32 %s_idx, i32 %s_idx2, { <4 x float>, <4 x float> } %dummy) #1 {
%rptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
@@ -9487,7 +9487,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
%r2 = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc1, <4 x i32> %s_srsrc1, i1 false, i32 0, i32 0)
%r3 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok3, <4 x float> %r2)
- %insert = insertvalue { <4 x float>, <4 x float> } undef, <4 x float> %r1, 0
+ %insert = insertvalue { <4 x float>, <4 x float> } %dummy, <4 x float> %r1, 0
%insert1 = insertvalue { <4 x float>, <4 x float> } %insert, <4 x float> %r3, 1
ret {<4 x float> , <4 x float>} %insert1
}
@@ -10737,7 +10737,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
; GFX12-GISEL-NEXT: ; return to shader part epilog
<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %s_in,
- i32 %idx1, i32 %idx2, i32 %s_idx, i32 %s_idx2) #1 {
+ i32 %idx1, i32 %idx2, i32 %s_idx, i32 %s_idx2, { <4 x float>, <4 x float> } %dummy) #1 {
%rptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
@@ -10751,7 +10751,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
%r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
%r3 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r2)
- %insert = insertvalue { <4 x float>, <4 x float> } undef, <4 x float> %r1, 0
+ %insert = insertvalue { <4 x float>, <4 x float> } %dummy, <4 x float> %r1, 0
%insert1 = insertvalue { <4 x float>, <4 x float> } %insert, <4 x float> %r3, 1
ret {<4 x float> , <4 x float>} %insert1
}
@@ -11346,7 +11346,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %s_in,
- i32 %idx1, i32 %idx2, i32 %s_idx, i32 %s_idx2) #1 {
+ i32 %idx1, i32 %idx2, i32 %s_idx, i32 %s_idx2, { <4 x float>, <4 x float> } %dummy) #1 {
%tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
%tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok, i32 %idx2)
%widx0 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %tok1, i32 %s_idx)
@@ -11363,7 +11363,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
%r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
%r3 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r2)
- %insert = insertvalue { <4 x float>, <4 x float> } undef, <4 x float> %r1, 0
+ %insert = insertvalue { <4 x float>, <4 x float> } %dummy, <4 x float> %r1, 0
%insert1 = insertvalue { <4 x float>, <4 x float> } %insert, <4 x float> %r3, 1
ret {<4 x float> , <4 x float>} %insert1
}
@@ -11782,7 +11782,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
- i32 %idx, <4 x i32> %s_idx, i32 %v_inp) #1 {
+ i32 %idx, <4 x i32> %s_idx, i32 %v_inp, { <4 x float>, float } %dummy) #1 {
%tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx)
%widx0 = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok, <4 x i32> %s_idx)
%val = call { <4 x float>, i32 } @llvm.amdgcn.struct.buffer.load.format.sl_v4f32i32s(<4 x i32> %widx0, i32 %v_inp, i32 0, i32 0, i32 0)
@@ -11792,7 +11792,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
%r1 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %tok, i32 %tfe)
%r1.float = bitcast i32 %r1 to float
- %insert = insertvalue { <4 x float>, float } undef, <4 x float> %r0, 0
+ %insert = insertvalue { <4 x float>, float } %dummy, <4 x float> %r0, 0
%insert1 = insertvalue { <4 x float>, float } %insert, float %r1.float, 1
ret {<4 x float> , float} %insert1
}
@@ -12078,7 +12078,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
- i32 inreg %idx, <4 x i32> %s_idx, i32 %v_inp) #1 {
+ i32 inreg %idx, <4 x i32> %s_idx, i32 %v_inp, { <4 x float>, float } %dummy) #1 {
%tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx)
%widx0 = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok, <4 x i32> %s_idx)
%val = call { <4 x float>, i32 } @llvm.amdgcn.struct.buffer.load.format.sl_v4f32i32s(<4 x i32> %widx0, i32 %v_inp, i32 0, i32 0, i32 0)
@@ -12088,7 +12088,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
%r1 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %tok, i32 %tfe)
%r1.float = bitcast i32 %r1 to float
- %insert = insertvalue { <4 x float>, float } undef, <4 x float> %r0, 0
+ %insert = insertvalue { <4 x float>, float } %dummy, <4 x float> %r0, 0
%insert1 = insertvalue { <4 x float>, float } %insert, float %r1.float, 1
ret {<4 x float> , float} %insert1
}
>From 13f09305f230c455d5fcd920a22446221bec8704 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Fri, 17 Apr 2026 16:42:37 +0100
Subject: [PATCH 05/35] Improve documentation and comments
---
llvm/docs/AMDGPUUsage.rst | 192 ++++++++++++++---------
llvm/include/llvm/IR/IntrinsicsAMDGPU.td | 4 +-
2 files changed, 116 insertions(+), 80 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 2e8b535a66486..b61fae2c20ce6 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -1893,84 +1893,6 @@ The AMDGPU backend implements the following LLVM IR intrinsics.
* :ref:`Synchronization Scope<amdgpu-intrinsics-syncscope-metadata-operand>`.
Note that the scope used must ensure that the L2 cache will be hit.
- llvm.amdgcn.ds.atomic.barrier.arrive.rtn.b64 Available starting GFX12.5.
- Corresponds to ``ds_atomic_barrier_arrive_rtn_b64``.
-
- For the purposes of the memory model, this is a monotonic atomic
- read-modify-write operation in the local address space.
-
- This intrinsic has 2 operands:
-
- * Local pointer to the LDS barrier data.
- * Update value; the pending count of the barrier will be
- decremented by this value (generally 1).
-
- Returns the LDS barrier data as it was before this operation
- was executed.
-
- llvm.amdgcn.ds.atomic.async.barrier.arrive.b64 Available starting GFX12.5.
- Corresponds to ``ds_atomic_async_barrier_arrive_b64``.
-
- For the purposes of the memory model, this is an asynchronous
- monotonic atomic read-modify-write operation in the local
- address space.
-
- This intrinsic has 1 operand:
-
- * Local pointer to the LDS barrier data.
- llvm.amdgcn.waterfall.begin Marks the beginning of a waterfall region of code.
-
- The compiler generates a waterfall loop around the region.
- A waterfall loop handles the case where an operation that requires
- a uniform operand (e.g., held in an SGPR) is applied to a divergent operand
- (held in a VGPR, with values varying per lane).
- Each iteration of the waterfall loop activates a subset of lanes that
- share the same value of the VGPR (the value in the first active lane).
- The operation is then executed using that value as the uniform operand.
- If the VGPR is already uniform, the waterfall loop executes only once.
- The worst case for a waterfall loop is one iteration per lane (all lanes
- have different values of the VGPR), but it is not common in practice.
-
- The intrinsic takes a previous token
- (``i32``; use a null/zero value if this is the first ``waterfall.begin`` in
- a waterfall group) and a VGPR.
-
- The intrinsic returns a new token that must be threaded through the
- corresponding ``waterfall.readfirstlane`` and ``waterfall.end`` or
- ``waterfall.last_use`` intrinsics, forming a waterfall group of intrinsics
- that together define a waterfall region.
-
- All intrinsics in a waterfall group must reside in the same basic block.
-
- Multiple ``waterfall.begin`` intrinsics can be chained by
- passing the token of the preceding ``waterfall.begin`` as the first argument.
- This allows a front-end to create one waterfall loop for
- multiple non-uniform values.
- Later compiler passes may remove values determined as uniform.
- The final token is used for other waterfall intrinsics in the same group.
-
- llvm.amdgcn.waterfall.readfirstlane Reads the first active lane's value of the VGPR and returns it as
- an SGPR for use within a waterfall region.
-
- Takes the ``i32`` token from the final ``waterfall.begin`` in the waterfall
- group and the VGPR. Returns the uniform (SGPR) result.
-
- If the VGPR is determined to be uniform at compile time, this intrinsic
- is optimized away (the input VGPR value is used directly).
-
- llvm.amdgcn.waterfall.end Marks the end of a waterfall region.
- Takes the ``i32`` token from the final ``waterfall.begin``
-
- llvm.amdgcn.waterfall.last_use Variant of ``waterfall.end`` for values whose last use is in a
- non-defining operation such as a store. Marks that the use of the value
- constitutes the end of the waterfall region.
-
- llvm.amdgcn.waterfall.last_use_vgpr Variant of ``waterfall.last_use`` for values that remain in a VGPR.
-
- llvm.amdgcn.waterfall.loop_end Inserted later by the compiler to be used with ```waterfall.last_use*``
- to mark the loop-end point for special
- handling such as SCC clobber tracking.
-
============================================== ==========================================================
.. TODO::
@@ -2068,6 +1990,120 @@ enabled this should map to ``scope:SCOPE_SE``.
+--------+--------------------------+---------------+---------------+---------------+---------------+---------------+
**Note:** Cache control bits for Store are not affected by WGP mode.
+'``llvm.amdgcn.waterfall``' Intrinsics
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``llvm.amdgcn.waterfall`` :ref:`family of intrinsics<amdgpu-waterfall-intrinsics-table>`
+describe how to generate a waterfall loop around a region of code.
+
+:Background:
+
+A waterfall loop handles the case where an operation that requires a uniform
+operand (e.g., in an SGPR) is applied to a non-uniform operand (held in a
+VGPR, with values varying per lane).
+
+Each iteration of the waterfall loop activates a subset of lanes that share
+the same value of the non-uniform operand (the value in the first active
+lane). The operation is then executed using that value as the uniform
+operand.
+
+If the operand is already uniform, the waterfall loop executes only
+once. The worst case for a waterfall loop is one iteration per lane (all
+lanes have different values of the operand), but this is not common in
+practice.
+
+:Motivation:
+
+The ``llvm.amdgc.waterfall.*`` intrinsics provide a practical way for a
+frontend to combine several operations into a single waterfall loop with an
+efficient iteration strategy over multiple non-uniform operands.
+
+In particular, a frontend can specify a subset of non-uniform operands that
+is sufficient to use as the "index" to loop over, i.e., lanes that have the
+same values for the specified operands must have the same value for all
+other non-uniform operands that need to be uniform. It is hard for the
+backend to find a fitting subset of non-uniform operands, but it can be easy
+for a frontend.
+
+:Implementation:
+
+A group of waterfall intrinsics that depend on the same token defines a
+single waterfall loop. The group identifies a region of code, the
+corresponding non-uniform operands that need to be uniform, and a subset of
+these operands that is sufficient to use as the "index" of the loop.
+
+A waterfall group must contain at least one
+``waterfall.begin``, at least one ``waterfall.readfirstlane``, and at least one
+of ``waterfall.end`` or ``waterfall.loop_end`` intrinsic.
+A group can contain more than one of each of the waterfall intrinsics.
+The token on the final
+``waterfall.begin`` must be used for other waterfall intrinsics in the same
+group. The waterfall loop will enclose all instructions from the earliest
+``waterfall.begin`` to the latest ``waterfall.end`` or ``waterfall.loop_end``
+intrinsic in the group.
+
+Each waterfall group must be contained within a single basic block.
+A single basic block can contain more than one waterfall group.
+
+Later compiler passes can remove operands determined as uniform.
+If all operands are uniform, the compiler will not insert the waterfall loop.
+
+.. table:: AMDGPU Waterfall Intrinsics
+ :name: amdgpu-waterfall-intrinsics-table
+
+ ============================================== =========================================================================
+ LLVM Intrinsic Description
+ ============================================== =========================================================================
+ ``llvm.amdgcn.waterfall.begin`` Marks the beginning of a waterfall region of code.
+ Specifies a non-uniform operand that needs to be uniform in the region
+ and should be used as part of the index for the waterfall loop to iterate
+ over.
+
+ Multiple non-uniform operands of different types can be used together as
+ an index by threading tokens through multiple ``waterfall.begin`` intrinsics::
+
+ %tok0 = llvm.amdgcn.waterfall.begin(i32 0, i32 %idx0)
+ %tok1 = llvm.amdgcn.waterfall.begin(i32 %tok0, i32 %idx1)
+ ...
+
+ The intrinsic takes a token
+ (use a null/zero value if this is the first ``waterfall.begin`` in a
+ waterfall group) and an operand.
+
+ The intrinsic returns a new token that must be threaded through the
+ corresponding ``waterfall.readfirstlane``, ``waterfall.end`` or
+ ``waterfall.last.use**`` intrinsics.
+
+ ``llvm.amdgcn.waterfall.readfirstlane`` Reads the first active lane's value of the non-uniform operand and
+ returns it for use within a waterfall region.
+
+ Takes a token from the final ``waterfall.begin`` in the waterfall
+ group and a non-uniform operand. Returns the uniform result.
+
+ Every non-inform operand from ``waterfall.begin`` must have a
+ corresponding ``waterfall.readfirstlane`` intrinsic in the group.
+ Additional ``waterfall.readfirstlane`` intrinsics in the group can be used
+ to specify non-uniform operands that are not part of the index,
+ but need to be uniform in the waterfall loop, as long as they satisfy
+ the correctness requirement (i.e., they have the same values in lanes
+ where the index has the same values).
+
+ ``llvm.amdgcn.waterfall.end`` Marks the end of a waterfall region.
+ Takes the token from the final ``waterfall.begin`` in the group.
+
+ ``llvm.amdgcn.waterfall.last.use`` Variant of ``waterfall.end`` for values whose last use is in a
+ non-defining operation such as a store. Marks that the use of the value
+ constitutes the end of the waterfall region.
+
+ ``llvm.amdgcn.waterfall.last.use.vgpr`` Variant of ``waterfall.last_use`` for values that remain in a VGPR.
+
+ ``llvm.amdgcn.waterfall.loop.end`` Inserted later by the compiler to be used with ``waterfall.last_use*``
+ to mark the loop-end point for special
+ handling such as SCC clobber tracking.
+ This intrinsic should not be generated by a frontend.
+
+
+ ============================================== =========================================================================
'``llvm.amdgcn.cooperative.atomic``' Intrinsics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
index c49535305e651..e9ad4df3079ea 100644
--- a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
+++ b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
@@ -2723,8 +2723,8 @@ def int_amdgcn_cs_chain:
// Waterfall intrinsics are used to mark a region as requiring waterfall loops to
// activate and deactivate lanes in a loop (and sometimes transform VGPR values
// into SGPR values).
-// Best case (uniform index in VGPR) the loop will execute once, worst case (all
-// values in the index are different) the loop will execute wave-size times.
+// Best case (all lanes have the same values) the loop will execute once, worst case (all lanes
+// have different values) the loop will execute wave-size times.
// The waterfall.begin intrinsic returns a new token that must be threaded through the
// corresponding waterfall.readfirstlane and waterfall.end or
// waterfall.last_use intrinsics, forming a waterfall group of intrinsics that together
>From 011f99ab3f22fc687ed2d79e1b5f818d6ffff76d Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Mon, 20 Apr 2026 14:22:00 +0100
Subject: [PATCH 06/35] Fix tests broken by renaming the new pass
---
...terfall-licm.ll => amdgpu-insert-waterfall-licm.ll} | 0
...nsert-waterfall.mir => amdgpu-insert-waterfall.mir} | 2 +-
llvm/test/CodeGen/AMDGPU/llc-pipeline.ll | 10 +++++-----
3 files changed, 6 insertions(+), 6 deletions(-)
rename llvm/test/CodeGen/AMDGPU/{si-insert-waterfall-licm.ll => amdgpu-insert-waterfall-licm.ll} (100%)
rename llvm/test/CodeGen/AMDGPU/{si-insert-waterfall.mir => amdgpu-insert-waterfall.mir} (98%)
diff --git a/llvm/test/CodeGen/AMDGPU/si-insert-waterfall-licm.ll b/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall-licm.ll
similarity index 100%
rename from llvm/test/CodeGen/AMDGPU/si-insert-waterfall-licm.ll
rename to llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall-licm.ll
diff --git a/llvm/test/CodeGen/AMDGPU/si-insert-waterfall.mir b/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall.mir
similarity index 98%
rename from llvm/test/CodeGen/AMDGPU/si-insert-waterfall.mir
rename to llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall.mir
index 30dedf0a89f18..16ecfafe49455 100644
--- a/llvm/test/CodeGen/AMDGPU/si-insert-waterfall.mir
+++ b/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall.mir
@@ -1,5 +1,5 @@
# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
-# RUN: llc -mtriple=amdgcn--amdpal -march=amdgcn -mcpu=gfx1030 -run-pass=si-insert-waterfall -o - %s | FileCheck %s
+# RUN: llc -mtriple=amdgcn--amdpal -march=amdgcn -mcpu=gfx1030 -run-pass=amdgpu-insert-waterfall -o - %s | FileCheck %s
# The test demonstrates a waterfall loop expansion in si-insert-waterfall pass.
---
diff --git a/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll b/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll
index e5ae5fdaff6ab..8c07cbbad64ec 100644
--- a/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll
+++ b/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll
@@ -109,7 +109,7 @@
; GCN-O0-NEXT: Finalize ISel and expand pseudo-instructions
; GCN-O0-NEXT: Local Stack Slot Allocation
; GCN-O0-NEXT: Register Usage Information Propagation
-; GCN-O0-NEXT: SI Insert waterfalls
+; GCN-O0-NEXT: AMDGPU Insert waterfalls
; GCN-O0-NEXT: Eliminate PHI nodes for register allocation
; GCN-O0-NEXT: SI Lower control flow pseudo instructions
; GCN-O0-NEXT: Two-Address instruction pass
@@ -344,7 +344,7 @@
; GCN-O1-NEXT: Remove dead machine instructions
; GCN-O1-NEXT: SI Shrink Instructions
; GCN-O1-NEXT: Register Usage Information Propagation
-; GCN-O1-NEXT: SI Insert waterfalls
+; GCN-O1-NEXT: AMDGPU Insert waterfalls
; GCN-O1-NEXT: AMDGPU Prepare AGPR Alloc
; GCN-O1-NEXT: Detect Dead Lanes
; GCN-O1-NEXT: Remove dead machine instructions
@@ -665,7 +665,7 @@
; GCN-O1-OPTS-NEXT: Remove dead machine instructions
; GCN-O1-OPTS-NEXT: SI Shrink Instructions
; GCN-O1-OPTS-NEXT: Register Usage Information Propagation
-; GCN-O1-OPTS-NEXT: SI Insert waterfalls
+; GCN-O1-OPTS-NEXT: AMDGPU Insert waterfalls
; GCN-O1-OPTS-NEXT: AMDGPU Prepare AGPR Alloc
; GCN-O1-OPTS-NEXT: Detect Dead Lanes
; GCN-O1-OPTS-NEXT: Remove dead machine instructions
@@ -991,7 +991,7 @@
; GCN-O2-NEXT: Remove dead machine instructions
; GCN-O2-NEXT: SI Shrink Instructions
; GCN-O2-NEXT: Register Usage Information Propagation
-; GCN-O2-NEXT: SI Insert waterfalls
+; GCN-O2-NEXT: AMDGPU Insert waterfalls
; GCN-O2-NEXT: AMDGPU Prepare AGPR Alloc
; GCN-O2-NEXT: Detect Dead Lanes
; GCN-O2-NEXT: Remove dead machine instructions
@@ -1331,7 +1331,7 @@
; GCN-O3-NEXT: Remove dead machine instructions
; GCN-O3-NEXT: SI Shrink Instructions
; GCN-O3-NEXT: Register Usage Information Propagation
-; GCN-O3-NEXT: SI Insert waterfalls
+; GCN-O3-NEXT: AMDGPU Insert waterfalls
; GCN-O3-NEXT: AMDGPU Prepare AGPR Alloc
; GCN-O3-NEXT: Detect Dead Lanes
; GCN-O3-NEXT: Remove dead machine instructions
>From ff6f27a87a1faaee7fa93155af6d9302c14ab2ec Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 21 Apr 2026 15:47:47 +0100
Subject: [PATCH 07/35] Format
---
llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp | 4 ++--
llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp | 4 ++--
llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp | 2 +-
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
index a7e1d97c36b7f..6f8075e5458e4 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
@@ -1,4 +1,4 @@
-//===- AMDGPUInsertWaterfall.cpp - insert waterfall loops at intrinsic markers -===//
+//===- AMDGPUInsertWaterfall.cpp - insert waterfall loops around markers -====//
//
// The LLVM Compiler Infrastructure
//
@@ -8,7 +8,7 @@
//===----------------------------------------------------------------------===//
//
/// \file
-/// Replace 3 intrinsics used to mark waterfall regions with actual waterfall
+/// Replace intrinsics used to mark waterfall regions with actual waterfall
/// loops. This is done at MachineIR level rather than LLVM-IR due to the use of
/// exec mask in this operation.
///
diff --git a/llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp b/llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp
index c612f90250992..6dfa795d1b7f7 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp
@@ -98,8 +98,8 @@ class AMDGPURegBankCombinerImpl : public Combiner {
// Combine support to remove amdgcn_waterfall_readfirstlane or
// amdgcn_waterfall_begin intrinsics if the index is determined to be
- // uniform. The AMDGPUInsertWaterfall pass can handle their removal and in some
- // cases remove the waterfall altogether
+ // uniform. The AMDGPUInsertWaterfall pass can handle their removal and in
+ // some cases remove the waterfall altogether
bool matchRmUniformWF(MachineInstr &MI,
RmUniformWFMatchInfo &MatchInfo) const;
void applyRmUniformWF(MachineInstr &MI,
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
index f12d9a9e33921..2368d7991de4c 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
@@ -25,6 +25,7 @@
#include "AMDGPUHazardLatency.h"
#include "AMDGPUIGroupLP.h"
#include "AMDGPUISelDAGToDAG.h"
+#include "AMDGPUInsertWaterfall.h"
#include "AMDGPULowerVGPREncoding.h"
#include "AMDGPUMacroFusion.h"
#include "AMDGPUNextUseAnalysis.h"
@@ -53,7 +54,6 @@
#include "SIFixVGPRCopies.h"
#include "SIFoldOperands.h"
#include "SIFormMemoryClauses.h"
-#include "AMDGPUInsertWaterfall.h"
#include "SILoadStoreOptimizer.h"
#include "SILowerControlFlow.h"
#include "SILowerSGPRSpills.h"
>From eb0fde540ff7951af99f34ccd6a8bf9e9bd44b10 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Fri, 22 May 2026 12:30:26 +0100
Subject: [PATCH 08/35] Fix after a rebase
---
.../Target/AMDGPU/AMDGPUCodeGenPrepare.cpp | 2 +-
.../CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll | 20 +++++++++----------
2 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp b/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp
index 52e517dffd43e..383081f914068 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp
@@ -2098,7 +2098,7 @@ bool AMDGPUCodeGenPrepareImpl::visitIntrinsicInst(IntrinsicInst &I) {
}
bool AMDGPUCodeGenPrepareImpl::visitGetElementPtrInst(GetElementPtrInst &I) {
- if (!CurrentWaterfall || UA.isUniform(&I))
+ if (!CurrentWaterfall || UA.isUniformAtDef(&I))
return false;
if (I.getParent() != CurrentWaterfall->getParent())
return false;
diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
index 755a730edab65..d7d717c4409a2 100644
--- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
+++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
@@ -9787,8 +9787,8 @@ define amdgpu_cs void @ds_write_8(i8 %value, i32 %index) #1 {
;
; GFX9-SDAG-LABEL: ds_write_8:
; GFX9-SDAG: ; %bb.0: ; %.entry
-; GFX9-SDAG-NEXT: s_mov_b32 s0, Lds at abs32@lo
-; GFX9-SDAG-NEXT: v_lshl_add_u32 v1, v1, 2, s0
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v2, Lds at abs32@lo
+; GFX9-SDAG-NEXT: v_lshl_add_u32 v1, v1, 2, v2
; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
; GFX9-SDAG-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s0, v1
@@ -11005,10 +11005,10 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
;
; GFX10-32-GISEL-LABEL: test_waterfall_multi_end_1loop_rsrc_load_inside:
; GFX10-32-GISEL: ; %bb.0:
-; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v8, v0
-; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v9, v1
; GFX10-32-GISEL-NEXT: s_mov_b32 s4, exec_lo
; GFX10-32-GISEL-NEXT: s_wqm_b32 exec_lo, exec_lo
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v8, v0
+; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v9, v1
; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v10, v2
; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v11, v3
; GFX10-32-GISEL-NEXT: s_mov_b32 s6, exec_lo
@@ -11097,10 +11097,10 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
;
; GFX10-64-GISEL-LABEL: test_waterfall_multi_end_1loop_rsrc_load_inside:
; GFX10-64-GISEL: ; %bb.0:
-; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v8, v0
-; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v9, v1
; GFX10-64-GISEL-NEXT: s_mov_b64 s[4:5], exec
; GFX10-64-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v8, v0
+; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v9, v1
; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v10, v2
; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v11, v3
; GFX10-64-GISEL-NEXT: s_mov_b64 s[8:9], exec
@@ -11194,10 +11194,10 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
;
; GFX1150-GISEL-LABEL: test_waterfall_multi_end_1loop_rsrc_load_inside:
; GFX1150-GISEL: ; %bb.0:
-; GFX1150-GISEL-NEXT: v_mov_b32_e32 v8, v0
-; GFX1150-GISEL-NEXT: v_mov_b32_e32 v9, v1
; GFX1150-GISEL-NEXT: s_mov_b64 s[4:5], exec
; GFX1150-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v8, v0
+; GFX1150-GISEL-NEXT: v_mov_b32_e32 v9, v1
; GFX1150-GISEL-NEXT: v_mov_b32_e32 v10, v2
; GFX1150-GISEL-NEXT: v_mov_b32_e32 v11, v3
; GFX1150-GISEL-NEXT: s_mov_b64 s[8:9], exec
@@ -11295,10 +11295,10 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
;
; GFX12-GISEL-LABEL: test_waterfall_multi_end_1loop_rsrc_load_inside:
; GFX12-GISEL: ; %bb.0:
-; GFX12-GISEL-NEXT: v_mov_b32_e32 v8, v0
-; GFX12-GISEL-NEXT: v_mov_b32_e32 v9, v1
; GFX12-GISEL-NEXT: s_mov_b64 s[4:5], exec
; GFX12-GISEL-NEXT: s_wqm_b64 exec, exec
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v8, v0
+; GFX12-GISEL-NEXT: v_mov_b32_e32 v9, v1
; GFX12-GISEL-NEXT: v_mov_b32_e32 v10, v2
; GFX12-GISEL-NEXT: v_mov_b32_e32 v11, v3
; GFX12-GISEL-NEXT: s_mov_b64 s[8:9], exec
>From dca9bd8058fc357a70dfd149afecd266d1ce0272 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 26 May 2026 12:20:02 +0100
Subject: [PATCH 09/35] Address review comments about documentation
---
llvm/docs/AMDGPUUsage.rst | 57 ++++++++++++------------
llvm/include/llvm/IR/IntrinsicsAMDGPU.td | 18 ++++----
2 files changed, 38 insertions(+), 37 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index b61fae2c20ce6..da297bb0b2a76 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -1990,11 +1990,12 @@ enabled this should map to ``scope:SCOPE_SE``.
+--------+--------------------------+---------------+---------------+---------------+---------------+---------------+
**Note:** Cache control bits for Store are not affected by WGP mode.
+
'``llvm.amdgcn.waterfall``' Intrinsics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``llvm.amdgcn.waterfall`` :ref:`family of intrinsics<amdgpu-waterfall-intrinsics-table>`
-describe how to generate a waterfall loop around a region of code.
+describes how to generate a waterfall loop around a region of code.
:Background:
@@ -2014,34 +2015,40 @@ practice.
:Motivation:
-The ``llvm.amdgc.waterfall.*`` intrinsics provide a practical way for a
+The ``llvm.amdgcn.waterfall.*`` intrinsics provide a practical way for a
frontend to combine several operations into a single waterfall loop with an
efficient iteration strategy over multiple non-uniform operands.
-In particular, a frontend can specify a subset of non-uniform operands that
+In particular, a frontend can specify a set of values that
is sufficient to use as the "index" to loop over, i.e., lanes that have the
-same values for the specified operands must have the same value for all
-other non-uniform operands that need to be uniform. It is hard for the
-backend to find a fitting subset of non-uniform operands, but it can be easy
+same index values must have the same values for all
+non-uniform operands that need to be uniform. It is hard for the
+backend to find a fitting index, but it can be easy
for a frontend.
:Implementation:
A group of waterfall intrinsics that depend on the same token defines a
single waterfall loop. The group identifies a region of code, the
-corresponding non-uniform operands that need to be uniform, and a subset of
-these operands that is sufficient to use as the "index" of the loop.
+non-uniform operands that need to be made uniform, and the operands
+to use as the "index" of the loop.
A waterfall group must contain at least one
-``waterfall.begin``, at least one ``waterfall.readfirstlane``, and at least one
-of ``waterfall.end`` or ``waterfall.loop_end`` intrinsic.
+``waterfall.begin``, at least one of ``waterfall.readfirstlane``, and at least one
+of ``waterfall.end`` or ``waterfall.last.use`` intrinsic.
A group can contain more than one of each of the waterfall intrinsics.
The token on the final
``waterfall.begin`` must be used for other waterfall intrinsics in the same
group. The waterfall loop will enclose all instructions from the earliest
-``waterfall.begin`` to the latest ``waterfall.end`` or ``waterfall.loop_end``
+``waterfall.begin`` to the latest ``waterfall.end`` or ``waterfall.last.use``
intrinsic in the group.
+Each ``waterfall.begin`` specifies a value that is part of the index. Each
+``waterfall.readfirstlane`` specifies a non-uniform operand that needs to be
+uniform. The index can be disjoint from the non-uniform operands. For
+example, ``waterfall.begin`` can take an index while
+``waterfall.readfirstlane`` takes a descriptor derived from that index.
+
Each waterfall group must be contained within a single basic block.
A single basic block can contain more than one waterfall group.
@@ -2055,11 +2062,10 @@ If all operands are uniform, the compiler will not insert the waterfall loop.
LLVM Intrinsic Description
============================================== =========================================================================
``llvm.amdgcn.waterfall.begin`` Marks the beginning of a waterfall region of code.
- Specifies a non-uniform operand that needs to be uniform in the region
- and should be used as part of the index for the waterfall loop to iterate
- over.
+ Specifies a value to be used as (part of) the index for the waterfall
+ loop to iterate over.
- Multiple non-uniform operands of different types can be used together as
+ Multiple values of different types can be combined as
an index by threading tokens through multiple ``waterfall.begin`` intrinsics::
%tok0 = llvm.amdgcn.waterfall.begin(i32 0, i32 %idx0)
@@ -2072,21 +2078,16 @@ If all operands are uniform, the compiler will not insert the waterfall loop.
The intrinsic returns a new token that must be threaded through the
corresponding ``waterfall.readfirstlane``, ``waterfall.end`` or
- ``waterfall.last.use**`` intrinsics.
+ ``waterfall.last.use`` intrinsics.
- ``llvm.amdgcn.waterfall.readfirstlane`` Reads the first active lane's value of the non-uniform operand and
- returns it for use within a waterfall region.
+ ``llvm.amdgcn.waterfall.readfirstlane`` Reads the first active lane's value of a given non-uniform operand and
+ returns it as a uniform value for use within a waterfall region.
Takes a token from the final ``waterfall.begin`` in the waterfall
group and a non-uniform operand. Returns the uniform result.
- Every non-inform operand from ``waterfall.begin`` must have a
- corresponding ``waterfall.readfirstlane`` intrinsic in the group.
- Additional ``waterfall.readfirstlane`` intrinsics in the group can be used
- to specify non-uniform operands that are not part of the index,
- but need to be uniform in the waterfall loop, as long as they satisfy
- the correctness requirement (i.e., they have the same values in lanes
- where the index has the same values).
+ The correctness requirement is that lanes with the same index values
+ must have the same values for all ``waterfall.readfirstlane`` operands.
``llvm.amdgcn.waterfall.end`` Marks the end of a waterfall region.
Takes the token from the final ``waterfall.begin`` in the group.
@@ -2095,12 +2096,12 @@ If all operands are uniform, the compiler will not insert the waterfall loop.
non-defining operation such as a store. Marks that the use of the value
constitutes the end of the waterfall region.
- ``llvm.amdgcn.waterfall.last.use.vgpr`` Variant of ``waterfall.last_use`` for values that remain in a VGPR.
+ ``llvm.amdgcn.waterfall.last.use.vgpr`` Variant of ``waterfall.last.use`` for values that remain in a VGPR.
- ``llvm.amdgcn.waterfall.loop.end`` Inserted later by the compiler to be used with ``waterfall.last_use*``
+ ``llvm.amdgcn.waterfall.loop.end`` Inserted later by the compiler to be used with ``waterfall.last.use``
to mark the loop-end point for special
handling such as SCC clobber tracking.
- This intrinsic should not be generated by a frontend.
+ This intrinsic is not intended to be generated by a frontend.
============================================== =========================================================================
diff --git a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
index e9ad4df3079ea..d4afd182133cb 100644
--- a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
+++ b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
@@ -2727,7 +2727,7 @@ def int_amdgcn_cs_chain:
// have different values) the loop will execute wave-size times.
// The waterfall.begin intrinsic returns a new token that must be threaded through the
// corresponding waterfall.readfirstlane and waterfall.end or
-// waterfall.last_use intrinsics, forming a waterfall group of intrinsics that together
+// waterfall.last.use intrinsics, forming a waterfall group of intrinsics that together
// define a waterfall region.
// All intrinsics in a waterfall group must reside within the same basic block.
// Where the intrinsic specifies "llvm_any_ty", the intrinsic will accept any of
@@ -2736,19 +2736,19 @@ def int_amdgcn_cs_chain:
// v4f32, v8f32
// TODO: extend the support to allow spanning multiple blocks
def int_amdgcn_waterfall_begin :
- Intrinsic<[llvm_i32_ty], // Returns sgpr token
+ Intrinsic<[llvm_i32_ty], // Returns token to be used by the rest of the waterfall loop
[llvm_i32_ty, // Previous begin token / null if first
- llvm_any_ty], // VGPR index for waterfall
+ llvm_any_ty], // Non-uniform value for waterfall index
[IntrConvergent]>;
-// Often the index from begin will be used as the readfirstlane value from
-// waterfall code - however, this isn't always the case. The readfirstlane style
-// intrinsic is required to be used, but often is a no-op if the index is the
-// used directly.
+// Often the index operand of begin is the same as the non-uniform operand
+// of readfirstlane, however, this isn't always the case.
+// The readfirstlane intrinsic is a no-op if the index is used directly
+// in the body of the waterfall loop.
def int_amdgcn_waterfall_readfirstlane :
Intrinsic<[llvm_any_ty], // Returns any value
[llvm_i32_ty, // Token from final begin
- llvm_any_ty], // VGPR value to be used in readfirstlane
+ llvm_any_ty], // Non-uniform value to be used in readfirstlane
[IntrConvergent]>;
// Effectively a no-op, but this intrinsic is used to indicate the end of the
@@ -2779,7 +2779,7 @@ def int_amdgcn_waterfall_last_use_vgpr :
LLVMMatchType<0>], // Same as return value
[]>;
-// Special waterfall loop end intrinsic to be used with last_use
+// Special waterfall loop end intrinsic to be used with last.use
// This is inserted late, before codegen, mainly to mark the loop end
// for special handling such as scc clobbers.
def int_amdgcn_waterfall_loop_end :
>From dde0e84a068eee92d2d75ab6c7d282d7a9cf0483 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 26 May 2026 13:25:48 +0100
Subject: [PATCH 10/35] Fix header
---
llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
index 6f8075e5458e4..7358f0c92c1e9 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
@@ -1,9 +1,8 @@
//===- AMDGPUInsertWaterfall.cpp - insert waterfall loops around markers -====//
//
-// The LLVM Compiler Infrastructure
-//
-// This file is distributed under the University of Illinois Open Source
-// License. See LICENSE.TXT for details.
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
//
>From f03d181d1b637eab3fa9621ab156d13c51f52b28 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 26 May 2026 14:43:30 +0100
Subject: [PATCH 11/35] Test with three waterfall.begin shows a bug in
AMDGPUAtomicOptimizer.
This is not a soundness bug, generated code is correct but
the compiler misses optimization opportunity in basic blocks
that have waterfall region with 3 begins.
---
.../AMDGPU/amdgcn.waterfall.atomic.opt.ll | 128 +++++++++++++++++-
1 file changed, 127 insertions(+), 1 deletion(-)
diff --git a/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll b/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
index 2f63293aaa09c..24cc9ffaaafb1 100644
--- a/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
+++ b/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
@@ -392,7 +392,133 @@ bb:
ret void
}
-
+; Test that the atomic optimizer correctly skips over a waterfall group
+; with 3 chained waterfall.begin intrinsics and still optimizes the
+; atomic that follows it.
+define dllexport amdgpu_cs void @atomic_add_after_triple_begin(ptr addrspace(1) %arg, i32 inreg %arg1, ptr addrspace(4) inreg noundef %arg2, i32 inreg %arg3) #0 {
+; GFX10-LABEL: atomic_add_after_triple_begin:
+; GFX10: ; %bb.0: ; %bb
+; GFX10-NEXT: s_ashr_i32 s4, s0, 31
+; GFX10-NEXT: v_add_co_u32 v0, vcc, v0, s0
+; GFX10-NEXT: v_add_co_ci_u32_e32 v1, vcc, s4, v1, vcc
+; GFX10-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-NEXT: global_load_dwordx3 v[0:2], v[0:1], off
+; GFX10-NEXT: .LBB4_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-NEXT: s_waitcnt vmcnt(0)
+; GFX10-NEXT: v_readfirstlane_b32 s0, v0
+; GFX10-NEXT: v_readfirstlane_b32 s8, v1
+; GFX10-NEXT: v_readfirstlane_b32 s9, v2
+; GFX10-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
+; GFX10-NEXT: v_cmpx_eq_u32_e64 s0, v0
+; GFX10-NEXT: v_cmpx_eq_u32_e64 s8, v1
+; GFX10-NEXT: v_cmpx_eq_u32_e64 s9, v2
+; GFX10-NEXT: s_ashr_i32 s8, s0, 31
+; GFX10-NEXT: s_add_u32 s12, s1, s0
+; GFX10-NEXT: s_addc_u32 s13, s2, s8
+; GFX10-NEXT: v_mov_b32_e32 v3, 0
+; GFX10-NEXT: s_load_dwordx4 s[8:11], s[12:13], 0x0
+; GFX10-NEXT: v_mov_b32_e32 v0, 1
+; GFX10-NEXT: s_ashr_i32 s13, s0, 31
+; GFX10-NEXT: s_add_u32 s12, s1, s0
+; GFX10-NEXT: s_addc_u32 s13, s2, s13
+; GFX10-NEXT: v_mov_b32_e32 v4, 1
+; GFX10-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-NEXT: buffer_atomic_add v0, v3, s[8:11], 0 idxen glc
+; GFX10-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-NEXT: s_load_dwordx4 s[8:11], s[12:13], 0x0
+; GFX10-NEXT: s_waitcnt vmcnt(0)
+; GFX10-NEXT: v_mov_b32_e32 v0, 1
+; GFX10-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-NEXT: buffer_atomic_add v0, v3, s[8:11], 0 idxen glc
+; GFX10-NEXT: s_andn2_wrexec_b64 s[6:7], s[6:7]
+; GFX10-NEXT: ; implicit-def: $vgpr0_vgpr1_vgpr2
+; GFX10-NEXT: s_cbranch_execnz .LBB4_1
+; GFX10-NEXT: ; %bb.2:
+; GFX10-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
+; GFX10-NEXT: s_mov_b64 exec, s[4:5]
+; GFX10-NEXT: s_ashr_i32 s0, s3, 31
+; GFX10-NEXT: s_add_u32 s4, s1, s3
+; GFX10-NEXT: s_addc_u32 s5, s2, s0
+; GFX10-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x0
+; GFX10-NEXT: s_waitcnt lgkmcnt(0)
+; GFX10-NEXT: buffer_atomic_add v4, v3, s[0:3], 0 idxen
+; GFX10-NEXT: s_endpgm
+;
+; GFX11-LABEL: atomic_add_after_triple_begin:
+; GFX11: ; %bb.0: ; %bb
+; GFX11-NEXT: v_add_co_u32 v0, vcc, v0, s0
+; GFX11-NEXT: s_ashr_i32 s4, s0, 31
+; GFX11-NEXT: s_mov_b64 s[6:7], exec
+; GFX11-NEXT: v_add_co_ci_u32_e64 v1, null, s4, v1, vcc
+; GFX11-NEXT: s_mov_b64 s[4:5], exec
+; GFX11-NEXT: global_load_b96 v[0:2], v[0:1], off
+; GFX11-NEXT: .LBB4_1: ; =>This Inner Loop Header: Depth=1
+; GFX11-NEXT: s_waitcnt vmcnt(0)
+; GFX11-NEXT: v_readfirstlane_b32 s0, v0
+; GFX11-NEXT: v_readfirstlane_b32 s8, v1
+; GFX11-NEXT: v_readfirstlane_b32 s9, v2
+; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_3) | instskip(NEXT) | instid1(VALU_DEP_3)
+; GFX11-NEXT: v_cmpx_eq_u32_e64 s0, v0
+; GFX11-NEXT: v_cmpx_eq_u32_e64 s8, v1
+; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_3)
+; GFX11-NEXT: v_cmpx_eq_u32_e64 s9, v2
+; GFX11-NEXT: s_ashr_i32 s9, s0, 31
+; GFX11-NEXT: s_add_u32 s8, s1, s0
+; GFX11-NEXT: s_addc_u32 s9, s2, s9
+; GFX11-NEXT: v_mov_b32_e32 v3, 0
+; GFX11-NEXT: s_load_b128 s[8:11], s[8:9], 0x0
+; GFX11-NEXT: v_mov_b32_e32 v0, 1
+; GFX11-NEXT: s_ashr_i32 s13, s0, 31
+; GFX11-NEXT: s_add_u32 s12, s1, s0
+; GFX11-NEXT: s_addc_u32 s13, s2, s13
+; GFX11-NEXT: v_mov_b32_e32 v4, 1
+; GFX11-NEXT: s_waitcnt lgkmcnt(0)
+; GFX11-NEXT: buffer_atomic_add_u32 v0, v3, s[8:11], 0 idxen glc
+; GFX11-NEXT: s_load_b128 s[8:11], s[12:13], 0x0
+; GFX11-NEXT: s_waitcnt vmcnt(0)
+; GFX11-NEXT: v_mov_b32_e32 v0, 1
+; GFX11-NEXT: s_waitcnt lgkmcnt(0)
+; GFX11-NEXT: buffer_atomic_add_u32 v0, v3, s[8:11], 0 idxen glc
+; GFX11-NEXT: s_and_not1_wrexec_b64 s[6:7], s[6:7]
+; GFX11-NEXT: ; implicit-def: $vgpr0_vgpr1_vgpr2
+; GFX11-NEXT: s_cbranch_execnz .LBB4_1
+; GFX11-NEXT: ; %bb.2:
+; GFX11-NEXT: s_mov_b64 exec, s[4:5]
+; GFX11-NEXT: s_ashr_i32 s4, s3, 31
+; GFX11-NEXT: s_add_u32 s0, s1, s3
+; GFX11-NEXT: s_addc_u32 s1, s2, s4
+; GFX11-NEXT: s_load_b128 s[0:3], s[0:1], 0x0
+; GFX11-NEXT: s_waitcnt lgkmcnt(0)
+; GFX11-NEXT: buffer_atomic_add_u32 v4, v3, s[0:3], 0 idxen
+; GFX11-NEXT: s_endpgm
+bb:
+ %getelementptr = getelementptr i8, ptr addrspace(1) %arg, i32 %arg1
+ %load = load <3 x i32>, ptr addrspace(1) %getelementptr, align 16
+ %idx1 = extractelement <3 x i32> %load, i32 0
+ %idx2 = extractelement <3 x i32> %load, i32 1
+ %idx3 = extractelement <3 x i32> %load, i32 2
+ %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
+ %tok2 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok1, i32 %idx2)
+ %tok3 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok2, i32 %idx3)
+ %rfl = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %tok3, i32 %idx3)
+ %sext = sext i32 %rfl to i64
+ %gep = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext
+ %rsrc = load <4 x i32>, ptr addrspace(4) %gep, align 4
+ %wf_atomic1 = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %rsrc, i32 0, i32 0, i32 0, i32 0)
+ %end1 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %tok3, i32 %wf_atomic1)
+ %rfl2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %tok3, i32 %idx1)
+ %sext2 = sext i32 %rfl2 to i64
+ %gep2 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext2
+ %rsrc2 = load <4 x i32>, ptr addrspace(4) %gep2, align 4
+ %wf_atomic2 = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %rsrc2, i32 0, i32 0, i32 0, i32 0)
+ %end2 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %tok3, i32 %wf_atomic2)
+ %sext3 = sext i32 %arg3 to i64
+ %gep3 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext3
+ %rsrc3 = load <4 x i32>, ptr addrspace(4) %gep3, align 4
+ %atomic = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %rsrc3, i32 0, i32 0, i32 0, i32 0)
+ ret void
+}
; Function Attrs: nocallback nofree nounwind willreturn
declare i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32, <4 x i32>, i32, i32, i32, i32 immarg) #3
>From ca458e8c3583a44fb8c22ae5a42d97e99b7a9025 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 26 May 2026 15:06:24 +0100
Subject: [PATCH 12/35] Fix waterfall.begin traversal in AMDGPUAtomicOptimizer
Handle more than two waterfall.begin intrinsics in the same waterfall region.
Iterate through all chained begins, instead of just the first two,
when detecting waterfall regions.
Assisted-by: Claude <noreply at anthropic.com>
---
.../Target/AMDGPU/AMDGPUAtomicOptimizer.cpp | 18 ++++-----
.../AMDGPU/amdgcn.waterfall.atomic.opt.ll | 40 ++++++++++++++-----
2 files changed, 38 insertions(+), 20 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index 5579202350b5d..a877c028e6456 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -193,17 +193,13 @@ static Instruction *findLastInWaterfall(Instruction &Begin) {
Instruction *FinalBegin = &Begin;
// Drill through any begin intrinsics
- do {
- if (FinalBegin->hasOneUse()) {
- User *U = *FinalBegin->user_begin();
- IntrinsicInst *Intrin = dyn_cast<IntrinsicInst>(U);
- if (Intrin &&
- Intrin->getIntrinsicID() == Intrinsic::amdgcn_waterfall_begin) {
- FinalBegin = Intrin;
- continue;
- }
- }
- } while (false);
+ while (FinalBegin->hasOneUse()) {
+ IntrinsicInst *Intrin = dyn_cast<IntrinsicInst>(*FinalBegin->user_begin());
+ if (!Intrin ||
+ Intrin->getIntrinsicID() != Intrinsic::amdgcn_waterfall_begin)
+ break;
+ FinalBegin = Intrin;
+ }
Instruction *Last = FinalBegin;
diff --git a/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll b/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
index 24cc9ffaaafb1..034d1ebecf592 100644
--- a/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
+++ b/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
@@ -422,7 +422,6 @@ define dllexport amdgpu_cs void @atomic_add_after_triple_begin(ptr addrspace(1)
; GFX10-NEXT: s_ashr_i32 s13, s0, 31
; GFX10-NEXT: s_add_u32 s12, s1, s0
; GFX10-NEXT: s_addc_u32 s13, s2, s13
-; GFX10-NEXT: v_mov_b32_e32 v4, 1
; GFX10-NEXT: s_waitcnt lgkmcnt(0)
; GFX10-NEXT: buffer_atomic_add v0, v3, s[8:11], 0 idxen glc
; GFX10-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
@@ -437,12 +436,23 @@ define dllexport amdgpu_cs void @atomic_add_after_triple_begin(ptr addrspace(1)
; GFX10-NEXT: ; %bb.2:
; GFX10-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-NEXT: s_mov_b64 exec, s[4:5]
+; GFX10-NEXT: s_mov_b64 s[4:5], exec
+; GFX10-NEXT: s_waitcnt vmcnt(0)
+; GFX10-NEXT: v_mbcnt_lo_u32_b32 v0, s4, 0
+; GFX10-NEXT: v_mbcnt_hi_u32_b32 v0, s5, v0
+; GFX10-NEXT: v_cmp_eq_u32_e32 vcc, 0, v0
+; GFX10-NEXT: s_and_saveexec_b64 s[6:7], vcc
+; GFX10-NEXT: s_cbranch_execz .LBB4_4
+; GFX10-NEXT: ; %bb.3:
; GFX10-NEXT: s_ashr_i32 s0, s3, 31
-; GFX10-NEXT: s_add_u32 s4, s1, s3
-; GFX10-NEXT: s_addc_u32 s5, s2, s0
-; GFX10-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x0
+; GFX10-NEXT: s_add_u32 s6, s1, s3
+; GFX10-NEXT: s_addc_u32 s7, s2, s0
+; GFX10-NEXT: s_bcnt1_i32_b64 s4, s[4:5]
+; GFX10-NEXT: s_load_dwordx4 s[0:3], s[6:7], 0x0
+; GFX10-NEXT: v_mov_b32_e32 v0, s4
; GFX10-NEXT: s_waitcnt lgkmcnt(0)
-; GFX10-NEXT: buffer_atomic_add v4, v3, s[0:3], 0 idxen
+; GFX10-NEXT: buffer_atomic_add v0, v3, s[0:3], 0 idxen
+; GFX10-NEXT: .LBB4_4:
; GFX10-NEXT: s_endpgm
;
; GFX11-LABEL: atomic_add_after_triple_begin:
@@ -472,7 +482,6 @@ define dllexport amdgpu_cs void @atomic_add_after_triple_begin(ptr addrspace(1)
; GFX11-NEXT: s_ashr_i32 s13, s0, 31
; GFX11-NEXT: s_add_u32 s12, s1, s0
; GFX11-NEXT: s_addc_u32 s13, s2, s13
-; GFX11-NEXT: v_mov_b32_e32 v4, 1
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: buffer_atomic_add_u32 v0, v3, s[8:11], 0 idxen glc
; GFX11-NEXT: s_load_b128 s[8:11], s[12:13], 0x0
@@ -485,12 +494,25 @@ define dllexport amdgpu_cs void @atomic_add_after_triple_begin(ptr addrspace(1)
; GFX11-NEXT: s_cbranch_execnz .LBB4_1
; GFX11-NEXT: ; %bb.2:
; GFX11-NEXT: s_mov_b64 exec, s[4:5]
-; GFX11-NEXT: s_ashr_i32 s4, s3, 31
+; GFX11-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(SKIP_3) | instid1(VALU_DEP_1)
+; GFX11-NEXT: s_mov_b64 s[4:5], exec
+; GFX11-NEXT: s_mov_b64 s[6:7], exec
+; GFX11-NEXT: s_waitcnt vmcnt(0)
+; GFX11-NEXT: v_mbcnt_lo_u32_b32 v0, s4, 0
+; GFX11-NEXT: v_mbcnt_hi_u32_b32 v0, s5, v0
+; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT: v_cmpx_eq_u32_e32 0, v0
+; GFX11-NEXT: s_cbranch_execz .LBB4_4
+; GFX11-NEXT: ; %bb.3:
+; GFX11-NEXT: s_ashr_i32 s6, s3, 31
; GFX11-NEXT: s_add_u32 s0, s1, s3
-; GFX11-NEXT: s_addc_u32 s1, s2, s4
+; GFX11-NEXT: s_addc_u32 s1, s2, s6
+; GFX11-NEXT: s_bcnt1_i32_b64 s4, s[4:5]
; GFX11-NEXT: s_load_b128 s[0:3], s[0:1], 0x0
+; GFX11-NEXT: v_mov_b32_e32 v0, s4
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
-; GFX11-NEXT: buffer_atomic_add_u32 v4, v3, s[0:3], 0 idxen
+; GFX11-NEXT: buffer_atomic_add_u32 v0, v3, s[0:3], 0 idxen
+; GFX11-NEXT: .LBB4_4:
; GFX11-NEXT: s_endpgm
bb:
%getelementptr = getelementptr i8, ptr addrspace(1) %arg, i32 %arg1
>From 4fef1d63b54ec8f5aa56f4ffaf6e9d03dfaa75fb Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Wed, 27 May 2026 12:32:14 +0100
Subject: [PATCH 13/35] Change waterfall intrinsic token type from i32 to token
Use llvm_token_ty instead of llvm_i32_ty for waterfall intrinsic token
parameters and return types.
Instruction selection for waterfall intrinsics is done in custom C++
because TableGen pattern matching fails for llvm_token_ty,
which is mapped to MVT::token and therefore
cannot appear in any register class type list.
Add new file AMDGPUWaterfallHelper.h to share code between SelectionDAG and
GlobalISel.
Assisted-by: Claude <noreply at anthropic.com>
---
llvm/include/llvm/IR/IntrinsicsAMDGPU.td | 16 +-
llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp | 37 +
llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.h | 2 +
.../Target/AMDGPU/AMDGPUInsertWaterfall.cpp | 9 +-
.../AMDGPU/AMDGPUInstructionSelector.cpp | 51 +
.../Target/AMDGPU/AMDGPUInstructionSelector.h | 2 +
.../lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp | 21 +
.../lib/Target/AMDGPU/AMDGPUWaterfallHelper.h | 74 ++
llvm/lib/Target/AMDGPU/SIInstructions.td | 80 +-
.../AMDGPU/llvm.amdgcn.waterfall.ll | 18 +-
.../AMDGPU/amdgcn.waterfall.atomic.opt.ll | 46 +-
.../AMDGPU/amdgpu-insert-waterfall-licm.ll | 6 +-
.../AMDGPU/amdgpu-insert-waterfall.mir | 16 +-
.../CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll | 1134 ++++++++---------
.../InstCombine/AMDGPU/waterfall.ll | 54 +-
15 files changed, 849 insertions(+), 717 deletions(-)
create mode 100644 llvm/lib/Target/AMDGPU/AMDGPUWaterfallHelper.h
diff --git a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
index d4afd182133cb..18b5cfe857f05 100644
--- a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
+++ b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
@@ -2736,9 +2736,9 @@ def int_amdgcn_cs_chain:
// v4f32, v8f32
// TODO: extend the support to allow spanning multiple blocks
def int_amdgcn_waterfall_begin :
- Intrinsic<[llvm_i32_ty], // Returns token to be used by the rest of the waterfall loop
- [llvm_i32_ty, // Previous begin token / null if first
- llvm_any_ty], // Non-uniform value for waterfall index
+ Intrinsic<[llvm_token_ty], // Returns token to be used by the rest of the waterfall loop
+ [llvm_token_ty, // Previous begin token / poison if first
+ llvm_any_ty], // Non-uniform value for waterfall index
[IntrConvergent]>;
// Often the index operand of begin is the same as the non-uniform operand
@@ -2747,7 +2747,7 @@ def int_amdgcn_waterfall_begin :
// in the body of the waterfall loop.
def int_amdgcn_waterfall_readfirstlane :
Intrinsic<[llvm_any_ty], // Returns any value
- [llvm_i32_ty, // Token from final begin
+ [llvm_token_ty, // Token from final begin
llvm_any_ty], // Non-uniform value to be used in readfirstlane
[IntrConvergent]>;
@@ -2757,7 +2757,7 @@ def int_amdgcn_waterfall_readfirstlane :
// intrinsic is reached,
def int_amdgcn_waterfall_end :
Intrinsic<[llvm_any_ty], // Returns any value
- [llvm_i32_ty, // Token from final begin
+ [llvm_token_ty, // Token from final begin
LLVMMatchType<0>], // Same as return value
[IntrConvergent]>;
@@ -2768,14 +2768,14 @@ def int_amdgcn_waterfall_end :
// store.
def int_amdgcn_waterfall_last_use :
Intrinsic<[llvm_any_ty], // Returns any value
- [llvm_i32_ty, // Token from begin
+ [llvm_token_ty, // Token from begin
LLVMMatchType<0>], // Same as return value
[IntrConvergent]>;
// Special variant where the last-use uses a VGPR that needs to be uniform.
def int_amdgcn_waterfall_last_use_vgpr :
Intrinsic<[llvm_any_ty], // Returns any value
- [llvm_i32_ty, // Token from begin
+ [llvm_token_ty, // Token from begin
LLVMMatchType<0>], // Same as return value
[]>;
@@ -2784,7 +2784,7 @@ def int_amdgcn_waterfall_last_use_vgpr :
// for special handling such as scc clobbers.
def int_amdgcn_waterfall_loop_end :
Intrinsic<[],
- [llvm_i32_ty], // Token from begin
+ [llvm_token_ty], // Token from begin
[IntrConvergent]>;
// Run a function with all the lanes enabled. Only direct calls are allowed. The
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp b/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp
index ecf8d957fc80f..a5574eccfff4f 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp
@@ -14,6 +14,7 @@
#include "AMDGPUISelDAGToDAG.h"
#include "AMDGPU.h"
#include "AMDGPUInstrInfo.h"
+#include "AMDGPUWaterfallHelper.h"
#include "AMDGPUSubtarget.h"
#include "AMDGPUTargetMachine.h"
#include "MCTargetDesc/AMDGPUMCTargetDesc.h"
@@ -3242,6 +3243,32 @@ void AMDGPUDAGToDAGISel::SelectInterpP1F16(SDNode *N) {
CurDAG->ReplaceAllUsesOfValueWith(SDValue(N, 0), SDValue(InterpP1LV, 0));
}
+void AMDGPUDAGToDAGISel::SelectWaterfallIntrinsic(SDNode *N, unsigned IntrID) {
+ // op0=chain, op1=intrinsicID, op2=token(Untyped), op3=value
+ SDValue Chain = N->getOperand(0);
+ SDValue Tok = N->getOperand(2);
+ SDValue Val = N->getOperand(3);
+ unsigned SizeDwords = Val.getValueSizeInBits() / 32;
+ unsigned Opc = AMDGPU::getWaterfallPseudoOpcode(IntrID, SizeDwords);
+ assert(Opc && "Unsupported waterfall intrinsic value size");
+
+ SDVTList VTs;
+ if (IntrID == Intrinsic::amdgcn_waterfall_begin)
+ VTs = CurDAG->getVTList(MVT::i32, MVT::Other);
+ else
+ VTs = CurDAG->getVTList(N->getValueType(0), MVT::Other);
+
+ CurDAG->SelectNodeTo(N, Opc, VTs, {Tok, Val, Chain});
+}
+
+void AMDGPUDAGToDAGISel::SelectWaterfallIntrinsicLoopEnd(SDNode *N) {
+ // op0=chain, op1=intrinsicID, op2=token(Untyped)
+ SDValue Chain = N->getOperand(0);
+ SDValue Tok = N->getOperand(2);
+ CurDAG->SelectNodeTo(N, AMDGPU::SI_WATERFALL_LOOP_END,
+ CurDAG->getVTList(MVT::Other), {Tok, Chain});
+}
+
void AMDGPUDAGToDAGISel::SelectINTRINSIC_W_CHAIN(SDNode *N) {
unsigned IntrID = N->getConstantOperandVal(1);
switch (IntrID) {
@@ -3263,6 +3290,13 @@ void AMDGPUDAGToDAGISel::SelectINTRINSIC_W_CHAIN(SDNode *N) {
.getInfo<SIMachineFunctionInfo>()
->setInitWholeWave();
break;
+ case Intrinsic::amdgcn_waterfall_begin:
+ case Intrinsic::amdgcn_waterfall_readfirstlane:
+ case Intrinsic::amdgcn_waterfall_end:
+ case Intrinsic::amdgcn_waterfall_last_use:
+ case Intrinsic::amdgcn_waterfall_last_use_vgpr:
+ SelectWaterfallIntrinsic(N, IntrID);
+ return;
}
SelectCode(N);
@@ -3356,6 +3390,9 @@ void AMDGPUDAGToDAGISel::SelectINTRINSIC_VOID(SDNode *N) {
case Intrinsic::amdgcn_tensor_store_from_lds:
SelectTensorLoadStore(N, IntrID);
return;
+ case Intrinsic::amdgcn_waterfall_loop_end:
+ SelectWaterfallIntrinsicLoopEnd(N);
+ return;
default:
break;
}
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.h b/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.h
index 0a3631e4dbf59..840c45421de4c 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.h
@@ -302,6 +302,8 @@ class AMDGPUDAGToDAGISel : public SelectionDAGISel {
void SelectDSBvhStackIntrinsic(SDNode *N, unsigned IntrID);
void SelectTensorLoadStore(SDNode *N, unsigned IntrID);
void SelectDS_GWS(SDNode *N, unsigned IntrID);
+ void SelectWaterfallIntrinsic(SDNode *N, unsigned IntrID);
+ void SelectWaterfallIntrinsicLoopEnd(SDNode *N);
void SelectInterpP1F16(SDNode *N);
void SelectINTRINSIC_W_CHAIN(SDNode *N);
void SelectINTRINSIC_WO_CHAIN(SDNode *N);
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
index 7358f0c92c1e9..cd25c3b361944 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
@@ -341,7 +341,14 @@ class AMDGPUInsertWaterfall : public MachineFunctionPass {
bool tokIsStart(const MachineOperand *MO) const {
MachineInstr *defInstr = getDefInstr(MO);
- if (defInstr && defInstr->getOpcode() == AMDGPU::S_MOV_B32) {
+ if (!defInstr)
+ return false;
+ if (defInstr->getOpcode() == AMDGPU::IMPLICIT_DEF)
+ return true;
+ if (defInstr->isCopy())
+ return tokIsStart(&defInstr->getOperand(1));
+ // S_MOV_B32 0 for backwards compatibility with hand-written MIR.
+ if (defInstr->getOpcode() == AMDGPU::S_MOV_B32) {
auto CopySrcOp = TII->getNamedOperand(*defInstr, AMDGPU::OpName::src0);
if (CopySrcOp && CopySrcOp->isImm()) {
if (CopySrcOp->getImm() == 0)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
index 463b8c40350b2..7de080a8f0b5a 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
@@ -15,6 +15,7 @@
#include "AMDGPU.h"
#include "AMDGPUGlobalISelUtils.h"
#include "AMDGPUInstrInfo.h"
+#include "AMDGPUWaterfallHelper.h"
#include "AMDGPURegisterBankInfo.h"
#include "AMDGPUTargetMachine.h"
#include "SIMachineFunctionInfo.h"
@@ -2520,6 +2521,14 @@ bool AMDGPUInstructionSelector::selectG_INTRINSIC_W_SIDE_EFFECTS(
return selectSGetBarrierState(I, IntrinsicID);
case Intrinsic::amdgcn_s_barrier_signal_isfirst:
return selectSBarrierSignalIsfirst(I, IntrinsicID);
+ case Intrinsic::amdgcn_waterfall_begin:
+ case Intrinsic::amdgcn_waterfall_readfirstlane:
+ case Intrinsic::amdgcn_waterfall_end:
+ case Intrinsic::amdgcn_waterfall_last_use:
+ case Intrinsic::amdgcn_waterfall_last_use_vgpr:
+ return selectWaterfallIntrinsic(I, IntrinsicID);
+ case Intrinsic::amdgcn_waterfall_loop_end:
+ return selectWaterfallIntrinsicLoopEnd(I);
}
return selectImpl(I, *CoverageInfo);
}
@@ -4439,6 +4448,48 @@ bool AMDGPUInstructionSelector::selectStackRestore(MachineInstr &MI) const {
return true;
}
+bool AMDGPUInstructionSelector::selectWaterfallIntrinsic(
+ MachineInstr &I, Intrinsic::ID IntrID) const {
+ // op0=result, op1=intrinsicID, op2=token_input, op3=value
+ MachineBasicBlock *MBB = I.getParent();
+ const DebugLoc &DL = I.getDebugLoc();
+ Register DstReg = I.getOperand(0).getReg();
+ Register TokReg = I.getOperand(2).getReg();
+ Register ValReg = I.getOperand(3).getReg();
+
+ unsigned SizeBits;
+ if (IntrID == Intrinsic::amdgcn_waterfall_begin)
+ SizeBits = MRI->getType(ValReg).getSizeInBits();
+ else
+ SizeBits = MRI->getType(DstReg).getSizeInBits();
+
+ unsigned SizeDwords = SizeBits / 32;
+ unsigned Opc = AMDGPU::getWaterfallPseudoOpcode(IntrID, SizeDwords);
+ if (!Opc)
+ return false;
+
+ MachineInstrBuilder MIB =
+ BuildMI(*MBB, &I, DL, TII.get(Opc), DstReg).addReg(TokReg).addReg(ValReg);
+ I.eraseFromParent();
+ constrainSelectedInstRegOperands(*MIB, TII, TRI, RBI);
+ return true;
+}
+
+bool AMDGPUInstructionSelector::selectWaterfallIntrinsicLoopEnd(
+ MachineInstr &I) const {
+ // op0=intrinsicID, op1=token
+ MachineBasicBlock *MBB = I.getParent();
+ const DebugLoc &DL = I.getDebugLoc();
+ Register TokReg = I.getOperand(1).getReg();
+
+ MachineInstrBuilder MIB =
+ BuildMI(*MBB, &I, DL, TII.get(AMDGPU::SI_WATERFALL_LOOP_END))
+ .addReg(TokReg);
+ I.eraseFromParent();
+ constrainSelectedInstRegOperands(*MIB, TII, TRI, RBI);
+ return true;
+}
+
bool AMDGPUInstructionSelector::select(MachineInstr &I) {
if (!I.isPreISelOpcode()) {
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h
index aa0d2385266d2..6184755060f06 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h
@@ -157,6 +157,8 @@ class AMDGPUInstructionSelector final : public InstructionSelector {
bool selectNamedBarrierInst(MachineInstr &I, Intrinsic::ID IID) const;
bool selectSBarrierSignalIsfirst(MachineInstr &I, Intrinsic::ID IID) const;
bool selectSGetBarrierState(MachineInstr &I, Intrinsic::ID IID) const;
+ bool selectWaterfallIntrinsic(MachineInstr &I, Intrinsic::ID IID) const;
+ bool selectWaterfallIntrinsicLoopEnd(MachineInstr &I) const;
bool selectSBarrierLeave(MachineInstr &I) const;
bool selectWaveShuffleIntrin(MachineInstr &I) const;
diff --git a/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp b/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
index 8b5076cdd7712..0b4c3c62f4e84 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
@@ -8061,6 +8061,10 @@ bool AMDGPULegalizerInfo::legalizeSetFPEnv(MachineInstr &MI,
return true;
}
+static void legalizeWaterfallTokenReg(MachineRegisterInfo &MRI, Register Reg) {
+ MRI.setType(Reg, LLT::scalar(32));
+}
+
bool AMDGPULegalizerInfo::legalizeIntrinsic(LegalizerHelper &Helper,
MachineInstr &MI) const {
MachineIRBuilder &B = Helper.MIRBuilder;
@@ -8561,6 +8565,23 @@ bool AMDGPULegalizerInfo::legalizeIntrinsic(LegalizerHelper &Helper,
.addMemOperand(*MI.memoperands_begin());
MI.eraseFromParent();
return true;
+ case Intrinsic::amdgcn_waterfall_begin:
+ // op0=token_result, op1=intrinsicID, op2=token_input, op3=value
+ legalizeWaterfallTokenReg(MRI, MI.getOperand(0).getReg());
+ legalizeWaterfallTokenReg(MRI, MI.getOperand(2).getReg());
+ return true;
+ case Intrinsic::amdgcn_waterfall_readfirstlane:
+ case Intrinsic::amdgcn_waterfall_end:
+ case Intrinsic::amdgcn_waterfall_last_use:
+ case Intrinsic::amdgcn_waterfall_last_use_vgpr:
+ // op0=value_result, op1=intrinsicID, op2=token_input, op3=value
+ legalizeWaterfallTokenReg(MRI, MI.getOperand(2).getReg());
+ return true;
+ case Intrinsic::amdgcn_waterfall_loop_end:
+ // op0=intrinsicID, op1=token_input
+ legalizeWaterfallTokenReg(MRI, MI.getOperand(1).getReg());
+ return true;
+
default: {
if (const AMDGPU::ImageDimIntrinsicInfo *ImageDimIntr =
AMDGPU::getImageDimIntrinsicInfo(IntrID))
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUWaterfallHelper.h b/llvm/lib/Target/AMDGPU/AMDGPUWaterfallHelper.h
new file mode 100644
index 0000000000000..1f63b680bb8c0
--- /dev/null
+++ b/llvm/lib/Target/AMDGPU/AMDGPUWaterfallHelper.h
@@ -0,0 +1,74 @@
+//===- AMDGPUWaterfallHelper.h - Waterfall intrinsic opcode mapping --------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Shared helper for mapping waterfall intrinsic IDs and value sizes to
+// SI_WATERFALL_* pseudo opcodes. Used by both SelectionDAG and GlobalISel
+// instruction selectors.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIB_TARGET_AMDGPU_AMDGPUWATERFALLHELPER_H
+#define LLVM_LIB_TARGET_AMDGPU_AMDGPUWATERFALLHELPER_H
+
+#include "MCTargetDesc/AMDGPUMCTargetDesc.h"
+#include "llvm/IR/IntrinsicsAMDGPU.h"
+
+namespace llvm::AMDGPU {
+
+inline unsigned getWaterfallPseudoOpcode(unsigned IntrID,
+ unsigned SizeDwords) {
+ switch (IntrID) {
+ case Intrinsic::amdgcn_waterfall_begin:
+ switch (SizeDwords) {
+ case 1: return AMDGPU::SI_WATERFALL_BEGIN_V1;
+ case 2: return AMDGPU::SI_WATERFALL_BEGIN_V2;
+ case 4: return AMDGPU::SI_WATERFALL_BEGIN_V4;
+ case 8: return AMDGPU::SI_WATERFALL_BEGIN_V8;
+ }
+ break;
+ case Intrinsic::amdgcn_waterfall_readfirstlane:
+ switch (SizeDwords) {
+ case 1: return AMDGPU::SI_WATERFALL_READFIRSTLANE_V1;
+ case 2: return AMDGPU::SI_WATERFALL_READFIRSTLANE_V2;
+ case 4: return AMDGPU::SI_WATERFALL_READFIRSTLANE_V4;
+ case 8: return AMDGPU::SI_WATERFALL_READFIRSTLANE_V8;
+ }
+ break;
+ case Intrinsic::amdgcn_waterfall_end:
+ switch (SizeDwords) {
+ case 1: return AMDGPU::SI_WATERFALL_END_V1;
+ case 2: return AMDGPU::SI_WATERFALL_END_V2;
+ case 4: return AMDGPU::SI_WATERFALL_END_V4;
+ case 8: return AMDGPU::SI_WATERFALL_END_V8;
+ }
+ break;
+ case Intrinsic::amdgcn_waterfall_last_use:
+ switch (SizeDwords) {
+ case 1: return AMDGPU::SI_WATERFALL_LAST_USE_V1;
+ case 2: return AMDGPU::SI_WATERFALL_LAST_USE_V2;
+ case 4: return AMDGPU::SI_WATERFALL_LAST_USE_V4;
+ case 8: return AMDGPU::SI_WATERFALL_LAST_USE_V8;
+ }
+ break;
+ case Intrinsic::amdgcn_waterfall_last_use_vgpr:
+ switch (SizeDwords) {
+ case 1: return AMDGPU::SI_WATERFALL_LAST_USE_V1_V;
+ case 2: return AMDGPU::SI_WATERFALL_LAST_USE_V2_V;
+ case 4: return AMDGPU::SI_WATERFALL_LAST_USE_V4_V;
+ case 8: return AMDGPU::SI_WATERFALL_LAST_USE_V8_V;
+ }
+ break;
+ case Intrinsic::amdgcn_waterfall_loop_end:
+ return AMDGPU::SI_WATERFALL_LOOP_END;
+ }
+ return 0;
+}
+
+} // namespace llvm::AMDGPU
+
+#endif // LLVM_LIB_TARGET_AMDGPU_AMDGPUWATERFALLHELPER_H
diff --git a/llvm/lib/Target/AMDGPU/SIInstructions.td b/llvm/lib/Target/AMDGPU/SIInstructions.td
index 1c2e435f88ca9..5ef5e2c31f98a 100644
--- a/llvm/lib/Target/AMDGPU/SIInstructions.td
+++ b/llvm/lib/Target/AMDGPU/SIInstructions.td
@@ -1396,28 +1396,39 @@ def : GCNPat <
>;
// Waterfall pseudo ops - these are effectively specialized control flow pseudo
-// instructions
-// They are tagged as defining SCC since the loop that gets inserted will usually clobber it
+// instructions.
+// They are tagged as defining SCC since the loop that gets inserted
+// will usually clobber it.
+// Selection is done in custom C++ because TableGen pattern matching fails for intrinsics
+// with token. TableGen type inference requires intrinsics and instruction types to be consistent,
+// but llvm_token_ty maps to MVT::token that cannot appear in any register class type list.
+// TODO: better definition for WaterfallTokenType. Tokens are eliminated
+// by AMDGPUInsertWaterfall pass, so WaterfallTokenType can use a dedicated "untyped"
+// zero-size non-allocatable type, but that would involve more changes in selection.
+def WaterfallTokenType : RegisterOperand<SReg_32>;
+
let UseNamedOperandTable = 1, Defs = [SCC] in {
class SI_WATERFALL_BEGIN<RegisterClass idx_rc> : SPseudoInstSI <
- (outs SReg_32:$tok_ret),
- (ins SReg_32:$tok, idx_rc:$idx)>;
+ (outs WaterfallTokenType:$tok_ret),
+ (ins WaterfallTokenType:$tok, idx_rc:$idx)>;
class SI_WATERFALL_READFIRSTLANE<RegisterClass dst_rc, RegisterClass src_rc> : SPseudoInstSI <
(outs dst_rc:$dst),
- (ins SReg_32:$tok, src_rc:$src)>;
+ (ins WaterfallTokenType:$tok, src_rc:$src)>;
class SI_WATERFALL_END<RegisterClass rc> : SPseudoInstSI <
(outs rc:$dst),
- (ins SReg_32:$tok, rc:$src)>;
+ (ins WaterfallTokenType:$tok, rc:$src)>;
class SI_WATERFALL_LAST_USE<RegisterClass rc> : SPseudoInstSI <
(outs rc:$dst),
- (ins SReg_32:$tok, rc:$src)>;
+ (ins WaterfallTokenType:$tok, rc:$src)>;
} // End UseNamedOperandTable = 1
+// TODO: Do we still need the size variants (_V1, _V2, etc.) when selection
+// is in custom code?
def SI_WATERFALL_BEGIN_V1 : SI_WATERFALL_BEGIN<VGPR_32>;
def SI_WATERFALL_BEGIN_V2 : SI_WATERFALL_BEGIN<VReg_64>;
def SI_WATERFALL_BEGIN_V4 : SI_WATERFALL_BEGIN<VReg_128>;
@@ -1446,7 +1457,7 @@ let Defs = [SCC],
UseNamedOperandTable = 1 in
def SI_WATERFALL_LOOP_END : SPseudoInstSI <
(outs),
- (ins SReg_32:$tok)
+ (ins WaterfallTokenType:$tok)
>;
//===----------------------------------------------------------------------===//
@@ -2759,59 +2770,6 @@ def : GCNPat <
(V_RCP_IFLAG_F32_e32 (V_CVT_F32_U32_e32 $src0))))
>;
-multiclass SI_WATERFALL_Pattern<ValueType dvt, ValueType svt, string VecSize> {
- def : GCNPat<(dvt(int_amdgcn_waterfall_readfirstlane i32:$tok, svt:$src)),
- (!cast<Instruction>("SI_WATERFALL_READFIRSTLANE_"#VecSize)
- i32:$tok,
- svt:$src)>;
-}
-
-multiclass SI_WATERFALL_S_Pattern<list<ValueType> vts, string VecSize> {
- foreach vt = vts in {
- def : GCNPat<(i32(int_amdgcn_waterfall_begin i32:$tok, vt:$idx)),
- (!cast<Instruction>("SI_WATERFALL_BEGIN_"#VecSize) i32:$tok,
- vt:$idx)>;
-
- def : GCNPat<(vt(int_amdgcn_waterfall_readfirstlane i32:$tok, vt:$src)),
- (!cast<Instruction>("SI_WATERFALL_READFIRSTLANE_"#VecSize)
- i32:$tok,
- vt:$src)>;
-
- def : GCNPat<(vt(int_amdgcn_waterfall_end i32:$tok, vt:$src)),
- (!cast<Instruction>("SI_WATERFALL_END_"#VecSize) i32:$tok,
- vt:$src)>;
-
- def : GCNPat<(vt(int_amdgcn_waterfall_last_use i32:$tok, vt:$src)),
- (!cast<Instruction>("SI_WATERFALL_LAST_USE_"#VecSize) i32:$tok,
- vt:$src)>;
-
- def : GCNPat<(vt(int_amdgcn_waterfall_last_use_vgpr i32:$tok, vt:$src)),
- (!cast<Instruction>("SI_WATERFALL_LAST_USE_"#VecSize#"_V")
- i32:$tok,
- vt:$src)>;
- }
-}
-
-defm : SI_WATERFALL_S_Pattern<Reg16Types.types, "V1">;
-defm : SI_WATERFALL_S_Pattern<Reg32Types.types, "V1">;
-defm : SI_WATERFALL_S_Pattern<Reg64Types.types, "V2">;
-defm : SI_WATERFALL_S_Pattern<Reg128Types.types, "V4">;
-defm : SI_WATERFALL_S_Pattern<Reg256Types.types, "V8">;
-
-defm : SI_WATERFALL_Pattern <i16, f16, "V1">;
-defm : SI_WATERFALL_Pattern <v2i16, v2f16, "V1">;
-defm : SI_WATERFALL_Pattern <v4i16, v4f16, "V2">;
-
-defm : SI_WATERFALL_Pattern <i32, f32, "V1">;
-defm : SI_WATERFALL_Pattern <v2i32, v2f32, "V2">;
-defm : SI_WATERFALL_Pattern <v4i32, v4f32, "V4">;
-defm : SI_WATERFALL_Pattern <v8i32, v8f32, "V8">;
-
-def : GCNPat<
- (int_amdgcn_waterfall_loop_end i32:$tok),
- (SI_WATERFALL_LOOP_END i32:$tok)
->;
-
//===----------------------------------------------------------------------===//
// VOP3 Patterns
//===----------------------------------------------------------------------===//
diff --git a/llvm/test/Analysis/UniformityAnalysis/AMDGPU/llvm.amdgcn.waterfall.ll b/llvm/test/Analysis/UniformityAnalysis/AMDGPU/llvm.amdgcn.waterfall.ll
index 7b934e85b68ab..7df5b210001c5 100644
--- a/llvm/test/Analysis/UniformityAnalysis/AMDGPU/llvm.amdgcn.waterfall.ll
+++ b/llvm/test/Analysis/UniformityAnalysis/AMDGPU/llvm.amdgcn.waterfall.ll
@@ -5,25 +5,25 @@
; CHECK: DIVERGENT: %args = load <2 x i32>, ptr addrspace(1) %gep.in, align 8
; CHECK: DIVERGENT: %value = extractelement <2 x i32> %args, i32 0
; CHECK: DIVERGENT: %lane = extractelement <2 x i32> %args, i32 1
-; CHECK-NOT: DIVERGENT: %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %lane)
-; CHECK-NOT: DIVERGENT: %readlane = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %lane)
+; CHECK-NOT: DIVERGENT: %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %lane)
+; CHECK-NOT: DIVERGENT: %readlane = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %lane)
; CHECK-NOT: DIVERGENT: %readlane1 = call i32 @llvm.amdgcn.readlane(i32 %value, i32 %readlane)
-; CHECK: DIVERGENT: %readlane2 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %wf_token, i32 %readlane1)
+; CHECK: DIVERGENT: %readlane2 = call i32 @llvm.amdgcn.waterfall.end.i32(token %wf_token, i32 %readlane1)
; CHECK: DIVERGENT: store i32 %readlane2, ptr addrspace(1) %out, align 4
define amdgpu_ps void @test_waterfall_readlane(i32 addrspace(1)* inreg %out, <2 x i32> addrspace(1)* inreg %in, i32 %tid) #1 {
%gep.in = getelementptr <2 x i32>, <2 x i32> addrspace(1)* %in, i32 %tid
%args = load <2 x i32>, <2 x i32> addrspace(1)* %gep.in
%value = extractelement <2 x i32> %args, i32 0
%lane = extractelement <2 x i32> %args, i32 1
- %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %lane)
- %readlane = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %lane)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %lane)
+ %readlane = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %lane)
%readlane1 = call i32 @llvm.amdgcn.readlane(i32 %value, i32 %readlane)
- %readlane2 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %wf_token, i32 %readlane1)
+ %readlane2 = call i32 @llvm.amdgcn.waterfall.end.i32(token %wf_token, i32 %readlane1)
store i32 %readlane2, i32 addrspace(1)* %out, align 4
ret void
}
-declare i32 @llvm.amdgcn.waterfall.begin.i32(i32, i32)
-declare i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32, i32)
+declare token @llvm.amdgcn.waterfall.begin.i32(token, i32)
+declare i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token, i32)
declare i32 @llvm.amdgcn.readlane(i32, i32)
-declare i32 @llvm.amdgcn.waterfall.end.i32(i32, i32)
+declare i32 @llvm.amdgcn.waterfall.end.i32(token, i32)
diff --git a/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll b/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
index 034d1ebecf592..8214232d84f09 100644
--- a/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
+++ b/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
@@ -61,13 +61,13 @@ bb:
%load = load <2 x i32>, ptr addrspace(1) %getelementptr, align 8
%extractelement = extractelement <2 x i32> %load, i32 0
%extractelement3 = extractelement <2 x i32> %load, i32 1
- %call = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %extractelement3)
- %call4 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %call, i32 %extractelement3)
+ %call = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %extractelement3)
+ %call4 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %call, i32 %extractelement3)
%sext = sext i32 %call4 to i64
%getelementptr5 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext
%load6 = load <4 x i32>, ptr addrspace(4) %getelementptr5, align 4
%call7 = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %load6, i32 0, i32 0, i32 0, i32 0)
- %call8 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %call, i32 %call7)
+ %call8 = call i32 @llvm.amdgcn.waterfall.end.i32(token %call, i32 %call7)
ret void
}
@@ -175,13 +175,13 @@ bb:
%getelementptr5 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext
%load6 = load <4 x i32>, ptr addrspace(4) %getelementptr5, align 4
%call = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %load6, i32 0, i32 0, i32 0, i32 0)
- %call7 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %extractelement4)
- %call8 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %call7, i32 %extractelement4)
+ %call7 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %extractelement4)
+ %call8 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %call7, i32 %extractelement4)
%sext9 = sext i32 %call8 to i64
%getelementptr10 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext9
%load11 = load <4 x i32>, ptr addrspace(4) %getelementptr10, align 4
%call12 = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %load11, i32 0, i32 0, i32 0, i32 0)
- %call13 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %call7, i32 %call12)
+ %call13 = call i32 @llvm.amdgcn.waterfall.end.i32(token %call7, i32 %call12)
ret void
}
@@ -283,13 +283,13 @@ bb:
%load = load <2 x i32>, ptr addrspace(1) %getelementptr, align 8
%extractelement = extractelement <2 x i32> %load, i32 0
%extractelement4 = extractelement <2 x i32> %load, i32 1
- %call7 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %extractelement4)
- %call8 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %call7, i32 %extractelement4)
+ %call7 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %extractelement4)
+ %call8 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %call7, i32 %extractelement4)
%sext9 = sext i32 %call8 to i64
%getelementptr10 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext9
%load11 = load <4 x i32>, ptr addrspace(4) %getelementptr10, align 4
%call12 = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %load11, i32 0, i32 0, i32 0, i32 0)
- %call13 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %call7, i32 %call12)
+ %call13 = call i32 @llvm.amdgcn.waterfall.end.i32(token %call7, i32 %call12)
%sext = sext i32 %arg3 to i64
%getelementptr5 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext
%load6 = load <4 x i32>, ptr addrspace(4) %getelementptr5, align 4
@@ -377,18 +377,18 @@ bb:
%load = load <2 x i32>, ptr addrspace(1) %getelementptr, align 8
%extractelement = extractelement <2 x i32> %load, i32 0
%extractelement4 = extractelement <2 x i32> %load, i32 1
- %token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %extractelement)
+ %token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %extractelement)
%sext = sext i32 %arg3 to i64
%getelementptr5 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext
%load6 = load <4 x i32>, ptr addrspace(4) %getelementptr5, align 4
%call = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %load6, i32 0, i32 0, i32 0, i32 0)
- %token2 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %token, i32 %extractelement4)
- %call8 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %token2, i32 %extractelement4)
+ %token2 = call token @llvm.amdgcn.waterfall.begin.i32(token %token, i32 %extractelement4)
+ %call8 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %token2, i32 %extractelement4)
%sext9 = sext i32 %call8 to i64
%getelementptr10 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext9
%load11 = load <4 x i32>, ptr addrspace(4) %getelementptr10, align 4
%call12 = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %load11, i32 0, i32 0, i32 0, i32 0)
- %call13 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %token2, i32 %call12)
+ %call13 = call i32 @llvm.amdgcn.waterfall.end.i32(token %token2, i32 %call12)
ret void
}
@@ -520,21 +520,21 @@ bb:
%idx1 = extractelement <3 x i32> %load, i32 0
%idx2 = extractelement <3 x i32> %load, i32 1
%idx3 = extractelement <3 x i32> %load, i32 2
- %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
- %tok2 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok1, i32 %idx2)
- %tok3 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok2, i32 %idx3)
- %rfl = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %tok3, i32 %idx3)
+ %tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok2 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok1, i32 %idx2)
+ %tok3 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok2, i32 %idx3)
+ %rfl = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %tok3, i32 %idx3)
%sext = sext i32 %rfl to i64
%gep = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext
%rsrc = load <4 x i32>, ptr addrspace(4) %gep, align 4
%wf_atomic1 = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %rsrc, i32 0, i32 0, i32 0, i32 0)
- %end1 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %tok3, i32 %wf_atomic1)
- %rfl2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %tok3, i32 %idx1)
+ %end1 = call i32 @llvm.amdgcn.waterfall.end.i32(token %tok3, i32 %wf_atomic1)
+ %rfl2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %tok3, i32 %idx1)
%sext2 = sext i32 %rfl2 to i64
%gep2 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext2
%rsrc2 = load <4 x i32>, ptr addrspace(4) %gep2, align 4
%wf_atomic2 = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %rsrc2, i32 0, i32 0, i32 0, i32 0)
- %end2 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %tok3, i32 %wf_atomic2)
+ %end2 = call i32 @llvm.amdgcn.waterfall.end.i32(token %tok3, i32 %wf_atomic2)
%sext3 = sext i32 %arg3 to i64
%gep3 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext3
%rsrc3 = load <4 x i32>, ptr addrspace(4) %gep3, align 4
@@ -545,10 +545,10 @@ bb:
; Function Attrs: nocallback nofree nounwind willreturn
declare i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32, <4 x i32>, i32, i32, i32, i32 immarg) #3
; Function Attrs: convergent nounwind
-declare i32 @llvm.amdgcn.waterfall.begin.i32(i32, i32) #4
+declare token @llvm.amdgcn.waterfall.begin.i32(token, i32) #4
; Function Attrs: convergent nounwind
-declare i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32, i32) #4
+declare i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token, i32) #4
; Function Attrs: convergent nounwind
-declare i32 @llvm.amdgcn.waterfall.end.i32(i32, i32) #4
+declare i32 @llvm.amdgcn.waterfall.end.i32(token, i32) #4
attributes #0 = { nounwind readnone convergent }
diff --git a/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall-licm.ll b/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall-licm.ll
index 3c9aa3f80a32c..69b7af14a23cd 100644
--- a/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall-licm.ll
+++ b/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall-licm.ll
@@ -29,12 +29,12 @@ loop.body:
br i1 %cmp, label %loop.body, label %loop.exit, !llvm.loop !1
loop.exit:
- %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
- %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %index)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %loop.ptr, i32 %s_idx
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %ptr, align 32
%r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %wf_token, <4 x float> %r)
ret <4 x float> %r1
}
diff --git a/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall.mir b/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall.mir
index 16ecfafe49455..5d2fabedc5e66 100644
--- a/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall.mir
+++ b/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall.mir
@@ -16,18 +16,18 @@ body: |
; CHECK-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr0
; CHECK-NEXT: [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr1
; CHECK-NEXT: [[REG_SEQUENCE:%[0-9]+]]:sgpr_128 = REG_SEQUENCE [[COPY]], %subreg.sub0, [[COPY]], %subreg.sub1, [[COPY]], %subreg.sub2, [[COPY]], %subreg.sub3
- ; CHECK-NEXT: [[S_MOV_B32_:%[0-9]+]]:sreg_32_xm0 = S_MOV_B32 0
+ ; CHECK-NEXT: [[DEF:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: .1:
; CHECK-NEXT: successors: %bb.2(0x80000000)
; CHECK-NEXT: {{ $}}
+ ; CHECK-NEXT: [[S_MOV_B32_:%[0-9]+]]:sreg_32_xm0_xexec = S_MOV_B32 $exec_lo
; CHECK-NEXT: [[S_MOV_B32_1:%[0-9]+]]:sreg_32_xm0_xexec = S_MOV_B32 $exec_lo
- ; CHECK-NEXT: [[S_MOV_B32_2:%[0-9]+]]:sreg_32_xm0_xexec = S_MOV_B32 $exec_lo
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: .2:
; CHECK-NEXT: successors: %bb.2(0x40000000), %bb.3(0x40000000)
; CHECK-NEXT: {{ $}}
- ; CHECK-NEXT: [[PHI:%[0-9]+]]:sreg_32_xm0_xexec = PHI [[S_MOV_B32_1]], %bb.1, %14, %bb.2
+ ; CHECK-NEXT: [[PHI:%[0-9]+]]:sreg_32_xm0_xexec = PHI [[S_MOV_B32_]], %bb.1, %14, %bb.2
; CHECK-NEXT: [[V_READFIRSTLANE_B32_:%[0-9]+]]:sreg_32_xm0 = V_READFIRSTLANE_B32 [[COPY2]], implicit $exec
; CHECK-NEXT: V_CMPX_EQ_U32_nosdst_e32 [[V_READFIRSTLANE_B32_]], [[COPY2]], implicit-def $exec, implicit $exec
; CHECK-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:sreg_64 = REG_SEQUENCE [[V_READFIRSTLANE_B32_]], %subreg.sub0, [[V_READFIRSTLANE_B32_]], %subreg.sub1
@@ -38,17 +38,17 @@ body: |
; CHECK-NEXT: SI_WATERFALL_LOOP %bb.2, implicit $exec
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: .3:
- ; CHECK-NEXT: $exec_lo = S_MOV_B32 [[S_MOV_B32_2]]
+ ; CHECK-NEXT: $exec_lo = S_MOV_B32 [[S_MOV_B32_1]]
%0:sgpr_32 = COPY $sgpr0
%1:vgpr_32 = COPY $vgpr0
%2:vgpr_32 = COPY $vgpr1
%3:sgpr_128 = REG_SEQUENCE %0:sgpr_32, %subreg.sub0, %0:sgpr_32, %subreg.sub1, %0:sgpr_32, %subreg.sub2, %0:sgpr_32, %subreg.sub3
- %4:sreg_32_xm0 = S_MOV_B32 0
- %5:sreg_32 = SI_WATERFALL_BEGIN_V1 killed %4:sreg_32_xm0, %2:vgpr_32, implicit-def $scc
- %6:sreg_32_xm0 = SI_WATERFALL_READFIRSTLANE_V1 %5:sreg_32, %2:vgpr_32, implicit-def $scc
+ %4:sgpr_32 = IMPLICIT_DEF
+ %5:sgpr_32 = SI_WATERFALL_BEGIN_V1 killed %4:sgpr_32, %2:vgpr_32, implicit-def $scc
+ %6:sreg_32_xm0 = SI_WATERFALL_READFIRSTLANE_V1 %5:sgpr_32, %2:vgpr_32, implicit-def $scc
%7:sreg_64 = REG_SEQUENCE %6:sreg_32_xm0, %subreg.sub0, %6:sreg_32_xm0, %subreg.sub1
%8:sgpr_256 = S_LOAD_DWORDX8_IMM killed %7:sreg_64, 0, 0
%9:vreg_128 = IMAGE_SAMPLE_V4_V1_gfx10 %1:vgpr_32, killed %8:sgpr_256, killed %3:sgpr_128, 15, 0, 0, 0, 0, 0, 0, 0, 0, implicit $exec :: (dereferenceable load (s128), addrspace 8)
- %10:vreg_128 = SI_WATERFALL_END_V4 %5:sreg_32, killed %9:vreg_128, implicit-def $scc
+ %10:vreg_128 = SI_WATERFALL_END_V4 %5:sgpr_32, killed %9:vreg_128, implicit-def $scc
...
diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
index d7d717c4409a2..b0d578f529e66 100644
--- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
+++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
@@ -346,10 +346,10 @@ define amdgpu_ps void @test_waterfall_readlane(i32 addrspace(1)* inreg %out, <2
%args = load <2 x i32>, <2 x i32> addrspace(1)* %gep.in
%value = extractelement <2 x i32> %args, i32 0
%lane = extractelement <2 x i32> %args, i32 1
- %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %lane)
- %readlane = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %lane)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %lane)
+ %readlane = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %lane)
%readlane1 = call i32 @llvm.amdgcn.readlane(i32 %value, i32 %readlane)
- %readlane2 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %wf_token, i32 %readlane1)
+ %readlane2 = call i32 @llvm.amdgcn.waterfall.end.i32(token %wf_token, i32 %readlane1)
; This store instruction should be outside the waterfall loop and the value
; being stored generated incrementally in the loop itself
store i32 %readlane2, i32 addrspace(1)* %out, align 4
@@ -766,12 +766,12 @@ define amdgpu_ps <4 x float> @test_waterfall_non_uniform_img(<8 x i32> addrspace
; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
- %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
- %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %index)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
%r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %wf_token, <4 x float> %r)
ret <4 x float> %r1
}
@@ -1341,10 +1341,10 @@ define amdgpu_ps <4 x float> @test_waterfall_non_uniform_img_single_read(<8 x i3
; GFX12-GISEL-NEXT: ; return to shader part epilog
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %index
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
- %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
- %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %wf_token, <8 x i32> %rsrc)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %wf_token, <8 x i32> %rsrc)
%r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %s_rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %wf_token, <4 x float> %r)
ret <4 x float> %r1
}
@@ -1775,18 +1775,18 @@ define amdgpu_ps void @test_multiple_groups(i32 addrspace(1)* inreg %out1, i32 a
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-GISEL-NEXT: global_store_b32 v0, v3, s[2:3]
; GFX12-GISEL-NEXT: s_endpgm
- %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
- %readlane1 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %idx1)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %readlane1 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %idx1)
%readlane1.1 = call i32 @llvm.amdgcn.readlane(i32 %val, i32 %readlane1)
- %readlane1.2 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %wf_token, i32 %readlane1.1)
+ %readlane1.2 = call i32 @llvm.amdgcn.waterfall.end.i32(token %wf_token, i32 %readlane1.1)
; This store instruction should be outside the waterfall loop and the value
; being stored generated incrementally in the loop itself
store i32 %readlane1.2, i32 addrspace(1)* %out1, align 4
- %wf_token2 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx2)
- %readlane2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token2, i32 %idx2)
+ %wf_token2 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx2)
+ %readlane2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token2, i32 %idx2)
%readlane2.1 = call i32 @llvm.amdgcn.readlane(i32 %val, i32 %readlane2)
- %readlane2.2 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %wf_token2, i32 %readlane2.1)
+ %readlane2.2 = call i32 @llvm.amdgcn.waterfall.end.i32(token %wf_token2, i32 %readlane2.1)
store i32 %readlane2.2, i32 addrspace(1)* %out2, align 4
ret void
@@ -2252,15 +2252,15 @@ define amdgpu_ps <4 x float> @test_waterfall_non_uniform_img_multi_rl(<8 x i32>
; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
- %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
- %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %index)
- %s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %val)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %index)
+ %s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %val)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
%ptr2 = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %samp_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
%samp = load <4 x i32>, <4 x i32> addrspace(4) * %ptr2, align 32
%r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %wf_token, <4 x float> %r)
ret <4 x float> %r1
}
@@ -2745,8 +2745,8 @@ define amdgpu_ps <4 x float> @test_waterfall_non_uni_img_2_idx(<8 x i32> addrspa
; GFX12-GISEL-NEXT: ; return to shader part epilog
%t_idx = insertelement <2 x i32> %dummy, i32 %index1, i32 0
%combined_idx = insertelement <2 x i32> %t_idx, i32 %index2, i32 1
- %wf_token = call i32 @llvm.amdgcn.waterfall.begin.v2i32(i32 0, <2 x i32> %combined_idx)
- %s_c_idx = call <2 x i32> @llvm.amdgcn.waterfall.readfirstlane.v2i32.v2i32(i32 %wf_token, <2 x i32> %combined_idx)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.v2i32(token poison, <2 x i32> %combined_idx)
+ %s_c_idx = call <2 x i32> @llvm.amdgcn.waterfall.readfirstlane.v2i32.v2i32(token %wf_token, <2 x i32> %combined_idx)
%s_idx1 = extractelement <2 x i32> %s_c_idx, i32 0
%s_idx2 = extractelement <2 x i32> %s_c_idx, i32 1
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx1
@@ -2754,7 +2754,7 @@ define amdgpu_ps <4 x float> @test_waterfall_non_uni_img_2_idx(<8 x i32> addrspa
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
%samp = load <4 x i32>, <4 x i32> addrspace(4) * %ptr2, align 32
%r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %wf_token, <4 x float> %r)
ret <4 x float> %r1
}
@@ -3207,9 +3207,9 @@ define amdgpu_ps void @test_waterfall_non_uniform_img_single_store(<8 x i32> add
; GFX12-GISEL-NEXT: s_endpgm
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %index
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
- %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
- %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %wf_token, <8 x i32> %rsrc)
- %s_rsrc_use = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(i32 %wf_token, <8 x i32> %s_rsrc)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %wf_token, <8 x i32> %rsrc)
+ %s_rsrc_use = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(token %wf_token, <8 x i32> %s_rsrc)
call void @llvm.amdgcn.image.store.1d.v4f32.i32(<4 x float> %data, i32 15, i32 %s, <8 x i32> %s_rsrc_use, i32 0, i32 0)
ret void
@@ -3244,9 +3244,9 @@ define amdgpu_ps void @test_remove_waterfall_last_use(<8 x i32> addrspace(4)* in
; GFX12-NEXT: image_store v[2:5], v1, s[0:7] dmask:0xf dim:SQ_RSRC_IMG_1D
; GFX12-NEXT: s_endpgm
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %in, align 32
- %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
- %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %wf_token, <8 x i32> %rsrc)
- %s_rsrc_use = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(i32 %wf_token, <8 x i32> %s_rsrc)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %wf_token, <8 x i32> %rsrc)
+ %s_rsrc_use = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(token %wf_token, <8 x i32> %s_rsrc)
call void @llvm.amdgcn.image.store.1d.v4f32.i32(<4 x float> %data, i32 15, i32 %s, <8 x i32> %s_rsrc_use, i32 0, i32 0)
ret void
@@ -3401,15 +3401,15 @@ define amdgpu_ps <4 x float> @test_remove_waterfall_multi_rl(<8 x i32> addrspace
; GFX12-GISEL-NEXT: image_sample v[0:3], v1, s[0:7], s[8:11] dmask:0xf dim:SQ_RSRC_IMG_1D
; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
- %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
- %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %val1)
- %s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %val2)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %val1)
+ %s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %val2)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
%ptr2 = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %samp_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
%samp = load <4 x i32>, <4 x i32> addrspace(4) * %ptr2, align 32
%r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %wf_token, <4 x float> %r)
ret <4 x float> %r1
}
@@ -3837,15 +3837,15 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
- %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
- %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %index)
- %s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %val)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %index)
+ %s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %val)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
%ptr2 = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %samp_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
%samp = load <4 x i32>, <4 x i32> addrspace(4) * %ptr2, align 32
%r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %wf_token, <4 x float> %r)
ret <4 x float> %r1
}
@@ -4218,15 +4218,15 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX12-GISEL-NEXT: s_mov_b64 exec, 0
; GFX12-GISEL-NEXT: export mrt0, off, off, off, off done
; GFX12-GISEL-NEXT: s_endpgm
- %wf_token = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
- %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %index)
- %s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token, i32 %val)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %index)
+ %s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %val)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
%ptr2 = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %samp_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
%samp = load <4 x i32>, <4 x i32> addrspace(4) * %ptr2, align 32
%r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %wf_token, <4 x float> %r)
%r2 = extractelement <4 x float> %r1, i32 0
%cond = fcmp olt float %r2, 0.000000e+00
br i1 %cond, label %.kill, label %.exit
@@ -4328,18 +4328,18 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
; VI-GISEL-NEXT: flat_load_dwordx4 v[10:13], v[0:1]
; VI-GISEL-NEXT: flat_load_dwordx4 v[14:17], v[2:3]
-; VI-GISEL-NEXT: s_mov_b32 s0, 0
-; VI-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
; VI-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
-; VI-GISEL-NEXT: v_readfirstlane_b32 s1, v4
-; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v5
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[6:7], s1, v4
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v5
-; VI-GISEL-NEXT: s_and_b64 s[6:7], s[6:7], s[8:9]
-; VI-GISEL-NEXT: s_and_saveexec_b64 s[6:7], s[6:7]
-; VI-GISEL-NEXT: s_mov_b32 s1, s0
+; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v4
+; VI-GISEL-NEXT: v_readfirstlane_b32 s6, v5
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[6:7], s6, v5
+; VI-GISEL-NEXT: s_and_b64 s[2:3], s[2:3], s[6:7]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-GISEL-NEXT: s_mov_b32 s6, 0
+; VI-GISEL-NEXT: s_mov_b32 s7, s6
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
-; VI-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s6
; VI-GISEL-NEXT: s_waitcnt vmcnt(2)
; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v6
; VI-GISEL-NEXT: v_readfirstlane_b32 s9, v7
@@ -4355,17 +4355,17 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v15
; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v16
; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v17
-; VI-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s7
; VI-GISEL-NEXT: ; implicit-def: $vgpr4
; VI-GISEL-NEXT: ; implicit-def: $vgpr5
; VI-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13
; VI-GISEL-NEXT: ; implicit-def: $vgpr14_vgpr15_vgpr16_vgpr17
; VI-GISEL-NEXT: s_nop 3
; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
-; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[6:7]
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
; VI-GISEL-NEXT: s_cbranch_execnz .LBB11_1
; VI-GISEL-NEXT: ; %bb.2:
-; VI-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; VI-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
; VI-GISEL-NEXT: ; return to shader part epilog
@@ -4451,18 +4451,18 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX9-GISEL-NEXT: global_load_dwordx4 v[6:9], v[0:1], off
; GFX9-GISEL-NEXT: global_load_dwordx4 v[10:13], v[0:1], off offset:16
; GFX9-GISEL-NEXT: global_load_dwordx4 v[14:17], v[2:3], off
-; GFX9-GISEL-NEXT: s_mov_b32 s0, 0
-; GFX9-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
; GFX9-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s1, v4
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v5
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[6:7], s1, v4
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v5
-; GFX9-GISEL-NEXT: s_and_b64 s[6:7], s[6:7], s[8:9]
-; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[6:7], s[6:7]
-; GFX9-GISEL-NEXT: s_mov_b32 s1, s0
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v4
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s6, v5
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[6:7], s6, v5
+; GFX9-GISEL-NEXT: s_and_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; GFX9-GISEL-NEXT: s_mov_b32 s6, 0
+; GFX9-GISEL-NEXT: s_mov_b32 s7, s6
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s6
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(2)
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v6
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s9, v7
@@ -4478,17 +4478,17 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v15
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v16
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v17
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s7
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr14_vgpr15_vgpr16_vgpr17
; GFX9-GISEL-NEXT: s_nop 3
; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
-; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[6:7]
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB11_1
; GFX9-GISEL-NEXT: ; %bb.2:
-; GFX9-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
; GFX9-GISEL-NEXT: ; return to shader part epilog
@@ -4983,12 +4983,12 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
- %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok, i32 %idx2)
- %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok1, <8 x i32> %rsrc)
- %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok1, <4 x i32> %srsrc)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
+ %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
%r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok1, <4 x float> %r)
ret <4 x float> %r1
}
@@ -5090,58 +5090,58 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
; VI-GISEL-NEXT: flat_load_dwordx4 v[8:11], v[0:1]
; VI-GISEL-NEXT: flat_load_dwordx4 v[12:15], v[2:3]
-; VI-GISEL-NEXT: s_mov_b32 s10, 0
-; VI-GISEL-NEXT: s_mov_b64 s[12:13], exec
+; VI-GISEL-NEXT: s_mov_b64 s[10:11], exec
; VI-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
; VI-GISEL-NEXT: s_waitcnt vmcnt(2)
; VI-GISEL-NEXT: v_readfirstlane_b32 s0, v4
; VI-GISEL-NEXT: v_readfirstlane_b32 s1, v5
; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v6
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[14:15], s0, v4
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s1, v5
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[16:17], s0, v4
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s1, v5
; VI-GISEL-NEXT: v_readfirstlane_b32 s3, v7
-; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s2, v6
+; VI-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s2, v6
; VI-GISEL-NEXT: s_waitcnt vmcnt(1)
; VI-GISEL-NEXT: v_readfirstlane_b32 s4, v8
-; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s3, v7
+; VI-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s3, v7
; VI-GISEL-NEXT: v_readfirstlane_b32 s5, v9
-; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s4, v8
+; VI-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s4, v8
; VI-GISEL-NEXT: v_readfirstlane_b32 s6, v10
-; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s5, v9
+; VI-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s5, v9
; VI-GISEL-NEXT: v_readfirstlane_b32 s7, v11
-; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s6, v10
+; VI-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s6, v10
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
-; VI-GISEL-NEXT: v_readfirstlane_b32 s16, v12
-; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s7, v11
-; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v13
-; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s16, v12
-; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v14
-; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s17, v13
-; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v15
-; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s18, v14
-; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s19, v15
-; VI-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; VI-GISEL-NEXT: s_and_saveexec_b64 s[14:15], s[14:15]
-; VI-GISEL-NEXT: s_mov_b32 s11, s10
-; VI-GISEL-NEXT: v_mov_b32_e32 v0, s10
-; VI-GISEL-NEXT: v_mov_b32_e32 v1, s11
-; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[0:7], s[16:19] dmask:0xf
+; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v12
+; VI-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s7, v11
+; VI-GISEL-NEXT: v_readfirstlane_b32 s13, v13
+; VI-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s12, v12
+; VI-GISEL-NEXT: v_readfirstlane_b32 s14, v14
+; VI-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s13, v13
+; VI-GISEL-NEXT: v_readfirstlane_b32 s15, v15
+; VI-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s14, v14
+; VI-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s15, v15
+; VI-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[16:17], s[16:17]
+; VI-GISEL-NEXT: s_mov_b32 s18, 0
+; VI-GISEL-NEXT: s_mov_b32 s19, s18
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s18
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s19
+; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[0:7], s[12:15] dmask:0xf
; VI-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
; VI-GISEL-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
-; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[14:15]
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[16:17]
; VI-GISEL-NEXT: s_cbranch_execnz .LBB12_1
; VI-GISEL-NEXT: ; %bb.2:
-; VI-GISEL-NEXT: s_mov_b64 exec, s[12:13]
+; VI-GISEL-NEXT: s_mov_b64 exec, s[10:11]
; VI-GISEL-NEXT: s_and_b64 exec, exec, s[8:9]
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
; VI-GISEL-NEXT: ; return to shader part epilog
@@ -5237,58 +5237,58 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX9-GISEL-NEXT: global_load_dwordx4 v[4:7], v[0:1], off
; GFX9-GISEL-NEXT: global_load_dwordx4 v[8:11], v[0:1], off offset:16
; GFX9-GISEL-NEXT: global_load_dwordx4 v[12:15], v[2:3], off
-; GFX9-GISEL-NEXT: s_mov_b32 s10, 0
-; GFX9-GISEL-NEXT: s_mov_b64 s[12:13], exec
+; GFX9-GISEL-NEXT: s_mov_b64 s[10:11], exec
; GFX9-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(2)
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s0, v4
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s1, v5
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v6
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[14:15], s0, v4
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s1, v5
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[16:17], s0, v4
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s1, v5
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s3, v7
-; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s2, v6
+; GFX9-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s2, v6
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(1)
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s4, v8
-; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s3, v7
+; GFX9-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s3, v7
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s5, v9
-; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s4, v8
+; GFX9-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s4, v8
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s6, v10
-; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s5, v9
+; GFX9-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s5, v9
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s7, v11
-; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s6, v10
+; GFX9-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s6, v10
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s16, v12
-; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s7, v11
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v13
-; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s16, v12
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v14
-; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s17, v13
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v15
-; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s18, v14
-; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[20:21], s19, v15
-; GFX9-GISEL-NEXT: s_and_b64 s[14:15], s[14:15], s[20:21]
-; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[14:15], s[14:15]
-; GFX9-GISEL-NEXT: s_mov_b32 s11, s10
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s10
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s11
-; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[0:7], s[16:19] dmask:0xf
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v12
+; GFX9-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s7, v11
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s13, v13
+; GFX9-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s12, v12
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s14, v14
+; GFX9-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s13, v13
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s15, v15
+; GFX9-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s14, v14
+; GFX9-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[18:19], s15, v15
+; GFX9-GISEL-NEXT: s_and_b64 s[16:17], s[16:17], s[18:19]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[16:17], s[16:17]
+; GFX9-GISEL-NEXT: s_mov_b32 s18, 0
+; GFX9-GISEL-NEXT: s_mov_b32 s19, s18
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s18
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s19
+; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[0:7], s[12:15] dmask:0xf
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
-; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[14:15]
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[16:17]
; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB12_1
; GFX9-GISEL-NEXT: ; %bb.2:
-; GFX9-GISEL-NEXT: s_mov_b64 exec, s[12:13]
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[10:11]
; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[8:9]
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
; GFX9-GISEL-NEXT: ; return to shader part epilog
@@ -5817,12 +5817,12 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call i32 @llvm.amdgcn.waterfall.begin.v8i32(i32 0, <8 x i32> %rsrc)
- %tok1 = call i32 @llvm.amdgcn.waterfall.begin.v4i32(i32 %tok, <4 x i32> %srsrc)
- %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok1, <8 x i32> %rsrc)
- %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok1, <4 x i32> %srsrc)
+ %tok = call token @llvm.amdgcn.waterfall.begin.v8i32(token poison, <8 x i32> %rsrc)
+ %tok1 = call token @llvm.amdgcn.waterfall.begin.v4i32(token %tok, <4 x i32> %srsrc)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
+ %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
%r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok1, <4 x float> %r)
ret <4 x float> %r1
}
@@ -5889,35 +5889,35 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; VI-GISEL-NEXT: s_add_u32 s0, s0, s2
; VI-GISEL-NEXT: s_addc_u32 s1, s1, s3
; VI-GISEL-NEXT: s_load_dwordx8 s[8:15], s[0:1], 0x0
-; VI-GISEL-NEXT: s_mov_b32 s0, 0
; VI-GISEL-NEXT: v_mov_b32_e32 v9, s4
-; VI-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
; VI-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
-; VI-GISEL-NEXT: v_readfirstlane_b32 s1, v9
-; VI-GISEL-NEXT: v_readfirstlane_b32 s16, v4
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s1, v9
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[16:17], s16, v4
-; VI-GISEL-NEXT: s_and_b64 s[4:5], s[4:5], s[16:17]
-; VI-GISEL-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
-; VI-GISEL-NEXT: s_mov_b32 s1, s0
+; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v9
+; VI-GISEL-NEXT: v_readfirstlane_b32 s4, v4
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v9
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s4, v4
+; VI-GISEL-NEXT: s_and_b64 s[2:3], s[2:3], s[4:5]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-GISEL-NEXT: s_mov_b32 s4, 0
+; VI-GISEL-NEXT: s_mov_b32 s5, s4
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
-; VI-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s4
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
; VI-GISEL-NEXT: v_readfirstlane_b32 s16, v5
; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v6
; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v7
; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v8
-; VI-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s5
; VI-GISEL-NEXT: ; implicit-def: $vgpr9
; VI-GISEL-NEXT: ; implicit-def: $vgpr4
; VI-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
; VI-GISEL-NEXT: s_waitcnt lgkmcnt(0)
; VI-GISEL-NEXT: s_nop 2
; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
-; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[4:5]
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
; VI-GISEL-NEXT: s_cbranch_execnz .LBB13_1
; VI-GISEL-NEXT: ; %bb.2:
-; VI-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; VI-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
; VI-GISEL-NEXT: ; return to shader part epilog
@@ -5983,35 +5983,35 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX9-GISEL-NEXT: s_add_u32 s0, s0, s2
; GFX9-GISEL-NEXT: s_addc_u32 s1, s1, s3
; GFX9-GISEL-NEXT: s_load_dwordx8 s[8:15], s[0:1], 0x0
-; GFX9-GISEL-NEXT: s_mov_b32 s0, 0
; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, s4
-; GFX9-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
; GFX9-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s1, v9
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s16, v4
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s1, v9
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[16:17], s16, v4
-; GFX9-GISEL-NEXT: s_and_b64 s[4:5], s[4:5], s[16:17]
-; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
-; GFX9-GISEL-NEXT: s_mov_b32 s1, s0
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v9
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s4, v4
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v9
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s4, v4
+; GFX9-GISEL-NEXT: s_and_b64 s[2:3], s[2:3], s[4:5]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; GFX9-GISEL-NEXT: s_mov_b32 s4, 0
+; GFX9-GISEL-NEXT: s_mov_b32 s5, s4
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s4
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s16, v5
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v6
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v7
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v8
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s5
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr9
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
; GFX9-GISEL-NEXT: s_waitcnt lgkmcnt(0)
; GFX9-GISEL-NEXT: s_nop 2
; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
-; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[4:5]
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB13_1
; GFX9-GISEL-NEXT: ; %bb.2:
-; GFX9-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
; GFX9-GISEL-NEXT: ; return to shader part epilog
@@ -6383,12 +6383,12 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
- %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok, i32 %idx2)
- %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok1, <8 x i32> %rsrc)
- %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok1, <4 x i32> %srsrc)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
+ %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
%r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok1, <4 x float> %r)
ret <4 x float> %r1
}
@@ -6466,37 +6466,37 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; VI-GISEL-NEXT: s_add_u32 s0, s2, s0
; VI-GISEL-NEXT: s_addc_u32 s1, s3, s1
; VI-GISEL-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x0
-; VI-GISEL-NEXT: s_mov_b32 s8, 0
; VI-GISEL-NEXT: v_mov_b32_e32 v13, s4
; VI-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; VI-GISEL-NEXT: s_mov_b32 s18, 0
; VI-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
-; VI-GISEL-NEXT: v_readfirstlane_b32 s9, v4
-; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v13
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s9, v4
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[12:13], s12, v13
-; VI-GISEL-NEXT: s_and_b64 s[10:11], s[10:11], s[12:13]
-; VI-GISEL-NEXT: s_and_saveexec_b64 s[10:11], s[10:11]
-; VI-GISEL-NEXT: s_mov_b32 s9, s8
+; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v4
+; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v13
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v4
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s10, v13
+; VI-GISEL-NEXT: s_and_b64 s[8:9], s[8:9], s[10:11]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[16:17], s[8:9]
+; VI-GISEL-NEXT: s_mov_b32 s19, s18
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
-; VI-GISEL-NEXT: v_mov_b32_e32 v0, s8
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s18
; VI-GISEL-NEXT: s_waitcnt vmcnt(1)
-; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v5
-; VI-GISEL-NEXT: v_readfirstlane_b32 s13, v6
-; VI-GISEL-NEXT: v_readfirstlane_b32 s14, v7
-; VI-GISEL-NEXT: v_readfirstlane_b32 s15, v8
+; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; VI-GISEL-NEXT: v_readfirstlane_b32 s9, v6
+; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v7
+; VI-GISEL-NEXT: v_readfirstlane_b32 s11, v8
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
-; VI-GISEL-NEXT: v_readfirstlane_b32 s16, v9
-; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v10
-; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v11
-; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v12
-; VI-GISEL-NEXT: v_mov_b32_e32 v1, s9
+; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v9
+; VI-GISEL-NEXT: v_readfirstlane_b32 s13, v10
+; VI-GISEL-NEXT: v_readfirstlane_b32 s14, v11
+; VI-GISEL-NEXT: v_readfirstlane_b32 s15, v12
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s19
; VI-GISEL-NEXT: ; implicit-def: $vgpr4
; VI-GISEL-NEXT: ; implicit-def: $vgpr13
; VI-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; VI-GISEL-NEXT: s_waitcnt lgkmcnt(0)
; VI-GISEL-NEXT: s_nop 2
-; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[12:19], s[0:3] dmask:0xf
-; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[10:11]
+; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[0:3] dmask:0xf
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[16:17]
; VI-GISEL-NEXT: s_cbranch_execnz .LBB14_1
; VI-GISEL-NEXT: ; %bb.2:
; VI-GISEL-NEXT: s_mov_b64 exec, s[4:5]
@@ -6571,37 +6571,37 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX9-GISEL-NEXT: s_add_u32 s0, s2, s0
; GFX9-GISEL-NEXT: s_addc_u32 s1, s3, s1
; GFX9-GISEL-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x0
-; GFX9-GISEL-NEXT: s_mov_b32 s8, 0
; GFX9-GISEL-NEXT: v_mov_b32_e32 v13, s4
; GFX9-GISEL-NEXT: s_mov_b64 s[4:5], exec
+; GFX9-GISEL-NEXT: s_mov_b32 s18, 0
; GFX9-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s9, v4
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v13
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s9, v4
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[12:13], s12, v13
-; GFX9-GISEL-NEXT: s_and_b64 s[10:11], s[10:11], s[12:13]
-; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[10:11], s[10:11]
-; GFX9-GISEL-NEXT: s_mov_b32 s9, s8
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v4
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v13
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v4
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s10, v13
+; GFX9-GISEL-NEXT: s_and_b64 s[8:9], s[8:9], s[10:11]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[16:17], s[8:9]
+; GFX9-GISEL-NEXT: s_mov_b32 s19, s18
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s8
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s18
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(1)
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v5
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s13, v6
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s14, v7
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s15, v8
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s9, v6
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v7
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s11, v8
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s16, v9
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v10
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v11
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v12
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s9
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v9
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s13, v10
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s14, v11
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s15, v12
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s19
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr13
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX9-GISEL-NEXT: s_waitcnt lgkmcnt(0)
; GFX9-GISEL-NEXT: s_nop 2
-; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[12:19], s[0:3] dmask:0xf
-; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[10:11]
+; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[0:3] dmask:0xf
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[16:17]
; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB14_1
; GFX9-GISEL-NEXT: ; %bb.2:
; GFX9-GISEL-NEXT: s_mov_b64 exec, s[4:5]
@@ -7027,12 +7027,12 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
- %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok, i32 %idx2)
- %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok1, <8 x i32> %rsrc)
- %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok1, <4 x i32> %srsrc)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
+ %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
%r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok1, <4 x float> %r)
ret <4 x float> %r1
}
@@ -7118,19 +7118,19 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
; VI-GISEL-NEXT: flat_load_dwordx4 v[9:12], v[0:1]
; VI-GISEL-NEXT: flat_load_dwordx4 v[13:16], v[2:3]
-; VI-GISEL-NEXT: s_mov_b32 s0, 0
; VI-GISEL-NEXT: v_mov_b32_e32 v17, s4
-; VI-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
; VI-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
-; VI-GISEL-NEXT: v_readfirstlane_b32 s1, v17
-; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v4
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s1, v17
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v4
-; VI-GISEL-NEXT: s_and_b64 s[4:5], s[4:5], s[8:9]
-; VI-GISEL-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
-; VI-GISEL-NEXT: s_mov_b32 s1, s0
+; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v17
+; VI-GISEL-NEXT: v_readfirstlane_b32 s4, v4
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v17
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s4, v4
+; VI-GISEL-NEXT: s_and_b64 s[2:3], s[2:3], s[4:5]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-GISEL-NEXT: s_mov_b32 s4, 0
+; VI-GISEL-NEXT: s_mov_b32 s5, s4
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
-; VI-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s4
; VI-GISEL-NEXT: s_waitcnt vmcnt(2)
; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v5
; VI-GISEL-NEXT: v_readfirstlane_b32 s9, v6
@@ -7146,17 +7146,17 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v14
; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v15
; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v16
-; VI-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s5
; VI-GISEL-NEXT: ; implicit-def: $vgpr17
; VI-GISEL-NEXT: ; implicit-def: $vgpr4
; VI-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; VI-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
; VI-GISEL-NEXT: s_nop 3
; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
-; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[4:5]
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
; VI-GISEL-NEXT: s_cbranch_execnz .LBB15_1
; VI-GISEL-NEXT: ; %bb.2:
-; VI-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; VI-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
; VI-GISEL-NEXT: ; return to shader part epilog
@@ -7236,19 +7236,19 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX9-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
; GFX9-GISEL-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
; GFX9-GISEL-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
-; GFX9-GISEL-NEXT: s_mov_b32 s0, 0
; GFX9-GISEL-NEXT: v_mov_b32_e32 v17, s4
-; GFX9-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
; GFX9-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s1, v17
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v4
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s1, v17
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v4
-; GFX9-GISEL-NEXT: s_and_b64 s[4:5], s[4:5], s[8:9]
-; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
-; GFX9-GISEL-NEXT: s_mov_b32 s1, s0
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v17
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s4, v4
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v17
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s4, v4
+; GFX9-GISEL-NEXT: s_and_b64 s[2:3], s[2:3], s[4:5]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; GFX9-GISEL-NEXT: s_mov_b32 s4, 0
+; GFX9-GISEL-NEXT: s_mov_b32 s5, s4
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s4
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(2)
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v5
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s9, v6
@@ -7264,17 +7264,17 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v14
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v15
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v16
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s5
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr17
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
; GFX9-GISEL-NEXT: s_nop 3
; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
-; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[4:5]
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB15_1
; GFX9-GISEL-NEXT: ; %bb.2:
-; GFX9-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
; GFX9-GISEL-NEXT: ; return to shader part epilog
@@ -7753,12 +7753,12 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
- %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok, i32 %idx2)
- %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok1, <8 x i32> %rsrc)
- %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok1, <4 x i32> %srsrc)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
+ %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
%r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok1, <4 x float> %r)
ret <4 x float> %r1
}
@@ -7844,19 +7844,19 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
; VI-GISEL-NEXT: flat_load_dwordx4 v[9:12], v[0:1]
; VI-GISEL-NEXT: flat_load_dwordx4 v[13:16], v[2:3]
-; VI-GISEL-NEXT: s_mov_b32 s0, 0
; VI-GISEL-NEXT: v_mov_b32_e32 v17, s4
-; VI-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
; VI-GISEL-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
-; VI-GISEL-NEXT: v_readfirstlane_b32 s1, v4
-; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v17
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s1, v4
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v17
-; VI-GISEL-NEXT: s_and_b64 s[4:5], s[4:5], s[8:9]
-; VI-GISEL-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
-; VI-GISEL-NEXT: s_mov_b32 s1, s0
+; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v4
+; VI-GISEL-NEXT: v_readfirstlane_b32 s4, v17
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s4, v17
+; VI-GISEL-NEXT: s_and_b64 s[2:3], s[2:3], s[4:5]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-GISEL-NEXT: s_mov_b32 s4, 0
+; VI-GISEL-NEXT: s_mov_b32 s5, s4
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
-; VI-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s4
; VI-GISEL-NEXT: s_waitcnt vmcnt(2)
; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v5
; VI-GISEL-NEXT: v_readfirstlane_b32 s9, v6
@@ -7872,17 +7872,17 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v14
; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v15
; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v16
-; VI-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s5
; VI-GISEL-NEXT: ; implicit-def: $vgpr4
; VI-GISEL-NEXT: ; implicit-def: $vgpr17
; VI-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; VI-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
; VI-GISEL-NEXT: s_nop 3
; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
-; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[4:5]
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
; VI-GISEL-NEXT: s_cbranch_execnz .LBB16_1
; VI-GISEL-NEXT: ; %bb.2:
-; VI-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; VI-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
; VI-GISEL-NEXT: ; return to shader part epilog
@@ -7962,19 +7962,19 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX9-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
; GFX9-GISEL-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
; GFX9-GISEL-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
-; GFX9-GISEL-NEXT: s_mov_b32 s0, 0
; GFX9-GISEL-NEXT: v_mov_b32_e32 v17, s4
-; GFX9-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
; GFX9-GISEL-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s1, v4
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v17
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s1, v4
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v17
-; GFX9-GISEL-NEXT: s_and_b64 s[4:5], s[4:5], s[8:9]
-; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[4:5], s[4:5]
-; GFX9-GISEL-NEXT: s_mov_b32 s1, s0
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v4
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s4, v17
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[4:5], s4, v17
+; GFX9-GISEL-NEXT: s_and_b64 s[2:3], s[2:3], s[4:5]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; GFX9-GISEL-NEXT: s_mov_b32 s4, 0
+; GFX9-GISEL-NEXT: s_mov_b32 s5, s4
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s4
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(2)
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v5
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s9, v6
@@ -7990,17 +7990,17 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v14
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v15
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v16
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s1
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s5
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr17
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
; GFX9-GISEL-NEXT: s_nop 3
; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
-; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[4:5]
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB16_1
; GFX9-GISEL-NEXT: ; %bb.2:
-; GFX9-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
; GFX9-GISEL-NEXT: ; return to shader part epilog
@@ -8479,12 +8479,12 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
- %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok, i32 %idx2)
- %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok1, <8 x i32> %rsrc)
- %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok1, <4 x i32> %srsrc)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
+ %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
%r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok1, <4 x float> %r)
ret <4 x float> %r1
}
@@ -8512,18 +8512,18 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; VI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
; VI-SDAG-NEXT: flat_load_dwordx4 v[15:18], v[0:1]
; VI-SDAG-NEXT: flat_load_dwordx4 v[7:10], v[2:3]
-; VI-SDAG-NEXT: s_mov_b32 s0, 0
-; VI-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
; VI-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
-; VI-SDAG-NEXT: v_readfirstlane_b32 s1, v5
-; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v6
-; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[6:7], s1, v5
-; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v6
-; VI-SDAG-NEXT: s_and_b64 s[6:7], s[6:7], s[8:9]
-; VI-SDAG-NEXT: s_and_saveexec_b64 s[6:7], s[6:7]
-; VI-SDAG-NEXT: s_mov_b32 s1, s0
-; VI-SDAG-NEXT: v_mov_b32_e32 v20, s1
+; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v5
+; VI-SDAG-NEXT: v_readfirstlane_b32 s6, v6
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v5
+; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[6:7], s6, v6
+; VI-SDAG-NEXT: s_and_b64 s[2:3], s[2:3], s[6:7]
+; VI-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-SDAG-NEXT: s_mov_b32 s6, 0
+; VI-SDAG-NEXT: s_mov_b32 s7, s6
+; VI-SDAG-NEXT: v_mov_b32_e32 v20, s7
; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v11
; VI-SDAG-NEXT: v_readfirstlane_b32 s9, v12
; VI-SDAG-NEXT: v_readfirstlane_b32 s10, v13
@@ -8536,15 +8536,15 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; VI-SDAG-NEXT: v_readfirstlane_b32 s17, v8
; VI-SDAG-NEXT: v_readfirstlane_b32 s18, v9
; VI-SDAG-NEXT: v_readfirstlane_b32 s19, v10
-; VI-SDAG-NEXT: v_mov_b32_e32 v19, s0
+; VI-SDAG-NEXT: v_mov_b32_e32 v19, s6
; VI-SDAG-NEXT: ; implicit-def: $vgpr5
; VI-SDAG-NEXT: ; implicit-def: $vgpr6
; VI-SDAG-NEXT: s_nop 3
; VI-SDAG-NEXT: image_sample v[0:3], v[19:20], s[8:15], s[16:19] dmask:0xf
-; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[6:7]
+; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
; VI-SDAG-NEXT: s_cbranch_execnz .LBB17_1
; VI-SDAG-NEXT: ; %bb.2:
-; VI-SDAG-NEXT: s_mov_b64 exec, s[2:3]
+; VI-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v11
; VI-SDAG-NEXT: v_readfirstlane_b32 s9, v12
; VI-SDAG-NEXT: v_readfirstlane_b32 s10, v13
@@ -8587,18 +8587,18 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
; VI-GISEL-NEXT: flat_load_dwordx4 v[12:15], v[0:1]
; VI-GISEL-NEXT: flat_load_dwordx4 v[16:19], v[2:3]
-; VI-GISEL-NEXT: s_mov_b32 s0, 0
-; VI-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; VI-GISEL-NEXT: s_mov_b32 s20, 0
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
; VI-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
-; VI-GISEL-NEXT: v_readfirstlane_b32 s1, v4
-; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v5
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s1, v4
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s10, v5
-; VI-GISEL-NEXT: s_and_b64 s[8:9], s[8:9], s[10:11]
-; VI-GISEL-NEXT: s_and_saveexec_b64 s[20:21], s[8:9]
-; VI-GISEL-NEXT: s_mov_b32 s1, s0
-; VI-GISEL-NEXT: v_mov_b32_e32 v21, s1
+; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v4
+; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v5
+; VI-GISEL-NEXT: s_and_b64 s[2:3], s[2:3], s[8:9]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-GISEL-NEXT: s_mov_b32 s21, s20
+; VI-GISEL-NEXT: v_mov_b32_e32 v20, s20
; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v8
; VI-GISEL-NEXT: v_readfirstlane_b32 s9, v9
; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v10
@@ -8611,15 +8611,15 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v17
; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v18
; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v19
-; VI-GISEL-NEXT: v_mov_b32_e32 v20, s0
+; VI-GISEL-NEXT: v_mov_b32_e32 v21, s21
; VI-GISEL-NEXT: ; implicit-def: $vgpr4
; VI-GISEL-NEXT: ; implicit-def: $vgpr5
; VI-GISEL-NEXT: s_nop 3
; VI-GISEL-NEXT: image_sample v[0:3], v[20:21], s[8:15], s[16:19] dmask:0xf
-; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[20:21]
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
; VI-GISEL-NEXT: s_cbranch_execnz .LBB17_1
; VI-GISEL-NEXT: ; %bb.2:
-; VI-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; VI-GISEL-NEXT: v_mov_b32_e32 v22, s4
; VI-GISEL-NEXT: v_mov_b32_e32 v23, s5
; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
@@ -8677,18 +8677,18 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX9-SDAG-NEXT: global_load_dwordx4 v[11:14], v[0:1], off offset:16
; GFX9-SDAG-NEXT: global_load_dwordx4 v[15:18], v[0:1], off
; GFX9-SDAG-NEXT: global_load_dwordx4 v[7:10], v[2:3], off
-; GFX9-SDAG-NEXT: s_mov_b32 s0, 0
-; GFX9-SDAG-NEXT: s_mov_b64 s[2:3], exec
+; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
; GFX9-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
-; GFX9-SDAG-NEXT: v_readfirstlane_b32 s1, v5
-; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v6
-; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[6:7], s1, v5
-; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v6
-; GFX9-SDAG-NEXT: s_and_b64 s[6:7], s[6:7], s[8:9]
-; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[6:7], s[6:7]
-; GFX9-SDAG-NEXT: s_mov_b32 s1, s0
-; GFX9-SDAG-NEXT: v_mov_b32_e32 v20, s1
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v5
+; GFX9-SDAG-NEXT: v_readfirstlane_b32 s6, v6
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v5
+; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[6:7], s6, v6
+; GFX9-SDAG-NEXT: s_and_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; GFX9-SDAG-NEXT: s_mov_b32 s6, 0
+; GFX9-SDAG-NEXT: s_mov_b32 s7, s6
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v20, s7
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v15
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s9, v16
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s10, v17
@@ -8701,15 +8701,15 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s17, v8
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s18, v9
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s19, v10
-; GFX9-SDAG-NEXT: v_mov_b32_e32 v19, s0
+; GFX9-SDAG-NEXT: v_mov_b32_e32 v19, s6
; GFX9-SDAG-NEXT: ; implicit-def: $vgpr5
; GFX9-SDAG-NEXT: ; implicit-def: $vgpr6
; GFX9-SDAG-NEXT: s_nop 3
; GFX9-SDAG-NEXT: image_sample v[0:3], v[19:20], s[8:15], s[16:19] dmask:0xf
-; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[6:7]
+; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB17_1
; GFX9-SDAG-NEXT: ; %bb.2:
-; GFX9-SDAG-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v15
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s9, v16
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s10, v17
@@ -8750,18 +8750,18 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX9-GISEL-NEXT: global_load_dwordx4 v[8:11], v[0:1], off
; GFX9-GISEL-NEXT: global_load_dwordx4 v[12:15], v[0:1], off offset:16
; GFX9-GISEL-NEXT: global_load_dwordx4 v[16:19], v[2:3], off
-; GFX9-GISEL-NEXT: s_mov_b32 s0, 0
-; GFX9-GISEL-NEXT: s_mov_b64 s[2:3], exec
+; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
+; GFX9-GISEL-NEXT: s_mov_b32 s20, 0
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
; GFX9-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s1, v4
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v5
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s1, v4
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s10, v5
-; GFX9-GISEL-NEXT: s_and_b64 s[8:9], s[8:9], s[10:11]
-; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[20:21], s[8:9]
-; GFX9-GISEL-NEXT: s_mov_b32 s1, s0
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v21, s1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v4
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v5
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v5
+; GFX9-GISEL-NEXT: s_and_b64 s[2:3], s[2:3], s[8:9]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; GFX9-GISEL-NEXT: s_mov_b32 s21, s20
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v20, s20
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v8
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s9, v9
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v10
@@ -8774,15 +8774,15 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v17
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v18
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v19
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v20, s0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v21, s21
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5
; GFX9-GISEL-NEXT: s_nop 3
; GFX9-GISEL-NEXT: image_sample v[0:3], v[20:21], s[8:15], s[16:19] dmask:0xf
-; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[20:21]
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB17_1
; GFX9-GISEL-NEXT: ; %bb.2:
-; GFX9-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-GISEL-NEXT: v_mov_b32_e32 v22, s4
; GFX9-GISEL-NEXT: v_mov_b32_e32 v23, s5
; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
@@ -9473,19 +9473,19 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
- %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok, i32 %idx2)
- %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok1, <8 x i32> %rsrc)
- %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok1, <4 x i32> %srsrc)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
+ %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
%r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok1, <4 x float> %r)
- %tok2 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx3)
- %tok3 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok2, i32 %idx4)
- %s_rsrc1 = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok3, <8 x i32> %rsrc)
- %s_srsrc1 = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok3, <4 x i32> %srsrc)
+ %tok2 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx3)
+ %tok3 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok2, i32 %idx4)
+ %s_rsrc1 = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok3, <8 x i32> %rsrc)
+ %s_srsrc1 = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok3, <4 x i32> %srsrc)
%r2 = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc1, <4 x i32> %s_srsrc1, i1 false, i32 0, i32 0)
- %r3 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok3, <4 x float> %r2)
+ %r3 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok3, <4 x float> %r2)
%insert = insertvalue { <4 x float>, <4 x float> } %dummy, <4 x float> %r1, 0
%insert1 = insertvalue { <4 x float>, <4 x float> } %insert, <4 x float> %r3, 1
@@ -9501,41 +9501,37 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; PRE-GFX10-NEXT: s_or_saveexec_b64 s[36:37], -1
; PRE-GFX10-NEXT: buffer_store_dword v40, off, s[0:3], s33 ; 4-byte Folded Spill
; PRE-GFX10-NEXT: s_mov_b64 exec, s[36:37]
-; PRE-GFX10-NEXT: v_writelane_b32 v40, s34, 8
+; PRE-GFX10-NEXT: v_writelane_b32 v40, s34, 6
; PRE-GFX10-NEXT: v_writelane_b32 v40, s4, 0
; PRE-GFX10-NEXT: v_writelane_b32 v40, s5, 1
; PRE-GFX10-NEXT: v_writelane_b32 v40, s6, 2
; PRE-GFX10-NEXT: v_writelane_b32 v40, s7, 3
-; PRE-GFX10-NEXT: v_writelane_b32 v40, s8, 4
-; PRE-GFX10-NEXT: v_writelane_b32 v40, s9, 5
-; PRE-GFX10-NEXT: v_writelane_b32 v40, s30, 6
-; PRE-GFX10-NEXT: s_mov_b32 s5, 0
-; PRE-GFX10-NEXT: s_mov_b64 s[6:7], exec
+; PRE-GFX10-NEXT: v_writelane_b32 v40, s30, 4
+; PRE-GFX10-NEXT: s_mov_b64 s[4:5], exec
; PRE-GFX10-NEXT: s_addk_i32 s32, 0x400
-; PRE-GFX10-NEXT: v_writelane_b32 v40, s31, 7
+; PRE-GFX10-NEXT: v_writelane_b32 v40, s31, 5
; PRE-GFX10-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
-; PRE-GFX10-NEXT: v_readfirstlane_b32 s4, v1
-; PRE-GFX10-NEXT: v_cmp_eq_u32_e64 s[34:35], s4, v1
-; PRE-GFX10-NEXT: s_and_saveexec_b64 s[8:9], s[34:35]
-; PRE-GFX10-NEXT: s_swappc_b64 s[30:31], s[4:5]
+; PRE-GFX10-NEXT: v_readfirstlane_b32 s34, v1
+; PRE-GFX10-NEXT: v_cmp_eq_u32_e64 s[36:37], s34, v1
+; PRE-GFX10-NEXT: s_and_saveexec_b64 s[6:7], s[36:37]
+; PRE-GFX10-NEXT: s_mov_b32 s35, 0
+; PRE-GFX10-NEXT: s_swappc_b64 s[30:31], s[34:35]
; PRE-GFX10-NEXT: v_mov_b32_e32 v2, v0
; PRE-GFX10-NEXT: ; implicit-def: $vgpr1
; PRE-GFX10-NEXT: ; implicit-def: $vgpr0
-; PRE-GFX10-NEXT: s_xor_b64 exec, exec, s[8:9]
+; PRE-GFX10-NEXT: s_xor_b64 exec, exec, s[6:7]
; PRE-GFX10-NEXT: s_cbranch_execnz .LBB18_1
; PRE-GFX10-NEXT: ; %bb.2:
-; PRE-GFX10-NEXT: s_mov_b64 exec, s[6:7]
+; PRE-GFX10-NEXT: s_mov_b64 exec, s[4:5]
; PRE-GFX10-NEXT: v_mov_b32_e32 v0, v2
-; PRE-GFX10-NEXT: v_readlane_b32 s31, v40, 7
-; PRE-GFX10-NEXT: v_readlane_b32 s30, v40, 6
-; PRE-GFX10-NEXT: v_readlane_b32 s9, v40, 5
-; PRE-GFX10-NEXT: v_readlane_b32 s8, v40, 4
+; PRE-GFX10-NEXT: v_readlane_b32 s31, v40, 5
+; PRE-GFX10-NEXT: v_readlane_b32 s30, v40, 4
; PRE-GFX10-NEXT: v_readlane_b32 s7, v40, 3
; PRE-GFX10-NEXT: v_readlane_b32 s6, v40, 2
; PRE-GFX10-NEXT: v_readlane_b32 s5, v40, 1
; PRE-GFX10-NEXT: v_readlane_b32 s4, v40, 0
; PRE-GFX10-NEXT: s_mov_b32 s32, s33
-; PRE-GFX10-NEXT: v_readlane_b32 s34, v40, 8
+; PRE-GFX10-NEXT: v_readlane_b32 s34, v40, 6
; PRE-GFX10-NEXT: s_or_saveexec_b64 s[36:37], -1
; PRE-GFX10-NEXT: buffer_load_dword v40, off, s[0:3], s33 ; 4-byte Folded Reload
; PRE-GFX10-NEXT: s_mov_b64 exec, s[36:37]
@@ -9552,38 +9548,34 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX10-32-NEXT: buffer_store_dword v40, off, s[0:3], s33 ; 4-byte Folded Spill
; GFX10-32-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-NEXT: s_mov_b32 exec_lo, s35
-; GFX10-32-NEXT: v_writelane_b32 v40, s34, 6
+; GFX10-32-NEXT: v_writelane_b32 v40, s34, 4
; GFX10-32-NEXT: s_addk_i32 s32, 0x200
; GFX10-32-NEXT: v_writelane_b32 v40, s4, 0
+; GFX10-32-NEXT: s_mov_b32 s4, exec_lo
; GFX10-32-NEXT: v_writelane_b32 v40, s5, 1
-; GFX10-32-NEXT: s_mov_b32 s5, 0
-; GFX10-32-NEXT: v_writelane_b32 v40, s6, 2
-; GFX10-32-NEXT: s_mov_b32 s6, exec_lo
-; GFX10-32-NEXT: v_writelane_b32 v40, s7, 3
-; GFX10-32-NEXT: s_mov_b32 s7, exec_lo
-; GFX10-32-NEXT: v_writelane_b32 v40, s30, 4
-; GFX10-32-NEXT: v_writelane_b32 v40, s31, 5
+; GFX10-32-NEXT: s_mov_b32 s5, exec_lo
+; GFX10-32-NEXT: v_writelane_b32 v40, s30, 2
+; GFX10-32-NEXT: v_writelane_b32 v40, s31, 3
; GFX10-32-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
-; GFX10-32-NEXT: v_readfirstlane_b32 s4, v1
+; GFX10-32-NEXT: v_readfirstlane_b32 s34, v1
; GFX10-32-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
-; GFX10-32-NEXT: v_cmpx_eq_u32_e32 s4, v1
-; GFX10-32-NEXT: s_swappc_b64 s[30:31], s[4:5]
+; GFX10-32-NEXT: v_cmpx_eq_u32_e32 s34, v1
+; GFX10-32-NEXT: s_mov_b32 s35, 0
+; GFX10-32-NEXT: s_swappc_b64 s[30:31], s[34:35]
; GFX10-32-NEXT: v_mov_b32_e32 v2, v0
-; GFX10-32-NEXT: s_andn2_wrexec_b32 s7, s7
+; GFX10-32-NEXT: s_andn2_wrexec_b32 s5, s5
; GFX10-32-NEXT: ; implicit-def: $vgpr1
; GFX10-32-NEXT: ; implicit-def: $vgpr0
; GFX10-32-NEXT: s_cbranch_execnz .LBB18_1
; GFX10-32-NEXT: ; %bb.2:
-; GFX10-32-NEXT: s_mov_b32 exec_lo, s6
+; GFX10-32-NEXT: s_mov_b32 exec_lo, s4
; GFX10-32-NEXT: v_mov_b32_e32 v0, v2
-; GFX10-32-NEXT: v_readlane_b32 s31, v40, 5
-; GFX10-32-NEXT: v_readlane_b32 s30, v40, 4
-; GFX10-32-NEXT: v_readlane_b32 s7, v40, 3
-; GFX10-32-NEXT: v_readlane_b32 s6, v40, 2
+; GFX10-32-NEXT: v_readlane_b32 s31, v40, 3
+; GFX10-32-NEXT: v_readlane_b32 s30, v40, 2
; GFX10-32-NEXT: v_readlane_b32 s5, v40, 1
; GFX10-32-NEXT: v_readlane_b32 s4, v40, 0
; GFX10-32-NEXT: s_mov_b32 s32, s33
-; GFX10-32-NEXT: v_readlane_b32 s34, v40, 6
+; GFX10-32-NEXT: v_readlane_b32 s34, v40, 4
; GFX10-32-NEXT: s_or_saveexec_b32 s35, -1
; GFX10-32-NEXT: buffer_load_dword v40, off, s[0:3], s33 ; 4-byte Folded Reload
; GFX10-32-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
@@ -9601,42 +9593,38 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX10-64-NEXT: buffer_store_dword v40, off, s[0:3], s33 ; 4-byte Folded Spill
; GFX10-64-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-NEXT: s_mov_b64 exec, s[36:37]
-; GFX10-64-NEXT: v_writelane_b32 v40, s34, 8
+; GFX10-64-NEXT: v_writelane_b32 v40, s34, 6
; GFX10-64-NEXT: s_addk_i32 s32, 0x400
; GFX10-64-NEXT: v_writelane_b32 v40, s4, 0
; GFX10-64-NEXT: v_writelane_b32 v40, s5, 1
-; GFX10-64-NEXT: s_mov_b32 s5, 0
+; GFX10-64-NEXT: s_mov_b64 s[4:5], exec
; GFX10-64-NEXT: v_writelane_b32 v40, s6, 2
; GFX10-64-NEXT: v_writelane_b32 v40, s7, 3
; GFX10-64-NEXT: s_mov_b64 s[6:7], exec
-; GFX10-64-NEXT: v_writelane_b32 v40, s8, 4
-; GFX10-64-NEXT: v_writelane_b32 v40, s9, 5
-; GFX10-64-NEXT: s_mov_b64 s[8:9], exec
-; GFX10-64-NEXT: v_writelane_b32 v40, s30, 6
-; GFX10-64-NEXT: v_writelane_b32 v40, s31, 7
+; GFX10-64-NEXT: v_writelane_b32 v40, s30, 4
+; GFX10-64-NEXT: v_writelane_b32 v40, s31, 5
; GFX10-64-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
-; GFX10-64-NEXT: v_readfirstlane_b32 s4, v1
+; GFX10-64-NEXT: v_readfirstlane_b32 s34, v1
; GFX10-64-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
-; GFX10-64-NEXT: v_cmpx_eq_u32_e64 s4, v1
-; GFX10-64-NEXT: s_swappc_b64 s[30:31], s[4:5]
+; GFX10-64-NEXT: v_cmpx_eq_u32_e64 s34, v1
+; GFX10-64-NEXT: s_mov_b32 s35, 0
+; GFX10-64-NEXT: s_swappc_b64 s[30:31], s[34:35]
; GFX10-64-NEXT: v_mov_b32_e32 v2, v0
-; GFX10-64-NEXT: s_andn2_wrexec_b64 s[8:9], s[8:9]
+; GFX10-64-NEXT: s_andn2_wrexec_b64 s[6:7], s[6:7]
; GFX10-64-NEXT: ; implicit-def: $vgpr1
; GFX10-64-NEXT: ; implicit-def: $vgpr0
; GFX10-64-NEXT: s_cbranch_execnz .LBB18_1
; GFX10-64-NEXT: ; %bb.2:
-; GFX10-64-NEXT: s_mov_b64 exec, s[6:7]
+; GFX10-64-NEXT: s_mov_b64 exec, s[4:5]
; GFX10-64-NEXT: v_mov_b32_e32 v0, v2
-; GFX10-64-NEXT: v_readlane_b32 s31, v40, 7
-; GFX10-64-NEXT: v_readlane_b32 s30, v40, 6
-; GFX10-64-NEXT: v_readlane_b32 s9, v40, 5
-; GFX10-64-NEXT: v_readlane_b32 s8, v40, 4
+; GFX10-64-NEXT: v_readlane_b32 s31, v40, 5
+; GFX10-64-NEXT: v_readlane_b32 s30, v40, 4
; GFX10-64-NEXT: v_readlane_b32 s7, v40, 3
; GFX10-64-NEXT: v_readlane_b32 s6, v40, 2
; GFX10-64-NEXT: v_readlane_b32 s5, v40, 1
; GFX10-64-NEXT: v_readlane_b32 s4, v40, 0
; GFX10-64-NEXT: s_mov_b32 s32, s33
-; GFX10-64-NEXT: v_readlane_b32 s34, v40, 8
+; GFX10-64-NEXT: v_readlane_b32 s34, v40, 6
; GFX10-64-NEXT: s_or_saveexec_b64 s[36:37], -1
; GFX10-64-NEXT: buffer_load_dword v40, off, s[0:3], s33 ; 4-byte Folded Reload
; GFX10-64-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
@@ -9653,43 +9641,39 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX1150-NEXT: s_or_saveexec_b64 s[2:3], -1
; GFX1150-NEXT: scratch_store_b32 off, v40, s33 ; 4-byte Folded Spill
; GFX1150-NEXT: s_mov_b64 exec, s[2:3]
-; GFX1150-NEXT: v_writelane_b32 v40, s0, 8
+; GFX1150-NEXT: v_writelane_b32 v40, s0, 6
; GFX1150-NEXT: s_add_i32 s32, s32, 16
; GFX1150-NEXT: v_writelane_b32 v40, s4, 0
; GFX1150-NEXT: v_writelane_b32 v40, s5, 1
-; GFX1150-NEXT: s_mov_b32 s5, 0
+; GFX1150-NEXT: s_mov_b64 s[4:5], exec
; GFX1150-NEXT: v_writelane_b32 v40, s6, 2
; GFX1150-NEXT: v_writelane_b32 v40, s7, 3
; GFX1150-NEXT: s_mov_b64 s[6:7], exec
-; GFX1150-NEXT: v_writelane_b32 v40, s8, 4
-; GFX1150-NEXT: v_writelane_b32 v40, s9, 5
-; GFX1150-NEXT: s_mov_b64 s[8:9], exec
-; GFX1150-NEXT: v_writelane_b32 v40, s30, 6
-; GFX1150-NEXT: v_writelane_b32 v40, s31, 7
+; GFX1150-NEXT: v_writelane_b32 v40, s30, 4
+; GFX1150-NEXT: v_writelane_b32 v40, s31, 5
; GFX1150-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
-; GFX1150-NEXT: v_readfirstlane_b32 s4, v1
-; GFX1150-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX1150-NEXT: v_cmpx_eq_u32_e64 s4, v1
-; GFX1150-NEXT: s_swappc_b64 s[30:31], s[4:5]
+; GFX1150-NEXT: v_readfirstlane_b32 s0, v1
+; GFX1150-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
+; GFX1150-NEXT: v_cmpx_eq_u32_e64 s0, v1
+; GFX1150-NEXT: s_mov_b32 s1, 0
+; GFX1150-NEXT: s_swappc_b64 s[30:31], s[0:1]
; GFX1150-NEXT: v_mov_b32_e32 v2, v0
-; GFX1150-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX1150-NEXT: s_and_not1_wrexec_b64 s[6:7], s[6:7]
; GFX1150-NEXT: ; implicit-def: $vgpr1
; GFX1150-NEXT: ; implicit-def: $vgpr0
; GFX1150-NEXT: s_cbranch_execnz .LBB18_1
; GFX1150-NEXT: ; %bb.2:
-; GFX1150-NEXT: s_mov_b64 exec, s[6:7]
+; GFX1150-NEXT: s_mov_b64 exec, s[4:5]
; GFX1150-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX1150-NEXT: v_mov_b32_e32 v0, v2
-; GFX1150-NEXT: v_readlane_b32 s31, v40, 7
-; GFX1150-NEXT: v_readlane_b32 s30, v40, 6
-; GFX1150-NEXT: v_readlane_b32 s9, v40, 5
-; GFX1150-NEXT: v_readlane_b32 s8, v40, 4
+; GFX1150-NEXT: v_readlane_b32 s31, v40, 5
+; GFX1150-NEXT: v_readlane_b32 s30, v40, 4
; GFX1150-NEXT: v_readlane_b32 s7, v40, 3
; GFX1150-NEXT: v_readlane_b32 s6, v40, 2
; GFX1150-NEXT: v_readlane_b32 s5, v40, 1
; GFX1150-NEXT: v_readlane_b32 s4, v40, 0
; GFX1150-NEXT: s_mov_b32 s32, s33
-; GFX1150-NEXT: v_readlane_b32 s0, v40, 8
+; GFX1150-NEXT: v_readlane_b32 s0, v40, 6
; GFX1150-NEXT: s_or_saveexec_b64 s[2:3], -1
; GFX1150-NEXT: scratch_load_b32 v40, off, s33 ; 4-byte Folded Reload
; GFX1150-NEXT: s_mov_b64 exec, s[2:3]
@@ -9710,45 +9694,41 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX12-NEXT: scratch_store_b32 off, v40, s33 ; 4-byte Folded Spill
; GFX12-NEXT: s_wait_alu depctr_sa_sdst(0)
; GFX12-NEXT: s_mov_b64 exec, s[2:3]
-; GFX12-NEXT: v_writelane_b32 v40, s0, 8
+; GFX12-NEXT: v_writelane_b32 v40, s0, 6
; GFX12-NEXT: s_add_co_i32 s32, s32, 16
; GFX12-NEXT: v_writelane_b32 v40, s4, 0
; GFX12-NEXT: v_writelane_b32 v40, s5, 1
-; GFX12-NEXT: s_mov_b32 s5, 0
+; GFX12-NEXT: s_mov_b64 s[4:5], exec
; GFX12-NEXT: v_writelane_b32 v40, s6, 2
; GFX12-NEXT: v_writelane_b32 v40, s7, 3
; GFX12-NEXT: s_mov_b64 s[6:7], exec
-; GFX12-NEXT: v_writelane_b32 v40, s8, 4
-; GFX12-NEXT: v_writelane_b32 v40, s9, 5
-; GFX12-NEXT: s_mov_b64 s[8:9], exec
-; GFX12-NEXT: v_writelane_b32 v40, s30, 6
-; GFX12-NEXT: v_writelane_b32 v40, s31, 7
+; GFX12-NEXT: v_writelane_b32 v40, s30, 4
+; GFX12-NEXT: v_writelane_b32 v40, s31, 5
; GFX12-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
-; GFX12-NEXT: v_readfirstlane_b32 s4, v1
+; GFX12-NEXT: v_readfirstlane_b32 s0, v1
; GFX12-NEXT: s_wait_alu depctr_va_sdst(0)
; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX12-NEXT: v_cmpx_eq_u32_e64 s4, v1
+; GFX12-NEXT: v_cmpx_eq_u32_e64 s0, v1
+; GFX12-NEXT: s_mov_b32 s1, 0
; GFX12-NEXT: s_wait_alu depctr_sa_sdst(0)
-; GFX12-NEXT: s_swappc_b64 s[30:31], s[4:5]
+; GFX12-NEXT: s_swappc_b64 s[30:31], s[0:1]
; GFX12-NEXT: v_mov_b32_e32 v2, v0
-; GFX12-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
+; GFX12-NEXT: s_and_not1_wrexec_b64 s[6:7], s[6:7]
; GFX12-NEXT: ; implicit-def: $vgpr1
; GFX12-NEXT: ; implicit-def: $vgpr0
; GFX12-NEXT: s_cbranch_execnz .LBB18_1
; GFX12-NEXT: ; %bb.2:
-; GFX12-NEXT: s_mov_b64 exec, s[6:7]
+; GFX12-NEXT: s_mov_b64 exec, s[4:5]
; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX12-NEXT: v_mov_b32_e32 v0, v2
-; GFX12-NEXT: v_readlane_b32 s31, v40, 7
-; GFX12-NEXT: v_readlane_b32 s30, v40, 6
-; GFX12-NEXT: v_readlane_b32 s9, v40, 5
-; GFX12-NEXT: v_readlane_b32 s8, v40, 4
+; GFX12-NEXT: v_readlane_b32 s31, v40, 5
+; GFX12-NEXT: v_readlane_b32 s30, v40, 4
; GFX12-NEXT: v_readlane_b32 s7, v40, 3
; GFX12-NEXT: v_readlane_b32 s6, v40, 2
; GFX12-NEXT: v_readlane_b32 s5, v40, 1
; GFX12-NEXT: v_readlane_b32 s4, v40, 0
; GFX12-NEXT: s_mov_b32 s32, s33
-; GFX12-NEXT: v_readlane_b32 s0, v40, 8
+; GFX12-NEXT: v_readlane_b32 s0, v40, 6
; GFX12-NEXT: s_or_saveexec_b64 s[2:3], -1
; GFX12-NEXT: scratch_load_b32 v40, off, s33 ; 4-byte Folded Reload
; GFX12-NEXT: s_wait_alu depctr_sa_sdst(0)
@@ -9757,12 +9737,12 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX12-NEXT: s_wait_loadcnt 0x0
; GFX12-NEXT: s_wait_alu depctr_sa_sdst(0)
; GFX12-NEXT: s_setpc_b64 s[30:31]
- %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %fptr)
- %s_fptr = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %tok, i32 %fptr)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %fptr)
+ %s_fptr = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %tok, i32 %fptr)
%ext = zext i32 %s_fptr to i64
%f = inttoptr i64 %ext to i32(i32)*
%callres = call amdgpu_gfx i32 %f(i32 %i)
- %r3 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %tok, i32 %callres)
+ %r3 = call i32 @llvm.amdgcn.waterfall.end.i32(token %tok, i32 %callres)
ret i32 %r3
}
@@ -9964,8 +9944,8 @@ define amdgpu_cs void @ds_write_8(i8 %value, i32 %index) #1 {
; GFX12-GISEL-NEXT: s_endpgm
.entry:
%gep = getelementptr [16384 x i32], ptr addrspace(3) @Lds, i32 0, i32 %index
- %0 = call i32 @llvm.amdgcn.waterfall.begin.p3(i32 0, ptr addrspace(3) %gep)
- %1 = call ptr addrspace(3) @llvm.amdgcn.waterfall.last.use.vgpr.p3(i32 %0, ptr addrspace(3) %gep)
+ %0 = call token @llvm.amdgcn.waterfall.begin.p3(token poison, ptr addrspace(3) %gep)
+ %1 = call ptr addrspace(3) @llvm.amdgcn.waterfall.last.use.vgpr.p3(token %0, ptr addrspace(3) %gep)
store i8 %value, ptr addrspace(3) %1, align 1
ret void
}
@@ -10061,22 +10041,23 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; VI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
; VI-GISEL-NEXT: flat_load_dwordx4 v[14:17], v[0:1]
; VI-GISEL-NEXT: flat_load_dwordx4 v[18:21], v[2:3]
-; VI-GISEL-NEXT: s_mov_b32 s0, 0
-; VI-GISEL-NEXT: s_mov_b64 s[2:3], exec
-; VI-GISEL-NEXT: s_mov_b32 s20, 0.5
+; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
; VI-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
-; VI-GISEL-NEXT: v_readfirstlane_b32 s1, v8
-; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v9
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[6:7], s1, v8
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v9
-; VI-GISEL-NEXT: s_and_b64 s[6:7], s[6:7], s[8:9]
-; VI-GISEL-NEXT: s_and_saveexec_b64 s[6:7], s[6:7]
-; VI-GISEL-NEXT: s_mov_b32 s1, s0
+; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v8
+; VI-GISEL-NEXT: v_readfirstlane_b32 s6, v9
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v8
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[6:7], s6, v9
+; VI-GISEL-NEXT: s_and_b64 s[2:3], s[2:3], s[6:7]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; VI-GISEL-NEXT: s_mov_b32 s6, 0
+; VI-GISEL-NEXT: s_mov_b32 s7, s6
; VI-GISEL-NEXT: s_waitcnt vmcnt(1)
-; VI-GISEL-NEXT: v_mov_b32_e32 v0, s0
-; VI-GISEL-NEXT: s_mov_b32 s21, s20
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s6
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s7
+; VI-GISEL-NEXT: s_mov_b32 s6, 0.5
+; VI-GISEL-NEXT: s_mov_b32 s7, s6
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
-; VI-GISEL-NEXT: v_mov_b32_e32 v4, s20
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, s6
; VI-GISEL-NEXT: s_waitcnt vmcnt(2)
; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v10
; VI-GISEL-NEXT: v_readfirstlane_b32 s9, v11
@@ -10092,19 +10073,18 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; VI-GISEL-NEXT: v_readfirstlane_b32 s17, v19
; VI-GISEL-NEXT: v_readfirstlane_b32 s18, v20
; VI-GISEL-NEXT: v_readfirstlane_b32 s19, v21
-; VI-GISEL-NEXT: v_mov_b32_e32 v1, s1
-; VI-GISEL-NEXT: v_mov_b32_e32 v5, s21
-; VI-GISEL-NEXT: s_nop 2
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, s7
+; VI-GISEL-NEXT: s_nop 3
; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
; VI-GISEL-NEXT: image_sample v[4:7], v[4:5], s[8:15], s[16:19] dmask:0xf
; VI-GISEL-NEXT: ; implicit-def: $vgpr8
; VI-GISEL-NEXT: ; implicit-def: $vgpr9
; VI-GISEL-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
; VI-GISEL-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
-; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[6:7]
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
; VI-GISEL-NEXT: s_cbranch_execnz .LBB20_1
; VI-GISEL-NEXT: ; %bb.2:
-; VI-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; VI-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
; VI-GISEL-NEXT: ; return to shader part epilog
@@ -10191,19 +10171,21 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX9-GISEL-NEXT: global_load_dwordx4 v[10:13], v[0:1], off
; GFX9-GISEL-NEXT: global_load_dwordx4 v[14:17], v[0:1], off offset:16
; GFX9-GISEL-NEXT: global_load_dwordx4 v[18:21], v[2:3], off
-; GFX9-GISEL-NEXT: s_mov_b32 s0, 0
-; GFX9-GISEL-NEXT: s_mov_b64 s[2:3], exec
-; GFX9-GISEL-NEXT: s_mov_b32 s20, 0.5
+; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
; GFX9-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s1, v8
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v9
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[6:7], s1, v8
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v9
-; GFX9-GISEL-NEXT: s_and_b64 s[6:7], s[6:7], s[8:9]
-; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[6:7], s[6:7]
-; GFX9-GISEL-NEXT: s_mov_b32 s1, s0
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, s1
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v8
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s6, v9
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v8
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[6:7], s6, v9
+; GFX9-GISEL-NEXT: s_and_b64 s[2:3], s[2:3], s[6:7]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
+; GFX9-GISEL-NEXT: s_mov_b32 s6, 0
+; GFX9-GISEL-NEXT: s_mov_b32 s7, s6
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, s7
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v8, s6
+; GFX9-GISEL-NEXT: s_mov_b32 s6, 0.5
+; GFX9-GISEL-NEXT: s_mov_b32 s7, s6
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v10
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s9, v11
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v12
@@ -10216,21 +10198,19 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s17, v19
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s18, v20
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s19, v21
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v8, s0
-; GFX9-GISEL-NEXT: s_mov_b32 s21, s20
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v10, s20
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v11, s21
-; GFX9-GISEL-NEXT: s_nop 0
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v11, s7
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v10, s6
+; GFX9-GISEL-NEXT: s_nop 2
; GFX9-GISEL-NEXT: image_sample v[0:3], v[8:9], s[8:15], s[16:19] dmask:0xf
; GFX9-GISEL-NEXT: image_sample v[4:7], v[10:11], s[8:15], s[16:19] dmask:0xf
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr8
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr9
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
-; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[6:7]
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB20_1
; GFX9-GISEL-NEXT: ; %bb.2:
-; GFX9-GISEL-NEXT: s_mov_b64 exec, s[2:3]
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
; GFX9-GISEL-NEXT: ; return to shader part epilog
@@ -10742,14 +10722,14 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
- %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok, i32 %idx2)
- %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32 %tok1, <8 x i32> %rsrc)
- %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok1, <4 x i32> %srsrc)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
+ %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
%r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
%r2 = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.500000e+00, float 0.500000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
- %r3 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r2)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok1, <4 x float> %r)
+ %r3 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok1, <4 x float> %r2)
%insert = insertvalue { <4 x float>, <4 x float> } %dummy, <4 x float> %r1, 0
%insert1 = insertvalue { <4 x float>, <4 x float> } %insert, <4 x float> %r3, 1
@@ -10810,51 +10790,51 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; VI-GISEL: ; %bb.0:
; VI-GISEL-NEXT: s_mov_b64 s[4:5], exec
; VI-GISEL-NEXT: s_wqm_b64 exec, exec
-; VI-GISEL-NEXT: s_mov_b32 s6, 0
; VI-GISEL-NEXT: v_mov_b32_e32 v8, v0
; VI-GISEL-NEXT: v_mov_b32_e32 v11, v1
; VI-GISEL-NEXT: v_mov_b32_e32 v9, v2
; VI-GISEL-NEXT: v_mov_b32_e32 v10, v3
-; VI-GISEL-NEXT: s_mov_b64 s[8:9], exec
-; VI-GISEL-NEXT: s_mov_b32 s24, 0.5
+; VI-GISEL-NEXT: s_mov_b64 s[6:7], exec
; VI-GISEL-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
-; VI-GISEL-NEXT: v_readfirstlane_b32 s7, v8
-; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v11
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s7, v8
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[12:13], s12, v11
-; VI-GISEL-NEXT: s_and_b64 s[10:11], s[10:11], s[12:13]
-; VI-GISEL-NEXT: s_and_saveexec_b64 s[10:11], s[10:11]
-; VI-GISEL-NEXT: v_readfirstlane_b32 s12, v9
-; VI-GISEL-NEXT: v_readfirstlane_b32 s14, v10
-; VI-GISEL-NEXT: s_ashr_i32 s13, s12, 31
-; VI-GISEL-NEXT: s_ashr_i32 s15, s14, 31
-; VI-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
-; VI-GISEL-NEXT: s_add_u32 s12, s0, s12
-; VI-GISEL-NEXT: s_addc_u32 s13, s1, s13
-; VI-GISEL-NEXT: s_lshl_b64 s[14:15], s[14:15], 4
-; VI-GISEL-NEXT: s_add_u32 s20, s2, s14
-; VI-GISEL-NEXT: s_addc_u32 s21, s3, s15
-; VI-GISEL-NEXT: s_load_dwordx8 s[12:19], s[12:13], 0x0
-; VI-GISEL-NEXT: s_load_dwordx4 s[20:23], s[20:21], 0x0
-; VI-GISEL-NEXT: s_mov_b32 s7, s6
+; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v8
+; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v11
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v8
+; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s10, v11
+; VI-GISEL-NEXT: s_and_b64 s[8:9], s[8:9], s[10:11]
+; VI-GISEL-NEXT: s_and_saveexec_b64 s[20:21], s[8:9]
+; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v9
+; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v10
+; VI-GISEL-NEXT: s_ashr_i32 s9, s8, 31
+; VI-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; VI-GISEL-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; VI-GISEL-NEXT: s_add_u32 s8, s0, s8
+; VI-GISEL-NEXT: s_addc_u32 s9, s1, s9
+; VI-GISEL-NEXT: s_lshl_b64 s[10:11], s[10:11], 4
+; VI-GISEL-NEXT: s_add_u32 s16, s2, s10
+; VI-GISEL-NEXT: s_addc_u32 s17, s3, s11
+; VI-GISEL-NEXT: s_load_dwordx8 s[8:15], s[8:9], 0x0
+; VI-GISEL-NEXT: s_load_dwordx4 s[16:19], s[16:17], 0x0
+; VI-GISEL-NEXT: s_mov_b32 s22, 0
+; VI-GISEL-NEXT: s_mov_b32 s23, s22
; VI-GISEL-NEXT: s_waitcnt vmcnt(1)
-; VI-GISEL-NEXT: v_mov_b32_e32 v0, s6
-; VI-GISEL-NEXT: s_mov_b32 s25, s24
+; VI-GISEL-NEXT: v_mov_b32_e32 v0, s22
+; VI-GISEL-NEXT: v_mov_b32_e32 v1, s23
+; VI-GISEL-NEXT: s_mov_b32 s22, 0.5
+; VI-GISEL-NEXT: s_mov_b32 s23, s22
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
-; VI-GISEL-NEXT: v_mov_b32_e32 v4, s24
-; VI-GISEL-NEXT: v_mov_b32_e32 v1, s7
-; VI-GISEL-NEXT: v_mov_b32_e32 v5, s25
+; VI-GISEL-NEXT: v_mov_b32_e32 v4, s22
+; VI-GISEL-NEXT: v_mov_b32_e32 v5, s23
; VI-GISEL-NEXT: s_waitcnt lgkmcnt(0)
-; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[12:19], s[20:23] dmask:0xf
-; VI-GISEL-NEXT: image_sample v[4:7], v[4:5], s[12:19], s[20:23] dmask:0xf
+; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
+; VI-GISEL-NEXT: image_sample v[4:7], v[4:5], s[8:15], s[16:19] dmask:0xf
; VI-GISEL-NEXT: ; implicit-def: $vgpr8
; VI-GISEL-NEXT: ; implicit-def: $vgpr11
; VI-GISEL-NEXT: ; implicit-def: $vgpr9
; VI-GISEL-NEXT: ; implicit-def: $vgpr10
-; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[10:11]
+; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[20:21]
; VI-GISEL-NEXT: s_cbranch_execnz .LBB21_1
; VI-GISEL-NEXT: ; %bb.2:
-; VI-GISEL-NEXT: s_mov_b64 exec, s[8:9]
+; VI-GISEL-NEXT: s_mov_b64 exec, s[6:7]
; VI-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
; VI-GISEL-NEXT: ; return to shader part epilog
@@ -10910,49 +10890,49 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX9-GISEL: ; %bb.0:
; GFX9-GISEL-NEXT: s_mov_b64 s[4:5], exec
; GFX9-GISEL-NEXT: s_wqm_b64 exec, exec
-; GFX9-GISEL-NEXT: s_mov_b32 s6, 0
; GFX9-GISEL-NEXT: v_mov_b32_e32 v8, v0
; GFX9-GISEL-NEXT: v_mov_b32_e32 v11, v1
; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, v2
; GFX9-GISEL-NEXT: v_mov_b32_e32 v10, v3
-; GFX9-GISEL-NEXT: s_mov_b64 s[8:9], exec
+; GFX9-GISEL-NEXT: s_mov_b64 s[6:7], exec
; GFX9-GISEL-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s7, v8
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v11
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s7, v8
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[12:13], s12, v11
-; GFX9-GISEL-NEXT: s_and_b64 s[10:11], s[10:11], s[12:13]
-; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[10:11], s[10:11]
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s12, v9
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s14, v10
-; GFX9-GISEL-NEXT: s_ashr_i32 s13, s12, 31
-; GFX9-GISEL-NEXT: s_ashr_i32 s15, s14, 31
-; GFX9-GISEL-NEXT: s_lshl_b64 s[12:13], s[12:13], 5
-; GFX9-GISEL-NEXT: s_add_u32 s24, s0, s12
-; GFX9-GISEL-NEXT: s_addc_u32 s25, s1, s13
-; GFX9-GISEL-NEXT: s_lshl_b64 s[12:13], s[14:15], 4
-; GFX9-GISEL-NEXT: s_add_u32 s26, s2, s12
-; GFX9-GISEL-NEXT: s_addc_u32 s27, s3, s13
-; GFX9-GISEL-NEXT: s_load_dwordx8 s[12:19], s[24:25], 0x0
-; GFX9-GISEL-NEXT: s_load_dwordx4 s[20:23], s[26:27], 0x0
-; GFX9-GISEL-NEXT: s_mov_b32 s7, s6
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, s7
-; GFX9-GISEL-NEXT: s_mov_b32 s24, 0.5
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v8, s6
-; GFX9-GISEL-NEXT: s_mov_b32 s25, s24
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v10, s24
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v11, s25
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v8
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v11
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v8
+; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[10:11], s10, v11
+; GFX9-GISEL-NEXT: s_and_b64 s[8:9], s[8:9], s[10:11]
+; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[20:21], s[8:9]
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v9
+; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v10
+; GFX9-GISEL-NEXT: s_ashr_i32 s9, s8, 31
+; GFX9-GISEL-NEXT: s_ashr_i32 s11, s10, 31
+; GFX9-GISEL-NEXT: s_lshl_b64 s[8:9], s[8:9], 5
+; GFX9-GISEL-NEXT: s_add_u32 s22, s0, s8
+; GFX9-GISEL-NEXT: s_addc_u32 s23, s1, s9
+; GFX9-GISEL-NEXT: s_lshl_b64 s[8:9], s[10:11], 4
+; GFX9-GISEL-NEXT: s_add_u32 s24, s2, s8
+; GFX9-GISEL-NEXT: s_addc_u32 s25, s3, s9
+; GFX9-GISEL-NEXT: s_load_dwordx8 s[8:15], s[22:23], 0x0
+; GFX9-GISEL-NEXT: s_load_dwordx4 s[16:19], s[24:25], 0x0
+; GFX9-GISEL-NEXT: s_mov_b32 s22, 0
+; GFX9-GISEL-NEXT: s_mov_b32 s23, s22
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v8, s22
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, s23
+; GFX9-GISEL-NEXT: s_mov_b32 s22, 0.5
+; GFX9-GISEL-NEXT: s_mov_b32 s23, s22
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v10, s22
+; GFX9-GISEL-NEXT: v_mov_b32_e32 v11, s23
; GFX9-GISEL-NEXT: s_waitcnt lgkmcnt(0)
-; GFX9-GISEL-NEXT: image_sample v[0:3], v[8:9], s[12:19], s[20:23] dmask:0xf
-; GFX9-GISEL-NEXT: image_sample v[4:7], v[10:11], s[12:19], s[20:23] dmask:0xf
+; GFX9-GISEL-NEXT: image_sample v[0:3], v[8:9], s[8:15], s[16:19] dmask:0xf
+; GFX9-GISEL-NEXT: image_sample v[4:7], v[10:11], s[8:15], s[16:19] dmask:0xf
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr8
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr11
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr9
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr10
-; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[10:11]
+; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[20:21]
; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB21_1
; GFX9-GISEL-NEXT: ; %bb.2:
-; GFX9-GISEL-NEXT: s_mov_b64 exec, s[8:9]
+; GFX9-GISEL-NEXT: s_mov_b64 exec, s[6:7]
; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
; GFX9-GISEL-NEXT: ; return to shader part epilog
@@ -11347,10 +11327,10 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX12-GISEL-NEXT: ; return to shader part epilog
<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %s_in,
i32 %idx1, i32 %idx2, i32 %s_idx, i32 %s_idx2, { <4 x float>, <4 x float> } %dummy) #1 {
- %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx1)
- %tok1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %tok, i32 %idx2)
- %widx0 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %tok1, i32 %s_idx)
- %widx1 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %tok1, i32 %s_idx2)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
+ %widx0 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %tok1, i32 %s_idx)
+ %widx1 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %tok1, i32 %s_idx2)
%widx0a = sext i32 %widx0 to i64
%widx1a = sext i32 %widx1 to i64
%rptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i64 %widx0a
@@ -11360,8 +11340,8 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
%r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %rsrc, <4 x i32> %srsrc, i1 false, i32 0, i32 0)
%r2 = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.500000e+00, float 0.500000e+00, <8 x i32> %rsrc, <4 x i32> %srsrc, i1 false, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r)
- %r3 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok1, <4 x float> %r2)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok1, <4 x float> %r)
+ %r3 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok1, <4 x float> %r2)
%insert = insertvalue { <4 x float>, <4 x float> } %dummy, <4 x float> %r1, 0
%insert1 = insertvalue { <4 x float>, <4 x float> } %insert, <4 x float> %r3, 1
@@ -11783,13 +11763,13 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
i32 %idx, <4 x i32> %s_idx, i32 %v_inp, { <4 x float>, float } %dummy) #1 {
- %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx)
- %widx0 = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok, <4 x i32> %s_idx)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx)
+ %widx0 = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok, <4 x i32> %s_idx)
%val = call { <4 x float>, i32 } @llvm.amdgcn.struct.buffer.load.format.sl_v4f32i32s(<4 x i32> %widx0, i32 %v_inp, i32 0, i32 0, i32 0)
%payl = extractvalue { <4 x float>, i32 } %val, 0
%tfe = extractvalue { <4 x float>, i32 } %val, 1
- %r0 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok, <4 x float> %payl)
- %r1 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %tok, i32 %tfe)
+ %r0 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok, <4 x float> %payl)
+ %r1 = call i32 @llvm.amdgcn.waterfall.end.i32(token %tok, i32 %tfe)
%r1.float = bitcast i32 %r1 to float
%insert = insertvalue { <4 x float>, float } %dummy, <4 x float> %r0, 0
@@ -12079,13 +12059,13 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
i32 inreg %idx, <4 x i32> %s_idx, i32 %v_inp, { <4 x float>, float } %dummy) #1 {
- %tok = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %idx)
- %widx0 = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32 %tok, <4 x i32> %s_idx)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx)
+ %widx0 = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok, <4 x i32> %s_idx)
%val = call { <4 x float>, i32 } @llvm.amdgcn.struct.buffer.load.format.sl_v4f32i32s(<4 x i32> %widx0, i32 %v_inp, i32 0, i32 0, i32 0)
%payl = extractvalue { <4 x float>, i32 } %val, 0
%tfe = extractvalue { <4 x float>, i32 } %val, 1
- %r0 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %tok, <4 x float> %payl)
- %r1 = call i32 @llvm.amdgcn.waterfall.end.i32(i32 %tok, i32 %tfe)
+ %r0 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok, <4 x float> %payl)
+ %r1 = call i32 @llvm.amdgcn.waterfall.end.i32(token %tok, i32 %tfe)
%r1.float = bitcast i32 %r1 to float
%insert = insertvalue { <4 x float>, float } %dummy, <4 x float> %r0, 0
@@ -12594,9 +12574,9 @@ define amdgpu_cs_chain void @wf_scc_check(i32 %arg, %struct.wobble %arg1, float
bb:
%call = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> zeroinitializer, i32 0, i32 0)
%icmp = icmp eq i32 %call, 0
- %call3 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %arg)
- %call4 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %call3, i32 %arg)
- %call5 = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(i32 %call3, <8 x i32> zeroinitializer)
+ %call3 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %arg)
+ %call4 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %call3, i32 %arg)
+ %call5 = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(token %call3, <8 x i32> zeroinitializer)
call void @llvm.amdgcn.image.store.2d.f32.i32.v8i32(float 0.000000e+00, i32 0, i32 0, i32 0, <8 x i32> %call5, i32 0, i32 0)
%select = select i1 %icmp, float 1.000000e+00, float 0.000000e+00
%fdiv = fdiv afn float %select, %arg2
@@ -12614,21 +12594,21 @@ bb7: ; preds = %bb
br label %bb6
}
-declare i32 @llvm.amdgcn.waterfall.begin.i32(i32, i32) #6
-declare i32 @llvm.amdgcn.waterfall.begin.v2i32(i32, <2 x i32>) #6
-declare i32 @llvm.amdgcn.waterfall.begin.v4i32(i32, <4 x i32>) #6
-declare i32 @llvm.amdgcn.waterfall.begin.v8i32(i32, <8 x i32>) #1
-declare i32 @llvm.amdgcn.waterfall.begin.p3(i32 , ptr addrspace(3))
-declare i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32, i32) #6
-declare <2 x i32> @llvm.amdgcn.waterfall.readfirstlane.v2i32.v2i32(i32, <2 x i32>) #6
-declare <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(i32, <8 x i32>) #6
-declare <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(i32, <4 x i32>) #1
-declare i16 @llvm.amdgcn.waterfall.end.i16(i32, i16) #6
-declare i32 @llvm.amdgcn.waterfall.end.i32(i32, i32) #6
-declare <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32, <4 x float>) #6
-declare <8 x i32> @llvm.amdgcn.waterfall.end.v8i32(i32, <8 x i32>) #6
-declare <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(i32, <8 x i32>) #6
-declare ptr addrspace(3) @llvm.amdgcn.waterfall.last.use.vgpr.p3(i32, ptr addrspace(3))
+declare token @llvm.amdgcn.waterfall.begin.i32(token, i32) #6
+declare token @llvm.amdgcn.waterfall.begin.v2i32(token, <2 x i32>) #6
+declare token @llvm.amdgcn.waterfall.begin.v4i32(token, <4 x i32>) #6
+declare token @llvm.amdgcn.waterfall.begin.v8i32(token, <8 x i32>) #1
+declare token @llvm.amdgcn.waterfall.begin.p3(token, ptr addrspace(3))
+declare i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token, i32) #6
+declare <2 x i32> @llvm.amdgcn.waterfall.readfirstlane.v2i32.v2i32(token, <2 x i32>) #6
+declare <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token, <8 x i32>) #6
+declare <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token, <4 x i32>) #1
+declare i16 @llvm.amdgcn.waterfall.end.i16(token, i16) #6
+declare i32 @llvm.amdgcn.waterfall.end.i32(token, i32) #6
+declare <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token, <4 x float>) #6
+declare <8 x i32> @llvm.amdgcn.waterfall.end.v8i32(token, <8 x i32>) #6
+declare <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(token, <8 x i32>) #6
+declare ptr addrspace(3) @llvm.amdgcn.waterfall.last.use.vgpr.p3(token, ptr addrspace(3))
declare i32 @llvm.amdgcn.readlane(i32, i32) #0
declare <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32, float, <8 x i32>, <4 x i32>, i1, i32, i32)
declare <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32, float, float, <8 x i32>, <4 x i32>, i1, i32, i32)
diff --git a/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll b/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
index 367e65fb68045..5e96f4a444f5b 100644
--- a/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
+++ b/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
@@ -3,71 +3,71 @@
define amdgpu_ps <4 x float> @test_waterfall_same_index2(<8 x i32> addrspace(4)* inreg %in, i32 %index, float %s, <4 x i32> inreg %samp) {
; CHECK-LABEL: @test_waterfall_same_index2(
-; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 [[INDEX:%.*]])
-; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 [[WF_TOKEN1]], i32 [[INDEX]])
+; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 [[INDEX:%.*]])
+; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token [[WF_TOKEN1]], i32 [[INDEX]])
; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
; CHECK-NEXT: [[PTR:%.*]] = getelementptr [32 x i8], ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
; CHECK-NEXT: [[RSRC:%.*]] = load <8 x i32>, ptr addrspace(4) [[PTR]], align 32
; CHECK-NEXT: [[R:%.*]] = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32.v8i32.v4i32(i32 15, float [[S:%.*]], <8 x i32> [[RSRC]], <4 x i32> [[SAMP:%.*]], i1 false, i32 0, i32 0)
-; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 [[WF_TOKEN1]], <4 x float> [[R]])
+; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token [[WF_TOKEN1]], <4 x float> [[R]])
; CHECK-NEXT: ret <4 x float> [[R1]]
;
- %wf_token1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
- %wf_token2 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %wf_token1, i32 %index)
- %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token2, i32 %index)
+ %wf_token1 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %wf_token2 = call token @llvm.amdgcn.waterfall.begin.i32(token %wf_token1, i32 %index)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token2, i32 %index)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
%r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token2, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %wf_token2, <4 x float> %r)
ret <4 x float> %r1
}
define amdgpu_ps <4 x float> @test_waterfall_same_index3(<8 x i32> addrspace(4)* inreg %in, i32 %index, float %s, <4 x i32> inreg %samp) {
; CHECK-LABEL: @test_waterfall_same_index3(
-; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 [[INDEX:%.*]])
-; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 [[WF_TOKEN1]], i32 [[INDEX]])
+; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 [[INDEX:%.*]])
+; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token [[WF_TOKEN1]], i32 [[INDEX]])
; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
; CHECK-NEXT: [[PTR:%.*]] = getelementptr [32 x i8], ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
; CHECK-NEXT: [[RSRC:%.*]] = load <8 x i32>, ptr addrspace(4) [[PTR]], align 32
; CHECK-NEXT: [[R:%.*]] = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32.v8i32.v4i32(i32 15, float [[S:%.*]], <8 x i32> [[RSRC]], <4 x i32> [[SAMP:%.*]], i1 false, i32 0, i32 0)
-; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 [[WF_TOKEN1]], <4 x float> [[R]])
+; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token [[WF_TOKEN1]], <4 x float> [[R]])
; CHECK-NEXT: ret <4 x float> [[R1]]
;
- %wf_token1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index)
- %wf_token2 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %wf_token1, i32 %index)
- %wf_token3 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %wf_token2, i32 %index)
- %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token3, i32 %index)
+ %wf_token1 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %wf_token2 = call token @llvm.amdgcn.waterfall.begin.i32(token %wf_token1, i32 %index)
+ %wf_token3 = call token @llvm.amdgcn.waterfall.begin.i32(token %wf_token2, i32 %index)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token3, i32 %index)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
%r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token3, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %wf_token3, <4 x float> %r)
ret <4 x float> %r1
}
define amdgpu_ps <4 x float> @test_waterfall_same_index_aba(<8 x i32> addrspace(4)* inreg %in, i32 %index1, i32 %index2, float %s, <4 x i32> inreg %samp) {
; CHECK-LABEL: @test_waterfall_same_index_aba(
-; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 [[INDEX1:%.*]])
-; CHECK-NEXT: [[WF_TOKEN2:%.*]] = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 [[WF_TOKEN1]], i32 [[INDEX2:%.*]])
-; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 [[WF_TOKEN2]], i32 [[INDEX2]])
+; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 [[INDEX1:%.*]])
+; CHECK-NEXT: [[WF_TOKEN2:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token [[WF_TOKEN1]], i32 [[INDEX2:%.*]])
+; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token [[WF_TOKEN2]], i32 [[INDEX2]])
; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
; CHECK-NEXT: [[PTR:%.*]] = getelementptr [32 x i8], ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
; CHECK-NEXT: [[RSRC:%.*]] = load <8 x i32>, ptr addrspace(4) [[PTR]], align 32
; CHECK-NEXT: [[R:%.*]] = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32.v8i32.v4i32(i32 15, float [[S:%.*]], <8 x i32> [[RSRC]], <4 x i32> [[SAMP:%.*]], i1 false, i32 0, i32 0)
-; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 [[WF_TOKEN2]], <4 x float> [[R]])
+; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token [[WF_TOKEN2]], <4 x float> [[R]])
; CHECK-NEXT: ret <4 x float> [[R1]]
;
- %wf_token1 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 0, i32 %index1)
- %wf_token2 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %wf_token1, i32 %index2)
- %wf_token3 = call i32 @llvm.amdgcn.waterfall.begin.i32(i32 %wf_token2, i32 %index1)
- %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32 %wf_token3, i32 %index2)
+ %wf_token1 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index1)
+ %wf_token2 = call token @llvm.amdgcn.waterfall.begin.i32(token %wf_token1, i32 %index2)
+ %wf_token3 = call token @llvm.amdgcn.waterfall.begin.i32(token %wf_token2, i32 %index1)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token3, i32 %index2)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
%r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
- %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32 %wf_token3, <4 x float> %r)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %wf_token3, <4 x float> %r)
ret <4 x float> %r1
}
-declare i32 @llvm.amdgcn.waterfall.begin.i32(i32, i32)
-declare i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(i32, i32)
-declare <4 x float> @llvm.amdgcn.waterfall.end.v4f32(i32, <4 x float>)
+declare token @llvm.amdgcn.waterfall.begin.i32(token, i32)
+declare i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token, i32)
+declare <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token, <4 x float>)
declare <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32, float, <8 x i32>, <4 x i32>, i1, i32, i32)
>From 64f72e7b3664144d21c59fd7589b31d4e17da9d6 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Wed, 27 May 2026 13:38:58 +0100
Subject: [PATCH 14/35] Replace AMDGPUWaterfallHelper.h with
SearchableTables-generated lookup
Assisted-by: Claude <noreply at anthropic.com>
---
llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp | 1 -
.../AMDGPU/AMDGPUInstructionSelector.cpp | 1 -
.../Target/AMDGPU/AMDGPUSearchableTables.td | 46 ++++++++++++
.../lib/Target/AMDGPU/AMDGPUWaterfallHelper.h | 74 -------------------
.../Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp | 7 ++
llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h | 10 +++
6 files changed, 63 insertions(+), 76 deletions(-)
delete mode 100644 llvm/lib/Target/AMDGPU/AMDGPUWaterfallHelper.h
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp b/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp
index a5574eccfff4f..1158330350247 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp
@@ -14,7 +14,6 @@
#include "AMDGPUISelDAGToDAG.h"
#include "AMDGPU.h"
#include "AMDGPUInstrInfo.h"
-#include "AMDGPUWaterfallHelper.h"
#include "AMDGPUSubtarget.h"
#include "AMDGPUTargetMachine.h"
#include "MCTargetDesc/AMDGPUMCTargetDesc.h"
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
index 7de080a8f0b5a..7a085d4aa096e 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
@@ -15,7 +15,6 @@
#include "AMDGPU.h"
#include "AMDGPUGlobalISelUtils.h"
#include "AMDGPUInstrInfo.h"
-#include "AMDGPUWaterfallHelper.h"
#include "AMDGPURegisterBankInfo.h"
#include "AMDGPUTargetMachine.h"
#include "SIMachineFunctionInfo.h"
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUSearchableTables.td b/llvm/lib/Target/AMDGPU/AMDGPUSearchableTables.td
index b83ea7b96afad..02178b6522c74 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUSearchableTables.td
+++ b/llvm/lib/Target/AMDGPU/AMDGPUSearchableTables.td
@@ -433,3 +433,49 @@ def AMDGPUImageDMaskIntrinsicTable : GenericTable {
let PrimaryKeyName = "getAMDGPUImageDMaskIntrinsic";
let PrimaryKeyEarlyOut = 1;
}
+
+//===----------------------------------------------------------------------===//
+// Waterfall intrinsic to pseudo opcode mapping.
+//===----------------------------------------------------------------------===//
+
+class WaterfallPseudoInfo<Intrinsic intr, int size, Instruction opcode> {
+ Intrinsic Intr = intr;
+ bits<8> SizeDwords = size;
+ Instruction Opcode = opcode;
+}
+
+def WaterfallPseudoTable : GenericTable {
+ let FilterClass = "WaterfallPseudoInfo";
+ let CppTypeName = "WaterfallPseudoInfo";
+ let Fields = ["Intr", "SizeDwords", "Opcode"];
+
+ let PrimaryKey = ["Intr", "SizeDwords"];
+ let PrimaryKeyName = "getWaterfallPseudoOpcodeHelper";
+}
+
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_begin, 1, SI_WATERFALL_BEGIN_V1>;
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_begin, 2, SI_WATERFALL_BEGIN_V2>;
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_begin, 4, SI_WATERFALL_BEGIN_V4>;
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_begin, 8, SI_WATERFALL_BEGIN_V8>;
+
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_readfirstlane, 1, SI_WATERFALL_READFIRSTLANE_V1>;
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_readfirstlane, 2, SI_WATERFALL_READFIRSTLANE_V2>;
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_readfirstlane, 4, SI_WATERFALL_READFIRSTLANE_V4>;
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_readfirstlane, 8, SI_WATERFALL_READFIRSTLANE_V8>;
+
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_end, 1, SI_WATERFALL_END_V1>;
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_end, 2, SI_WATERFALL_END_V2>;
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_end, 4, SI_WATERFALL_END_V4>;
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_end, 8, SI_WATERFALL_END_V8>;
+
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_last_use, 1, SI_WATERFALL_LAST_USE_V1>;
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_last_use, 2, SI_WATERFALL_LAST_USE_V2>;
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_last_use, 4, SI_WATERFALL_LAST_USE_V4>;
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_last_use, 8, SI_WATERFALL_LAST_USE_V8>;
+
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_last_use_vgpr, 1, SI_WATERFALL_LAST_USE_V1_V>;
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_last_use_vgpr, 2, SI_WATERFALL_LAST_USE_V2_V>;
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_last_use_vgpr, 4, SI_WATERFALL_LAST_USE_V4_V>;
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_last_use_vgpr, 8, SI_WATERFALL_LAST_USE_V8_V>;
+
+def : WaterfallPseudoInfo<int_amdgcn_waterfall_loop_end, 0, SI_WATERFALL_LOOP_END>;
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUWaterfallHelper.h b/llvm/lib/Target/AMDGPU/AMDGPUWaterfallHelper.h
deleted file mode 100644
index 1f63b680bb8c0..0000000000000
--- a/llvm/lib/Target/AMDGPU/AMDGPUWaterfallHelper.h
+++ /dev/null
@@ -1,74 +0,0 @@
-//===- AMDGPUWaterfallHelper.h - Waterfall intrinsic opcode mapping --------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// Shared helper for mapping waterfall intrinsic IDs and value sizes to
-// SI_WATERFALL_* pseudo opcodes. Used by both SelectionDAG and GlobalISel
-// instruction selectors.
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_LIB_TARGET_AMDGPU_AMDGPUWATERFALLHELPER_H
-#define LLVM_LIB_TARGET_AMDGPU_AMDGPUWATERFALLHELPER_H
-
-#include "MCTargetDesc/AMDGPUMCTargetDesc.h"
-#include "llvm/IR/IntrinsicsAMDGPU.h"
-
-namespace llvm::AMDGPU {
-
-inline unsigned getWaterfallPseudoOpcode(unsigned IntrID,
- unsigned SizeDwords) {
- switch (IntrID) {
- case Intrinsic::amdgcn_waterfall_begin:
- switch (SizeDwords) {
- case 1: return AMDGPU::SI_WATERFALL_BEGIN_V1;
- case 2: return AMDGPU::SI_WATERFALL_BEGIN_V2;
- case 4: return AMDGPU::SI_WATERFALL_BEGIN_V4;
- case 8: return AMDGPU::SI_WATERFALL_BEGIN_V8;
- }
- break;
- case Intrinsic::amdgcn_waterfall_readfirstlane:
- switch (SizeDwords) {
- case 1: return AMDGPU::SI_WATERFALL_READFIRSTLANE_V1;
- case 2: return AMDGPU::SI_WATERFALL_READFIRSTLANE_V2;
- case 4: return AMDGPU::SI_WATERFALL_READFIRSTLANE_V4;
- case 8: return AMDGPU::SI_WATERFALL_READFIRSTLANE_V8;
- }
- break;
- case Intrinsic::amdgcn_waterfall_end:
- switch (SizeDwords) {
- case 1: return AMDGPU::SI_WATERFALL_END_V1;
- case 2: return AMDGPU::SI_WATERFALL_END_V2;
- case 4: return AMDGPU::SI_WATERFALL_END_V4;
- case 8: return AMDGPU::SI_WATERFALL_END_V8;
- }
- break;
- case Intrinsic::amdgcn_waterfall_last_use:
- switch (SizeDwords) {
- case 1: return AMDGPU::SI_WATERFALL_LAST_USE_V1;
- case 2: return AMDGPU::SI_WATERFALL_LAST_USE_V2;
- case 4: return AMDGPU::SI_WATERFALL_LAST_USE_V4;
- case 8: return AMDGPU::SI_WATERFALL_LAST_USE_V8;
- }
- break;
- case Intrinsic::amdgcn_waterfall_last_use_vgpr:
- switch (SizeDwords) {
- case 1: return AMDGPU::SI_WATERFALL_LAST_USE_V1_V;
- case 2: return AMDGPU::SI_WATERFALL_LAST_USE_V2_V;
- case 4: return AMDGPU::SI_WATERFALL_LAST_USE_V4_V;
- case 8: return AMDGPU::SI_WATERFALL_LAST_USE_V8_V;
- }
- break;
- case Intrinsic::amdgcn_waterfall_loop_end:
- return AMDGPU::SI_WATERFALL_LOOP_END;
- }
- return 0;
-}
-
-} // namespace llvm::AMDGPU
-
-#endif // LLVM_LIB_TARGET_AMDGPU_AMDGPUWATERFALLHELPER_H
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
index 33df51e8a7e07..4023fe06615e5 100644
--- a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
@@ -309,8 +309,15 @@ unsigned getCompletionActionImplicitArgPosition(unsigned CodeObjectVersion) {
#define GET_MIMGG16MappingTable_IMPL
#define GET_MAIInstInfoTable_IMPL
#define GET_WMMAInstInfoTable_IMPL
+#define GET_WaterfallPseudoTable_IMPL
#include "AMDGPUGenSearchableTables.inc"
+unsigned getWaterfallPseudoOpcode(unsigned IntrID, unsigned SizeDwords) {
+ if (auto *Info = getWaterfallPseudoOpcodeHelper(IntrID, SizeDwords))
+ return Info->Opcode;
+ return 0;
+}
+
int getMIMGOpcode(unsigned BaseOpcode, unsigned MIMGEncoding,
unsigned VDataDwords, unsigned VAddrDwords) {
const MIMGInfo *Info =
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
index 1f7084c8d25ae..b9f4ed41d0fe4 100644
--- a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
+++ b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
@@ -129,6 +129,13 @@ struct WMMAInstInfo {
bool HasMatrixScale;
};
+struct WaterfallPseudoInfo {
+ unsigned Intr;
+ uint8_t SizeDwords;
+ unsigned Opcode;
+};
+
+#define GET_WaterfallPseudoTable_DECL
#define GET_MIMGBaseOpcode_DECL
#define GET_MIMGDim_DECL
#define GET_MIMGEncoding_DECL
@@ -1920,6 +1927,9 @@ class ClusterDimsAttr {
Kind AttrKind = Kind::Unknown;
};
+LLVM_READONLY
+unsigned getWaterfallPseudoOpcode(unsigned IntrID, unsigned SizeDwords);
+
} // namespace AMDGPU
raw_ostream &operator<<(raw_ostream &OS,
>From b7a1ad2d4549d14f9202823103fa51b91507ff85 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Wed, 27 May 2026 14:01:31 +0100
Subject: [PATCH 15/35] Use token none instead of token poison for waterfall
intrinsics
Handle ConstantTokenNone in SelectionDAGBuilder and IRTranslator
so that token none can be used as the initial waterfall token.
Assisted-by: Claude <noreply at anthropic.com>
---
llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp | 2 +
.../SelectionDAG/SelectionDAGBuilder.cpp | 3 ++
.../AMDGPU/llvm.amdgcn.waterfall.ll | 4 +-
.../AMDGPU/amdgcn.waterfall.atomic.opt.ll | 10 ++--
.../AMDGPU/amdgpu-insert-waterfall-licm.ll | 2 +-
.../CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll | 54 +++++++++----------
.../InstCombine/AMDGPU/waterfall.ll | 12 ++---
7 files changed, 46 insertions(+), 41 deletions(-)
diff --git a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
index 8c8e08865744a..cde5da9d987a5 100644
--- a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
@@ -3853,6 +3853,8 @@ bool IRTranslator::translate(const Constant &C, Register Reg) {
EntryBuilder->buildFConstant(Reg, *CF);
} else if (isa<UndefValue>(C))
EntryBuilder->buildUndef(Reg);
+ else if (isa<ConstantTokenNone>(C))
+ EntryBuilder->buildUndef(Reg);
else if (isa<ConstantPointerNull>(C))
EntryBuilder->buildConstant(Reg, 0);
else if (auto GV = dyn_cast<GlobalValue>(&C))
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index eca5bb1598ae0..26a074911e801 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -1891,6 +1891,9 @@ SDValue SelectionDAGBuilder::getValueImpl(const Value *V) {
if (isa<UndefValue>(C) && !V->getType()->isAggregateType())
return isa<PoisonValue>(C) ? DAG.getPOISON(VT) : DAG.getUNDEF(VT);
+ if (isa<ConstantTokenNone>(C))
+ return DAG.getUNDEF(VT);
+
if (const ConstantExpr *CE = dyn_cast<ConstantExpr>(C)) {
visit(CE->getOpcode(), *CE);
SDValue N1 = NodeMap[V];
diff --git a/llvm/test/Analysis/UniformityAnalysis/AMDGPU/llvm.amdgcn.waterfall.ll b/llvm/test/Analysis/UniformityAnalysis/AMDGPU/llvm.amdgcn.waterfall.ll
index 7df5b210001c5..7a2e991980c6b 100644
--- a/llvm/test/Analysis/UniformityAnalysis/AMDGPU/llvm.amdgcn.waterfall.ll
+++ b/llvm/test/Analysis/UniformityAnalysis/AMDGPU/llvm.amdgcn.waterfall.ll
@@ -5,7 +5,7 @@
; CHECK: DIVERGENT: %args = load <2 x i32>, ptr addrspace(1) %gep.in, align 8
; CHECK: DIVERGENT: %value = extractelement <2 x i32> %args, i32 0
; CHECK: DIVERGENT: %lane = extractelement <2 x i32> %args, i32 1
-; CHECK-NOT: DIVERGENT: %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %lane)
+; CHECK-NOT: DIVERGENT: %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %lane)
; CHECK-NOT: DIVERGENT: %readlane = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %lane)
; CHECK-NOT: DIVERGENT: %readlane1 = call i32 @llvm.amdgcn.readlane(i32 %value, i32 %readlane)
; CHECK: DIVERGENT: %readlane2 = call i32 @llvm.amdgcn.waterfall.end.i32(token %wf_token, i32 %readlane1)
@@ -15,7 +15,7 @@ define amdgpu_ps void @test_waterfall_readlane(i32 addrspace(1)* inreg %out, <2
%args = load <2 x i32>, <2 x i32> addrspace(1)* %gep.in
%value = extractelement <2 x i32> %args, i32 0
%lane = extractelement <2 x i32> %args, i32 1
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %lane)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %lane)
%readlane = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %lane)
%readlane1 = call i32 @llvm.amdgcn.readlane(i32 %value, i32 %readlane)
%readlane2 = call i32 @llvm.amdgcn.waterfall.end.i32(token %wf_token, i32 %readlane1)
diff --git a/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll b/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
index 8214232d84f09..2e5f7c0e7c94b 100644
--- a/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
+++ b/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
@@ -61,7 +61,7 @@ bb:
%load = load <2 x i32>, ptr addrspace(1) %getelementptr, align 8
%extractelement = extractelement <2 x i32> %load, i32 0
%extractelement3 = extractelement <2 x i32> %load, i32 1
- %call = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %extractelement3)
+ %call = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %extractelement3)
%call4 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %call, i32 %extractelement3)
%sext = sext i32 %call4 to i64
%getelementptr5 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext
@@ -175,7 +175,7 @@ bb:
%getelementptr5 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext
%load6 = load <4 x i32>, ptr addrspace(4) %getelementptr5, align 4
%call = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %load6, i32 0, i32 0, i32 0, i32 0)
- %call7 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %extractelement4)
+ %call7 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %extractelement4)
%call8 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %call7, i32 %extractelement4)
%sext9 = sext i32 %call8 to i64
%getelementptr10 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext9
@@ -283,7 +283,7 @@ bb:
%load = load <2 x i32>, ptr addrspace(1) %getelementptr, align 8
%extractelement = extractelement <2 x i32> %load, i32 0
%extractelement4 = extractelement <2 x i32> %load, i32 1
- %call7 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %extractelement4)
+ %call7 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %extractelement4)
%call8 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %call7, i32 %extractelement4)
%sext9 = sext i32 %call8 to i64
%getelementptr10 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext9
@@ -377,7 +377,7 @@ bb:
%load = load <2 x i32>, ptr addrspace(1) %getelementptr, align 8
%extractelement = extractelement <2 x i32> %load, i32 0
%extractelement4 = extractelement <2 x i32> %load, i32 1
- %token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %extractelement)
+ %token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %extractelement)
%sext = sext i32 %arg3 to i64
%getelementptr5 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext
%load6 = load <4 x i32>, ptr addrspace(4) %getelementptr5, align 4
@@ -520,7 +520,7 @@ bb:
%idx1 = extractelement <3 x i32> %load, i32 0
%idx2 = extractelement <3 x i32> %load, i32 1
%idx3 = extractelement <3 x i32> %load, i32 2
- %tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
%tok2 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok1, i32 %idx2)
%tok3 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok2, i32 %idx3)
%rfl = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %tok3, i32 %idx3)
diff --git a/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall-licm.ll b/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall-licm.ll
index 69b7af14a23cd..166676bdf6e5c 100644
--- a/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall-licm.ll
+++ b/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall-licm.ll
@@ -29,7 +29,7 @@ loop.body:
br i1 %cmp, label %loop.body, label %loop.exit, !llvm.loop !1
loop.exit:
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %index)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %loop.ptr, i32 %s_idx
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %ptr, align 32
diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
index b0d578f529e66..f311e8d292880 100644
--- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
+++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
@@ -346,7 +346,7 @@ define amdgpu_ps void @test_waterfall_readlane(i32 addrspace(1)* inreg %out, <2
%args = load <2 x i32>, <2 x i32> addrspace(1)* %gep.in
%value = extractelement <2 x i32> %args, i32 0
%lane = extractelement <2 x i32> %args, i32 1
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %lane)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %lane)
%readlane = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %lane)
%readlane1 = call i32 @llvm.amdgcn.readlane(i32 %value, i32 %readlane)
%readlane2 = call i32 @llvm.amdgcn.waterfall.end.i32(token %wf_token, i32 %readlane1)
@@ -766,7 +766,7 @@ define amdgpu_ps <4 x float> @test_waterfall_non_uniform_img(<8 x i32> addrspace
; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %index)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
@@ -1341,7 +1341,7 @@ define amdgpu_ps <4 x float> @test_waterfall_non_uniform_img_single_read(<8 x i3
; GFX12-GISEL-NEXT: ; return to shader part epilog
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %index
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %wf_token, <8 x i32> %rsrc)
%r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %s_rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
%r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %wf_token, <4 x float> %r)
@@ -1775,7 +1775,7 @@ define amdgpu_ps void @test_multiple_groups(i32 addrspace(1)* inreg %out1, i32 a
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-GISEL-NEXT: global_store_b32 v0, v3, s[2:3]
; GFX12-GISEL-NEXT: s_endpgm
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
%readlane1 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %idx1)
%readlane1.1 = call i32 @llvm.amdgcn.readlane(i32 %val, i32 %readlane1)
%readlane1.2 = call i32 @llvm.amdgcn.waterfall.end.i32(token %wf_token, i32 %readlane1.1)
@@ -1783,7 +1783,7 @@ define amdgpu_ps void @test_multiple_groups(i32 addrspace(1)* inreg %out1, i32 a
; being stored generated incrementally in the loop itself
store i32 %readlane1.2, i32 addrspace(1)* %out1, align 4
- %wf_token2 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx2)
+ %wf_token2 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx2)
%readlane2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token2, i32 %idx2)
%readlane2.1 = call i32 @llvm.amdgcn.readlane(i32 %val, i32 %readlane2)
%readlane2.2 = call i32 @llvm.amdgcn.waterfall.end.i32(token %wf_token2, i32 %readlane2.1)
@@ -2252,7 +2252,7 @@ define amdgpu_ps <4 x float> @test_waterfall_non_uniform_img_multi_rl(<8 x i32>
; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %index)
%s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %val)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
@@ -2745,7 +2745,7 @@ define amdgpu_ps <4 x float> @test_waterfall_non_uni_img_2_idx(<8 x i32> addrspa
; GFX12-GISEL-NEXT: ; return to shader part epilog
%t_idx = insertelement <2 x i32> %dummy, i32 %index1, i32 0
%combined_idx = insertelement <2 x i32> %t_idx, i32 %index2, i32 1
- %wf_token = call token @llvm.amdgcn.waterfall.begin.v2i32(token poison, <2 x i32> %combined_idx)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.v2i32(token none, <2 x i32> %combined_idx)
%s_c_idx = call <2 x i32> @llvm.amdgcn.waterfall.readfirstlane.v2i32.v2i32(token %wf_token, <2 x i32> %combined_idx)
%s_idx1 = extractelement <2 x i32> %s_c_idx, i32 0
%s_idx2 = extractelement <2 x i32> %s_c_idx, i32 1
@@ -3207,7 +3207,7 @@ define amdgpu_ps void @test_waterfall_non_uniform_img_single_store(<8 x i32> add
; GFX12-GISEL-NEXT: s_endpgm
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %index
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %wf_token, <8 x i32> %rsrc)
%s_rsrc_use = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(token %wf_token, <8 x i32> %s_rsrc)
call void @llvm.amdgcn.image.store.1d.v4f32.i32(<4 x float> %data, i32 15, i32 %s, <8 x i32> %s_rsrc_use, i32 0, i32 0)
@@ -3244,7 +3244,7 @@ define amdgpu_ps void @test_remove_waterfall_last_use(<8 x i32> addrspace(4)* in
; GFX12-NEXT: image_store v[2:5], v1, s[0:7] dmask:0xf dim:SQ_RSRC_IMG_1D
; GFX12-NEXT: s_endpgm
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %in, align 32
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %wf_token, <8 x i32> %rsrc)
%s_rsrc_use = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(token %wf_token, <8 x i32> %s_rsrc)
call void @llvm.amdgcn.image.store.1d.v4f32.i32(<4 x float> %data, i32 15, i32 %s, <8 x i32> %s_rsrc_use, i32 0, i32 0)
@@ -3401,7 +3401,7 @@ define amdgpu_ps <4 x float> @test_remove_waterfall_multi_rl(<8 x i32> addrspace
; GFX12-GISEL-NEXT: image_sample v[0:3], v1, s[0:7], s[8:11] dmask:0xf dim:SQ_RSRC_IMG_1D
; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %val1)
%s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %val2)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
@@ -3837,7 +3837,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %index)
%s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %val)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
@@ -4218,7 +4218,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX12-GISEL-NEXT: s_mov_b64 exec, 0
; GFX12-GISEL-NEXT: export mrt0, off, off, off, off done
; GFX12-GISEL-NEXT: s_endpgm
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %index)
%s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %val)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
@@ -4983,7 +4983,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
%tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
%s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
@@ -5817,7 +5817,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call token @llvm.amdgcn.waterfall.begin.v8i32(token poison, <8 x i32> %rsrc)
+ %tok = call token @llvm.amdgcn.waterfall.begin.v8i32(token none, <8 x i32> %rsrc)
%tok1 = call token @llvm.amdgcn.waterfall.begin.v4i32(token %tok, <4 x i32> %srsrc)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
%s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
@@ -6383,7 +6383,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
%tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
%s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
@@ -7027,7 +7027,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
%tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
%s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
@@ -7753,7 +7753,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
%tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
%s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
@@ -8479,7 +8479,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
%tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
%s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
@@ -9473,14 +9473,14 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
%tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
%s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
%r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
%r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok1, <4 x float> %r)
- %tok2 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx3)
+ %tok2 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx3)
%tok3 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok2, i32 %idx4)
%s_rsrc1 = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok3, <8 x i32> %rsrc)
%s_srsrc1 = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok3, <4 x i32> %srsrc)
@@ -9737,7 +9737,7 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX12-NEXT: s_wait_loadcnt 0x0
; GFX12-NEXT: s_wait_alu depctr_sa_sdst(0)
; GFX12-NEXT: s_setpc_b64 s[30:31]
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %fptr)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %fptr)
%s_fptr = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %tok, i32 %fptr)
%ext = zext i32 %s_fptr to i64
%f = inttoptr i64 %ext to i32(i32)*
@@ -9944,7 +9944,7 @@ define amdgpu_cs void @ds_write_8(i8 %value, i32 %index) #1 {
; GFX12-GISEL-NEXT: s_endpgm
.entry:
%gep = getelementptr [16384 x i32], ptr addrspace(3) @Lds, i32 0, i32 %index
- %0 = call token @llvm.amdgcn.waterfall.begin.p3(token poison, ptr addrspace(3) %gep)
+ %0 = call token @llvm.amdgcn.waterfall.begin.p3(token none, ptr addrspace(3) %gep)
%1 = call ptr addrspace(3) @llvm.amdgcn.waterfall.last.use.vgpr.p3(token %0, ptr addrspace(3) %gep)
store i8 %value, ptr addrspace(3) %1, align 1
ret void
@@ -10722,7 +10722,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
%tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
%s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
@@ -11327,7 +11327,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX12-GISEL-NEXT: ; return to shader part epilog
<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %s_in,
i32 %idx1, i32 %idx2, i32 %s_idx, i32 %s_idx2, { <4 x float>, <4 x float> } %dummy) #1 {
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
%tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
%widx0 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %tok1, i32 %s_idx)
%widx1 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %tok1, i32 %s_idx2)
@@ -11763,7 +11763,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
i32 %idx, <4 x i32> %s_idx, i32 %v_inp, { <4 x float>, float } %dummy) #1 {
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx)
%widx0 = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok, <4 x i32> %s_idx)
%val = call { <4 x float>, i32 } @llvm.amdgcn.struct.buffer.load.format.sl_v4f32i32s(<4 x i32> %widx0, i32 %v_inp, i32 0, i32 0, i32 0)
%payl = extractvalue { <4 x float>, i32 } %val, 0
@@ -12059,7 +12059,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
i32 inreg %idx, <4 x i32> %s_idx, i32 %v_inp, { <4 x float>, float } %dummy) #1 {
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx)
%widx0 = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok, <4 x i32> %s_idx)
%val = call { <4 x float>, i32 } @llvm.amdgcn.struct.buffer.load.format.sl_v4f32i32s(<4 x i32> %widx0, i32 %v_inp, i32 0, i32 0, i32 0)
%payl = extractvalue { <4 x float>, i32 } %val, 0
@@ -12574,7 +12574,7 @@ define amdgpu_cs_chain void @wf_scc_check(i32 %arg, %struct.wobble %arg1, float
bb:
%call = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> zeroinitializer, i32 0, i32 0)
%icmp = icmp eq i32 %call, 0
- %call3 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %arg)
+ %call3 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %arg)
%call4 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %call3, i32 %arg)
%call5 = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(token %call3, <8 x i32> zeroinitializer)
call void @llvm.amdgcn.image.store.2d.f32.i32.v8i32(float 0.000000e+00, i32 0, i32 0, i32 0, <8 x i32> %call5, i32 0, i32 0)
diff --git a/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll b/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
index 5e96f4a444f5b..2633388bae49d 100644
--- a/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
+++ b/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
@@ -3,7 +3,7 @@
define amdgpu_ps <4 x float> @test_waterfall_same_index2(<8 x i32> addrspace(4)* inreg %in, i32 %index, float %s, <4 x i32> inreg %samp) {
; CHECK-LABEL: @test_waterfall_same_index2(
-; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 [[INDEX:%.*]])
+; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 [[INDEX:%.*]])
; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token [[WF_TOKEN1]], i32 [[INDEX]])
; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
; CHECK-NEXT: [[PTR:%.*]] = getelementptr [32 x i8], ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
@@ -12,7 +12,7 @@ define amdgpu_ps <4 x float> @test_waterfall_same_index2(<8 x i32> addrspace(4)*
; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token [[WF_TOKEN1]], <4 x float> [[R]])
; CHECK-NEXT: ret <4 x float> [[R1]]
;
- %wf_token1 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %wf_token1 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
%wf_token2 = call token @llvm.amdgcn.waterfall.begin.i32(token %wf_token1, i32 %index)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token2, i32 %index)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
@@ -24,7 +24,7 @@ define amdgpu_ps <4 x float> @test_waterfall_same_index2(<8 x i32> addrspace(4)*
define amdgpu_ps <4 x float> @test_waterfall_same_index3(<8 x i32> addrspace(4)* inreg %in, i32 %index, float %s, <4 x i32> inreg %samp) {
; CHECK-LABEL: @test_waterfall_same_index3(
-; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 [[INDEX:%.*]])
+; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 [[INDEX:%.*]])
; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token [[WF_TOKEN1]], i32 [[INDEX]])
; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
; CHECK-NEXT: [[PTR:%.*]] = getelementptr [32 x i8], ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
@@ -33,7 +33,7 @@ define amdgpu_ps <4 x float> @test_waterfall_same_index3(<8 x i32> addrspace(4)*
; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token [[WF_TOKEN1]], <4 x float> [[R]])
; CHECK-NEXT: ret <4 x float> [[R1]]
;
- %wf_token1 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
+ %wf_token1 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
%wf_token2 = call token @llvm.amdgcn.waterfall.begin.i32(token %wf_token1, i32 %index)
%wf_token3 = call token @llvm.amdgcn.waterfall.begin.i32(token %wf_token2, i32 %index)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token3, i32 %index)
@@ -46,7 +46,7 @@ define amdgpu_ps <4 x float> @test_waterfall_same_index3(<8 x i32> addrspace(4)*
define amdgpu_ps <4 x float> @test_waterfall_same_index_aba(<8 x i32> addrspace(4)* inreg %in, i32 %index1, i32 %index2, float %s, <4 x i32> inreg %samp) {
; CHECK-LABEL: @test_waterfall_same_index_aba(
-; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 [[INDEX1:%.*]])
+; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 [[INDEX1:%.*]])
; CHECK-NEXT: [[WF_TOKEN2:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token [[WF_TOKEN1]], i32 [[INDEX2:%.*]])
; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token [[WF_TOKEN2]], i32 [[INDEX2]])
; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
@@ -56,7 +56,7 @@ define amdgpu_ps <4 x float> @test_waterfall_same_index_aba(<8 x i32> addrspace(
; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token [[WF_TOKEN2]], <4 x float> [[R]])
; CHECK-NEXT: ret <4 x float> [[R1]]
;
- %wf_token1 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index1)
+ %wf_token1 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index1)
%wf_token2 = call token @llvm.amdgcn.waterfall.begin.i32(token %wf_token1, i32 %index2)
%wf_token3 = call token @llvm.amdgcn.waterfall.begin.i32(token %wf_token2, i32 %index1)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token3, i32 %index2)
>From fac26acfe46a22c7fe7cd6b6d4ac6f37a97e55d7 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Wed, 27 May 2026 14:16:41 +0100
Subject: [PATCH 16/35] Revert "Use token none instead of token poison for
waterfall intrinsics"
The current version is equivalent to using poison at LLVM IR level. It's not
clear how to lower token none to 0 or something more accurate than undef in
a target specific way, to make detection of start token easier.
This reverts commit 28c92865aa2628434fcb43e6768b63ad384fdef8.
---
llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp | 2 -
.../SelectionDAG/SelectionDAGBuilder.cpp | 3 --
.../AMDGPU/llvm.amdgcn.waterfall.ll | 4 +-
.../AMDGPU/amdgcn.waterfall.atomic.opt.ll | 10 ++--
.../AMDGPU/amdgpu-insert-waterfall-licm.ll | 2 +-
.../CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll | 54 +++++++++----------
.../InstCombine/AMDGPU/waterfall.ll | 12 ++---
7 files changed, 41 insertions(+), 46 deletions(-)
diff --git a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
index cde5da9d987a5..8c8e08865744a 100644
--- a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
@@ -3853,8 +3853,6 @@ bool IRTranslator::translate(const Constant &C, Register Reg) {
EntryBuilder->buildFConstant(Reg, *CF);
} else if (isa<UndefValue>(C))
EntryBuilder->buildUndef(Reg);
- else if (isa<ConstantTokenNone>(C))
- EntryBuilder->buildUndef(Reg);
else if (isa<ConstantPointerNull>(C))
EntryBuilder->buildConstant(Reg, 0);
else if (auto GV = dyn_cast<GlobalValue>(&C))
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 26a074911e801..eca5bb1598ae0 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -1891,9 +1891,6 @@ SDValue SelectionDAGBuilder::getValueImpl(const Value *V) {
if (isa<UndefValue>(C) && !V->getType()->isAggregateType())
return isa<PoisonValue>(C) ? DAG.getPOISON(VT) : DAG.getUNDEF(VT);
- if (isa<ConstantTokenNone>(C))
- return DAG.getUNDEF(VT);
-
if (const ConstantExpr *CE = dyn_cast<ConstantExpr>(C)) {
visit(CE->getOpcode(), *CE);
SDValue N1 = NodeMap[V];
diff --git a/llvm/test/Analysis/UniformityAnalysis/AMDGPU/llvm.amdgcn.waterfall.ll b/llvm/test/Analysis/UniformityAnalysis/AMDGPU/llvm.amdgcn.waterfall.ll
index 7a2e991980c6b..7df5b210001c5 100644
--- a/llvm/test/Analysis/UniformityAnalysis/AMDGPU/llvm.amdgcn.waterfall.ll
+++ b/llvm/test/Analysis/UniformityAnalysis/AMDGPU/llvm.amdgcn.waterfall.ll
@@ -5,7 +5,7 @@
; CHECK: DIVERGENT: %args = load <2 x i32>, ptr addrspace(1) %gep.in, align 8
; CHECK: DIVERGENT: %value = extractelement <2 x i32> %args, i32 0
; CHECK: DIVERGENT: %lane = extractelement <2 x i32> %args, i32 1
-; CHECK-NOT: DIVERGENT: %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %lane)
+; CHECK-NOT: DIVERGENT: %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %lane)
; CHECK-NOT: DIVERGENT: %readlane = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %lane)
; CHECK-NOT: DIVERGENT: %readlane1 = call i32 @llvm.amdgcn.readlane(i32 %value, i32 %readlane)
; CHECK: DIVERGENT: %readlane2 = call i32 @llvm.amdgcn.waterfall.end.i32(token %wf_token, i32 %readlane1)
@@ -15,7 +15,7 @@ define amdgpu_ps void @test_waterfall_readlane(i32 addrspace(1)* inreg %out, <2
%args = load <2 x i32>, <2 x i32> addrspace(1)* %gep.in
%value = extractelement <2 x i32> %args, i32 0
%lane = extractelement <2 x i32> %args, i32 1
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %lane)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %lane)
%readlane = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %lane)
%readlane1 = call i32 @llvm.amdgcn.readlane(i32 %value, i32 %readlane)
%readlane2 = call i32 @llvm.amdgcn.waterfall.end.i32(token %wf_token, i32 %readlane1)
diff --git a/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll b/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
index 2e5f7c0e7c94b..8214232d84f09 100644
--- a/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
+++ b/llvm/test/CodeGen/AMDGPU/amdgcn.waterfall.atomic.opt.ll
@@ -61,7 +61,7 @@ bb:
%load = load <2 x i32>, ptr addrspace(1) %getelementptr, align 8
%extractelement = extractelement <2 x i32> %load, i32 0
%extractelement3 = extractelement <2 x i32> %load, i32 1
- %call = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %extractelement3)
+ %call = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %extractelement3)
%call4 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %call, i32 %extractelement3)
%sext = sext i32 %call4 to i64
%getelementptr5 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext
@@ -175,7 +175,7 @@ bb:
%getelementptr5 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext
%load6 = load <4 x i32>, ptr addrspace(4) %getelementptr5, align 4
%call = call i32 @llvm.amdgcn.struct.buffer.atomic.add.i32(i32 1, <4 x i32> %load6, i32 0, i32 0, i32 0, i32 0)
- %call7 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %extractelement4)
+ %call7 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %extractelement4)
%call8 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %call7, i32 %extractelement4)
%sext9 = sext i32 %call8 to i64
%getelementptr10 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext9
@@ -283,7 +283,7 @@ bb:
%load = load <2 x i32>, ptr addrspace(1) %getelementptr, align 8
%extractelement = extractelement <2 x i32> %load, i32 0
%extractelement4 = extractelement <2 x i32> %load, i32 1
- %call7 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %extractelement4)
+ %call7 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %extractelement4)
%call8 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %call7, i32 %extractelement4)
%sext9 = sext i32 %call8 to i64
%getelementptr10 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext9
@@ -377,7 +377,7 @@ bb:
%load = load <2 x i32>, ptr addrspace(1) %getelementptr, align 8
%extractelement = extractelement <2 x i32> %load, i32 0
%extractelement4 = extractelement <2 x i32> %load, i32 1
- %token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %extractelement)
+ %token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %extractelement)
%sext = sext i32 %arg3 to i64
%getelementptr5 = getelementptr i8, ptr addrspace(4) %arg2, i64 %sext
%load6 = load <4 x i32>, ptr addrspace(4) %getelementptr5, align 4
@@ -520,7 +520,7 @@ bb:
%idx1 = extractelement <3 x i32> %load, i32 0
%idx2 = extractelement <3 x i32> %load, i32 1
%idx3 = extractelement <3 x i32> %load, i32 2
- %tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
+ %tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
%tok2 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok1, i32 %idx2)
%tok3 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok2, i32 %idx3)
%rfl = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %tok3, i32 %idx3)
diff --git a/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall-licm.ll b/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall-licm.ll
index 166676bdf6e5c..69b7af14a23cd 100644
--- a/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall-licm.ll
+++ b/llvm/test/CodeGen/AMDGPU/amdgpu-insert-waterfall-licm.ll
@@ -29,7 +29,7 @@ loop.body:
br i1 %cmp, label %loop.body, label %loop.exit, !llvm.loop !1
loop.exit:
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %index)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %loop.ptr, i32 %s_idx
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %ptr, align 32
diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
index f311e8d292880..b0d578f529e66 100644
--- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
+++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
@@ -346,7 +346,7 @@ define amdgpu_ps void @test_waterfall_readlane(i32 addrspace(1)* inreg %out, <2
%args = load <2 x i32>, <2 x i32> addrspace(1)* %gep.in
%value = extractelement <2 x i32> %args, i32 0
%lane = extractelement <2 x i32> %args, i32 1
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %lane)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %lane)
%readlane = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %lane)
%readlane1 = call i32 @llvm.amdgcn.readlane(i32 %value, i32 %readlane)
%readlane2 = call i32 @llvm.amdgcn.waterfall.end.i32(token %wf_token, i32 %readlane1)
@@ -766,7 +766,7 @@ define amdgpu_ps <4 x float> @test_waterfall_non_uniform_img(<8 x i32> addrspace
; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %index)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
@@ -1341,7 +1341,7 @@ define amdgpu_ps <4 x float> @test_waterfall_non_uniform_img_single_read(<8 x i3
; GFX12-GISEL-NEXT: ; return to shader part epilog
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %index
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %wf_token, <8 x i32> %rsrc)
%r = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32 15, float %s, <8 x i32> %s_rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
%r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %wf_token, <4 x float> %r)
@@ -1775,7 +1775,7 @@ define amdgpu_ps void @test_multiple_groups(i32 addrspace(1)* inreg %out1, i32 a
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-GISEL-NEXT: global_store_b32 v0, v3, s[2:3]
; GFX12-GISEL-NEXT: s_endpgm
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
%readlane1 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %idx1)
%readlane1.1 = call i32 @llvm.amdgcn.readlane(i32 %val, i32 %readlane1)
%readlane1.2 = call i32 @llvm.amdgcn.waterfall.end.i32(token %wf_token, i32 %readlane1.1)
@@ -1783,7 +1783,7 @@ define amdgpu_ps void @test_multiple_groups(i32 addrspace(1)* inreg %out1, i32 a
; being stored generated incrementally in the loop itself
store i32 %readlane1.2, i32 addrspace(1)* %out1, align 4
- %wf_token2 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx2)
+ %wf_token2 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx2)
%readlane2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token2, i32 %idx2)
%readlane2.1 = call i32 @llvm.amdgcn.readlane(i32 %val, i32 %readlane2)
%readlane2.2 = call i32 @llvm.amdgcn.waterfall.end.i32(token %wf_token2, i32 %readlane2.1)
@@ -2252,7 +2252,7 @@ define amdgpu_ps <4 x float> @test_waterfall_non_uniform_img_multi_rl(<8 x i32>
; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %index)
%s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %val)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
@@ -2745,7 +2745,7 @@ define amdgpu_ps <4 x float> @test_waterfall_non_uni_img_2_idx(<8 x i32> addrspa
; GFX12-GISEL-NEXT: ; return to shader part epilog
%t_idx = insertelement <2 x i32> %dummy, i32 %index1, i32 0
%combined_idx = insertelement <2 x i32> %t_idx, i32 %index2, i32 1
- %wf_token = call token @llvm.amdgcn.waterfall.begin.v2i32(token none, <2 x i32> %combined_idx)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.v2i32(token poison, <2 x i32> %combined_idx)
%s_c_idx = call <2 x i32> @llvm.amdgcn.waterfall.readfirstlane.v2i32.v2i32(token %wf_token, <2 x i32> %combined_idx)
%s_idx1 = extractelement <2 x i32> %s_c_idx, i32 0
%s_idx2 = extractelement <2 x i32> %s_c_idx, i32 1
@@ -3207,7 +3207,7 @@ define amdgpu_ps void @test_waterfall_non_uniform_img_single_store(<8 x i32> add
; GFX12-GISEL-NEXT: s_endpgm
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %index
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %wf_token, <8 x i32> %rsrc)
%s_rsrc_use = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(token %wf_token, <8 x i32> %s_rsrc)
call void @llvm.amdgcn.image.store.1d.v4f32.i32(<4 x float> %data, i32 15, i32 %s, <8 x i32> %s_rsrc_use, i32 0, i32 0)
@@ -3244,7 +3244,7 @@ define amdgpu_ps void @test_remove_waterfall_last_use(<8 x i32> addrspace(4)* in
; GFX12-NEXT: image_store v[2:5], v1, s[0:7] dmask:0xf dim:SQ_RSRC_IMG_1D
; GFX12-NEXT: s_endpgm
%rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %in, align 32
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %wf_token, <8 x i32> %rsrc)
%s_rsrc_use = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(token %wf_token, <8 x i32> %s_rsrc)
call void @llvm.amdgcn.image.store.1d.v4f32.i32(<4 x float> %data, i32 15, i32 %s, <8 x i32> %s_rsrc_use, i32 0, i32 0)
@@ -3401,7 +3401,7 @@ define amdgpu_ps <4 x float> @test_remove_waterfall_multi_rl(<8 x i32> addrspace
; GFX12-GISEL-NEXT: image_sample v[0:3], v1, s[0:7], s[8:11] dmask:0xf dim:SQ_RSRC_IMG_1D
; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %val1)
%s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %val2)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
@@ -3837,7 +3837,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX12-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %index)
%s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %val)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
@@ -4218,7 +4218,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX12-GISEL-NEXT: s_mov_b64 exec, 0
; GFX12-GISEL-NEXT: export mrt0, off, off, off, off done
; GFX12-GISEL-NEXT: s_endpgm
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
+ %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %index)
%s_idx2 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token, i32 %val)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
@@ -4983,7 +4983,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
%tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
%s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
@@ -5817,7 +5817,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call token @llvm.amdgcn.waterfall.begin.v8i32(token none, <8 x i32> %rsrc)
+ %tok = call token @llvm.amdgcn.waterfall.begin.v8i32(token poison, <8 x i32> %rsrc)
%tok1 = call token @llvm.amdgcn.waterfall.begin.v4i32(token %tok, <4 x i32> %srsrc)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
%s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
@@ -6383,7 +6383,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
%tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
%s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
@@ -7027,7 +7027,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
%tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
%s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
@@ -7753,7 +7753,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
%tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
%s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
@@ -8479,7 +8479,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
%tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
%s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
@@ -9473,14 +9473,14 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
%tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
%s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
%r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32 15, float 0.000000e+00, float 0.000000e+00, <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, i1 false, i32 0, i32 0)
%r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok1, <4 x float> %r)
- %tok2 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx3)
+ %tok2 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx3)
%tok3 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok2, i32 %idx4)
%s_rsrc1 = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok3, <8 x i32> %rsrc)
%s_srsrc1 = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok3, <4 x i32> %srsrc)
@@ -9737,7 +9737,7 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX12-NEXT: s_wait_loadcnt 0x0
; GFX12-NEXT: s_wait_alu depctr_sa_sdst(0)
; GFX12-NEXT: s_setpc_b64 s[30:31]
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %fptr)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %fptr)
%s_fptr = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %tok, i32 %fptr)
%ext = zext i32 %s_fptr to i64
%f = inttoptr i64 %ext to i32(i32)*
@@ -9944,7 +9944,7 @@ define amdgpu_cs void @ds_write_8(i8 %value, i32 %index) #1 {
; GFX12-GISEL-NEXT: s_endpgm
.entry:
%gep = getelementptr [16384 x i32], ptr addrspace(3) @Lds, i32 0, i32 %index
- %0 = call token @llvm.amdgcn.waterfall.begin.p3(token none, ptr addrspace(3) %gep)
+ %0 = call token @llvm.amdgcn.waterfall.begin.p3(token poison, ptr addrspace(3) %gep)
%1 = call ptr addrspace(3) @llvm.amdgcn.waterfall.last.use.vgpr.p3(token %0, ptr addrspace(3) %gep)
store i8 %value, ptr addrspace(3) %1, align 1
ret void
@@ -10722,7 +10722,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
%sptr = getelementptr <4 x i32>, <4 x i32> addrspace(4)* %s_in, i32 %s_idx2
%rsrc = load <8 x i32>, <8 x i32> addrspace(4)* %rptr, align 16
%srsrc = load <4 x i32>, <4 x i32> addrspace(4)* %sptr, align 16
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
%tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
%s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
%s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
@@ -11327,7 +11327,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX12-GISEL-NEXT: ; return to shader part epilog
<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %s_in,
i32 %idx1, i32 %idx2, i32 %s_idx, i32 %s_idx2, { <4 x float>, <4 x float> } %dummy) #1 {
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx1)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx1)
%tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx2)
%widx0 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %tok1, i32 %s_idx)
%widx1 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %tok1, i32 %s_idx2)
@@ -11763,7 +11763,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
i32 %idx, <4 x i32> %s_idx, i32 %v_inp, { <4 x float>, float } %dummy) #1 {
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx)
%widx0 = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok, <4 x i32> %s_idx)
%val = call { <4 x float>, i32 } @llvm.amdgcn.struct.buffer.load.format.sl_v4f32i32s(<4 x i32> %widx0, i32 %v_inp, i32 0, i32 0, i32 0)
%payl = extractvalue { <4 x float>, i32 } %val, 0
@@ -12059,7 +12059,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
; GFX12-GISEL-NEXT: ; return to shader part epilog
i32 inreg %idx, <4 x i32> %s_idx, i32 %v_inp, { <4 x float>, float } %dummy) #1 {
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %idx)
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx)
%widx0 = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok, <4 x i32> %s_idx)
%val = call { <4 x float>, i32 } @llvm.amdgcn.struct.buffer.load.format.sl_v4f32i32s(<4 x i32> %widx0, i32 %v_inp, i32 0, i32 0, i32 0)
%payl = extractvalue { <4 x float>, i32 } %val, 0
@@ -12574,7 +12574,7 @@ define amdgpu_cs_chain void @wf_scc_check(i32 %arg, %struct.wobble %arg1, float
bb:
%call = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> zeroinitializer, i32 0, i32 0)
%icmp = icmp eq i32 %call, 0
- %call3 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %arg)
+ %call3 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %arg)
%call4 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %call3, i32 %arg)
%call5 = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(token %call3, <8 x i32> zeroinitializer)
call void @llvm.amdgcn.image.store.2d.f32.i32.v8i32(float 0.000000e+00, i32 0, i32 0, i32 0, <8 x i32> %call5, i32 0, i32 0)
diff --git a/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll b/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
index 2633388bae49d..5e96f4a444f5b 100644
--- a/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
+++ b/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
@@ -3,7 +3,7 @@
define amdgpu_ps <4 x float> @test_waterfall_same_index2(<8 x i32> addrspace(4)* inreg %in, i32 %index, float %s, <4 x i32> inreg %samp) {
; CHECK-LABEL: @test_waterfall_same_index2(
-; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 [[INDEX:%.*]])
+; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 [[INDEX:%.*]])
; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token [[WF_TOKEN1]], i32 [[INDEX]])
; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
; CHECK-NEXT: [[PTR:%.*]] = getelementptr [32 x i8], ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
@@ -12,7 +12,7 @@ define amdgpu_ps <4 x float> @test_waterfall_same_index2(<8 x i32> addrspace(4)*
; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token [[WF_TOKEN1]], <4 x float> [[R]])
; CHECK-NEXT: ret <4 x float> [[R1]]
;
- %wf_token1 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
+ %wf_token1 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
%wf_token2 = call token @llvm.amdgcn.waterfall.begin.i32(token %wf_token1, i32 %index)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token2, i32 %index)
%ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %s_idx
@@ -24,7 +24,7 @@ define amdgpu_ps <4 x float> @test_waterfall_same_index2(<8 x i32> addrspace(4)*
define amdgpu_ps <4 x float> @test_waterfall_same_index3(<8 x i32> addrspace(4)* inreg %in, i32 %index, float %s, <4 x i32> inreg %samp) {
; CHECK-LABEL: @test_waterfall_same_index3(
-; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 [[INDEX:%.*]])
+; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 [[INDEX:%.*]])
; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token [[WF_TOKEN1]], i32 [[INDEX]])
; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
; CHECK-NEXT: [[PTR:%.*]] = getelementptr [32 x i8], ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
@@ -33,7 +33,7 @@ define amdgpu_ps <4 x float> @test_waterfall_same_index3(<8 x i32> addrspace(4)*
; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token [[WF_TOKEN1]], <4 x float> [[R]])
; CHECK-NEXT: ret <4 x float> [[R1]]
;
- %wf_token1 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index)
+ %wf_token1 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
%wf_token2 = call token @llvm.amdgcn.waterfall.begin.i32(token %wf_token1, i32 %index)
%wf_token3 = call token @llvm.amdgcn.waterfall.begin.i32(token %wf_token2, i32 %index)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token3, i32 %index)
@@ -46,7 +46,7 @@ define amdgpu_ps <4 x float> @test_waterfall_same_index3(<8 x i32> addrspace(4)*
define amdgpu_ps <4 x float> @test_waterfall_same_index_aba(<8 x i32> addrspace(4)* inreg %in, i32 %index1, i32 %index2, float %s, <4 x i32> inreg %samp) {
; CHECK-LABEL: @test_waterfall_same_index_aba(
-; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 [[INDEX1:%.*]])
+; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 [[INDEX1:%.*]])
; CHECK-NEXT: [[WF_TOKEN2:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token [[WF_TOKEN1]], i32 [[INDEX2:%.*]])
; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token [[WF_TOKEN2]], i32 [[INDEX2]])
; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
@@ -56,7 +56,7 @@ define amdgpu_ps <4 x float> @test_waterfall_same_index_aba(<8 x i32> addrspace(
; CHECK-NEXT: [[R1:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token [[WF_TOKEN2]], <4 x float> [[R]])
; CHECK-NEXT: ret <4 x float> [[R1]]
;
- %wf_token1 = call token @llvm.amdgcn.waterfall.begin.i32(token none, i32 %index1)
+ %wf_token1 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index1)
%wf_token2 = call token @llvm.amdgcn.waterfall.begin.i32(token %wf_token1, i32 %index2)
%wf_token3 = call token @llvm.amdgcn.waterfall.begin.i32(token %wf_token2, i32 %index1)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %wf_token3, i32 %index2)
>From 035413a5128b4250c1c5abd1ec94d1ccd4bf645e Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Wed, 27 May 2026 15:53:40 +0100
Subject: [PATCH 17/35] Remove waterfall last_use, last_use_vgpr, and loop_end
intrinsics
Split the waterfall intrinsics into two parts for easier review.
This commit removes the last_use, last_use_vgpr, and loop_end
intrinsics, keeping only the core begin/readfirstlane/end triple.
Assisted-by: Claude <noreply at anthropic.com>
---
llvm/docs/AMDGPUUsage.rst | 23 +-
llvm/include/llvm/IR/IntrinsicsAMDGPU.td | 31 +-
.../Target/AMDGPU/AMDGPUAtomicOptimizer.cpp | 14 -
.../Target/AMDGPU/AMDGPUCodeGenPrepare.cpp | 15 -
llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp | 13 -
llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.h | 1 -
.../Target/AMDGPU/AMDGPUInsertWaterfall.cpp | 112 +-
.../AMDGPU/AMDGPUInstructionSelector.cpp | 19 -
.../Target/AMDGPU/AMDGPUInstructionSelector.h | 1 -
.../lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp | 6 -
.../Target/AMDGPU/AMDGPURegisterBankInfo.cpp | 23 -
.../Target/AMDGPU/AMDGPUSearchableTables.td | 12 -
llvm/lib/Target/AMDGPU/SIInstructions.td | 20 -
.../CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll | 1980 ++++-------------
14 files changed, 400 insertions(+), 1870 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index da297bb0b2a76..c526e62bcdfd1 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -2034,14 +2034,13 @@ non-uniform operands that need to be made uniform, and the operands
to use as the "index" of the loop.
A waterfall group must contain at least one
-``waterfall.begin``, at least one of ``waterfall.readfirstlane``, and at least one
-of ``waterfall.end`` or ``waterfall.last.use`` intrinsic.
+``waterfall.begin``, at least one ``waterfall.readfirstlane``, and at least one
+``waterfall.end`` intrinsic.
A group can contain more than one of each of the waterfall intrinsics.
The token on the final
``waterfall.begin`` must be used for other waterfall intrinsics in the same
group. The waterfall loop will enclose all instructions from the earliest
-``waterfall.begin`` to the latest ``waterfall.end`` or ``waterfall.last.use``
-intrinsic in the group.
+``waterfall.begin`` to the latest ``waterfall.end`` intrinsic in the group.
Each ``waterfall.begin`` specifies a value that is part of the index. Each
``waterfall.readfirstlane`` specifies a non-uniform operand that needs to be
@@ -2077,8 +2076,8 @@ If all operands are uniform, the compiler will not insert the waterfall loop.
waterfall group) and an operand.
The intrinsic returns a new token that must be threaded through the
- corresponding ``waterfall.readfirstlane``, ``waterfall.end`` or
- ``waterfall.last.use`` intrinsics.
+ corresponding ``waterfall.readfirstlane`` and ``waterfall.end``
+ intrinsics.
``llvm.amdgcn.waterfall.readfirstlane`` Reads the first active lane's value of a given non-uniform operand and
returns it as a uniform value for use within a waterfall region.
@@ -2092,18 +2091,6 @@ If all operands are uniform, the compiler will not insert the waterfall loop.
``llvm.amdgcn.waterfall.end`` Marks the end of a waterfall region.
Takes the token from the final ``waterfall.begin`` in the group.
- ``llvm.amdgcn.waterfall.last.use`` Variant of ``waterfall.end`` for values whose last use is in a
- non-defining operation such as a store. Marks that the use of the value
- constitutes the end of the waterfall region.
-
- ``llvm.amdgcn.waterfall.last.use.vgpr`` Variant of ``waterfall.last.use`` for values that remain in a VGPR.
-
- ``llvm.amdgcn.waterfall.loop.end`` Inserted later by the compiler to be used with ``waterfall.last.use``
- to mark the loop-end point for special
- handling such as SCC clobber tracking.
- This intrinsic is not intended to be generated by a frontend.
-
-
============================================== =========================================================================
'``llvm.amdgcn.cooperative.atomic``' Intrinsics
diff --git a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
index 18b5cfe857f05..94d810b466bc2 100644
--- a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
+++ b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
@@ -2726,9 +2726,8 @@ def int_amdgcn_cs_chain:
// Best case (all lanes have the same values) the loop will execute once, worst case (all lanes
// have different values) the loop will execute wave-size times.
// The waterfall.begin intrinsic returns a new token that must be threaded through the
-// corresponding waterfall.readfirstlane and waterfall.end or
-// waterfall.last.use intrinsics, forming a waterfall group of intrinsics that together
-// define a waterfall region.
+// corresponding waterfall.readfirstlane and waterfall.end intrinsics, forming a
+// waterfall group of intrinsics that together define a waterfall region.
// All intrinsics in a waterfall group must reside within the same basic block.
// Where the intrinsic specifies "llvm_any_ty", the intrinsic will accept any of
// the following types:
@@ -2761,32 +2760,6 @@ def int_amdgcn_waterfall_end :
LLVMMatchType<0>], // Same as return value
[IntrConvergent]>;
-// Effectively a no-op, no code results directly from this intrinsic. This is
-// used to indicate that the use of the defined value constitutes the end of the
-// waterfall propagation. This is roughly equivalent to the end intrinsic, but
-// can be used to tag values that are used in non-defining operations such as a
-// store.
-def int_amdgcn_waterfall_last_use :
- Intrinsic<[llvm_any_ty], // Returns any value
- [llvm_token_ty, // Token from begin
- LLVMMatchType<0>], // Same as return value
- [IntrConvergent]>;
-
-// Special variant where the last-use uses a VGPR that needs to be uniform.
-def int_amdgcn_waterfall_last_use_vgpr :
- Intrinsic<[llvm_any_ty], // Returns any value
- [llvm_token_ty, // Token from begin
- LLVMMatchType<0>], // Same as return value
- []>;
-
-// Special waterfall loop end intrinsic to be used with last.use
-// This is inserted late, before codegen, mainly to mark the loop end
-// for special handling such as scc clobbers.
-def int_amdgcn_waterfall_loop_end :
- Intrinsic<[],
- [llvm_token_ty], // Token from begin
- [IntrConvergent]>;
-
// Run a function with all the lanes enabled. Only direct calls are allowed. The
// first argument is the callee, which must have the `amdgpu_gfx_whole_wave`
// calling convention and must not be variadic. The remaining arguments to the
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
index a877c028e6456..fc746fa2c9812 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAtomicOptimizer.cpp
@@ -218,20 +218,6 @@ static Instruction *findLastInWaterfall(Instruction &Begin) {
if (Last->comesBefore(Intrin))
Last = Intrin;
break;
- case Intrinsic::amdgcn_waterfall_last_use:
- case Intrinsic::amdgcn_waterfall_last_use_vgpr: {
- // Find the actual last use
- for (auto &LastUse : Intrin->uses()) {
- auto *LUI = static_cast<Instruction *>(LastUse.getUser());
- if (LUI->getParent() != Intrin->getParent()) {
- // LUI has to be in same BB as waterfall intrinsics.
- return nullptr;
- }
- if (Last->comesBefore(LUI))
- Last = LUI;
- }
- break;
- }
case Intrinsic::amdgcn_waterfall_readfirstlane:
// Always in the middle of a group - so doesn't have any effect on group
// detection
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp b/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp
index 383081f914068..ba58f3ed57e38 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp
@@ -264,7 +264,6 @@ class AMDGPUCodeGenPrepareImpl
bool visitAddrSpaceCastInst(AddrSpaceCastInst &I);
bool visitIntrinsicInst(IntrinsicInst &I);
- bool visitWaterfallLastUseIntrinsic(IntrinsicInst &I);
bool visitFMinLike(IntrinsicInst &I);
bool visitSqrt(IntrinsicInst &I);
bool visitLog(FPMathOperator &Log, Intrinsic::ID IID);
@@ -2090,8 +2089,6 @@ bool AMDGPUCodeGenPrepareImpl::visitIntrinsicInst(IntrinsicInst &I) {
case Intrinsic::amdgcn_waterfall_end:
CurrentWaterfall = &I;
return false;
- case Intrinsic::amdgcn_waterfall_last_use:
- return visitWaterfallLastUseIntrinsic(I);
default:
return false;
}
@@ -2126,18 +2123,6 @@ bool AMDGPUCodeGenPrepareImpl::visitGetElementPtrInst(GetElementPtrInst &I) {
return true;
}
-bool AMDGPUCodeGenPrepareImpl::visitWaterfallLastUseIntrinsic(
- IntrinsicInst &I) {
- auto *Token = I.getOperand(0);
- for (auto *U : I.users()) {
- auto *UI = cast<Instruction>(U);
- BasicBlock::iterator InsertPt = std::next(UI->getIterator());
- IRBuilder<> Builder(I.getParent(), InsertPt);
- Builder.CreateIntrinsic(Intrinsic::amdgcn_waterfall_loop_end, {}, {Token});
- }
- return true;
-}
-
/// Match the core sequence in the fract pattern (x - floor(x), which doesn't
/// need to consider edge case handling.
Value *AMDGPUCodeGenPrepareImpl::matchFractPatImpl(Value &FractSrc,
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp b/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp
index 1158330350247..ba70b0ba46ea1 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp
@@ -3260,14 +3260,6 @@ void AMDGPUDAGToDAGISel::SelectWaterfallIntrinsic(SDNode *N, unsigned IntrID) {
CurDAG->SelectNodeTo(N, Opc, VTs, {Tok, Val, Chain});
}
-void AMDGPUDAGToDAGISel::SelectWaterfallIntrinsicLoopEnd(SDNode *N) {
- // op0=chain, op1=intrinsicID, op2=token(Untyped)
- SDValue Chain = N->getOperand(0);
- SDValue Tok = N->getOperand(2);
- CurDAG->SelectNodeTo(N, AMDGPU::SI_WATERFALL_LOOP_END,
- CurDAG->getVTList(MVT::Other), {Tok, Chain});
-}
-
void AMDGPUDAGToDAGISel::SelectINTRINSIC_W_CHAIN(SDNode *N) {
unsigned IntrID = N->getConstantOperandVal(1);
switch (IntrID) {
@@ -3292,8 +3284,6 @@ void AMDGPUDAGToDAGISel::SelectINTRINSIC_W_CHAIN(SDNode *N) {
case Intrinsic::amdgcn_waterfall_begin:
case Intrinsic::amdgcn_waterfall_readfirstlane:
case Intrinsic::amdgcn_waterfall_end:
- case Intrinsic::amdgcn_waterfall_last_use:
- case Intrinsic::amdgcn_waterfall_last_use_vgpr:
SelectWaterfallIntrinsic(N, IntrID);
return;
}
@@ -3389,9 +3379,6 @@ void AMDGPUDAGToDAGISel::SelectINTRINSIC_VOID(SDNode *N) {
case Intrinsic::amdgcn_tensor_store_from_lds:
SelectTensorLoadStore(N, IntrID);
return;
- case Intrinsic::amdgcn_waterfall_loop_end:
- SelectWaterfallIntrinsicLoopEnd(N);
- return;
default:
break;
}
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.h b/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.h
index 840c45421de4c..f2bb7deec183e 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.h
@@ -303,7 +303,6 @@ class AMDGPUDAGToDAGISel : public SelectionDAGISel {
void SelectTensorLoadStore(SDNode *N, unsigned IntrID);
void SelectDS_GWS(SDNode *N, unsigned IntrID);
void SelectWaterfallIntrinsic(SDNode *N, unsigned IntrID);
- void SelectWaterfallIntrinsicLoopEnd(SDNode *N);
void SelectInterpP1F16(SDNode *N);
void SelectINTRINSIC_W_CHAIN(SDNode *N);
void SelectINTRINSIC_WO_CHAIN(SDNode *N);
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
index cd25c3b361944..11b2d38067528 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
@@ -77,45 +77,6 @@ static unsigned getWFEndSize(const unsigned Opcode) {
return 0; // Not SI_WATERFALL_END_*
}
-static unsigned getWFLastUseSize(const unsigned Opcode) {
- switch (Opcode) {
- case AMDGPU::SI_WATERFALL_LAST_USE_V1:
- case AMDGPU::SI_WATERFALL_LAST_USE_V1_V:
- return 1;
- case AMDGPU::SI_WATERFALL_LAST_USE_V2:
- case AMDGPU::SI_WATERFALL_LAST_USE_V2_V:
- return 2;
- case AMDGPU::SI_WATERFALL_LAST_USE_V4:
- case AMDGPU::SI_WATERFALL_LAST_USE_V4_V:
- return 4;
- case AMDGPU::SI_WATERFALL_LAST_USE_V8:
- case AMDGPU::SI_WATERFALL_LAST_USE_V8_V:
- return 8;
- default:
- break;
- }
-
- return 0; // Not SI_WATERFALL_LAST_USE_*
-}
-
-static bool isWFLastUseVGPR(const unsigned Opcode) {
- switch (Opcode) {
- case AMDGPU::SI_WATERFALL_LAST_USE_V1_V:
- case AMDGPU::SI_WATERFALL_LAST_USE_V2_V:
- case AMDGPU::SI_WATERFALL_LAST_USE_V4_V:
- case AMDGPU::SI_WATERFALL_LAST_USE_V8_V:
- return true;
- default:
- break;
- }
-
- return false;
-}
-
-static bool isWFLoopEnd(const unsigned Opcode) {
- return Opcode == AMDGPU::SI_WATERFALL_LOOP_END;
-}
-
static void readFirstLaneReg(MachineBasicBlock &MBB, MachineRegisterInfo *MRI,
const SIRegisterInfo *RI, const SIInstrInfo *TII,
MachineBasicBlock::iterator &I, const DebugLoc &DL,
@@ -278,13 +239,10 @@ class AMDGPUInsertWaterfall : public MachineFunctionPass {
const MachineRegisterInfo *MRI;
Register TokReg; // This is always the token from the last begin intrinsic
MachineInstr *Final;
- bool hasVGPRLastUse;
std::vector<MachineInstr *> BeginList;
std::vector<MachineInstr *> RFLList;
std::vector<MachineInstr *> EndList;
- std::vector<MachineInstr *> LastUseList;
- std::vector<MachineInstr *> LoopEndList;
// List of corresponding init, newdst and phi registers used in loop for
// end pseudos
@@ -294,7 +252,7 @@ class AMDGPUInsertWaterfall : public MachineFunctionPass {
WaterfallWorkitem() = default;
WaterfallWorkitem(MachineInstr *_Begin, const SIInstrInfo *_TII,
MachineRegisterInfo *_MRI)
- : TII(_TII), MRI(_MRI), Final(nullptr), hasVGPRLastUse(false) {
+ : TII(_TII), MRI(_MRI), Final(nullptr) {
auto TokMO = TII->getNamedOperand(*_Begin, AMDGPU::OpName::tok_ret);
@@ -312,24 +270,9 @@ class AMDGPUInsertWaterfall : public MachineFunctionPass {
void processCandidate(MachineInstr *Cand) {
unsigned Opcode = Cand->getOpcode();
- // Trivially end any waterfall intrinsic instructions
if (getWFBeginSize(Opcode) || getWFRFLSize(Opcode) ||
- getWFEndSize(Opcode) || getWFLastUseSize(Opcode) ||
- isWFLoopEnd(Opcode)) {
- // TODO: A new waterfall clause shouldn't overlap with any uses
- // tagged by a last_use intrinsic
+ getWFEndSize(Opcode))
return;
- }
-
- // Iterate over the LastUseList to determine if this instruction has a
- // later use of a tagged last_use
- for (auto Use : LastUseList) {
- auto UseMO = TII->getNamedOperand(*Use, AMDGPU::OpName::dst);
- Register UseReg = UseMO->getReg();
-
- if (Cand->findRegisterUseOperand(UseReg, /*TRI=*/nullptr))
- Final = Cand;
- }
}
MachineInstr *getDefInstr(const MachineOperand *MO) const {
@@ -362,8 +305,7 @@ class AMDGPUInsertWaterfall : public MachineFunctionPass {
unsigned Opcode = Cand->getOpcode();
assert((getWFBeginSize(Opcode) || getWFRFLSize(Opcode) ||
- getWFEndSize(Opcode) || getWFLastUseSize(Opcode) ||
- isWFLoopEnd(Opcode)) &&
+ getWFEndSize(Opcode)) &&
"expected a waterfall instruction in addCandidate");
auto CandTokMO = TII->getNamedOperand(*Cand, AMDGPU::OpName::tok);
@@ -401,14 +343,6 @@ class AMDGPUInsertWaterfall : public MachineFunctionPass {
EndList.push_back(Cand);
Final = Cand;
return true;
- } else if (getWFLastUseSize(Opcode)) {
- LastUseList.push_back(Cand);
- if (isWFLastUseVGPR(Opcode))
- hasVGPRLastUse = true;
- return true;
- } else if (isWFLoopEnd(Opcode)) {
- LoopEndList.push_back(Cand);
- return true;
} else {
report_fatal_error("Unknown opcode, expected waterfall intrinsic");
}
@@ -424,10 +358,6 @@ class AMDGPUInsertWaterfall : public MachineFunctionPass {
RFLMI->eraseFromParent();
for (auto EndMI : EndList)
EndMI->eraseFromParent();
- for (auto LUMI : LastUseList)
- LUMI->eraseFromParent();
- for (auto LEMI : LoopEndList)
- LEMI->eraseFromParent();
}
};
@@ -523,20 +453,16 @@ bool AMDGPUInsertWaterfall::removeRedundantWaterfall(WaterfallWorkitem &Item) {
// Note: this test also returns true when there are NO RFL intrinsics, the
// case where a prior pass has removed all of them and the loop is now
// redundant
- // Also check for the special case where there are no RFL intrinsics, but
- // there are some last.use with VGPR uses.
- if (!Item.BeginList.size() ||
- (Removed == Item.RFLList.size() && !Item.hasVGPRLastUse)) {
+ if (!Item.BeginList.size() || Removed == Item.RFLList.size()) {
// Removed all of the RFLs
// We can remove the waterfall loop entirely
// Protocol is to replace all dst operands for the waterfall_end intrinsics
- // with their src operands. Replace all last_use dst operands with their src
- // operands We don't need to check that the loop index isn't used anywhere
- // as the protocol for waterfall intrinsics is to only use the begin index
- // via a readfirstlane intrinsic anyway (which should also be removed) Any
- // problems due to errors in this pass around loop removal will be picked up
- // later by e.g. use before def errors
+ // with their src operands. We don't need to check that the loop index isn't
+ // used anywhere as the protocol for waterfall intrinsics is to only use the
+ // begin index via a readfirstlane intrinsic anyway (which should also be
+ // removed). Any problems due to errors in this pass around loop removal
+ // will be picked up later by e.g. use before def errors
LLVM_DEBUG(
dbgs()
<< "detected case for waterfall loop removal - already all uniform\n");
@@ -545,11 +471,6 @@ bool AMDGPUInsertWaterfall::removeRedundantWaterfall(WaterfallWorkitem &Item) {
auto EndSrcOp = TII->getNamedOperand(*EndMI, AMDGPU::OpName::src);
replaceRegIncSubReg(MRI, RI, EndDstOp, EndSrcOp);
}
- for (auto LUMI : Item.LastUseList) {
- auto LUDstOp = TII->getNamedOperand(*LUMI, AMDGPU::OpName::dst);
- auto LUSrcOp = TII->getNamedOperand(*LUMI, AMDGPU::OpName::src);
- replaceRegIncSubReg(MRI, RI, LUDstOp, LUSrcOp);
- }
// If all the begins were removed, we have to replace the RFL with actual
// RFL, these will show up in the NewRLFList
for (auto RFLMI : NewRFLList) {
@@ -611,8 +532,7 @@ bool AMDGPUInsertWaterfall::processWaterfall(MachineBasicBlock &MBB) {
continue;
}
- assert((Item.RFLList.size() || Item.hasVGPRLastUse) &&
- (Item.EndList.size() || Item.LastUseList.size()) &&
+ assert(Item.RFLList.size() && Item.EndList.size() &&
"SI_WATERFALL* pseudo instruction group must have at least 1 of "
"each type");
@@ -663,12 +583,6 @@ bool AMDGPUInsertWaterfall::processWaterfall(MachineBasicBlock &MBB) {
Item.EndRegs.emplace_back(
TII->getNamedOperand(*EndMI, AMDGPU::OpName::dst),
TII->getNamedOperand(*EndMI, AMDGPU::OpName::src));
- for (auto LUMI : Item.LastUseList) {
- auto LUSrc = TII->getNamedOperand(*LUMI, AMDGPU::OpName::src);
- auto LUDst = TII->getNamedOperand(*LUMI, AMDGPU::OpName::dst);
- replaceRegIncSubReg(MRI, RI, LUDst, LUSrc);
- }
-
// EXEC mask handling
Register Exec = ST->isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
unsigned SaveExecOpc = ST->isWave32() ? AMDGPU::S_AND_SAVEEXEC_B32
@@ -721,8 +635,7 @@ bool AMDGPUInsertWaterfall::processWaterfall(MachineBasicBlock &MBB) {
BuildMI(LoopHeaderBB, LH, DL, TII->get(MovOpc), SaveExec).addReg(Exec);
// Move all instructions from the SI_WATERFALL_BEGIN to the last
- // SI_WATERFALL_END or last use tagged from SI_WATERFALL_LAST_USE
- // into the new LoopBB
+ // SI_WATERFALL_END into the new LoopBB
MachineBasicBlock::iterator SpliceE(Item.Final);
++SpliceE;
LoopBB.splice(LoopBB.begin(), CurrMBB, I, SpliceE);
@@ -872,8 +785,7 @@ bool AMDGPUInsertWaterfall::runOnMachineFunction(MachineFunction &MF) {
llvm_unreachable("Incorrect SI_WATERFALL_* groups");
}
}
- } else if (getWFRFLSize(Opcode) || getWFEndSize(Opcode) ||
- getWFLastUseSize(Opcode) || isWFLoopEnd(Opcode)) {
+ } else if (getWFRFLSize(Opcode) || getWFEndSize(Opcode)) {
// On to the body of the group intrinsics,
// Tag StartNew as true if we encounter another begin
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
index 7a085d4aa096e..30d90de280bd8 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
@@ -2523,11 +2523,7 @@ bool AMDGPUInstructionSelector::selectG_INTRINSIC_W_SIDE_EFFECTS(
case Intrinsic::amdgcn_waterfall_begin:
case Intrinsic::amdgcn_waterfall_readfirstlane:
case Intrinsic::amdgcn_waterfall_end:
- case Intrinsic::amdgcn_waterfall_last_use:
- case Intrinsic::amdgcn_waterfall_last_use_vgpr:
return selectWaterfallIntrinsic(I, IntrinsicID);
- case Intrinsic::amdgcn_waterfall_loop_end:
- return selectWaterfallIntrinsicLoopEnd(I);
}
return selectImpl(I, *CoverageInfo);
}
@@ -4474,21 +4470,6 @@ bool AMDGPUInstructionSelector::selectWaterfallIntrinsic(
return true;
}
-bool AMDGPUInstructionSelector::selectWaterfallIntrinsicLoopEnd(
- MachineInstr &I) const {
- // op0=intrinsicID, op1=token
- MachineBasicBlock *MBB = I.getParent();
- const DebugLoc &DL = I.getDebugLoc();
- Register TokReg = I.getOperand(1).getReg();
-
- MachineInstrBuilder MIB =
- BuildMI(*MBB, &I, DL, TII.get(AMDGPU::SI_WATERFALL_LOOP_END))
- .addReg(TokReg);
- I.eraseFromParent();
- constrainSelectedInstRegOperands(*MIB, TII, TRI, RBI);
- return true;
-}
-
bool AMDGPUInstructionSelector::select(MachineInstr &I) {
if (!I.isPreISelOpcode()) {
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h
index 6184755060f06..df8cea5171461 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.h
@@ -158,7 +158,6 @@ class AMDGPUInstructionSelector final : public InstructionSelector {
bool selectSBarrierSignalIsfirst(MachineInstr &I, Intrinsic::ID IID) const;
bool selectSGetBarrierState(MachineInstr &I, Intrinsic::ID IID) const;
bool selectWaterfallIntrinsic(MachineInstr &I, Intrinsic::ID IID) const;
- bool selectWaterfallIntrinsicLoopEnd(MachineInstr &I) const;
bool selectSBarrierLeave(MachineInstr &I) const;
bool selectWaveShuffleIntrin(MachineInstr &I) const;
diff --git a/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp b/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
index 0b4c3c62f4e84..9ef9eebe71195 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
@@ -8572,15 +8572,9 @@ bool AMDGPULegalizerInfo::legalizeIntrinsic(LegalizerHelper &Helper,
return true;
case Intrinsic::amdgcn_waterfall_readfirstlane:
case Intrinsic::amdgcn_waterfall_end:
- case Intrinsic::amdgcn_waterfall_last_use:
- case Intrinsic::amdgcn_waterfall_last_use_vgpr:
// op0=value_result, op1=intrinsicID, op2=token_input, op3=value
legalizeWaterfallTokenReg(MRI, MI.getOperand(2).getReg());
return true;
- case Intrinsic::amdgcn_waterfall_loop_end:
- // op0=intrinsicID, op1=token_input
- legalizeWaterfallTokenReg(MRI, MI.getOperand(1).getReg());
- return true;
default: {
if (const AMDGPU::ImageDimIntrinsicInfo *ImageDimIntr =
diff --git a/llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp b/llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp
index e680e658835bb..41e2f0238e332 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp
@@ -5547,29 +5547,6 @@ AMDGPURegisterBankInfo::getInstrMapping(const MachineInstr &MI) const {
OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, SizeSrc2);
break;
}
- case Intrinsic::amdgcn_waterfall_last_use: {
- unsigned SizeDst = getSizeInBits(MI.getOperand(0).getReg(), MRI, *TRI);
- unsigned SizeSrc1 = getSizeInBits(MI.getOperand(2).getReg(), MRI, *TRI);
- unsigned SizeSrc2 = getSizeInBits(MI.getOperand(3).getReg(), MRI, *TRI);
- OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, SizeDst);
- OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, SizeSrc1);
- OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, SizeSrc2);
- break;
- }
- case Intrinsic::amdgcn_waterfall_last_use_vgpr: {
- unsigned SizeDst = getSizeInBits(MI.getOperand(0).getReg(), MRI, *TRI);
- unsigned SizeSrc1 = getSizeInBits(MI.getOperand(2).getReg(), MRI, *TRI);
- unsigned SizeSrc2 = getSizeInBits(MI.getOperand(3).getReg(), MRI, *TRI);
- OpdsMapping[0] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, SizeDst);
- OpdsMapping[2] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, SizeSrc1);
- OpdsMapping[3] = AMDGPU::getValueMapping(AMDGPU::VGPRRegBankID, SizeSrc2);
- break;
- }
- case Intrinsic::amdgcn_waterfall_loop_end: {
- unsigned SizeSrc1 = getSizeInBits(MI.getOperand(1).getReg(), MRI, *TRI);
- OpdsMapping[1] = AMDGPU::getValueMapping(AMDGPU::SGPRRegBankID, SizeSrc1);
- break;
- }
case Intrinsic::amdgcn_ds_gws_init:
case Intrinsic::amdgcn_ds_gws_barrier:
case Intrinsic::amdgcn_ds_gws_sema_br: {
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUSearchableTables.td b/llvm/lib/Target/AMDGPU/AMDGPUSearchableTables.td
index 02178b6522c74..71c34b7adc6b7 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUSearchableTables.td
+++ b/llvm/lib/Target/AMDGPU/AMDGPUSearchableTables.td
@@ -405,7 +405,6 @@ def : AlwaysUniform<int_amdgcn_ballot>;
def : AlwaysUniform<int_amdgcn_if_break>;
def : AlwaysUniform<int_amdgcn_waterfall_readfirstlane>;
def : AlwaysUniform<int_amdgcn_waterfall_begin>;
-def : AlwaysUniform<int_amdgcn_waterfall_last_use>;
def : AlwaysUniform<int_amdgcn_cluster_workgroup_id_x>;
def : AlwaysUniform<int_amdgcn_cluster_workgroup_id_y>;
def : AlwaysUniform<int_amdgcn_cluster_workgroup_id_z>;
@@ -468,14 +467,3 @@ def : WaterfallPseudoInfo<int_amdgcn_waterfall_end, 2, SI_WATERFALL_END_V2>;
def : WaterfallPseudoInfo<int_amdgcn_waterfall_end, 4, SI_WATERFALL_END_V4>;
def : WaterfallPseudoInfo<int_amdgcn_waterfall_end, 8, SI_WATERFALL_END_V8>;
-def : WaterfallPseudoInfo<int_amdgcn_waterfall_last_use, 1, SI_WATERFALL_LAST_USE_V1>;
-def : WaterfallPseudoInfo<int_amdgcn_waterfall_last_use, 2, SI_WATERFALL_LAST_USE_V2>;
-def : WaterfallPseudoInfo<int_amdgcn_waterfall_last_use, 4, SI_WATERFALL_LAST_USE_V4>;
-def : WaterfallPseudoInfo<int_amdgcn_waterfall_last_use, 8, SI_WATERFALL_LAST_USE_V8>;
-
-def : WaterfallPseudoInfo<int_amdgcn_waterfall_last_use_vgpr, 1, SI_WATERFALL_LAST_USE_V1_V>;
-def : WaterfallPseudoInfo<int_amdgcn_waterfall_last_use_vgpr, 2, SI_WATERFALL_LAST_USE_V2_V>;
-def : WaterfallPseudoInfo<int_amdgcn_waterfall_last_use_vgpr, 4, SI_WATERFALL_LAST_USE_V4_V>;
-def : WaterfallPseudoInfo<int_amdgcn_waterfall_last_use_vgpr, 8, SI_WATERFALL_LAST_USE_V8_V>;
-
-def : WaterfallPseudoInfo<int_amdgcn_waterfall_loop_end, 0, SI_WATERFALL_LOOP_END>;
diff --git a/llvm/lib/Target/AMDGPU/SIInstructions.td b/llvm/lib/Target/AMDGPU/SIInstructions.td
index 5ef5e2c31f98a..57a91142026f5 100644
--- a/llvm/lib/Target/AMDGPU/SIInstructions.td
+++ b/llvm/lib/Target/AMDGPU/SIInstructions.td
@@ -1421,10 +1421,6 @@ class SI_WATERFALL_END<RegisterClass rc> : SPseudoInstSI <
(outs rc:$dst),
(ins WaterfallTokenType:$tok, rc:$src)>;
-class SI_WATERFALL_LAST_USE<RegisterClass rc> : SPseudoInstSI <
- (outs rc:$dst),
- (ins WaterfallTokenType:$tok, rc:$src)>;
-
} // End UseNamedOperandTable = 1
// TODO: Do we still need the size variants (_V1, _V2, etc.) when selection
@@ -1444,22 +1440,6 @@ def SI_WATERFALL_END_V2 : SI_WATERFALL_END<VReg_64>;
def SI_WATERFALL_END_V4 : SI_WATERFALL_END<VReg_128>;
def SI_WATERFALL_END_V8 : SI_WATERFALL_END<VReg_256>;
-def SI_WATERFALL_LAST_USE_V1 : SI_WATERFALL_LAST_USE<SGPR_32>;
-def SI_WATERFALL_LAST_USE_V1_V : SI_WATERFALL_LAST_USE<VGPR_32>;
-def SI_WATERFALL_LAST_USE_V2 : SI_WATERFALL_LAST_USE<SReg_64>;
-def SI_WATERFALL_LAST_USE_V2_V : SI_WATERFALL_LAST_USE<VReg_64>;
-def SI_WATERFALL_LAST_USE_V4 : SI_WATERFALL_LAST_USE<SGPR_128>;
-def SI_WATERFALL_LAST_USE_V4_V : SI_WATERFALL_LAST_USE<SReg_128>;
-def SI_WATERFALL_LAST_USE_V8 : SI_WATERFALL_LAST_USE<SReg_256>;
-def SI_WATERFALL_LAST_USE_V8_V : SI_WATERFALL_LAST_USE<VReg_256>;
-
-let Defs = [SCC],
- UseNamedOperandTable = 1 in
-def SI_WATERFALL_LOOP_END : SPseudoInstSI <
- (outs),
- (ins WaterfallTokenType:$tok)
->;
-
//===----------------------------------------------------------------------===//
// VOP1 Patterns
//===----------------------------------------------------------------------===//
diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
index b0d578f529e66..80b5f35f0c7ad 100644
--- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
+++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
@@ -3,10 +3,10 @@
; RUN: llc -global-isel=1 -march=amdgcn -mcpu=fiji -verify-machineinstrs < %s | FileCheck -check-prefixes=PRE-GFX10,VI,VI-GISEL %s
; RUN: llc -global-isel=0 -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=PRE-GFX10,GFX9,GFX9-SDAG %s
; RUN: llc -global-isel=1 -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=PRE-GFX10,GFX9,GFX9-GISEL %s
-; RUN: llc -global-isel=0 -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX10GFX11-SDAG,GFX10,GFX10-32,GFX10-32-SDAG %s
-; RUN: llc -global-isel=1 -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX10,GFX10-32,GFX10-32-GISEL %s
-; RUN: llc -global-isel=0 -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX10GFX11-SDAG,GFX10,GFX10-64,GFX10-64-SDAG %s
-; RUN: llc -global-isel=1 -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX10,GFX10-64,GFX10-64-GISEL %s
+; RUN: llc -global-isel=0 -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX10GFX11-SDAG,GFX10-32,GFX10-32-SDAG %s
+; RUN: llc -global-isel=1 -march=amdgcn -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX10-32,GFX10-32-GISEL %s
+; RUN: llc -global-isel=0 -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX10GFX11-SDAG,GFX10-64,GFX10-64-SDAG %s
+; RUN: llc -global-isel=1 -march=amdgcn -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX10-64,GFX10-64-GISEL %s
; RUN: llc -global-isel=0 -march=amdgcn -mcpu=gfx1150 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX10GFX11-SDAG,GFX1150,GFX1150-SDAG %s
; RUN: llc -global-isel=1 -march=amdgcn -mcpu=gfx1150 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX1150,GFX1150-GISEL %s
; RUN: llc -global-isel=0 -march=amdgcn -mcpu=gfx1200 -mattr=-wavefrontsize32,+wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefixes=GFX12,GFX12-SDAG %s
@@ -2759,499 +2759,6 @@ define amdgpu_ps <4 x float> @test_waterfall_non_uni_img_2_idx(<8 x i32> addrspa
ret <4 x float> %r1
}
-define amdgpu_ps void @test_waterfall_non_uniform_img_single_store(<8 x i32> addrspace(4)* inreg %in, i32 %index, i32 %s, <4 x float> %data) #1 {
-; VI-SDAG-LABEL: test_waterfall_non_uniform_img_single_store:
-; VI-SDAG: ; %bb.0:
-; VI-SDAG-NEXT: v_mov_b32_e32 v6, v1
-; VI-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
-; VI-SDAG-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
-; VI-SDAG-NEXT: v_mov_b32_e32 v1, s1
-; VI-SDAG-NEXT: v_add_u32_e32 v7, vcc, s0, v7
-; VI-SDAG-NEXT: v_addc_u32_e32 v8, vcc, v1, v8, vcc
-; VI-SDAG-NEXT: v_add_u32_e32 v11, vcc, 16, v7
-; VI-SDAG-NEXT: v_addc_u32_e32 v12, vcc, 0, v8, vcc
-; VI-SDAG-NEXT: flat_load_dwordx4 v[7:10], v[7:8]
-; VI-SDAG-NEXT: flat_load_dwordx4 v[11:14], v[11:12]
-; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
-; VI-SDAG-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
-; VI-SDAG-NEXT: v_readfirstlane_b32 s0, v0
-; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[0:1], s0, v0
-; VI-SDAG-NEXT: s_and_saveexec_b64 s[8:9], s[0:1]
-; VI-SDAG-NEXT: v_readfirstlane_b32 s0, v7
-; VI-SDAG-NEXT: v_readfirstlane_b32 s1, v8
-; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v9
-; VI-SDAG-NEXT: v_readfirstlane_b32 s3, v10
-; VI-SDAG-NEXT: v_readfirstlane_b32 s4, v11
-; VI-SDAG-NEXT: v_readfirstlane_b32 s5, v12
-; VI-SDAG-NEXT: v_readfirstlane_b32 s6, v13
-; VI-SDAG-NEXT: v_readfirstlane_b32 s7, v14
-; VI-SDAG-NEXT: ; implicit-def: $vgpr0
-; VI-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
-; VI-SDAG-NEXT: s_nop 4
-; VI-SDAG-NEXT: image_store v[2:5], v6, s[0:7] dmask:0xf unorm
-; VI-SDAG-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
-; VI-SDAG-NEXT: ; implicit-def: $vgpr6
-; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[8:9]
-; VI-SDAG-NEXT: s_cbranch_execnz .LBB6_1
-; VI-SDAG-NEXT: ; %bb.2:
-; VI-SDAG-NEXT: s_endpgm
-;
-; VI-GISEL-LABEL: test_waterfall_non_uniform_img_single_store:
-; VI-GISEL: ; %bb.0:
-; VI-GISEL-NEXT: v_mov_b32_e32 v6, v1
-; VI-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
-; VI-GISEL-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
-; VI-GISEL-NEXT: v_mov_b32_e32 v10, s1
-; VI-GISEL-NEXT: v_mov_b32_e32 v9, s0
-; VI-GISEL-NEXT: v_add_u32_e32 v7, vcc, v9, v7
-; VI-GISEL-NEXT: v_addc_u32_e32 v8, vcc, v10, v8, vcc
-; VI-GISEL-NEXT: v_add_u32_e32 v11, vcc, 16, v7
-; VI-GISEL-NEXT: v_addc_u32_e32 v12, vcc, 0, v8, vcc
-; VI-GISEL-NEXT: flat_load_dwordx4 v[7:10], v[7:8]
-; VI-GISEL-NEXT: flat_load_dwordx4 v[11:14], v[11:12]
-; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
-; VI-GISEL-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
-; VI-GISEL-NEXT: v_readfirstlane_b32 s0, v0
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[0:1], s0, v0
-; VI-GISEL-NEXT: s_and_saveexec_b64 s[8:9], s[0:1]
-; VI-GISEL-NEXT: v_readfirstlane_b32 s0, v7
-; VI-GISEL-NEXT: v_readfirstlane_b32 s1, v8
-; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v9
-; VI-GISEL-NEXT: v_readfirstlane_b32 s3, v10
-; VI-GISEL-NEXT: v_readfirstlane_b32 s4, v11
-; VI-GISEL-NEXT: v_readfirstlane_b32 s5, v12
-; VI-GISEL-NEXT: v_readfirstlane_b32 s6, v13
-; VI-GISEL-NEXT: v_readfirstlane_b32 s7, v14
-; VI-GISEL-NEXT: ; implicit-def: $vgpr0
-; VI-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
-; VI-GISEL-NEXT: s_nop 4
-; VI-GISEL-NEXT: image_store v[2:5], v6, s[0:7] dmask:0xf unorm
-; VI-GISEL-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
-; VI-GISEL-NEXT: ; implicit-def: $vgpr6
-; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[8:9]
-; VI-GISEL-NEXT: s_cbranch_execnz .LBB6_1
-; VI-GISEL-NEXT: ; %bb.2:
-; VI-GISEL-NEXT: s_endpgm
-;
-; GFX9-SDAG-LABEL: test_waterfall_non_uniform_img_single_store:
-; GFX9-SDAG: ; %bb.0:
-; GFX9-SDAG-NEXT: v_mov_b32_e32 v6, v1
-; GFX9-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
-; GFX9-SDAG-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
-; GFX9-SDAG-NEXT: v_mov_b32_e32 v1, s1
-; GFX9-SDAG-NEXT: v_add_co_u32_e32 v15, vcc, s0, v7
-; GFX9-SDAG-NEXT: v_addc_co_u32_e32 v16, vcc, v1, v8, vcc
-; GFX9-SDAG-NEXT: global_load_dwordx4 v[11:14], v[15:16], off offset:16
-; GFX9-SDAG-NEXT: global_load_dwordx4 v[7:10], v[15:16], off
-; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
-; GFX9-SDAG-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
-; GFX9-SDAG-NEXT: v_readfirstlane_b32 s0, v0
-; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[0:1], s0, v0
-; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[8:9], s[0:1]
-; GFX9-SDAG-NEXT: v_readfirstlane_b32 s0, v7
-; GFX9-SDAG-NEXT: v_readfirstlane_b32 s1, v8
-; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v9
-; GFX9-SDAG-NEXT: v_readfirstlane_b32 s3, v10
-; GFX9-SDAG-NEXT: v_readfirstlane_b32 s4, v11
-; GFX9-SDAG-NEXT: v_readfirstlane_b32 s5, v12
-; GFX9-SDAG-NEXT: v_readfirstlane_b32 s6, v13
-; GFX9-SDAG-NEXT: v_readfirstlane_b32 s7, v14
-; GFX9-SDAG-NEXT: ; implicit-def: $vgpr0
-; GFX9-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
-; GFX9-SDAG-NEXT: s_nop 4
-; GFX9-SDAG-NEXT: image_store v[2:5], v6, s[0:7] dmask:0xf unorm
-; GFX9-SDAG-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
-; GFX9-SDAG-NEXT: ; implicit-def: $vgpr6
-; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[8:9]
-; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB6_1
-; GFX9-SDAG-NEXT: ; %bb.2:
-; GFX9-SDAG-NEXT: s_endpgm
-;
-; GFX9-GISEL-LABEL: test_waterfall_non_uniform_img_single_store:
-; GFX9-GISEL: ; %bb.0:
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v6, v1
-; GFX9-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
-; GFX9-GISEL-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v10, s1
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, s0
-; GFX9-GISEL-NEXT: v_add_co_u32_e32 v15, vcc, v9, v7
-; GFX9-GISEL-NEXT: v_addc_co_u32_e32 v16, vcc, v10, v8, vcc
-; GFX9-GISEL-NEXT: global_load_dwordx4 v[7:10], v[15:16], off
-; GFX9-GISEL-NEXT: global_load_dwordx4 v[11:14], v[15:16], off offset:16
-; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX9-GISEL-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s0, v0
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[0:1], s0, v0
-; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[8:9], s[0:1]
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s0, v7
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s1, v8
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v9
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s3, v10
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s4, v11
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s5, v12
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s6, v13
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s7, v14
-; GFX9-GISEL-NEXT: ; implicit-def: $vgpr0
-; GFX9-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
-; GFX9-GISEL-NEXT: s_nop 4
-; GFX9-GISEL-NEXT: image_store v[2:5], v6, s[0:7] dmask:0xf unorm
-; GFX9-GISEL-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
-; GFX9-GISEL-NEXT: ; implicit-def: $vgpr6
-; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[8:9]
-; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB6_1
-; GFX9-GISEL-NEXT: ; %bb.2:
-; GFX9-GISEL-NEXT: s_endpgm
-;
-; GFX10-32-SDAG-LABEL: test_waterfall_non_uniform_img_single_store:
-; GFX10-32-SDAG: ; %bb.0:
-; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v6, v1
-; GFX10-32-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
-; GFX10-32-SDAG-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
-; GFX10-32-SDAG-NEXT: v_add_co_u32 v15, vcc_lo, s0, v7
-; GFX10-32-SDAG-NEXT: v_add_co_ci_u32_e32 v16, vcc_lo, s1, v8, vcc_lo
-; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
-; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
-; GFX10-32-SDAG-NEXT: s_clause 0x1
-; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[11:14], v[15:16], off offset:16
-; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[7:10], v[15:16], off
-; GFX10-32-SDAG-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
-; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s1, v0
-; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
-; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s1, v0
-; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
-; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s4, v7
-; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s5, v8
-; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s6, v9
-; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s7, v10
-; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s8, v11
-; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s9, v12
-; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s10, v13
-; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s11, v14
-; GFX10-32-SDAG-NEXT: image_store v[2:5], v6, s[4:11] dmask:0xf dim:SQ_RSRC_IMG_1D unorm
-; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s0, s0
-; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr0
-; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
-; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
-; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr6
-; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB6_1
-; GFX10-32-SDAG-NEXT: ; %bb.2:
-; GFX10-32-SDAG-NEXT: s_endpgm
-;
-; GFX10-32-GISEL-LABEL: test_waterfall_non_uniform_img_single_store:
-; GFX10-32-GISEL: ; %bb.0:
-; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v6, v1
-; GFX10-32-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
-; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v10, s1
-; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v9, s0
-; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
-; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
-; GFX10-32-GISEL-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
-; GFX10-32-GISEL-NEXT: v_add_co_u32 v15, vcc_lo, v9, v7
-; GFX10-32-GISEL-NEXT: v_add_co_ci_u32_e32 v16, vcc_lo, v10, v8, vcc_lo
-; GFX10-32-GISEL-NEXT: s_clause 0x1
-; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[7:10], v[15:16], off
-; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[11:14], v[15:16], off offset:16
-; GFX10-32-GISEL-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
-; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s1, v0
-; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
-; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s1, v0
-; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(1)
-; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s4, v7
-; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s5, v8
-; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s6, v9
-; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s7, v10
-; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s8, v11
-; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s9, v12
-; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s10, v13
-; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s11, v14
-; GFX10-32-GISEL-NEXT: image_store v[2:5], v6, s[4:11] dmask:0xf dim:SQ_RSRC_IMG_1D unorm
-; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s0, s0
-; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr0
-; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
-; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
-; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr6
-; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB6_1
-; GFX10-32-GISEL-NEXT: ; %bb.2:
-; GFX10-32-GISEL-NEXT: s_endpgm
-;
-; GFX10-64-SDAG-LABEL: test_waterfall_non_uniform_img_single_store:
-; GFX10-64-SDAG: ; %bb.0:
-; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v6, v1
-; GFX10-64-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
-; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
-; GFX10-64-SDAG-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
-; GFX10-64-SDAG-NEXT: v_add_co_u32 v15, vcc, s0, v7
-; GFX10-64-SDAG-NEXT: v_add_co_ci_u32_e32 v16, vcc, s1, v8, vcc
-; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX10-64-SDAG-NEXT: s_clause 0x1
-; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[11:14], v[15:16], off offset:16
-; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[7:10], v[15:16], off
-; GFX10-64-SDAG-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
-; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s2, v0
-; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
-; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s2, v0
-; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
-; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s4, v7
-; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s5, v8
-; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s6, v9
-; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s7, v10
-; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s8, v11
-; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s9, v12
-; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s10, v13
-; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s11, v14
-; GFX10-64-SDAG-NEXT: image_store v[2:5], v6, s[4:11] dmask:0xf dim:SQ_RSRC_IMG_1D unorm
-; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[0:1], s[0:1]
-; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr0
-; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
-; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
-; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr6
-; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB6_1
-; GFX10-64-SDAG-NEXT: ; %bb.2:
-; GFX10-64-SDAG-NEXT: s_endpgm
-;
-; GFX10-64-GISEL-LABEL: test_waterfall_non_uniform_img_single_store:
-; GFX10-64-GISEL: ; %bb.0:
-; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v6, v1
-; GFX10-64-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
-; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v10, s1
-; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v9, s0
-; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
-; GFX10-64-GISEL-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
-; GFX10-64-GISEL-NEXT: v_add_co_u32 v15, vcc, v9, v7
-; GFX10-64-GISEL-NEXT: v_add_co_ci_u32_e32 v16, vcc, v10, v8, vcc
-; GFX10-64-GISEL-NEXT: s_clause 0x1
-; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[7:10], v[15:16], off
-; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[11:14], v[15:16], off offset:16
-; GFX10-64-GISEL-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
-; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s2, v0
-; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
-; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s2, v0
-; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(1)
-; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s4, v7
-; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s5, v8
-; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s6, v9
-; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s7, v10
-; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s8, v11
-; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s9, v12
-; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s10, v13
-; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s11, v14
-; GFX10-64-GISEL-NEXT: image_store v[2:5], v6, s[4:11] dmask:0xf dim:SQ_RSRC_IMG_1D unorm
-; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[0:1], s[0:1]
-; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr0
-; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
-; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
-; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr6
-; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB6_1
-; GFX10-64-GISEL-NEXT: ; %bb.2:
-; GFX10-64-GISEL-NEXT: s_endpgm
-;
-; GFX1150-SDAG-LABEL: test_waterfall_non_uniform_img_single_store:
-; GFX1150-SDAG: ; %bb.0:
-; GFX1150-SDAG-NEXT: v_mov_b32_e32 v6, v1
-; GFX1150-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
-; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
-; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1150-SDAG-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
-; GFX1150-SDAG-NEXT: v_add_co_u32 v7, vcc, s0, v7
-; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX1150-SDAG-NEXT: v_add_co_ci_u32_e64 v8, null, s1, v8, vcc
-; GFX1150-SDAG-NEXT: s_clause 0x1
-; GFX1150-SDAG-NEXT: global_load_b128 v[11:14], v[7:8], off offset:16
-; GFX1150-SDAG-NEXT: global_load_b128 v[7:10], v[7:8], off
-; GFX1150-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX1150-SDAG-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
-; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s2, v0
-; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s2, v0
-; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
-; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s4, v7
-; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s5, v8
-; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s6, v9
-; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s7, v10
-; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s8, v11
-; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s9, v12
-; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s10, v13
-; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s11, v14
-; GFX1150-SDAG-NEXT: image_store v[2:5], v6, s[4:11] dmask:0xf dim:SQ_RSRC_IMG_1D unorm
-; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[0:1], s[0:1]
-; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr0
-; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
-; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
-; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr6
-; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB6_1
-; GFX1150-SDAG-NEXT: ; %bb.2:
-; GFX1150-SDAG-NEXT: s_endpgm
-;
-; GFX1150-GISEL-LABEL: test_waterfall_non_uniform_img_single_store:
-; GFX1150-GISEL: ; %bb.0:
-; GFX1150-GISEL-NEXT: v_mov_b32_e32 v6, v1
-; GFX1150-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
-; GFX1150-GISEL-NEXT: v_mov_b32_e32 v10, s1
-; GFX1150-GISEL-NEXT: v_mov_b32_e32 v9, s0
-; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
-; GFX1150-GISEL-NEXT: v_lshlrev_b64 v[7:8], 5, v[0:1]
-; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1150-GISEL-NEXT: v_add_co_u32 v11, vcc, v9, v7
-; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v12, null, v10, v8, vcc
-; GFX1150-GISEL-NEXT: s_clause 0x1
-; GFX1150-GISEL-NEXT: global_load_b128 v[7:10], v[11:12], off
-; GFX1150-GISEL-NEXT: global_load_b128 v[11:14], v[11:12], off offset:16
-; GFX1150-GISEL-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
-; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s2, v0
-; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s2, v0
-; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(1)
-; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s4, v7
-; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s5, v8
-; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s6, v9
-; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s7, v10
-; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s8, v11
-; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s9, v12
-; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s10, v13
-; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s11, v14
-; GFX1150-GISEL-NEXT: image_store v[2:5], v6, s[4:11] dmask:0xf dim:SQ_RSRC_IMG_1D unorm
-; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[0:1], s[0:1]
-; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr0
-; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
-; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
-; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr6
-; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB6_1
-; GFX1150-GISEL-NEXT: ; %bb.2:
-; GFX1150-GISEL-NEXT: s_endpgm
-;
-; GFX12-SDAG-LABEL: test_waterfall_non_uniform_img_single_store:
-; GFX12-SDAG: ; %bb.0:
-; GFX12-SDAG-NEXT: v_mov_b32_e32 v6, v1
-; GFX12-SDAG-NEXT: v_ashrrev_i32_e32 v1, 31, v0
-; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
-; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX12-SDAG-NEXT: v_lshlrev_b64_e32 v[7:8], 5, v[0:1]
-; GFX12-SDAG-NEXT: v_add_co_u32 v7, vcc, s0, v7
-; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX12-SDAG-NEXT: v_add_co_ci_u32_e64 v8, null, s1, v8, vcc
-; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX12-SDAG-NEXT: s_clause 0x1
-; GFX12-SDAG-NEXT: global_load_b128 v[11:14], v[7:8], off offset:16
-; GFX12-SDAG-NEXT: global_load_b128 v[7:10], v[7:8], off
-; GFX12-SDAG-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
-; GFX12-SDAG-NEXT: v_readfirstlane_b32 s2, v0
-; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
-; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s2, v0
-; GFX12-SDAG-NEXT: s_wait_loadcnt 0x0
-; GFX12-SDAG-NEXT: v_readfirstlane_b32 s4, v7
-; GFX12-SDAG-NEXT: v_readfirstlane_b32 s5, v8
-; GFX12-SDAG-NEXT: v_readfirstlane_b32 s6, v9
-; GFX12-SDAG-NEXT: v_readfirstlane_b32 s7, v10
-; GFX12-SDAG-NEXT: v_readfirstlane_b32 s8, v11
-; GFX12-SDAG-NEXT: v_readfirstlane_b32 s9, v12
-; GFX12-SDAG-NEXT: v_readfirstlane_b32 s10, v13
-; GFX12-SDAG-NEXT: v_readfirstlane_b32 s11, v14
-; GFX12-SDAG-NEXT: image_store v[2:5], v6, s[4:11] dmask:0xf dim:SQ_RSRC_IMG_1D
-; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[0:1], s[0:1]
-; GFX12-SDAG-NEXT: ; implicit-def: $vgpr0
-; GFX12-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
-; GFX12-SDAG-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
-; GFX12-SDAG-NEXT: ; implicit-def: $vgpr6
-; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB6_1
-; GFX12-SDAG-NEXT: ; %bb.2:
-; GFX12-SDAG-NEXT: s_endpgm
-;
-; GFX12-GISEL-LABEL: test_waterfall_non_uniform_img_single_store:
-; GFX12-GISEL: ; %bb.0:
-; GFX12-GISEL-NEXT: v_mov_b32_e32 v6, v1
-; GFX12-GISEL-NEXT: v_ashrrev_i32_e32 v1, 31, v0
-; GFX12-GISEL-NEXT: v_mov_b32_e32 v10, s1
-; GFX12-GISEL-NEXT: v_mov_b32_e32 v9, s0
-; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
-; GFX12-GISEL-NEXT: v_lshlrev_b64_e32 v[7:8], 5, v[0:1]
-; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX12-GISEL-NEXT: v_add_co_u32 v11, vcc, v9, v7
-; GFX12-GISEL-NEXT: v_add_co_ci_u32_e64 v12, null, v10, v8, vcc
-; GFX12-GISEL-NEXT: s_clause 0x1
-; GFX12-GISEL-NEXT: global_load_b128 v[7:10], v[11:12], off
-; GFX12-GISEL-NEXT: global_load_b128 v[11:14], v[11:12], off offset:16
-; GFX12-GISEL-NEXT: .LBB6_1: ; =>This Inner Loop Header: Depth=1
-; GFX12-GISEL-NEXT: v_readfirstlane_b32 s2, v0
-; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
-; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s2, v0
-; GFX12-GISEL-NEXT: s_wait_loadcnt 0x1
-; GFX12-GISEL-NEXT: v_readfirstlane_b32 s4, v7
-; GFX12-GISEL-NEXT: v_readfirstlane_b32 s5, v8
-; GFX12-GISEL-NEXT: v_readfirstlane_b32 s6, v9
-; GFX12-GISEL-NEXT: v_readfirstlane_b32 s7, v10
-; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
-; GFX12-GISEL-NEXT: v_readfirstlane_b32 s8, v11
-; GFX12-GISEL-NEXT: v_readfirstlane_b32 s9, v12
-; GFX12-GISEL-NEXT: v_readfirstlane_b32 s10, v13
-; GFX12-GISEL-NEXT: v_readfirstlane_b32 s11, v14
-; GFX12-GISEL-NEXT: image_store v[2:5], v6, s[4:11] dmask:0xf dim:SQ_RSRC_IMG_1D
-; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[0:1], s[0:1]
-; GFX12-GISEL-NEXT: ; implicit-def: $vgpr0
-; GFX12-GISEL-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
-; GFX12-GISEL-NEXT: ; implicit-def: $vgpr2_vgpr3_vgpr4_vgpr5
-; GFX12-GISEL-NEXT: ; implicit-def: $vgpr6
-; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB6_1
-; GFX12-GISEL-NEXT: ; %bb.2:
-; GFX12-GISEL-NEXT: s_endpgm
- %ptr = getelementptr <8 x i32>, <8 x i32> addrspace(4)* %in, i32 %index
- %rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %ptr, align 32
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
- %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %wf_token, <8 x i32> %rsrc)
- %s_rsrc_use = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(token %wf_token, <8 x i32> %s_rsrc)
- call void @llvm.amdgcn.image.store.1d.v4f32.i32(<4 x float> %data, i32 15, i32 %s, <8 x i32> %s_rsrc_use, i32 0, i32 0)
-
- ret void
-}
-
-define amdgpu_ps void @test_remove_waterfall_last_use(<8 x i32> addrspace(4)* inreg %in, i32 %index, i32 %s, <4 x float> %data) #1 {
-; PRE-GFX10-LABEL: test_remove_waterfall_last_use:
-; PRE-GFX10: ; %bb.0:
-; PRE-GFX10-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x0
-; PRE-GFX10-NEXT: s_waitcnt lgkmcnt(0)
-; PRE-GFX10-NEXT: image_store v[2:5], v1, s[0:7] dmask:0xf unorm
-; PRE-GFX10-NEXT: s_endpgm
-;
-; GFX10-LABEL: test_remove_waterfall_last_use:
-; GFX10: ; %bb.0:
-; GFX10-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x0
-; GFX10-NEXT: s_waitcnt lgkmcnt(0)
-; GFX10-NEXT: image_store v[2:5], v1, s[0:7] dmask:0xf dim:SQ_RSRC_IMG_1D unorm
-; GFX10-NEXT: s_endpgm
-;
-; GFX1150-LABEL: test_remove_waterfall_last_use:
-; GFX1150: ; %bb.0:
-; GFX1150-NEXT: s_load_b256 s[0:7], s[0:1], 0x0
-; GFX1150-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1150-NEXT: image_store v[2:5], v1, s[0:7] dmask:0xf dim:SQ_RSRC_IMG_1D unorm
-; GFX1150-NEXT: s_endpgm
-;
-; GFX12-LABEL: test_remove_waterfall_last_use:
-; GFX12: ; %bb.0:
-; GFX12-NEXT: s_load_b256 s[0:7], s[0:1], 0x0
-; GFX12-NEXT: s_wait_kmcnt 0x0
-; GFX12-NEXT: image_store v[2:5], v1, s[0:7] dmask:0xf dim:SQ_RSRC_IMG_1D
-; GFX12-NEXT: s_endpgm
- %rsrc = load <8 x i32>, <8 x i32> addrspace(4) * %in, align 32
- %wf_token = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %index)
- %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %wf_token, <8 x i32> %rsrc)
- %s_rsrc_use = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(token %wf_token, <8 x i32> %s_rsrc)
- call void @llvm.amdgcn.image.store.1d.v4f32.i32(<4 x float> %data, i32 15, i32 %s, <8 x i32> %s_rsrc_use, i32 0, i32 0)
-
- ret void
-}
-
define amdgpu_ps <4 x float> @test_remove_waterfall_multi_rl(<8 x i32> addrspace(4)* inreg %in, <4 x i32> addrspace(4)* inreg %samp_in, i32 %index, float %s, i32 inreg %val1, i32 inreg %val2) #1 {
; VI-LABEL: test_remove_waterfall_multi_rl:
; VI: ; %bb.0:
@@ -3422,7 +2929,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; VI-SDAG-NEXT: v_mov_b32_e32 v4, v1
; VI-SDAG-NEXT: v_mov_b32_e32 v5, v0
; VI-SDAG-NEXT: s_mov_b64 s[8:9], exec
-; VI-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: .LBB7_1: ; =>This Inner Loop Header: Depth=1
; VI-SDAG-NEXT: v_readfirstlane_b32 s10, v5
; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[12:13], s10, v5
; VI-SDAG-NEXT: s_and_saveexec_b64 s[24:25], s[12:13]
@@ -3441,7 +2948,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; VI-SDAG-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf
; VI-SDAG-NEXT: ; implicit-def: $vgpr4
; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[24:25]
-; VI-SDAG-NEXT: s_cbranch_execnz .LBB9_1
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB7_1
; VI-SDAG-NEXT: ; %bb.2:
; VI-SDAG-NEXT: s_mov_b64 exec, s[8:9]
; VI-SDAG-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -3455,7 +2962,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; VI-GISEL-NEXT: v_mov_b32_e32 v5, v0
; VI-GISEL-NEXT: v_mov_b32_e32 v4, v1
; VI-GISEL-NEXT: s_mov_b64 s[8:9], exec
-; VI-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: .LBB7_1: ; =>This Inner Loop Header: Depth=1
; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v5
; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[12:13], s10, v5
; VI-GISEL-NEXT: s_and_saveexec_b64 s[24:25], s[12:13]
@@ -3474,7 +2981,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; VI-GISEL-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf
; VI-GISEL-NEXT: ; implicit-def: $vgpr4
; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[24:25]
-; VI-GISEL-NEXT: s_cbranch_execnz .LBB9_1
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB7_1
; VI-GISEL-NEXT: ; %bb.2:
; VI-GISEL-NEXT: s_mov_b64 exec, s[8:9]
; VI-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -3488,7 +2995,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX9-SDAG-NEXT: v_mov_b32_e32 v4, v1
; GFX9-SDAG-NEXT: v_mov_b32_e32 v5, v0
; GFX9-SDAG-NEXT: s_mov_b64 s[8:9], exec
-; GFX9-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: .LBB7_1: ; =>This Inner Loop Header: Depth=1
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s10, v5
; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[12:13], s10, v5
; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[24:25], s[12:13]
@@ -3507,7 +3014,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX9-SDAG-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf
; GFX9-SDAG-NEXT: ; implicit-def: $vgpr4
; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[24:25]
-; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB9_1
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB7_1
; GFX9-SDAG-NEXT: ; %bb.2:
; GFX9-SDAG-NEXT: s_mov_b64 exec, s[8:9]
; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -3521,7 +3028,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX9-GISEL-NEXT: v_mov_b32_e32 v5, v0
; GFX9-GISEL-NEXT: v_mov_b32_e32 v4, v1
; GFX9-GISEL-NEXT: s_mov_b64 s[8:9], exec
-; GFX9-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: .LBB7_1: ; =>This Inner Loop Header: Depth=1
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v5
; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[12:13], s10, v5
; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[24:25], s[12:13]
@@ -3540,7 +3047,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX9-GISEL-NEXT: image_sample v[0:3], v4, s[12:19], s[20:23] dmask:0xf
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[24:25]
-; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB9_1
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB7_1
; GFX9-GISEL-NEXT: ; %bb.2:
; GFX9-GISEL-NEXT: s_mov_b64 exec, s[8:9]
; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -3555,7 +3062,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v5, v0
; GFX10-32-SDAG-NEXT: s_mov_b32 s8, exec_lo
; GFX10-32-SDAG-NEXT: s_mov_b32 s7, exec_lo
-; GFX10-32-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: .LBB7_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s10, v5
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s10, v5
@@ -3574,7 +3081,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s8, s8
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr5
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr4
-; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB9_1
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB7_1
; GFX10-32-SDAG-NEXT: ; %bb.2:
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s7
@@ -3590,7 +3097,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v4, v1
; GFX10-32-GISEL-NEXT: s_mov_b32 s8, exec_lo
; GFX10-32-GISEL-NEXT: s_mov_b32 s7, exec_lo
-; GFX10-32-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: .LBB7_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s10, v5
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s10, v5
@@ -3609,7 +3116,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s8, s8
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4
-; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB9_1
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB7_1
; GFX10-32-GISEL-NEXT: ; %bb.2:
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s7
@@ -3625,7 +3132,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v5, v0
; GFX10-64-SDAG-NEXT: s_mov_b64 s[10:11], exec
; GFX10-64-SDAG-NEXT: s_mov_b64 s[8:9], exec
-; GFX10-64-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: .LBB7_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s12, v5
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s12, v5
@@ -3644,7 +3151,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[10:11], s[10:11]
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr5
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr4
-; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB9_1
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB7_1
; GFX10-64-SDAG-NEXT: ; %bb.2:
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[8:9]
@@ -3660,7 +3167,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v4, v1
; GFX10-64-GISEL-NEXT: s_mov_b64 s[10:11], exec
; GFX10-64-GISEL-NEXT: s_mov_b64 s[8:9], exec
-; GFX10-64-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: .LBB7_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s12, v5
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s12, v5
@@ -3679,7 +3186,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[10:11], s[10:11]
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4
-; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB9_1
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB7_1
; GFX10-64-GISEL-NEXT: ; %bb.2:
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[8:9]
@@ -3695,7 +3202,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX1150-SDAG-NEXT: v_mov_b32_e32 v5, v0
; GFX1150-SDAG-NEXT: s_mov_b64 s[10:11], exec
; GFX1150-SDAG-NEXT: s_mov_b64 s[8:9], exec
-; GFX1150-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: .LBB7_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s12, v5
; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s12, v5
@@ -3716,7 +3223,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr5
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr4
-; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB9_1
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB7_1
; GFX1150-SDAG-NEXT: ; %bb.2:
; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[8:9]
; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -3732,7 +3239,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX1150-GISEL-NEXT: v_mov_b32_e32 v4, v1
; GFX1150-GISEL-NEXT: s_mov_b64 s[10:11], exec
; GFX1150-GISEL-NEXT: s_mov_b64 s[8:9], exec
-; GFX1150-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: .LBB7_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s12, v5
; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s12, v5
@@ -3753,7 +3260,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4
-; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB9_1
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB7_1
; GFX1150-GISEL-NEXT: ; %bb.2:
; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[8:9]
; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -3769,7 +3276,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX12-SDAG-NEXT: v_mov_b32_e32 v5, v0
; GFX12-SDAG-NEXT: s_mov_b64 s[10:11], exec
; GFX12-SDAG-NEXT: s_mov_b64 s[8:9], exec
-; GFX12-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: .LBB7_1: ; =>This Inner Loop Header: Depth=1
; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s12, v5
; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
@@ -3790,7 +3297,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr5
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr4
-; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB9_1
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB7_1
; GFX12-SDAG-NEXT: ; %bb.2:
; GFX12-SDAG-NEXT: s_mov_b64 exec, s[8:9]
; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -3806,7 +3313,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX12-GISEL-NEXT: v_mov_b32_e32 v4, v1
; GFX12-GISEL-NEXT: s_mov_b64 s[10:11], exec
; GFX12-GISEL-NEXT: s_mov_b64 s[8:9], exec
-; GFX12-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: .LBB7_1: ; =>This Inner Loop Header: Depth=1
; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(SKIP_1) | instid1(VALU_DEP_1)
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s12, v5
; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
@@ -3830,7 +3337,7 @@ define amdgpu_ps <4 x float> @test_keep_waterfall_multi_rl(<8 x i32> addrspace(4
; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4
-; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB9_1
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB7_1
; GFX12-GISEL-NEXT: ; %bb.2:
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[8:9]
; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -3856,7 +3363,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; VI-NEXT: s_mov_b64 s[6:7], exec
; VI-NEXT: s_wqm_b64 exec, exec
; VI-NEXT: s_mov_b64 s[8:9], exec
-; VI-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
+; VI-NEXT: .LBB8_1: ; =>This Inner Loop Header: Depth=1
; VI-NEXT: v_readfirstlane_b32 s10, v0
; VI-NEXT: v_cmp_eq_u32_e64 s[12:13], s10, v0
; VI-NEXT: s_and_saveexec_b64 s[24:25], s[12:13]
@@ -3875,7 +3382,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; VI-NEXT: image_sample v[5:8], v1, s[12:19], s[20:23] dmask:0xf
; VI-NEXT: ; implicit-def: $vgpr1
; VI-NEXT: s_xor_b64 exec, exec, s[24:25]
-; VI-NEXT: s_cbranch_execnz .LBB10_1
+; VI-NEXT: s_cbranch_execnz .LBB8_1
; VI-NEXT: ; %bb.2:
; VI-NEXT: s_mov_b64 exec, s[8:9]
; VI-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -3883,18 +3390,18 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; VI-NEXT: v_cmp_gt_f32_e32 vcc, 0, v5
; VI-NEXT: s_and_saveexec_b64 s[0:1], vcc
; VI-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
-; VI-NEXT: s_cbranch_execz .LBB10_5
+; VI-NEXT: s_cbranch_execz .LBB8_5
; VI-NEXT: ; %bb.3: ; %.kill
; VI-NEXT: s_andn2_b64 s[6:7], s[6:7], exec
-; VI-NEXT: s_cbranch_scc0 .LBB10_6
+; VI-NEXT: s_cbranch_scc0 .LBB8_6
; VI-NEXT: ; %bb.4: ; %.kill
; VI-NEXT: s_mov_b64 exec, 0
-; VI-NEXT: .LBB10_5: ; %.exit
+; VI-NEXT: .LBB8_5: ; %.exit
; VI-NEXT: s_or_b64 exec, exec, s[0:1]
; VI-NEXT: v_mov_b32_e32 v0, 0
; VI-NEXT: exp mrt0, v0, off, off, off done vm
; VI-NEXT: s_endpgm
-; VI-NEXT: .LBB10_6:
+; VI-NEXT: .LBB8_6:
; VI-NEXT: s_mov_b64 exec, 0
; VI-NEXT: exp null, off, off, off, off done vm
; VI-NEXT: s_endpgm
@@ -3904,7 +3411,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX9-NEXT: s_mov_b64 s[6:7], exec
; GFX9-NEXT: s_wqm_b64 exec, exec
; GFX9-NEXT: s_mov_b64 s[8:9], exec
-; GFX9-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT: .LBB8_1: ; =>This Inner Loop Header: Depth=1
; GFX9-NEXT: v_readfirstlane_b32 s10, v0
; GFX9-NEXT: v_cmp_eq_u32_e64 s[12:13], s10, v0
; GFX9-NEXT: s_and_saveexec_b64 s[24:25], s[12:13]
@@ -3923,7 +3430,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX9-NEXT: image_sample v[5:8], v1, s[12:19], s[20:23] dmask:0xf
; GFX9-NEXT: ; implicit-def: $vgpr1
; GFX9-NEXT: s_xor_b64 exec, exec, s[24:25]
-; GFX9-NEXT: s_cbranch_execnz .LBB10_1
+; GFX9-NEXT: s_cbranch_execnz .LBB8_1
; GFX9-NEXT: ; %bb.2:
; GFX9-NEXT: s_mov_b64 exec, s[8:9]
; GFX9-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -3931,18 +3438,18 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX9-NEXT: v_cmp_gt_f32_e32 vcc, 0, v5
; GFX9-NEXT: s_and_saveexec_b64 s[0:1], vcc
; GFX9-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
-; GFX9-NEXT: s_cbranch_execz .LBB10_5
+; GFX9-NEXT: s_cbranch_execz .LBB8_5
; GFX9-NEXT: ; %bb.3: ; %.kill
; GFX9-NEXT: s_andn2_b64 s[6:7], s[6:7], exec
-; GFX9-NEXT: s_cbranch_scc0 .LBB10_6
+; GFX9-NEXT: s_cbranch_scc0 .LBB8_6
; GFX9-NEXT: ; %bb.4: ; %.kill
; GFX9-NEXT: s_mov_b64 exec, 0
-; GFX9-NEXT: .LBB10_5: ; %.exit
+; GFX9-NEXT: .LBB8_5: ; %.exit
; GFX9-NEXT: s_or_b64 exec, exec, s[0:1]
; GFX9-NEXT: v_mov_b32_e32 v0, 0
; GFX9-NEXT: exp mrt0, v0, off, off, off done vm
; GFX9-NEXT: s_endpgm
-; GFX9-NEXT: .LBB10_6:
+; GFX9-NEXT: .LBB8_6:
; GFX9-NEXT: s_mov_b64 exec, 0
; GFX9-NEXT: exp null, off, off, off, off done vm
; GFX9-NEXT: s_endpgm
@@ -3953,7 +3460,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX10-32-NEXT: s_wqm_b32 exec_lo, exec_lo
; GFX10-32-NEXT: s_mov_b32 s8, exec_lo
; GFX10-32-NEXT: s_mov_b32 s7, exec_lo
-; GFX10-32-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-NEXT: .LBB8_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-NEXT: v_readfirstlane_b32 s10, v0
; GFX10-32-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-32-NEXT: v_cmpx_eq_u32_e32 s10, v0
@@ -3972,7 +3479,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX10-32-NEXT: s_andn2_wrexec_b32 s8, s8
; GFX10-32-NEXT: ; implicit-def: $vgpr0
; GFX10-32-NEXT: ; implicit-def: $vgpr1
-; GFX10-32-NEXT: s_cbranch_execnz .LBB10_1
+; GFX10-32-NEXT: s_cbranch_execnz .LBB8_1
; GFX10-32-NEXT: ; %bb.2:
; GFX10-32-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-NEXT: s_mov_b32 exec_lo, s7
@@ -3981,18 +3488,18 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX10-32-NEXT: v_cmp_gt_f32_e32 vcc_lo, 0, v5
; GFX10-32-NEXT: s_and_saveexec_b32 s0, vcc_lo
; GFX10-32-NEXT: s_xor_b32 s0, exec_lo, s0
-; GFX10-32-NEXT: s_cbranch_execz .LBB10_5
+; GFX10-32-NEXT: s_cbranch_execz .LBB8_5
; GFX10-32-NEXT: ; %bb.3: ; %.kill
; GFX10-32-NEXT: s_andn2_b32 s6, s6, exec_lo
-; GFX10-32-NEXT: s_cbranch_scc0 .LBB10_6
+; GFX10-32-NEXT: s_cbranch_scc0 .LBB8_6
; GFX10-32-NEXT: ; %bb.4: ; %.kill
; GFX10-32-NEXT: s_mov_b32 exec_lo, 0
-; GFX10-32-NEXT: .LBB10_5: ; %.exit
+; GFX10-32-NEXT: .LBB8_5: ; %.exit
; GFX10-32-NEXT: s_or_b32 exec_lo, exec_lo, s0
; GFX10-32-NEXT: v_mov_b32_e32 v0, 0
; GFX10-32-NEXT: exp mrt0, v0, off, off, off done vm
; GFX10-32-NEXT: s_endpgm
-; GFX10-32-NEXT: .LBB10_6:
+; GFX10-32-NEXT: .LBB8_6:
; GFX10-32-NEXT: s_mov_b32 exec_lo, 0
; GFX10-32-NEXT: exp null, off, off, off, off done vm
; GFX10-32-NEXT: s_endpgm
@@ -4003,7 +3510,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX10-64-NEXT: s_wqm_b64 exec, exec
; GFX10-64-NEXT: s_mov_b64 s[10:11], exec
; GFX10-64-NEXT: s_mov_b64 s[8:9], exec
-; GFX10-64-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-NEXT: .LBB8_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-NEXT: v_readfirstlane_b32 s12, v0
; GFX10-64-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-64-NEXT: v_cmpx_eq_u32_e64 s12, v0
@@ -4022,7 +3529,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX10-64-NEXT: s_andn2_wrexec_b64 s[10:11], s[10:11]
; GFX10-64-NEXT: ; implicit-def: $vgpr0
; GFX10-64-NEXT: ; implicit-def: $vgpr1
-; GFX10-64-NEXT: s_cbranch_execnz .LBB10_1
+; GFX10-64-NEXT: s_cbranch_execnz .LBB8_1
; GFX10-64-NEXT: ; %bb.2:
; GFX10-64-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-NEXT: s_mov_b64 exec, s[8:9]
@@ -4031,18 +3538,18 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX10-64-NEXT: v_cmp_gt_f32_e32 vcc, 0, v5
; GFX10-64-NEXT: s_and_saveexec_b64 s[0:1], vcc
; GFX10-64-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
-; GFX10-64-NEXT: s_cbranch_execz .LBB10_5
+; GFX10-64-NEXT: s_cbranch_execz .LBB8_5
; GFX10-64-NEXT: ; %bb.3: ; %.kill
; GFX10-64-NEXT: s_andn2_b64 s[6:7], s[6:7], exec
-; GFX10-64-NEXT: s_cbranch_scc0 .LBB10_6
+; GFX10-64-NEXT: s_cbranch_scc0 .LBB8_6
; GFX10-64-NEXT: ; %bb.4: ; %.kill
; GFX10-64-NEXT: s_mov_b64 exec, 0
-; GFX10-64-NEXT: .LBB10_5: ; %.exit
+; GFX10-64-NEXT: .LBB8_5: ; %.exit
; GFX10-64-NEXT: s_or_b64 exec, exec, s[0:1]
; GFX10-64-NEXT: v_mov_b32_e32 v0, 0
; GFX10-64-NEXT: exp mrt0, v0, off, off, off done vm
; GFX10-64-NEXT: s_endpgm
-; GFX10-64-NEXT: .LBB10_6:
+; GFX10-64-NEXT: .LBB8_6:
; GFX10-64-NEXT: s_mov_b64 exec, 0
; GFX10-64-NEXT: exp null, off, off, off, off done vm
; GFX10-64-NEXT: s_endpgm
@@ -4055,7 +3562,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX1150-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
; GFX1150-NEXT: s_mov_b64 s[10:11], exec
; GFX1150-NEXT: s_mov_b64 s[8:9], exec
-; GFX1150-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-NEXT: .LBB8_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-NEXT: v_readfirstlane_b32 s12, v0
; GFX1150-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
; GFX1150-NEXT: v_cmpx_eq_u32_e64 s12, v0
@@ -4076,7 +3583,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX1150-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
; GFX1150-NEXT: ; implicit-def: $vgpr0
; GFX1150-NEXT: ; implicit-def: $vgpr1
-; GFX1150-NEXT: s_cbranch_execnz .LBB10_1
+; GFX1150-NEXT: s_cbranch_execnz .LBB8_1
; GFX1150-NEXT: ; %bb.2:
; GFX1150-NEXT: s_mov_b64 exec, s[8:9]
; GFX1150-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
@@ -4085,13 +3592,13 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX1150-NEXT: s_waitcnt vmcnt(0)
; GFX1150-NEXT: v_cmpx_gt_f32_e32 0, v5
; GFX1150-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
-; GFX1150-NEXT: s_cbranch_execz .LBB10_5
+; GFX1150-NEXT: s_cbranch_execz .LBB8_5
; GFX1150-NEXT: ; %bb.3: ; %.kill
; GFX1150-NEXT: s_and_not1_b64 s[6:7], s[6:7], exec
-; GFX1150-NEXT: s_cbranch_scc0 .LBB10_6
+; GFX1150-NEXT: s_cbranch_scc0 .LBB8_6
; GFX1150-NEXT: ; %bb.4: ; %.kill
; GFX1150-NEXT: s_mov_b64 exec, 0
-; GFX1150-NEXT: .LBB10_5: ; %.exit
+; GFX1150-NEXT: .LBB8_5: ; %.exit
; GFX1150-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
; GFX1150-NEXT: s_or_b64 exec, exec, s[0:1]
; GFX1150-NEXT: v_mov_b32_e32 v0, 0
@@ -4100,7 +3607,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX1150-NEXT: s_nop 0
; GFX1150-NEXT: s_nop 0
; GFX1150-NEXT: s_endpgm
-; GFX1150-NEXT: .LBB10_6:
+; GFX1150-NEXT: .LBB8_6:
; GFX1150-NEXT: s_mov_b64 exec, 0
; GFX1150-NEXT: exp mrt0, off, off, off, off done
; GFX1150-NEXT: s_setprio 0
@@ -4115,7 +3622,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
; GFX12-SDAG-NEXT: s_mov_b64 s[10:11], exec
; GFX12-SDAG-NEXT: s_mov_b64 s[8:9], exec
-; GFX12-SDAG-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: .LBB8_1: ; =>This Inner Loop Header: Depth=1
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s12, v0
; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
@@ -4136,7 +3643,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr0
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr1
-; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB10_1
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB8_1
; GFX12-SDAG-NEXT: ; %bb.2:
; GFX12-SDAG-NEXT: s_mov_b64 exec, s[8:9]
; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
@@ -4145,19 +3652,19 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
; GFX12-SDAG-NEXT: v_cmpx_gt_f32_e32 0, v5
; GFX12-SDAG-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
-; GFX12-SDAG-NEXT: s_cbranch_execz .LBB10_5
+; GFX12-SDAG-NEXT: s_cbranch_execz .LBB8_5
; GFX12-SDAG-NEXT: ; %bb.3: ; %.kill
; GFX12-SDAG-NEXT: s_and_not1_b64 s[6:7], s[6:7], exec
-; GFX12-SDAG-NEXT: s_cbranch_scc0 .LBB10_6
+; GFX12-SDAG-NEXT: s_cbranch_scc0 .LBB8_6
; GFX12-SDAG-NEXT: ; %bb.4: ; %.kill
; GFX12-SDAG-NEXT: s_mov_b64 exec, 0
-; GFX12-SDAG-NEXT: .LBB10_5: ; %.exit
+; GFX12-SDAG-NEXT: .LBB8_5: ; %.exit
; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
; GFX12-SDAG-NEXT: s_or_b64 exec, exec, s[0:1]
; GFX12-SDAG-NEXT: v_mov_b32_e32 v0, 0
; GFX12-SDAG-NEXT: export mrt0, v0, off, off, off done
; GFX12-SDAG-NEXT: s_endpgm
-; GFX12-SDAG-NEXT: .LBB10_6:
+; GFX12-SDAG-NEXT: .LBB8_6:
; GFX12-SDAG-NEXT: s_mov_b64 exec, 0
; GFX12-SDAG-NEXT: export mrt0, off, off, off, off done
; GFX12-SDAG-NEXT: s_endpgm
@@ -4169,7 +3676,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
; GFX12-GISEL-NEXT: s_mov_b64 s[10:11], exec
; GFX12-GISEL-NEXT: s_mov_b64 s[8:9], exec
-; GFX12-GISEL-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: .LBB8_1: ; =>This Inner Loop Header: Depth=1
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s12, v0
; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
@@ -4193,7 +3700,7 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr0
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr1
-; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB10_1
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB8_1
; GFX12-GISEL-NEXT: ; %bb.2:
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[8:9]
; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
@@ -4202,19 +3709,19 @@ define amdgpu_ps void @test_waterfall_sample_with_kill(<8 x i32> addrspace(4)* i
; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
; GFX12-GISEL-NEXT: v_cmpx_gt_f32_e32 0, v5
; GFX12-GISEL-NEXT: s_xor_b64 s[0:1], exec, s[0:1]
-; GFX12-GISEL-NEXT: s_cbranch_execz .LBB10_5
+; GFX12-GISEL-NEXT: s_cbranch_execz .LBB8_5
; GFX12-GISEL-NEXT: ; %bb.3: ; %.kill
; GFX12-GISEL-NEXT: s_and_not1_b64 s[6:7], s[6:7], exec
-; GFX12-GISEL-NEXT: s_cbranch_scc0 .LBB10_6
+; GFX12-GISEL-NEXT: s_cbranch_scc0 .LBB8_6
; GFX12-GISEL-NEXT: ; %bb.4: ; %.kill
; GFX12-GISEL-NEXT: s_mov_b64 exec, 0
-; GFX12-GISEL-NEXT: .LBB10_5: ; %.exit
+; GFX12-GISEL-NEXT: .LBB8_5: ; %.exit
; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
; GFX12-GISEL-NEXT: s_or_b64 exec, exec, s[0:1]
; GFX12-GISEL-NEXT: v_mov_b32_e32 v0, 0
; GFX12-GISEL-NEXT: export mrt0, v0, off, off, off done
; GFX12-GISEL-NEXT: s_endpgm
-; GFX12-GISEL-NEXT: .LBB10_6:
+; GFX12-GISEL-NEXT: .LBB8_6:
; GFX12-GISEL-NEXT: s_mov_b64 exec, 0
; GFX12-GISEL-NEXT: export mrt0, off, off, off, off done
; GFX12-GISEL-NEXT: s_endpgm
@@ -4265,7 +3772,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; VI-SDAG-NEXT: flat_load_dwordx4 v[11:14], v[0:1]
; VI-SDAG-NEXT: flat_load_dwordx4 v[15:18], v[2:3]
; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; VI-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v5
; VI-SDAG-NEXT: v_readfirstlane_b32 s6, v6
; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v5
@@ -4297,7 +3804,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; VI-SDAG-NEXT: s_nop 3
; VI-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
-; VI-SDAG-NEXT: s_cbranch_execnz .LBB11_1
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB9_1
; VI-SDAG-NEXT: ; %bb.2:
; VI-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; VI-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
@@ -4329,7 +3836,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; VI-GISEL-NEXT: flat_load_dwordx4 v[10:13], v[0:1]
; VI-GISEL-NEXT: flat_load_dwordx4 v[14:17], v[2:3]
; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; VI-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v4
; VI-GISEL-NEXT: v_readfirstlane_b32 s6, v5
; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
@@ -4363,7 +3870,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; VI-GISEL-NEXT: s_nop 3
; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; VI-GISEL-NEXT: s_cbranch_execnz .LBB11_1
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB9_1
; VI-GISEL-NEXT: ; %bb.2:
; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; VI-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
@@ -4391,7 +3898,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX9-SDAG-NEXT: global_load_dwordx4 v[7:10], v[0:1], off
; GFX9-SDAG-NEXT: global_load_dwordx4 v[15:18], v[2:3], off
; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX9-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v5
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s6, v6
; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v5
@@ -4422,7 +3929,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX9-SDAG-NEXT: s_nop 3
; GFX9-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
-; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB11_1
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB9_1
; GFX9-SDAG-NEXT: ; %bb.2:
; GFX9-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
@@ -4452,7 +3959,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX9-GISEL-NEXT: global_load_dwordx4 v[10:13], v[0:1], off offset:16
; GFX9-GISEL-NEXT: global_load_dwordx4 v[14:17], v[2:3], off
; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX9-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v4
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s6, v5
; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
@@ -4486,7 +3993,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX9-GISEL-NEXT: s_nop 3
; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB11_1
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB9_1
; GFX9-GISEL-NEXT: ; %bb.2:
; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
@@ -4514,7 +4021,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[15:18], v[2:3], off
; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
-; GFX10-32-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s2, v5
; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s3, v6
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -4541,7 +4048,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr6
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr15_vgpr16_vgpr17_vgpr18
-; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB11_1
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB9_1
; GFX10-32-SDAG-NEXT: ; %bb.2:
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s0
@@ -4574,7 +4081,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[6:9], v[0:1], off
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[10:13], v[0:1], off offset:16
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[14:17], v[2:3], off
-; GFX10-32-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v4
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s3, v5
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -4602,7 +4109,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr14_vgpr15_vgpr16_vgpr17
-; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB11_1
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB9_1
; GFX10-32-GISEL-NEXT: ; %bb.2:
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
@@ -4631,7 +4138,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[15:18], v[2:3], off
; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX10-64-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s6, v5
; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s7, v6
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -4658,7 +4165,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr6
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr15_vgpr16_vgpr17_vgpr18
-; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB11_1
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB9_1
; GFX10-64-SDAG-NEXT: ; %bb.2:
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[0:1]
@@ -4691,7 +4198,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[6:9], v[0:1], off
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[10:13], v[0:1], off offset:16
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[14:17], v[2:3], off
-; GFX10-64-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s6, v4
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s7, v5
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -4719,7 +4226,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr14_vgpr15_vgpr16_vgpr17
-; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB11_1
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB9_1
; GFX10-64-GISEL-NEXT: ; %bb.2:
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
@@ -4752,7 +4259,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX1150-SDAG-NEXT: global_load_b128 v[7:10], v[0:1], off
; GFX1150-SDAG-NEXT: global_load_b128 v[15:18], v[2:3], off
; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
-; GFX1150-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s6, v5
; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s7, v6
; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
@@ -4779,7 +4286,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr6
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr15_vgpr16_vgpr17_vgpr18
-; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB11_1
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB9_1
; GFX1150-SDAG-NEXT: ; %bb.2:
; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -4814,7 +4321,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX1150-GISEL-NEXT: global_load_b128 v[6:9], v[0:1], off
; GFX1150-GISEL-NEXT: global_load_b128 v[10:13], v[0:1], off offset:16
; GFX1150-GISEL-NEXT: global_load_b128 v[14:17], v[2:3], off
-; GFX1150-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s6, v4
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s7, v5
; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
@@ -4842,7 +4349,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr14_vgpr15_vgpr16_vgpr17
-; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB11_1
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB9_1
; GFX1150-GISEL-NEXT: ; %bb.2:
; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -4875,7 +4382,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX12-SDAG-NEXT: global_load_b128 v[15:18], v[2:3], off
; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX12-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s6, v5
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s7, v6
; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
@@ -4904,7 +4411,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr6
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr15_vgpr16_vgpr17_vgpr18
-; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB11_1
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB9_1
; GFX12-SDAG-NEXT: ; %bb.2:
; GFX12-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -4940,7 +4447,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX12-GISEL-NEXT: global_load_b128 v[6:9], v[0:1], off
; GFX12-GISEL-NEXT: global_load_b128 v[10:13], v[0:1], off offset:16
; GFX12-GISEL-NEXT: global_load_b128 v[14:17], v[2:3], off
-; GFX12-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: .LBB9_1: ; =>This Inner Loop Header: Depth=1
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s6, v4
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s7, v5
; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
@@ -4970,7 +4477,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin(<8 x i32> addrspace(4)*
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr14_vgpr15_vgpr16_vgpr17
-; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB11_1
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB9_1
; GFX12-GISEL-NEXT: ; %bb.2:
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -5015,7 +4522,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; VI-SDAG-NEXT: flat_load_dwordx4 v[8:11], v[2:3]
; VI-SDAG-NEXT: flat_load_dwordx4 v[12:15], v[0:1]
; VI-SDAG-NEXT: s_mov_b64 s[10:11], exec
-; VI-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
; VI-SDAG-NEXT: s_waitcnt vmcnt(2)
; VI-SDAG-NEXT: v_readfirstlane_b32 s0, v4
; VI-SDAG-NEXT: v_readfirstlane_b32 s1, v5
@@ -5061,7 +4568,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; VI-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
; VI-SDAG-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[16:17]
-; VI-SDAG-NEXT: s_cbranch_execnz .LBB12_1
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB10_1
; VI-SDAG-NEXT: ; %bb.2:
; VI-SDAG-NEXT: s_mov_b64 exec, s[10:11]
; VI-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
@@ -5091,7 +4598,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; VI-GISEL-NEXT: flat_load_dwordx4 v[8:11], v[0:1]
; VI-GISEL-NEXT: flat_load_dwordx4 v[12:15], v[2:3]
; VI-GISEL-NEXT: s_mov_b64 s[10:11], exec
-; VI-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
; VI-GISEL-NEXT: s_waitcnt vmcnt(2)
; VI-GISEL-NEXT: v_readfirstlane_b32 s0, v4
; VI-GISEL-NEXT: v_readfirstlane_b32 s1, v5
@@ -5139,7 +4646,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; VI-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
; VI-GISEL-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[16:17]
-; VI-GISEL-NEXT: s_cbranch_execnz .LBB12_1
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB10_1
; VI-GISEL-NEXT: ; %bb.2:
; VI-GISEL-NEXT: s_mov_b64 exec, s[10:11]
; VI-GISEL-NEXT: s_and_b64 exec, exec, s[8:9]
@@ -5165,7 +4672,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX9-SDAG-NEXT: global_load_dwordx4 v[4:7], v[16:17], off
; GFX9-SDAG-NEXT: global_load_dwordx4 v[12:15], v[0:1], off
; GFX9-SDAG-NEXT: s_mov_b64 s[10:11], exec
-; GFX9-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
; GFX9-SDAG-NEXT: s_waitcnt vmcnt(1)
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s0, v4
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s1, v5
@@ -5210,7 +4717,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX9-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
; GFX9-SDAG-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[16:17]
-; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB12_1
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB10_1
; GFX9-SDAG-NEXT: ; %bb.2:
; GFX9-SDAG-NEXT: s_mov_b64 exec, s[10:11]
; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
@@ -5238,7 +4745,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX9-GISEL-NEXT: global_load_dwordx4 v[8:11], v[0:1], off offset:16
; GFX9-GISEL-NEXT: global_load_dwordx4 v[12:15], v[2:3], off
; GFX9-GISEL-NEXT: s_mov_b64 s[10:11], exec
-; GFX9-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(2)
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s0, v4
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s1, v5
@@ -5286,7 +4793,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[16:17]
-; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB12_1
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB10_1
; GFX9-GISEL-NEXT: ; %bb.2:
; GFX9-GISEL-NEXT: s_mov_b64 exec, s[10:11]
; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[8:9]
@@ -5312,7 +4819,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[12:15], v[0:1], off
; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
-; GFX10-32-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(1)
; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s8, v4
; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s9, v5
@@ -5345,7 +4852,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s1, s1
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
-; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB12_1
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB10_1
; GFX10-32-SDAG-NEXT: ; %bb.2:
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s0
@@ -5376,7 +4883,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[4:7], v[0:1], off
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[8:11], v[0:1], off offset:16
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[12:15], v[2:3], off
-; GFX10-32-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(2)
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s8, v4
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s9, v5
@@ -5410,7 +4917,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s1, s1
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
-; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB12_1
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB10_1
; GFX10-32-GISEL-NEXT: ; %bb.2:
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
@@ -5437,7 +4944,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[12:15], v[0:1], off
; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX10-64-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(1)
; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s8, v4
; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s9, v5
@@ -5470,7 +4977,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
-; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB12_1
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB10_1
; GFX10-64-SDAG-NEXT: ; %bb.2:
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[0:1]
@@ -5501,7 +5008,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[4:7], v[0:1], off
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[8:11], v[0:1], off offset:16
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[12:15], v[2:3], off
-; GFX10-64-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(2)
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s8, v4
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s9, v5
@@ -5535,7 +5042,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
-; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB12_1
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB10_1
; GFX10-64-GISEL-NEXT: ; %bb.2:
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
@@ -5567,7 +5074,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX1150-SDAG-NEXT: global_load_b128 v[12:15], v[0:1], off
; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
; GFX1150-SDAG-NEXT: s_set_inst_prefetch_distance 0x1
-; GFX1150-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(1)
; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s8, v4
; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s9, v5
@@ -5599,7 +5106,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
-; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB12_1
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB10_1
; GFX1150-SDAG-NEXT: ; %bb.2:
; GFX1150-SDAG-NEXT: s_set_inst_prefetch_distance 0x2
; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[0:1]
@@ -5635,7 +5142,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX1150-GISEL-NEXT: global_load_b128 v[8:11], v[0:1], off offset:16
; GFX1150-GISEL-NEXT: global_load_b128 v[12:15], v[2:3], off
; GFX1150-GISEL-NEXT: s_set_inst_prefetch_distance 0x1
-; GFX1150-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(2)
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s8, v4
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s9, v5
@@ -5668,7 +5175,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
-; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB12_1
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB10_1
; GFX1150-GISEL-NEXT: ; %bb.2:
; GFX1150-GISEL-NEXT: s_set_inst_prefetch_distance 0x2
; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
@@ -5700,7 +5207,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX12-SDAG-NEXT: global_load_b128 v[12:15], v[0:1], off
; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX12-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
; GFX12-SDAG-NEXT: s_wait_loadcnt 0x1
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s8, v4
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s9, v5
@@ -5734,7 +5241,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
-; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB12_1
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB10_1
; GFX12-SDAG-NEXT: ; %bb.2:
; GFX12-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -5769,7 +5276,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX12-GISEL-NEXT: global_load_b128 v[4:7], v[0:1], off
; GFX12-GISEL-NEXT: global_load_b128 v[8:11], v[0:1], off offset:16
; GFX12-GISEL-NEXT: global_load_b128 v[12:15], v[2:3], off
-; GFX12-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: .LBB10_1: ; =>This Inner Loop Header: Depth=1
; GFX12-GISEL-NEXT: s_wait_loadcnt 0x2
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s8, v4
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s9, v5
@@ -5804,7 +5311,7 @@ define amdgpu_ps <4 x float> @test_waterfall_full_idx_multi_begin(<8 x i32> addr
; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr12_vgpr13_vgpr14_vgpr15
-; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB12_1
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB10_1
; GFX12-GISEL-NEXT: ; %bb.2:
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -5846,7 +5353,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; VI-SDAG-NEXT: s_addc_u32 s1, s1, s3
; VI-SDAG-NEXT: s_load_dwordx8 s[0:7], s[0:1], 0x0
; VI-SDAG-NEXT: s_mov_b64 s[10:11], exec
-; VI-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
; VI-SDAG-NEXT: v_readfirstlane_b32 s12, v4
; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[12:13], s12, v4
; VI-SDAG-NEXT: s_and_saveexec_b64 s[16:17], s[12:13]
@@ -5864,7 +5371,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; VI-SDAG-NEXT: s_nop 2
; VI-SDAG-NEXT: image_sample v[0:3], v[0:1], s[0:7], s[12:15] dmask:0xf
; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[16:17]
-; VI-SDAG-NEXT: s_cbranch_execnz .LBB13_1
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB11_1
; VI-SDAG-NEXT: ; %bb.2:
; VI-SDAG-NEXT: s_mov_b64 exec, s[10:11]
; VI-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
@@ -5891,7 +5398,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; VI-GISEL-NEXT: s_load_dwordx8 s[8:15], s[0:1], 0x0
; VI-GISEL-NEXT: v_mov_b32_e32 v9, s4
; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; VI-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v9
; VI-GISEL-NEXT: v_readfirstlane_b32 s4, v4
; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v9
@@ -5915,7 +5422,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; VI-GISEL-NEXT: s_nop 2
; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; VI-GISEL-NEXT: s_cbranch_execnz .LBB13_1
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB11_1
; VI-GISEL-NEXT: ; %bb.2:
; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; VI-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -5940,7 +5447,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX9-SDAG-NEXT: s_addc_u32 s11, s1, s3
; GFX9-SDAG-NEXT: s_load_dwordx8 s[0:7], s[10:11], 0x0
; GFX9-SDAG-NEXT: s_mov_b64 s[10:11], exec
-; GFX9-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s12, v4
; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[12:13], s12, v4
; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[16:17], s[12:13]
@@ -5958,7 +5465,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX9-SDAG-NEXT: s_nop 2
; GFX9-SDAG-NEXT: image_sample v[0:3], v[0:1], s[0:7], s[12:15] dmask:0xf
; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[16:17]
-; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB13_1
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB11_1
; GFX9-SDAG-NEXT: ; %bb.2:
; GFX9-SDAG-NEXT: s_mov_b64 exec, s[10:11]
; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[8:9]
@@ -5985,7 +5492,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX9-GISEL-NEXT: s_load_dwordx8 s[8:15], s[0:1], 0x0
; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, s4
; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX9-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v9
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s4, v4
; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v9
@@ -6009,7 +5516,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX9-GISEL-NEXT: s_nop 2
; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB13_1
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB11_1
; GFX9-GISEL-NEXT: ; %bb.2:
; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -6034,7 +5541,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX10-32-SDAG-NEXT: s_addc_u32 s11, s1, s3
; GFX10-32-SDAG-NEXT: s_load_dwordx8 s[0:7], s[10:11], 0x0
; GFX10-32-SDAG-NEXT: s_mov_b32 s10, exec_lo
-; GFX10-32-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s11, v4
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s11, v4
@@ -6049,7 +5556,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s10, s10
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr4
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
-; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB13_1
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB11_1
; GFX10-32-SDAG-NEXT: ; %bb.2:
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s9
@@ -6078,7 +5585,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
-; GFX10-32-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v9
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s3, v4
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -6096,7 +5603,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr9
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
-; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB13_1
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB11_1
; GFX10-32-GISEL-NEXT: ; %bb.2:
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
@@ -6122,7 +5629,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX10-64-SDAG-NEXT: s_addc_u32 s11, s1, s3
; GFX10-64-SDAG-NEXT: s_load_dwordx8 s[0:7], s[10:11], 0x0
; GFX10-64-SDAG-NEXT: s_mov_b64 s[10:11], exec
-; GFX10-64-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s14, v4
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s14, v4
@@ -6137,7 +5644,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[12:13], s[12:13]
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr4
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
-; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB13_1
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB11_1
; GFX10-64-SDAG-NEXT: ; %bb.2:
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[10:11]
@@ -6166,7 +5673,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX10-64-GISEL-NEXT: s_load_dwordx8 s[8:15], s[0:1], 0x0
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX10-64-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s4, v9
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s5, v4
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -6184,7 +5691,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr9
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
-; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB13_1
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB11_1
; GFX10-64-GISEL-NEXT: ; %bb.2:
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
@@ -6212,7 +5719,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX1150-SDAG-NEXT: global_load_b128 v[5:8], v[0:1], off
; GFX1150-SDAG-NEXT: s_addc_u32 s1, s1, s3
; GFX1150-SDAG-NEXT: s_load_b256 s[0:7], s[0:1], 0x0
-; GFX1150-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s14, v4
; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s14, v4
@@ -6227,7 +5734,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[12:13], s[12:13]
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr4
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
-; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB13_1
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB11_1
; GFX1150-SDAG-NEXT: ; %bb.2:
; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[10:11]
; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -6257,7 +5764,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
; GFX1150-GISEL-NEXT: v_add_co_ci_u32_e64 v1, null, v3, v1, vcc
; GFX1150-GISEL-NEXT: global_load_b128 v[5:8], v[0:1], off
-; GFX1150-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s4, v9
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s5, v4
; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
@@ -6275,7 +5782,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr9
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
-; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB13_1
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB11_1
; GFX1150-GISEL-NEXT: ; %bb.2:
; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -6302,7 +5809,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX12-SDAG-NEXT: s_add_nc_u64 s[0:1], s[0:1], s[2:3]
; GFX12-SDAG-NEXT: global_load_b128 v[5:8], v[0:1], off
; GFX12-SDAG-NEXT: s_load_b256 s[0:7], s[0:1], 0x0
-; GFX12-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s14, v4
; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
@@ -6319,7 +5826,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[12:13], s[12:13]
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr4
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
-; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB13_1
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB11_1
; GFX12-SDAG-NEXT: ; %bb.2:
; GFX12-SDAG-NEXT: s_mov_b64 exec, s[10:11]
; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -6350,7 +5857,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX12-GISEL-NEXT: s_load_b256 s[8:15], s[0:1], 0x0
; GFX12-GISEL-NEXT: global_load_b128 v[5:8], v[0:1], off
; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX12-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: .LBB11_1: ; =>This Inner Loop Header: Depth=1
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s4, v9
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s5, v4
; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
@@ -6370,7 +5877,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_1(<8 x i32>
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr9
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
-; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB13_1
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB11_1
; GFX12-GISEL-NEXT: ; %bb.2:
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -6415,7 +5922,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; VI-SDAG-NEXT: s_addc_u32 s1, s3, s1
; VI-SDAG-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x0
; VI-SDAG-NEXT: s_mov_b64 s[4:5], exec
-; VI-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v4
; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v4
; VI-SDAG-NEXT: s_and_saveexec_b64 s[16:17], s[8:9]
@@ -6438,7 +5945,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; VI-SDAG-NEXT: s_nop 2
; VI-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[0:3] dmask:0xf
; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[16:17]
-; VI-SDAG-NEXT: s_cbranch_execnz .LBB14_1
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB12_1
; VI-SDAG-NEXT: ; %bb.2:
; VI-SDAG-NEXT: s_mov_b64 exec, s[4:5]
; VI-SDAG-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -6469,7 +5976,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; VI-GISEL-NEXT: v_mov_b32_e32 v13, s4
; VI-GISEL-NEXT: s_mov_b64 s[4:5], exec
; VI-GISEL-NEXT: s_mov_b32 s18, 0
-; VI-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v4
; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v13
; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v4
@@ -6497,7 +6004,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; VI-GISEL-NEXT: s_nop 2
; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[0:3] dmask:0xf
; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[16:17]
-; VI-GISEL-NEXT: s_cbranch_execnz .LBB14_1
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB12_1
; VI-GISEL-NEXT: ; %bb.2:
; VI-GISEL-NEXT: s_mov_b64 exec, s[4:5]
; VI-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -6523,7 +6030,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX9-SDAG-NEXT: s_addc_u32 s1, s3, s1
; GFX9-SDAG-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x0
; GFX9-SDAG-NEXT: s_mov_b64 s[4:5], exec
-; GFX9-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v4
; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v4
; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[16:17], s[8:9]
@@ -6545,7 +6052,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX9-SDAG-NEXT: s_nop 2
; GFX9-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[0:3] dmask:0xf
; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[16:17]
-; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB14_1
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB12_1
; GFX9-SDAG-NEXT: ; %bb.2:
; GFX9-SDAG-NEXT: s_mov_b64 exec, s[4:5]
; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -6574,7 +6081,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX9-GISEL-NEXT: v_mov_b32_e32 v13, s4
; GFX9-GISEL-NEXT: s_mov_b64 s[4:5], exec
; GFX9-GISEL-NEXT: s_mov_b32 s18, 0
-; GFX9-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v4
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v13
; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v4
@@ -6602,7 +6109,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX9-GISEL-NEXT: s_nop 2
; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[0:3] dmask:0xf
; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[16:17]
-; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB14_1
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB12_1
; GFX9-GISEL-NEXT: ; %bb.2:
; GFX9-GISEL-NEXT: s_mov_b64 exec, s[4:5]
; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -6629,7 +6136,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
; GFX10-32-SDAG-NEXT: s_addc_u32 s1, s3, s1
; GFX10-32-SDAG-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x0
-; GFX10-32-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s7, v4
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s7, v4
@@ -6648,7 +6155,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s6, s6
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr4
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
-; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB14_1
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB12_1
; GFX10-32-SDAG-NEXT: ; %bb.2:
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s5
@@ -6679,7 +6186,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX10-32-GISEL-NEXT: s_clause 0x1
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
-; GFX10-32-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s7, v4
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s8, v13
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -6702,7 +6209,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr13
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
-; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB14_1
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB12_1
; GFX10-32-GISEL-NEXT: ; %bb.2:
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s4
@@ -6730,7 +6237,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
; GFX10-64-SDAG-NEXT: s_addc_u32 s1, s3, s1
; GFX10-64-SDAG-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x0
-; GFX10-64-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s10, v4
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v4
@@ -6749,7 +6256,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[8:9], s[8:9]
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr4
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
-; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB14_1
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB12_1
; GFX10-64-SDAG-NEXT: ; %bb.2:
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[4:5]
@@ -6780,7 +6287,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX10-64-GISEL-NEXT: s_clause 0x1
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
-; GFX10-64-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s10, v4
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s11, v13
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -6803,7 +6310,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr13
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
-; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB14_1
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB12_1
; GFX10-64-GISEL-NEXT: ; %bb.2:
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[4:5]
@@ -6833,7 +6340,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX1150-SDAG-NEXT: s_add_u32 s0, s2, s0
; GFX1150-SDAG-NEXT: s_addc_u32 s1, s3, s1
; GFX1150-SDAG-NEXT: s_load_b128 s[0:3], s[0:1], 0x0
-; GFX1150-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s10, v4
; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s10, v4
@@ -6852,7 +6359,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr4
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
-; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB14_1
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB12_1
; GFX1150-SDAG-NEXT: ; %bb.2:
; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[4:5]
; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -6883,7 +6390,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX1150-GISEL-NEXT: s_clause 0x1
; GFX1150-GISEL-NEXT: global_load_b128 v[5:8], v[0:1], off
; GFX1150-GISEL-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
-; GFX1150-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s10, v4
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s11, v13
; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
@@ -6906,7 +6413,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr13
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
-; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB14_1
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB12_1
; GFX1150-GISEL-NEXT: ; %bb.2:
; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[4:5]
; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -6936,7 +6443,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX12-SDAG-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
; GFX12-SDAG-NEXT: global_load_b128 v[5:8], v[0:1], off
; GFX12-SDAG-NEXT: s_load_b128 s[0:3], s[0:1], 0x0
-; GFX12-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s10, v4
; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
@@ -6957,7 +6464,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[8:9], s[8:9]
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr4
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
-; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB14_1
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB12_1
; GFX12-SDAG-NEXT: ; %bb.2:
; GFX12-SDAG-NEXT: s_mov_b64 exec, s[4:5]
; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -6990,7 +6497,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX12-GISEL-NEXT: s_clause 0x1
; GFX12-GISEL-NEXT: global_load_b128 v[5:8], v[0:1], off
; GFX12-GISEL-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
-; GFX12-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: .LBB12_1: ; =>This Inner Loop Header: Depth=1
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s10, v4
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s11, v13
; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
@@ -7015,7 +6522,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_2(<8 x i32>
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr13
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
-; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB14_1
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB12_1
; GFX12-GISEL-NEXT: ; %bb.2:
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[4:5]
; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -7060,7 +6567,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; VI-SDAG-NEXT: flat_load_dwordx4 v[9:12], v[0:1]
; VI-SDAG-NEXT: flat_load_dwordx4 v[13:16], v[2:3]
; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; VI-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v4
; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
; VI-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
@@ -7088,7 +6595,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; VI-SDAG-NEXT: s_nop 3
; VI-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
-; VI-SDAG-NEXT: s_cbranch_execnz .LBB15_1
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB13_1
; VI-SDAG-NEXT: ; %bb.2:
; VI-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; VI-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
@@ -7120,7 +6627,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; VI-GISEL-NEXT: flat_load_dwordx4 v[13:16], v[2:3]
; VI-GISEL-NEXT: v_mov_b32_e32 v17, s4
; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; VI-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v17
; VI-GISEL-NEXT: v_readfirstlane_b32 s4, v4
; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v17
@@ -7154,7 +6661,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; VI-GISEL-NEXT: s_nop 3
; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; VI-GISEL-NEXT: s_cbranch_execnz .LBB15_1
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB13_1
; VI-GISEL-NEXT: ; %bb.2:
; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; VI-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -7181,7 +6688,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX9-SDAG-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
; GFX9-SDAG-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX9-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v4
; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
@@ -7208,7 +6715,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX9-SDAG-NEXT: s_nop 3
; GFX9-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
-; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB15_1
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB13_1
; GFX9-SDAG-NEXT: ; %bb.2:
; GFX9-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
@@ -7238,7 +6745,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX9-GISEL-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
; GFX9-GISEL-NEXT: v_mov_b32_e32 v17, s4
; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX9-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v17
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s4, v4
; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v17
@@ -7272,7 +6779,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX9-GISEL-NEXT: s_nop 3
; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB15_1
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB13_1
; GFX9-GISEL-NEXT: ; %bb.2:
; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -7299,7 +6806,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
-; GFX10-32-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s2, v4
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s2, v4
@@ -7323,7 +6830,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr4
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
-; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB15_1
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB13_1
; GFX10-32-SDAG-NEXT: ; %bb.2:
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s0
@@ -7356,7 +6863,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
-; GFX10-32-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v17
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s3, v4
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -7384,7 +6891,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
-; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB15_1
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB13_1
; GFX10-32-GISEL-NEXT: ; %bb.2:
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
@@ -7412,7 +6919,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX10-64-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s6, v4
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v4
@@ -7436,7 +6943,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr4
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
-; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB15_1
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB13_1
; GFX10-64-SDAG-NEXT: ; %bb.2:
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[0:1]
@@ -7469,7 +6976,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
-; GFX10-64-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s4, v17
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s5, v4
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -7497,7 +7004,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
-; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB15_1
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB13_1
; GFX10-64-GISEL-NEXT: ; %bb.2:
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
@@ -7529,7 +7036,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX1150-SDAG-NEXT: global_load_b128 v[5:8], v[0:1], off
; GFX1150-SDAG-NEXT: global_load_b128 v[13:16], v[2:3], off
; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
-; GFX1150-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s6, v4
; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v4
@@ -7553,7 +7060,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr4
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
-; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB15_1
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB13_1
; GFX1150-SDAG-NEXT: ; %bb.2:
; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -7588,7 +7095,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX1150-GISEL-NEXT: global_load_b128 v[5:8], v[0:1], off
; GFX1150-GISEL-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
; GFX1150-GISEL-NEXT: global_load_b128 v[13:16], v[2:3], off
-; GFX1150-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s4, v17
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s5, v4
; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
@@ -7616,7 +7123,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
-; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB15_1
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB13_1
; GFX1150-GISEL-NEXT: ; %bb.2:
; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -7648,7 +7155,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX12-SDAG-NEXT: global_load_b128 v[13:16], v[2:3], off
; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX12-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s6, v4
; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
@@ -7674,7 +7181,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr4
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
-; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB15_1
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB13_1
; GFX12-SDAG-NEXT: ; %bb.2:
; GFX12-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -7710,7 +7217,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX12-GISEL-NEXT: global_load_b128 v[5:8], v[0:1], off
; GFX12-GISEL-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
; GFX12-GISEL-NEXT: global_load_b128 v[13:16], v[2:3], off
-; GFX12-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: .LBB13_1: ; =>This Inner Loop Header: Depth=1
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s4, v17
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s5, v4
; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
@@ -7740,7 +7247,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_3(<8 x i32>
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
-; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB15_1
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB13_1
; GFX12-GISEL-NEXT: ; %bb.2:
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -7786,7 +7293,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; VI-SDAG-NEXT: flat_load_dwordx4 v[9:12], v[0:1]
; VI-SDAG-NEXT: flat_load_dwordx4 v[13:16], v[2:3]
; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; VI-SDAG-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v4
; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
; VI-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
@@ -7814,7 +7321,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; VI-SDAG-NEXT: s_nop 3
; VI-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
-; VI-SDAG-NEXT: s_cbranch_execnz .LBB16_1
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB14_1
; VI-SDAG-NEXT: ; %bb.2:
; VI-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; VI-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
@@ -7846,7 +7353,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; VI-GISEL-NEXT: flat_load_dwordx4 v[13:16], v[2:3]
; VI-GISEL-NEXT: v_mov_b32_e32 v17, s4
; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; VI-GISEL-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v4
; VI-GISEL-NEXT: v_readfirstlane_b32 s4, v17
; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
@@ -7880,7 +7387,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; VI-GISEL-NEXT: s_nop 3
; VI-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; VI-GISEL-NEXT: s_cbranch_execnz .LBB16_1
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB14_1
; VI-GISEL-NEXT: ; %bb.2:
; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; VI-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -7907,7 +7414,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX9-SDAG-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
; GFX9-SDAG-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX9-SDAG-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v4
; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
@@ -7934,7 +7441,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX9-SDAG-NEXT: s_nop 3
; GFX9-SDAG-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
-; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB16_1
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB14_1
; GFX9-SDAG-NEXT: ; %bb.2:
; GFX9-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
@@ -7964,7 +7471,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX9-GISEL-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
; GFX9-GISEL-NEXT: v_mov_b32_e32 v17, s4
; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX9-GISEL-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v4
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s4, v17
; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
@@ -7998,7 +7505,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX9-GISEL-NEXT: s_nop 3
; GFX9-GISEL-NEXT: image_sample v[0:3], v[0:1], s[8:15], s[16:19] dmask:0xf
; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB16_1
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB14_1
; GFX9-GISEL-NEXT: ; %bb.2:
; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -8025,7 +7532,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX10-32-SDAG-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
-; GFX10-32-SDAG-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s2, v4
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s2, v4
@@ -8049,7 +7556,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr4
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
-; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB16_1
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB14_1
; GFX10-32-SDAG-NEXT: ; %bb.2:
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s0
@@ -8082,7 +7589,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
-; GFX10-32-GISEL-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v4
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s3, v17
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -8110,7 +7617,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr17
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
-; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB16_1
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB14_1
; GFX10-32-GISEL-NEXT: ; %bb.2:
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
@@ -8138,7 +7645,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX10-64-SDAG-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX10-64-SDAG-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s6, v4
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v4
@@ -8162,7 +7669,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr4
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
-; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB16_1
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB14_1
; GFX10-64-SDAG-NEXT: ; %bb.2:
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[0:1]
@@ -8195,7 +7702,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[5:8], v[0:1], off
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[9:12], v[0:1], off offset:16
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[13:16], v[2:3], off
-; GFX10-64-GISEL-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s4, v4
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s5, v17
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -8223,7 +7730,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr17
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
-; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB16_1
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB14_1
; GFX10-64-GISEL-NEXT: ; %bb.2:
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
@@ -8255,7 +7762,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX1150-SDAG-NEXT: global_load_b128 v[5:8], v[0:1], off
; GFX1150-SDAG-NEXT: global_load_b128 v[13:16], v[2:3], off
; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
-; GFX1150-SDAG-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s6, v4
; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s6, v4
@@ -8279,7 +7786,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr4
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
-; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB16_1
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB14_1
; GFX1150-SDAG-NEXT: ; %bb.2:
; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -8314,7 +7821,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX1150-GISEL-NEXT: global_load_b128 v[5:8], v[0:1], off
; GFX1150-GISEL-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
; GFX1150-GISEL-NEXT: global_load_b128 v[13:16], v[2:3], off
-; GFX1150-GISEL-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s4, v4
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s5, v17
; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
@@ -8342,7 +7849,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr17
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
-; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB16_1
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB14_1
; GFX1150-GISEL-NEXT: ; %bb.2:
; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -8374,7 +7881,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX12-SDAG-NEXT: global_load_b128 v[13:16], v[2:3], off
; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX12-SDAG-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s6, v4
; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
@@ -8400,7 +7907,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr4
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
-; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB16_1
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB14_1
; GFX12-SDAG-NEXT: ; %bb.2:
; GFX12-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -8436,7 +7943,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX12-GISEL-NEXT: global_load_b128 v[5:8], v[0:1], off
; GFX12-GISEL-NEXT: global_load_b128 v[9:12], v[0:1], off offset:16
; GFX12-GISEL-NEXT: global_load_b128 v[13:16], v[2:3], off
-; GFX12-GISEL-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: .LBB14_1: ; =>This Inner Loop Header: Depth=1
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s4, v4
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s5, v17
; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
@@ -8466,7 +7973,7 @@ define amdgpu_ps <4 x float> @test_waterfall_multi_begin_uniform_idx_4(<8 x i32>
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr17
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr13_vgpr14_vgpr15_vgpr16
-; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB16_1
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB14_1
; GFX12-GISEL-NEXT: ; %bb.2:
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -8514,7 +8021,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; VI-SDAG-NEXT: flat_load_dwordx4 v[7:10], v[2:3]
; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
-; VI-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v5
; VI-SDAG-NEXT: v_readfirstlane_b32 s6, v6
; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v5
@@ -8542,7 +8049,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; VI-SDAG-NEXT: s_nop 3
; VI-SDAG-NEXT: image_sample v[0:3], v[19:20], s[8:15], s[16:19] dmask:0xf
; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
-; VI-SDAG-NEXT: s_cbranch_execnz .LBB17_1
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB15_1
; VI-SDAG-NEXT: ; %bb.2:
; VI-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v11
@@ -8590,7 +8097,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
; VI-GISEL-NEXT: s_mov_b32 s20, 0
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
-; VI-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v4
; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v5
; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
@@ -8617,13 +8124,13 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; VI-GISEL-NEXT: s_nop 3
; VI-GISEL-NEXT: image_sample v[0:3], v[20:21], s[8:15], s[16:19] dmask:0xf
; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; VI-GISEL-NEXT: s_cbranch_execnz .LBB17_1
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB15_1
; VI-GISEL-NEXT: ; %bb.2:
; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; VI-GISEL-NEXT: v_mov_b32_e32 v22, s4
; VI-GISEL-NEXT: v_mov_b32_e32 v23, s5
; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; VI-GISEL-NEXT: .LBB17_3: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: .LBB15_3: ; =>This Inner Loop Header: Depth=1
; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v22
; VI-GISEL-NEXT: v_readfirstlane_b32 s4, v23
; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v22
@@ -8650,7 +8157,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; VI-GISEL-NEXT: image_sample v[4:7], v[20:21], s[8:15], s[16:19] dmask:0xf
; VI-GISEL-NEXT: ; implicit-def: $vgpr20_vgpr21
; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; VI-GISEL-NEXT: s_cbranch_execnz .LBB17_3
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB15_3
; VI-GISEL-NEXT: ; %bb.4:
; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; VI-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -8679,7 +8186,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX9-SDAG-NEXT: global_load_dwordx4 v[7:10], v[2:3], off
; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
-; GFX9-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v5
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s6, v6
; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v5
@@ -8707,7 +8214,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX9-SDAG-NEXT: s_nop 3
; GFX9-SDAG-NEXT: image_sample v[0:3], v[19:20], s[8:15], s[16:19] dmask:0xf
; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
-; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB17_1
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB15_1
; GFX9-SDAG-NEXT: ; %bb.2:
; GFX9-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v15
@@ -8753,7 +8260,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
; GFX9-GISEL-NEXT: s_mov_b32 s20, 0
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX9-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v4
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v5
; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v4
@@ -8780,13 +8287,13 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX9-GISEL-NEXT: s_nop 3
; GFX9-GISEL-NEXT: image_sample v[0:3], v[20:21], s[8:15], s[16:19] dmask:0xf
; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB17_1
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB15_1
; GFX9-GISEL-NEXT: ; %bb.2:
; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-GISEL-NEXT: v_mov_b32_e32 v22, s4
; GFX9-GISEL-NEXT: v_mov_b32_e32 v23, s5
; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX9-GISEL-NEXT: .LBB17_3: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: .LBB15_3: ; =>This Inner Loop Header: Depth=1
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v22
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s4, v23
; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v22
@@ -8813,7 +8320,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX9-GISEL-NEXT: image_sample v[4:7], v[20:21], s[8:15], s[16:19] dmask:0xf
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr20_vgpr21
; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB17_3
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB15_3
; GFX9-GISEL-NEXT: ; %bb.4:
; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[6:7]
@@ -8842,7 +8349,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
-; GFX10-32-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s2, v5
; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s3, v6
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -8865,7 +8372,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s1, s1
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr5
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr6
-; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB17_1
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB15_1
; GFX10-32-SDAG-NEXT: ; %bb.2:
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s0
@@ -8912,7 +8419,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[12:15], v[0:1], off offset:16
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[16:19], v[2:3], off
; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX10-32-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v4
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s3, v5
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -8935,7 +8442,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s1, s1
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5
-; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB17_1
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB15_1
; GFX10-32-GISEL-NEXT: ; %bb.2:
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
@@ -8943,7 +8450,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v22, s5
; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
-; GFX10-32-GISEL-NEXT: .LBB17_3: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: .LBB15_3: ; =>This Inner Loop Header: Depth=1
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v21
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s3, v22
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -8968,7 +8475,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr16_vgpr17_vgpr18_vgpr19
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr20
-; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB17_3
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB15_3
; GFX10-32-GISEL-NEXT: ; %bb.4:
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
@@ -8998,7 +8505,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
-; GFX10-64-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s6, v5
; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s7, v6
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -9021,7 +8528,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr5
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr6
-; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB17_1
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB15_1
; GFX10-64-SDAG-NEXT: ; %bb.2:
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[0:1]
@@ -9068,7 +8575,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[12:15], v[0:1], off offset:16
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[16:19], v[2:3], off
; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX10-64-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s8, v4
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s9, v5
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -9091,7 +8598,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[2:3], s[2:3]
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5
-; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB17_1
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB15_1
; GFX10-64-GISEL-NEXT: ; %bb.2:
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
@@ -9099,7 +8606,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v22, s5
; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX10-64-GISEL-NEXT: .LBB17_3: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: .LBB15_3: ; =>This Inner Loop Header: Depth=1
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s4, v21
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s5, v22
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -9124,7 +8631,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr16_vgpr17_vgpr18_vgpr19
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr20
-; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB17_3
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB15_3
; GFX10-64-GISEL-NEXT: ; %bb.4:
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
@@ -9158,7 +8665,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX1150-SDAG-NEXT: global_load_b128 v[7:10], v[2:3], off
; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
-; GFX1150-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s6, v5
; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s7, v6
; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
@@ -9182,7 +8689,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr5
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr6
-; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB17_1
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB15_1
; GFX1150-SDAG-NEXT: ; %bb.2:
; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s8, v15
@@ -9230,7 +8737,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX1150-GISEL-NEXT: global_load_b128 v[12:15], v[0:1], off offset:16
; GFX1150-GISEL-NEXT: global_load_b128 v[16:19], v[2:3], off
; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX1150-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s8, v4
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s9, v5
; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
@@ -9254,14 +8761,14 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5
-; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB17_1
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB15_1
; GFX1150-GISEL-NEXT: ; %bb.2:
; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX1150-GISEL-NEXT: v_mov_b32_e32 v21, s4
; GFX1150-GISEL-NEXT: v_mov_b32_e32 v22, s5
; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX1150-GISEL-NEXT: .LBB17_3: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: .LBB15_3: ; =>This Inner Loop Header: Depth=1
; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s4, v21
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s5, v22
@@ -9288,7 +8795,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr16_vgpr17_vgpr18_vgpr19
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr20
-; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB17_3
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB15_3
; GFX1150-GISEL-NEXT: ; %bb.4:
; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -9321,7 +8828,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX12-SDAG-NEXT: global_load_b128 v[7:10], v[2:3], off
; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX12-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s6, v5
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s7, v6
; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
@@ -9348,7 +8855,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr5
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr6
-; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB17_1
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB15_1
; GFX12-SDAG-NEXT: ; %bb.2:
; GFX12-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s8, v15
@@ -9395,7 +8902,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX12-GISEL-NEXT: global_load_b128 v[8:11], v[0:1], off
; GFX12-GISEL-NEXT: global_load_b128 v[12:15], v[0:1], off offset:16
; GFX12-GISEL-NEXT: global_load_b128 v[16:19], v[2:3], off
-; GFX12-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: .LBB15_1: ; =>This Inner Loop Header: Depth=1
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s8, v4
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s9, v5
; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
@@ -9423,14 +8930,14 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[2:3], s[2:3]
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr4
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5
-; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB17_1
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB15_1
; GFX12-GISEL-NEXT: ; %bb.2:
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-GISEL-NEXT: v_mov_b32_e32 v21, s4
; GFX12-GISEL-NEXT: v_mov_b32_e32 v22, s5
; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX12-GISEL-NEXT: .LBB17_3: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: .LBB15_3: ; =>This Inner Loop Header: Depth=1
; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s4, v21
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s5, v22
@@ -9458,7 +8965,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_begin_uniform_i
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr16_vgpr17_vgpr18_vgpr19
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr20
-; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB17_3
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB15_3
; GFX12-GISEL-NEXT: ; %bb.4:
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -9510,7 +9017,7 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; PRE-GFX10-NEXT: s_mov_b64 s[4:5], exec
; PRE-GFX10-NEXT: s_addk_i32 s32, 0x400
; PRE-GFX10-NEXT: v_writelane_b32 v40, s31, 5
-; PRE-GFX10-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
+; PRE-GFX10-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
; PRE-GFX10-NEXT: v_readfirstlane_b32 s34, v1
; PRE-GFX10-NEXT: v_cmp_eq_u32_e64 s[36:37], s34, v1
; PRE-GFX10-NEXT: s_and_saveexec_b64 s[6:7], s[36:37]
@@ -9520,7 +9027,7 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; PRE-GFX10-NEXT: ; implicit-def: $vgpr1
; PRE-GFX10-NEXT: ; implicit-def: $vgpr0
; PRE-GFX10-NEXT: s_xor_b64 exec, exec, s[6:7]
-; PRE-GFX10-NEXT: s_cbranch_execnz .LBB18_1
+; PRE-GFX10-NEXT: s_cbranch_execnz .LBB16_1
; PRE-GFX10-NEXT: ; %bb.2:
; PRE-GFX10-NEXT: s_mov_b64 exec, s[4:5]
; PRE-GFX10-NEXT: v_mov_b32_e32 v0, v2
@@ -9556,7 +9063,7 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX10-32-NEXT: s_mov_b32 s5, exec_lo
; GFX10-32-NEXT: v_writelane_b32 v40, s30, 2
; GFX10-32-NEXT: v_writelane_b32 v40, s31, 3
-; GFX10-32-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-NEXT: v_readfirstlane_b32 s34, v1
; GFX10-32-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-32-NEXT: v_cmpx_eq_u32_e32 s34, v1
@@ -9566,7 +9073,7 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX10-32-NEXT: s_andn2_wrexec_b32 s5, s5
; GFX10-32-NEXT: ; implicit-def: $vgpr1
; GFX10-32-NEXT: ; implicit-def: $vgpr0
-; GFX10-32-NEXT: s_cbranch_execnz .LBB18_1
+; GFX10-32-NEXT: s_cbranch_execnz .LBB16_1
; GFX10-32-NEXT: ; %bb.2:
; GFX10-32-NEXT: s_mov_b32 exec_lo, s4
; GFX10-32-NEXT: v_mov_b32_e32 v0, v2
@@ -9603,7 +9110,7 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX10-64-NEXT: s_mov_b64 s[6:7], exec
; GFX10-64-NEXT: v_writelane_b32 v40, s30, 4
; GFX10-64-NEXT: v_writelane_b32 v40, s31, 5
-; GFX10-64-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-NEXT: v_readfirstlane_b32 s34, v1
; GFX10-64-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-64-NEXT: v_cmpx_eq_u32_e64 s34, v1
@@ -9613,7 +9120,7 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX10-64-NEXT: s_andn2_wrexec_b64 s[6:7], s[6:7]
; GFX10-64-NEXT: ; implicit-def: $vgpr1
; GFX10-64-NEXT: ; implicit-def: $vgpr0
-; GFX10-64-NEXT: s_cbranch_execnz .LBB18_1
+; GFX10-64-NEXT: s_cbranch_execnz .LBB16_1
; GFX10-64-NEXT: ; %bb.2:
; GFX10-64-NEXT: s_mov_b64 exec, s[4:5]
; GFX10-64-NEXT: v_mov_b32_e32 v0, v2
@@ -9651,7 +9158,7 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX1150-NEXT: s_mov_b64 s[6:7], exec
; GFX1150-NEXT: v_writelane_b32 v40, s30, 4
; GFX1150-NEXT: v_writelane_b32 v40, s31, 5
-; GFX1150-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-NEXT: v_readfirstlane_b32 s0, v1
; GFX1150-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
; GFX1150-NEXT: v_cmpx_eq_u32_e64 s0, v1
@@ -9661,7 +9168,7 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX1150-NEXT: s_and_not1_wrexec_b64 s[6:7], s[6:7]
; GFX1150-NEXT: ; implicit-def: $vgpr1
; GFX1150-NEXT: ; implicit-def: $vgpr0
-; GFX1150-NEXT: s_cbranch_execnz .LBB18_1
+; GFX1150-NEXT: s_cbranch_execnz .LBB16_1
; GFX1150-NEXT: ; %bb.2:
; GFX1150-NEXT: s_mov_b64 exec, s[4:5]
; GFX1150-NEXT: s_delay_alu instid0(VALU_DEP_1)
@@ -9704,250 +9211,46 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX12-NEXT: s_mov_b64 s[6:7], exec
; GFX12-NEXT: v_writelane_b32 v40, s30, 4
; GFX12-NEXT: v_writelane_b32 v40, s31, 5
-; GFX12-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
; GFX12-NEXT: v_readfirstlane_b32 s0, v1
-; GFX12-NEXT: s_wait_alu depctr_va_sdst(0)
-; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX12-NEXT: v_cmpx_eq_u32_e64 s0, v1
-; GFX12-NEXT: s_mov_b32 s1, 0
-; GFX12-NEXT: s_wait_alu depctr_sa_sdst(0)
-; GFX12-NEXT: s_swappc_b64 s[30:31], s[0:1]
-; GFX12-NEXT: v_mov_b32_e32 v2, v0
-; GFX12-NEXT: s_and_not1_wrexec_b64 s[6:7], s[6:7]
-; GFX12-NEXT: ; implicit-def: $vgpr1
-; GFX12-NEXT: ; implicit-def: $vgpr0
-; GFX12-NEXT: s_cbranch_execnz .LBB18_1
-; GFX12-NEXT: ; %bb.2:
-; GFX12-NEXT: s_mov_b64 exec, s[4:5]
-; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX12-NEXT: v_mov_b32_e32 v0, v2
-; GFX12-NEXT: v_readlane_b32 s31, v40, 5
-; GFX12-NEXT: v_readlane_b32 s30, v40, 4
-; GFX12-NEXT: v_readlane_b32 s7, v40, 3
-; GFX12-NEXT: v_readlane_b32 s6, v40, 2
-; GFX12-NEXT: v_readlane_b32 s5, v40, 1
-; GFX12-NEXT: v_readlane_b32 s4, v40, 0
-; GFX12-NEXT: s_mov_b32 s32, s33
-; GFX12-NEXT: v_readlane_b32 s0, v40, 6
-; GFX12-NEXT: s_or_saveexec_b64 s[2:3], -1
-; GFX12-NEXT: scratch_load_b32 v40, off, s33 ; 4-byte Folded Reload
-; GFX12-NEXT: s_wait_alu depctr_sa_sdst(0)
-; GFX12-NEXT: s_mov_b64 exec, s[2:3]
-; GFX12-NEXT: s_mov_b32 s33, s0
-; GFX12-NEXT: s_wait_loadcnt 0x0
-; GFX12-NEXT: s_wait_alu depctr_sa_sdst(0)
-; GFX12-NEXT: s_setpc_b64 s[30:31]
- %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %fptr)
- %s_fptr = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %tok, i32 %fptr)
- %ext = zext i32 %s_fptr to i64
- %f = inttoptr i64 %ext to i32(i32)*
- %callres = call amdgpu_gfx i32 %f(i32 %i)
- %r3 = call i32 @llvm.amdgcn.waterfall.end.i32(token %tok, i32 %callres)
- ret i32 %r3
-}
-
-define amdgpu_cs void @ds_write_8(i8 %value, i32 %index) #1 {
-; VI-LABEL: ds_write_8:
-; VI: ; %bb.0: ; %.entry
-; VI-NEXT: v_lshlrev_b32_e32 v1, 2, v1
-; VI-NEXT: v_add_u32_e32 v1, vcc, Lds at abs32@lo, v1
-; VI-NEXT: s_mov_b32 m0, -1
-; VI-NEXT: s_mov_b64 s[0:1], exec
-; VI-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
-; VI-NEXT: v_readfirstlane_b32 s0, v1
-; VI-NEXT: v_cmp_eq_u32_e64 s[0:1], s0, v1
-; VI-NEXT: s_and_saveexec_b64 s[0:1], s[0:1]
-; VI-NEXT: ds_write_b8 v1, v0
-; VI-NEXT: ; implicit-def: $vgpr1
-; VI-NEXT: ; implicit-def: $vgpr0
-; VI-NEXT: s_xor_b64 exec, exec, s[0:1]
-; VI-NEXT: s_cbranch_execnz .LBB19_1
-; VI-NEXT: ; %bb.2:
-; VI-NEXT: s_endpgm
-;
-; GFX9-SDAG-LABEL: ds_write_8:
-; GFX9-SDAG: ; %bb.0: ; %.entry
-; GFX9-SDAG-NEXT: v_mov_b32_e32 v2, Lds at abs32@lo
-; GFX9-SDAG-NEXT: v_lshl_add_u32 v1, v1, 2, v2
-; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX9-SDAG-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
-; GFX9-SDAG-NEXT: v_readfirstlane_b32 s0, v1
-; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[0:1], s0, v1
-; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[0:1], s[0:1]
-; GFX9-SDAG-NEXT: ds_write_b8 v1, v0
-; GFX9-SDAG-NEXT: ; implicit-def: $vgpr1
-; GFX9-SDAG-NEXT: ; implicit-def: $vgpr0
-; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[0:1]
-; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB19_1
-; GFX9-SDAG-NEXT: ; %bb.2:
-; GFX9-SDAG-NEXT: s_endpgm
-;
-; GFX9-GISEL-LABEL: ds_write_8:
-; GFX9-GISEL: ; %bb.0: ; %.entry
-; GFX9-GISEL-NEXT: v_lshlrev_b32_e32 v1, 2, v1
-; GFX9-GISEL-NEXT: v_add_u32_e32 v1, Lds at abs32@lo, v1
-; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX9-GISEL-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s0, v1
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[0:1], s0, v1
-; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[0:1], s[0:1]
-; GFX9-GISEL-NEXT: ds_write_b8 v1, v0
-; GFX9-GISEL-NEXT: ; implicit-def: $vgpr1
-; GFX9-GISEL-NEXT: ; implicit-def: $vgpr0
-; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[0:1]
-; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB19_1
-; GFX9-GISEL-NEXT: ; %bb.2:
-; GFX9-GISEL-NEXT: s_endpgm
-;
-; GFX10-32-SDAG-LABEL: ds_write_8:
-; GFX10-32-SDAG: ; %bb.0: ; %.entry
-; GFX10-32-SDAG-NEXT: v_lshl_add_u32 v1, v1, 2, Lds at abs32@lo
-; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
-; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
-; GFX10-32-SDAG-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
-; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s1, v1
-; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
-; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s1, v1
-; GFX10-32-SDAG-NEXT: ds_write_b8 v1, v0
-; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s0, s0
-; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr1
-; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr0
-; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB19_1
-; GFX10-32-SDAG-NEXT: ; %bb.2:
-; GFX10-32-SDAG-NEXT: s_endpgm
-;
-; GFX10-32-GISEL-LABEL: ds_write_8:
-; GFX10-32-GISEL: ; %bb.0: ; %.entry
-; GFX10-32-GISEL-NEXT: v_lshlrev_b32_e32 v1, 2, v1
-; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
-; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
-; GFX10-32-GISEL-NEXT: v_add_nc_u32_e32 v1, Lds at abs32@lo, v1
-; GFX10-32-GISEL-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
-; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s1, v1
-; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
-; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s1, v1
-; GFX10-32-GISEL-NEXT: ds_write_b8 v1, v0
-; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s0, s0
-; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr1
-; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr0
-; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB19_1
-; GFX10-32-GISEL-NEXT: ; %bb.2:
-; GFX10-32-GISEL-NEXT: s_endpgm
-;
-; GFX10-64-SDAG-LABEL: ds_write_8:
-; GFX10-64-SDAG: ; %bb.0: ; %.entry
-; GFX10-64-SDAG-NEXT: v_lshl_add_u32 v1, v1, 2, Lds at abs32@lo
-; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
-; GFX10-64-SDAG-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
-; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s2, v1
-; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
-; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s2, v1
-; GFX10-64-SDAG-NEXT: ds_write_b8 v1, v0
-; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[0:1], s[0:1]
-; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr1
-; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr0
-; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB19_1
-; GFX10-64-SDAG-NEXT: ; %bb.2:
-; GFX10-64-SDAG-NEXT: s_endpgm
-;
-; GFX10-64-GISEL-LABEL: ds_write_8:
-; GFX10-64-GISEL: ; %bb.0: ; %.entry
-; GFX10-64-GISEL-NEXT: v_lshlrev_b32_e32 v1, 2, v1
-; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
-; GFX10-64-GISEL-NEXT: v_add_nc_u32_e32 v1, Lds at abs32@lo, v1
-; GFX10-64-GISEL-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
-; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s2, v1
-; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
-; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s2, v1
-; GFX10-64-GISEL-NEXT: ds_write_b8 v1, v0
-; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[0:1], s[0:1]
-; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr1
-; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr0
-; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB19_1
-; GFX10-64-GISEL-NEXT: ; %bb.2:
-; GFX10-64-GISEL-NEXT: s_endpgm
-;
-; GFX1150-SDAG-LABEL: ds_write_8:
-; GFX1150-SDAG: ; %bb.0: ; %.entry
-; GFX1150-SDAG-NEXT: v_lshl_add_u32 v1, v1, 2, Lds at abs32@lo
-; GFX1150-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
-; GFX1150-SDAG-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
-; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s2, v1
-; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s2, v1
-; GFX1150-SDAG-NEXT: ds_store_b8 v1, v0
-; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[0:1], s[0:1]
-; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr1
-; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr0
-; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB19_1
-; GFX1150-SDAG-NEXT: ; %bb.2:
-; GFX1150-SDAG-NEXT: s_endpgm
-;
-; GFX1150-GISEL-LABEL: ds_write_8:
-; GFX1150-GISEL: ; %bb.0: ; %.entry
-; GFX1150-GISEL-NEXT: v_lshlrev_b32_e32 v1, 2, v1
-; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
-; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX1150-GISEL-NEXT: v_add_nc_u32_e32 v1, Lds at abs32@lo, v1
-; GFX1150-GISEL-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
-; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s2, v1
-; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s2, v1
-; GFX1150-GISEL-NEXT: ds_store_b8 v1, v0
-; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[0:1], s[0:1]
-; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr1
-; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr0
-; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB19_1
-; GFX1150-GISEL-NEXT: ; %bb.2:
-; GFX1150-GISEL-NEXT: s_endpgm
-;
-; GFX12-SDAG-LABEL: ds_write_8:
-; GFX12-SDAG: ; %bb.0: ; %.entry
-; GFX12-SDAG-NEXT: v_lshl_add_u32 v1, v1, 2, Lds at abs32@lo
-; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
-; GFX12-SDAG-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
-; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
-; GFX12-SDAG-NEXT: v_readfirstlane_b32 s2, v1
-; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
-; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s2, v1
-; GFX12-SDAG-NEXT: ds_store_b8 v1, v0
-; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[0:1], s[0:1]
-; GFX12-SDAG-NEXT: ; implicit-def: $vgpr1
-; GFX12-SDAG-NEXT: ; implicit-def: $vgpr0
-; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB19_1
-; GFX12-SDAG-NEXT: ; %bb.2:
-; GFX12-SDAG-NEXT: s_endpgm
-;
-; GFX12-GISEL-LABEL: ds_write_8:
-; GFX12-GISEL: ; %bb.0: ; %.entry
-; GFX12-GISEL-NEXT: v_lshlrev_b32_e32 v1, 2, v1
-; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
-; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX12-GISEL-NEXT: v_add_nc_u32_e32 v1, Lds at abs32@lo, v1
-; GFX12-GISEL-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
-; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
-; GFX12-GISEL-NEXT: v_readfirstlane_b32 s2, v1
-; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
-; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s2, v1
-; GFX12-GISEL-NEXT: ds_store_b8 v1, v0
-; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[0:1], s[0:1]
-; GFX12-GISEL-NEXT: ; implicit-def: $vgpr1
-; GFX12-GISEL-NEXT: ; implicit-def: $vgpr0
-; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB19_1
-; GFX12-GISEL-NEXT: ; %bb.2:
-; GFX12-GISEL-NEXT: s_endpgm
-.entry:
- %gep = getelementptr [16384 x i32], ptr addrspace(3) @Lds, i32 0, i32 %index
- %0 = call token @llvm.amdgcn.waterfall.begin.p3(token poison, ptr addrspace(3) %gep)
- %1 = call ptr addrspace(3) @llvm.amdgcn.waterfall.last.use.vgpr.p3(token %0, ptr addrspace(3) %gep)
- store i8 %value, ptr addrspace(3) %1, align 1
- ret void
+; GFX12-NEXT: s_wait_alu depctr_va_sdst(0)
+; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-NEXT: v_cmpx_eq_u32_e64 s0, v1
+; GFX12-NEXT: s_mov_b32 s1, 0
+; GFX12-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-NEXT: s_swappc_b64 s[30:31], s[0:1]
+; GFX12-NEXT: v_mov_b32_e32 v2, v0
+; GFX12-NEXT: s_and_not1_wrexec_b64 s[6:7], s[6:7]
+; GFX12-NEXT: ; implicit-def: $vgpr1
+; GFX12-NEXT: ; implicit-def: $vgpr0
+; GFX12-NEXT: s_cbranch_execnz .LBB16_1
+; GFX12-NEXT: ; %bb.2:
+; GFX12-NEXT: s_mov_b64 exec, s[4:5]
+; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-NEXT: v_mov_b32_e32 v0, v2
+; GFX12-NEXT: v_readlane_b32 s31, v40, 5
+; GFX12-NEXT: v_readlane_b32 s30, v40, 4
+; GFX12-NEXT: v_readlane_b32 s7, v40, 3
+; GFX12-NEXT: v_readlane_b32 s6, v40, 2
+; GFX12-NEXT: v_readlane_b32 s5, v40, 1
+; GFX12-NEXT: v_readlane_b32 s4, v40, 0
+; GFX12-NEXT: s_mov_b32 s32, s33
+; GFX12-NEXT: v_readlane_b32 s0, v40, 6
+; GFX12-NEXT: s_or_saveexec_b64 s[2:3], -1
+; GFX12-NEXT: scratch_load_b32 v40, off, s33 ; 4-byte Folded Reload
+; GFX12-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-NEXT: s_mov_b64 exec, s[2:3]
+; GFX12-NEXT: s_mov_b32 s33, s0
+; GFX12-NEXT: s_wait_loadcnt 0x0
+; GFX12-NEXT: s_wait_alu depctr_sa_sdst(0)
+; GFX12-NEXT: s_setpc_b64 s[30:31]
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %fptr)
+ %s_fptr = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %tok, i32 %fptr)
+ %ext = zext i32 %s_fptr to i64
+ %f = inttoptr i64 %ext to i32(i32)*
+ %callres = call amdgpu_gfx i32 %f(i32 %i)
+ %r3 = call i32 @llvm.amdgcn.waterfall.end.i32(token %tok, i32 %callres)
+ ret i32 %r3
}
define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
@@ -9974,7 +9277,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; VI-SDAG-NEXT: flat_load_dwordx4 v[14:17], v[0:1]
; VI-SDAG-NEXT: flat_load_dwordx4 v[18:21], v[2:3]
; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; VI-SDAG-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v9
; VI-SDAG-NEXT: v_readfirstlane_b32 s6, v8
; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v9
@@ -10010,7 +9313,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; VI-SDAG-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
; VI-SDAG-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
-; VI-SDAG-NEXT: s_cbranch_execnz .LBB20_1
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB17_1
; VI-SDAG-NEXT: ; %bb.2:
; VI-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; VI-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
@@ -10042,7 +9345,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; VI-GISEL-NEXT: flat_load_dwordx4 v[14:17], v[0:1]
; VI-GISEL-NEXT: flat_load_dwordx4 v[18:21], v[2:3]
; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; VI-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v8
; VI-GISEL-NEXT: v_readfirstlane_b32 s6, v9
; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v8
@@ -10082,7 +9385,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; VI-GISEL-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
; VI-GISEL-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; VI-GISEL-NEXT: s_cbranch_execnz .LBB20_1
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB17_1
; VI-GISEL-NEXT: ; %bb.2:
; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; VI-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
@@ -10111,7 +9414,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX9-SDAG-NEXT: global_load_dwordx4 v[18:21], v[2:3], off
; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
-; GFX9-SDAG-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v9
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s6, v8
; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v9
@@ -10142,7 +9445,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX9-SDAG-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
; GFX9-SDAG-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
-; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB20_1
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB17_1
; GFX9-SDAG-NEXT: ; %bb.2:
; GFX9-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
@@ -10173,7 +9476,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX9-GISEL-NEXT: global_load_dwordx4 v[18:21], v[2:3], off
; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX9-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v8
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s6, v9
; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v8
@@ -10208,7 +9511,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB20_1
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB17_1
; GFX9-GISEL-NEXT: ; %bb.2:
; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
@@ -10237,7 +9540,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0)
-; GFX10-32-SDAG-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s2, v9
; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s3, v8
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -10264,7 +9567,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr8
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
-; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB20_1
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB17_1
; GFX10-32-SDAG-NEXT: ; %bb.2:
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s0
@@ -10298,7 +9601,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[14:17], v[0:1], off offset:16
; GFX10-32-GISEL-NEXT: global_load_dwordx4 v[18:21], v[2:3], off
; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX10-32-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v8
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s3, v9
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -10325,7 +9628,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr9
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
-; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB20_1
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB17_1
; GFX10-32-GISEL-NEXT: ; %bb.2:
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
@@ -10355,7 +9658,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0)
-; GFX10-64-SDAG-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s6, v9
; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s7, v8
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -10382,7 +9685,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr8
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
-; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB20_1
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB17_1
; GFX10-64-SDAG-NEXT: ; %bb.2:
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[0:1]
@@ -10416,7 +9719,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[14:17], v[0:1], off offset:16
; GFX10-64-GISEL-NEXT: global_load_dwordx4 v[18:21], v[2:3], off
; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0)
-; GFX10-64-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s6, v8
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s7, v9
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -10443,7 +9746,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr9
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
-; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB20_1
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB17_1
; GFX10-64-GISEL-NEXT: ; %bb.2:
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
@@ -10476,7 +9779,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX1150-SDAG-NEXT: global_load_b128 v[10:13], v[0:1], off
; GFX1150-SDAG-NEXT: global_load_b128 v[18:21], v[2:3], off
; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
-; GFX1150-SDAG-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s6, v9
; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s7, v8
; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
@@ -10506,7 +9809,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr8
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
-; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB20_1
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB17_1
; GFX1150-SDAG-NEXT: ; %bb.2:
; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -10541,7 +9844,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX1150-GISEL-NEXT: global_load_b128 v[10:13], v[0:1], off
; GFX1150-GISEL-NEXT: global_load_b128 v[14:17], v[0:1], off offset:16
; GFX1150-GISEL-NEXT: global_load_b128 v[18:21], v[2:3], off
-; GFX1150-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s6, v8
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s7, v9
; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
@@ -10572,7 +9875,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr9
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
-; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB20_1
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB17_1
; GFX1150-GISEL-NEXT: ; %bb.2:
; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -10605,7 +9908,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX12-SDAG-NEXT: global_load_b128 v[18:21], v[2:3], off
; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX12-SDAG-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s6, v9
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s7, v8
; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
@@ -10638,7 +9941,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr8
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
-; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB20_1
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB17_1
; GFX12-SDAG-NEXT: ; %bb.2:
; GFX12-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -10674,7 +9977,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX12-GISEL-NEXT: global_load_b128 v[10:13], v[0:1], off
; GFX12-GISEL-NEXT: global_load_b128 v[14:17], v[0:1], off offset:16
; GFX12-GISEL-NEXT: global_load_b128 v[18:21], v[2:3], off
-; GFX12-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: .LBB17_1: ; =>This Inner Loop Header: Depth=1
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s6, v8
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s7, v9
; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
@@ -10708,7 +10011,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop(
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr9
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr18_vgpr19_vgpr20_vgpr21
-; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB20_1
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB17_1
; GFX12-GISEL-NEXT: ; %bb.2:
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -10746,7 +10049,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; VI-SDAG-NEXT: v_mov_b32_e32 v10, v1
; VI-SDAG-NEXT: v_mov_b32_e32 v11, v0
; VI-SDAG-NEXT: s_mov_b64 s[6:7], exec
-; VI-SDAG-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
; VI-SDAG-NEXT: v_readfirstlane_b32 s8, v11
; VI-SDAG-NEXT: v_readfirstlane_b32 s10, v10
; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v11
@@ -10779,7 +10082,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; VI-SDAG-NEXT: ; implicit-def: $vgpr9
; VI-SDAG-NEXT: ; implicit-def: $vgpr8
; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[20:21]
-; VI-SDAG-NEXT: s_cbranch_execnz .LBB21_1
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB18_1
; VI-SDAG-NEXT: ; %bb.2:
; VI-SDAG-NEXT: s_mov_b64 exec, s[6:7]
; VI-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
@@ -10795,7 +10098,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; VI-GISEL-NEXT: v_mov_b32_e32 v9, v2
; VI-GISEL-NEXT: v_mov_b32_e32 v10, v3
; VI-GISEL-NEXT: s_mov_b64 s[6:7], exec
-; VI-GISEL-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
; VI-GISEL-NEXT: v_readfirstlane_b32 s8, v8
; VI-GISEL-NEXT: v_readfirstlane_b32 s10, v11
; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v8
@@ -10832,7 +10135,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; VI-GISEL-NEXT: ; implicit-def: $vgpr9
; VI-GISEL-NEXT: ; implicit-def: $vgpr10
; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[20:21]
-; VI-GISEL-NEXT: s_cbranch_execnz .LBB21_1
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB18_1
; VI-GISEL-NEXT: ; %bb.2:
; VI-GISEL-NEXT: s_mov_b64 exec, s[6:7]
; VI-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
@@ -10848,7 +10151,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX9-SDAG-NEXT: v_mov_b32_e32 v10, v1
; GFX9-SDAG-NEXT: v_mov_b32_e32 v11, v0
; GFX9-SDAG-NEXT: s_mov_b64 s[6:7], exec
-; GFX9-SDAG-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s8, v11
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s10, v10
; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v11
@@ -10879,7 +10182,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX9-SDAG-NEXT: ; implicit-def: $vgpr9
; GFX9-SDAG-NEXT: ; implicit-def: $vgpr8
; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[20:21]
-; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB21_1
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB18_1
; GFX9-SDAG-NEXT: ; %bb.2:
; GFX9-SDAG-NEXT: s_mov_b64 exec, s[6:7]
; GFX9-SDAG-NEXT: s_and_b64 exec, exec, s[4:5]
@@ -10895,7 +10198,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, v2
; GFX9-GISEL-NEXT: v_mov_b32_e32 v10, v3
; GFX9-GISEL-NEXT: s_mov_b64 s[6:7], exec
-; GFX9-GISEL-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s8, v8
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s10, v11
; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[8:9], s8, v8
@@ -10930,7 +10233,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr9
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr10
; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[20:21]
-; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB21_1
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB18_1
; GFX9-GISEL-NEXT: ; %bb.2:
; GFX9-GISEL-NEXT: s_mov_b64 exec, s[6:7]
; GFX9-GISEL-NEXT: s_and_b64 exec, exec, s[4:5]
@@ -10947,7 +10250,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v11, v0
; GFX10-32-SDAG-NEXT: s_mov_b32 s6, exec_lo
; GFX10-32-SDAG-NEXT: s_mov_b32 s5, exec_lo
-; GFX10-32-SDAG-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s7, v11
; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s8, v10
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -10975,7 +10278,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr10
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr9
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr8
-; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB21_1
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB18_1
; GFX10-32-SDAG-NEXT: ; %bb.2:
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s5
@@ -10993,7 +10296,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v11, v3
; GFX10-32-GISEL-NEXT: s_mov_b32 s6, exec_lo
; GFX10-32-GISEL-NEXT: s_mov_b32 s5, exec_lo
-; GFX10-32-GISEL-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s7, v8
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s8, v9
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -11021,7 +10324,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr9
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr10
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr11
-; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB21_1
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB18_1
; GFX10-32-GISEL-NEXT: ; %bb.2:
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s5
@@ -11039,7 +10342,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v11, v0
; GFX10-64-SDAG-NEXT: s_mov_b64 s[8:9], exec
; GFX10-64-SDAG-NEXT: s_mov_b64 s[6:7], exec
-; GFX10-64-SDAG-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s10, v11
; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s11, v10
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -11067,7 +10370,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr10
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr9
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr8
-; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB21_1
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB18_1
; GFX10-64-SDAG-NEXT: ; %bb.2:
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[6:7]
@@ -11085,7 +10388,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v11, v3
; GFX10-64-GISEL-NEXT: s_mov_b64 s[8:9], exec
; GFX10-64-GISEL-NEXT: s_mov_b64 s[6:7], exec
-; GFX10-64-GISEL-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s10, v8
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s11, v9
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -11113,7 +10416,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr9
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr10
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr11
-; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB21_1
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB18_1
; GFX10-64-GISEL-NEXT: ; %bb.2:
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[6:7]
@@ -11131,7 +10434,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX1150-SDAG-NEXT: v_mov_b32_e32 v11, v0
; GFX1150-SDAG-NEXT: s_mov_b64 s[8:9], exec
; GFX1150-SDAG-NEXT: s_mov_b64 s[6:7], exec
-; GFX1150-SDAG-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s10, v11
; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s11, v10
@@ -11164,7 +10467,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr10
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr9
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr8
-; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB21_1
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB18_1
; GFX1150-SDAG-NEXT: ; %bb.2:
; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[6:7]
; GFX1150-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -11182,7 +10485,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX1150-GISEL-NEXT: v_mov_b32_e32 v11, v3
; GFX1150-GISEL-NEXT: s_mov_b64 s[8:9], exec
; GFX1150-GISEL-NEXT: s_mov_b64 s[6:7], exec
-; GFX1150-GISEL-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s10, v8
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s11, v9
; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_2)
@@ -11214,7 +10517,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr9
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr10
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr11
-; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB21_1
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB18_1
; GFX1150-GISEL-NEXT: ; %bb.2:
; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[6:7]
; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -11232,7 +10535,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX12-SDAG-NEXT: v_mov_b32_e32 v11, v0
; GFX12-SDAG-NEXT: s_mov_b64 s[8:9], exec
; GFX12-SDAG-NEXT: s_mov_b64 s[6:7], exec
-; GFX12-SDAG-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_2)
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s10, v11
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s11, v10
@@ -11265,7 +10568,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr10
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr9
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr8
-; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB21_1
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB18_1
; GFX12-SDAG-NEXT: ; %bb.2:
; GFX12-SDAG-NEXT: s_mov_b64 exec, s[6:7]
; GFX12-SDAG-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -11283,7 +10586,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX12-GISEL-NEXT: v_mov_b32_e32 v11, v3
; GFX12-GISEL-NEXT: s_mov_b64 s[8:9], exec
; GFX12-GISEL-NEXT: s_mov_b64 s[6:7], exec
-; GFX12-GISEL-NEXT: .LBB21_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: .LBB18_1: ; =>This Inner Loop Header: Depth=1
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s10, v8
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s11, v9
; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
@@ -11318,7 +10621,7 @@ define amdgpu_ps {<4 x float>,<4 x float>} @test_waterfall_multi_end_1loop_rsrc_
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr9
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr10
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr11
-; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB21_1
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB18_1
; GFX12-GISEL-NEXT: ; %bb.2:
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[6:7]
; GFX12-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
@@ -11357,7 +10660,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; VI-SDAG-NEXT: v_mov_b32_e32 v6, v1
; VI-SDAG-NEXT: v_mov_b32_e32 v10, v0
; VI-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; VI-SDAG-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; VI-SDAG-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
; VI-SDAG-NEXT: v_readfirstlane_b32 s2, v10
; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v10
; VI-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
@@ -11377,7 +10680,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; VI-SDAG-NEXT: buffer_load_format_xyzw v[0:4], v5, s[4:7], 0 idxen tfe
; VI-SDAG-NEXT: ; implicit-def: $vgpr5
; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
-; VI-SDAG-NEXT: s_cbranch_execnz .LBB22_1
+; VI-SDAG-NEXT: s_cbranch_execnz .LBB19_1
; VI-SDAG-NEXT: ; %bb.2:
; VI-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; VI-SDAG-NEXT: s_waitcnt vmcnt(0)
@@ -11391,7 +10694,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; VI-GISEL-NEXT: v_mov_b32_e32 v8, v3
; VI-GISEL-NEXT: v_mov_b32_e32 v9, v4
; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; VI-GISEL-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v10
; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v10
; VI-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
@@ -11411,7 +10714,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; VI-GISEL-NEXT: buffer_load_format_xyzw v[0:4], v5, s[4:7], 0 idxen tfe
; VI-GISEL-NEXT: ; implicit-def: $vgpr5
; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; VI-GISEL-NEXT: s_cbranch_execnz .LBB22_1
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB19_1
; VI-GISEL-NEXT: ; %bb.2:
; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
@@ -11425,7 +10728,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX9-SDAG-NEXT: v_mov_b32_e32 v6, v1
; GFX9-SDAG-NEXT: v_mov_b32_e32 v10, v0
; GFX9-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX9-SDAG-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
; GFX9-SDAG-NEXT: v_readfirstlane_b32 s2, v10
; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v10
; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
@@ -11445,7 +10748,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX9-SDAG-NEXT: buffer_load_format_xyzw v[0:4], v5, s[4:7], 0 idxen tfe
; GFX9-SDAG-NEXT: ; implicit-def: $vgpr5
; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[2:3]
-; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB22_1
+; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB19_1
; GFX9-SDAG-NEXT: ; %bb.2:
; GFX9-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0)
@@ -11459,7 +10762,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX9-GISEL-NEXT: v_mov_b32_e32 v8, v3
; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, v4
; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX9-GISEL-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v10
; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v10
; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
@@ -11479,7 +10782,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX9-GISEL-NEXT: buffer_load_format_xyzw v[0:4], v5, s[4:7], 0 idxen tfe
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr5
; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB22_1
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB19_1
; GFX9-GISEL-NEXT: ; %bb.2:
; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
@@ -11494,7 +10797,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v10, v0
; GFX10-32-SDAG-NEXT: s_mov_b32 s1, exec_lo
; GFX10-32-SDAG-NEXT: s_mov_b32 s0, exec_lo
-; GFX10-32-SDAG-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-SDAG-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s2, v10
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s2, v10
@@ -11513,7 +10816,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr10
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr5
-; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB22_1
+; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB19_1
; GFX10-32-SDAG-NEXT: ; %bb.2:
; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s0
@@ -11529,7 +10832,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v9, v4
; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
-; GFX10-32-GISEL-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v10
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s2, v10
@@ -11548,7 +10851,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr10
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5
-; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB22_1
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB19_1
; GFX10-32-GISEL-NEXT: ; %bb.2:
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
@@ -11564,7 +10867,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v10, v0
; GFX10-64-SDAG-NEXT: s_mov_b64 s[2:3], exec
; GFX10-64-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX10-64-SDAG-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-SDAG-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s4, v10
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s4, v10
@@ -11583,7 +10886,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr10
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr5
-; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB22_1
+; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB19_1
; GFX10-64-SDAG-NEXT: ; %bb.2:
; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[0:1]
@@ -11599,7 +10902,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v9, v4
; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX10-64-GISEL-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s4, v10
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v10
@@ -11618,7 +10921,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr10
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5
-; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB22_1
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB19_1
; GFX10-64-GISEL-NEXT: ; %bb.2:
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
@@ -11634,7 +10937,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX1150-SDAG-NEXT: v_mov_b32_e32 v10, v0
; GFX1150-SDAG-NEXT: s_mov_b64 s[2:3], exec
; GFX1150-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX1150-SDAG-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-SDAG-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s4, v10
; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s4, v10
@@ -11653,7 +10956,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr10
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr5
-; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB22_1
+; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB19_1
; GFX1150-SDAG-NEXT: ; %bb.2:
; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0)
@@ -11668,7 +10971,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX1150-GISEL-NEXT: v_mov_b32_e32 v9, v4
; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX1150-GISEL-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s4, v10
; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v10
@@ -11687,7 +10990,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr10
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5
-; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB22_1
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB19_1
; GFX1150-GISEL-NEXT: ; %bb.2:
; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
@@ -11702,7 +11005,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX12-SDAG-NEXT: v_mov_b32_e32 v10, v0
; GFX12-SDAG-NEXT: s_mov_b64 s[2:3], exec
; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX12-SDAG-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-SDAG-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
; GFX12-SDAG-NEXT: v_readfirstlane_b32 s4, v10
; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
@@ -11722,7 +11025,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr10
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
; GFX12-SDAG-NEXT: ; implicit-def: $vgpr5
-; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB22_1
+; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB19_1
; GFX12-SDAG-NEXT: ; %bb.2:
; GFX12-SDAG-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-SDAG-NEXT: s_wait_loadcnt 0x0
@@ -11737,7 +11040,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX12-GISEL-NEXT: v_mov_b32_e32 v9, v4
; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX12-GISEL-NEXT: .LBB22_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: .LBB19_1: ; =>This Inner Loop Header: Depth=1
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s4, v10
; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
@@ -11757,7 +11060,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct(
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr10
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr6_vgpr7_vgpr8_vgpr9
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5
-; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB22_1
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB19_1
; GFX12-GISEL-NEXT: ; %bb.2:
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
@@ -11804,7 +11107,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
; VI-GISEL-NEXT: v_mov_b32_e32 v9, v4
; VI-GISEL-NEXT: v_mov_b32_e32 v10, s0
; VI-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; VI-GISEL-NEXT: .LBB23_1: ; =>This Inner Loop Header: Depth=1
+; VI-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
; VI-GISEL-NEXT: v_readfirstlane_b32 s2, v10
; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v10
; VI-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
@@ -11824,7 +11127,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
; VI-GISEL-NEXT: buffer_load_format_xyzw v[0:4], v9, s[4:7], 0 idxen tfe
; VI-GISEL-NEXT: ; implicit-def: $vgpr9
; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; VI-GISEL-NEXT: s_cbranch_execnz .LBB23_1
+; VI-GISEL-NEXT: s_cbranch_execnz .LBB20_1
; VI-GISEL-NEXT: ; %bb.2:
; VI-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; VI-GISEL-NEXT: s_waitcnt vmcnt(0)
@@ -11856,7 +11159,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
; GFX9-GISEL-NEXT: v_mov_b32_e32 v9, v4
; GFX9-GISEL-NEXT: v_mov_b32_e32 v10, s0
; GFX9-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX9-GISEL-NEXT: .LBB23_1: ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
; GFX9-GISEL-NEXT: v_readfirstlane_b32 s2, v10
; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[2:3], s2, v10
; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[2:3], s[2:3]
@@ -11876,7 +11179,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
; GFX9-GISEL-NEXT: buffer_load_format_xyzw v[0:4], v9, s[4:7], 0 idxen tfe
; GFX9-GISEL-NEXT: ; implicit-def: $vgpr9
; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[2:3]
-; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB23_1
+; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB20_1
; GFX9-GISEL-NEXT: ; %bb.2:
; GFX9-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0)
@@ -11909,7 +11212,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v10, s0
; GFX10-32-GISEL-NEXT: s_mov_b32 s1, exec_lo
; GFX10-32-GISEL-NEXT: s_mov_b32 s0, exec_lo
-; GFX10-32-GISEL-NEXT: .LBB23_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-32-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s2, v10
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s2, v10
@@ -11928,7 +11231,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr10
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr9
-; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB23_1
+; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB20_1
; GFX10-32-GISEL-NEXT: ; %bb.2:
; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s0
@@ -11945,7 +11248,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v10, s0
; GFX10-64-GISEL-NEXT: s_mov_b64 s[2:3], exec
; GFX10-64-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX10-64-GISEL-NEXT: .LBB23_1: ; =>This Inner Loop Header: Depth=1
+; GFX10-64-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s4, v10
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v10
@@ -11964,7 +11267,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr10
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr9
-; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB23_1
+; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB20_1
; GFX10-64-GISEL-NEXT: ; %bb.2:
; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[0:1]
@@ -11981,7 +11284,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
; GFX1150-GISEL-NEXT: v_mov_b32_e32 v10, s0
; GFX1150-GISEL-NEXT: s_mov_b64 s[2:3], exec
; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX1150-GISEL-NEXT: .LBB23_1: ; =>This Inner Loop Header: Depth=1
+; GFX1150-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s4, v10
; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s4, v10
@@ -12000,7 +11303,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr10
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr9
-; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB23_1
+; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB20_1
; GFX1150-GISEL-NEXT: ; %bb.2:
; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0)
@@ -12033,7 +11336,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
; GFX12-GISEL-NEXT: v_mov_b32_e32 v10, s0
; GFX12-GISEL-NEXT: s_mov_b64 s[2:3], exec
; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX12-GISEL-NEXT: .LBB23_1: ; =>This Inner Loop Header: Depth=1
+; GFX12-GISEL-NEXT: .LBB20_1: ; =>This Inner Loop Header: Depth=1
; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
; GFX12-GISEL-NEXT: v_readfirstlane_b32 s4, v10
; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
@@ -12053,7 +11356,7 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr10
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr5_vgpr6_vgpr7_vgpr8
; GFX12-GISEL-NEXT: ; implicit-def: $vgpr9
-; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB23_1
+; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB20_1
; GFX12-GISEL-NEXT: ; %bb.2:
; GFX12-GISEL-NEXT: s_mov_b64 exec, s[0:1]
; GFX12-GISEL-NEXT: s_wait_loadcnt 0x0
@@ -12075,525 +11378,6 @@ define amdgpu_ps {<4 x float>,float} @test_waterfall_multi_end_struct_uniform(
%struct.wobble = type { float, float }
-define amdgpu_cs_chain void @wf_scc_check(i32 %arg, %struct.wobble %arg1, float %arg2) {
-; VI-SDAG-LABEL: wf_scc_check:
-; VI-SDAG: ; %bb.0: ; %bb
-; VI-SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; VI-SDAG-NEXT: s_mov_b32 s0, 0
-; VI-SDAG-NEXT: s_mov_b32 s1, s0
-; VI-SDAG-NEXT: s_mov_b32 s2, s0
-; VI-SDAG-NEXT: s_mov_b32 s3, s0
-; VI-SDAG-NEXT: s_buffer_load_dword s1, s[0:3], 0x0
-; VI-SDAG-NEXT: s_mov_b64 s[10:11], exec
-; VI-SDAG-NEXT: s_waitcnt lgkmcnt(0)
-; VI-SDAG-NEXT: s_cmp_eq_u32 s1, 0
-; VI-SDAG-NEXT: s_cselect_b64 s[8:9], -1, 0
-; VI-SDAG-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
-; VI-SDAG-NEXT: v_readfirstlane_b32 s1, v8
-; VI-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s1, v8
-; VI-SDAG-NEXT: s_and_saveexec_b64 s[12:13], s[2:3]
-; VI-SDAG-NEXT: v_mov_b32_e32 v0, 0
-; VI-SDAG-NEXT: s_mov_b32 s1, s0
-; VI-SDAG-NEXT: s_mov_b32 s2, s0
-; VI-SDAG-NEXT: s_mov_b32 s3, s0
-; VI-SDAG-NEXT: s_mov_b32 s4, s0
-; VI-SDAG-NEXT: s_mov_b32 s5, s0
-; VI-SDAG-NEXT: s_mov_b32 s6, s0
-; VI-SDAG-NEXT: s_mov_b32 s7, s0
-; VI-SDAG-NEXT: v_mov_b32_e32 v1, v0
-; VI-SDAG-NEXT: image_store v0, v[0:1], s[0:7] unorm
-; VI-SDAG-NEXT: ; implicit-def: $vgpr8
-; VI-SDAG-NEXT: s_xor_b64 exec, exec, s[12:13]
-; VI-SDAG-NEXT: s_cbranch_execnz .LBB24_1
-; VI-SDAG-NEXT: ; %bb.2:
-; VI-SDAG-NEXT: s_mov_b64 exec, s[10:11]
-; VI-SDAG-NEXT: v_rcp_f32_e32 v0, v11
-; VI-SDAG-NEXT: v_cndmask_b32_e64 v1, 0, 1.0, s[8:9]
-; VI-SDAG-NEXT: v_mul_f32_e32 v0, v1, v0
-; VI-SDAG-NEXT: v_cmp_le_f32_e32 vcc, 0, v0
-; VI-SDAG-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; VI-SDAG-NEXT: s_endpgm
-;
-; VI-GISEL-LABEL: wf_scc_check:
-; VI-GISEL: ; %bb.0: ; %bb
-; VI-GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; VI-GISEL-NEXT: s_mov_b32 s8, 0
-; VI-GISEL-NEXT: s_mov_b32 s9, s8
-; VI-GISEL-NEXT: s_mov_b32 s10, s8
-; VI-GISEL-NEXT: s_mov_b32 s11, s8
-; VI-GISEL-NEXT: s_buffer_load_dword s9, s[8:11], 0x0
-; VI-GISEL-NEXT: s_mov_b32 s0, 0
-; VI-GISEL-NEXT: s_mov_b32 s1, s8
-; VI-GISEL-NEXT: s_mov_b32 s2, s8
-; VI-GISEL-NEXT: s_mov_b32 s3, s8
-; VI-GISEL-NEXT: s_waitcnt lgkmcnt(0)
-; VI-GISEL-NEXT: s_cmp_eq_u32 s9, 0
-; VI-GISEL-NEXT: s_mov_b32 s4, s8
-; VI-GISEL-NEXT: s_mov_b32 s5, s8
-; VI-GISEL-NEXT: s_mov_b32 s6, s8
-; VI-GISEL-NEXT: s_mov_b32 s7, s8
-; VI-GISEL-NEXT: s_cselect_b32 s12, 1, 0
-; VI-GISEL-NEXT: s_mov_b64 s[10:11], exec
-; VI-GISEL-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
-; VI-GISEL-NEXT: v_readfirstlane_b32 s9, v8
-; VI-GISEL-NEXT: v_cmp_eq_u32_e64 s[14:15], s9, v8
-; VI-GISEL-NEXT: s_and_saveexec_b64 s[14:15], s[14:15]
-; VI-GISEL-NEXT: s_mov_b32 s9, s8
-; VI-GISEL-NEXT: v_mov_b32_e32 v0, s8
-; VI-GISEL-NEXT: v_mov_b32_e32 v2, 0
-; VI-GISEL-NEXT: v_mov_b32_e32 v1, s9
-; VI-GISEL-NEXT: image_store v2, v[0:1], s[0:7] unorm
-; VI-GISEL-NEXT: ; implicit-def: $vgpr8
-; VI-GISEL-NEXT: s_xor_b64 exec, exec, s[14:15]
-; VI-GISEL-NEXT: s_cbranch_execnz .LBB24_1
-; VI-GISEL-NEXT: ; %bb.2:
-; VI-GISEL-NEXT: s_mov_b64 exec, s[10:11]
-; VI-GISEL-NEXT: v_rcp_f32_e32 v0, v11
-; VI-GISEL-NEXT: s_cmp_lg_u32 s12, 0
-; VI-GISEL-NEXT: s_cselect_b32 s0, 1.0, 0
-; VI-GISEL-NEXT: v_mul_f32_e32 v0, s0, v0
-; VI-GISEL-NEXT: v_cmp_le_f32_e32 vcc, 0, v0
-; VI-GISEL-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; VI-GISEL-NEXT: s_endpgm
-;
-; GFX9-SDAG-LABEL: wf_scc_check:
-; GFX9-SDAG: ; %bb.0: ; %bb
-; GFX9-SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX9-SDAG-NEXT: s_mov_b32 s0, 0
-; GFX9-SDAG-NEXT: s_mov_b32 s1, s0
-; GFX9-SDAG-NEXT: s_mov_b32 s2, s0
-; GFX9-SDAG-NEXT: s_mov_b32 s3, s0
-; GFX9-SDAG-NEXT: s_buffer_load_dword s1, s[0:3], 0x0
-; GFX9-SDAG-NEXT: s_mov_b64 s[10:11], exec
-; GFX9-SDAG-NEXT: s_waitcnt lgkmcnt(0)
-; GFX9-SDAG-NEXT: s_cmp_eq_u32 s1, 0
-; GFX9-SDAG-NEXT: s_cselect_b64 s[8:9], -1, 0
-; GFX9-SDAG-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
-; GFX9-SDAG-NEXT: v_readfirstlane_b32 s1, v8
-; GFX9-SDAG-NEXT: v_cmp_eq_u32_e64 s[2:3], s1, v8
-; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[12:13], s[2:3]
-; GFX9-SDAG-NEXT: v_mov_b32_e32 v0, 0
-; GFX9-SDAG-NEXT: s_mov_b32 s1, s0
-; GFX9-SDAG-NEXT: s_mov_b32 s2, s0
-; GFX9-SDAG-NEXT: s_mov_b32 s3, s0
-; GFX9-SDAG-NEXT: s_mov_b32 s4, s0
-; GFX9-SDAG-NEXT: s_mov_b32 s5, s0
-; GFX9-SDAG-NEXT: s_mov_b32 s6, s0
-; GFX9-SDAG-NEXT: s_mov_b32 s7, s0
-; GFX9-SDAG-NEXT: v_mov_b32_e32 v1, v0
-; GFX9-SDAG-NEXT: image_store v0, v[0:1], s[0:7] unorm
-; GFX9-SDAG-NEXT: ; implicit-def: $vgpr8
-; GFX9-SDAG-NEXT: s_xor_b64 exec, exec, s[12:13]
-; GFX9-SDAG-NEXT: s_cbranch_execnz .LBB24_1
-; GFX9-SDAG-NEXT: ; %bb.2:
-; GFX9-SDAG-NEXT: s_mov_b64 exec, s[10:11]
-; GFX9-SDAG-NEXT: v_rcp_f32_e32 v0, v11
-; GFX9-SDAG-NEXT: v_cndmask_b32_e64 v1, 0, 1.0, s[8:9]
-; GFX9-SDAG-NEXT: v_mul_f32_e32 v0, v1, v0
-; GFX9-SDAG-NEXT: v_cmp_le_f32_e32 vcc, 0, v0
-; GFX9-SDAG-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX9-SDAG-NEXT: s_endpgm
-;
-; GFX9-GISEL-LABEL: wf_scc_check:
-; GFX9-GISEL: ; %bb.0: ; %bb
-; GFX9-GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX9-GISEL-NEXT: s_mov_b32 s8, 0
-; GFX9-GISEL-NEXT: s_mov_b32 s9, s8
-; GFX9-GISEL-NEXT: s_mov_b32 s10, s8
-; GFX9-GISEL-NEXT: s_mov_b32 s11, s8
-; GFX9-GISEL-NEXT: s_buffer_load_dword s9, s[8:11], 0x0
-; GFX9-GISEL-NEXT: s_mov_b32 s0, 0
-; GFX9-GISEL-NEXT: s_mov_b32 s1, s8
-; GFX9-GISEL-NEXT: s_mov_b32 s2, s8
-; GFX9-GISEL-NEXT: s_mov_b32 s3, s8
-; GFX9-GISEL-NEXT: s_waitcnt lgkmcnt(0)
-; GFX9-GISEL-NEXT: s_cmp_eq_u32 s9, 0
-; GFX9-GISEL-NEXT: s_mov_b32 s4, s8
-; GFX9-GISEL-NEXT: s_mov_b32 s5, s8
-; GFX9-GISEL-NEXT: s_mov_b32 s6, s8
-; GFX9-GISEL-NEXT: s_mov_b32 s7, s8
-; GFX9-GISEL-NEXT: s_cselect_b32 s12, 1, 0
-; GFX9-GISEL-NEXT: s_mov_b64 s[10:11], exec
-; GFX9-GISEL-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
-; GFX9-GISEL-NEXT: v_readfirstlane_b32 s9, v8
-; GFX9-GISEL-NEXT: v_cmp_eq_u32_e64 s[14:15], s9, v8
-; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[14:15], s[14:15]
-; GFX9-GISEL-NEXT: s_mov_b32 s9, s8
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v0, s8
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v2, 0
-; GFX9-GISEL-NEXT: v_mov_b32_e32 v1, s9
-; GFX9-GISEL-NEXT: image_store v2, v[0:1], s[0:7] unorm
-; GFX9-GISEL-NEXT: ; implicit-def: $vgpr8
-; GFX9-GISEL-NEXT: s_xor_b64 exec, exec, s[14:15]
-; GFX9-GISEL-NEXT: s_cbranch_execnz .LBB24_1
-; GFX9-GISEL-NEXT: ; %bb.2:
-; GFX9-GISEL-NEXT: s_mov_b64 exec, s[10:11]
-; GFX9-GISEL-NEXT: v_rcp_f32_e32 v0, v11
-; GFX9-GISEL-NEXT: s_cmp_lg_u32 s12, 0
-; GFX9-GISEL-NEXT: s_cselect_b32 s0, 1.0, 0
-; GFX9-GISEL-NEXT: v_mul_f32_e32 v0, s0, v0
-; GFX9-GISEL-NEXT: v_cmp_le_f32_e32 vcc, 0, v0
-; GFX9-GISEL-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX9-GISEL-NEXT: s_endpgm
-;
-; GFX10-32-SDAG-LABEL: wf_scc_check:
-; GFX10-32-SDAG: ; %bb.0: ; %bb
-; GFX10-32-SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX10-32-SDAG-NEXT: s_mov_b32 s0, 0
-; GFX10-32-SDAG-NEXT: s_mov_b32 s10, exec_lo
-; GFX10-32-SDAG-NEXT: s_mov_b32 s1, s0
-; GFX10-32-SDAG-NEXT: s_mov_b32 s2, s0
-; GFX10-32-SDAG-NEXT: s_mov_b32 s3, s0
-; GFX10-32-SDAG-NEXT: s_mov_b32 s9, exec_lo
-; GFX10-32-SDAG-NEXT: s_buffer_load_dword s1, s[0:3], 0x0
-; GFX10-32-SDAG-NEXT: s_waitcnt lgkmcnt(0)
-; GFX10-32-SDAG-NEXT: s_cmp_eq_u32 s1, 0
-; GFX10-32-SDAG-NEXT: s_cselect_b32 s8, -1, 0
-; GFX10-32-SDAG-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
-; GFX10-32-SDAG-NEXT: v_readfirstlane_b32 s1, v8
-; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
-; GFX10-32-SDAG-NEXT: v_cmpx_eq_u32_e32 s1, v8
-; GFX10-32-SDAG-NEXT: v_mov_b32_e32 v0, 0
-; GFX10-32-SDAG-NEXT: s_mov_b32 s1, s0
-; GFX10-32-SDAG-NEXT: s_mov_b32 s2, s0
-; GFX10-32-SDAG-NEXT: s_mov_b32 s3, s0
-; GFX10-32-SDAG-NEXT: s_mov_b32 s4, s0
-; GFX10-32-SDAG-NEXT: s_mov_b32 s5, s0
-; GFX10-32-SDAG-NEXT: s_mov_b32 s6, s0
-; GFX10-32-SDAG-NEXT: s_mov_b32 s7, s0
-; GFX10-32-SDAG-NEXT: image_store v0, [v0, v0], s[0:7] dim:SQ_RSRC_IMG_2D unorm
-; GFX10-32-SDAG-NEXT: s_andn2_wrexec_b32 s10, s10
-; GFX10-32-SDAG-NEXT: ; implicit-def: $vgpr8
-; GFX10-32-SDAG-NEXT: s_cbranch_execnz .LBB24_1
-; GFX10-32-SDAG-NEXT: ; %bb.2:
-; GFX10-32-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX10-32-SDAG-NEXT: s_mov_b32 exec_lo, s9
-; GFX10-32-SDAG-NEXT: v_rcp_f32_e32 v0, v11
-; GFX10-32-SDAG-NEXT: v_cndmask_b32_e64 v1, 0, 1.0, s8
-; GFX10-32-SDAG-NEXT: v_mul_f32_e32 v0, v1, v0
-; GFX10-32-SDAG-NEXT: v_cmp_le_f32_e32 vcc_lo, 0, v0
-; GFX10-32-SDAG-NEXT: s_and_saveexec_b32 s0, vcc_lo
-; GFX10-32-SDAG-NEXT: s_endpgm
-;
-; GFX10-32-GISEL-LABEL: wf_scc_check:
-; GFX10-32-GISEL: ; %bb.0: ; %bb
-; GFX10-32-GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX10-32-GISEL-NEXT: s_mov_b32 s12, 0
-; GFX10-32-GISEL-NEXT: s_mov_b32 s10, exec_lo
-; GFX10-32-GISEL-NEXT: s_mov_b32 s13, s12
-; GFX10-32-GISEL-NEXT: s_mov_b32 s14, s12
-; GFX10-32-GISEL-NEXT: s_mov_b32 s15, s12
-; GFX10-32-GISEL-NEXT: s_mov_b32 s0, s12
-; GFX10-32-GISEL-NEXT: s_buffer_load_dword s7, s[12:15], 0x0
-; GFX10-32-GISEL-NEXT: s_mov_b32 s1, s12
-; GFX10-32-GISEL-NEXT: s_mov_b32 s2, s12
-; GFX10-32-GISEL-NEXT: s_mov_b32 s3, s12
-; GFX10-32-GISEL-NEXT: s_mov_b32 s4, s12
-; GFX10-32-GISEL-NEXT: s_mov_b32 s5, s12
-; GFX10-32-GISEL-NEXT: s_mov_b32 s6, s12
-; GFX10-32-GISEL-NEXT: s_mov_b32 s9, exec_lo
-; GFX10-32-GISEL-NEXT: s_waitcnt lgkmcnt(0)
-; GFX10-32-GISEL-NEXT: s_cmp_eq_u32 s7, 0
-; GFX10-32-GISEL-NEXT: s_mov_b32 s7, s12
-; GFX10-32-GISEL-NEXT: s_cselect_b32 s8, 1, 0
-; GFX10-32-GISEL-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
-; GFX10-32-GISEL-NEXT: v_readfirstlane_b32 s11, v8
-; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
-; GFX10-32-GISEL-NEXT: v_cmpx_eq_u32_e32 s11, v8
-; GFX10-32-GISEL-NEXT: v_mov_b32_e32 v0, 0
-; GFX10-32-GISEL-NEXT: image_store v0, [v0, v0], s[0:7] dim:SQ_RSRC_IMG_2D unorm
-; GFX10-32-GISEL-NEXT: s_andn2_wrexec_b32 s10, s10
-; GFX10-32-GISEL-NEXT: ; implicit-def: $vgpr8
-; GFX10-32-GISEL-NEXT: s_cbranch_execnz .LBB24_1
-; GFX10-32-GISEL-NEXT: ; %bb.2:
-; GFX10-32-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX10-32-GISEL-NEXT: s_mov_b32 exec_lo, s9
-; GFX10-32-GISEL-NEXT: v_rcp_f32_e32 v0, v11
-; GFX10-32-GISEL-NEXT: s_cmp_lg_u32 s8, 0
-; GFX10-32-GISEL-NEXT: s_cselect_b32 s0, 1.0, 0
-; GFX10-32-GISEL-NEXT: v_mul_f32_e32 v0, s0, v0
-; GFX10-32-GISEL-NEXT: v_cmp_le_f32_e32 vcc_lo, 0, v0
-; GFX10-32-GISEL-NEXT: s_and_saveexec_b32 s0, vcc_lo
-; GFX10-32-GISEL-NEXT: s_endpgm
-;
-; GFX10-64-SDAG-LABEL: wf_scc_check:
-; GFX10-64-SDAG: ; %bb.0: ; %bb
-; GFX10-64-SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX10-64-SDAG-NEXT: s_mov_b32 s0, 0
-; GFX10-64-SDAG-NEXT: s_mov_b64 s[12:13], exec
-; GFX10-64-SDAG-NEXT: s_mov_b32 s1, s0
-; GFX10-64-SDAG-NEXT: s_mov_b32 s2, s0
-; GFX10-64-SDAG-NEXT: s_mov_b32 s3, s0
-; GFX10-64-SDAG-NEXT: s_mov_b64 s[10:11], exec
-; GFX10-64-SDAG-NEXT: s_buffer_load_dword s1, s[0:3], 0x0
-; GFX10-64-SDAG-NEXT: s_waitcnt lgkmcnt(0)
-; GFX10-64-SDAG-NEXT: s_cmp_eq_u32 s1, 0
-; GFX10-64-SDAG-NEXT: s_cselect_b64 s[8:9], -1, 0
-; GFX10-64-SDAG-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
-; GFX10-64-SDAG-NEXT: v_readfirstlane_b32 s1, v8
-; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
-; GFX10-64-SDAG-NEXT: v_cmpx_eq_u32_e64 s1, v8
-; GFX10-64-SDAG-NEXT: v_mov_b32_e32 v0, 0
-; GFX10-64-SDAG-NEXT: s_mov_b32 s1, s0
-; GFX10-64-SDAG-NEXT: s_mov_b32 s2, s0
-; GFX10-64-SDAG-NEXT: s_mov_b32 s3, s0
-; GFX10-64-SDAG-NEXT: s_mov_b32 s4, s0
-; GFX10-64-SDAG-NEXT: s_mov_b32 s5, s0
-; GFX10-64-SDAG-NEXT: s_mov_b32 s6, s0
-; GFX10-64-SDAG-NEXT: s_mov_b32 s7, s0
-; GFX10-64-SDAG-NEXT: image_store v0, [v0, v0], s[0:7] dim:SQ_RSRC_IMG_2D unorm
-; GFX10-64-SDAG-NEXT: s_andn2_wrexec_b64 s[12:13], s[12:13]
-; GFX10-64-SDAG-NEXT: ; implicit-def: $vgpr8
-; GFX10-64-SDAG-NEXT: s_cbranch_execnz .LBB24_1
-; GFX10-64-SDAG-NEXT: ; %bb.2:
-; GFX10-64-SDAG-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX10-64-SDAG-NEXT: s_mov_b64 exec, s[10:11]
-; GFX10-64-SDAG-NEXT: v_rcp_f32_e32 v0, v11
-; GFX10-64-SDAG-NEXT: v_cndmask_b32_e64 v1, 0, 1.0, s[8:9]
-; GFX10-64-SDAG-NEXT: v_mul_f32_e32 v0, v1, v0
-; GFX10-64-SDAG-NEXT: v_cmp_le_f32_e32 vcc, 0, v0
-; GFX10-64-SDAG-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX10-64-SDAG-NEXT: s_endpgm
-;
-; GFX10-64-GISEL-LABEL: wf_scc_check:
-; GFX10-64-GISEL: ; %bb.0: ; %bb
-; GFX10-64-GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX10-64-GISEL-NEXT: s_mov_b32 s8, 0
-; GFX10-64-GISEL-NEXT: s_mov_b32 s9, s8
-; GFX10-64-GISEL-NEXT: s_mov_b32 s10, s8
-; GFX10-64-GISEL-NEXT: s_mov_b32 s11, s8
-; GFX10-64-GISEL-NEXT: s_mov_b32 s0, s8
-; GFX10-64-GISEL-NEXT: s_buffer_load_dword s7, s[8:11], 0x0
-; GFX10-64-GISEL-NEXT: s_mov_b32 s1, s8
-; GFX10-64-GISEL-NEXT: s_mov_b32 s2, s8
-; GFX10-64-GISEL-NEXT: s_mov_b32 s3, s8
-; GFX10-64-GISEL-NEXT: s_mov_b32 s4, s8
-; GFX10-64-GISEL-NEXT: s_mov_b32 s5, s8
-; GFX10-64-GISEL-NEXT: s_mov_b32 s6, s8
-; GFX10-64-GISEL-NEXT: s_mov_b64 s[10:11], exec
-; GFX10-64-GISEL-NEXT: s_waitcnt lgkmcnt(0)
-; GFX10-64-GISEL-NEXT: s_cmp_eq_u32 s7, 0
-; GFX10-64-GISEL-NEXT: s_mov_b32 s7, s8
-; GFX10-64-GISEL-NEXT: s_cselect_b32 s12, 1, 0
-; GFX10-64-GISEL-NEXT: s_mov_b64 s[8:9], exec
-; GFX10-64-GISEL-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
-; GFX10-64-GISEL-NEXT: v_readfirstlane_b32 s13, v8
-; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
-; GFX10-64-GISEL-NEXT: v_cmpx_eq_u32_e64 s13, v8
-; GFX10-64-GISEL-NEXT: v_mov_b32_e32 v0, 0
-; GFX10-64-GISEL-NEXT: image_store v0, [v0, v0], s[0:7] dim:SQ_RSRC_IMG_2D unorm
-; GFX10-64-GISEL-NEXT: s_andn2_wrexec_b64 s[10:11], s[10:11]
-; GFX10-64-GISEL-NEXT: ; implicit-def: $vgpr8
-; GFX10-64-GISEL-NEXT: s_cbranch_execnz .LBB24_1
-; GFX10-64-GISEL-NEXT: ; %bb.2:
-; GFX10-64-GISEL-NEXT: s_waitcnt_depctr depctr_vm_vsrc(0)
-; GFX10-64-GISEL-NEXT: s_mov_b64 exec, s[8:9]
-; GFX10-64-GISEL-NEXT: v_rcp_f32_e32 v0, v11
-; GFX10-64-GISEL-NEXT: s_cmp_lg_u32 s12, 0
-; GFX10-64-GISEL-NEXT: s_cselect_b32 s0, 1.0, 0
-; GFX10-64-GISEL-NEXT: v_mul_f32_e32 v0, s0, v0
-; GFX10-64-GISEL-NEXT: v_cmp_le_f32_e32 vcc, 0, v0
-; GFX10-64-GISEL-NEXT: s_and_saveexec_b64 s[0:1], vcc
-; GFX10-64-GISEL-NEXT: s_endpgm
-;
-; GFX1150-SDAG-LABEL: wf_scc_check:
-; GFX1150-SDAG: ; %bb.0: ; %bb
-; GFX1150-SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX1150-SDAG-NEXT: s_mov_b32 s0, 0
-; GFX1150-SDAG-NEXT: s_mov_b64 s[12:13], exec
-; GFX1150-SDAG-NEXT: s_mov_b32 s1, s0
-; GFX1150-SDAG-NEXT: s_mov_b32 s2, s0
-; GFX1150-SDAG-NEXT: s_mov_b32 s3, s0
-; GFX1150-SDAG-NEXT: s_mov_b64 s[10:11], exec
-; GFX1150-SDAG-NEXT: s_buffer_load_b32 s1, s[0:3], 0x0
-; GFX1150-SDAG-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1150-SDAG-NEXT: s_cmp_eq_u32 s1, 0
-; GFX1150-SDAG-NEXT: s_cselect_b64 s[8:9], -1, 0
-; GFX1150-SDAG-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
-; GFX1150-SDAG-NEXT: v_readfirstlane_b32 s1, v8
-; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX1150-SDAG-NEXT: v_cmpx_eq_u32_e64 s1, v8
-; GFX1150-SDAG-NEXT: v_mov_b32_e32 v0, 0
-; GFX1150-SDAG-NEXT: s_mov_b32 s1, s0
-; GFX1150-SDAG-NEXT: s_mov_b32 s2, s0
-; GFX1150-SDAG-NEXT: s_mov_b32 s3, s0
-; GFX1150-SDAG-NEXT: s_mov_b32 s4, s0
-; GFX1150-SDAG-NEXT: s_mov_b32 s5, s0
-; GFX1150-SDAG-NEXT: s_mov_b32 s6, s0
-; GFX1150-SDAG-NEXT: s_mov_b32 s7, s0
-; GFX1150-SDAG-NEXT: image_store v0, [v0, v0], s[0:7] dim:SQ_RSRC_IMG_2D unorm
-; GFX1150-SDAG-NEXT: s_and_not1_wrexec_b64 s[12:13], s[12:13]
-; GFX1150-SDAG-NEXT: ; implicit-def: $vgpr8
-; GFX1150-SDAG-NEXT: s_cbranch_execnz .LBB24_1
-; GFX1150-SDAG-NEXT: ; %bb.2:
-; GFX1150-SDAG-NEXT: s_mov_b64 exec, s[10:11]
-; GFX1150-SDAG-NEXT: v_rcp_f32_e32 v0, v11
-; GFX1150-SDAG-NEXT: v_cndmask_b32_e64 v1, 0, 1.0, s[8:9]
-; GFX1150-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX1150-SDAG-NEXT: s_delay_alu instid0(TRANS32_DEP_1) | instid1(VALU_DEP_1)
-; GFX1150-SDAG-NEXT: v_mul_f32_e32 v0, v1, v0
-; GFX1150-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX1150-SDAG-NEXT: v_cmpx_le_f32_e32 0, v0
-; GFX1150-SDAG-NEXT: s_endpgm
-;
-; GFX1150-GISEL-LABEL: wf_scc_check:
-; GFX1150-GISEL: ; %bb.0: ; %bb
-; GFX1150-GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX1150-GISEL-NEXT: s_mov_b32 s8, 0
-; GFX1150-GISEL-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
-; GFX1150-GISEL-NEXT: s_mov_b32 s9, s8
-; GFX1150-GISEL-NEXT: s_mov_b32 s10, s8
-; GFX1150-GISEL-NEXT: s_mov_b32 s11, s8
-; GFX1150-GISEL-NEXT: s_mov_b32 s0, s8
-; GFX1150-GISEL-NEXT: s_buffer_load_b32 s7, s[8:11], 0x0
-; GFX1150-GISEL-NEXT: s_mov_b32 s1, s8
-; GFX1150-GISEL-NEXT: s_mov_b32 s2, s8
-; GFX1150-GISEL-NEXT: s_mov_b32 s3, s8
-; GFX1150-GISEL-NEXT: s_mov_b32 s4, s8
-; GFX1150-GISEL-NEXT: s_mov_b32 s5, s8
-; GFX1150-GISEL-NEXT: s_mov_b32 s6, s8
-; GFX1150-GISEL-NEXT: s_mov_b64 s[10:11], exec
-; GFX1150-GISEL-NEXT: s_waitcnt lgkmcnt(0)
-; GFX1150-GISEL-NEXT: s_cmp_eq_u32 s7, 0
-; GFX1150-GISEL-NEXT: s_mov_b32 s7, s8
-; GFX1150-GISEL-NEXT: s_cselect_b32 s12, 1, 0
-; GFX1150-GISEL-NEXT: s_mov_b64 s[8:9], exec
-; GFX1150-GISEL-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
-; GFX1150-GISEL-NEXT: v_readfirstlane_b32 s13, v8
-; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX1150-GISEL-NEXT: v_cmpx_eq_u32_e64 s13, v8
-; GFX1150-GISEL-NEXT: v_mov_b32_e32 v0, 0
-; GFX1150-GISEL-NEXT: image_store v0, [v0, v0], s[0:7] dim:SQ_RSRC_IMG_2D unorm
-; GFX1150-GISEL-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
-; GFX1150-GISEL-NEXT: ; implicit-def: $vgpr8
-; GFX1150-GISEL-NEXT: s_cbranch_execnz .LBB24_1
-; GFX1150-GISEL-NEXT: ; %bb.2:
-; GFX1150-GISEL-NEXT: s_mov_b64 exec, s[8:9]
-; GFX1150-GISEL-NEXT: v_rcp_f32_e32 v0, v11
-; GFX1150-GISEL-NEXT: s_cmp_lg_u32 s12, 0
-; GFX1150-GISEL-NEXT: s_cselect_b32 s0, 1.0, 0
-; GFX1150-GISEL-NEXT: s_delay_alu instid0(TRANS32_DEP_1) | instid1(SALU_CYCLE_1)
-; GFX1150-GISEL-NEXT: v_mul_f32_e32 v0, s0, v0
-; GFX1150-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX1150-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX1150-GISEL-NEXT: v_cmpx_le_f32_e32 0, v0
-; GFX1150-GISEL-NEXT: s_endpgm
-;
-; GFX12-SDAG-LABEL: wf_scc_check:
-; GFX12-SDAG: ; %bb.0: ; %bb
-; GFX12-SDAG-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX12-SDAG-NEXT: s_wait_expcnt 0x0
-; GFX12-SDAG-NEXT: s_wait_samplecnt 0x0
-; GFX12-SDAG-NEXT: s_wait_bvhcnt 0x0
-; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
-; GFX12-SDAG-NEXT: s_mov_b32 s0, 0
-; GFX12-SDAG-NEXT: s_mov_b64 s[10:11], exec
-; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
-; GFX12-SDAG-NEXT: s_mov_b32 s1, s0
-; GFX12-SDAG-NEXT: s_mov_b32 s2, s0
-; GFX12-SDAG-NEXT: s_mov_b32 s3, s0
-; GFX12-SDAG-NEXT: s_mov_b64 s[8:9], exec
-; GFX12-SDAG-NEXT: s_buffer_load_b32 s12, s[0:3], 0x0
-; GFX12-SDAG-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
-; GFX12-SDAG-NEXT: v_readfirstlane_b32 s1, v8
-; GFX12-SDAG-NEXT: s_wait_alu depctr_va_sdst(0)
-; GFX12-SDAG-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX12-SDAG-NEXT: v_cmpx_eq_u32_e64 s1, v8
-; GFX12-SDAG-NEXT: v_mov_b32_e32 v0, 0
-; GFX12-SDAG-NEXT: s_mov_b32 s1, s0
-; GFX12-SDAG-NEXT: s_mov_b32 s2, s0
-; GFX12-SDAG-NEXT: s_mov_b32 s3, s0
-; GFX12-SDAG-NEXT: s_mov_b32 s4, s0
-; GFX12-SDAG-NEXT: s_mov_b32 s5, s0
-; GFX12-SDAG-NEXT: s_mov_b32 s6, s0
-; GFX12-SDAG-NEXT: s_mov_b32 s7, s0
-; GFX12-SDAG-NEXT: image_store v0, [v0, v0], s[0:7] dim:SQ_RSRC_IMG_2D
-; GFX12-SDAG-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
-; GFX12-SDAG-NEXT: ; implicit-def: $vgpr8
-; GFX12-SDAG-NEXT: s_cbranch_execnz .LBB24_1
-; GFX12-SDAG-NEXT: ; %bb.2:
-; GFX12-SDAG-NEXT: s_mov_b64 exec, s[8:9]
-; GFX12-SDAG-NEXT: v_rcp_f32_e32 v0, v11
-; GFX12-SDAG-NEXT: s_wait_kmcnt 0x0
-; GFX12-SDAG-NEXT: s_cmp_eq_u32 s12, 0
-; GFX12-SDAG-NEXT: s_cselect_b32 s0, 1.0, 0
-; GFX12-SDAG-NEXT: s_wait_alu depctr_sa_sdst(0)
-; GFX12-SDAG-NEXT: s_delay_alu instid0(TRANS32_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
-; GFX12-SDAG-NEXT: v_mul_f32_e32 v0, s0, v0
-; GFX12-SDAG-NEXT: s_mov_b64 s[0:1], exec
-; GFX12-SDAG-NEXT: v_cmpx_le_f32_e32 0, v0
-; GFX12-SDAG-NEXT: s_endpgm
-;
-; GFX12-GISEL-LABEL: wf_scc_check:
-; GFX12-GISEL: ; %bb.0: ; %bb
-; GFX12-GISEL-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX12-GISEL-NEXT: s_wait_expcnt 0x0
-; GFX12-GISEL-NEXT: s_wait_samplecnt 0x0
-; GFX12-GISEL-NEXT: s_wait_bvhcnt 0x0
-; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
-; GFX12-GISEL-NEXT: s_mov_b32 s8, 0
-; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
-; GFX12-GISEL-NEXT: s_mov_b32 s9, s8
-; GFX12-GISEL-NEXT: s_mov_b32 s10, s8
-; GFX12-GISEL-NEXT: s_mov_b32 s11, s8
-; GFX12-GISEL-NEXT: s_mov_b32 s0, s8
-; GFX12-GISEL-NEXT: s_buffer_load_b32 s7, s[8:11], 0x0
-; GFX12-GISEL-NEXT: s_mov_b32 s1, s8
-; GFX12-GISEL-NEXT: s_mov_b32 s2, s8
-; GFX12-GISEL-NEXT: s_mov_b32 s3, s8
-; GFX12-GISEL-NEXT: s_mov_b32 s4, s8
-; GFX12-GISEL-NEXT: s_mov_b32 s5, s8
-; GFX12-GISEL-NEXT: s_mov_b32 s6, s8
-; GFX12-GISEL-NEXT: s_mov_b64 s[10:11], exec
-; GFX12-GISEL-NEXT: s_wait_kmcnt 0x0
-; GFX12-GISEL-NEXT: s_cmp_eq_u32 s7, 0
-; GFX12-GISEL-NEXT: s_mov_b32 s7, s8
-; GFX12-GISEL-NEXT: s_cselect_b32 s12, 1, 0
-; GFX12-GISEL-NEXT: s_mov_b64 s[8:9], exec
-; GFX12-GISEL-NEXT: .LBB24_1: ; =>This Inner Loop Header: Depth=1
-; GFX12-GISEL-NEXT: v_readfirstlane_b32 s13, v8
-; GFX12-GISEL-NEXT: s_wait_alu depctr_va_sdst(0)
-; GFX12-GISEL-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX12-GISEL-NEXT: v_cmpx_eq_u32_e64 s13, v8
-; GFX12-GISEL-NEXT: v_mov_b32_e32 v0, 0
-; GFX12-GISEL-NEXT: image_store v0, [v0, v0], s[0:7] dim:SQ_RSRC_IMG_2D
-; GFX12-GISEL-NEXT: s_and_not1_wrexec_b64 s[10:11], s[10:11]
-; GFX12-GISEL-NEXT: ; implicit-def: $vgpr8
-; GFX12-GISEL-NEXT: s_cbranch_execnz .LBB24_1
-; GFX12-GISEL-NEXT: ; %bb.2:
-; GFX12-GISEL-NEXT: s_mov_b64 exec, s[8:9]
-; GFX12-GISEL-NEXT: v_rcp_f32_e32 v0, v11
-; GFX12-GISEL-NEXT: s_cmp_lg_u32 s12, 0
-; GFX12-GISEL-NEXT: s_cselect_b32 s0, 1.0, 0
-; GFX12-GISEL-NEXT: s_wait_alu depctr_sa_sdst(0)
-; GFX12-GISEL-NEXT: s_delay_alu instid0(TRANS32_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
-; GFX12-GISEL-NEXT: v_mul_f32_e32 v0, s0, v0
-; GFX12-GISEL-NEXT: s_mov_b64 s[0:1], exec
-; GFX12-GISEL-NEXT: v_cmpx_le_f32_e32 0, v0
-; GFX12-GISEL-NEXT: s_endpgm
-bb:
- %call = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> zeroinitializer, i32 0, i32 0)
- %icmp = icmp eq i32 %call, 0
- %call3 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %arg)
- %call4 = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %call3, i32 %arg)
- %call5 = call <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(token %call3, <8 x i32> zeroinitializer)
- call void @llvm.amdgcn.image.store.2d.f32.i32.v8i32(float 0.000000e+00, i32 0, i32 0, i32 0, <8 x i32> %call5, i32 0, i32 0)
- %select = select i1 %icmp, float 1.000000e+00, float 0.000000e+00
- %fdiv = fdiv afn float %select, %arg2
- %fcmp = fcmp ult float %fdiv, 0.000000e+00
- br i1 %fcmp, label %bb6, label %bb7
-
-bb6: ; preds = %bb7, %bb
- ret void
-
-bb7: ; preds = %bb
- %extractvalue = extractvalue %struct.wobble %arg1, 0
- %fsub = fsub float 0.000000e+00, %extractvalue
- %fcmp8 = fcmp olt float %fsub, 0.000000e+00
- %or = or i1 false, %fcmp8
- br label %bb6
-}
-
declare token @llvm.amdgcn.waterfall.begin.i32(token, i32) #6
declare token @llvm.amdgcn.waterfall.begin.v2i32(token, <2 x i32>) #6
declare token @llvm.amdgcn.waterfall.begin.v4i32(token, <4 x i32>) #6
@@ -12607,8 +11391,6 @@ declare i16 @llvm.amdgcn.waterfall.end.i16(token, i16) #6
declare i32 @llvm.amdgcn.waterfall.end.i32(token, i32) #6
declare <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token, <4 x float>) #6
declare <8 x i32> @llvm.amdgcn.waterfall.end.v8i32(token, <8 x i32>) #6
-declare <8 x i32> @llvm.amdgcn.waterfall.last.use.v8i32(token, <8 x i32>) #6
-declare ptr addrspace(3) @llvm.amdgcn.waterfall.last.use.vgpr.p3(token, ptr addrspace(3))
declare i32 @llvm.amdgcn.readlane(i32, i32) #0
declare <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32, float, <8 x i32>, <4 x i32>, i1, i32, i32)
declare <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(i32, float, float, <8 x i32>, <4 x i32>, i1, i32, i32)
>From d1b393b2b901dc78452dee5d0737eff71d00a55f Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Thu, 28 May 2026 14:23:00 +0100
Subject: [PATCH 18/35] fix test after rebase
---
.../CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll | 32 +++++++++----------
1 file changed, 15 insertions(+), 17 deletions(-)
diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
index 80b5f35f0c7ad..8b28c1266d041 100644
--- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
+++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.waterfall.ll
@@ -9009,14 +9009,14 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; PRE-GFX10-NEXT: buffer_store_dword v40, off, s[0:3], s33 ; 4-byte Folded Spill
; PRE-GFX10-NEXT: s_mov_b64 exec, s[36:37]
; PRE-GFX10-NEXT: v_writelane_b32 v40, s34, 6
+; PRE-GFX10-NEXT: s_addk_i32 s32, 0x400
; PRE-GFX10-NEXT: v_writelane_b32 v40, s4, 0
; PRE-GFX10-NEXT: v_writelane_b32 v40, s5, 1
; PRE-GFX10-NEXT: v_writelane_b32 v40, s6, 2
; PRE-GFX10-NEXT: v_writelane_b32 v40, s7, 3
; PRE-GFX10-NEXT: v_writelane_b32 v40, s30, 4
-; PRE-GFX10-NEXT: s_mov_b64 s[4:5], exec
-; PRE-GFX10-NEXT: s_addk_i32 s32, 0x400
; PRE-GFX10-NEXT: v_writelane_b32 v40, s31, 5
+; PRE-GFX10-NEXT: s_mov_b64 s[4:5], exec
; PRE-GFX10-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
; PRE-GFX10-NEXT: v_readfirstlane_b32 s34, v1
; PRE-GFX10-NEXT: v_cmp_eq_u32_e64 s[36:37], s34, v1
@@ -9030,9 +9030,9 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; PRE-GFX10-NEXT: s_cbranch_execnz .LBB16_1
; PRE-GFX10-NEXT: ; %bb.2:
; PRE-GFX10-NEXT: s_mov_b64 exec, s[4:5]
+; PRE-GFX10-NEXT: v_readlane_b32 s30, v40, 4
; PRE-GFX10-NEXT: v_mov_b32_e32 v0, v2
; PRE-GFX10-NEXT: v_readlane_b32 s31, v40, 5
-; PRE-GFX10-NEXT: v_readlane_b32 s30, v40, 4
; PRE-GFX10-NEXT: v_readlane_b32 s7, v40, 3
; PRE-GFX10-NEXT: v_readlane_b32 s6, v40, 2
; PRE-GFX10-NEXT: v_readlane_b32 s5, v40, 1
@@ -9058,11 +9058,11 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX10-32-NEXT: v_writelane_b32 v40, s34, 4
; GFX10-32-NEXT: s_addk_i32 s32, 0x200
; GFX10-32-NEXT: v_writelane_b32 v40, s4, 0
-; GFX10-32-NEXT: s_mov_b32 s4, exec_lo
; GFX10-32-NEXT: v_writelane_b32 v40, s5, 1
-; GFX10-32-NEXT: s_mov_b32 s5, exec_lo
; GFX10-32-NEXT: v_writelane_b32 v40, s30, 2
; GFX10-32-NEXT: v_writelane_b32 v40, s31, 3
+; GFX10-32-NEXT: s_mov_b32 s5, exec_lo
+; GFX10-32-NEXT: s_mov_b32 s4, exec_lo
; GFX10-32-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
; GFX10-32-NEXT: v_readfirstlane_b32 s34, v1
; GFX10-32-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -9076,9 +9076,9 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX10-32-NEXT: s_cbranch_execnz .LBB16_1
; GFX10-32-NEXT: ; %bb.2:
; GFX10-32-NEXT: s_mov_b32 exec_lo, s4
+; GFX10-32-NEXT: v_readlane_b32 s30, v40, 2
; GFX10-32-NEXT: v_mov_b32_e32 v0, v2
; GFX10-32-NEXT: v_readlane_b32 s31, v40, 3
-; GFX10-32-NEXT: v_readlane_b32 s30, v40, 2
; GFX10-32-NEXT: v_readlane_b32 s5, v40, 1
; GFX10-32-NEXT: v_readlane_b32 s4, v40, 0
; GFX10-32-NEXT: s_mov_b32 s32, s33
@@ -9104,12 +9104,12 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX10-64-NEXT: s_addk_i32 s32, 0x400
; GFX10-64-NEXT: v_writelane_b32 v40, s4, 0
; GFX10-64-NEXT: v_writelane_b32 v40, s5, 1
-; GFX10-64-NEXT: s_mov_b64 s[4:5], exec
; GFX10-64-NEXT: v_writelane_b32 v40, s6, 2
; GFX10-64-NEXT: v_writelane_b32 v40, s7, 3
-; GFX10-64-NEXT: s_mov_b64 s[6:7], exec
; GFX10-64-NEXT: v_writelane_b32 v40, s30, 4
; GFX10-64-NEXT: v_writelane_b32 v40, s31, 5
+; GFX10-64-NEXT: s_mov_b64 s[6:7], exec
+; GFX10-64-NEXT: s_mov_b64 s[4:5], exec
; GFX10-64-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
; GFX10-64-NEXT: v_readfirstlane_b32 s34, v1
; GFX10-64-NEXT: s_waitcnt_depctr depctr_sa_sdst(0)
@@ -9123,9 +9123,9 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX10-64-NEXT: s_cbranch_execnz .LBB16_1
; GFX10-64-NEXT: ; %bb.2:
; GFX10-64-NEXT: s_mov_b64 exec, s[4:5]
+; GFX10-64-NEXT: v_readlane_b32 s30, v40, 4
; GFX10-64-NEXT: v_mov_b32_e32 v0, v2
; GFX10-64-NEXT: v_readlane_b32 s31, v40, 5
-; GFX10-64-NEXT: v_readlane_b32 s30, v40, 4
; GFX10-64-NEXT: v_readlane_b32 s7, v40, 3
; GFX10-64-NEXT: v_readlane_b32 s6, v40, 2
; GFX10-64-NEXT: v_readlane_b32 s5, v40, 1
@@ -9152,12 +9152,12 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX1150-NEXT: s_add_i32 s32, s32, 16
; GFX1150-NEXT: v_writelane_b32 v40, s4, 0
; GFX1150-NEXT: v_writelane_b32 v40, s5, 1
-; GFX1150-NEXT: s_mov_b64 s[4:5], exec
; GFX1150-NEXT: v_writelane_b32 v40, s6, 2
; GFX1150-NEXT: v_writelane_b32 v40, s7, 3
-; GFX1150-NEXT: s_mov_b64 s[6:7], exec
; GFX1150-NEXT: v_writelane_b32 v40, s30, 4
; GFX1150-NEXT: v_writelane_b32 v40, s31, 5
+; GFX1150-NEXT: s_mov_b64 s[6:7], exec
+; GFX1150-NEXT: s_mov_b64 s[4:5], exec
; GFX1150-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
; GFX1150-NEXT: v_readfirstlane_b32 s0, v1
; GFX1150-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
@@ -9171,10 +9171,9 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX1150-NEXT: s_cbranch_execnz .LBB16_1
; GFX1150-NEXT: ; %bb.2:
; GFX1150-NEXT: s_mov_b64 exec, s[4:5]
-; GFX1150-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX1150-NEXT: v_readlane_b32 s30, v40, 4
; GFX1150-NEXT: v_mov_b32_e32 v0, v2
; GFX1150-NEXT: v_readlane_b32 s31, v40, 5
-; GFX1150-NEXT: v_readlane_b32 s30, v40, 4
; GFX1150-NEXT: v_readlane_b32 s7, v40, 3
; GFX1150-NEXT: v_readlane_b32 s6, v40, 2
; GFX1150-NEXT: v_readlane_b32 s5, v40, 1
@@ -9205,12 +9204,12 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX12-NEXT: s_add_co_i32 s32, s32, 16
; GFX12-NEXT: v_writelane_b32 v40, s4, 0
; GFX12-NEXT: v_writelane_b32 v40, s5, 1
-; GFX12-NEXT: s_mov_b64 s[4:5], exec
; GFX12-NEXT: v_writelane_b32 v40, s6, 2
; GFX12-NEXT: v_writelane_b32 v40, s7, 3
-; GFX12-NEXT: s_mov_b64 s[6:7], exec
; GFX12-NEXT: v_writelane_b32 v40, s30, 4
; GFX12-NEXT: v_writelane_b32 v40, s31, 5
+; GFX12-NEXT: s_mov_b64 s[6:7], exec
+; GFX12-NEXT: s_mov_b64 s[4:5], exec
; GFX12-NEXT: .LBB16_1: ; =>This Inner Loop Header: Depth=1
; GFX12-NEXT: v_readfirstlane_b32 s0, v1
; GFX12-NEXT: s_wait_alu depctr_va_sdst(0)
@@ -9226,10 +9225,9 @@ define amdgpu_gfx i32 @test_indirect_call_vgpr_ptr_arg_and_reuse(i32 %i, i32 %fp
; GFX12-NEXT: s_cbranch_execnz .LBB16_1
; GFX12-NEXT: ; %bb.2:
; GFX12-NEXT: s_mov_b64 exec, s[4:5]
-; GFX12-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX12-NEXT: v_readlane_b32 s30, v40, 4
; GFX12-NEXT: v_mov_b32_e32 v0, v2
; GFX12-NEXT: v_readlane_b32 s31, v40, 5
-; GFX12-NEXT: v_readlane_b32 s30, v40, 4
; GFX12-NEXT: v_readlane_b32 s7, v40, 3
; GFX12-NEXT: v_readlane_b32 s6, v40, 2
; GFX12-NEXT: v_readlane_b32 s5, v40, 1
>From 30e56daceed361a9829811f35efb35cd8f2fa76f Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 2 Jun 2026 11:38:50 +0100
Subject: [PATCH 19/35] fixup! Improve documentation and comments
Fix bad conflict resolution
---
llvm/docs/AMDGPUUsage.rst | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index c526e62bcdfd1..3cdcaa85c99de 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -1893,6 +1893,32 @@ The AMDGPU backend implements the following LLVM IR intrinsics.
* :ref:`Synchronization Scope<amdgpu-intrinsics-syncscope-metadata-operand>`.
Note that the scope used must ensure that the L2 cache will be hit.
+ llvm.amdgcn.ds.atomic.barrier.arrive.rtn.b64 Available starting GFX12.5.
+ Corresponds to ``ds_atomic_barrier_arrive_rtn_b64``.
+
+ For the purposes of the memory model, this is a monotonic atomic
+ read-modify-write operation in the local address space.
+
+ This intrinsic has 2 operands:
+
+ * Local pointer to the LDS barrier data.
+ * Update value; the pending count of the barrier will be
+ decremented by this value (generally 1).
+
+ Returns the LDS barrier data as it was before this operation
+ was executed.
+
+ llvm.amdgcn.ds.atomic.async.barrier.arrive.b64 Available starting GFX12.5.
+ Corresponds to ``ds_atomic_async_barrier_arrive_b64``.
+
+ For the purposes of the memory model, this is an asynchronous
+ monotonic atomic read-modify-write operation in the local
+ address space.
+
+ This intrinsic has 1 operand:
+
+ * Local pointer to the LDS barrier data.
+
============================================== ==========================================================
.. TODO::
>From 446e259534113396065b73ae974a55b9e812ba91 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 2 Jun 2026 13:33:26 +0100
Subject: [PATCH 20/35] waterfall.readfirstlane type of input and output value
match
---
llvm/include/llvm/IR/IntrinsicsAMDGPU.td | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
index 94d810b466bc2..63a3edb149b57 100644
--- a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
+++ b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
@@ -2745,9 +2745,9 @@ def int_amdgcn_waterfall_begin :
// The readfirstlane intrinsic is a no-op if the index is used directly
// in the body of the waterfall loop.
def int_amdgcn_waterfall_readfirstlane :
- Intrinsic<[llvm_any_ty], // Returns any value
+ Intrinsic<[llvm_any_ty], // Uniform value read from the first active lane
[llvm_token_ty, // Token from final begin
- llvm_any_ty], // Non-uniform value to be used in readfirstlane
+ LLVMMatchType<0>], // Non-uniform value to read
[IntrConvergent]>;
// Effectively a no-op, but this intrinsic is used to indicate the end of the
>From df718ed2e995bbda1e9471ffe9fa52dcb35ec8e6 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 2 Jun 2026 14:14:38 +0100
Subject: [PATCH 21/35] Improve doc and comments
---
llvm/docs/AMDGPUUsage.rst | 28 ++++++++++++++++++------
llvm/include/llvm/IR/IntrinsicsAMDGPU.td | 21 +++++++++---------
2 files changed, 31 insertions(+), 18 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 3cdcaa85c99de..7ef17c0c548e3 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -2056,7 +2056,7 @@ for a frontend.
A group of waterfall intrinsics that depend on the same token defines a
single waterfall loop. The group identifies a region of code, the
-non-uniform operands that need to be made uniform, and the operands
+non-uniform operands that need to be made uniform, and the values
to use as the "index" of the loop.
A waterfall group must contain at least one
@@ -2074,6 +2074,16 @@ uniform. The index can be disjoint from the non-uniform operands. For
example, ``waterfall.begin`` can take an index while
``waterfall.readfirstlane`` takes a descriptor derived from that index.
+For example::
+
+ %tok0 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx0)
+ %tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok, i32 %idx1)
+ %s_rsrc = call <8 x i32> @llvm.amdgcn.waterfall.readfirstlane.v8i32.v8i32(token %tok1, <8 x i32> %rsrc)
+ %s_srsrc = call <4 x i32> @llvm.amdgcn.waterfall.readfirstlane.v4i32.v4i32(token %tok1, <4 x i32> %srsrc)
+ %r = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f32(..., <8 x i32> %s_rsrc, <4 x i32> %s_srsrc, ...)
+ %r1 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok1, <4 x float> %r)
+ ...
+
Each waterfall group must be contained within a single basic block.
A single basic block can contain more than one waterfall group.
@@ -2093,13 +2103,13 @@ If all operands are uniform, the compiler will not insert the waterfall loop.
Multiple values of different types can be combined as
an index by threading tokens through multiple ``waterfall.begin`` intrinsics::
- %tok0 = llvm.amdgcn.waterfall.begin(i32 0, i32 %idx0)
- %tok1 = llvm.amdgcn.waterfall.begin(i32 %tok0, i32 %idx1)
+ %tok0 = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %idx0)
+ %tok1 = call token @llvm.amdgcn.waterfall.begin.i32(token %tok0, i32 %idx1)
...
- The intrinsic takes a token
- (use a null/zero value if this is the first ``waterfall.begin`` in a
- waterfall group) and an operand.
+ The intrinsic takes a token and a non-uniform value.
+ Use ``poison`` as a token if this is the first ``waterfall.begin`` in a
+ waterfall group.
The intrinsic returns a new token that must be threaded through the
corresponding ``waterfall.readfirstlane`` and ``waterfall.end``
@@ -2114,8 +2124,12 @@ If all operands are uniform, the compiler will not insert the waterfall loop.
The correctness requirement is that lanes with the same index values
must have the same values for all ``waterfall.readfirstlane`` operands.
+ The ``waterfall.readfirstlane`` intrinsic is a no-op if the index value
+ is used directly in the body of the waterfall loop.
+
``llvm.amdgcn.waterfall.end`` Marks the end of a waterfall region.
- Takes the token from the final ``waterfall.begin`` in the group.
+ Takes the token from the final ``waterfall.begin`` in the group and
+ a value. Effectively a no-op tagging a value as the end of the section.
============================================== =========================================================================
diff --git a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
index 63a3edb149b57..5f2732024b014 100644
--- a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
+++ b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
@@ -2723,37 +2723,36 @@ def int_amdgcn_cs_chain:
// Waterfall intrinsics are used to mark a region as requiring waterfall loops to
// activate and deactivate lanes in a loop (and sometimes transform VGPR values
// into SGPR values).
-// Best case (all lanes have the same values) the loop will execute once, worst case (all lanes
-// have different values) the loop will execute wave-size times.
+// Best case (all lanes have the same values) the loop will execute once,
+// worst case (all lanes have different values) the loop will execute wave-size times.
+//
// The waterfall.begin intrinsic returns a new token that must be threaded through the
// corresponding waterfall.readfirstlane and waterfall.end intrinsics, forming a
// waterfall group of intrinsics that together define a waterfall region.
// All intrinsics in a waterfall group must reside within the same basic block.
+// TODO: extend the support to allow spanning multiple blocks
+//
// Where the intrinsic specifies "llvm_any_ty", the intrinsic will accept any of
// the following types:
// i16, v2i16, v4i16, i32, v2i32, v4i32, v8i32, f16, v2f16, v4f16, f32, v2f32,
// v4f32, v8f32
-// TODO: extend the support to allow spanning multiple blocks
+//
+// The waterfall.begin intrinsic marks the beginning of a waterfall region of code.
def int_amdgcn_waterfall_begin :
Intrinsic<[llvm_token_ty], // Returns token to be used by the rest of the waterfall loop
[llvm_token_ty, // Previous begin token / poison if first
llvm_any_ty], // Non-uniform value for waterfall index
[IntrConvergent]>;
-// Often the index operand of begin is the same as the non-uniform operand
-// of readfirstlane, however, this isn't always the case.
-// The readfirstlane intrinsic is a no-op if the index is used directly
-// in the body of the waterfall loop.
+// The waterfall.readfirstlane intrinsic reads the first active lane's value
+// and returns it as a uniform value for use within a waterfall region.
def int_amdgcn_waterfall_readfirstlane :
Intrinsic<[llvm_any_ty], // Uniform value read from the first active lane
[llvm_token_ty, // Token from final begin
LLVMMatchType<0>], // Non-uniform value to read
[IntrConvergent]>;
-// Effectively a no-op, but this intrinsic is used to indicate the end of the
-// region for waterfall by tagging a value as the end of the section.
-// Waterfall will happen for all values dependent on the begin until the end
-// intrinsic is reached,
+// The waterfall.end intrinsic marks the end of a waterfall region of code.
def int_amdgcn_waterfall_end :
Intrinsic<[llvm_any_ty], // Returns any value
[llvm_token_ty, // Token from final begin
>From 7107ef20febbe612aaaa2973c1310089fbb22b11 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 2 Jun 2026 15:04:52 +0100
Subject: [PATCH 22/35] Cleanup in AMDGPUInsertWaterfall
---
llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
index 11b2d38067528..d89bd6d9b7541 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
@@ -112,8 +112,7 @@ static void readFirstLaneReg(MachineBasicBlock &MBB, MachineRegisterInfo *MRI,
// Check if operand is uniform by checking:
// 1. Trivially detectable as operand in SGPR
// 2. Direct def is from an SGPR->VGPR copy (which may happen if assumed
-// non-uniform value
-// turns out to be uniform)
+// non-uniform value turns out to be uniform)
static Register getUniformOperandReplacementReg(MachineRegisterInfo *MRI,
const SIRegisterInfo *RI,
Register Reg) {
@@ -254,13 +253,15 @@ class AMDGPUInsertWaterfall : public MachineFunctionPass {
MachineRegisterInfo *_MRI)
: TII(_TII), MRI(_MRI), Final(nullptr) {
- auto TokMO = TII->getNamedOperand(*_Begin, AMDGPU::OpName::tok_ret);
+ const MachineOperand *TokMO = TII->getNamedOperand(*_Begin, AMDGPU::OpName::tok_ret);
assert(tokIsStart(TII->getNamedOperand(*_Begin, AMDGPU::OpName::tok)) &&
"first begin does not have an undefined input token as expected");
+
assert(TokMO &&
"Unable to extract tok operand from SI_WATERFALL_BEGIN pseudo op");
+
BeginList.push_back(_Begin);
TokReg = TokMO->getReg();
}
@@ -277,7 +278,7 @@ class AMDGPUInsertWaterfall : public MachineFunctionPass {
MachineInstr *getDefInstr(const MachineOperand *MO) const {
if (MO->isReg() && MRI->hasOneDef(MO->getReg())) {
- return (*MRI->def_begin(MO->getReg())).getParent();
+ return (MRI->def_begin(MO->getReg()))->getParent();
}
return nullptr;
}
>From 1dc39628285f25e80042573535b382f76df6fbf3 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 2 Jun 2026 15:24:54 +0100
Subject: [PATCH 23/35] Remove processCandidate (only needed with
waterfall.last.use)
---
llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp | 10 ----------
1 file changed, 10 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
index d89bd6d9b7541..449ab1dbe9d3a 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
@@ -269,13 +269,6 @@ class AMDGPUInsertWaterfall : public MachineFunctionPass {
WaterfallWorkitem(const SIInstrInfo *_TII, MachineRegisterInfo *_MRI)
: TII(_TII), MRI(_MRI), TokReg(AMDGPU::NoRegister), Final(nullptr) {}
- void processCandidate(MachineInstr *Cand) {
- unsigned Opcode = Cand->getOpcode();
- if (getWFBeginSize(Opcode) || getWFRFLSize(Opcode) ||
- getWFEndSize(Opcode))
- return;
- }
-
MachineInstr *getDefInstr(const MachineOperand *MO) const {
if (MO->isReg() && MRI->hasOneDef(MO->getReg())) {
return (MRI->def_begin(MO->getReg()))->getParent();
@@ -803,9 +796,6 @@ bool AMDGPUInsertWaterfall::runOnMachineFunction(MachineFunction &MF) {
if (!Worklist.back().addCandidate(&MI)) {
llvm_unreachable("Overlapping SI_WATERFALL_* groups");
}
- } else {
- if (Worklist.size())
- Worklist.back().processCandidate(&MI);
}
}
Changed |= processWaterfall(MBB);
>From dd8257fc5fcfa842bfc4a23904d9bd26d7faa6a6 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 2 Jun 2026 15:28:44 +0100
Subject: [PATCH 24/35] Remove Removed variable in removeRedundantWaterfall
---
llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp | 9 +++------
1 file changed, 3 insertions(+), 6 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
index 449ab1dbe9d3a..ed33f44d54036 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
@@ -421,7 +421,6 @@ bool AMDGPUInsertWaterfall::removeRedundantWaterfall(WaterfallWorkitem &Item) {
// uniform - unless there are no begin instructions, in which case we always
// remove
bool LoopRemoved = false;
- unsigned Removed = 0;
std::vector<MachineInstr *> NewRFLList;
std::vector<MachineInstr *> ToRemoveRFLList;
@@ -434,10 +433,8 @@ bool AMDGPUInsertWaterfall::removeRedundantWaterfall(WaterfallWorkitem &Item) {
Register ReplaceReg = getUniformOperandReplacementReg(MRI, RI, RFLSrcReg);
if (ReplaceReg != AMDGPU::NoRegister) {
MRI->replaceRegWith(RFLDstReg, ReplaceReg);
- Removed++;
ToRemoveRFLList.push_back(RFLMI);
} else if (RFLDstOp->isDead()) {
- Removed++;
ToRemoveRFLList.push_back(RFLMI);
} else {
NewRFLList.push_back(RFLMI);
@@ -447,7 +444,7 @@ bool AMDGPUInsertWaterfall::removeRedundantWaterfall(WaterfallWorkitem &Item) {
// Note: this test also returns true when there are NO RFL intrinsics, the
// case where a prior pass has removed all of them and the loop is now
// redundant
- if (!Item.BeginList.size() || Removed == Item.RFLList.size()) {
+ if (!Item.BeginList.size() || ToRemoveRFLList.size() == Item.RFLList.size()) {
// Removed all of the RFLs
// We can remove the waterfall loop entirely
@@ -481,8 +478,8 @@ bool AMDGPUInsertWaterfall::removeRedundantWaterfall(WaterfallWorkitem &Item) {
LoopRemoved = true;
- } else if (Removed) {
- LLVM_DEBUG(dbgs() << "Removed " << Removed
+ } else if (!ToRemoveRFLList.empty()) {
+ LLVM_DEBUG(dbgs() << "Removed " << ToRemoveRFLList.size()
<< " waterfall rfl intrinsics due to being uniform - "
"updating remaining rfl list\n");
// TODO: there's an opportunity to pull the DAG involving the (removed) rfls
>From b008aa82315dd174bd536ef4465f0f035af9602f Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 2 Jun 2026 15:45:20 +0100
Subject: [PATCH 25/35] Use empty instead of !size
---
llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
index ed33f44d54036..693181e9da8fa 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
@@ -444,7 +444,7 @@ bool AMDGPUInsertWaterfall::removeRedundantWaterfall(WaterfallWorkitem &Item) {
// Note: this test also returns true when there are NO RFL intrinsics, the
// case where a prior pass has removed all of them and the loop is now
// redundant
- if (!Item.BeginList.size() || ToRemoveRFLList.size() == Item.RFLList.size()) {
+ if (Item.BeginList.empty() || ToRemoveRFLList.size() == Item.RFLList.size()) {
// Removed all of the RFLs
// We can remove the waterfall loop entirely
>From d0b0520655b921f07b8ba958b88ac81d50b5421f Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 2 Jun 2026 15:45:51 +0100
Subject: [PATCH 26/35] Use SmallVector
---
llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
index 693181e9da8fa..b4662a2eba741 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
@@ -421,8 +421,8 @@ bool AMDGPUInsertWaterfall::removeRedundantWaterfall(WaterfallWorkitem &Item) {
// uniform - unless there are no begin instructions, in which case we always
// remove
bool LoopRemoved = false;
- std::vector<MachineInstr *> NewRFLList;
- std::vector<MachineInstr *> ToRemoveRFLList;
+ SmallVector<MachineInstr *, 4> NewRFLList;
+ SmallVector<MachineInstr *, 4> ToRemoveRFLList;
for (auto RFLMI : Item.RFLList) {
auto RFLSrcOp = TII->getNamedOperand(*RFLMI, AMDGPU::OpName::src);
>From e375e28da4f2f62f47a03ef3ead11c931329329e Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 2 Jun 2026 15:46:14 +0100
Subject: [PATCH 27/35] readFirstLaneReg doesn't need "source register"
argument
it can be obtained from source operand
---
llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp | 16 +++++++---------
1 file changed, 7 insertions(+), 9 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
index b4662a2eba741..7a3321be834c5 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
@@ -80,9 +80,9 @@ static unsigned getWFEndSize(const unsigned Opcode) {
static void readFirstLaneReg(MachineBasicBlock &MBB, MachineRegisterInfo *MRI,
const SIRegisterInfo *RI, const SIInstrInfo *TII,
MachineBasicBlock::iterator &I, const DebugLoc &DL,
- Register RFLReg, Register RFLSrcReg,
- const MachineOperand &RFLSrcOp) {
- auto RFLRegRC = MRI->getRegClass(RFLReg);
+ Register RFLReg, const MachineOperand &RFLSrcOp) {
+ Register RFLSrcReg = RFLSrcOp.getReg();
+ const TargetRegisterClass *RFLRegRC = MRI->getRegClass(RFLReg);
uint32_t RegSize = RI->getRegSizeInBits(*RFLRegRC) / 32;
assert(RI->hasVGPRs(MRI->getRegClass(RFLSrcReg)) &&
"unexpected uniform operand for readfirstlane");
@@ -467,11 +467,10 @@ bool AMDGPUInsertWaterfall::removeRedundantWaterfall(WaterfallWorkitem &Item) {
for (auto RFLMI : NewRFLList) {
auto DstReg = TII->getNamedOperand(*RFLMI, AMDGPU::OpName::dst)->getReg();
auto SrcOp = TII->getNamedOperand(*RFLMI, AMDGPU::OpName::src);
- Register SrcReg = SrcOp->getReg();
MachineBasicBlock::iterator RFLInsert(RFLMI);
readFirstLaneReg(*RFLMI->getParent(), MRI, RI, TII, RFLInsert,
- RFLMI->getDebugLoc(), DstReg, SrcReg, *SrcOp);
+ RFLMI->getDebugLoc(), DstReg, *SrcOp);
}
Item.eraseFromParent();
@@ -661,19 +660,18 @@ bool AMDGPUInsertWaterfall::processWaterfall(MachineBasicBlock &MBB) {
// Get the next index to use from the first enabled lane
for (auto &CurrIdx : IndexList)
readFirstLaneReg(LoopBB, MRI, RI, TII, J, DL, CurrIdx.CurrentIdxReg,
- CurrIdx.Index->getReg(), *CurrIdx.Index);
+ *CurrIdx.Index);
// Also process the readlane pseudo ops - if readfirstlane is using the
// index then just replace with the CurrentIdxReg instead
for (auto RFLMI : Item.RFLList) {
auto RFLSrcOp = TII->getNamedOperand(*RFLMI, AMDGPU::OpName::src);
auto RFLDstOp = TII->getNamedOperand(*RFLMI, AMDGPU::OpName::dst);
- Register RFLSrcReg = RFLSrcOp->getReg();
Register RFLDstReg = RFLDstOp->getReg();
bool MatchedIdx = false;
for (auto &CurrIdx : IndexList) {
- if (RFLSrcReg == CurrIdx.Index->getReg()) {
+ if (RFLSrcOp->getReg() == CurrIdx.Index->getReg()) {
// Use the CurrentIdxReg for this
Item.RFLRegs.push_back(CurrIdx.CurrentIdxReg);
MRI->replaceRegWith(RFLDstReg, CurrIdx.CurrentIdxReg);
@@ -686,7 +684,7 @@ bool AMDGPUInsertWaterfall::processWaterfall(MachineBasicBlock &MBB) {
// Insert function to expand to required size here
MachineBasicBlock::iterator RFLInsert(RFLMI);
readFirstLaneReg(LoopBB, MRI, RI, TII, RFLInsert, DL, RFLDstReg,
- RFLSrcReg, *RFLSrcOp);
+ *RFLSrcOp);
}
}
>From 72f42efb756792a827e32ea085e89c9212507a5c Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 2 Jun 2026 16:17:54 +0100
Subject: [PATCH 28/35] More cleanup
---
.../Target/AMDGPU/AMDGPUInsertWaterfall.cpp | 29 +++++++++----------
1 file changed, 14 insertions(+), 15 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
index 7a3321be834c5..772542f6e563d 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
@@ -239,9 +239,9 @@ class AMDGPUInsertWaterfall : public MachineFunctionPass {
Register TokReg; // This is always the token from the last begin intrinsic
MachineInstr *Final;
- std::vector<MachineInstr *> BeginList;
- std::vector<MachineInstr *> RFLList;
- std::vector<MachineInstr *> EndList;
+ SmallVector<MachineInstr *, 4> BeginList;
+ SmallVector<MachineInstr *, 4> RFLList;
+ SmallVector<MachineInstr *, 4> EndList;
// List of corresponding init, newdst and phi registers used in loop for
// end pseudos
@@ -483,9 +483,7 @@ bool AMDGPUInsertWaterfall::removeRedundantWaterfall(WaterfallWorkitem &Item) {
"updating remaining rfl list\n");
// TODO: there's an opportunity to pull the DAG involving the (removed) rfls
// out of the waterfall loop
- Item.RFLList.clear();
- std::copy(NewRFLList.begin(), NewRFLList.end(),
- std::back_inserter(Item.RFLList));
+ Item.RFLList = std::move(NewRFLList);
for (auto RFLMI : ToRemoveRFLList)
RFLMI->eraseFromParent();
}
@@ -499,14 +497,15 @@ bool AMDGPUInsertWaterfall::processWaterfall(MachineBasicBlock &MBB) {
MachineBasicBlock *CurrMBB = &MBB;
// Firstly we check that there are at least 3 related waterfall instructions
- // for this begin
- // SI_WATERFALL_BEGIN [ SI_WATERFALL_BEGIN ]*
- // [ SI_WATERFALL_READFIRSTLANE ]+ [ SI_WATERFALL_END ]+ If there are multiple
- // waterfall loops they must also be disjoint
+ // for this begin:
+ // [ SI_WATERFALL_BEGIN ]+
+ // [ SI_WATERFALL_READFIRSTLANE ]+
+ // [ SI_WATERFALL_END ]+
+ // If there are multiple waterfall loops they must also be disjoint.
for (WaterfallWorkitem &Item : Worklist) {
LLVM_DEBUG(
- if (Item.BeginList.size()) {
+ if (!Item.BeginList.empty()) {
dbgs() << "Processing " << *Item.BeginList[0] << "\n";
for (auto RUse = MRI->use_begin(Item.TokReg), RSE = MRI->use_end();
@@ -522,7 +521,7 @@ bool AMDGPUInsertWaterfall::processWaterfall(MachineBasicBlock &MBB) {
continue;
}
- assert(Item.RFLList.size() && Item.EndList.size() &&
+ assert(!Item.RFLList.empty() && !Item.EndList.empty() &&
"SI_WATERFALL* pseudo instruction group must have at least 1 of "
"each type");
@@ -541,7 +540,7 @@ bool AMDGPUInsertWaterfall::processWaterfall(MachineBasicBlock &MBB) {
Register CurrentIdxReg;
} IdxInfo;
- std::vector<IdxInfo> IndexList;
+ SmallVector<IdxInfo, 4> IndexList;
#ifndef NDEBUG
bool IsUniform = true;
#endif
@@ -701,7 +700,7 @@ bool AMDGPUInsertWaterfall::processWaterfall(MachineBasicBlock &MBB) {
CondReg = compareIdx(LoopBB, MRI, RI, TII, J, DL, CurrIdx.CurrentIdxReg,
*CurrIdx.Index, CondReg, ST->isWave32());
- // Update EXEC, save the original EXEC value to VCC
+ // Update EXEC, save the original EXEC value to NewExec
BuildMI(LoopBB, J, DL, TII->get(SaveExecOpc), NewExec)
.addReg(CondReg, RegState::Kill);
@@ -780,7 +779,7 @@ bool AMDGPUInsertWaterfall::runOnMachineFunction(MachineFunction &MF) {
// Tag StartNew as true if we encounter another begin
StartNew = true;
- if (!Worklist.size() || getToken(&MI) != Worklist.back().TokReg) {
+ if (Worklist.empty() || getToken(&MI) != Worklist.back().TokReg) {
// There's no associated begin for these body intrinsics
// That means it's either an error - or all the begin intrinsics
// were removed due to being uniform
>From 410db47daec2b036892af562ea201dd764828c46 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 2 Jun 2026 16:35:25 +0100
Subject: [PATCH 29/35] Format
---
llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
index 772542f6e563d..7bc4d2fd03a96 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
@@ -253,7 +253,8 @@ class AMDGPUInsertWaterfall : public MachineFunctionPass {
MachineRegisterInfo *_MRI)
: TII(_TII), MRI(_MRI), Final(nullptr) {
- const MachineOperand *TokMO = TII->getNamedOperand(*_Begin, AMDGPU::OpName::tok_ret);
+ const MachineOperand *TokMO =
+ TII->getNamedOperand(*_Begin, AMDGPU::OpName::tok_ret);
assert(tokIsStart(TII->getNamedOperand(*_Begin, AMDGPU::OpName::tok)) &&
"first begin does not have an undefined input token as expected");
@@ -261,7 +262,6 @@ class AMDGPUInsertWaterfall : public MachineFunctionPass {
assert(TokMO &&
"Unable to extract tok operand from SI_WATERFALL_BEGIN pseudo op");
-
BeginList.push_back(_Begin);
TokReg = TokMO->getReg();
}
>From 28cf22c872033cdf19ff061b89e0289f6d9eaafa Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 2 Jun 2026 19:14:08 +0100
Subject: [PATCH 30/35] Address review: improve docs
---
llvm/docs/AMDGPUUsage.rst | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 7ef17c0c548e3..12aba1f7c0d59 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -2086,9 +2086,11 @@ For example::
Each waterfall group must be contained within a single basic block.
A single basic block can contain more than one waterfall group.
+Waterfall regions must not overlap.
-Later compiler passes can remove operands determined as uniform.
-If all operands are uniform, the compiler will not insert the waterfall loop.
+Later compiler passes can remove parts of the index that are statically determined to be uniform
+(i.e., remove the corresponding ``waterfall.begin`` intrinsics).
+If all parts of the index are uniform, the compiler will not insert the waterfall loop.
.. table:: AMDGPU Waterfall Intrinsics
:name: amdgpu-waterfall-intrinsics-table
@@ -2129,7 +2131,8 @@ If all operands are uniform, the compiler will not insert the waterfall loop.
``llvm.amdgcn.waterfall.end`` Marks the end of a waterfall region.
Takes the token from the final ``waterfall.begin`` in the group and
- a value. Effectively a no-op tagging a value as the end of the section.
+ a value that needs to be available after the waterfall loop.
+ Effectively a no-op tagging a value as the end of the section.
============================================== =========================================================================
>From 5af98acd6a3a2e109653d4069c4b4a9e851928ef Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Tue, 2 Jun 2026 19:43:08 +0100
Subject: [PATCH 31/35] Don't copy call-site attributes
---
llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp | 6 ------
1 file changed, 6 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp
index 7d3428d794e29..d3515855adcef 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp
@@ -143,10 +143,6 @@ static std::optional<Instruction *> modifyIntrinsicCall(
NewCall->copyMetadata(OldIntr);
if (isa<FPMathOperator>(NewCall))
NewCall->copyFastMathFlags(&OldIntr);
- // Copy attributes
- AttributeList OldAttrList = OldIntr.getAttributes();
- NewCall->setAttributes(OldAttrList);
-
// Erase and replace uses
if (!InstToReplace.getType()->isVoidTy())
IC.replaceInstUsesWith(InstToReplace, NewCall);
@@ -2398,8 +2394,6 @@ static Value *simplifyAMDGCNMemoryIntrinsicDemanded(InstCombiner &IC,
IC.Builder.CreateIntrinsic(II.getIntrinsicID(), OverloadTys, Args);
NewCall->takeName(&II);
NewCall->copyMetadata(II);
- AttributeList OldAttrList = II.getAttributes();
- NewCall->setAttributes(OldAttrList);
if (IsLoad) {
if (NewNumElts == 1) {
>From faee3075f796cd168e164b45dae9c26ead3d0aba Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Wed, 3 Jun 2026 08:47:47 +0100
Subject: [PATCH 32/35] Fix inst combine waterfall test
---
llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll b/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
index 5e96f4a444f5b..f74b70281aa94 100644
--- a/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
+++ b/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
@@ -4,7 +4,7 @@
define amdgpu_ps <4 x float> @test_waterfall_same_index2(<8 x i32> addrspace(4)* inreg %in, i32 %index, float %s, <4 x i32> inreg %samp) {
; CHECK-LABEL: @test_waterfall_same_index2(
; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 [[INDEX:%.*]])
-; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token [[WF_TOKEN1]], i32 [[INDEX]])
+; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32(token [[WF_TOKEN1]], i32 [[INDEX]])
; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
; CHECK-NEXT: [[PTR:%.*]] = getelementptr [32 x i8], ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
; CHECK-NEXT: [[RSRC:%.*]] = load <8 x i32>, ptr addrspace(4) [[PTR]], align 32
@@ -25,7 +25,7 @@ define amdgpu_ps <4 x float> @test_waterfall_same_index2(<8 x i32> addrspace(4)*
define amdgpu_ps <4 x float> @test_waterfall_same_index3(<8 x i32> addrspace(4)* inreg %in, i32 %index, float %s, <4 x i32> inreg %samp) {
; CHECK-LABEL: @test_waterfall_same_index3(
; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 [[INDEX:%.*]])
-; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token [[WF_TOKEN1]], i32 [[INDEX]])
+; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32(token [[WF_TOKEN1]], i32 [[INDEX]])
; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
; CHECK-NEXT: [[PTR:%.*]] = getelementptr [32 x i8], ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
; CHECK-NEXT: [[RSRC:%.*]] = load <8 x i32>, ptr addrspace(4) [[PTR]], align 32
@@ -48,7 +48,7 @@ define amdgpu_ps <4 x float> @test_waterfall_same_index_aba(<8 x i32> addrspace(
; CHECK-LABEL: @test_waterfall_same_index_aba(
; CHECK-NEXT: [[WF_TOKEN1:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 [[INDEX1:%.*]])
; CHECK-NEXT: [[WF_TOKEN2:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token [[WF_TOKEN1]], i32 [[INDEX2:%.*]])
-; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token [[WF_TOKEN2]], i32 [[INDEX2]])
+; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32(token [[WF_TOKEN2]], i32 [[INDEX2]])
; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
; CHECK-NEXT: [[PTR:%.*]] = getelementptr [32 x i8], ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
; CHECK-NEXT: [[RSRC:%.*]] = load <8 x i32>, ptr addrspace(4) [[PTR]], align 32
>From 253bda0a4135b131c143b3dafb73cfb124e40672 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Wed, 3 Jun 2026 09:37:00 +0100
Subject: [PATCH 33/35] Add waterfall InstCombine test to show call-site
attribute is lost
composing L->LZ rewrite and dmask shrink
Demonstrates two InstCombine transforms firing in sequence on the same
image.sample call inside a waterfall region: lod=0.0 rewrites
image.sample.l.1d to image.sample.lz.1d, and demanded-elements then
shrinks the dmask from 15 to 3 and the return type from <4 x float> to
<2 x float>. Both transforms rebuild the call via
IC.Builder.CreateIntrinsic, which drops call-site-only attributes
(memory(argmem: read) here).
Assisted-by: Claude <noreply at anthropic.com>
---
.../InstCombine/AMDGPU/waterfall.ll | 41 +++++++++++++++++++
1 file changed, 41 insertions(+)
diff --git a/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll b/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
index f74b70281aa94..83b2d89260a36 100644
--- a/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
+++ b/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
@@ -67,7 +67,48 @@ define amdgpu_ps <4 x float> @test_waterfall_same_index_aba(<8 x i32> addrspace(
ret <4 x float> %r1
}
+; Inside a waterfall region, InstCombine runs two transforms on the same
+; image.sample call:
+; 1. modifyIntrinsicCall: lod = 0.0 rewrites image.sample.l.1d to
+; image.sample.lz.1d, dropping the lod operand.
+; 2. simplifyAMDGCNMemoryIntrinsicDemanded: only lanes 0 and 1 escape
+; waterfall.end, so the dmask shrinks from 15 to 3 and the return
+; type shrinks from <4 x float> to <2 x float>. This is reached via
+; waterfall.end's SimplifyAndSetOp.
+; Each transform rebuilds the call with IC.Builder.CreateIntrinsic, which
+; would drop any call-site-only attributes (here, memory(argmem: read)
+; attached via #1). The new call is guaranteed to inherit attributes from
+; the intrinsic declaration, but not call-site attributes.
+define amdgpu_ps <2 x float> @waterfall_image_l_to_lz_shrink(<8 x i32> addrspace(4)* inreg %in, i32 %vidx, float %s, <4 x i32> inreg %samp) {
+; CHECK-LABEL: @waterfall_image_l_to_lz_shrink(
+; CHECK-NEXT: [[TOK:%.*]] = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 [[VIDX:%.*]])
+; CHECK-NEXT: [[S_IDX:%.*]] = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32(token [[TOK]], i32 [[VIDX]])
+; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
+; CHECK-NEXT: [[PTR:%.*]] = getelementptr [32 x i8], ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
+; CHECK-NEXT: [[R:%.*]] = load <8 x i32>, ptr addrspace(4) [[PTR]], align 32
+; CHECK-NEXT: [[V4:%.*]] = call <2 x float> @llvm.amdgcn.image.sample.lz.1d.v2f32.f32.v8i32.v4i32(i32 3, float [[S:%.*]], <8 x i32> [[R]], <4 x i32> [[SAMP:%.*]], i1 false, i32 0, i32 0)
+; CHECK-NEXT: [[TMP2:%.*]] = shufflevector <2 x float> [[V4]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 poison, i32 poison>
+; CHECK-NEXT: [[W4:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token [[TOK]], <4 x float> [[TMP2]])
+; CHECK-NEXT: [[R1:%.*]] = shufflevector <4 x float> [[W4]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
+; CHECK-NEXT: ret <2 x float> [[R1]]
+;
+ %tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %vidx)
+ %s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %tok, i32 %vidx)
+ %ptr = getelementptr <8 x i32>, ptr addrspace(4) %in, i32 %s_idx
+ %r = load <8 x i32>, ptr addrspace(4) %ptr, align 32
+ %v4 = call <4 x float> @llvm.amdgcn.image.sample.l.1d.v4f32.f32.v8i32.v4i32(i32 15, float %s, float 0.0, <8 x i32> %r, <4 x i32> %samp, i1 false, i32 0, i32 0) #1
+ %w4 = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token %tok, <4 x float> %v4)
+ %e0 = extractelement <4 x float> %w4, i32 0
+ %e1 = extractelement <4 x float> %w4, i32 1
+ %r0 = insertelement <2 x float> poison, float %e0, i32 0
+ %r1 = insertelement <2 x float> %r0, float %e1, i32 1
+ ret <2 x float> %r1
+}
+
declare token @llvm.amdgcn.waterfall.begin.i32(token, i32)
declare i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token, i32)
declare <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token, <4 x float>)
declare <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f32(i32, float, <8 x i32>, <4 x i32>, i1, i32, i32)
+declare <4 x float> @llvm.amdgcn.image.sample.l.1d.v4f32.f32.v8i32.v4i32(i32, float, float, <8 x i32>, <4 x i32>, i1, i32, i32)
+
+attributes #1 = { memory(argmem: read) }
>From 053b0d4a674f7e3fd482a771e393d0167fcb8a34 Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Wed, 3 Jun 2026 09:44:14 +0100
Subject: [PATCH 34/35] Reapply "Don't copy call-site attributes" and tighten
test
Reverts the removal of setAttributes in modifyIntrinsicCall and
simplifyAMDGCNMemoryIntrinsicDemanded. The copies are needed to
preserve call-site-only attributes (e.g. memory(argmem: read)) across
the L->LZ rewrite and the demanded-elements dmask shrink that fire in
sequence inside a waterfall region.
Tightens waterfall_image_l_to_lz_shrink to observe the surviving
attribute group via #[[ATTR]] / { memory(argmem: read) }, so future
removal of the call-site attribute copy will fail this test.
Assisted-by: Claude <noreply at anthropic.com>
---
llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp | 6 ++++++
llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll | 3 ++-
2 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp
index d3515855adcef..7d3428d794e29 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp
@@ -143,6 +143,10 @@ static std::optional<Instruction *> modifyIntrinsicCall(
NewCall->copyMetadata(OldIntr);
if (isa<FPMathOperator>(NewCall))
NewCall->copyFastMathFlags(&OldIntr);
+ // Copy attributes
+ AttributeList OldAttrList = OldIntr.getAttributes();
+ NewCall->setAttributes(OldAttrList);
+
// Erase and replace uses
if (!InstToReplace.getType()->isVoidTy())
IC.replaceInstUsesWith(InstToReplace, NewCall);
@@ -2394,6 +2398,8 @@ static Value *simplifyAMDGCNMemoryIntrinsicDemanded(InstCombiner &IC,
IC.Builder.CreateIntrinsic(II.getIntrinsicID(), OverloadTys, Args);
NewCall->takeName(&II);
NewCall->copyMetadata(II);
+ AttributeList OldAttrList = II.getAttributes();
+ NewCall->setAttributes(OldAttrList);
if (IsLoad) {
if (NewNumElts == 1) {
diff --git a/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll b/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
index 83b2d89260a36..7ed8565c1de49 100644
--- a/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
+++ b/llvm/test/Transforms/InstCombine/AMDGPU/waterfall.ll
@@ -86,12 +86,13 @@ define amdgpu_ps <2 x float> @waterfall_image_l_to_lz_shrink(<8 x i32> addrspace
; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[S_IDX]] to i64
; CHECK-NEXT: [[PTR:%.*]] = getelementptr [32 x i8], ptr addrspace(4) [[IN:%.*]], i64 [[TMP1]]
; CHECK-NEXT: [[R:%.*]] = load <8 x i32>, ptr addrspace(4) [[PTR]], align 32
-; CHECK-NEXT: [[V4:%.*]] = call <2 x float> @llvm.amdgcn.image.sample.lz.1d.v2f32.f32.v8i32.v4i32(i32 3, float [[S:%.*]], <8 x i32> [[R]], <4 x i32> [[SAMP:%.*]], i1 false, i32 0, i32 0)
+; CHECK-NEXT: [[V4:%.*]] = call <2 x float> @llvm.amdgcn.image.sample.lz.1d.v2f32.f32.v8i32.v4i32(i32 3, float [[S:%.*]], <8 x i32> [[R]], <4 x i32> [[SAMP:%.*]], i1 false, i32 0, i32 0) #[[ATTR:[0-9]+]]
; CHECK-NEXT: [[TMP2:%.*]] = shufflevector <2 x float> [[V4]], <2 x float> poison, <4 x i32> <i32 0, i32 1, i32 poison, i32 poison>
; CHECK-NEXT: [[W4:%.*]] = call <4 x float> @llvm.amdgcn.waterfall.end.v4f32(token [[TOK]], <4 x float> [[TMP2]])
; CHECK-NEXT: [[R1:%.*]] = shufflevector <4 x float> [[W4]], <4 x float> poison, <2 x i32> <i32 0, i32 1>
; CHECK-NEXT: ret <2 x float> [[R1]]
;
+; CHECK: attributes #[[ATTR]] = { memory(argmem: read) }
%tok = call token @llvm.amdgcn.waterfall.begin.i32(token poison, i32 %vidx)
%s_idx = call i32 @llvm.amdgcn.waterfall.readfirstlane.i32.i32(token %tok, i32 %vidx)
%ptr = getelementptr <8 x i32>, ptr addrspace(4) %in, i32 %s_idx
>From b413b95e28002c81cd2bf52e4dea28aa172f38af Mon Sep 17 00:00:00 2001
From: Greta Yorsh <Greta.Yorsh at amd.com>
Date: Wed, 3 Jun 2026 09:55:13 +0100
Subject: [PATCH 35/35] Remove parenthesis (why can't clang-format do it?)
---
llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
index 7bc4d2fd03a96..638078146f26c 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInsertWaterfall.cpp
@@ -271,7 +271,7 @@ class AMDGPUInsertWaterfall : public MachineFunctionPass {
MachineInstr *getDefInstr(const MachineOperand *MO) const {
if (MO->isReg() && MRI->hasOneDef(MO->getReg())) {
- return (MRI->def_begin(MO->getReg()))->getParent();
+ return MRI->def_begin(MO->getReg())->getParent();
}
return nullptr;
}
More information about the llvm-commits
mailing list