[Mlir-commits] [llvm] [mlir] [mlir][tosa] Limit shape op level checks to MAX_SHAPE_LEN (PR #212193)
llvmlistbot at llvm.org
llvmlistbot at llvm.org
Mon Jul 27 01:21:34 PDT 2026
https://github.com/atabakp created https://github.com/llvm/llvm-project/pull/212193
Cherry-pick of llvm/llvm-project at bc010989719d. Replaces #13.
#13 works around the failure by globally disabling shape-type rank validation. This is the upstream root-cause fix: shape ops are level-checked against a new MAX_SHAPE_LEN (16 at level 8K, 64 at NONE), and the incorrect blanket MAX_RANK check on all shapeTypes is removed. A pad shape operand is 2*rank, so it was never correct to check it against MAX_RANK.
Confirmed on the #13 case (in16x1x48, k3x1, oc48, s2x1): the stride-2 decomposition emits `tosa.pad` with `!tosa.shape<8>`, which fails `exceeds MAX_RANK` on torq-3.10 and passes with this change.
Adapted: only AddShape, DivCeilShape, DivFloorShape, MulShape and SubShape exist in this tree, so the remaining conversions and their tests are omitted.
>From 007de72dd03e260664c054d56f653c1451c41287 Mon Sep 17 00:00:00 2001
From: Benoit Jacob <benoit.jacob at amd.com>
Date: Wed, 17 Dec 2025 19:08:47 +0000
Subject: [PATCH 01/10] Revert "Reland "Redesign Straight-Line Strength
Reduction (SLSR) (#162930)" (#169614)"
This reverts commit f803e463f921f9cf460f4c3e96372fb3755cc934.
---
.../Scalar/StraightLineStrengthReduce.cpp | 1160 ++++-------------
.../AMDGPU/agpr-copy-no-free-registers.ll | 20 +-
.../AMDGPU/dagcombine-reassociate-bug.ll | 2 +-
llvm/test/CodeGen/AMDGPU/idot2.ll | 12 +-
llvm/test/CodeGen/AMDGPU/idot4s.ll | 161 ++-
llvm/test/CodeGen/AMDGPU/idot8u.ll | 6 +-
.../AMDGPU/promote-constOffset-to-imm.ll | 467 +++----
.../AMDGPU/splitkit-getsubrangeformask.ll | 18 +-
llvm/test/CodeGen/AMDGPU/waitcnt-vscnt.ll | 329 +++--
.../AMDGPU/pr23975.ll | 2 +-
.../reassociate-geps-and-slsr-addrspace.ll | 10 +-
.../NVPTX/slsr-i8-gep.ll | 271 ----
.../NVPTX/slsr-invalid.ll | 140 --
.../NVPTX/slsr-var-delta.ll | 70 -
.../path-compression.ll | 35 -
.../pick-candidate.ll | 32 -
.../StraightLineStrengthReduce/slsr-add.ll | 120 --
.../StraightLineStrengthReduce/slsr-gep.ll | 163 +--
18 files changed, 828 insertions(+), 2190 deletions(-)
delete mode 100644 llvm/test/Transforms/StraightLineStrengthReduce/NVPTX/slsr-i8-gep.ll
delete mode 100644 llvm/test/Transforms/StraightLineStrengthReduce/NVPTX/slsr-invalid.ll
delete mode 100644 llvm/test/Transforms/StraightLineStrengthReduce/NVPTX/slsr-var-delta.ll
delete mode 100644 llvm/test/Transforms/StraightLineStrengthReduce/path-compression.ll
delete mode 100644 llvm/test/Transforms/StraightLineStrengthReduce/pick-candidate.ll
diff --git a/llvm/lib/Transforms/Scalar/StraightLineStrengthReduce.cpp b/llvm/lib/Transforms/Scalar/StraightLineStrengthReduce.cpp
index c371a9d2f6416..259c747b4a5d9 100644
--- a/llvm/lib/Transforms/Scalar/StraightLineStrengthReduce.cpp
+++ b/llvm/lib/Transforms/Scalar/StraightLineStrengthReduce.cpp
@@ -12,16 +12,17 @@
// effective in simplifying arithmetic statements derived from an unrolled loop.
// It can also simplify the logic of SeparateConstOffsetFromGEP.
//
-// There are many optimizations we can perform in the domain of SLSR.
-// We look for strength reduction candidates in the following forms:
+// There are many optimizations we can perform in the domain of SLSR. This file
+// for now contains only an initial step. Specifically, we look for strength
+// reduction candidates in the following forms:
//
-// Form Add: B + i * S
-// Form Mul: (B + i) * S
-// Form GEP: &B[i * S]
+// Form 1: B + i * S
+// Form 2: (B + i) * S
+// Form 3: &B[i * S]
//
// where S is an integer variable, and i is a constant integer. If we found two
// candidates S1 and S2 in the same form and S1 dominates S2, we may rewrite S2
-// in a simpler way with respect to S1 (index delta). For example,
+// in a simpler way with respect to S1. For example,
//
// S1: X = B + i * S
// S2: Y = B + i' * S => X + (i' - i) * S
@@ -34,29 +35,8 @@
//
// Note: (i' - i) * S is folded to the extent possible.
//
-// For Add and GEP forms, we can also rewrite a candidate in a simpler way
-// with respect to other dominating candidates if their B or S are different
-// but other parts are the same. For example,
-//
-// Base Delta:
-// S1: X = B + i * S
-// S2: Y = B' + i * S => X + (B' - B)
-//
-// S1: X = &B [i * S]
-// S2: Y = &B'[i * S] => X + (B' - B)
-//
-// Stride Delta:
-// S1: X = B + i * S
-// S2: Y = B + i * S' => X + i * (S' - S)
-//
-// S1: X = &B[i * S]
-// S2: Y = &B[i * S'] => X + i * (S' - S)
-//
-// PS: Stride delta rewrite on Mul form is usually non-profitable, and Base
-// delta rewrite sometimes is profitable, so we do not support them on Mul.
-//
// This rewriting is in general a good idea. The code patterns we focus on
-// usually come from loop unrolling, so the delta is likely the same
+// usually come from loop unrolling, so (i' - i) * S is likely the same
// across iterations and can be reused. When that happens, the optimized form
// takes only one add starting from the second iteration.
//
@@ -67,14 +47,19 @@
// TODO:
//
// - Floating point arithmetics when fast math is enabled.
+//
+// - SLSR may decrease ILP at the architecture level. Targets that are very
+// sensitive to ILP may want to disable it. Having SLSR to consider ILP is
+// left as future work.
+//
+// - When (i' - i) is constant but i and i' are not, we could still perform
+// SLSR.
#include "llvm/Transforms/Scalar/StraightLineStrengthReduce.h"
#include "llvm/ADT/APInt.h"
#include "llvm/ADT/DepthFirstIterator.h"
-#include "llvm/ADT/SetVector.h"
#include "llvm/ADT/SmallVector.h"
#include "llvm/Analysis/ScalarEvolution.h"
-#include "llvm/Analysis/ScalarEvolutionExpressions.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/Analysis/ValueTracking.h"
#include "llvm/IR/Constants.h"
@@ -101,24 +86,16 @@
#include <cstdint>
#include <limits>
#include <list>
-#include <queue>
#include <vector>
using namespace llvm;
using namespace PatternMatch;
-#define DEBUG_TYPE "slsr"
-
static const unsigned UnknownAddressSpace =
std::numeric_limits<unsigned>::max();
DEBUG_COUNTER(StraightLineStrengthReduceCounter, "slsr-counter",
- "Controls whether rewriteCandidate is executed.");
-
-// Only for testing.
-static cl::opt<bool>
- EnablePoisonReuseGuard("enable-poison-reuse-guard", cl::init(true),
- cl::desc("Enable poison-reuse guard"));
+ "Controls whether rewriteCandidateWithBasis is executed.");
namespace {
@@ -165,23 +142,15 @@ class StraightLineStrengthReduce {
GEP, // &B[..][i * S][..]
};
- enum DKind {
- InvalidDelta, // reserved for the default constructor
- IndexDelta, // Delta is a constant from Index
- BaseDelta, // Delta is a constant or variable from Base
- StrideDelta, // Delta is a constant or variable from Stride
- };
-
Candidate() = default;
Candidate(Kind CT, const SCEV *B, ConstantInt *Idx, Value *S,
- Instruction *I, const SCEV *StrideSCEV)
- : CandidateKind(CT), Base(B), Index(Idx), Stride(S), Ins(I),
- StrideSCEV(StrideSCEV) {}
+ Instruction *I)
+ : CandidateKind(CT), Base(B), Index(Idx), Stride(S), Ins(I) {}
Kind CandidateKind = Invalid;
const SCEV *Base = nullptr;
- // TODO: Swap Index and Stride's name.
+
// Note that Index and Stride of a GEP candidate do not necessarily have the
// same integer type. In that case, during rewriting, Stride will be
// sign-extended or truncated to Index's type.
@@ -208,164 +177,22 @@ class StraightLineStrengthReduce {
// Points to the immediate basis of this candidate, or nullptr if we cannot
// find any basis for this candidate.
Candidate *Basis = nullptr;
-
- DKind DeltaKind = InvalidDelta;
-
- // Store SCEV of Stride to compute delta from different strides
- const SCEV *StrideSCEV = nullptr;
-
- // Points to (Y - X) that will be used to rewrite this candidate.
- Value *Delta = nullptr;
-
- /// Cost model: Evaluate the computational efficiency of the candidate.
- ///
- /// Efficiency levels (higher is better):
- /// ZeroInst (5) - [Variable] or [Const]
- /// OneInstOneVar (4) - [Variable + Const] or [Variable * Const]
- /// OneInstTwoVar (3) - [Variable + Variable] or [Variable * Variable]
- /// TwoInstOneVar (2) - [Const + Const * Variable]
- /// TwoInstTwoVar (1) - [Variable + Const * Variable]
- enum EfficiencyLevel : unsigned {
- Unknown = 0,
- TwoInstTwoVar = 1,
- TwoInstOneVar = 2,
- OneInstTwoVar = 3,
- OneInstOneVar = 4,
- ZeroInst = 5
- };
-
- static EfficiencyLevel
- getComputationEfficiency(Kind CandidateKind, const ConstantInt *Index,
- const Value *Stride, const SCEV *Base = nullptr) {
- bool IsConstantBase = false;
- bool IsZeroBase = false;
- // When evaluating the efficiency of a rewrite, if the Base's SCEV is
- // not available, conservatively assume the base is not constant.
- if (auto *ConstBase = dyn_cast_or_null<SCEVConstant>(Base)) {
- IsConstantBase = true;
- IsZeroBase = ConstBase->getValue()->isZero();
- }
-
- bool IsConstantStride = isa<ConstantInt>(Stride);
- bool IsZeroStride =
- IsConstantStride && cast<ConstantInt>(Stride)->isZero();
- // All constants
- if (IsConstantBase && IsConstantStride)
- return ZeroInst;
-
- // (Base + Index) * Stride
- if (CandidateKind == Mul) {
- if (IsZeroStride)
- return ZeroInst;
- if (Index->isZero())
- return (IsConstantStride || IsConstantBase) ? OneInstOneVar
- : OneInstTwoVar;
-
- if (IsConstantBase)
- return IsZeroBase && (Index->isOne() || Index->isMinusOne())
- ? ZeroInst
- : OneInstOneVar;
-
- if (IsConstantStride) {
- auto *CI = cast<ConstantInt>(Stride);
- return (CI->isOne() || CI->isMinusOne()) ? OneInstOneVar
- : TwoInstOneVar;
- }
- return TwoInstTwoVar;
- }
-
- // Base + Index * Stride
- assert(CandidateKind == Add || CandidateKind == GEP);
- if (Index->isZero() || IsZeroStride)
- return ZeroInst;
-
- bool IsSimpleIndex = Index->isOne() || Index->isMinusOne();
-
- if (IsConstantBase)
- return IsZeroBase ? (IsSimpleIndex ? ZeroInst : OneInstOneVar)
- : (IsSimpleIndex ? OneInstOneVar : TwoInstOneVar);
-
- if (IsConstantStride)
- return IsZeroStride ? ZeroInst : OneInstOneVar;
-
- if (IsSimpleIndex)
- return OneInstTwoVar;
-
- return TwoInstTwoVar;
- }
-
- // Evaluate if the given delta is profitable to rewrite this candidate.
- bool isProfitableRewrite(const Value &Delta, const DKind DeltaKind) const {
- // This function cannot accurately evaluate the profit of whole expression
- // with context. A candidate (B + I * S) cannot express whether this
- // instruction needs to compute on its own (I * S), which may be shared
- // with other candidates or may need instructions to compute.
- // If the rewritten form has the same strength, still rewrite to
- // (X + Delta) since it may expose more CSE opportunities on Delta, as
- // unrolled loops usually have identical Delta for each unrolled body.
- //
- // Note, this function should only be used on Index Delta rewrite.
- // Base and Stride delta need context info to evaluate the register
- // pressure impact from variable delta.
- return getComputationEfficiency(CandidateKind, Index, Stride, Base) <=
- getRewriteEfficiency(Delta, DeltaKind);
- }
-
- // Evaluate the rewrite efficiency of this candidate with its Basis
- EfficiencyLevel getRewriteEfficiency() const {
- return Basis ? getRewriteEfficiency(*Delta, DeltaKind) : Unknown;
- }
-
- // Evaluate the rewrite efficiency of this candidate with a given delta
- EfficiencyLevel getRewriteEfficiency(const Value &Delta,
- const DKind DeltaKind) const {
- switch (DeltaKind) {
- case BaseDelta: // [X + Delta]
- return getComputationEfficiency(
- CandidateKind,
- ConstantInt::get(cast<IntegerType>(Delta.getType()), 1), &Delta);
- case StrideDelta: // [X + Index * Delta]
- return getComputationEfficiency(CandidateKind, Index, &Delta);
- case IndexDelta: // [X + Delta * Stride]
- return getComputationEfficiency(CandidateKind,
- cast<ConstantInt>(&Delta), Stride);
- default:
- return Unknown;
- }
- }
-
- bool isHighEfficiency() const {
- return getComputationEfficiency(CandidateKind, Index, Stride, Base) >=
- OneInstOneVar;
- }
-
- // Verify that this candidate has valid delta components relative to the
- // basis
- bool hasValidDelta(const Candidate &Basis) const {
- switch (DeltaKind) {
- case IndexDelta:
- // Index differs, Base and Stride must match
- return Base == Basis.Base && StrideSCEV == Basis.StrideSCEV;
- case StrideDelta:
- // Stride differs, Base and Index must match
- return Base == Basis.Base && Index == Basis.Index;
- case BaseDelta:
- // Base differs, Stride and Index must match
- return StrideSCEV == Basis.StrideSCEV && Index == Basis.Index;
- default:
- return false;
- }
- }
};
bool runOnFunction(Function &F);
private:
- // Fetch straight-line basis for rewriting C, update C.Basis to point to it,
- // and store the delta between C and its Basis in C.Delta.
- void setBasisAndDeltaFor(Candidate &C);
+ // Returns true if Basis is a basis for C, i.e., Basis dominates C and they
+ // share the same base and stride.
+ bool isBasisFor(const Candidate &Basis, const Candidate &C);
+
// Returns whether the candidate can be folded into an addressing mode.
- bool isFoldable(const Candidate &C, TargetTransformInfo *TTI);
+ bool isFoldable(const Candidate &C, TargetTransformInfo *TTI,
+ const DataLayout *DL);
+
+ // Returns true if C is already in a simplest form and not worth being
+ // rewritten.
+ bool isSimplestForm(const Candidate &C);
// Checks whether I is in a candidate form. If so, adds all the matching forms
// to Candidates, and tries to find the immediate basis for each of them.
@@ -389,6 +216,12 @@ class StraightLineStrengthReduce {
// Allocate candidates and find bases for GetElementPtr instructions.
void allocateCandidatesAndFindBasisForGEP(GetElementPtrInst *GEP);
+ // A helper function that scales Idx with ElementSize before invoking
+ // allocateCandidatesAndFindBasis.
+ void allocateCandidatesAndFindBasisForGEP(const SCEV *B, ConstantInt *Idx,
+ Value *S, uint64_t ElementSize,
+ Instruction *I);
+
// Adds the given form <CT, B, Idx, S> to Candidates, and finds its immediate
// basis.
void allocateCandidatesAndFindBasis(Candidate::Kind CT, const SCEV *B,
@@ -396,7 +229,13 @@ class StraightLineStrengthReduce {
Instruction *I);
// Rewrites candidate C with respect to Basis.
- void rewriteCandidate(const Candidate &C);
+ void rewriteCandidateWithBasis(const Candidate &C, const Candidate &Basis);
+
+ // A helper function that factors ArrayIdx to a product of a stride and a
+ // constant index, and invokes allocateCandidatesAndFindBasis with the
+ // factorings.
+ void factorArrayIndex(Value *ArrayIdx, const SCEV *Base, uint64_t ElementSize,
+ GetElementPtrInst *GEP);
// Emit code that computes the "bump" from Basis to C.
static Value *emitBump(const Candidate &Basis, const Candidate &C,
@@ -408,209 +247,12 @@ class StraightLineStrengthReduce {
TargetTransformInfo *TTI = nullptr;
std::list<Candidate> Candidates;
- // Map from SCEV to instructions that represent the value,
- // instructions are sorted in depth-first order.
- DenseMap<const SCEV *, SmallSetVector<Instruction *, 2>> SCEVToInsts;
-
- // Record the dependency between instructions. If C.Basis == B, we would have
- // {B.Ins -> {C.Ins, ...}}.
- MapVector<Instruction *, std::vector<Instruction *>> DependencyGraph;
-
- // Map between each instruction and its possible candidates.
- DenseMap<Instruction *, SmallVector<Candidate *, 3>> RewriteCandidates;
-
- // All instructions that have candidates sort in topological order based on
- // dependency graph, from roots to leaves.
- std::vector<Instruction *> SortedCandidateInsts;
-
- // Record all instructions that are already rewritten and will be removed
- // later.
- std::vector<Instruction *> DeadInstructions;
-
- // Classify candidates against Delta kind
- class CandidateDictTy {
- public:
- using CandsTy = SmallVector<Candidate *, 8>;
- using BBToCandsTy = DenseMap<const BasicBlock *, CandsTy>;
-
- private:
- // Index delta Basis must have the same (Base, StrideSCEV, Inst.Type)
- using IndexDeltaKeyTy = std::tuple<const SCEV *, const SCEV *, Type *>;
- DenseMap<IndexDeltaKeyTy, BBToCandsTy> IndexDeltaCandidates;
-
- // Base delta Basis must have the same (StrideSCEV, Index, Inst.Type)
- using BaseDeltaKeyTy = std::tuple<const SCEV *, ConstantInt *, Type *>;
- DenseMap<BaseDeltaKeyTy, BBToCandsTy> BaseDeltaCandidates;
-
- // Stride delta Basis must have the same (Base, Index, Inst.Type)
- using StrideDeltaKeyTy = std::tuple<const SCEV *, ConstantInt *, Type *>;
- DenseMap<StrideDeltaKeyTy, BBToCandsTy> StrideDeltaCandidates;
-
- public:
- // TODO: Disable index delta on GEP after we completely move
- // from typed GEP to PtrAdd.
- const BBToCandsTy *getCandidatesWithDeltaKind(const Candidate &C,
- Candidate::DKind K) const {
- assert(K != Candidate::InvalidDelta);
- if (K == Candidate::IndexDelta) {
- IndexDeltaKeyTy IndexDeltaKey(C.Base, C.StrideSCEV, C.Ins->getType());
- auto It = IndexDeltaCandidates.find(IndexDeltaKey);
- if (It != IndexDeltaCandidates.end())
- return &It->second;
- } else if (K == Candidate::BaseDelta) {
- BaseDeltaKeyTy BaseDeltaKey(C.StrideSCEV, C.Index, C.Ins->getType());
- auto It = BaseDeltaCandidates.find(BaseDeltaKey);
- if (It != BaseDeltaCandidates.end())
- return &It->second;
- } else {
- assert(K == Candidate::StrideDelta);
- StrideDeltaKeyTy StrideDeltaKey(C.Base, C.Index, C.Ins->getType());
- auto It = StrideDeltaCandidates.find(StrideDeltaKey);
- if (It != StrideDeltaCandidates.end())
- return &It->second;
- }
- return nullptr;
- }
-
- // Pointers to C must remain valid until CandidateDict is cleared.
- void add(Candidate &C) {
- Type *ValueType = C.Ins->getType();
- BasicBlock *BB = C.Ins->getParent();
- IndexDeltaKeyTy IndexDeltaKey(C.Base, C.StrideSCEV, ValueType);
- BaseDeltaKeyTy BaseDeltaKey(C.StrideSCEV, C.Index, ValueType);
- StrideDeltaKeyTy StrideDeltaKey(C.Base, C.Index, ValueType);
- IndexDeltaCandidates[IndexDeltaKey][BB].push_back(&C);
- BaseDeltaCandidates[BaseDeltaKey][BB].push_back(&C);
- StrideDeltaCandidates[StrideDeltaKey][BB].push_back(&C);
- }
- // Remove all mappings from set
- void clear() {
- IndexDeltaCandidates.clear();
- BaseDeltaCandidates.clear();
- StrideDeltaCandidates.clear();
- }
- } CandidateDict;
-
- const SCEV *getAndRecordSCEV(Value *V) {
- auto *S = SE->getSCEV(V);
- if (isa<Instruction>(V) && !(isa<SCEVCouldNotCompute>(S) ||
- isa<SCEVUnknown>(S) || isa<SCEVConstant>(S)))
- SCEVToInsts[S].insert(cast<Instruction>(V));
-
- return S;
- }
-
- bool candidatePredicate(Candidate *Basis, Candidate &C, Candidate::DKind K);
-
- bool searchFrom(const CandidateDictTy::BBToCandsTy &BBToCands, Candidate &C,
- Candidate::DKind K);
-
- // Get the nearest instruction before CI that represents the value of S,
- // return nullptr if no instruction is associated with S or S is not a
- // reusable expression.
- Value *getNearestValueOfSCEV(const SCEV *S, const Instruction *CI) const {
- if (isa<SCEVCouldNotCompute>(S))
- return nullptr;
-
- if (auto *SU = dyn_cast<SCEVUnknown>(S))
- return SU->getValue();
- if (auto *SC = dyn_cast<SCEVConstant>(S))
- return SC->getValue();
-
- auto It = SCEVToInsts.find(S);
- if (It == SCEVToInsts.end())
- return nullptr;
-
- // Instructions are sorted in depth-first order, so search for the nearest
- // instruction by walking the list in reverse order.
- for (Instruction *I : reverse(It->second))
- if (DT->dominates(I, CI))
- return I;
-
- return nullptr;
- }
-
- struct DeltaInfo {
- Candidate *Cand;
- Candidate::DKind DeltaKind;
- Value *Delta;
-
- DeltaInfo()
- : Cand(nullptr), DeltaKind(Candidate::InvalidDelta), Delta(nullptr) {}
- DeltaInfo(Candidate *Cand, Candidate::DKind DeltaKind, Value *Delta)
- : Cand(Cand), DeltaKind(DeltaKind), Delta(Delta) {}
- operator bool() const { return Cand != nullptr; }
- };
-
- friend raw_ostream &operator<<(raw_ostream &OS, const DeltaInfo &DI);
-
- DeltaInfo compressPath(Candidate &C, Candidate *Basis) const;
-
- Candidate *pickRewriteCandidate(Instruction *I) const;
- void sortCandidateInstructions();
- Value *getDelta(const Candidate &C, const Candidate &Basis,
- Candidate::DKind K) const;
- static bool isSimilar(Candidate &C, Candidate &Basis, Candidate::DKind K);
-
- // Add Basis -> C in DependencyGraph and propagate
- // C.Stride and C.Delta's dependency to C
- void addDependency(Candidate &C, Candidate *Basis) {
- if (Basis)
- DependencyGraph[Basis->Ins].emplace_back(C.Ins);
-
- // If any candidate of Inst has a basis, then Inst will be rewritten,
- // C must be rewritten after rewriting Inst, so we need to propagate
- // the dependency to C
- auto PropagateDependency = [&](Instruction *Inst) {
- if (auto CandsIt = RewriteCandidates.find(Inst);
- CandsIt != RewriteCandidates.end() &&
- llvm::any_of(CandsIt->second,
- [](Candidate *Cand) { return Cand->Basis; }))
- DependencyGraph[Inst].emplace_back(C.Ins);
- };
-
- // If C has a variable delta and the delta is a candidate,
- // propagate its dependency to C
- if (auto *DeltaInst = dyn_cast_or_null<Instruction>(C.Delta))
- PropagateDependency(DeltaInst);
-
- // If the stride is a candidate, propagate its dependency to C
- if (auto *StrideInst = dyn_cast<Instruction>(C.Stride))
- PropagateDependency(StrideInst);
- };
+ // Temporarily holds all instructions that are unlinked (but not deleted) by
+ // rewriteCandidateWithBasis. These instructions will be actually removed
+ // after all rewriting finishes.
+ std::vector<Instruction *> UnlinkedInstructions;
};
-inline raw_ostream &operator<<(raw_ostream &OS,
- const StraightLineStrengthReduce::Candidate &C) {
- OS << "Ins: " << *C.Ins << "\n Base: " << *C.Base
- << "\n Index: " << *C.Index << "\n Stride: " << *C.Stride
- << "\n StrideSCEV: " << *C.StrideSCEV;
- if (C.Basis)
- OS << "\n Delta: " << *C.Delta << "\n Basis: \n [ " << *C.Basis << " ]";
- return OS;
-}
-
-[[maybe_unused]] LLVM_DUMP_METHOD inline raw_ostream &
-operator<<(raw_ostream &OS, const StraightLineStrengthReduce::DeltaInfo &DI) {
- OS << "Cand: " << *DI.Cand << "\n";
- OS << "Delta Kind: ";
- switch (DI.DeltaKind) {
- case StraightLineStrengthReduce::Candidate::IndexDelta:
- OS << "Index";
- break;
- case StraightLineStrengthReduce::Candidate::BaseDelta:
- OS << "Base";
- break;
- case StraightLineStrengthReduce::Candidate::StrideDelta:
- OS << "Stride";
- break;
- default:
- break;
- }
- OS << "\nDelta: " << *DI.Delta;
- return OS;
-}
-
} // end anonymous namespace
char StraightLineStrengthReduceLegacyPass::ID = 0;
@@ -627,301 +269,17 @@ FunctionPass *llvm::createStraightLineStrengthReducePass() {
return new StraightLineStrengthReduceLegacyPass();
}
-// A helper function that unifies the bitwidth of A and B.
-static void unifyBitWidth(APInt &A, APInt &B) {
- if (A.getBitWidth() < B.getBitWidth())
- A = A.sext(B.getBitWidth());
- else if (A.getBitWidth() > B.getBitWidth())
- B = B.sext(A.getBitWidth());
-}
-
-Value *StraightLineStrengthReduce::getDelta(const Candidate &C,
- const Candidate &Basis,
- Candidate::DKind K) const {
- if (K == Candidate::IndexDelta) {
- APInt Idx = C.Index->getValue();
- APInt BasisIdx = Basis.Index->getValue();
- unifyBitWidth(Idx, BasisIdx);
- APInt IndexDelta = Idx - BasisIdx;
- IntegerType *DeltaType =
- IntegerType::get(C.Ins->getContext(), IndexDelta.getBitWidth());
- return ConstantInt::get(DeltaType, IndexDelta);
- } else if (K == Candidate::BaseDelta || K == Candidate::StrideDelta) {
- const SCEV *BasisPart =
- (K == Candidate::BaseDelta) ? Basis.Base : Basis.StrideSCEV;
- const SCEV *CandPart = (K == Candidate::BaseDelta) ? C.Base : C.StrideSCEV;
- const SCEV *Diff = SE->getMinusSCEV(CandPart, BasisPart);
- return getNearestValueOfSCEV(Diff, C.Ins);
- }
- return nullptr;
-}
-
-bool StraightLineStrengthReduce::isSimilar(Candidate &C, Candidate &Basis,
- Candidate::DKind K) {
- bool SameType = false;
- switch (K) {
- case Candidate::StrideDelta:
- SameType = C.StrideSCEV->getType() == Basis.StrideSCEV->getType();
- break;
- case Candidate::BaseDelta:
- SameType = C.Base->getType() == Basis.Base->getType();
- break;
- case Candidate::IndexDelta:
- SameType = true;
- break;
- default:;
- }
- return SameType && Basis.Ins != C.Ins &&
- Basis.CandidateKind == C.CandidateKind;
-}
-
-// Try to find a Delta that C can reuse Basis to rewrite.
-// Set C.Delta, C.Basis, and C.DeltaKind if found.
-// Return true if found a constant delta.
-// Return false if not found or the delta is not a constant.
-bool StraightLineStrengthReduce::candidatePredicate(Candidate *Basis,
- Candidate &C,
- Candidate::DKind K) {
- SmallVector<Instruction *> DropPoisonGeneratingInsts;
- // Ensure the IR of Basis->Ins is not more poisonous than its SCEV.
- if (!isSimilar(C, *Basis, K) ||
- (EnablePoisonReuseGuard &&
- !SE->canReuseInstruction(SE->getSCEV(Basis->Ins), Basis->Ins,
- DropPoisonGeneratingInsts)))
- return false;
-
- assert(DT->dominates(Basis->Ins, C.Ins));
- Value *Delta = getDelta(C, *Basis, K);
- if (!Delta)
- return false;
-
- // IndexDelta rewrite is not always profitable, e.g.,
- // X = B + 8 * S
- // Y = B + S,
- // rewriting Y to X - 7 * S is probably a bad idea.
- // So, we need to check if the rewrite form's computation efficiency
- // is better than the original form.
- if (K == Candidate::IndexDelta &&
- !C.isProfitableRewrite(*Delta, Candidate::IndexDelta))
- return false;
-
- // If there is a Delta that we can reuse Basis to rewrite C,
- // clean up DropPoisonGeneratingInsts returned by successful
- // SE->canReuseInstruction()
- for (Instruction *I : DropPoisonGeneratingInsts)
- I->dropPoisonGeneratingAnnotations();
-
- // Record delta if none has been found yet, or the new delta is
- // a constant that is better than the existing delta.
- if (!C.Delta || isa<ConstantInt>(Delta)) {
- C.Delta = Delta;
- C.Basis = Basis;
- C.DeltaKind = K;
- }
- return isa<ConstantInt>(C.Delta);
-}
-
-// return true if find a Basis with constant delta and stop searching,
-// return false if did not find a Basis or the delta is not a constant
-// and continue searching for a Basis with constant delta
-bool StraightLineStrengthReduce::searchFrom(
- const CandidateDictTy::BBToCandsTy &BBToCands, Candidate &C,
- Candidate::DKind K) {
-
- // Stride delta rewrite on Mul form is usually non-profitable, and Base
- // delta rewrite sometimes is profitable, so we do not support them on Mul.
- if (C.CandidateKind == Candidate::Mul && K != Candidate::IndexDelta)
- return false;
-
- // Search dominating candidates by walking the immediate-dominator chain
- // from the candidate's defining block upward. Visiting blocks in this
- // order ensures we prefer the closest dominating basis.
- const BasicBlock *BB = C.Ins->getParent();
- while (BB) {
- auto It = BBToCands.find(BB);
- if (It != BBToCands.end())
- for (Candidate *Basis : reverse(It->second))
- if (candidatePredicate(Basis, C, K))
- return true;
-
- const DomTreeNode *Node = DT->getNode(BB);
- if (!Node)
- break;
- Node = Node->getIDom();
- BB = Node ? Node->getBlock() : nullptr;
- }
- return false;
-}
-
-void StraightLineStrengthReduce::setBasisAndDeltaFor(Candidate &C) {
- if (const auto *BaseDeltaCandidates =
- CandidateDict.getCandidatesWithDeltaKind(C, Candidate::BaseDelta))
- if (searchFrom(*BaseDeltaCandidates, C, Candidate::BaseDelta)) {
- LLVM_DEBUG(dbgs() << "Found delta from Base: " << *C.Delta << "\n");
- return;
- }
-
- if (const auto *StrideDeltaCandidates =
- CandidateDict.getCandidatesWithDeltaKind(C, Candidate::StrideDelta))
- if (searchFrom(*StrideDeltaCandidates, C, Candidate::StrideDelta)) {
- LLVM_DEBUG(dbgs() << "Found delta from Stride: " << *C.Delta << "\n");
- return;
- }
-
- if (const auto *IndexDeltaCandidates =
- CandidateDict.getCandidatesWithDeltaKind(C, Candidate::IndexDelta))
- if (searchFrom(*IndexDeltaCandidates, C, Candidate::IndexDelta)) {
- LLVM_DEBUG(dbgs() << "Found delta from Index: " << *C.Delta << "\n");
- return;
- }
-
- // If we did not find a constant delta, we might have found a variable delta
- if (C.Delta) {
- LLVM_DEBUG({
- dbgs() << "Found delta from ";
- if (C.DeltaKind == Candidate::BaseDelta)
- dbgs() << "Base: ";
- else
- dbgs() << "Stride: ";
- dbgs() << *C.Delta << "\n";
- });
- assert(C.DeltaKind != Candidate::InvalidDelta && C.Basis);
- }
-}
-
-// Compress the path from `Basis` to the deepest Basis in the Basis chain
-// to avoid non-profitable data dependency and improve ILP.
-// X = A + 1
-// Y = X + 1
-// Z = Y + 1
-// ->
-// X = A + 1
-// Y = A + 2
-// Z = A + 3
-// Return the delta info for C aginst the new Basis
-auto StraightLineStrengthReduce::compressPath(Candidate &C,
- Candidate *Basis) const
- -> DeltaInfo {
- if (!Basis || !Basis->Basis || C.CandidateKind == Candidate::Mul)
- return {};
- Candidate *Root = Basis;
- Value *NewDelta = nullptr;
- auto NewKind = Candidate::InvalidDelta;
-
- while (Root->Basis) {
- Candidate *NextRoot = Root->Basis;
- if (C.Base == NextRoot->Base && C.StrideSCEV == NextRoot->StrideSCEV &&
- isSimilar(C, *NextRoot, Candidate::IndexDelta)) {
- ConstantInt *CI =
- cast<ConstantInt>(getDelta(C, *NextRoot, Candidate::IndexDelta));
- if (CI->isZero() || CI->isOne() || isa<SCEVConstant>(C.StrideSCEV)) {
- Root = NextRoot;
- NewKind = Candidate::IndexDelta;
- NewDelta = CI;
- continue;
- }
- }
-
- const SCEV *CandPart = nullptr;
- const SCEV *BasisPart = nullptr;
- auto CurrKind = Candidate::InvalidDelta;
- if (C.Base == NextRoot->Base && C.Index == NextRoot->Index) {
- CandPart = C.StrideSCEV;
- BasisPart = NextRoot->StrideSCEV;
- CurrKind = Candidate::StrideDelta;
- } else if (C.StrideSCEV == NextRoot->StrideSCEV &&
- C.Index == NextRoot->Index) {
- CandPart = C.Base;
- BasisPart = NextRoot->Base;
- CurrKind = Candidate::BaseDelta;
- } else
- break;
-
- assert(CandPart && BasisPart);
- if (!isSimilar(C, *NextRoot, CurrKind))
- break;
-
- if (auto DeltaVal =
- dyn_cast<SCEVConstant>(SE->getMinusSCEV(CandPart, BasisPart))) {
- Root = NextRoot;
- NewDelta = DeltaVal->getValue();
- NewKind = CurrKind;
- } else
- break;
- }
-
- if (Root != Basis) {
- assert(NewKind != Candidate::InvalidDelta && NewDelta);
- LLVM_DEBUG(dbgs() << "Found new Basis with " << *NewDelta
- << " from path compression.\n");
- return {Root, NewKind, NewDelta};
- }
-
- return {};
-}
-
-// Topologically sort candidate instructions based on their relationship in
-// dependency graph.
-void StraightLineStrengthReduce::sortCandidateInstructions() {
- SortedCandidateInsts.clear();
- // An instruction may have multiple candidates that get different Basis
- // instructions, and each candidate can get dependencies from Basis and
- // Stride when Stride will also be rewritten by SLSR. Hence, an instruction
- // may have multiple dependencies. Use InDegree to ensure all dependencies
- // processed before processing itself.
- DenseMap<Instruction *, int> InDegree;
- for (auto &KV : DependencyGraph) {
- InDegree.try_emplace(KV.first, 0);
-
- for (auto *Child : KV.second) {
- InDegree[Child]++;
- }
- }
- std::queue<Instruction *> WorkList;
- DenseSet<Instruction *> Visited;
-
- for (auto &KV : DependencyGraph)
- if (InDegree[KV.first] == 0)
- WorkList.push(KV.first);
-
- while (!WorkList.empty()) {
- Instruction *I = WorkList.front();
- WorkList.pop();
- if (!Visited.insert(I).second)
- continue;
-
- SortedCandidateInsts.push_back(I);
-
- for (auto *Next : DependencyGraph[I]) {
- auto &Degree = InDegree[Next];
- if (--Degree == 0)
- WorkList.push(Next);
- }
- }
-
- assert(SortedCandidateInsts.size() == DependencyGraph.size() &&
- "Dependency graph should not have cycles");
-}
-
-auto StraightLineStrengthReduce::pickRewriteCandidate(Instruction *I) const
- -> Candidate * {
- // Return the candidate of instruction I that has the highest profit.
- auto It = RewriteCandidates.find(I);
- if (It == RewriteCandidates.end())
- return nullptr;
-
- Candidate *BestC = nullptr;
- auto BestEfficiency = Candidate::Unknown;
- for (Candidate *C : reverse(It->second))
- if (C->Basis) {
- auto Efficiency = C->getRewriteEfficiency();
- if (Efficiency > BestEfficiency) {
- BestEfficiency = Efficiency;
- BestC = C;
- }
- }
-
- return BestC;
+bool StraightLineStrengthReduce::isBasisFor(const Candidate &Basis,
+ const Candidate &C) {
+ return (Basis.Ins != C.Ins && // skip the same instruction
+ // They must have the same type too. Basis.Base == C.Base
+ // doesn't guarantee their types are the same (PR23975).
+ Basis.Ins->getType() == C.Ins->getType() &&
+ // Basis must dominate C in order to rewrite C with respect to Basis.
+ DT->dominates(Basis.Ins->getParent(), C.Ins->getParent()) &&
+ // They share the same base, stride, and candidate kind.
+ Basis.Base == C.Base && Basis.Stride == C.Stride &&
+ Basis.CandidateKind == C.CandidateKind);
}
static bool isGEPFoldable(GetElementPtrInst *GEP,
@@ -941,7 +299,8 @@ static bool isAddFoldable(const SCEV *Base, ConstantInt *Index, Value *Stride,
}
bool StraightLineStrengthReduce::isFoldable(const Candidate &C,
- TargetTransformInfo *TTI) {
+ TargetTransformInfo *TTI,
+ const DataLayout *DL) {
if (C.CandidateKind == Candidate::Add)
return isAddFoldable(C.Base, C.Index, C.Stride, TTI);
if (C.CandidateKind == Candidate::GEP)
@@ -949,39 +308,75 @@ bool StraightLineStrengthReduce::isFoldable(const Candidate &C,
return false;
}
+// Returns true if GEP has zero or one non-zero index.
+static bool hasOnlyOneNonZeroIndex(GetElementPtrInst *GEP) {
+ unsigned NumNonZeroIndices = 0;
+ for (Use &Idx : GEP->indices()) {
+ ConstantInt *ConstIdx = dyn_cast<ConstantInt>(Idx);
+ if (ConstIdx == nullptr || !ConstIdx->isZero())
+ ++NumNonZeroIndices;
+ }
+ return NumNonZeroIndices <= 1;
+}
+
+bool StraightLineStrengthReduce::isSimplestForm(const Candidate &C) {
+ if (C.CandidateKind == Candidate::Add) {
+ // B + 1 * S or B + (-1) * S
+ return C.Index->isOne() || C.Index->isMinusOne();
+ }
+ if (C.CandidateKind == Candidate::Mul) {
+ // (B + 0) * S
+ return C.Index->isZero();
+ }
+ if (C.CandidateKind == Candidate::GEP) {
+ // (char*)B + S or (char*)B - S
+ return ((C.Index->isOne() || C.Index->isMinusOne()) &&
+ hasOnlyOneNonZeroIndex(cast<GetElementPtrInst>(C.Ins)));
+ }
+ return false;
+}
+
+// TODO: We currently implement an algorithm whose time complexity is linear in
+// the number of existing candidates. However, we could do better by using
+// ScopedHashTable. Specifically, while traversing the dominator tree, we could
+// maintain all the candidates that dominate the basic block being traversed in
+// a ScopedHashTable. This hash table is indexed by the base and the stride of
+// a candidate. Therefore, finding the immediate basis of a candidate boils down
+// to one hash-table look up.
void StraightLineStrengthReduce::allocateCandidatesAndFindBasis(
Candidate::Kind CT, const SCEV *B, ConstantInt *Idx, Value *S,
Instruction *I) {
- // Record the SCEV of S that we may use it as a variable delta.
- // Ensure that we rewrite C with a existing IR that reproduces delta value.
-
- Candidate C(CT, B, Idx, S, I, getAndRecordSCEV(S));
- // If we can fold I into an addressing mode, computing I is likely free or
- // takes only one instruction. So, we don't need to analyze or rewrite it.
+ Candidate C(CT, B, Idx, S, I);
+ // SLSR can complicate an instruction in two cases:
+ //
+ // 1. If we can fold I into an addressing mode, computing I is likely free or
+ // takes only one instruction.
+ //
+ // 2. I is already in a simplest form. For example, when
+ // X = B + 8 * S
+ // Y = B + S,
+ // rewriting Y to X - 7 * S is probably a bad idea.
//
- // Currently, this algorithm can at best optimize complex computations into
- // a `variable +/* constant` form. However, some targets have stricter
- // constraints on the their addressing mode.
- // For example, a `variable + constant` can only be folded to an addressing
- // mode if the constant falls within a certain range.
- // So, we also check if the instruction is already high efficient enough
- // for the strength reduction algorithm.
- if (!isFoldable(C, TTI) && !C.isHighEfficiency()) {
- setBasisAndDeltaFor(C);
-
- // Compress unnecessary rewrite to improve ILP
- if (auto Res = compressPath(C, C.Basis)) {
- C.Basis = Res.Cand;
- C.DeltaKind = Res.DeltaKind;
- C.Delta = Res.Delta;
+ // In the above cases, we still add I to the candidate list so that I can be
+ // the basis of other candidates, but we leave I's basis blank so that I
+ // won't be rewritten.
+ if (!isFoldable(C, TTI, DL) && !isSimplestForm(C)) {
+ // Try to compute the immediate basis of C.
+ unsigned NumIterations = 0;
+ // Limit the scan radius to avoid running in quadratice time.
+ static const unsigned MaxNumIterations = 50;
+ for (auto Basis = Candidates.rbegin();
+ Basis != Candidates.rend() && NumIterations < MaxNumIterations;
+ ++Basis, ++NumIterations) {
+ if (isBasisFor(*Basis, C)) {
+ C.Basis = &(*Basis);
+ break;
+ }
}
}
// Regardless of whether we find a basis for C, we need to push C to the
// candidate list so that it can be the basis of other candidates.
- LLVM_DEBUG(dbgs() << "Allocated Candidate: " << C << "\n");
Candidates.push_back(C);
- RewriteCandidates[C.Ins].push_back(&Candidates.back());
- CandidateDict.add(Candidates.back());
}
void StraightLineStrengthReduce::allocateCandidatesAndFindBasis(
@@ -1080,6 +475,54 @@ void StraightLineStrengthReduce::allocateCandidatesAndFindBasisForMul(
}
}
+void StraightLineStrengthReduce::allocateCandidatesAndFindBasisForGEP(
+ const SCEV *B, ConstantInt *Idx, Value *S, uint64_t ElementSize,
+ Instruction *I) {
+ // I = B + sext(Idx *nsw S) * ElementSize
+ // = B + (sext(Idx) * sext(S)) * ElementSize
+ // = B + (sext(Idx) * ElementSize) * sext(S)
+ // Casting to IntegerType is safe because we skipped vector GEPs.
+ IntegerType *PtrIdxTy = cast<IntegerType>(DL->getIndexType(I->getType()));
+ ConstantInt *ScaledIdx = ConstantInt::get(
+ PtrIdxTy, Idx->getSExtValue() * (int64_t)ElementSize, true);
+ allocateCandidatesAndFindBasis(Candidate::GEP, B, ScaledIdx, S, I);
+}
+
+void StraightLineStrengthReduce::factorArrayIndex(Value *ArrayIdx,
+ const SCEV *Base,
+ uint64_t ElementSize,
+ GetElementPtrInst *GEP) {
+ // At least, ArrayIdx = ArrayIdx *nsw 1.
+ allocateCandidatesAndFindBasisForGEP(
+ Base, ConstantInt::get(cast<IntegerType>(ArrayIdx->getType()), 1),
+ ArrayIdx, ElementSize, GEP);
+ Value *LHS = nullptr;
+ ConstantInt *RHS = nullptr;
+ // One alternative is matching the SCEV of ArrayIdx instead of ArrayIdx
+ // itself. This would allow us to handle the shl case for free. However,
+ // matching SCEVs has two issues:
+ //
+ // 1. this would complicate rewriting because the rewriting procedure
+ // would have to translate SCEVs back to IR instructions. This translation
+ // is difficult when LHS is further evaluated to a composite SCEV.
+ //
+ // 2. ScalarEvolution is designed to be control-flow oblivious. It tends
+ // to strip nsw/nuw flags which are critical for SLSR to trace into
+ // sext'ed multiplication.
+ if (match(ArrayIdx, m_NSWMul(m_Value(LHS), m_ConstantInt(RHS)))) {
+ // SLSR is currently unsafe if i * S may overflow.
+ // GEP = Base + sext(LHS *nsw RHS) * ElementSize
+ allocateCandidatesAndFindBasisForGEP(Base, RHS, LHS, ElementSize, GEP);
+ } else if (match(ArrayIdx, m_NSWShl(m_Value(LHS), m_ConstantInt(RHS)))) {
+ // GEP = Base + sext(LHS <<nsw RHS) * ElementSize
+ // = Base + sext(LHS *nsw (1 << RHS)) * ElementSize
+ APInt One(RHS->getBitWidth(), 1);
+ ConstantInt *PowerOf2 =
+ ConstantInt::get(RHS->getContext(), One << RHS->getValue());
+ allocateCandidatesAndFindBasisForGEP(Base, PowerOf2, LHS, ElementSize, GEP);
+ }
+}
+
void StraightLineStrengthReduce::allocateCandidatesAndFindBasisForGEP(
GetElementPtrInst *GEP) {
// TODO: handle vector GEPs
@@ -1111,8 +554,7 @@ void StraightLineStrengthReduce::allocateCandidatesAndFindBasisForGEP(
DL->getIndexSizeInBits(GEP->getAddressSpace())) {
// Skip factoring if ArrayIdx is wider than the index size, because
// ArrayIdx is implicitly truncated to the index size.
- allocateCandidatesAndFindBasis(Candidate::GEP, BaseExpr, ElementSizeIdx,
- ArrayIdx, GEP);
+ factorArrayIndex(ArrayIdx, BaseExpr, ElementSize, GEP);
}
// When ArrayIdx is the sext of a value, we try to factor that value as
// well. Handling this case is important because array indices are
@@ -1123,159 +565,118 @@ void StraightLineStrengthReduce::allocateCandidatesAndFindBasisForGEP(
DL->getIndexSizeInBits(GEP->getAddressSpace())) {
// Skip factoring if TruncatedArrayIdx is wider than the pointer size,
// because TruncatedArrayIdx is implicitly truncated to the pointer size.
- allocateCandidatesAndFindBasis(Candidate::GEP, BaseExpr, ElementSizeIdx,
- TruncatedArrayIdx, GEP);
+ factorArrayIndex(TruncatedArrayIdx, BaseExpr, ElementSize, GEP);
}
IndexExprs[I - 1] = OrigIndexExpr;
}
}
+// A helper function that unifies the bitwidth of A and B.
+static void unifyBitWidth(APInt &A, APInt &B) {
+ if (A.getBitWidth() < B.getBitWidth())
+ A = A.sext(B.getBitWidth());
+ else if (A.getBitWidth() > B.getBitWidth())
+ B = B.sext(A.getBitWidth());
+}
+
Value *StraightLineStrengthReduce::emitBump(const Candidate &Basis,
const Candidate &C,
IRBuilder<> &Builder,
const DataLayout *DL) {
- auto CreateMul = [&](Value *LHS, Value *RHS) {
- if (ConstantInt *CR = dyn_cast<ConstantInt>(RHS)) {
- const APInt &ConstRHS = CR->getValue();
- IntegerType *DeltaType =
- IntegerType::get(C.Ins->getContext(), ConstRHS.getBitWidth());
- if (ConstRHS.isPowerOf2()) {
- ConstantInt *Exponent =
- ConstantInt::get(DeltaType, ConstRHS.logBase2());
- return Builder.CreateShl(LHS, Exponent);
- }
- if (ConstRHS.isNegatedPowerOf2()) {
- ConstantInt *Exponent =
- ConstantInt::get(DeltaType, (-ConstRHS).logBase2());
- return Builder.CreateNeg(Builder.CreateShl(LHS, Exponent));
- }
- }
-
- return Builder.CreateMul(LHS, RHS);
- };
-
- Value *Delta = C.Delta;
- // If Delta is 0, C is a fully redundant of C.Basis,
- // just replace C.Ins with Basis.Ins
- if (ConstantInt *CI = dyn_cast<ConstantInt>(Delta);
- CI && CI->getValue().isZero())
- return nullptr;
-
- if (C.DeltaKind == Candidate::IndexDelta) {
- APInt IndexDelta = cast<ConstantInt>(C.Delta)->getValue();
- // IndexDelta
- // X = B + i * S
- // Y = B + i` * S
- // = B + (i + IndexDelta) * S
- // = B + i * S + IndexDelta * S
- // = X + IndexDelta * S
- // Bump = (i' - i) * S
-
- // Common case 1: if (i' - i) is 1, Bump = S.
- if (IndexDelta == 1)
- return C.Stride;
- // Common case 2: if (i' - i) is -1, Bump = -S.
- if (IndexDelta.isAllOnes())
- return Builder.CreateNeg(C.Stride);
-
- IntegerType *DeltaType =
- IntegerType::get(Basis.Ins->getContext(), IndexDelta.getBitWidth());
- Value *ExtendedStride = Builder.CreateSExtOrTrunc(C.Stride, DeltaType);
-
- return CreateMul(ExtendedStride, C.Delta);
+ APInt Idx = C.Index->getValue(), BasisIdx = Basis.Index->getValue();
+ unifyBitWidth(Idx, BasisIdx);
+ APInt IndexOffset = Idx - BasisIdx;
+
+ // Compute Bump = C - Basis = (i' - i) * S.
+ // Common case 1: if (i' - i) is 1, Bump = S.
+ if (IndexOffset == 1)
+ return C.Stride;
+ // Common case 2: if (i' - i) is -1, Bump = -S.
+ if (IndexOffset.isAllOnes())
+ return Builder.CreateNeg(C.Stride);
+
+ // Otherwise, Bump = (i' - i) * sext/trunc(S). Note that (i' - i) and S may
+ // have different bit widths.
+ IntegerType *DeltaType =
+ IntegerType::get(Basis.Ins->getContext(), IndexOffset.getBitWidth());
+ Value *ExtendedStride = Builder.CreateSExtOrTrunc(C.Stride, DeltaType);
+ if (IndexOffset.isPowerOf2()) {
+ // If (i' - i) is a power of 2, Bump = sext/trunc(S) << log(i' - i).
+ ConstantInt *Exponent = ConstantInt::get(DeltaType, IndexOffset.logBase2());
+ return Builder.CreateShl(ExtendedStride, Exponent);
}
-
- assert(C.DeltaKind == Candidate::StrideDelta ||
- C.DeltaKind == Candidate::BaseDelta);
- assert(C.CandidateKind != Candidate::Mul);
- // StrideDelta
- // X = B + i * S
- // Y = B + i * S'
- // = B + i * (S + StrideDelta)
- // = B + i * S + i * StrideDelta
- // = X + i * StrideDelta
- // Bump = i * (S' - S)
- //
- // BaseDelta
- // X = B + i * S
- // Y = B' + i * S
- // = (B + BaseDelta) + i * S
- // = X + BaseDelta
- // Bump = (B' - B).
- Value *Bump = C.Delta;
- if (C.DeltaKind == Candidate::StrideDelta) {
- // If this value is consumed by a GEP, promote StrideDelta before doing
- // StrideDelta * Index to ensure the same semantics as the original GEP.
- if (C.CandidateKind == Candidate::GEP) {
- auto *GEP = cast<GetElementPtrInst>(C.Ins);
- Type *NewScalarIndexTy =
- DL->getIndexType(GEP->getPointerOperandType()->getScalarType());
- Bump = Builder.CreateSExtOrTrunc(Bump, NewScalarIndexTy);
- }
- if (!C.Index->isOne()) {
- Value *ExtendedIndex =
- Builder.CreateSExtOrTrunc(C.Index, Bump->getType());
- Bump = CreateMul(Bump, ExtendedIndex);
- }
+ if (IndexOffset.isNegatedPowerOf2()) {
+ // If (i - i') is a power of 2, Bump = -sext/trunc(S) << log(i' - i).
+ ConstantInt *Exponent =
+ ConstantInt::get(DeltaType, (-IndexOffset).logBase2());
+ return Builder.CreateNeg(Builder.CreateShl(ExtendedStride, Exponent));
}
- return Bump;
+ Constant *Delta = ConstantInt::get(DeltaType, IndexOffset);
+ return Builder.CreateMul(ExtendedStride, Delta);
}
-void StraightLineStrengthReduce::rewriteCandidate(const Candidate &C) {
+void StraightLineStrengthReduce::rewriteCandidateWithBasis(
+ const Candidate &C, const Candidate &Basis) {
if (!DebugCounter::shouldExecute(StraightLineStrengthReduceCounter))
return;
- const Candidate &Basis = *C.Basis;
- assert(C.Delta && C.CandidateKind == Basis.CandidateKind &&
- C.hasValidDelta(Basis));
+ assert(C.CandidateKind == Basis.CandidateKind && C.Base == Basis.Base &&
+ C.Stride == Basis.Stride);
+ // We run rewriteCandidateWithBasis on all candidates in a post-order, so the
+ // basis of a candidate cannot be unlinked before the candidate.
+ assert(Basis.Ins->getParent() != nullptr && "the basis is unlinked");
+
+ // An instruction can correspond to multiple candidates. Therefore, instead of
+ // simply deleting an instruction when we rewrite it, we mark its parent as
+ // nullptr (i.e. unlink it) so that we can skip the candidates whose
+ // instruction is already rewritten.
+ if (!C.Ins->getParent())
+ return;
IRBuilder<> Builder(C.Ins);
Value *Bump = emitBump(Basis, C, Builder, DL);
Value *Reduced = nullptr; // equivalent to but weaker than C.Ins
- // If delta is 0, C is a fully redundant of Basis, and Bump is nullptr,
- // just replace C.Ins with Basis.Ins
- if (!Bump)
- Reduced = Basis.Ins;
- else {
- switch (C.CandidateKind) {
- case Candidate::Add:
- case Candidate::Mul: {
- // C = Basis + Bump
- Value *NegBump;
- if (match(Bump, m_Neg(m_Value(NegBump)))) {
- // If Bump is a neg instruction, emit C = Basis - (-Bump).
- Reduced = Builder.CreateSub(Basis.Ins, NegBump);
- // We only use the negative argument of Bump, and Bump itself may be
- // trivially dead.
- RecursivelyDeleteTriviallyDeadInstructions(Bump);
- } else {
- // It's tempting to preserve nsw on Bump and/or Reduced. However, it's
- // usually unsound, e.g.,
- //
- // X = (-2 +nsw 1) *nsw INT_MAX
- // Y = (-2 +nsw 3) *nsw INT_MAX
- // =>
- // Y = X + 2 * INT_MAX
- //
- // Neither + and * in the resultant expression are nsw.
- Reduced = Builder.CreateAdd(Basis.Ins, Bump);
- }
- break;
- }
- case Candidate::GEP: {
- bool InBounds = cast<GetElementPtrInst>(C.Ins)->isInBounds();
- // C = (char *)Basis + Bump
- Reduced = Builder.CreatePtrAdd(Basis.Ins, Bump, "", InBounds);
- break;
+ switch (C.CandidateKind) {
+ case Candidate::Add:
+ case Candidate::Mul: {
+ // C = Basis + Bump
+ Value *NegBump;
+ if (match(Bump, m_Neg(m_Value(NegBump)))) {
+ // If Bump is a neg instruction, emit C = Basis - (-Bump).
+ Reduced = Builder.CreateSub(Basis.Ins, NegBump);
+ // We only use the negative argument of Bump, and Bump itself may be
+ // trivially dead.
+ RecursivelyDeleteTriviallyDeadInstructions(Bump);
+ } else {
+ // It's tempting to preserve nsw on Bump and/or Reduced. However, it's
+ // usually unsound, e.g.,
+ //
+ // X = (-2 +nsw 1) *nsw INT_MAX
+ // Y = (-2 +nsw 3) *nsw INT_MAX
+ // =>
+ // Y = X + 2 * INT_MAX
+ //
+ // Neither + and * in the resultant expression are nsw.
+ Reduced = Builder.CreateAdd(Basis.Ins, Bump);
}
- default:
- llvm_unreachable("C.CandidateKind is invalid");
- };
- Reduced->takeName(C.Ins);
+ break;
+ }
+ case Candidate::GEP: {
+ bool InBounds = cast<GetElementPtrInst>(C.Ins)->isInBounds();
+ // C = (char *)Basis + Bump
+ Reduced = Builder.CreatePtrAdd(Basis.Ins, Bump, "", InBounds);
+ break;
}
+ default:
+ llvm_unreachable("C.CandidateKind is invalid");
+ };
+ Reduced->takeName(C.Ins);
C.Ins->replaceAllUsesWith(Reduced);
- DeadInstructions.push_back(C.Ins);
+ // Unlink C.Ins so that we can skip other candidates also corresponding to
+ // C.Ins. The actual deletion is postponed to the end of runOnFunction.
+ C.Ins->removeFromParent();
+ UnlinkedInstructions.push_back(C.Ins);
}
bool StraightLineStrengthReduceLegacyPass::runOnFunction(Function &F) {
@@ -1289,42 +690,33 @@ bool StraightLineStrengthReduceLegacyPass::runOnFunction(Function &F) {
}
bool StraightLineStrengthReduce::runOnFunction(Function &F) {
- LLVM_DEBUG(dbgs() << "SLSR on Function: " << F.getName() << "\n");
// Traverse the dominator tree in the depth-first order. This order makes sure
// all bases of a candidate are in Candidates when we process it.
for (const auto Node : depth_first(DT))
for (auto &I : *(Node->getBlock()))
allocateCandidatesAndFindBasis(&I);
- // Build the dependency graph and sort candidate instructions from dependency
- // roots to leaves
- for (auto &C : Candidates) {
- DependencyGraph.try_emplace(C.Ins);
- addDependency(C, C.Basis);
+ // Rewrite candidates in the reverse depth-first order. This order makes sure
+ // a candidate being rewritten is not a basis for any other candidate.
+ while (!Candidates.empty()) {
+ const Candidate &C = Candidates.back();
+ if (C.Basis != nullptr) {
+ rewriteCandidateWithBasis(C, *C.Basis);
+ }
+ Candidates.pop_back();
+ }
+
+ // Delete all unlink instructions.
+ for (auto *UnlinkedInst : UnlinkedInstructions) {
+ for (unsigned I = 0, E = UnlinkedInst->getNumOperands(); I != E; ++I) {
+ Value *Op = UnlinkedInst->getOperand(I);
+ UnlinkedInst->setOperand(I, nullptr);
+ RecursivelyDeleteTriviallyDeadInstructions(Op);
+ }
+ UnlinkedInst->deleteValue();
}
- sortCandidateInstructions();
-
- // Rewrite candidates in the topological order that rewrites a Candidate
- // always before rewriting its Basis
- for (Instruction *I : reverse(SortedCandidateInsts))
- if (Candidate *C = pickRewriteCandidate(I))
- rewriteCandidate(*C);
-
- for (auto *DeadIns : DeadInstructions)
- // A dead instruction may be another dead instruction's op,
- // don't delete an instruction twice
- if (DeadIns->getParent())
- RecursivelyDeleteTriviallyDeadInstructions(DeadIns);
-
- bool Ret = !DeadInstructions.empty();
- DeadInstructions.clear();
- DependencyGraph.clear();
- RewriteCandidates.clear();
- SortedCandidateInsts.clear();
- // First clear all references to candidates in the list
- CandidateDict.clear();
- // Then destroy the list
- Candidates.clear();
+ bool Ret = !UnlinkedInstructions.empty();
+ UnlinkedInstructions.clear();
return Ret;
}
diff --git a/llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll b/llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll
index ef7a13819a799..ebbeab94066d6 100644
--- a/llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll
+++ b/llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll
@@ -541,9 +541,10 @@ define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg
; GFX908-NEXT: s_lshr_b32 s2, s0, 16
; GFX908-NEXT: v_cvt_f32_f16_e32 v19, s2
; GFX908-NEXT: s_lshl_b64 s[6:7], s[4:5], 5
-; GFX908-NEXT: v_mov_b32_e32 v0, 0
; GFX908-NEXT: s_lshl_b64 s[14:15], s[10:11], 5
+; GFX908-NEXT: v_mov_b32_e32 v0, 0
; GFX908-NEXT: s_and_b64 s[0:1], exec, s[0:1]
+; GFX908-NEXT: s_or_b32 s14, s14, 28
; GFX908-NEXT: s_lshl_b64 s[16:17], s[8:9], 5
; GFX908-NEXT: v_mov_b32_e32 v1, 0
; GFX908-NEXT: s_waitcnt vmcnt(0)
@@ -609,13 +610,13 @@ define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg
; GFX908-NEXT: ; => This Inner Loop Header: Depth=2
; GFX908-NEXT: s_add_u32 s22, s20, s9
; GFX908-NEXT: s_addc_u32 s23, s21, s13
-; GFX908-NEXT: global_load_dword v21, v17, s[22:23] offset:16 glc
+; GFX908-NEXT: global_load_dword v21, v17, s[22:23] offset:-12 glc
; GFX908-NEXT: s_waitcnt vmcnt(0)
-; GFX908-NEXT: global_load_dword v20, v17, s[22:23] offset:20 glc
+; GFX908-NEXT: global_load_dword v20, v17, s[22:23] offset:-8 glc
; GFX908-NEXT: s_waitcnt vmcnt(0)
-; GFX908-NEXT: global_load_dword v12, v17, s[22:23] offset:24 glc
+; GFX908-NEXT: global_load_dword v12, v17, s[22:23] offset:-4 glc
; GFX908-NEXT: s_waitcnt vmcnt(0)
-; GFX908-NEXT: global_load_dword v12, v17, s[22:23] offset:28 glc
+; GFX908-NEXT: global_load_dword v12, v17, s[22:23] glc
; GFX908-NEXT: s_waitcnt vmcnt(0)
; GFX908-NEXT: ds_read_b64 v[12:13], v17
; GFX908-NEXT: ds_read_b64 v[14:15], v0
@@ -709,6 +710,7 @@ define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg
; GFX90A-NEXT: s_lshl_b64 s[6:7], s[4:5], 5
; GFX90A-NEXT: s_lshl_b64 s[14:15], s[10:11], 5
; GFX90A-NEXT: s_and_b64 s[0:1], exec, s[0:1]
+; GFX90A-NEXT: s_or_b32 s14, s14, 28
; GFX90A-NEXT: s_lshl_b64 s[16:17], s[8:9], 5
; GFX90A-NEXT: s_waitcnt vmcnt(0)
; GFX90A-NEXT: v_readfirstlane_b32 s2, v18
@@ -769,13 +771,13 @@ define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg
; GFX90A-NEXT: ; => This Inner Loop Header: Depth=2
; GFX90A-NEXT: s_add_u32 s22, s20, s9
; GFX90A-NEXT: s_addc_u32 s23, s21, s13
-; GFX90A-NEXT: global_load_dword v21, v19, s[22:23] offset:16 glc
+; GFX90A-NEXT: global_load_dword v21, v19, s[22:23] offset:-12 glc
; GFX90A-NEXT: s_waitcnt vmcnt(0)
-; GFX90A-NEXT: global_load_dword v20, v19, s[22:23] offset:20 glc
+; GFX90A-NEXT: global_load_dword v20, v19, s[22:23] offset:-8 glc
; GFX90A-NEXT: s_waitcnt vmcnt(0)
-; GFX90A-NEXT: global_load_dword v14, v19, s[22:23] offset:24 glc
+; GFX90A-NEXT: global_load_dword v14, v19, s[22:23] offset:-4 glc
; GFX90A-NEXT: s_waitcnt vmcnt(0)
-; GFX90A-NEXT: global_load_dword v14, v19, s[22:23] offset:28 glc
+; GFX90A-NEXT: global_load_dword v14, v19, s[22:23] glc
; GFX90A-NEXT: s_waitcnt vmcnt(0)
; GFX90A-NEXT: ds_read_b64 v[14:15], v19
; GFX90A-NEXT: ds_read_b64 v[16:17], v0
diff --git a/llvm/test/CodeGen/AMDGPU/dagcombine-reassociate-bug.ll b/llvm/test/CodeGen/AMDGPU/dagcombine-reassociate-bug.ll
index 1b447571efaf2..af1c64321222b 100644
--- a/llvm/test/CodeGen/AMDGPU/dagcombine-reassociate-bug.ll
+++ b/llvm/test/CodeGen/AMDGPU/dagcombine-reassociate-bug.ll
@@ -1,4 +1,4 @@
-; RUN: llc -mtriple=amdgcn -amdgpu-scalar-ir-passes=false < %s | FileCheck %s
+; RUN: llc -mtriple=amdgcn < %s | FileCheck %s
; Test for a bug where DAGCombiner::ReassociateOps() was creating adds
; with offset in the first operand and base pointers in the second.
diff --git a/llvm/test/CodeGen/AMDGPU/idot2.ll b/llvm/test/CodeGen/AMDGPU/idot2.ll
index bf65657ff841c..22907ca28c47f 100644
--- a/llvm/test/CodeGen/AMDGPU/idot2.ll
+++ b/llvm/test/CodeGen/AMDGPU/idot2.ll
@@ -2396,7 +2396,7 @@ define amdgpu_kernel void @udot2_MultipleUses_mul2(ptr addrspace(1) %src1,
; GFX9-NODL-NEXT: v_mul_u32_u24_e32 v4, v2, v1
; GFX9-NODL-NEXT: s_waitcnt lgkmcnt(0)
; GFX9-NODL-NEXT: v_mad_u32_u24 v1, v2, v1, s0
-; GFX9-NODL-NEXT: v_add3_u32 v1, v1, v4, v3
+; GFX9-NODL-NEXT: v_add3_u32 v1, v4, v1, v3
; GFX9-NODL-NEXT: global_store_dword v0, v1, s[6:7]
; GFX9-NODL-NEXT: s_endpgm
;
@@ -2417,7 +2417,7 @@ define amdgpu_kernel void @udot2_MultipleUses_mul2(ptr addrspace(1) %src1,
; GFX9-DL-NEXT: v_mul_u32_u24_e32 v4, v2, v1
; GFX9-DL-NEXT: s_waitcnt lgkmcnt(0)
; GFX9-DL-NEXT: v_mad_u32_u24 v1, v2, v1, s0
-; GFX9-DL-NEXT: v_add3_u32 v1, v1, v4, v3
+; GFX9-DL-NEXT: v_add3_u32 v1, v4, v1, v3
; GFX9-DL-NEXT: global_store_dword v0, v1, s[6:7]
; GFX9-DL-NEXT: s_endpgm
;
@@ -2442,7 +2442,7 @@ define amdgpu_kernel void @udot2_MultipleUses_mul2(ptr addrspace(1) %src1,
; GFX10-DL-NEXT: s_waitcnt lgkmcnt(0)
; GFX10-DL-NEXT: v_mad_u32_u24 v0, v3, v0, s0
; GFX10-DL-NEXT: v_mov_b32_e32 v3, 0
-; GFX10-DL-NEXT: v_add3_u32 v0, v0, v2, v1
+; GFX10-DL-NEXT: v_add3_u32 v0, v2, v0, v1
; GFX10-DL-NEXT: global_store_dword v3, v0, s[6:7]
; GFX10-DL-NEXT: s_endpgm
ptr addrspace(1) %src2,
@@ -2553,7 +2553,7 @@ define amdgpu_kernel void @idot2_MultipleUses_mul2(ptr addrspace(1) %src1,
; GFX9-NODL-NEXT: v_mul_i32_i24_e32 v4, v2, v1
; GFX9-NODL-NEXT: s_waitcnt lgkmcnt(0)
; GFX9-NODL-NEXT: v_mad_i32_i24 v1, v2, v1, s0
-; GFX9-NODL-NEXT: v_add3_u32 v1, v1, v4, v3
+; GFX9-NODL-NEXT: v_add3_u32 v1, v4, v1, v3
; GFX9-NODL-NEXT: global_store_dword v0, v1, s[6:7]
; GFX9-NODL-NEXT: s_endpgm
;
@@ -2574,7 +2574,7 @@ define amdgpu_kernel void @idot2_MultipleUses_mul2(ptr addrspace(1) %src1,
; GFX9-DL-NEXT: v_mul_i32_i24_e32 v4, v2, v1
; GFX9-DL-NEXT: s_waitcnt lgkmcnt(0)
; GFX9-DL-NEXT: v_mad_i32_i24 v1, v2, v1, s0
-; GFX9-DL-NEXT: v_add3_u32 v1, v1, v4, v3
+; GFX9-DL-NEXT: v_add3_u32 v1, v4, v1, v3
; GFX9-DL-NEXT: global_store_dword v0, v1, s[6:7]
; GFX9-DL-NEXT: s_endpgm
;
@@ -2599,7 +2599,7 @@ define amdgpu_kernel void @idot2_MultipleUses_mul2(ptr addrspace(1) %src1,
; GFX10-DL-NEXT: s_waitcnt lgkmcnt(0)
; GFX10-DL-NEXT: v_mad_i32_i24 v0, v3, v0, s0
; GFX10-DL-NEXT: v_mov_b32_e32 v3, 0
-; GFX10-DL-NEXT: v_add3_u32 v0, v0, v2, v1
+; GFX10-DL-NEXT: v_add3_u32 v0, v2, v0, v1
; GFX10-DL-NEXT: global_store_dword v3, v0, s[6:7]
; GFX10-DL-NEXT: s_endpgm
ptr addrspace(1) %src2,
diff --git a/llvm/test/CodeGen/AMDGPU/idot4s.ll b/llvm/test/CodeGen/AMDGPU/idot4s.ll
index 1a22fa805a5a7..fd1f7b000472a 100644
--- a/llvm/test/CodeGen/AMDGPU/idot4s.ll
+++ b/llvm/test/CodeGen/AMDGPU/idot4s.ll
@@ -3268,19 +3268,19 @@ define amdgpu_kernel void @idot4_nonstandard_signed(ptr addrspace(1) %src1,
; GFX7-NEXT: buffer_load_dword v0, v[0:1], s[4:7], 0 addr64
; GFX7-NEXT: s_mov_b32 s2, -1
; GFX7-NEXT: s_waitcnt vmcnt(1)
-; GFX7-NEXT: v_bfe_i32 v3, v2, 8, 8
; GFX7-NEXT: v_bfe_i32 v1, v2, 0, 8
+; GFX7-NEXT: v_bfe_i32 v3, v2, 8, 8
+; GFX7-NEXT: v_and_b32_e32 v1, 0xffff, v1
; GFX7-NEXT: s_waitcnt vmcnt(0)
+; GFX7-NEXT: v_and_b32_e32 v5, 0xff, v0
+; GFX7-NEXT: v_bfe_i32 v4, v2, 16, 8
; GFX7-NEXT: v_bfe_u32 v6, v0, 8, 8
; GFX7-NEXT: v_and_b32_e32 v3, 0xffff, v3
-; GFX7-NEXT: v_bfe_i32 v4, v2, 16, 8
-; GFX7-NEXT: v_and_b32_e32 v1, 0xffff, v1
-; GFX7-NEXT: v_and_b32_e32 v5, 0xff, v0
-; GFX7-NEXT: v_mul_u32_u24_e32 v3, v6, v3
+; GFX7-NEXT: v_mul_u32_u24_e32 v1, v1, v5
; GFX7-NEXT: v_ashrrev_i32_e32 v2, 24, v2
; GFX7-NEXT: v_bfe_u32 v7, v0, 16, 8
; GFX7-NEXT: v_and_b32_e32 v4, 0xffff, v4
-; GFX7-NEXT: v_mad_u32_u24 v1, v1, v5, v3
+; GFX7-NEXT: v_mad_u32_u24 v1, v6, v3, v1
; GFX7-NEXT: v_lshrrev_b32_e32 v0, 24, v0
; GFX7-NEXT: v_mad_u32_u24 v1, v7, v4, v1
; GFX7-NEXT: v_and_b32_e32 v2, 0xffff, v2
@@ -3307,18 +3307,18 @@ define amdgpu_kernel void @idot4_nonstandard_signed(ptr addrspace(1) %src1,
; GFX8-NEXT: v_mov_b32_e32 v0, s4
; GFX8-NEXT: v_mov_b32_e32 v1, s5
; GFX8-NEXT: s_waitcnt vmcnt(1)
-; GFX8-NEXT: v_lshrrev_b32_e32 v8, 8, v3
+; GFX8-NEXT: v_lshrrev_b32_e32 v7, 8, v3
; GFX8-NEXT: v_lshrrev_b32_e32 v5, 16, v3
-; GFX8-NEXT: v_bfe_i32 v6, v3, 0, 8
-; GFX8-NEXT: v_lshrrev_b32_e32 v3, 24, v3
+; GFX8-NEXT: v_bfe_i32 v7, v7, 0, 8
; GFX8-NEXT: v_bfe_i32 v5, v5, 0, 8
-; GFX8-NEXT: v_bfe_i32 v3, v3, 0, 8
; GFX8-NEXT: s_waitcnt vmcnt(0)
-; GFX8-NEXT: v_lshrrev_b32_e32 v9, 8, v2
-; GFX8-NEXT: v_and_b32_e32 v7, 0xff, v2
-; GFX8-NEXT: v_mul_lo_u16_sdwa v8, v9, sext(v8) dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX8-NEXT: v_lshrrev_b32_e32 v8, 8, v2
+; GFX8-NEXT: v_mul_lo_u16_sdwa v6, sext(v3), v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX8-NEXT: v_and_b32_e32 v8, 0xff, v8
; GFX8-NEXT: v_and_b32_sdwa v4, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX8-NEXT: v_mad_u16 v6, v6, v7, v8
+; GFX8-NEXT: v_lshrrev_b32_e32 v3, 24, v3
+; GFX8-NEXT: v_mad_u16 v6, v8, v7, v6
+; GFX8-NEXT: v_bfe_i32 v3, v3, 0, 8
; GFX8-NEXT: v_mad_u16 v4, v4, v5, v6
; GFX8-NEXT: v_lshrrev_b32_e32 v2, 24, v2
; GFX8-NEXT: v_mad_u16 v2, v3, v2, v4
@@ -3337,19 +3337,19 @@ define amdgpu_kernel void @idot4_nonstandard_signed(ptr addrspace(1) %src1,
; GFX9-NODL-NEXT: s_movk_i32 s0, 0xff
; GFX9-NODL-NEXT: v_mov_b32_e32 v0, 0
; GFX9-NODL-NEXT: s_waitcnt vmcnt(1)
-; GFX9-NODL-NEXT: v_lshrrev_b32_e32 v6, 8, v1
+; GFX9-NODL-NEXT: v_lshrrev_b32_e32 v5, 8, v1
; GFX9-NODL-NEXT: s_waitcnt vmcnt(0)
-; GFX9-NODL-NEXT: v_lshrrev_b32_e32 v7, 8, v2
+; GFX9-NODL-NEXT: v_lshrrev_b32_e32 v6, 8, v2
; GFX9-NODL-NEXT: v_lshrrev_b32_e32 v3, 16, v1
-; GFX9-NODL-NEXT: v_bfe_i32 v4, v1, 0, 8
-; GFX9-NODL-NEXT: v_and_b32_e32 v5, 0xff, v2
-; GFX9-NODL-NEXT: v_mul_lo_u16_sdwa v6, v7, sext(v6) dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
-; GFX9-NODL-NEXT: v_and_b32_sdwa v8, v2, s0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NODL-NEXT: v_mul_lo_u16_sdwa v4, sext(v1), v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX9-NODL-NEXT: v_bfe_i32 v5, v5, 0, 8
+; GFX9-NODL-NEXT: v_and_b32_e32 v6, 0xff, v6
+; GFX9-NODL-NEXT: v_and_b32_sdwa v7, v2, s0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
; GFX9-NODL-NEXT: v_lshrrev_b32_e32 v1, 24, v1
; GFX9-NODL-NEXT: v_bfe_i32 v3, v3, 0, 8
-; GFX9-NODL-NEXT: v_mad_legacy_u16 v4, v4, v5, v6
+; GFX9-NODL-NEXT: v_mad_legacy_u16 v4, v6, v5, v4
; GFX9-NODL-NEXT: v_bfe_i32 v1, v1, 0, 8
-; GFX9-NODL-NEXT: v_mad_legacy_u16 v3, v8, v3, v4
+; GFX9-NODL-NEXT: v_mad_legacy_u16 v3, v7, v3, v4
; GFX9-NODL-NEXT: v_lshrrev_b32_e32 v2, 24, v2
; GFX9-NODL-NEXT: v_mad_legacy_u16 v1, v1, v2, v3
; GFX9-NODL-NEXT: v_bfe_i32 v1, v1, 0, 16
@@ -3367,19 +3367,19 @@ define amdgpu_kernel void @idot4_nonstandard_signed(ptr addrspace(1) %src1,
; GFX9-DL-NEXT: s_movk_i32 s0, 0xff
; GFX9-DL-NEXT: v_mov_b32_e32 v0, 0
; GFX9-DL-NEXT: s_waitcnt vmcnt(1)
-; GFX9-DL-NEXT: v_lshrrev_b32_e32 v6, 8, v1
+; GFX9-DL-NEXT: v_lshrrev_b32_e32 v5, 8, v1
; GFX9-DL-NEXT: s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT: v_lshrrev_b32_e32 v7, 8, v2
+; GFX9-DL-NEXT: v_lshrrev_b32_e32 v6, 8, v2
; GFX9-DL-NEXT: v_lshrrev_b32_e32 v3, 16, v1
-; GFX9-DL-NEXT: v_bfe_i32 v4, v1, 0, 8
-; GFX9-DL-NEXT: v_and_b32_e32 v5, 0xff, v2
-; GFX9-DL-NEXT: v_mul_lo_u16_sdwa v6, v7, sext(v6) dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
-; GFX9-DL-NEXT: v_and_b32_sdwa v8, v2, s0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-DL-NEXT: v_mul_lo_u16_sdwa v4, sext(v1), v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX9-DL-NEXT: v_bfe_i32 v5, v5, 0, 8
+; GFX9-DL-NEXT: v_and_b32_e32 v6, 0xff, v6
+; GFX9-DL-NEXT: v_and_b32_sdwa v7, v2, s0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
; GFX9-DL-NEXT: v_lshrrev_b32_e32 v1, 24, v1
; GFX9-DL-NEXT: v_bfe_i32 v3, v3, 0, 8
-; GFX9-DL-NEXT: v_mad_legacy_u16 v4, v4, v5, v6
+; GFX9-DL-NEXT: v_mad_legacy_u16 v4, v6, v5, v4
; GFX9-DL-NEXT: v_bfe_i32 v1, v1, 0, 8
-; GFX9-DL-NEXT: v_mad_legacy_u16 v3, v8, v3, v4
+; GFX9-DL-NEXT: v_mad_legacy_u16 v3, v7, v3, v4
; GFX9-DL-NEXT: v_lshrrev_b32_e32 v2, 24, v2
; GFX9-DL-NEXT: v_mad_legacy_u16 v1, v1, v2, v3
; GFX9-DL-NEXT: v_bfe_i32 v1, v1, 0, 16
@@ -3392,28 +3392,28 @@ define amdgpu_kernel void @idot4_nonstandard_signed(ptr addrspace(1) %src1,
; GFX10-DL-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
; GFX10-DL-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x34
; GFX10-DL-NEXT: v_lshlrev_b32_e32 v0, 2, v0
-; GFX10-DL-NEXT: v_mov_b32_e32 v4, 0xff
+; GFX10-DL-NEXT: v_mov_b32_e32 v6, 0xff
; GFX10-DL-NEXT: s_waitcnt lgkmcnt(0)
; GFX10-DL-NEXT: s_clause 0x1
; GFX10-DL-NEXT: global_load_dword v1, v0, s[0:1]
; GFX10-DL-NEXT: global_load_dword v2, v0, s[2:3]
; GFX10-DL-NEXT: s_waitcnt vmcnt(1)
-; GFX10-DL-NEXT: v_lshrrev_b32_e32 v0, 8, v1
+; GFX10-DL-NEXT: v_bfe_i32 v0, v1, 0, 8
; GFX10-DL-NEXT: s_waitcnt vmcnt(0)
-; GFX10-DL-NEXT: v_lshrrev_b32_e32 v3, 8, v2
-; GFX10-DL-NEXT: v_lshrrev_b32_e32 v5, 16, v1
-; GFX10-DL-NEXT: v_bfe_i32 v6, v1, 0, 8
-; GFX10-DL-NEXT: v_and_b32_e32 v7, 0xff, v2
-; GFX10-DL-NEXT: v_bfe_i32 v0, v0, 0, 8
-; GFX10-DL-NEXT: v_and_b32_e32 v3, 0xff, v3
+; GFX10-DL-NEXT: v_and_b32_e32 v3, 0xff, v2
+; GFX10-DL-NEXT: v_lshrrev_b32_e32 v4, 8, v1
+; GFX10-DL-NEXT: v_lshrrev_b32_e32 v5, 8, v2
+; GFX10-DL-NEXT: v_lshrrev_b32_e32 v7, 16, v1
; GFX10-DL-NEXT: v_lshrrev_b32_e32 v1, 24, v1
-; GFX10-DL-NEXT: v_mul_lo_u16 v0, v3, v0
-; GFX10-DL-NEXT: v_and_b32_sdwa v3, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX10-DL-NEXT: v_bfe_i32 v4, v5, 0, 8
+; GFX10-DL-NEXT: v_mul_lo_u16 v0, v0, v3
+; GFX10-DL-NEXT: v_bfe_i32 v3, v4, 0, 8
+; GFX10-DL-NEXT: v_and_b32_e32 v4, 0xff, v5
+; GFX10-DL-NEXT: v_and_b32_sdwa v5, v2, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-DL-NEXT: v_bfe_i32 v6, v7, 0, 8
; GFX10-DL-NEXT: v_bfe_i32 v1, v1, 0, 8
; GFX10-DL-NEXT: v_lshrrev_b32_e32 v2, 24, v2
-; GFX10-DL-NEXT: v_mad_u16 v0, v6, v7, v0
-; GFX10-DL-NEXT: v_mad_u16 v0, v3, v4, v0
+; GFX10-DL-NEXT: v_mad_u16 v0, v4, v3, v0
+; GFX10-DL-NEXT: v_mad_u16 v0, v5, v6, v0
; GFX10-DL-NEXT: v_mad_u16 v0, v1, v2, v0
; GFX10-DL-NEXT: v_mov_b32_e32 v1, 0
; GFX10-DL-NEXT: v_bfe_i32 v0, v0, 0, 16
@@ -3429,34 +3429,32 @@ define amdgpu_kernel void @idot4_nonstandard_signed(ptr addrspace(1) %src1,
; GFX11-DL-TRUE16-NEXT: v_lshlrev_b32_e32 v0, 2, v0
; GFX11-DL-TRUE16-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-DL-TRUE16-NEXT: s_clause 0x1
-; GFX11-DL-TRUE16-NEXT: global_load_b32 v3, v0, s[0:1]
-; GFX11-DL-TRUE16-NEXT: global_load_b32 v4, v0, s[2:3]
+; GFX11-DL-TRUE16-NEXT: global_load_b32 v2, v0, s[0:1]
+; GFX11-DL-TRUE16-NEXT: global_load_b32 v3, v0, s[2:3]
; GFX11-DL-TRUE16-NEXT: s_waitcnt vmcnt(1)
-; GFX11-DL-TRUE16-NEXT: v_lshrrev_b32_e32 v0, 8, v3
+; GFX11-DL-TRUE16-NEXT: v_lshrrev_b32_e32 v4, 8, v2
+; GFX11-DL-TRUE16-NEXT: v_bfe_i32 v1, v2, 0, 8
; GFX11-DL-TRUE16-NEXT: s_waitcnt vmcnt(0)
-; GFX11-DL-TRUE16-NEXT: v_lshrrev_b32_e32 v1, 8, v4
-; GFX11-DL-TRUE16-NEXT: v_bfe_i32 v5, v3, 0, 8
-; GFX11-DL-TRUE16-NEXT: v_mov_b16_e32 v6.l, v3.h
-; GFX11-DL-TRUE16-NEXT: v_bfe_i32 v2, v0, 0, 8
-; GFX11-DL-TRUE16-NEXT: s_delay_alu instid0(VALU_DEP_4) | instskip(SKIP_1) | instid1(VALU_DEP_3)
-; GFX11-DL-TRUE16-NEXT: v_and_b16 v0.l, 0xff, v1.l
-; GFX11-DL-TRUE16-NEXT: v_and_b16 v0.h, 0xff, v4.l
-; GFX11-DL-TRUE16-NEXT: v_mov_b16_e32 v1.l, v2.l
-; GFX11-DL-TRUE16-NEXT: v_mov_b16_e32 v2.l, v5.l
+; GFX11-DL-TRUE16-NEXT: v_and_b16 v0.l, 0xff, v3.l
+; GFX11-DL-TRUE16-NEXT: v_lshrrev_b32_e32 v5, 8, v3
+; GFX11-DL-TRUE16-NEXT: v_mov_b16_e32 v6.l, v2.h
+; GFX11-DL-TRUE16-NEXT: v_bfe_i32 v4, v4, 0, 8
+; GFX11-DL-TRUE16-NEXT: v_and_b16 v1.h, 0xff, v3.h
+; GFX11-DL-TRUE16-NEXT: v_mul_lo_u16 v0.l, v1.l, v0.l
+; GFX11-DL-TRUE16-NEXT: v_and_b16 v0.h, 0xff, v5.l
; GFX11-DL-TRUE16-NEXT: v_bfe_i32 v5, v6, 0, 8
-; GFX11-DL-TRUE16-NEXT: v_lshrrev_b32_e32 v6, 24, v3
-; GFX11-DL-TRUE16-NEXT: s_delay_alu instid0(VALU_DEP_4) | instskip(SKIP_1) | instid1(VALU_DEP_4)
-; GFX11-DL-TRUE16-NEXT: v_mul_lo_u16 v0.l, v0.l, v1.l
-; GFX11-DL-TRUE16-NEXT: v_and_b16 v1.l, 0xff, v4.h
-; GFX11-DL-TRUE16-NEXT: v_mov_b16_e32 v3.l, v5.l
-; GFX11-DL-TRUE16-NEXT: v_lshrrev_b32_e32 v4, 24, v4
-; GFX11-DL-TRUE16-NEXT: s_delay_alu instid0(VALU_DEP_4) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11-DL-TRUE16-NEXT: v_mad_u16 v0.l, v2.l, v0.h, v0.l
-; GFX11-DL-TRUE16-NEXT: v_bfe_i32 v2, v6, 0, 8
-; GFX11-DL-TRUE16-NEXT: v_mad_u16 v0.l, v1.l, v3.l, v0.l
+; GFX11-DL-TRUE16-NEXT: v_mov_b16_e32 v1.l, v4.l
+; GFX11-DL-TRUE16-NEXT: v_lshrrev_b32_e32 v4, 24, v2
+; GFX11-DL-TRUE16-NEXT: v_lshrrev_b32_e32 v3, 24, v3
+; GFX11-DL-TRUE16-NEXT: s_delay_alu instid0(VALU_DEP_4) | instskip(NEXT) | instid1(VALU_DEP_4)
+; GFX11-DL-TRUE16-NEXT: v_mov_b16_e32 v2.l, v5.l
+; GFX11-DL-TRUE16-NEXT: v_mad_u16 v0.l, v0.h, v1.l, v0.l
+; GFX11-DL-TRUE16-NEXT: s_delay_alu instid0(VALU_DEP_4) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-DL-TRUE16-NEXT: v_bfe_i32 v4, v4, 0, 8
+; GFX11-DL-TRUE16-NEXT: v_mad_u16 v0.l, v1.h, v2.l, v0.l
; GFX11-DL-TRUE16-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11-DL-TRUE16-NEXT: v_mov_b16_e32 v1.l, v2.l
-; GFX11-DL-TRUE16-NEXT: v_mad_u16 v0.l, v1.l, v4.l, v0.l
+; GFX11-DL-TRUE16-NEXT: v_mov_b16_e32 v1.l, v4.l
+; GFX11-DL-TRUE16-NEXT: v_mad_u16 v0.l, v1.l, v3.l, v0.l
; GFX11-DL-TRUE16-NEXT: v_mov_b32_e32 v1, 0
; GFX11-DL-TRUE16-NEXT: s_delay_alu instid0(VALU_DEP_2)
; GFX11-DL-TRUE16-NEXT: v_bfe_i32 v0, v0, 0, 16
@@ -3475,25 +3473,24 @@ define amdgpu_kernel void @idot4_nonstandard_signed(ptr addrspace(1) %src1,
; GFX11-DL-FAKE16-NEXT: global_load_b32 v1, v0, s[0:1]
; GFX11-DL-FAKE16-NEXT: global_load_b32 v0, v0, s[2:3]
; GFX11-DL-FAKE16-NEXT: s_waitcnt vmcnt(1)
-; GFX11-DL-FAKE16-NEXT: v_lshrrev_b32_e32 v2, 8, v1
+; GFX11-DL-FAKE16-NEXT: v_bfe_i32 v2, v1, 0, 8
; GFX11-DL-FAKE16-NEXT: s_waitcnt vmcnt(0)
-; GFX11-DL-FAKE16-NEXT: v_lshrrev_b32_e32 v3, 8, v0
-; GFX11-DL-FAKE16-NEXT: v_lshrrev_b32_e32 v4, 16, v1
-; GFX11-DL-FAKE16-NEXT: v_lshrrev_b32_e32 v5, 16, v0
-; GFX11-DL-FAKE16-NEXT: v_bfe_i32 v6, v1, 0, 8
-; GFX11-DL-FAKE16-NEXT: v_bfe_i32 v2, v2, 0, 8
-; GFX11-DL-FAKE16-NEXT: v_and_b32_e32 v3, 0xff, v3
-; GFX11-DL-FAKE16-NEXT: v_and_b32_e32 v7, 0xff, v0
-; GFX11-DL-FAKE16-NEXT: v_lshrrev_b32_e32 v1, 24, v1
-; GFX11-DL-FAKE16-NEXT: v_lshrrev_b32_e32 v0, 24, v0
-; GFX11-DL-FAKE16-NEXT: s_delay_alu instid0(VALU_DEP_4) | instskip(SKIP_3) | instid1(VALU_DEP_4)
-; GFX11-DL-FAKE16-NEXT: v_mul_lo_u16 v2, v3, v2
+; GFX11-DL-FAKE16-NEXT: v_and_b32_e32 v3, 0xff, v0
+; GFX11-DL-FAKE16-NEXT: v_lshrrev_b32_e32 v4, 8, v1
+; GFX11-DL-FAKE16-NEXT: v_lshrrev_b32_e32 v5, 8, v0
+; GFX11-DL-FAKE16-NEXT: v_lshrrev_b32_e32 v6, 16, v1
+; GFX11-DL-FAKE16-NEXT: v_lshrrev_b32_e32 v7, 16, v0
+; GFX11-DL-FAKE16-NEXT: v_mul_lo_u16 v2, v2, v3
; GFX11-DL-FAKE16-NEXT: v_bfe_i32 v3, v4, 0, 8
; GFX11-DL-FAKE16-NEXT: v_and_b32_e32 v4, 0xff, v5
-; GFX11-DL-FAKE16-NEXT: v_bfe_i32 v1, v1, 0, 8
-; GFX11-DL-FAKE16-NEXT: v_mad_u16 v2, v6, v7, v2
-; GFX11-DL-FAKE16-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX11-DL-FAKE16-NEXT: v_lshrrev_b32_e32 v1, 24, v1
+; GFX11-DL-FAKE16-NEXT: v_bfe_i32 v5, v6, 0, 8
+; GFX11-DL-FAKE16-NEXT: v_and_b32_e32 v6, 0xff, v7
+; GFX11-DL-FAKE16-NEXT: v_lshrrev_b32_e32 v0, 24, v0
; GFX11-DL-FAKE16-NEXT: v_mad_u16 v2, v4, v3, v2
+; GFX11-DL-FAKE16-NEXT: v_bfe_i32 v1, v1, 0, 8
+; GFX11-DL-FAKE16-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
+; GFX11-DL-FAKE16-NEXT: v_mad_u16 v2, v6, v5, v2
; GFX11-DL-FAKE16-NEXT: v_mad_u16 v0, v1, v0, v2
; GFX11-DL-FAKE16-NEXT: v_mov_b32_e32 v1, 0
; GFX11-DL-FAKE16-NEXT: s_delay_alu instid0(VALU_DEP_2)
diff --git a/llvm/test/CodeGen/AMDGPU/idot8u.ll b/llvm/test/CodeGen/AMDGPU/idot8u.ll
index c03802e144d5b..dfc1e3c088129 100644
--- a/llvm/test/CodeGen/AMDGPU/idot8u.ll
+++ b/llvm/test/CodeGen/AMDGPU/idot8u.ll
@@ -1684,7 +1684,7 @@ define amdgpu_kernel void @udot8_multiuses_mul1(ptr addrspace(1) %src1,
; GFX9-NEXT: v_mul_u32_u24_e32 v4, v4, v11
; GFX9-NEXT: v_add3_u32 v2, v2, v7, v6
; GFX9-NEXT: v_add3_u32 v2, v2, v5, v4
-; GFX9-NEXT: v_add3_u32 v1, v1, v17, v2
+; GFX9-NEXT: v_add3_u32 v1, v17, v1, v2
; GFX9-NEXT: global_store_dword v0, v1, s[6:7]
; GFX9-NEXT: s_endpgm
;
@@ -1735,7 +1735,7 @@ define amdgpu_kernel void @udot8_multiuses_mul1(ptr addrspace(1) %src1,
; GFX9-DL-NEXT: v_mul_u32_u24_e32 v4, v4, v11
; GFX9-DL-NEXT: v_add3_u32 v2, v2, v7, v6
; GFX9-DL-NEXT: v_add3_u32 v2, v2, v5, v4
-; GFX9-DL-NEXT: v_add3_u32 v1, v1, v17, v2
+; GFX9-DL-NEXT: v_add3_u32 v1, v17, v1, v2
; GFX9-DL-NEXT: global_store_dword v0, v1, s[6:7]
; GFX9-DL-NEXT: s_endpgm
;
@@ -1789,7 +1789,7 @@ define amdgpu_kernel void @udot8_multiuses_mul1(ptr addrspace(1) %src1,
; GFX10-DL-NEXT: v_add3_u32 v0, v0, v6, v5
; GFX10-DL-NEXT: v_add3_u32 v0, v0, v1, v2
; GFX10-DL-NEXT: v_mov_b32_e32 v1, 0
-; GFX10-DL-NEXT: v_add3_u32 v0, v13, v3, v0
+; GFX10-DL-NEXT: v_add3_u32 v0, v3, v13, v0
; GFX10-DL-NEXT: global_store_dword v1, v0, s[6:7]
; GFX10-DL-NEXT: s_endpgm
ptr addrspace(1) %src2,
diff --git a/llvm/test/CodeGen/AMDGPU/promote-constOffset-to-imm.ll b/llvm/test/CodeGen/AMDGPU/promote-constOffset-to-imm.ll
index 1156f2718cf1e..3329c9a761900 100644
--- a/llvm/test/CodeGen/AMDGPU/promote-constOffset-to-imm.ll
+++ b/llvm/test/CodeGen/AMDGPU/promote-constOffset-to-imm.ll
@@ -365,110 +365,107 @@ define hidden amdgpu_kernel void @clmem_read(ptr addrspace(1) %buffer) {
; GFX8-NEXT: s_waitcnt lgkmcnt(0)
; GFX8-NEXT: s_swappc_b64 s[30:31], s[4:5]
; GFX8-NEXT: v_lshlrev_b32_e32 v1, 17, v0
-; GFX8-NEXT: v_and_b32_e32 v10, 0xfe000000, v1
+; GFX8-NEXT: v_and_b32_e32 v12, 0xfe000000, v1
; GFX8-NEXT: v_mov_b32_e32 v1, 3
; GFX8-NEXT: v_lshlrev_b32_sdwa v0, v1, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
-; GFX8-NEXT: v_or_b32_e32 v0, v10, v0
+; GFX8-NEXT: v_or_b32_e32 v0, v12, v0
; GFX8-NEXT: v_mov_b32_e32 v1, s35
; GFX8-NEXT: v_add_u32_e32 v0, vcc, s34, v0
; GFX8-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
-; GFX8-NEXT: s_movk_i32 s0, 0x2800
+; GFX8-NEXT: s_movk_i32 s0, 0x5000
; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0
-; GFX8-NEXT: v_mov_b32_e32 v6, 0
+; GFX8-NEXT: v_mov_b32_e32 v10, 0
; GFX8-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
-; GFX8-NEXT: v_mov_b32_e32 v7, 0
-; GFX8-NEXT: v_mov_b32_e32 v11, 0x7f
-; GFX8-NEXT: s_movk_i32 s1, 0x800
-; GFX8-NEXT: s_movk_i32 s2, 0x1000
-; GFX8-NEXT: s_movk_i32 s3, 0x1800
-; GFX8-NEXT: s_movk_i32 s4, 0x2000
+; GFX8-NEXT: v_mov_b32_e32 v11, 0
+; GFX8-NEXT: v_mov_b32_e32 v13, 0x7f
; GFX8-NEXT: .LBB1_1: ; %for.cond.preheader
; GFX8-NEXT: ; =>This Loop Header: Depth=1
; GFX8-NEXT: ; Child Loop BB1_2 Depth 2
; GFX8-NEXT: v_mov_b32_e32 v3, v1
-; GFX8-NEXT: s_mov_b32 s5, 0
+; GFX8-NEXT: s_mov_b32 s0, 0
; GFX8-NEXT: v_mov_b32_e32 v2, v0
; GFX8-NEXT: .LBB1_2: ; %for.body
; GFX8-NEXT: ; Parent Loop BB1_1 Depth=1
; GFX8-NEXT: ; => This Inner Loop Header: Depth=2
-; GFX8-NEXT: v_add_u32_e32 v8, vcc, 0xffffd800, v2
-; GFX8-NEXT: v_addc_u32_e32 v9, vcc, -1, v3, vcc
-; GFX8-NEXT: flat_load_dwordx2 v[4:5], v[2:3]
-; GFX8-NEXT: flat_load_dwordx2 v[14:15], v[8:9]
-; GFX8-NEXT: v_add_u32_e32 v12, vcc, 0xffffe000, v2
-; GFX8-NEXT: v_addc_u32_e32 v13, vcc, -1, v3, vcc
-; GFX8-NEXT: flat_load_dwordx2 v[12:13], v[12:13]
-; GFX8-NEXT: v_add_u32_e32 v8, vcc, 0xffffe800, v2
-; GFX8-NEXT: v_addc_u32_e32 v9, vcc, -1, v3, vcc
-; GFX8-NEXT: flat_load_dwordx2 v[18:19], v[8:9]
-; GFX8-NEXT: v_add_u32_e32 v16, vcc, 0xfffff000, v2
-; GFX8-NEXT: v_addc_u32_e32 v17, vcc, -1, v3, vcc
-; GFX8-NEXT: v_add_u32_e32 v20, vcc, 0xfffff800, v2
+; GFX8-NEXT: v_add_u32_e32 v4, vcc, 0xffffb000, v2
+; GFX8-NEXT: v_addc_u32_e32 v5, vcc, -1, v3, vcc
+; GFX8-NEXT: flat_load_dwordx2 v[14:15], v[4:5]
+; GFX8-NEXT: v_add_u32_e32 v6, vcc, 0xffffb800, v2
+; GFX8-NEXT: v_addc_u32_e32 v7, vcc, -1, v3, vcc
+; GFX8-NEXT: flat_load_dwordx2 v[16:17], v[6:7]
+; GFX8-NEXT: v_add_u32_e32 v4, vcc, 0xffffc000, v2
+; GFX8-NEXT: v_addc_u32_e32 v5, vcc, -1, v3, vcc
+; GFX8-NEXT: flat_load_dwordx2 v[18:19], v[4:5]
+; GFX8-NEXT: v_add_u32_e32 v6, vcc, 0xffffc800, v2
+; GFX8-NEXT: v_addc_u32_e32 v7, vcc, -1, v3, vcc
+; GFX8-NEXT: v_add_u32_e32 v4, vcc, 0xffffd000, v2
+; GFX8-NEXT: v_addc_u32_e32 v5, vcc, -1, v3, vcc
+; GFX8-NEXT: v_add_u32_e32 v20, vcc, 0xffffd800, v2
; GFX8-NEXT: v_addc_u32_e32 v21, vcc, -1, v3, vcc
-; GFX8-NEXT: flat_load_dwordx2 v[8:9], v[16:17]
-; GFX8-NEXT: v_add_u32_e32 v16, vcc, s1, v2
-; GFX8-NEXT: v_addc_u32_e32 v17, vcc, 0, v3, vcc
-; GFX8-NEXT: s_addk_i32 s5, 0x2000
-; GFX8-NEXT: s_cmp_gt_u32 s5, 0x3fffff
-; GFX8-NEXT: s_waitcnt vmcnt(3)
-; GFX8-NEXT: v_add_u32_e32 v22, vcc, v14, v6
-; GFX8-NEXT: v_addc_u32_e32 v23, vcc, v15, v7, vcc
-; GFX8-NEXT: v_add_u32_e32 v6, vcc, s2, v2
-; GFX8-NEXT: flat_load_dwordx2 v[14:15], v[20:21]
-; GFX8-NEXT: flat_load_dwordx2 v[16:17], v[16:17]
-; GFX8-NEXT: v_addc_u32_e32 v7, vcc, 0, v3, vcc
-; GFX8-NEXT: v_add_u32_e32 v20, vcc, s3, v2
-; GFX8-NEXT: v_addc_u32_e32 v21, vcc, 0, v3, vcc
-; GFX8-NEXT: s_waitcnt vmcnt(4)
-; GFX8-NEXT: v_add_u32_e32 v22, vcc, v12, v22
-; GFX8-NEXT: v_addc_u32_e32 v23, vcc, v13, v23, vcc
-; GFX8-NEXT: v_add_u32_e32 v12, vcc, s4, v2
; GFX8-NEXT: flat_load_dwordx2 v[6:7], v[6:7]
-; GFX8-NEXT: flat_load_dwordx2 v[20:21], v[20:21]
-; GFX8-NEXT: v_addc_u32_e32 v13, vcc, 0, v3, vcc
+; GFX8-NEXT: v_add_u32_e32 v22, vcc, 0xffffe000, v2
+; GFX8-NEXT: v_addc_u32_e32 v23, vcc, -1, v3, vcc
+; GFX8-NEXT: flat_load_dwordx2 v[8:9], v[4:5]
+; GFX8-NEXT: flat_load_dwordx2 v[4:5], v[20:21]
+; GFX8-NEXT: s_addk_i32 s0, 0x2000
+; GFX8-NEXT: s_cmp_gt_u32 s0, 0x3fffff
; GFX8-NEXT: s_waitcnt vmcnt(5)
+; GFX8-NEXT: v_add_u32_e32 v24, vcc, v14, v10
+; GFX8-NEXT: v_addc_u32_e32 v25, vcc, v15, v11, vcc
+; GFX8-NEXT: v_add_u32_e32 v10, vcc, 0xffffe800, v2
+; GFX8-NEXT: v_addc_u32_e32 v11, vcc, -1, v3, vcc
+; GFX8-NEXT: v_add_u32_e32 v14, vcc, 0xfffff000, v2
+; GFX8-NEXT: flat_load_dwordx2 v[20:21], v[22:23]
+; GFX8-NEXT: flat_load_dwordx2 v[10:11], v[10:11]
+; GFX8-NEXT: v_addc_u32_e32 v15, vcc, -1, v3, vcc
+; GFX8-NEXT: s_waitcnt vmcnt(6)
+; GFX8-NEXT: v_add_u32_e32 v22, vcc, v16, v24
+; GFX8-NEXT: v_addc_u32_e32 v23, vcc, v17, v25, vcc
+; GFX8-NEXT: v_add_u32_e32 v16, vcc, 0xfffff800, v2
+; GFX8-NEXT: flat_load_dwordx2 v[14:15], v[14:15]
+; GFX8-NEXT: v_addc_u32_e32 v17, vcc, -1, v3, vcc
+; GFX8-NEXT: flat_load_dwordx2 v[16:17], v[16:17]
+; GFX8-NEXT: s_waitcnt vmcnt(7)
; GFX8-NEXT: v_add_u32_e32 v22, vcc, v18, v22
; GFX8-NEXT: v_addc_u32_e32 v23, vcc, v19, v23, vcc
-; GFX8-NEXT: v_add_u32_e32 v18, vcc, s0, v2
-; GFX8-NEXT: flat_load_dwordx2 v[12:13], v[12:13]
-; GFX8-NEXT: v_addc_u32_e32 v19, vcc, 0, v3, vcc
-; GFX8-NEXT: flat_load_dwordx2 v[18:19], v[18:19]
+; GFX8-NEXT: flat_load_dwordx2 v[18:19], v[2:3]
; GFX8-NEXT: v_add_u32_e32 v2, vcc, 0x10000, v2
; GFX8-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc
+; GFX8-NEXT: s_waitcnt vmcnt(7)
+; GFX8-NEXT: v_add_u32_e32 v6, vcc, v6, v22
+; GFX8-NEXT: v_addc_u32_e32 v7, vcc, v7, v23, vcc
; GFX8-NEXT: s_waitcnt vmcnt(6)
-; GFX8-NEXT: v_add_u32_e32 v8, vcc, v8, v22
-; GFX8-NEXT: v_addc_u32_e32 v9, vcc, v9, v23, vcc
+; GFX8-NEXT: v_add_u32_e32 v6, vcc, v8, v6
+; GFX8-NEXT: v_addc_u32_e32 v7, vcc, v9, v7, vcc
; GFX8-NEXT: s_waitcnt vmcnt(5)
-; GFX8-NEXT: v_add_u32_e32 v8, vcc, v14, v8
-; GFX8-NEXT: v_addc_u32_e32 v9, vcc, v15, v9, vcc
-; GFX8-NEXT: v_add_u32_e32 v4, vcc, v4, v8
-; GFX8-NEXT: v_addc_u32_e32 v5, vcc, v5, v9, vcc
+; GFX8-NEXT: v_add_u32_e32 v4, vcc, v4, v6
+; GFX8-NEXT: v_addc_u32_e32 v5, vcc, v5, v7, vcc
; GFX8-NEXT: s_waitcnt vmcnt(4)
-; GFX8-NEXT: v_add_u32_e32 v4, vcc, v16, v4
-; GFX8-NEXT: v_addc_u32_e32 v5, vcc, v17, v5, vcc
-; GFX8-NEXT: s_waitcnt vmcnt(3)
-; GFX8-NEXT: v_add_u32_e32 v4, vcc, v6, v4
-; GFX8-NEXT: v_addc_u32_e32 v5, vcc, v7, v5, vcc
-; GFX8-NEXT: s_waitcnt vmcnt(2)
; GFX8-NEXT: v_add_u32_e32 v4, vcc, v20, v4
; GFX8-NEXT: v_addc_u32_e32 v5, vcc, v21, v5, vcc
+; GFX8-NEXT: s_waitcnt vmcnt(3)
+; GFX8-NEXT: v_add_u32_e32 v4, vcc, v10, v4
+; GFX8-NEXT: v_addc_u32_e32 v5, vcc, v11, v5, vcc
+; GFX8-NEXT: s_waitcnt vmcnt(2)
+; GFX8-NEXT: v_add_u32_e32 v4, vcc, v14, v4
+; GFX8-NEXT: v_addc_u32_e32 v5, vcc, v15, v5, vcc
; GFX8-NEXT: s_waitcnt vmcnt(1)
-; GFX8-NEXT: v_add_u32_e32 v4, vcc, v12, v4
-; GFX8-NEXT: v_addc_u32_e32 v5, vcc, v13, v5, vcc
+; GFX8-NEXT: v_add_u32_e32 v4, vcc, v16, v4
+; GFX8-NEXT: v_addc_u32_e32 v5, vcc, v17, v5, vcc
; GFX8-NEXT: s_waitcnt vmcnt(0)
-; GFX8-NEXT: v_add_u32_e32 v6, vcc, v18, v4
-; GFX8-NEXT: v_addc_u32_e32 v7, vcc, v19, v5, vcc
+; GFX8-NEXT: v_add_u32_e32 v10, vcc, v18, v4
+; GFX8-NEXT: v_addc_u32_e32 v11, vcc, v19, v5, vcc
; GFX8-NEXT: s_cbranch_scc0 .LBB1_2
; GFX8-NEXT: ; %bb.3: ; %while.cond.loopexit
; GFX8-NEXT: ; in Loop: Header=BB1_1 Depth=1
-; GFX8-NEXT: v_subrev_u32_e32 v11, vcc, 1, v11
+; GFX8-NEXT: v_subrev_u32_e32 v13, vcc, 1, v13
; GFX8-NEXT: s_and_b64 vcc, exec, vcc
; GFX8-NEXT: s_cbranch_vccz .LBB1_1
; GFX8-NEXT: ; %bb.4: ; %while.end
; GFX8-NEXT: v_mov_b32_e32 v1, s35
-; GFX8-NEXT: v_add_u32_e32 v0, vcc, s34, v10
+; GFX8-NEXT: v_add_u32_e32 v0, vcc, s34, v12
; GFX8-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
-; GFX8-NEXT: flat_store_dwordx2 v[0:1], v[6:7]
+; GFX8-NEXT: flat_store_dwordx2 v[0:1], v[10:11]
; GFX8-NEXT: s_endpgm
;
; GFX900-LABEL: clmem_read:
@@ -498,76 +495,79 @@ define hidden amdgpu_kernel void @clmem_read(ptr addrspace(1) %buffer) {
; GFX900-NEXT: v_mov_b32_e32 v1, s35
; GFX900-NEXT: v_add_co_u32_e32 v0, vcc, s34, v0
; GFX900-NEXT: v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
-; GFX900-NEXT: v_add_co_u32_e32 v0, vcc, 0x2800, v0
+; GFX900-NEXT: v_add_co_u32_e32 v0, vcc, 0x5000, v0
; GFX900-NEXT: v_mov_b32_e32 v4, 0
; GFX900-NEXT: v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
; GFX900-NEXT: v_mov_b32_e32 v5, 0
; GFX900-NEXT: v_mov_b32_e32 v7, 0x7f
-; GFX900-NEXT: s_movk_i32 s2, 0xf000
-; GFX900-NEXT: s_movk_i32 s3, 0x1000
-; GFX900-NEXT: s_movk_i32 s4, 0x2000
+; GFX900-NEXT: s_movk_i32 s2, 0xd000
+; GFX900-NEXT: s_movk_i32 s3, 0xe000
+; GFX900-NEXT: s_movk_i32 s4, 0xf000
; GFX900-NEXT: .LBB1_1: ; %for.cond.preheader
; GFX900-NEXT: ; =>This Loop Header: Depth=1
; GFX900-NEXT: ; Child Loop BB1_2 Depth 2
; GFX900-NEXT: v_mov_b32_e32 v3, v1
-; GFX900-NEXT: v_mov_b32_e32 v2, v0
; GFX900-NEXT: s_mov_b32 s5, 0
+; GFX900-NEXT: v_mov_b32_e32 v2, v0
; GFX900-NEXT: .LBB1_2: ; %for.body
; GFX900-NEXT: ; Parent Loop BB1_1 Depth=1
; GFX900-NEXT: ; => This Inner Loop Header: Depth=2
-; GFX900-NEXT: v_add_co_u32_e32 v8, vcc, 0xffffe000, v2
+; GFX900-NEXT: v_add_co_u32_e32 v8, vcc, 0xffffb000, v2
; GFX900-NEXT: v_addc_co_u32_e32 v9, vcc, -1, v3, vcc
-; GFX900-NEXT: global_load_dwordx2 v[14:15], v[8:9], off offset:-2048
+; GFX900-NEXT: global_load_dwordx2 v[8:9], v[8:9], off
+; GFX900-NEXT: v_add_co_u32_e32 v14, vcc, 0xffffc000, v2
+; GFX900-NEXT: v_addc_co_u32_e32 v15, vcc, -1, v3, vcc
+; GFX900-NEXT: global_load_dwordx2 v[18:19], v[14:15], off offset:-2048
+; GFX900-NEXT: global_load_dwordx2 v[20:21], v[14:15], off
+; GFX900-NEXT: v_add_co_u32_e32 v16, vcc, s2, v2
+; GFX900-NEXT: v_addc_co_u32_e32 v17, vcc, -1, v3, vcc
+; GFX900-NEXT: v_add_co_u32_e32 v14, vcc, s3, v2
+; GFX900-NEXT: global_load_dwordx2 v[16:17], v[16:17], off offset:-2048
+; GFX900-NEXT: v_addc_co_u32_e32 v15, vcc, -1, v3, vcc
; GFX900-NEXT: global_load_dwordx2 v[10:11], v[2:3], off offset:-4096
; GFX900-NEXT: global_load_dwordx2 v[12:13], v[2:3], off offset:-2048
; GFX900-NEXT: s_addk_i32 s5, 0x2000
; GFX900-NEXT: s_cmp_gt_u32 s5, 0x3fffff
-; GFX900-NEXT: s_waitcnt vmcnt(2)
-; GFX900-NEXT: v_add_co_u32_e32 v16, vcc, v14, v4
-; GFX900-NEXT: v_addc_co_u32_e32 v17, vcc, v15, v5, vcc
-; GFX900-NEXT: global_load_dwordx2 v[4:5], v[2:3], off
-; GFX900-NEXT: global_load_dwordx2 v[14:15], v[8:9], off
-; GFX900-NEXT: s_waitcnt vmcnt(0)
-; GFX900-NEXT: v_add_co_u32_e32 v14, vcc, v14, v16
-; GFX900-NEXT: v_addc_co_u32_e32 v15, vcc, v15, v17, vcc
-; GFX900-NEXT: v_add_co_u32_e32 v8, vcc, s2, v2
-; GFX900-NEXT: v_addc_co_u32_e32 v9, vcc, -1, v3, vcc
-; GFX900-NEXT: global_load_dwordx2 v[8:9], v[8:9], off offset:-2048
-; GFX900-NEXT: s_waitcnt vmcnt(0)
-; GFX900-NEXT: v_add_co_u32_e32 v14, vcc, v8, v14
-; GFX900-NEXT: v_addc_co_u32_e32 v15, vcc, v9, v15, vcc
-; GFX900-NEXT: global_load_dwordx2 v[8:9], v[2:3], off offset:2048
-; GFX900-NEXT: v_add_co_u32_e32 v14, vcc, v10, v14
-; GFX900-NEXT: v_addc_co_u32_e32 v11, vcc, v11, v15, vcc
-; GFX900-NEXT: v_add_co_u32_e64 v14, s[0:1], v12, v14
-; GFX900-NEXT: v_addc_co_u32_e64 v15, s[0:1], v13, v11, s[0:1]
-; GFX900-NEXT: v_add_co_u32_e32 v10, vcc, s3, v2
-; GFX900-NEXT: v_add_co_u32_e64 v12, s[0:1], s4, v2
-; GFX900-NEXT: v_addc_co_u32_e32 v11, vcc, 0, v3, vcc
-; GFX900-NEXT: v_addc_co_u32_e64 v13, vcc, 0, v3, s[0:1]
-; GFX900-NEXT: v_add_co_u32_e32 v16, vcc, v4, v14
-; GFX900-NEXT: v_addc_co_u32_e32 v17, vcc, v5, v15, vcc
-; GFX900-NEXT: global_load_dwordx2 v[4:5], v[12:13], off offset:-4096
-; GFX900-NEXT: global_load_dwordx2 v[14:15], v[10:11], off offset:2048
-; GFX900-NEXT: s_waitcnt vmcnt(2)
-; GFX900-NEXT: v_add_co_u32_e32 v16, vcc, v8, v16
-; GFX900-NEXT: v_addc_co_u32_e32 v17, vcc, v9, v17, vcc
-; GFX900-NEXT: global_load_dwordx2 v[8:9], v[12:13], off
-; GFX900-NEXT: global_load_dwordx2 v[10:11], v[12:13], off offset:2048
+; GFX900-NEXT: s_waitcnt vmcnt(5)
+; GFX900-NEXT: v_add_co_u32_e32 v22, vcc, v8, v4
+; GFX900-NEXT: v_addc_co_u32_e32 v5, vcc, v9, v5, vcc
+; GFX900-NEXT: global_load_dwordx2 v[8:9], v[14:15], off offset:-4096
+; GFX900-NEXT: s_waitcnt vmcnt(5)
+; GFX900-NEXT: v_add_co_u32_e64 v24, s[0:1], v18, v22
+; GFX900-NEXT: v_addc_co_u32_e64 v25, s[0:1], v19, v5, s[0:1]
+; GFX900-NEXT: global_load_dwordx2 v[18:19], v[14:15], off offset:-2048
+; GFX900-NEXT: global_load_dwordx2 v[22:23], v[14:15], off
+; GFX900-NEXT: v_add_co_u32_e32 v4, vcc, s4, v2
+; GFX900-NEXT: v_addc_co_u32_e32 v5, vcc, -1, v3, vcc
+; GFX900-NEXT: global_load_dwordx2 v[4:5], v[4:5], off offset:-2048
+; GFX900-NEXT: s_waitcnt vmcnt(7)
+; GFX900-NEXT: v_add_co_u32_e32 v20, vcc, v20, v24
+; GFX900-NEXT: global_load_dwordx2 v[14:15], v[2:3], off
+; GFX900-NEXT: v_addc_co_u32_e32 v21, vcc, v21, v25, vcc
; GFX900-NEXT: v_add_co_u32_e32 v2, vcc, 0x10000, v2
; GFX900-NEXT: v_addc_co_u32_e32 v3, vcc, 0, v3, vcc
+; GFX900-NEXT: s_waitcnt vmcnt(7)
+; GFX900-NEXT: v_add_co_u32_e32 v16, vcc, v16, v20
+; GFX900-NEXT: v_addc_co_u32_e32 v17, vcc, v17, v21, vcc
+; GFX900-NEXT: s_waitcnt vmcnt(4)
+; GFX900-NEXT: v_add_co_u32_e32 v8, vcc, v8, v16
+; GFX900-NEXT: v_addc_co_u32_e32 v9, vcc, v9, v17, vcc
; GFX900-NEXT: s_waitcnt vmcnt(3)
-; GFX900-NEXT: v_add_co_u32_e32 v4, vcc, v4, v16
-; GFX900-NEXT: v_addc_co_u32_e32 v5, vcc, v5, v17, vcc
+; GFX900-NEXT: v_add_co_u32_e32 v8, vcc, v18, v8
+; GFX900-NEXT: v_addc_co_u32_e32 v9, vcc, v19, v9, vcc
; GFX900-NEXT: s_waitcnt vmcnt(2)
-; GFX900-NEXT: v_add_co_u32_e32 v4, vcc, v14, v4
-; GFX900-NEXT: v_addc_co_u32_e32 v5, vcc, v15, v5, vcc
+; GFX900-NEXT: v_add_co_u32_e32 v8, vcc, v22, v8
+; GFX900-NEXT: v_addc_co_u32_e32 v9, vcc, v23, v9, vcc
; GFX900-NEXT: s_waitcnt vmcnt(1)
-; GFX900-NEXT: v_add_co_u32_e32 v4, vcc, v8, v4
-; GFX900-NEXT: v_addc_co_u32_e32 v5, vcc, v9, v5, vcc
-; GFX900-NEXT: s_waitcnt vmcnt(0)
+; GFX900-NEXT: v_add_co_u32_e32 v4, vcc, v4, v8
+; GFX900-NEXT: v_addc_co_u32_e32 v5, vcc, v5, v9, vcc
; GFX900-NEXT: v_add_co_u32_e32 v4, vcc, v10, v4
; GFX900-NEXT: v_addc_co_u32_e32 v5, vcc, v11, v5, vcc
+; GFX900-NEXT: v_add_co_u32_e32 v4, vcc, v12, v4
+; GFX900-NEXT: v_addc_co_u32_e32 v5, vcc, v13, v5, vcc
+; GFX900-NEXT: s_waitcnt vmcnt(0)
+; GFX900-NEXT: v_add_co_u32_e32 v4, vcc, v14, v4
+; GFX900-NEXT: v_addc_co_u32_e32 v5, vcc, v15, v5, vcc
; GFX900-NEXT: s_cbranch_scc0 .LBB1_2
; GFX900-NEXT: ; %bb.3: ; %while.cond.loopexit
; GFX900-NEXT: ; in Loop: Header=BB1_1 Depth=1
@@ -610,7 +610,7 @@ define hidden amdgpu_kernel void @clmem_read(ptr addrspace(1) %buffer) {
; GFX10-NEXT: v_lshl_or_b32 v0, v0, 3, v6
; GFX10-NEXT: v_add_co_u32 v0, s0, s34, v0
; GFX10-NEXT: v_add_co_ci_u32_e64 v1, s0, s35, 0, s0
-; GFX10-NEXT: v_add_co_u32 v0, vcc_lo, 0x2800, v0
+; GFX10-NEXT: v_add_co_u32 v0, vcc_lo, 0x5000, v0
; GFX10-NEXT: v_add_co_ci_u32_e32 v1, vcc_lo, 0, v1, vcc_lo
; GFX10-NEXT: .LBB1_1: ; %for.cond.preheader
; GFX10-NEXT: ; =>This Loop Header: Depth=1
@@ -621,30 +621,29 @@ define hidden amdgpu_kernel void @clmem_read(ptr addrspace(1) %buffer) {
; GFX10-NEXT: .LBB1_2: ; %for.body
; GFX10-NEXT: ; Parent Loop BB1_1 Depth=1
; GFX10-NEXT: ; => This Inner Loop Header: Depth=2
-; GFX10-NEXT: v_add_co_u32 v8, vcc_lo, v4, 0xffffe000
+; GFX10-NEXT: v_add_co_u32 v8, vcc_lo, v4, 0xffffb800
; GFX10-NEXT: v_add_co_ci_u32_e32 v9, vcc_lo, -1, v5, vcc_lo
-; GFX10-NEXT: v_add_co_u32 v10, vcc_lo, v4, 0xfffff000
+; GFX10-NEXT: v_add_co_u32 v10, vcc_lo, v4, 0xffffc800
; GFX10-NEXT: v_add_co_ci_u32_e32 v11, vcc_lo, -1, v5, vcc_lo
-; GFX10-NEXT: s_clause 0x5
+; GFX10-NEXT: v_add_co_u32 v14, vcc_lo, v4, 0xffffd800
+; GFX10-NEXT: v_add_co_ci_u32_e32 v15, vcc_lo, -1, v5, vcc_lo
+; GFX10-NEXT: v_add_co_u32 v18, vcc_lo, v4, 0xffffe800
+; GFX10-NEXT: s_clause 0x2
; GFX10-NEXT: global_load_dwordx2 v[12:13], v[8:9], off offset:-2048
-; GFX10-NEXT: global_load_dwordx2 v[14:15], v[8:9], off
; GFX10-NEXT: global_load_dwordx2 v[16:17], v[10:11], off offset:-2048
-; GFX10-NEXT: global_load_dwordx2 v[18:19], v[10:11], off
-; GFX10-NEXT: global_load_dwordx2 v[20:21], v[4:5], off offset:-2048
-; GFX10-NEXT: global_load_dwordx2 v[22:23], v[4:5], off
-; GFX10-NEXT: v_add_co_u32 v8, vcc_lo, v4, 0x1000
-; GFX10-NEXT: v_add_co_ci_u32_e32 v9, vcc_lo, 0, v5, vcc_lo
-; GFX10-NEXT: v_add_co_u32 v10, vcc_lo, v4, 0x2000
-; GFX10-NEXT: v_add_co_ci_u32_e32 v11, vcc_lo, 0, v5, vcc_lo
-; GFX10-NEXT: global_load_dwordx2 v[24:25], v[8:9], off offset:-2048
-; GFX10-NEXT: v_add_co_u32 v26, vcc_lo, 0x2800, v4
-; GFX10-NEXT: s_clause 0x1
-; GFX10-NEXT: global_load_dwordx2 v[28:29], v[10:11], off offset:-2048
+; GFX10-NEXT: global_load_dwordx2 v[20:21], v[14:15], off offset:-2048
+; GFX10-NEXT: v_add_co_ci_u32_e32 v19, vcc_lo, -1, v5, vcc_lo
+; GFX10-NEXT: v_add_co_u32 v22, vcc_lo, 0xfffff000, v4
+; GFX10-NEXT: v_add_co_ci_u32_e32 v23, vcc_lo, -1, v5, vcc_lo
+; GFX10-NEXT: s_clause 0x7
+; GFX10-NEXT: global_load_dwordx2 v[24:25], v[18:19], off offset:-2048
; GFX10-NEXT: global_load_dwordx2 v[8:9], v[8:9], off
-; GFX10-NEXT: v_add_co_ci_u32_e32 v27, vcc_lo, 0, v5, vcc_lo
-; GFX10-NEXT: s_clause 0x1
-; GFX10-NEXT: global_load_dwordx2 v[30:31], v[10:11], off
-; GFX10-NEXT: global_load_dwordx2 v[32:33], v[26:27], off
+; GFX10-NEXT: global_load_dwordx2 v[10:11], v[10:11], off
+; GFX10-NEXT: global_load_dwordx2 v[14:15], v[14:15], off
+; GFX10-NEXT: global_load_dwordx2 v[26:27], v[18:19], off
+; GFX10-NEXT: global_load_dwordx2 v[28:29], v[22:23], off
+; GFX10-NEXT: global_load_dwordx2 v[30:31], v[4:5], off offset:-2048
+; GFX10-NEXT: global_load_dwordx2 v[32:33], v[4:5], off
; GFX10-NEXT: v_add_co_u32 v4, vcc_lo, 0x10000, v4
; GFX10-NEXT: v_add_co_ci_u32_e32 v5, vcc_lo, 0, v5, vcc_lo
; GFX10-NEXT: s_addk_i32 s1, 0x2000
@@ -652,27 +651,25 @@ define hidden amdgpu_kernel void @clmem_read(ptr addrspace(1) %buffer) {
; GFX10-NEXT: s_waitcnt vmcnt(10)
; GFX10-NEXT: v_add_co_u32 v2, s0, v12, v2
; GFX10-NEXT: v_add_co_ci_u32_e64 v3, s0, v13, v3, s0
-; GFX10-NEXT: s_waitcnt vmcnt(9)
-; GFX10-NEXT: v_add_co_u32 v2, s0, v14, v2
-; GFX10-NEXT: v_add_co_ci_u32_e64 v3, s0, v15, v3, s0
-; GFX10-NEXT: s_waitcnt vmcnt(8)
+; GFX10-NEXT: s_waitcnt vmcnt(6)
+; GFX10-NEXT: v_add_co_u32 v2, s0, v8, v2
+; GFX10-NEXT: v_add_co_ci_u32_e64 v3, s0, v9, v3, s0
; GFX10-NEXT: v_add_co_u32 v2, s0, v16, v2
; GFX10-NEXT: v_add_co_ci_u32_e64 v3, s0, v17, v3, s0
-; GFX10-NEXT: s_waitcnt vmcnt(7)
-; GFX10-NEXT: v_add_co_u32 v2, s0, v18, v2
-; GFX10-NEXT: v_add_co_ci_u32_e64 v3, s0, v19, v3, s0
-; GFX10-NEXT: s_waitcnt vmcnt(6)
+; GFX10-NEXT: s_waitcnt vmcnt(5)
+; GFX10-NEXT: v_add_co_u32 v2, s0, v10, v2
+; GFX10-NEXT: v_add_co_ci_u32_e64 v3, s0, v11, v3, s0
; GFX10-NEXT: v_add_co_u32 v2, s0, v20, v2
; GFX10-NEXT: v_add_co_ci_u32_e64 v3, s0, v21, v3, s0
-; GFX10-NEXT: s_waitcnt vmcnt(5)
-; GFX10-NEXT: v_add_co_u32 v2, s0, v22, v2
-; GFX10-NEXT: v_add_co_ci_u32_e64 v3, s0, v23, v3, s0
; GFX10-NEXT: s_waitcnt vmcnt(4)
+; GFX10-NEXT: v_add_co_u32 v2, s0, v14, v2
+; GFX10-NEXT: v_add_co_ci_u32_e64 v3, s0, v15, v3, s0
; GFX10-NEXT: v_add_co_u32 v2, s0, v24, v2
; GFX10-NEXT: v_add_co_ci_u32_e64 v3, s0, v25, v3, s0
+; GFX10-NEXT: s_waitcnt vmcnt(3)
+; GFX10-NEXT: v_add_co_u32 v2, s0, v26, v2
+; GFX10-NEXT: v_add_co_ci_u32_e64 v3, s0, v27, v3, s0
; GFX10-NEXT: s_waitcnt vmcnt(2)
-; GFX10-NEXT: v_add_co_u32 v2, s0, v8, v2
-; GFX10-NEXT: v_add_co_ci_u32_e64 v3, s0, v9, v3, s0
; GFX10-NEXT: v_add_co_u32 v2, s0, v28, v2
; GFX10-NEXT: v_add_co_ci_u32_e64 v3, s0, v29, v3, s0
; GFX10-NEXT: s_waitcnt vmcnt(1)
@@ -720,76 +717,78 @@ define hidden amdgpu_kernel void @clmem_read(ptr addrspace(1) %buffer) {
; GFX90A-NEXT: v_mov_b32_e32 v2, s35
; GFX90A-NEXT: v_add_co_u32_e32 v1, vcc, s34, v1
; GFX90A-NEXT: v_addc_co_u32_e32 v3, vcc, 0, v2, vcc
-; GFX90A-NEXT: v_add_co_u32_e32 v2, vcc, 0x2800, v1
+; GFX90A-NEXT: v_add_co_u32_e32 v2, vcc, 0x5000, v1
; GFX90A-NEXT: v_addc_co_u32_e32 v3, vcc, 0, v3, vcc
; GFX90A-NEXT: v_mov_b32_e32 v1, 0x7f
; GFX90A-NEXT: v_pk_mov_b32 v[4:5], 0, 0
+; GFX90A-NEXT: s_movk_i32 s0, 0xd000
+; GFX90A-NEXT: s_movk_i32 s1, 0xe000
; GFX90A-NEXT: s_movk_i32 s2, 0xf000
-; GFX90A-NEXT: s_movk_i32 s3, 0x1000
-; GFX90A-NEXT: s_movk_i32 s4, 0x2000
; GFX90A-NEXT: .LBB1_1: ; %for.cond.preheader
; GFX90A-NEXT: ; =>This Loop Header: Depth=1
; GFX90A-NEXT: ; Child Loop BB1_2 Depth 2
+; GFX90A-NEXT: s_mov_b32 s3, 0
; GFX90A-NEXT: v_pk_mov_b32 v[6:7], v[2:3], v[2:3] op_sel:[0,1]
-; GFX90A-NEXT: s_mov_b32 s5, 0
; GFX90A-NEXT: .LBB1_2: ; %for.body
; GFX90A-NEXT: ; Parent Loop BB1_1 Depth=1
; GFX90A-NEXT: ; => This Inner Loop Header: Depth=2
-; GFX90A-NEXT: v_add_co_u32_e64 v18, s[0:1], s3, v6
-; GFX90A-NEXT: v_addc_co_u32_e64 v19, s[0:1], 0, v7, s[0:1]
-; GFX90A-NEXT: v_add_co_u32_e64 v20, s[0:1], s4, v6
-; GFX90A-NEXT: v_add_co_u32_e32 v8, vcc, 0xffffe000, v6
-; GFX90A-NEXT: v_addc_co_u32_e64 v21, s[0:1], 0, v7, s[0:1]
-; GFX90A-NEXT: v_addc_co_u32_e32 v9, vcc, -1, v7, vcc
-; GFX90A-NEXT: global_load_dwordx2 v[24:25], v[20:21], off offset:-4096
-; GFX90A-NEXT: global_load_dwordx2 v[26:27], v[20:21], off
-; GFX90A-NEXT: global_load_dwordx2 v[28:29], v[8:9], off offset:-2048
-; GFX90A-NEXT: global_load_dwordx2 v[30:31], v[8:9], off
+; GFX90A-NEXT: v_add_co_u32_e32 v12, vcc, 0xffffb000, v6
+; GFX90A-NEXT: v_addc_co_u32_e32 v13, vcc, -1, v7, vcc
+; GFX90A-NEXT: global_load_dwordx2 v[12:13], v[12:13], off
+; GFX90A-NEXT: v_add_co_u32_e32 v14, vcc, 0xffffc000, v6
+; GFX90A-NEXT: v_addc_co_u32_e32 v15, vcc, -1, v7, vcc
+; GFX90A-NEXT: global_load_dwordx2 v[18:19], v[14:15], off offset:-2048
+; GFX90A-NEXT: global_load_dwordx2 v[20:21], v[14:15], off
+; GFX90A-NEXT: v_add_co_u32_e32 v16, vcc, s0, v6
+; GFX90A-NEXT: v_addc_co_u32_e32 v17, vcc, -1, v7, vcc
+; GFX90A-NEXT: global_load_dwordx2 v[16:17], v[16:17], off offset:-2048
+; GFX90A-NEXT: v_add_co_u32_e32 v14, vcc, s1, v6
+; GFX90A-NEXT: v_addc_co_u32_e32 v15, vcc, -1, v7, vcc
+; GFX90A-NEXT: global_load_dwordx2 v[24:25], v[14:15], off offset:-4096
+; GFX90A-NEXT: global_load_dwordx2 v[26:27], v[14:15], off offset:-2048
+; GFX90A-NEXT: global_load_dwordx2 v[28:29], v[14:15], off
; GFX90A-NEXT: v_add_co_u32_e32 v22, vcc, s2, v6
; GFX90A-NEXT: v_addc_co_u32_e32 v23, vcc, -1, v7, vcc
-; GFX90A-NEXT: global_load_dwordx2 v[8:9], v[22:23], off offset:-2048
-; GFX90A-NEXT: s_nop 0
-; GFX90A-NEXT: global_load_dwordx2 v[18:19], v[18:19], off offset:2048
-; GFX90A-NEXT: s_nop 0
-; GFX90A-NEXT: global_load_dwordx2 v[20:21], v[20:21], off offset:2048
-; GFX90A-NEXT: s_nop 0
-; GFX90A-NEXT: global_load_dwordx2 v[10:11], v[6:7], off offset:-4096
-; GFX90A-NEXT: global_load_dwordx2 v[12:13], v[6:7], off offset:-2048
-; GFX90A-NEXT: global_load_dwordx2 v[14:15], v[6:7], off
-; GFX90A-NEXT: global_load_dwordx2 v[16:17], v[6:7], off offset:2048
+; GFX90A-NEXT: global_load_dwordx2 v[14:15], v[22:23], off offset:-2048
+; GFX90A-NEXT: global_load_dwordx2 v[30:31], v[6:7], off
+; GFX90A-NEXT: global_load_dwordx2 v[8:9], v[6:7], off offset:-4096
+; GFX90A-NEXT: global_load_dwordx2 v[10:11], v[6:7], off offset:-2048
; GFX90A-NEXT: v_add_co_u32_e32 v6, vcc, 0x10000, v6
; GFX90A-NEXT: v_addc_co_u32_e32 v7, vcc, 0, v7, vcc
-; GFX90A-NEXT: s_addk_i32 s5, 0x2000
-; GFX90A-NEXT: s_cmp_gt_u32 s5, 0x3fffff
-; GFX90A-NEXT: s_waitcnt vmcnt(8)
-; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v28, v4
-; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v29, v5, vcc
-; GFX90A-NEXT: s_waitcnt vmcnt(7)
-; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v30, v4
-; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v31, v5, vcc
-; GFX90A-NEXT: s_waitcnt vmcnt(6)
-; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v8, v4
-; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v9, v5, vcc
-; GFX90A-NEXT: s_waitcnt vmcnt(3)
-; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v10, v4
-; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v11, v5, vcc
-; GFX90A-NEXT: s_waitcnt vmcnt(2)
+; GFX90A-NEXT: s_addk_i32 s3, 0x2000
+; GFX90A-NEXT: s_cmp_gt_u32 s3, 0x3fffff
+; GFX90A-NEXT: s_waitcnt vmcnt(10)
; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v12, v4
; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v13, v5, vcc
-; GFX90A-NEXT: s_waitcnt vmcnt(1)
-; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v14, v4
-; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v15, v5, vcc
-; GFX90A-NEXT: s_waitcnt vmcnt(0)
+; GFX90A-NEXT: s_waitcnt vmcnt(9)
+; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v18, v4
+; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v19, v5, vcc
+; GFX90A-NEXT: s_waitcnt vmcnt(8)
+; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v20, v4
+; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v21, v5, vcc
+; GFX90A-NEXT: s_waitcnt vmcnt(7)
; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v16, v4
; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v17, v5, vcc
+; GFX90A-NEXT: s_waitcnt vmcnt(6)
; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v24, v4
; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v25, v5, vcc
-; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v18, v4
-; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v19, v5, vcc
+; GFX90A-NEXT: s_waitcnt vmcnt(5)
; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v26, v4
; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v27, v5, vcc
-; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v20, v4
-; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v21, v5, vcc
+; GFX90A-NEXT: s_waitcnt vmcnt(4)
+; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v28, v4
+; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v29, v5, vcc
+; GFX90A-NEXT: s_waitcnt vmcnt(3)
+; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v14, v4
+; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v15, v5, vcc
+; GFX90A-NEXT: s_waitcnt vmcnt(1)
+; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v8, v4
+; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v9, v5, vcc
+; GFX90A-NEXT: s_waitcnt vmcnt(0)
+; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v10, v4
+; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v11, v5, vcc
+; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, v30, v4
+; GFX90A-NEXT: v_addc_co_u32_e32 v5, vcc, v31, v5, vcc
; GFX90A-NEXT: s_cbranch_scc0 .LBB1_2
; GFX90A-NEXT: ; %bb.3: ; %while.cond.loopexit
; GFX90A-NEXT: ; in Loop: Header=BB1_1 Depth=1
@@ -824,7 +823,7 @@ define hidden amdgpu_kernel void @clmem_read(ptr addrspace(1) %buffer) {
; GFX11-NEXT: v_add_co_u32 v0, s0, s34, v0
; GFX11-NEXT: v_add_co_ci_u32_e64 v1, null, s35, 0, s0
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_2) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11-NEXT: v_add_co_u32 v0, vcc_lo, 0x2800, v0
+; GFX11-NEXT: v_add_co_u32 v0, vcc_lo, 0x5000, v0
; GFX11-NEXT: v_add_co_ci_u32_e64 v1, null, 0, v1, vcc_lo
; GFX11-NEXT: .LBB1_1: ; %for.cond.preheader
; GFX11-NEXT: ; =>This Loop Header: Depth=1
@@ -836,74 +835,76 @@ define hidden amdgpu_kernel void @clmem_read(ptr addrspace(1) %buffer) {
; GFX11-NEXT: ; Parent Loop BB1_1 Depth=1
; GFX11-NEXT: ; => This Inner Loop Header: Depth=2
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_1)
-; GFX11-NEXT: v_add_co_u32 v8, vcc_lo, 0xffffe000, v4
+; GFX11-NEXT: v_add_co_u32 v8, vcc_lo, v4, 0xffffc000
; GFX11-NEXT: v_add_co_ci_u32_e64 v9, null, -1, v5, vcc_lo
-; GFX11-NEXT: v_add_co_u32 v10, vcc_lo, 0xfffff000, v4
+; GFX11-NEXT: v_add_co_u32 v10, vcc_lo, 0xffffc000, v4
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_2) | instid1(VALU_DEP_1)
; GFX11-NEXT: v_add_co_ci_u32_e64 v11, null, -1, v5, vcc_lo
-; GFX11-NEXT: global_load_b64 v[12:13], v[8:9], off offset:-2048
-; GFX11-NEXT: v_add_co_u32 v22, vcc_lo, v4, 0x2000
-; GFX11-NEXT: v_add_co_ci_u32_e64 v23, null, 0, v5, vcc_lo
-; GFX11-NEXT: v_add_co_u32 v24, vcc_lo, 0x1000, v4
-; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
-; GFX11-NEXT: v_add_co_ci_u32_e64 v25, null, 0, v5, vcc_lo
-; GFX11-NEXT: global_load_b64 v[26:27], v[22:23], off offset:-4096
-; GFX11-NEXT: v_add_co_u32 v28, vcc_lo, 0x2000, v4
-; GFX11-NEXT: s_clause 0x6
-; GFX11-NEXT: global_load_b64 v[24:25], v[24:25], off offset:2048
-; GFX11-NEXT: global_load_b64 v[8:9], v[8:9], off
+; GFX11-NEXT: global_load_b64 v[14:15], v[8:9], off offset:-4096
+; GFX11-NEXT: v_add_co_u32 v12, vcc_lo, 0xffffd000, v4
+; GFX11-NEXT: v_add_co_ci_u32_e64 v13, null, -1, v5, vcc_lo
+; GFX11-NEXT: v_add_co_u32 v16, vcc_lo, v4, 0xffffe000
; GFX11-NEXT: global_load_b64 v[10:11], v[10:11], off offset:-2048
-; GFX11-NEXT: global_load_b64 v[14:15], v[4:5], off offset:-4096
-; GFX11-NEXT: global_load_b64 v[16:17], v[4:5], off offset:-2048
-; GFX11-NEXT: global_load_b64 v[18:19], v[4:5], off
-; GFX11-NEXT: global_load_b64 v[20:21], v[4:5], off offset:2048
-; GFX11-NEXT: v_add_co_ci_u32_e64 v29, null, 0, v5, vcc_lo
+; GFX11-NEXT: v_add_co_ci_u32_e64 v17, null, -1, v5, vcc_lo
+; GFX11-NEXT: global_load_b64 v[12:13], v[12:13], off offset:-2048
+; GFX11-NEXT: v_add_co_u32 v18, vcc_lo, 0xffffe000, v4
; GFX11-NEXT: s_clause 0x1
-; GFX11-NEXT: global_load_b64 v[22:23], v[22:23], off
-; GFX11-NEXT: global_load_b64 v[28:29], v[28:29], off offset:2048
+; GFX11-NEXT: global_load_b64 v[20:21], v[16:17], off offset:-4096
+; GFX11-NEXT: global_load_b64 v[8:9], v[8:9], off
+; GFX11-NEXT: v_add_co_ci_u32_e64 v19, null, -1, v5, vcc_lo
+; GFX11-NEXT: v_add_co_u32 v22, vcc_lo, 0xfffff000, v4
+; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
+; GFX11-NEXT: v_add_co_ci_u32_e64 v23, null, -1, v5, vcc_lo
+; GFX11-NEXT: s_clause 0x5
+; GFX11-NEXT: global_load_b64 v[18:19], v[18:19], off offset:-2048
+; GFX11-NEXT: global_load_b64 v[16:17], v[16:17], off
+; GFX11-NEXT: global_load_b64 v[22:23], v[22:23], off offset:-2048
+; GFX11-NEXT: global_load_b64 v[24:25], v[4:5], off offset:-4096
+; GFX11-NEXT: global_load_b64 v[26:27], v[4:5], off offset:-2048
+; GFX11-NEXT: global_load_b64 v[28:29], v[4:5], off
; GFX11-NEXT: v_add_co_u32 v4, vcc_lo, 0x10000, v4
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(SALU_CYCLE_1)
; GFX11-NEXT: v_add_co_ci_u32_e64 v5, null, 0, v5, vcc_lo
; GFX11-NEXT: s_addk_i32 s1, 0x2000
; GFX11-NEXT: s_cmp_gt_u32 s1, 0x3fffff
; GFX11-NEXT: s_waitcnt vmcnt(10)
-; GFX11-NEXT: v_add_co_u32 v2, s0, v12, v2
-; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11-NEXT: v_add_co_ci_u32_e64 v3, null, v13, v3, s0
-; GFX11-NEXT: s_waitcnt vmcnt(7)
-; GFX11-NEXT: v_add_co_u32 v2, s0, v8, v2
+; GFX11-NEXT: v_add_co_u32 v2, s0, v14, v2
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11-NEXT: v_add_co_ci_u32_e64 v3, null, v9, v3, s0
-; GFX11-NEXT: s_waitcnt vmcnt(6)
+; GFX11-NEXT: v_add_co_ci_u32_e64 v3, null, v15, v3, s0
+; GFX11-NEXT: s_waitcnt vmcnt(9)
; GFX11-NEXT: v_add_co_u32 v2, s0, v10, v2
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
; GFX11-NEXT: v_add_co_ci_u32_e64 v3, null, v11, v3, s0
+; GFX11-NEXT: s_waitcnt vmcnt(6)
+; GFX11-NEXT: v_add_co_u32 v2, s0, v8, v2
+; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT: v_add_co_ci_u32_e64 v3, null, v9, v3, s0
+; GFX11-NEXT: v_add_co_u32 v2, s0, v12, v2
+; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT: v_add_co_ci_u32_e64 v3, null, v13, v3, s0
+; GFX11-NEXT: v_add_co_u32 v2, s0, v20, v2
+; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
+; GFX11-NEXT: v_add_co_ci_u32_e64 v3, null, v21, v3, s0
; GFX11-NEXT: s_waitcnt vmcnt(5)
-; GFX11-NEXT: v_add_co_u32 v2, s0, v14, v2
+; GFX11-NEXT: v_add_co_u32 v2, s0, v18, v2
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11-NEXT: v_add_co_ci_u32_e64 v3, null, v15, v3, s0
+; GFX11-NEXT: v_add_co_ci_u32_e64 v3, null, v19, v3, s0
; GFX11-NEXT: s_waitcnt vmcnt(4)
; GFX11-NEXT: v_add_co_u32 v2, s0, v16, v2
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
; GFX11-NEXT: v_add_co_ci_u32_e64 v3, null, v17, v3, s0
; GFX11-NEXT: s_waitcnt vmcnt(3)
-; GFX11-NEXT: v_add_co_u32 v2, s0, v18, v2
+; GFX11-NEXT: v_add_co_u32 v2, s0, v22, v2
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11-NEXT: v_add_co_ci_u32_e64 v3, null, v19, v3, s0
+; GFX11-NEXT: v_add_co_ci_u32_e64 v3, null, v23, v3, s0
; GFX11-NEXT: s_waitcnt vmcnt(2)
-; GFX11-NEXT: v_add_co_u32 v2, s0, v20, v2
-; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT: v_add_co_ci_u32_e64 v3, null, v21, v3, s0
-; GFX11-NEXT: v_add_co_u32 v2, s0, v26, v2
-; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
-; GFX11-NEXT: v_add_co_ci_u32_e64 v3, null, v27, v3, s0
; GFX11-NEXT: v_add_co_u32 v2, s0, v24, v2
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
; GFX11-NEXT: v_add_co_ci_u32_e64 v3, null, v25, v3, s0
; GFX11-NEXT: s_waitcnt vmcnt(1)
-; GFX11-NEXT: v_add_co_u32 v2, s0, v22, v2
+; GFX11-NEXT: v_add_co_u32 v2, s0, v26, v2
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_2)
-; GFX11-NEXT: v_add_co_ci_u32_e64 v3, null, v23, v3, s0
+; GFX11-NEXT: v_add_co_ci_u32_e64 v3, null, v27, v3, s0
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: v_add_co_u32 v2, vcc_lo, v28, v2
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_1)
diff --git a/llvm/test/CodeGen/AMDGPU/splitkit-getsubrangeformask.ll b/llvm/test/CodeGen/AMDGPU/splitkit-getsubrangeformask.ll
index c0c1763d54cc0..67dae136afb72 100644
--- a/llvm/test/CodeGen/AMDGPU/splitkit-getsubrangeformask.ll
+++ b/llvm/test/CodeGen/AMDGPU/splitkit-getsubrangeformask.ll
@@ -146,7 +146,7 @@ define amdgpu_gs void @_amdgpu_gs_main(i32 inreg %primShaderTableAddrLow, <31 x
; CHECK-NEXT: [[S_ASHR_I32_5:%[0-9]+]]:sreg_32_xm0 = S_ASHR_I32 [[S_LSHL_B32_4]], 31, implicit-def dead $scc
; CHECK-NEXT: undef [[S_ADD_U32_18:%[0-9]+]].sub0:sreg_64 = S_ADD_U32 [[COPY]].sub0, [[S_LSHL_B32_4]], implicit-def $scc
; CHECK-NEXT: [[S_ADD_U32_18:%[0-9]+]].sub1:sreg_64 = S_ADDC_U32 undef %57:sreg_32, [[S_ASHR_I32_5]], implicit-def dead $scc, implicit $scc
- ; CHECK-NEXT: [[S_LOAD_DWORD_IMM:%[0-9]+]]:sreg_32_xm0_xexec = S_LOAD_DWORD_IMM [[S_ADD_U32_18]], 168, 0 :: (invariant load (s32) from %ir.276, align 8, addrspace 4)
+ ; CHECK-NEXT: [[S_LOAD_DWORD_IMM:%[0-9]+]]:sreg_32_xm0_xexec = S_LOAD_DWORD_IMM [[S_ADD_U32_18]], 168, 0 :: (invariant load (s32) from %ir.275, align 8, addrspace 4)
; CHECK-NEXT: [[S_LOAD_DWORDX4_IMM14:%[0-9]+]]:sgpr_128 = S_LOAD_DWORDX4_IMM [[S_ADD_U32_8]], 576, 0 :: (invariant load (s128) from %ir.159, addrspace 4)
; CHECK-NEXT: [[BUFFER_LOAD_FORMAT_X_IDXEN11:%[0-9]+]]:vgpr_32 = BUFFER_LOAD_FORMAT_X_IDXEN [[V_MOV_B32_e32_]], [[S_LOAD_DWORDX4_IMM13]], 0, 0, 0, 0, implicit $exec :: (dereferenceable load (s32), align 1, addrspace 8)
; CHECK-NEXT: [[BUFFER_LOAD_FORMAT_X_IDXEN12:%[0-9]+]]:vgpr_32 = BUFFER_LOAD_FORMAT_X_IDXEN [[V_MOV_B32_e32_]], [[S_LOAD_DWORDX4_IMM9]], 0, 0, 0, 0, implicit $exec :: (dereferenceable load (s32), align 1, addrspace 8)
@@ -169,7 +169,7 @@ define amdgpu_gs void @_amdgpu_gs_main(i32 inreg %primShaderTableAddrLow, <31 x
; CHECK-NEXT: [[S_ADD_I32_14:%[0-9]+]]:sreg_32 = S_ADD_I32 [[S_BUFFER_LOAD_DWORD_IMM4]], -467, implicit-def dead $scc
; CHECK-NEXT: undef [[S_ADD_U32_19:%[0-9]+]].sub0:sreg_64 = S_ADD_U32 [[COPY]].sub0, [[S_LSHL_B32_5]], implicit-def $scc
; CHECK-NEXT: [[S_ADD_U32_19:%[0-9]+]].sub1:sreg_64 = S_ADDC_U32 undef %57:sreg_32, [[S_ASHR_I32_6]], implicit-def dead $scc, implicit $scc
- ; CHECK-NEXT: [[S_LOAD_DWORDX2_IMM1:%[0-9]+]]:sreg_64_xexec = S_LOAD_DWORDX2_IMM [[S_ADD_U32_19]], 168, 0 :: (invariant load (s64) from %ir.285, addrspace 4)
+ ; CHECK-NEXT: [[S_LOAD_DWORDX2_IMM1:%[0-9]+]]:sreg_64_xexec = S_LOAD_DWORDX2_IMM [[S_ADD_U32_19]], 168, 0 :: (invariant load (s64) from %ir.284, addrspace 4)
; CHECK-NEXT: [[BUFFER_LOAD_DWORD_OFFSET2:%[0-9]+]]:vgpr_32 = BUFFER_LOAD_DWORD_OFFSET [[S_LOAD_DWORDX4_IMM16]], 0, 0, 0, 0, implicit $exec :: (dereferenceable load (s32), align 1, addrspace 8)
; CHECK-NEXT: [[BUFFER_LOAD_DWORD_OFFSET3:%[0-9]+]]:vgpr_32 = BUFFER_LOAD_DWORD_OFFSET [[S_LOAD_DWORDX4_IMM17]], 0, 0, 0, 0, implicit $exec :: (dereferenceable load (s32), align 1, addrspace 8)
; CHECK-NEXT: [[S_LOAD_DWORDX4_IMM18:%[0-9]+]]:sgpr_128 = S_LOAD_DWORDX4_IMM [[S_ADD_U32_12]], 0, 0 :: (invariant load (s128) from %ir.207, addrspace 4)
@@ -190,20 +190,20 @@ define amdgpu_gs void @_amdgpu_gs_main(i32 inreg %primShaderTableAddrLow, <31 x
; CHECK-NEXT: [[S_ADD_I32_15:%[0-9]+]]:sreg_32 = S_ADD_I32 [[S_BUFFER_LOAD_DWORD_IMM5]], -468, implicit-def dead $scc
; CHECK-NEXT: undef [[S_ADD_U32_20:%[0-9]+]].sub0:sreg_64 = S_ADD_U32 [[COPY]].sub0, [[S_LSHL_B32_6]], implicit-def $scc
; CHECK-NEXT: [[S_ADD_U32_20:%[0-9]+]].sub1:sreg_64 = S_ADDC_U32 undef %57:sreg_32, [[S_ASHR_I32_7]], implicit-def dead $scc, implicit $scc
- ; CHECK-NEXT: [[S_LOAD_DWORDX2_IMM2:%[0-9]+]]:sreg_64_xexec = S_LOAD_DWORDX2_IMM [[S_ADD_U32_20]], 168, 0 :: (invariant load (s64) from %ir.296, addrspace 4)
+ ; CHECK-NEXT: [[S_LOAD_DWORDX2_IMM2:%[0-9]+]]:sreg_64_xexec = S_LOAD_DWORDX2_IMM [[S_ADD_U32_20]], 168, 0 :: (invariant load (s64) from %ir.295, addrspace 4)
; CHECK-NEXT: [[COPY17:%[0-9]+]]:sgpr_128 = COPY [[S_LOAD_DWORDX2_IMM]]
; CHECK-NEXT: [[S_AND_B32_1:%[0-9]+]]:sreg_32 = S_AND_B32 [[S_LOAD_DWORDX2_IMM2]].sub1, 65535, implicit-def dead $scc
; CHECK-NEXT: [[COPY17:%[0-9]+]].sub0:sgpr_128 = COPY [[S_LOAD_DWORDX2_IMM2]].sub0
; CHECK-NEXT: [[COPY17:%[0-9]+]].sub1:sgpr_128 = COPY [[S_AND_B32_1]]
; CHECK-NEXT: [[S_BUFFER_LOAD_DWORD_IMM6:%[0-9]+]]:sreg_32_xm0_xexec = S_BUFFER_LOAD_DWORD_IMM [[COPY17]], 0, 0 :: (dereferenceable invariant load (s32))
- ; CHECK-NEXT: [[S_LOAD_DWORDX4_IMM22:%[0-9]+]]:sgpr_128 = S_LOAD_DWORDX4_IMM [[S_ADD_U32_16]], 160, 0 :: (invariant load (s128) from %ir.259, addrspace 4)
+ ; CHECK-NEXT: [[S_LOAD_DWORDX4_IMM22:%[0-9]+]]:sgpr_128 = S_LOAD_DWORDX4_IMM [[S_ADD_U32_16]], 160, 0 :: (invariant load (s128) from %ir.258, addrspace 4)
; CHECK-NEXT: [[S_LSHL_B32_7:%[0-9]+]]:sreg_32 = S_LSHL_B32 [[COPY14]], 3, implicit-def dead $scc
- ; CHECK-NEXT: [[S_LOAD_DWORDX4_IMM23:%[0-9]+]]:sgpr_128 = S_LOAD_DWORDX4_IMM [[S_ADD_U32_17]], 160, 0 :: (invariant load (s128) from %ir.268, addrspace 4)
+ ; CHECK-NEXT: [[S_LOAD_DWORDX4_IMM23:%[0-9]+]]:sgpr_128 = S_LOAD_DWORDX4_IMM [[S_ADD_U32_17]], 160, 0 :: (invariant load (s128) from %ir.267, addrspace 4)
; CHECK-NEXT: [[S_ASHR_I32_8:%[0-9]+]]:sreg_32_xm0 = S_ASHR_I32 [[S_LSHL_B32_7]], 31, implicit-def dead $scc
; CHECK-NEXT: [[S_ADD_I32_16:%[0-9]+]]:sreg_32 = S_ADD_I32 [[S_BUFFER_LOAD_DWORD_IMM6]], -469, implicit-def dead $scc
; CHECK-NEXT: undef [[S_ADD_U32_21:%[0-9]+]].sub0:sreg_64 = S_ADD_U32 [[COPY]].sub0, [[S_LSHL_B32_7]], implicit-def $scc
; CHECK-NEXT: [[S_ADD_U32_21:%[0-9]+]].sub1:sreg_64 = S_ADDC_U32 undef %57:sreg_32, [[S_ASHR_I32_8]], implicit-def dead $scc, implicit $scc
- ; CHECK-NEXT: [[S_LOAD_DWORD_IMM1:%[0-9]+]]:sreg_32_xm0_xexec = S_LOAD_DWORD_IMM [[S_ADD_U32_21]], 168, 0 :: (invariant load (s32) from %ir.308, align 8, addrspace 4)
+ ; CHECK-NEXT: [[S_LOAD_DWORD_IMM1:%[0-9]+]]:sreg_32_xm0_xexec = S_LOAD_DWORD_IMM [[S_ADD_U32_21]], 168, 0 :: (invariant load (s32) from %ir.307, align 8, addrspace 4)
; CHECK-NEXT: [[BUFFER_LOAD_FORMAT_X_IDXEN21:%[0-9]+]]:vgpr_32 = BUFFER_LOAD_FORMAT_X_IDXEN [[V_MOV_B32_e32_]], [[S_LOAD_DWORDX4_IMM22]], 0, 0, 0, 0, implicit $exec :: (dereferenceable load (s32), align 1, addrspace 8)
; CHECK-NEXT: [[BUFFER_LOAD_FORMAT_X_IDXEN22:%[0-9]+]]:vgpr_32 = BUFFER_LOAD_FORMAT_X_IDXEN [[V_MOV_B32_e32_]], [[S_LOAD_DWORDX4_IMM23]], 0, 0, 0, 0, implicit $exec :: (dereferenceable load (s32), align 1, addrspace 8)
; CHECK-NEXT: KILL [[S_LOAD_DWORDX4_IMM23]]
@@ -221,13 +221,13 @@ define amdgpu_gs void @_amdgpu_gs_main(i32 inreg %primShaderTableAddrLow, <31 x
; CHECK-NEXT: [[S_ADD_I32_22:%[0-9]+]]:sreg_32 = S_ADD_I32 [[S_BUFFER_LOAD_DWORD_IMM7]], -473, implicit-def dead $scc
; CHECK-NEXT: undef [[S_ADD_U32_22:%[0-9]+]].sub0:sreg_64 = S_ADD_U32 [[COPY1]], [[S_LSHL_B32_]], implicit-def $scc
; CHECK-NEXT: [[S_ADD_U32_22:%[0-9]+]].sub1:sreg_64 = S_ADDC_U32 undef %33:sreg_32, [[S_ASHR_I32_]], implicit-def dead $scc, implicit $scc
- ; CHECK-NEXT: [[S_LOAD_DWORDX4_IMM24:%[0-9]+]]:sgpr_128 = S_LOAD_DWORDX4_IMM [[S_ADD_U32_22]], 96, 0 :: (invariant load (s128) from %ir.326, addrspace 4)
+ ; CHECK-NEXT: [[S_LOAD_DWORDX4_IMM24:%[0-9]+]]:sgpr_128 = S_LOAD_DWORDX4_IMM [[S_ADD_U32_22]], 96, 0 :: (invariant load (s128) from %ir.325, addrspace 4)
; CHECK-NEXT: undef [[S_ADD_U32_23:%[0-9]+]].sub0:sreg_64 = S_ADD_U32 [[COPY1]], [[S_LSHL_B32_1]], implicit-def $scc
; CHECK-NEXT: [[S_ADD_U32_23:%[0-9]+]].sub1:sreg_64 = S_ADDC_U32 undef %33:sreg_32, [[S_ASHR_I32_1]], implicit-def dead $scc, implicit $scc
- ; CHECK-NEXT: [[S_LOAD_DWORDX4_IMM25:%[0-9]+]]:sgpr_128 = S_LOAD_DWORDX4_IMM [[S_ADD_U32_23]], 96, 0 :: (invariant load (s128) from %ir.332, addrspace 4)
+ ; CHECK-NEXT: [[S_LOAD_DWORDX4_IMM25:%[0-9]+]]:sgpr_128 = S_LOAD_DWORDX4_IMM [[S_ADD_U32_23]], 96, 0 :: (invariant load (s128) from %ir.331, addrspace 4)
; CHECK-NEXT: undef [[S_ADD_U32_24:%[0-9]+]].sub0:sreg_64 = S_ADD_U32 [[COPY1]], [[S_LSHL_B32_2]], implicit-def $scc
; CHECK-NEXT: [[S_ADD_U32_24:%[0-9]+]].sub1:sreg_64 = S_ADDC_U32 undef %33:sreg_32, [[S_ASHR_I32_2]], implicit-def dead $scc, implicit $scc
- ; CHECK-NEXT: [[S_LOAD_DWORDX4_IMM26:%[0-9]+]]:sgpr_128 = S_LOAD_DWORDX4_IMM [[S_ADD_U32_24]], 96, 0 :: (invariant load (s128) from %ir.338, addrspace 4)
+ ; CHECK-NEXT: [[S_LOAD_DWORDX4_IMM26:%[0-9]+]]:sgpr_128 = S_LOAD_DWORDX4_IMM [[S_ADD_U32_24]], 96, 0 :: (invariant load (s128) from %ir.337, addrspace 4)
; CHECK-NEXT: [[BUFFER_LOAD_FORMAT_X_IDXEN23:%[0-9]+]]:vgpr_32 = BUFFER_LOAD_FORMAT_X_IDXEN [[V_MOV_B32_e32_]], [[S_LOAD_DWORDX4_IMM24]], 0, 0, 0, 0, implicit $exec :: (dereferenceable load (s32), align 1, addrspace 8)
; CHECK-NEXT: [[BUFFER_LOAD_FORMAT_X_IDXEN24:%[0-9]+]]:vgpr_32 = BUFFER_LOAD_FORMAT_X_IDXEN [[V_MOV_B32_e32_]], [[S_LOAD_DWORDX4_IMM25]], 0, 0, 0, 0, implicit $exec :: (dereferenceable load (s32), align 1, addrspace 8)
; CHECK-NEXT: [[BUFFER_LOAD_FORMAT_X_IDXEN25:%[0-9]+]]:vgpr_32 = BUFFER_LOAD_FORMAT_X_IDXEN [[V_MOV_B32_e32_]], [[S_LOAD_DWORDX4_IMM26]], 0, 0, 0, 0, implicit $exec :: (dereferenceable load (s32), align 1, addrspace 8)
diff --git a/llvm/test/CodeGen/AMDGPU/waitcnt-vscnt.ll b/llvm/test/CodeGen/AMDGPU/waitcnt-vscnt.ll
index f6922c75ff848..a0aee6c80703f 100644
--- a/llvm/test/CodeGen/AMDGPU/waitcnt-vscnt.ll
+++ b/llvm/test/CodeGen/AMDGPU/waitcnt-vscnt.ll
@@ -7,27 +7,36 @@
define amdgpu_kernel void @barrier_vmcnt_global(ptr addrspace(1) %arg) {
; GFX8-LABEL: barrier_vmcnt_global:
; GFX8: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX8-NEXT: v_lshlrev_b32_e32 v0, 2, v0
+; GFX8-NEXT: v_lshlrev_b32_e32 v1, 2, v0
; GFX8-NEXT: s_waitcnt lgkmcnt(0)
-; GFX8-NEXT: v_mov_b32_e32 v1, s1
+; GFX8-NEXT: v_mov_b32_e32 v3, s1
+; GFX8-NEXT: v_add_u32_e32 v1, vcc, s0, v1
+; GFX8-NEXT: v_addc_u32_e32 v2, vcc, 0, v3, vcc
+; GFX8-NEXT: flat_load_dword v4, v[1:2]
+; GFX8-NEXT: v_mov_b32_e32 v1, 0
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, 1, v0
+; GFX8-NEXT: v_lshrrev_b64 v[0:1], 30, v[1:2]
; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0
-; GFX8-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
-; GFX8-NEXT: flat_load_dword v2, v[0:1]
-; GFX8-NEXT: v_add_u32_e32 v0, vcc, 4, v0
-; GFX8-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GFX8-NEXT: v_addc_u32_e32 v1, vcc, v3, v1, vcc
; GFX8-NEXT: s_waitcnt vmcnt(0)
; GFX8-NEXT: s_barrier
-; GFX8-NEXT: flat_store_dword v[0:1], v2
+; GFX8-NEXT: flat_store_dword v[0:1], v4
; GFX8-NEXT: s_endpgm
;
; GFX9-LABEL: barrier_vmcnt_global:
; GFX9: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX9-NEXT: v_lshlrev_b32_e32 v0, 2, v0
+; GFX9-NEXT: v_lshlrev_b32_e32 v1, 2, v0
; GFX9-NEXT: s_waitcnt lgkmcnt(0)
-; GFX9-NEXT: global_load_dword v1, v0, s[0:1]
+; GFX9-NEXT: global_load_dword v2, v1, s[0:1]
+; GFX9-NEXT: v_add_u32_e32 v1, 1, v0
+; GFX9-NEXT: v_mov_b32_e32 v0, 0
+; GFX9-NEXT: v_lshrrev_b64 v[0:1], 30, v[0:1]
+; GFX9-NEXT: v_mov_b32_e32 v3, s1
+; GFX9-NEXT: v_add_co_u32_e32 v0, vcc, s0, v0
+; GFX9-NEXT: v_addc_co_u32_e32 v1, vcc, v3, v1, vcc
; GFX9-NEXT: s_waitcnt vmcnt(0)
; GFX9-NEXT: s_barrier
-; GFX9-NEXT: global_store_dword v0, v1, s[0:1] offset:4
+; GFX9-NEXT: global_store_dword v[0:1], v2, off
; GFX9-NEXT: s_endpgm
bb:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
@@ -48,20 +57,22 @@ bb:
define amdgpu_kernel void @barrier_vscnt_global(ptr addrspace(1) %arg) {
; GFX8-LABEL: barrier_vscnt_global:
; GFX8: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX8-NEXT: v_add_u32_e32 v1, vcc, 2, v0
-; GFX8-NEXT: v_mov_b32_e32 v0, 0
-; GFX8-NEXT: v_lshrrev_b64 v[1:2], 30, v[0:1]
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, 2, v0
+; GFX8-NEXT: v_mov_b32_e32 v1, 0
+; GFX8-NEXT: v_lshrrev_b64 v[2:3], 30, v[1:2]
; GFX8-NEXT: s_waitcnt lgkmcnt(0)
-; GFX8-NEXT: v_mov_b32_e32 v3, s1
-; GFX8-NEXT: v_add_u32_e32 v1, vcc, s0, v1
-; GFX8-NEXT: v_addc_u32_e32 v2, vcc, v3, v2, vcc
-; GFX8-NEXT: flat_store_dword v[1:2], v0
-; GFX8-NEXT: v_add_u32_e32 v0, vcc, -4, v1
-; GFX8-NEXT: v_addc_u32_e32 v1, vcc, -1, v2, vcc
-; GFX8-NEXT: v_mov_b32_e32 v2, 1
+; GFX8-NEXT: v_mov_b32_e32 v4, s1
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, s0, v2
+; GFX8-NEXT: v_addc_u32_e32 v3, vcc, v4, v3, vcc
+; GFX8-NEXT: flat_store_dword v[2:3], v1
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, 1, v0
+; GFX8-NEXT: v_lshrrev_b64 v[0:1], 30, v[1:2]
+; GFX8-NEXT: v_mov_b32_e32 v3, 1
+; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0
+; GFX8-NEXT: v_addc_u32_e32 v1, vcc, v4, v1, vcc
; GFX8-NEXT: s_waitcnt vmcnt(0)
; GFX8-NEXT: s_barrier
-; GFX8-NEXT: flat_store_dword v[0:1], v2
+; GFX8-NEXT: flat_store_dword v[0:1], v3
; GFX8-NEXT: s_endpgm
;
; GFX9-LABEL: barrier_vscnt_global:
@@ -70,14 +81,18 @@ define amdgpu_kernel void @barrier_vscnt_global(ptr addrspace(1) %arg) {
; GFX9-NEXT: v_add_u32_e32 v2, 2, v0
; GFX9-NEXT: v_lshrrev_b64 v[2:3], 30, v[1:2]
; GFX9-NEXT: s_waitcnt lgkmcnt(0)
-; GFX9-NEXT: v_mov_b32_e32 v0, s1
+; GFX9-NEXT: v_mov_b32_e32 v4, s1
; GFX9-NEXT: v_add_co_u32_e32 v2, vcc, s0, v2
-; GFX9-NEXT: v_addc_co_u32_e32 v3, vcc, v0, v3, vcc
-; GFX9-NEXT: v_mov_b32_e32 v0, 1
+; GFX9-NEXT: v_addc_co_u32_e32 v3, vcc, v4, v3, vcc
; GFX9-NEXT: global_store_dword v[2:3], v1, off
+; GFX9-NEXT: v_add_u32_e32 v2, 1, v0
+; GFX9-NEXT: v_lshrrev_b64 v[0:1], 30, v[1:2]
+; GFX9-NEXT: v_mov_b32_e32 v3, 1
+; GFX9-NEXT: v_add_co_u32_e32 v0, vcc, s0, v0
+; GFX9-NEXT: v_addc_co_u32_e32 v1, vcc, v4, v1, vcc
; GFX9-NEXT: s_waitcnt vmcnt(0)
; GFX9-NEXT: s_barrier
-; GFX9-NEXT: global_store_dword v[2:3], v0, off offset:-4
+; GFX9-NEXT: global_store_dword v[0:1], v3, off
; GFX9-NEXT: s_endpgm
bb:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
@@ -100,19 +115,22 @@ bb:
define amdgpu_kernel void @barrier_vmcnt_vscnt_global(ptr addrspace(1) %arg) {
; GFX8-LABEL: barrier_vmcnt_vscnt_global:
; GFX8: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX8-NEXT: v_add_u32_e32 v1, vcc, 2, v0
-; GFX8-NEXT: v_mov_b32_e32 v0, 0
-; GFX8-NEXT: v_lshrrev_b64 v[1:2], 30, v[0:1]
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, 2, v0
+; GFX8-NEXT: v_mov_b32_e32 v1, 0
+; GFX8-NEXT: v_lshrrev_b64 v[2:3], 30, v[1:2]
; GFX8-NEXT: s_waitcnt lgkmcnt(0)
-; GFX8-NEXT: v_mov_b32_e32 v3, s1
-; GFX8-NEXT: v_add_u32_e32 v1, vcc, s0, v1
-; GFX8-NEXT: v_addc_u32_e32 v2, vcc, v3, v2, vcc
-; GFX8-NEXT: v_add_u32_e32 v3, vcc, -8, v1
-; GFX8-NEXT: v_addc_u32_e32 v4, vcc, -1, v2, vcc
-; GFX8-NEXT: flat_load_dword v3, v[3:4]
-; GFX8-NEXT: flat_store_dword v[1:2], v0
-; GFX8-NEXT: v_add_u32_e32 v0, vcc, -4, v1
-; GFX8-NEXT: v_addc_u32_e32 v1, vcc, -1, v2, vcc
+; GFX8-NEXT: v_mov_b32_e32 v4, s1
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, s0, v2
+; GFX8-NEXT: v_addc_u32_e32 v3, vcc, v4, v3, vcc
+; GFX8-NEXT: flat_store_dword v[2:3], v1
+; GFX8-NEXT: v_lshlrev_b32_e32 v2, 2, v0
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, s0, v2
+; GFX8-NEXT: v_addc_u32_e32 v3, vcc, 0, v4, vcc
+; GFX8-NEXT: flat_load_dword v3, v[2:3]
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, 1, v0
+; GFX8-NEXT: v_lshrrev_b64 v[0:1], 30, v[1:2]
+; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0
+; GFX8-NEXT: v_addc_u32_e32 v1, vcc, v4, v1, vcc
; GFX8-NEXT: s_waitcnt vmcnt(0)
; GFX8-NEXT: s_barrier
; GFX8-NEXT: flat_store_dword v[0:1], v3
@@ -124,15 +142,19 @@ define amdgpu_kernel void @barrier_vmcnt_vscnt_global(ptr addrspace(1) %arg) {
; GFX9-NEXT: v_add_u32_e32 v2, 2, v0
; GFX9-NEXT: v_lshrrev_b64 v[2:3], 30, v[1:2]
; GFX9-NEXT: s_waitcnt lgkmcnt(0)
-; GFX9-NEXT: v_mov_b32_e32 v0, s1
+; GFX9-NEXT: v_mov_b32_e32 v4, s1
; GFX9-NEXT: v_add_co_u32_e32 v2, vcc, s0, v2
-; GFX9-NEXT: v_addc_co_u32_e32 v3, vcc, v0, v3, vcc
-; GFX9-NEXT: global_load_dword v0, v[2:3], off offset:-8
-; GFX9-NEXT: s_nop 0
+; GFX9-NEXT: v_addc_co_u32_e32 v3, vcc, v4, v3, vcc
; GFX9-NEXT: global_store_dword v[2:3], v1, off
+; GFX9-NEXT: v_lshlrev_b32_e32 v2, 2, v0
+; GFX9-NEXT: global_load_dword v3, v2, s[0:1]
+; GFX9-NEXT: v_add_u32_e32 v2, 1, v0
+; GFX9-NEXT: v_lshrrev_b64 v[0:1], 30, v[1:2]
+; GFX9-NEXT: v_add_co_u32_e32 v0, vcc, s0, v0
+; GFX9-NEXT: v_addc_co_u32_e32 v1, vcc, v4, v1, vcc
; GFX9-NEXT: s_waitcnt vmcnt(0)
; GFX9-NEXT: s_barrier
-; GFX9-NEXT: global_store_dword v[2:3], v0, off offset:-4
+; GFX9-NEXT: global_store_dword v[0:1], v3, off
; GFX9-NEXT: s_endpgm
bb:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
@@ -157,30 +179,38 @@ bb:
define amdgpu_kernel void @barrier_vmcnt_flat(ptr %arg) {
; GFX8-LABEL: barrier_vmcnt_flat:
; GFX8: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX8-NEXT: v_lshlrev_b32_e32 v0, 2, v0
+; GFX8-NEXT: v_lshlrev_b32_e32 v1, 2, v0
; GFX8-NEXT: s_waitcnt lgkmcnt(0)
-; GFX8-NEXT: v_mov_b32_e32 v1, s1
+; GFX8-NEXT: v_mov_b32_e32 v3, s1
+; GFX8-NEXT: v_add_u32_e32 v1, vcc, s0, v1
+; GFX8-NEXT: v_addc_u32_e32 v2, vcc, 0, v3, vcc
+; GFX8-NEXT: flat_load_dword v4, v[1:2]
+; GFX8-NEXT: v_mov_b32_e32 v1, 0
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, 1, v0
+; GFX8-NEXT: v_lshrrev_b64 v[0:1], 30, v[1:2]
; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0
-; GFX8-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
-; GFX8-NEXT: flat_load_dword v2, v[0:1]
-; GFX8-NEXT: v_add_u32_e32 v0, vcc, 4, v0
-; GFX8-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GFX8-NEXT: v_addc_u32_e32 v1, vcc, v3, v1, vcc
; GFX8-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
; GFX8-NEXT: s_barrier
-; GFX8-NEXT: flat_store_dword v[0:1], v2
+; GFX8-NEXT: flat_store_dword v[0:1], v4
; GFX8-NEXT: s_endpgm
;
; GFX9-LABEL: barrier_vmcnt_flat:
; GFX9: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX9-NEXT: v_lshlrev_b32_e32 v0, 2, v0
+; GFX9-NEXT: v_lshlrev_b32_e32 v1, 2, v0
; GFX9-NEXT: s_waitcnt lgkmcnt(0)
-; GFX9-NEXT: v_mov_b32_e32 v1, s1
+; GFX9-NEXT: v_mov_b32_e32 v3, s1
+; GFX9-NEXT: v_add_co_u32_e32 v1, vcc, s0, v1
+; GFX9-NEXT: v_addc_co_u32_e32 v2, vcc, 0, v3, vcc
+; GFX9-NEXT: flat_load_dword v4, v[1:2]
+; GFX9-NEXT: v_mov_b32_e32 v1, 0
+; GFX9-NEXT: v_add_u32_e32 v2, 1, v0
+; GFX9-NEXT: v_lshrrev_b64 v[0:1], 30, v[1:2]
; GFX9-NEXT: v_add_co_u32_e32 v0, vcc, s0, v0
-; GFX9-NEXT: v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
-; GFX9-NEXT: flat_load_dword v2, v[0:1]
+; GFX9-NEXT: v_addc_co_u32_e32 v1, vcc, v3, v1, vcc
; GFX9-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
; GFX9-NEXT: s_barrier
-; GFX9-NEXT: flat_store_dword v[0:1], v2 offset:4
+; GFX9-NEXT: flat_store_dword v[0:1], v4
; GFX9-NEXT: s_endpgm
bb:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
@@ -201,20 +231,22 @@ bb:
define amdgpu_kernel void @barrier_vscnt_flat(ptr %arg) {
; GFX8-LABEL: barrier_vscnt_flat:
; GFX8: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX8-NEXT: v_add_u32_e32 v1, vcc, 2, v0
-; GFX8-NEXT: v_mov_b32_e32 v0, 0
-; GFX8-NEXT: v_lshrrev_b64 v[1:2], 30, v[0:1]
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, 2, v0
+; GFX8-NEXT: v_mov_b32_e32 v1, 0
+; GFX8-NEXT: v_lshrrev_b64 v[2:3], 30, v[1:2]
; GFX8-NEXT: s_waitcnt lgkmcnt(0)
-; GFX8-NEXT: v_mov_b32_e32 v3, s1
-; GFX8-NEXT: v_add_u32_e32 v1, vcc, s0, v1
-; GFX8-NEXT: v_addc_u32_e32 v2, vcc, v3, v2, vcc
-; GFX8-NEXT: flat_store_dword v[1:2], v0
-; GFX8-NEXT: v_add_u32_e32 v0, vcc, -4, v1
-; GFX8-NEXT: v_addc_u32_e32 v1, vcc, -1, v2, vcc
-; GFX8-NEXT: v_mov_b32_e32 v2, 1
+; GFX8-NEXT: v_mov_b32_e32 v4, s1
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, s0, v2
+; GFX8-NEXT: v_addc_u32_e32 v3, vcc, v4, v3, vcc
+; GFX8-NEXT: flat_store_dword v[2:3], v1
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, 1, v0
+; GFX8-NEXT: v_lshrrev_b64 v[0:1], 30, v[1:2]
+; GFX8-NEXT: v_mov_b32_e32 v3, 1
+; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0
+; GFX8-NEXT: v_addc_u32_e32 v1, vcc, v4, v1, vcc
; GFX8-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
; GFX8-NEXT: s_barrier
-; GFX8-NEXT: flat_store_dword v[0:1], v2
+; GFX8-NEXT: flat_store_dword v[0:1], v3
; GFX8-NEXT: s_endpgm
;
; GFX9-LABEL: barrier_vscnt_flat:
@@ -223,16 +255,18 @@ define amdgpu_kernel void @barrier_vscnt_flat(ptr %arg) {
; GFX9-NEXT: v_add_u32_e32 v2, 2, v0
; GFX9-NEXT: v_lshrrev_b64 v[2:3], 30, v[1:2]
; GFX9-NEXT: s_waitcnt lgkmcnt(0)
-; GFX9-NEXT: v_mov_b32_e32 v0, s1
+; GFX9-NEXT: v_mov_b32_e32 v4, s1
; GFX9-NEXT: v_add_co_u32_e32 v2, vcc, s0, v2
-; GFX9-NEXT: v_addc_co_u32_e32 v3, vcc, v0, v3, vcc
-; GFX9-NEXT: v_add_co_u32_e32 v0, vcc, -4, v2
+; GFX9-NEXT: v_addc_co_u32_e32 v3, vcc, v4, v3, vcc
; GFX9-NEXT: flat_store_dword v[2:3], v1
-; GFX9-NEXT: v_addc_co_u32_e32 v1, vcc, -1, v3, vcc
-; GFX9-NEXT: v_mov_b32_e32 v2, 1
+; GFX9-NEXT: v_add_u32_e32 v2, 1, v0
+; GFX9-NEXT: v_lshrrev_b64 v[0:1], 30, v[1:2]
+; GFX9-NEXT: v_mov_b32_e32 v3, 1
+; GFX9-NEXT: v_add_co_u32_e32 v0, vcc, s0, v0
+; GFX9-NEXT: v_addc_co_u32_e32 v1, vcc, v4, v1, vcc
; GFX9-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
; GFX9-NEXT: s_barrier
-; GFX9-NEXT: flat_store_dword v[0:1], v2
+; GFX9-NEXT: flat_store_dword v[0:1], v3
; GFX9-NEXT: s_endpgm
bb:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
@@ -255,19 +289,22 @@ bb:
define amdgpu_kernel void @barrier_vmcnt_vscnt_flat(ptr %arg) {
; GFX8-LABEL: barrier_vmcnt_vscnt_flat:
; GFX8: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX8-NEXT: v_add_u32_e32 v1, vcc, 2, v0
-; GFX8-NEXT: v_mov_b32_e32 v0, 0
-; GFX8-NEXT: v_lshrrev_b64 v[1:2], 30, v[0:1]
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, 2, v0
+; GFX8-NEXT: v_mov_b32_e32 v1, 0
+; GFX8-NEXT: v_lshrrev_b64 v[2:3], 30, v[1:2]
; GFX8-NEXT: s_waitcnt lgkmcnt(0)
-; GFX8-NEXT: v_mov_b32_e32 v3, s1
-; GFX8-NEXT: v_add_u32_e32 v1, vcc, s0, v1
-; GFX8-NEXT: v_addc_u32_e32 v2, vcc, v3, v2, vcc
-; GFX8-NEXT: v_add_u32_e32 v3, vcc, -8, v1
-; GFX8-NEXT: v_addc_u32_e32 v4, vcc, -1, v2, vcc
-; GFX8-NEXT: flat_load_dword v3, v[3:4]
-; GFX8-NEXT: flat_store_dword v[1:2], v0
-; GFX8-NEXT: v_add_u32_e32 v0, vcc, -4, v1
-; GFX8-NEXT: v_addc_u32_e32 v1, vcc, -1, v2, vcc
+; GFX8-NEXT: v_mov_b32_e32 v4, s1
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, s0, v2
+; GFX8-NEXT: v_addc_u32_e32 v3, vcc, v4, v3, vcc
+; GFX8-NEXT: flat_store_dword v[2:3], v1
+; GFX8-NEXT: v_lshlrev_b32_e32 v2, 2, v0
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, s0, v2
+; GFX8-NEXT: v_addc_u32_e32 v3, vcc, 0, v4, vcc
+; GFX8-NEXT: flat_load_dword v3, v[2:3]
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, 1, v0
+; GFX8-NEXT: v_lshrrev_b64 v[0:1], 30, v[1:2]
+; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0
+; GFX8-NEXT: v_addc_u32_e32 v1, vcc, v4, v1, vcc
; GFX8-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
; GFX8-NEXT: s_barrier
; GFX8-NEXT: flat_store_dword v[0:1], v3
@@ -279,18 +316,21 @@ define amdgpu_kernel void @barrier_vmcnt_vscnt_flat(ptr %arg) {
; GFX9-NEXT: v_add_u32_e32 v2, 2, v0
; GFX9-NEXT: v_lshrrev_b64 v[2:3], 30, v[1:2]
; GFX9-NEXT: s_waitcnt lgkmcnt(0)
-; GFX9-NEXT: v_mov_b32_e32 v0, s1
+; GFX9-NEXT: v_mov_b32_e32 v4, s1
; GFX9-NEXT: v_add_co_u32_e32 v2, vcc, s0, v2
-; GFX9-NEXT: v_addc_co_u32_e32 v3, vcc, v0, v3, vcc
-; GFX9-NEXT: v_add_co_u32_e32 v4, vcc, -8, v2
-; GFX9-NEXT: v_addc_co_u32_e32 v5, vcc, -1, v3, vcc
-; GFX9-NEXT: flat_load_dword v4, v[4:5]
-; GFX9-NEXT: v_add_co_u32_e32 v0, vcc, -4, v2
+; GFX9-NEXT: v_addc_co_u32_e32 v3, vcc, v4, v3, vcc
; GFX9-NEXT: flat_store_dword v[2:3], v1
-; GFX9-NEXT: v_addc_co_u32_e32 v1, vcc, -1, v3, vcc
+; GFX9-NEXT: v_lshlrev_b32_e32 v2, 2, v0
+; GFX9-NEXT: v_add_co_u32_e32 v2, vcc, s0, v2
+; GFX9-NEXT: v_addc_co_u32_e32 v3, vcc, 0, v4, vcc
+; GFX9-NEXT: flat_load_dword v3, v[2:3]
+; GFX9-NEXT: v_add_u32_e32 v2, 1, v0
+; GFX9-NEXT: v_lshrrev_b64 v[0:1], 30, v[1:2]
+; GFX9-NEXT: v_add_co_u32_e32 v0, vcc, s0, v0
+; GFX9-NEXT: v_addc_co_u32_e32 v1, vcc, v4, v1, vcc
; GFX9-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
; GFX9-NEXT: s_barrier
-; GFX9-NEXT: flat_store_dword v[0:1], v4
+; GFX9-NEXT: flat_store_dword v[0:1], v3
; GFX9-NEXT: s_endpgm
bb:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
@@ -315,19 +355,22 @@ bb:
define amdgpu_kernel void @barrier_vmcnt_vscnt_flat_workgroup(ptr %arg) {
; GFX8-LABEL: barrier_vmcnt_vscnt_flat_workgroup:
; GFX8: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX8-NEXT: v_add_u32_e32 v1, vcc, 2, v0
-; GFX8-NEXT: v_mov_b32_e32 v0, 0
-; GFX8-NEXT: v_lshrrev_b64 v[1:2], 30, v[0:1]
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, 2, v0
+; GFX8-NEXT: v_mov_b32_e32 v1, 0
+; GFX8-NEXT: v_lshrrev_b64 v[2:3], 30, v[1:2]
; GFX8-NEXT: s_waitcnt lgkmcnt(0)
-; GFX8-NEXT: v_mov_b32_e32 v3, s1
-; GFX8-NEXT: v_add_u32_e32 v1, vcc, s0, v1
-; GFX8-NEXT: v_addc_u32_e32 v2, vcc, v3, v2, vcc
-; GFX8-NEXT: v_add_u32_e32 v3, vcc, -8, v1
-; GFX8-NEXT: v_addc_u32_e32 v4, vcc, -1, v2, vcc
-; GFX8-NEXT: flat_load_dword v3, v[3:4]
-; GFX8-NEXT: flat_store_dword v[1:2], v0
-; GFX8-NEXT: v_add_u32_e32 v0, vcc, -4, v1
-; GFX8-NEXT: v_addc_u32_e32 v1, vcc, -1, v2, vcc
+; GFX8-NEXT: v_mov_b32_e32 v4, s1
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, s0, v2
+; GFX8-NEXT: v_addc_u32_e32 v3, vcc, v4, v3, vcc
+; GFX8-NEXT: flat_store_dword v[2:3], v1
+; GFX8-NEXT: v_lshlrev_b32_e32 v2, 2, v0
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, s0, v2
+; GFX8-NEXT: v_addc_u32_e32 v3, vcc, 0, v4, vcc
+; GFX8-NEXT: flat_load_dword v3, v[2:3]
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, 1, v0
+; GFX8-NEXT: v_lshrrev_b64 v[0:1], 30, v[1:2]
+; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0
+; GFX8-NEXT: v_addc_u32_e32 v1, vcc, v4, v1, vcc
; GFX8-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
; GFX8-NEXT: s_barrier
; GFX8-NEXT: flat_store_dword v[0:1], v3
@@ -339,18 +382,21 @@ define amdgpu_kernel void @barrier_vmcnt_vscnt_flat_workgroup(ptr %arg) {
; GFX9-NEXT: v_add_u32_e32 v2, 2, v0
; GFX9-NEXT: v_lshrrev_b64 v[2:3], 30, v[1:2]
; GFX9-NEXT: s_waitcnt lgkmcnt(0)
-; GFX9-NEXT: v_mov_b32_e32 v0, s1
+; GFX9-NEXT: v_mov_b32_e32 v4, s1
; GFX9-NEXT: v_add_co_u32_e32 v2, vcc, s0, v2
-; GFX9-NEXT: v_addc_co_u32_e32 v3, vcc, v0, v3, vcc
-; GFX9-NEXT: v_add_co_u32_e32 v4, vcc, -8, v2
-; GFX9-NEXT: v_addc_co_u32_e32 v5, vcc, -1, v3, vcc
-; GFX9-NEXT: flat_load_dword v4, v[4:5]
-; GFX9-NEXT: v_add_co_u32_e32 v0, vcc, -4, v2
+; GFX9-NEXT: v_addc_co_u32_e32 v3, vcc, v4, v3, vcc
; GFX9-NEXT: flat_store_dword v[2:3], v1
-; GFX9-NEXT: v_addc_co_u32_e32 v1, vcc, -1, v3, vcc
+; GFX9-NEXT: v_lshlrev_b32_e32 v2, 2, v0
+; GFX9-NEXT: v_add_co_u32_e32 v2, vcc, s0, v2
+; GFX9-NEXT: v_addc_co_u32_e32 v3, vcc, 0, v4, vcc
+; GFX9-NEXT: flat_load_dword v3, v[2:3]
+; GFX9-NEXT: v_add_u32_e32 v2, 1, v0
+; GFX9-NEXT: v_lshrrev_b64 v[0:1], 30, v[1:2]
+; GFX9-NEXT: v_add_co_u32_e32 v0, vcc, s0, v0
+; GFX9-NEXT: v_addc_co_u32_e32 v1, vcc, v4, v1, vcc
; GFX9-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
; GFX9-NEXT: s_barrier
-; GFX9-NEXT: flat_store_dword v[0:1], v4
+; GFX9-NEXT: flat_store_dword v[0:1], v3
; GFX9-NEXT: s_endpgm
bb:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
@@ -375,25 +421,34 @@ bb:
define amdgpu_kernel void @load_vmcnt_global(ptr addrspace(1) %arg) {
; GFX8-LABEL: load_vmcnt_global:
; GFX8: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX8-NEXT: v_lshlrev_b32_e32 v0, 2, v0
+; GFX8-NEXT: v_lshlrev_b32_e32 v1, 2, v0
; GFX8-NEXT: s_waitcnt lgkmcnt(0)
-; GFX8-NEXT: v_mov_b32_e32 v1, s1
+; GFX8-NEXT: v_mov_b32_e32 v3, s1
+; GFX8-NEXT: v_add_u32_e32 v1, vcc, s0, v1
+; GFX8-NEXT: v_addc_u32_e32 v2, vcc, 0, v3, vcc
+; GFX8-NEXT: flat_load_dword v4, v[1:2]
+; GFX8-NEXT: v_mov_b32_e32 v1, 0
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, 1, v0
+; GFX8-NEXT: v_lshrrev_b64 v[0:1], 30, v[1:2]
; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0
-; GFX8-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
-; GFX8-NEXT: flat_load_dword v2, v[0:1]
-; GFX8-NEXT: v_add_u32_e32 v0, vcc, 4, v0
-; GFX8-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GFX8-NEXT: v_addc_u32_e32 v1, vcc, v3, v1, vcc
; GFX8-NEXT: s_waitcnt vmcnt(0)
-; GFX8-NEXT: flat_store_dword v[0:1], v2
+; GFX8-NEXT: flat_store_dword v[0:1], v4
; GFX8-NEXT: s_endpgm
;
; GFX9-LABEL: load_vmcnt_global:
; GFX9: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX9-NEXT: v_lshlrev_b32_e32 v0, 2, v0
+; GFX9-NEXT: v_lshlrev_b32_e32 v1, 2, v0
; GFX9-NEXT: s_waitcnt lgkmcnt(0)
-; GFX9-NEXT: global_load_dword v1, v0, s[0:1]
+; GFX9-NEXT: global_load_dword v2, v1, s[0:1]
+; GFX9-NEXT: v_add_u32_e32 v1, 1, v0
+; GFX9-NEXT: v_mov_b32_e32 v0, 0
+; GFX9-NEXT: v_lshrrev_b64 v[0:1], 30, v[0:1]
+; GFX9-NEXT: v_mov_b32_e32 v3, s1
+; GFX9-NEXT: v_add_co_u32_e32 v0, vcc, s0, v0
+; GFX9-NEXT: v_addc_co_u32_e32 v1, vcc, v3, v1, vcc
; GFX9-NEXT: s_waitcnt vmcnt(0)
-; GFX9-NEXT: global_store_dword v0, v1, s[0:1] offset:4
+; GFX9-NEXT: global_store_dword v[0:1], v2, off
; GFX9-NEXT: s_endpgm
bb:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
@@ -411,28 +466,36 @@ bb:
define amdgpu_kernel void @load_vmcnt_flat(ptr %arg) {
; GFX8-LABEL: load_vmcnt_flat:
; GFX8: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX8-NEXT: v_lshlrev_b32_e32 v0, 2, v0
+; GFX8-NEXT: v_lshlrev_b32_e32 v1, 2, v0
; GFX8-NEXT: s_waitcnt lgkmcnt(0)
-; GFX8-NEXT: v_mov_b32_e32 v1, s1
+; GFX8-NEXT: v_mov_b32_e32 v3, s1
+; GFX8-NEXT: v_add_u32_e32 v1, vcc, s0, v1
+; GFX8-NEXT: v_addc_u32_e32 v2, vcc, 0, v3, vcc
+; GFX8-NEXT: flat_load_dword v4, v[1:2]
+; GFX8-NEXT: v_mov_b32_e32 v1, 0
+; GFX8-NEXT: v_add_u32_e32 v2, vcc, 1, v0
+; GFX8-NEXT: v_lshrrev_b64 v[0:1], 30, v[1:2]
; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0
-; GFX8-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
-; GFX8-NEXT: flat_load_dword v2, v[0:1]
-; GFX8-NEXT: v_add_u32_e32 v0, vcc, 4, v0
-; GFX8-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GFX8-NEXT: v_addc_u32_e32 v1, vcc, v3, v1, vcc
; GFX8-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
-; GFX8-NEXT: flat_store_dword v[0:1], v2
+; GFX8-NEXT: flat_store_dword v[0:1], v4
; GFX8-NEXT: s_endpgm
;
; GFX9-LABEL: load_vmcnt_flat:
; GFX9: s_load_dwordx2 s[0:1], s[4:5], 0x24
-; GFX9-NEXT: v_lshlrev_b32_e32 v0, 2, v0
+; GFX9-NEXT: v_lshlrev_b32_e32 v1, 2, v0
; GFX9-NEXT: s_waitcnt lgkmcnt(0)
-; GFX9-NEXT: v_mov_b32_e32 v1, s1
+; GFX9-NEXT: v_mov_b32_e32 v3, s1
+; GFX9-NEXT: v_add_co_u32_e32 v1, vcc, s0, v1
+; GFX9-NEXT: v_addc_co_u32_e32 v2, vcc, 0, v3, vcc
+; GFX9-NEXT: flat_load_dword v4, v[1:2]
+; GFX9-NEXT: v_mov_b32_e32 v1, 0
+; GFX9-NEXT: v_add_u32_e32 v2, 1, v0
+; GFX9-NEXT: v_lshrrev_b64 v[0:1], 30, v[1:2]
; GFX9-NEXT: v_add_co_u32_e32 v0, vcc, s0, v0
-; GFX9-NEXT: v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
-; GFX9-NEXT: flat_load_dword v2, v[0:1]
+; GFX9-NEXT: v_addc_co_u32_e32 v1, vcc, v3, v1, vcc
; GFX9-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0)
-; GFX9-NEXT: flat_store_dword v[0:1], v2 offset:4
+; GFX9-NEXT: flat_store_dword v[0:1], v4
; GFX9-NEXT: s_endpgm
bb:
%tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
diff --git a/llvm/test/Transforms/StraightLineStrengthReduce/AMDGPU/pr23975.ll b/llvm/test/Transforms/StraightLineStrengthReduce/AMDGPU/pr23975.ll
index 51b42463d81ba..0d3846d1eac16 100644
--- a/llvm/test/Transforms/StraightLineStrengthReduce/AMDGPU/pr23975.ll
+++ b/llvm/test/Transforms/StraightLineStrengthReduce/AMDGPU/pr23975.ll
@@ -13,7 +13,7 @@ entry:
%tmp = sext i32 undef to i64
%arrayidx114 = getelementptr inbounds %struct.Matrix4x4, ptr addrspace(1) %leafTransformations, i64 %tmp
%tmp1 = getelementptr %struct.Matrix4x4, ptr addrspace(1) %leafTransformations, i64 %tmp, i32 0, i64 0, i64 1
-; CHECK: %tmp1 = getelementptr i8, ptr addrspace(1) %arrayidx114, i64 4
+; CHECK: %tmp1 = getelementptr %struct.Matrix4x4, ptr addrspace(1) %leafTransformations, i64 %tmp, i32 0, i64 0, i64 1
%tmp2 = load <4 x float>, ptr addrspace(1) undef, align 4
ret void
}
diff --git a/llvm/test/Transforms/StraightLineStrengthReduce/AMDGPU/reassociate-geps-and-slsr-addrspace.ll b/llvm/test/Transforms/StraightLineStrengthReduce/AMDGPU/reassociate-geps-and-slsr-addrspace.ll
index 8662e61c95d66..03edfdceab325 100644
--- a/llvm/test/Transforms/StraightLineStrengthReduce/AMDGPU/reassociate-geps-and-slsr-addrspace.ll
+++ b/llvm/test/Transforms/StraightLineStrengthReduce/AMDGPU/reassociate-geps-and-slsr-addrspace.ll
@@ -46,9 +46,9 @@ define amdgpu_kernel void @slsr_after_reassociate_global_geps_over_mubuf_max_off
; CHECK-NEXT: [[P1:%.*]] = getelementptr inbounds float, ptr addrspace(1) [[ARR]], i64 [[TMP]]
; CHECK-NEXT: [[V11:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
; CHECK-NEXT: store i32 [[V11]], ptr addrspace(1) [[OUT]], align 4
-; CHECK-NEXT: [[OFFSET:%.*]] = sext i32 [[I]] to i64
-; CHECK-NEXT: [[TMP5:%.*]] = shl i64 [[OFFSET]], 2
-; CHECK-NEXT: [[P2:%.*]] = getelementptr inbounds i8, ptr addrspace(1) [[P1]], i64 [[TMP5]]
+; CHECK-NEXT: [[J2:%.*]] = add i32 [[J1]], [[I]]
+; CHECK-NEXT: [[TMP5:%.*]] = sext i32 [[J2]] to i64
+; CHECK-NEXT: [[P2:%.*]] = getelementptr inbounds float, ptr addrspace(1) [[ARR]], i64 [[TMP5]]
; CHECK-NEXT: [[V22:%.*]] = load i32, ptr addrspace(1) [[P2]], align 4
; CHECK-NEXT: store i32 [[V22]], ptr addrspace(1) [[OUT]], align 4
; CHECK-NEXT: ret void
@@ -109,8 +109,8 @@ define amdgpu_kernel void @slsr_after_reassociate_lds_geps_over_ds_max_offset(pt
; CHECK-NEXT: [[P1:%.*]] = getelementptr inbounds float, ptr addrspace(3) [[ARR]], i32 [[J1]]
; CHECK-NEXT: [[V11:%.*]] = load i32, ptr addrspace(3) [[P1]], align 4
; CHECK-NEXT: store i32 [[V11]], ptr addrspace(1) [[OUT]], align 4
-; CHECK-NEXT: [[J2:%.*]] = shl i32 [[I]], 2
-; CHECK-NEXT: [[P2:%.*]] = getelementptr inbounds i8, ptr addrspace(3) [[P1]], i32 [[J2]]
+; CHECK-NEXT: [[J2:%.*]] = add i32 [[J1]], [[I]]
+; CHECK-NEXT: [[P2:%.*]] = getelementptr inbounds float, ptr addrspace(3) [[ARR]], i32 [[J2]]
; CHECK-NEXT: [[V22:%.*]] = load i32, ptr addrspace(3) [[P2]], align 4
; CHECK-NEXT: store i32 [[V22]], ptr addrspace(1) [[OUT]], align 4
; CHECK-NEXT: ret void
diff --git a/llvm/test/Transforms/StraightLineStrengthReduce/NVPTX/slsr-i8-gep.ll b/llvm/test/Transforms/StraightLineStrengthReduce/NVPTX/slsr-i8-gep.ll
deleted file mode 100644
index b28ca07084495..0000000000000
--- a/llvm/test/Transforms/StraightLineStrengthReduce/NVPTX/slsr-i8-gep.ll
+++ /dev/null
@@ -1,271 +0,0 @@
-; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
-; RUN: opt < %s -passes=slsr -S | FileCheck %s
-; RUN: llc < %s -march=nvptx64 -mcpu=sm_75 | FileCheck %s --check-prefix=PTX
-
-target triple = "nvptx64-nvidia-cuda"
-
-define void @slsr_i8_zero_delta(ptr %in, ptr %out, i64 %add) {
-; PTX-LABEL: slsr_i8_zero_delta(
-; PTX: {
-; PTX-NEXT: .reg .b16 %rs<6>;
-; PTX-NEXT: .reg .b64 %rd<5>;
-; PTX-EMPTY:
-; PTX-NEXT: // %bb.0:
-; PTX-NEXT: ld.param.b64 %rd1, [slsr_i8_zero_delta_param_0];
-; PTX-NEXT: ld.param.b64 %rd2, [slsr_i8_zero_delta_param_2];
-; PTX-NEXT: add.s64 %rd3, %rd1, %rd2;
-; PTX-NEXT: ld.param.b64 %rd4, [slsr_i8_zero_delta_param_1];
-; PTX-NEXT: ld.b8 %rs1, [%rd3+32];
-; PTX-NEXT: ld.b8 %rs2, [%rd3+64];
-; PTX-NEXT: ld.b8 %rs3, [%rd3+96];
-; PTX-NEXT: add.s16 %rs4, %rs1, %rs2;
-; PTX-NEXT: add.s16 %rs5, %rs4, %rs3;
-; PTX-NEXT: st.b8 [%rd4], %rs5;
-; PTX-NEXT: ret;
-; CHECK-LABEL: define void @slsr_i8_zero_delta(
-; CHECK-SAME: ptr [[IN:%.*]], ptr [[OUT:%.*]], i64 [[ADD:%.*]]) {
-; CHECK-NEXT: [[GETELEM0_0:%.*]] = getelementptr inbounds i8, ptr [[IN]], i64 [[ADD]]
-; CHECK-NEXT: [[GETELEM0_1:%.*]] = getelementptr inbounds i8, ptr [[GETELEM0_0]], i64 32
-; CHECK-NEXT: [[LOAD0:%.*]] = load i8, ptr [[GETELEM0_1]], align 1
-; CHECK-NEXT: [[GETELEM1_1:%.*]] = getelementptr inbounds i8, ptr [[GETELEM0_0]], i64 64
-; CHECK-NEXT: [[LOAD1:%.*]] = load i8, ptr [[GETELEM1_1]], align 1
-; CHECK-NEXT: [[GETELEM2_1:%.*]] = getelementptr inbounds i8, ptr [[GETELEM0_0]], i64 96
-; CHECK-NEXT: [[LOAD2:%.*]] = load i8, ptr [[GETELEM2_1]], align 1
-; CHECK-NEXT: [[OUT0:%.*]] = add i8 [[LOAD0]], [[LOAD1]]
-; CHECK-NEXT: [[OUT1:%.*]] = add i8 [[OUT0]], [[LOAD2]]
-; CHECK-NEXT: store i8 [[OUT1]], ptr [[OUT]], align 1
-; CHECK-NEXT: ret void
-;
- %getElem0.0 = getelementptr inbounds i8, ptr %in, i64 %add
- %getElem0.1 = getelementptr inbounds i8, ptr %getElem0.0, i64 32
- %load0 = load i8, ptr %getElem0.1
-
- %getElem1.0 = getelementptr inbounds i8, ptr %in, i64 %add
- %getElem1.1 = getelementptr inbounds i8, ptr %getElem1.0, i64 64
- %load1 = load i8, ptr %getElem1.1
-
- %getElem2.0 = getelementptr inbounds i8, ptr %in, i64 %add
- %getElem2.1 = getelementptr inbounds i8, ptr %getElem2.0, i64 96
- %load2 = load i8, ptr %getElem2.1
-
- %out0 = add i8 %load0, %load1
- %out1 = add i8 %out0, %load2
- store i8 %out1, ptr %out
-
- ret void
-}
-
-define void @slsr_i8_zero_delta_2(ptr %in, ptr %out, i64 %add) {
-; PTX-LABEL: slsr_i8_zero_delta_2(
-; PTX: {
-; PTX-NEXT: .reg .b16 %rs<6>;
-; PTX-NEXT: .reg .b64 %rd<5>;
-; PTX-EMPTY:
-; PTX-NEXT: // %bb.0:
-; PTX-NEXT: ld.param.b64 %rd1, [slsr_i8_zero_delta_2_param_0];
-; PTX-NEXT: ld.param.b64 %rd2, [slsr_i8_zero_delta_2_param_2];
-; PTX-NEXT: add.s64 %rd3, %rd1, %rd2;
-; PTX-NEXT: ld.param.b64 %rd4, [slsr_i8_zero_delta_2_param_1];
-; PTX-NEXT: ld.b8 %rs1, [%rd3];
-; PTX-NEXT: ld.b8 %rs2, [%rd3+32];
-; PTX-NEXT: ld.b8 %rs3, [%rd3+64];
-; PTX-NEXT: add.s16 %rs4, %rs1, %rs2;
-; PTX-NEXT: add.s16 %rs5, %rs4, %rs3;
-; PTX-NEXT: st.b8 [%rd4], %rs5;
-; PTX-NEXT: ret;
-; CHECK-LABEL: define void @slsr_i8_zero_delta_2(
-; CHECK-SAME: ptr [[IN:%.*]], ptr [[OUT:%.*]], i64 [[ADD:%.*]]) {
-; CHECK-NEXT: [[GETELEM0_0:%.*]] = getelementptr inbounds i8, ptr [[IN]], i64 [[ADD]]
-; CHECK-NEXT: [[LOAD0:%.*]] = load i8, ptr [[GETELEM0_0]], align 1
-; CHECK-NEXT: [[GETELEM1_1:%.*]] = getelementptr inbounds i8, ptr [[GETELEM0_0]], i64 32
-; CHECK-NEXT: [[LOAD1:%.*]] = load i8, ptr [[GETELEM1_1]], align 1
-; CHECK-NEXT: [[GETELEM2_1:%.*]] = getelementptr inbounds i8, ptr [[GETELEM0_0]], i64 64
-; CHECK-NEXT: [[LOAD2:%.*]] = load i8, ptr [[GETELEM2_1]], align 1
-; CHECK-NEXT: [[OUT0:%.*]] = add i8 [[LOAD0]], [[LOAD1]]
-; CHECK-NEXT: [[OUT1:%.*]] = add i8 [[OUT0]], [[LOAD2]]
-; CHECK-NEXT: store i8 [[OUT1]], ptr [[OUT]], align 1
-; CHECK-NEXT: ret void
-;
- %getElem0.0 = getelementptr inbounds i8, ptr %in, i64 %add
- %load0 = load i8, ptr %getElem0.0
-
- %getElem1.0 = getelementptr i8, ptr %in, i64 %add
- %getElem1.1 = getelementptr inbounds i8, ptr %getElem1.0, i64 32
- %load1 = load i8, ptr %getElem1.1
-
- %getElem2.0 = getelementptr i8, ptr %in, i64 %add
- %getElem2.1 = getelementptr inbounds i8, ptr %getElem2.0, i64 64
- %load2 = load i8, ptr %getElem2.1
-
- %out0 = add i8 %load0, %load1
- %out1 = add i8 %out0, %load2
- store i8 %out1, ptr %out
-
- ret void
-}
-
-define void @slsr_i8_base_delta(ptr %in, ptr %out, i64 %add) {
-; PTX-LABEL: slsr_i8_base_delta(
-; PTX: {
-; PTX-NEXT: .reg .b16 %rs<6>;
-; PTX-NEXT: .reg .b64 %rd<5>;
-; PTX-EMPTY:
-; PTX-NEXT: // %bb.0:
-; PTX-NEXT: ld.param.b64 %rd1, [slsr_i8_base_delta_param_0];
-; PTX-NEXT: ld.param.b64 %rd2, [slsr_i8_base_delta_param_2];
-; PTX-NEXT: add.s64 %rd3, %rd1, %rd2;
-; PTX-NEXT: ld.param.b64 %rd4, [slsr_i8_base_delta_param_1];
-; PTX-NEXT: ld.b8 %rs1, [%rd3+32];
-; PTX-NEXT: ld.b8 %rs2, [%rd3+65];
-; PTX-NEXT: ld.b8 %rs3, [%rd3+98];
-; PTX-NEXT: add.s16 %rs4, %rs1, %rs2;
-; PTX-NEXT: add.s16 %rs5, %rs4, %rs3;
-; PTX-NEXT: st.b8 [%rd4], %rs5;
-; PTX-NEXT: ret;
-; CHECK-LABEL: define void @slsr_i8_base_delta(
-; CHECK-SAME: ptr [[IN:%.*]], ptr [[OUT:%.*]], i64 [[ADD:%.*]]) {
-; CHECK-NEXT: [[GETELEM0_0:%.*]] = getelementptr inbounds i8, ptr [[IN]], i64 [[ADD]]
-; CHECK-NEXT: [[GETELEM0_1:%.*]] = getelementptr inbounds i8, ptr [[GETELEM0_0]], i64 32
-; CHECK-NEXT: [[LOAD0:%.*]] = load i8, ptr [[GETELEM0_1]], align 1
-; CHECK-NEXT: [[GETELEM1_1:%.*]] = getelementptr inbounds i8, ptr [[GETELEM0_0]], i64 1
-; CHECK-NEXT: [[GETELEM1_2:%.*]] = getelementptr inbounds i8, ptr [[GETELEM1_1]], i64 64
-; CHECK-NEXT: [[LOAD1:%.*]] = load i8, ptr [[GETELEM1_2]], align 1
-; CHECK-NEXT: [[GETELEM2_1:%.*]] = getelementptr inbounds i8, ptr [[GETELEM0_0]], i64 2
-; CHECK-NEXT: [[GETELEM2_2:%.*]] = getelementptr inbounds i8, ptr [[GETELEM2_1]], i64 96
-; CHECK-NEXT: [[LOAD2:%.*]] = load i8, ptr [[GETELEM2_2]], align 1
-; CHECK-NEXT: [[OUT0:%.*]] = add i8 [[LOAD0]], [[LOAD1]]
-; CHECK-NEXT: [[OUT1:%.*]] = add i8 [[OUT0]], [[LOAD2]]
-; CHECK-NEXT: store i8 [[OUT1]], ptr [[OUT]], align 1
-; CHECK-NEXT: ret void
-;
- %getElem0.0 = getelementptr inbounds i8, ptr %in, i64 %add
- %getElem0.1 = getelementptr inbounds i8, ptr %getElem0.0, i64 32
- %load0 = load i8, ptr %getElem0.1
-
- %getElem1.0 = getelementptr inbounds i8, ptr %in, i64 1
- %getElem1.1 = getelementptr inbounds i8, ptr %getElem1.0, i64 %add
- %getElem1.2 = getelementptr inbounds i8, ptr %getElem1.1, i64 64
- %load1 = load i8, ptr %getElem1.2
-
- %getElem2.0 = getelementptr inbounds i8, ptr %in, i64 2
- %getElem2.1 = getelementptr inbounds i8, ptr %getElem2.0, i64 %add
- %getElem2.2 = getelementptr inbounds i8, ptr %getElem2.1, i64 96
- %load2 = load i8, ptr %getElem2.2
-
- %out0 = add i8 %load0, %load1
- %out1 = add i8 %out0, %load2
- store i8 %out1, ptr %out
-
- ret void
-}
-
-define void @slsr_i8_index_delta(ptr %in, ptr %out, i64 %add) {
-; PTX-LABEL: slsr_i8_index_delta(
-; PTX: {
-; PTX-NEXT: .reg .b16 %rs<6>;
-; PTX-NEXT: .reg .b64 %rd<7>;
-; PTX-EMPTY:
-; PTX-NEXT: // %bb.0:
-; PTX-NEXT: ld.param.b64 %rd1, [slsr_i8_index_delta_param_0];
-; PTX-NEXT: ld.param.b64 %rd2, [slsr_i8_index_delta_param_2];
-; PTX-NEXT: shl.b64 %rd3, %rd2, 3;
-; PTX-NEXT: add.s64 %rd4, %rd1, %rd3;
-; PTX-NEXT: ld.param.b64 %rd5, [slsr_i8_index_delta_param_1];
-; PTX-NEXT: ld.b8 %rs1, [%rd4+32];
-; PTX-NEXT: add.s64 %rd6, %rd1, %rd2;
-; PTX-NEXT: ld.b8 %rs2, [%rd6+64];
-; PTX-NEXT: ld.b8 %rs3, [%rd6+96];
-; PTX-NEXT: add.s16 %rs4, %rs1, %rs2;
-; PTX-NEXT: add.s16 %rs5, %rs4, %rs3;
-; PTX-NEXT: st.b8 [%rd5], %rs5;
-; PTX-NEXT: ret;
-; CHECK-LABEL: define void @slsr_i8_index_delta(
-; CHECK-SAME: ptr [[IN:%.*]], ptr [[OUT:%.*]], i64 [[ADD:%.*]]) {
-; CHECK-NEXT: [[GETELEM0_0:%.*]] = getelementptr inbounds double, ptr [[IN]], i64 [[ADD]]
-; CHECK-NEXT: [[GETELEM0_1:%.*]] = getelementptr inbounds i8, ptr [[GETELEM0_0]], i64 32
-; CHECK-NEXT: [[LOAD0:%.*]] = load i8, ptr [[GETELEM0_1]], align 1
-; CHECK-NEXT: [[GETELEM1_0:%.*]] = getelementptr inbounds i8, ptr [[IN]], i64 [[ADD]]
-; CHECK-NEXT: [[GETELEM1_1:%.*]] = getelementptr inbounds i8, ptr [[GETELEM1_0]], i64 64
-; CHECK-NEXT: [[LOAD1:%.*]] = load i8, ptr [[GETELEM1_1]], align 1
-; CHECK-NEXT: [[GETELEM2_1:%.*]] = getelementptr inbounds i8, ptr [[GETELEM1_0]], i64 96
-; CHECK-NEXT: [[LOAD2:%.*]] = load i8, ptr [[GETELEM2_1]], align 1
-; CHECK-NEXT: [[OUT0:%.*]] = add i8 [[LOAD0]], [[LOAD1]]
-; CHECK-NEXT: [[OUT1:%.*]] = add i8 [[OUT0]], [[LOAD2]]
-; CHECK-NEXT: store i8 [[OUT1]], ptr [[OUT]], align 1
-; CHECK-NEXT: ret void
-;
- %getElem0.0 = getelementptr inbounds double, ptr %in, i64 %add
- %getElem0.1 = getelementptr inbounds i8, ptr %getElem0.0, i64 32
- %load0 = load i8, ptr %getElem0.1
-
- %getElem1.0 = getelementptr inbounds i8, ptr %in, i64 %add
- %getElem1.1 = getelementptr inbounds i8, ptr %getElem1.0, i64 64
- %load1 = load i8, ptr %getElem1.1
-
- %getElem2.0 = getelementptr inbounds i8, ptr %in, i64 %add
- %getElem2.1 = getelementptr inbounds i8, ptr %getElem2.0, i64 96
- %load2 = load i8, ptr %getElem2.1
-
- %out0 = add i8 %load0, %load1
- %out1 = add i8 %out0, %load2
- store i8 %out1, ptr %out
-
- ret void
-}
-
-define void @slsr_i8_stride_delta(ptr %in, ptr %out, i64 %add, i64 %offset) {
-; PTX-LABEL: slsr_i8_stride_delta(
-; PTX: {
-; PTX-NEXT: .reg .b16 %rs<6>;
-; PTX-NEXT: .reg .b64 %rd<7>;
-; PTX-EMPTY:
-; PTX-NEXT: // %bb.0:
-; PTX-NEXT: ld.param.b64 %rd1, [slsr_i8_stride_delta_param_0];
-; PTX-NEXT: ld.param.b64 %rd2, [slsr_i8_stride_delta_param_2];
-; PTX-NEXT: add.s64 %rd3, %rd1, %rd2;
-; PTX-NEXT: ld.param.b64 %rd4, [slsr_i8_stride_delta_param_1];
-; PTX-NEXT: ld.b8 %rs1, [%rd3+32];
-; PTX-NEXT: ld.param.b64 %rd5, [slsr_i8_stride_delta_param_3];
-; PTX-NEXT: ld.b8 %rs2, [%rd3+65];
-; PTX-NEXT: add.s64 %rd6, %rd3, %rd5;
-; PTX-NEXT: ld.b8 %rs3, [%rd6+96];
-; PTX-NEXT: add.s16 %rs4, %rs1, %rs2;
-; PTX-NEXT: add.s16 %rs5, %rs4, %rs3;
-; PTX-NEXT: st.b8 [%rd4], %rs5;
-; PTX-NEXT: ret;
-; CHECK-LABEL: define void @slsr_i8_stride_delta(
-; CHECK-SAME: ptr [[IN:%.*]], ptr [[OUT:%.*]], i64 [[ADD:%.*]], i64 [[OFFSET:%.*]]) {
-; CHECK-NEXT: [[GETELEM0_0:%.*]] = getelementptr inbounds i8, ptr [[IN]], i64 [[ADD]]
-; CHECK-NEXT: [[GETELEM0_1:%.*]] = getelementptr inbounds i8, ptr [[GETELEM0_0]], i64 32
-; CHECK-NEXT: [[LOAD0:%.*]] = load i8, ptr [[GETELEM0_1]], align 1
-; CHECK-NEXT: [[GETELEM1_0:%.*]] = getelementptr inbounds i8, ptr [[GETELEM0_0]], i64 1
-; CHECK-NEXT: [[GETELEM1_1:%.*]] = getelementptr inbounds i8, ptr [[GETELEM1_0]], i64 64
-; CHECK-NEXT: [[LOAD1:%.*]] = load i8, ptr [[GETELEM1_1]], align 1
-; CHECK-NEXT: [[GETELEM2_0:%.*]] = getelementptr inbounds i8, ptr [[GETELEM0_0]], i64 [[OFFSET]]
-; CHECK-NEXT: [[GETELEM2_1:%.*]] = getelementptr inbounds i8, ptr [[GETELEM2_0]], i64 96
-; CHECK-NEXT: [[LOAD2:%.*]] = load i8, ptr [[GETELEM2_1]], align 1
-; CHECK-NEXT: [[OUT0:%.*]] = add i8 [[LOAD0]], [[LOAD1]]
-; CHECK-NEXT: [[OUT1:%.*]] = add i8 [[OUT0]], [[LOAD2]]
-; CHECK-NEXT: store i8 [[OUT1]], ptr [[OUT]], align 1
-; CHECK-NEXT: ret void
-;
- %getElem0.0 = getelementptr inbounds i8, ptr %in, i64 %add
- %getElem0.1 = getelementptr inbounds i8, ptr %getElem0.0, i64 32
- %load0 = load i8, ptr %getElem0.1
-
- %add1 = add i64 %add, 1
- %getElem1.0 = getelementptr inbounds i8, ptr %in, i64 %add1
- %getElem1.1 = getelementptr inbounds i8, ptr %getElem1.0, i64 64
- %load1 = load i8, ptr %getElem1.1
-
- %add2 = add i64 %add, %offset
- %getElem2.0 = getelementptr inbounds i8, ptr %in, i64 %add2
- %getElem2.1 = getelementptr inbounds i8, ptr %getElem2.0, i64 96
- %load2 = load i8, ptr %getElem2.1
-
- %out0 = add i8 %load0, %load1
- %out1 = add i8 %out0, %load2
- store i8 %out1, ptr %out
-
- ret void
-}
diff --git a/llvm/test/Transforms/StraightLineStrengthReduce/NVPTX/slsr-invalid.ll b/llvm/test/Transforms/StraightLineStrengthReduce/NVPTX/slsr-invalid.ll
deleted file mode 100644
index 6377701a4d289..0000000000000
--- a/llvm/test/Transforms/StraightLineStrengthReduce/NVPTX/slsr-invalid.ll
+++ /dev/null
@@ -1,140 +0,0 @@
-; RUN: opt < %s -passes=slsr -S | FileCheck %s
-
-target triple = "nvptx64-nvidia-cuda"
-
-;;; This test encodes a regression where SCEV based reuse of an instruction
-;;; derived from `or disjoint` is unsound. In the source function there is an
-;;; arithmetic expression `row*4 + column` that is materialized twice. The
-;;; first materialization uses `or disjoint` which can become poison if the
-;;; disjointness promise is violated. The second materialization recomputes the
-;;; value with ordinary `shl` and `add` so it is not poisoned on that path.
-;;;
-;;; The buggy optimization tries to "reuse" the first materialization on a
-;;; different path. In the pointer version it reuses a GEP that is based on
-;;; the `or disjoint` value. In the integer version it reuses an `add i64`
-;;; based on the same value. For some inputs the original program never uses
-;;; the poisoned `or disjoint` result on that path so the behavior is defined.
-;;; After the transformation the reused instruction is consumed by a load or
-;;; call argument which observes poison and turns the program into UB.
-;;;
-;;; These tests pin the behavior of ScalarEvolution::canReuseInstruction. It
-;;; must reject reuse of the `or disjoint` based instruction in favor of the
-;;; recomputed expression, since the candidate IR is strictly more poison
-;;; generating than the SCEV expression we want to realize.
-
-define void @invalid_gep_reuse(ptr readonly align 16 captures(none) dereferenceable(12) %0, ptr writeonly align 256 captures(none) dereferenceable(15) %1, i32 %row, i32 %column) {
-; CHECK-LABEL: define void @invalid_gep_reuse(
-; CHECK-SAME: ptr readonly align 16 captures(none) dereferenceable(12) [[TMP0:%.*]], ptr writeonly align 256 captures(none) dereferenceable(15) [[TMP1:%.*]], i32 [[ROW:%.*]], i32 [[COLUMN:%.*]]) {
-; CHECK-NEXT: [[TMP3:%.*]] = icmp samesign ult i32 [[COLUMN]], 4
-; CHECK-NEXT: [[TMP4:%.*]] = shl nuw nsw i32 [[ROW]], 2
-; CHECK-NEXT: [[TMP5:%.*]] = or disjoint i32 [[TMP4]], [[COLUMN]]
-; CHECK-NEXT: [[TMP6:%.*]] = zext nneg i32 [[TMP5]] to i64
-; CHECK-NEXT: [[TMP7:%.*]] = getelementptr inbounds i8, ptr [[TMP0]], i64 [[TMP6]]
-; CHECK-NEXT: br i1 [[TMP3]], label %[[BB8:.*]], label %[[BB10:.*]]
-; CHECK: [[BB8]]:
-; CHECK-NEXT: [[TMP9:%.*]] = load i8, ptr [[TMP7]], align 1
-; CHECK-NEXT: br label %[[BB10]]
-; CHECK: [[BB10]]:
-; CHECK-NEXT: [[TMP11:%.*]] = phi i8 [ [[TMP9]], %[[BB8]] ], [ 1, [[TMP2:%.*]] ]
-; CHECK-NEXT: [[DOTNOT:%.*]] = icmp eq i32 [[COLUMN]], 0
-; CHECK-NEXT: [[TMP16:%.*]] = shl nuw nsw i32 [[ROW]], 2
-; CHECK-NEXT: [[TMP13:%.*]] = add nuw nsw i32 [[TMP16]], [[COLUMN]]
-; CHECK-NEXT: [[TMP17:%.*]] = zext nneg i32 [[TMP13]] to i64
-; CHECK-NEXT: [[TMP15:%.*]] = getelementptr i8, ptr [[TMP0]], i64 [[TMP17]]
-; CHECK-NEXT: [[TMP12:%.*]] = getelementptr i8, ptr [[TMP15]], i64 -1
-; CHECK-NEXT: br i1 [[DOTNOT]], label %[[BB19:.*]], label %[[BB17:.*]]
-; CHECK: [[BB17]]:
-; CHECK-NEXT: [[TMP14:%.*]] = load i8, ptr [[TMP12]], align 1
-; CHECK-NEXT: br label %[[BB19]]
-; CHECK: [[BB19]]:
-; CHECK-NEXT: ret void
-;
- %3 = icmp samesign ult i32 %column, 4
- %4 = shl nuw nsw i32 %row, 2
- %5 = or disjoint i32 %4, %column
- %6 = zext nneg i32 %5 to i64
- %7 = getelementptr inbounds i8, ptr %0, i64 %6
- br i1 %3, label %8, label %10
-
-8: ; preds = %2
- %9 = load i8, ptr %7, align 1
- br label %10
-
-10: ; preds = %8, %2
- %11 = phi i8 [ %9, %8 ], [ 1, %2 ]
- %.not = icmp eq i32 %column, 0
- %12 = shl nuw nsw i32 %row, 2
- %13 = add nuw nsw i32 %12, %column
- %14 = zext nneg i32 %13 to i64
- %15 = getelementptr i8, ptr %0, i64 %14
- %16 = getelementptr i8, ptr %15, i64 -1
- br i1 %.not, label %19, label %17
-
-17: ; preds = %10
- %18 = load i8, ptr %16, align 1
- br label %19
-
-19: ; preds = %17, %10
- ret void
-}
-
-define void @invalid_add_reuse(i64 %0, ptr writeonly align 256 captures(none) dereferenceable(15) %1, i32 %row, i32 %column) {
-; CHECK-LABEL: define void @invalid_add_reuse(
-; CHECK-SAME: i64 [[TMP0:%.*]], ptr writeonly align 256 captures(none) dereferenceable(15) [[TMP1:%.*]], i32 [[ROW:%.*]], i32 [[COLUMN:%.*]]) {
-; CHECK-NEXT: [[TMP3:%.*]] = icmp samesign ult i32 [[COLUMN]], 4
-; CHECK-NEXT: [[TMP4:%.*]] = shl nuw nsw i32 [[ROW]], 2
-; CHECK-NEXT: [[TMP5:%.*]] = or disjoint i32 [[TMP4]], [[COLUMN]]
-; CHECK-NEXT: [[TMP6:%.*]] = zext nneg i32 [[TMP5]] to i64
-; CHECK-NEXT: [[TMP7:%.*]] = add i64 [[TMP0]], [[TMP6]]
-; CHECK-NEXT: br i1 [[TMP3]], label %[[BB8:.*]], label %[[BB10:.*]]
-; CHECK: [[BB8]]:
-; CHECK-NEXT: [[TMP9:%.*]] = call i64 @foo(i64 [[TMP7]])
-; CHECK-NEXT: br label %[[BB10]]
-; CHECK: [[BB10]]:
-; CHECK-NEXT: [[TMP11:%.*]] = phi i64 [ [[TMP9]], %[[BB8]] ], [ 1, [[TMP2:%.*]] ]
-; CHECK-NEXT: [[DOTNOT:%.*]] = icmp eq i32 [[COLUMN]], 0
-; CHECK-NEXT: [[TMP12:%.*]] = shl nuw nsw i32 [[ROW]], 2
-; CHECK-NEXT: [[TMP17:%.*]] = add nuw nsw i32 [[TMP12]], [[COLUMN]]
-; CHECK-NEXT: [[TMP18:%.*]] = zext nneg i32 [[TMP17]] to i64
-; CHECK-NEXT: [[TMP15:%.*]] = mul i64 [[TMP0]], 4
-; CHECK-NEXT: [[TMP13:%.*]] = add i64 [[TMP15]], [[TMP18]]
-; CHECK-NEXT: [[TMP14:%.*]] = add i64 [[TMP13]], -1
-; CHECK-NEXT: br i1 [[DOTNOT]], label %[[BB20:.*]], label %[[BB18:.*]]
-; CHECK: [[BB18]]:
-; CHECK-NEXT: [[TMP16:%.*]] = call i64 @bar(i64 [[TMP14]])
-; CHECK-NEXT: br label %[[BB20]]
-; CHECK: [[BB20]]:
-; CHECK-NEXT: ret void
-;
- %3 = icmp samesign ult i32 %column, 4
- %4 = shl nuw nsw i32 %row, 2
- %5 = or disjoint i32 %4, %column
- %6 = zext nneg i32 %5 to i64
- %7 = add i64 %0, %6
- br i1 %3, label %8, label %10
-
-8: ; preds = %2
- %9 = call i64 @foo(i64 %7)
- br label %10
-
-10: ; preds = %8, %2
- %11 = phi i64 [ %9, %8 ], [ 1, %2 ]
- %.not = icmp eq i32 %column, 0
- %12 = shl nuw nsw i32 %row, 2
- %13 = add nuw nsw i32 %12, %column
- %14 = zext nneg i32 %13 to i64
- %15 = mul i64 %0, 4
- %16 = add i64 %15, %14
- %17 = add i64 %16, -1
- br i1 %.not, label %20, label %18
-
-18: ; preds = %10
- %19 = call i64 @bar(i64 %17)
- br label %20
-
-20: ; preds = %18, %10
- ret void
-}
-
-declare i64 @foo(i64)
-declare i64 @bar(i64)
diff --git a/llvm/test/Transforms/StraightLineStrengthReduce/NVPTX/slsr-var-delta.ll b/llvm/test/Transforms/StraightLineStrengthReduce/NVPTX/slsr-var-delta.ll
deleted file mode 100644
index ce6f720cc4352..0000000000000
--- a/llvm/test/Transforms/StraightLineStrengthReduce/NVPTX/slsr-var-delta.ll
+++ /dev/null
@@ -1,70 +0,0 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
-; RUN: opt < %s -passes=slsr -S | FileCheck %s
-; RUN: llc < %s -march=nvptx64 -mcpu=sm_75 | FileCheck %s --check-prefix=PTX
-
-target triple = "nvptx64-nvidia-cuda"
-
-; Test SLSR can reuse the computation by complex variable delta.
-; The original program needs 4 mul.wide.s32, after SLSR with
-; variable-delta, it can reduce to 1 mul.wide.s32.
-define void @foo(ptr %a, ptr %b, i32 %j) {
-; PTX-LABEL: foo(
-; PTX: {
-; PTX-NEXT: .reg .b32 %r<4>;
-; PTX-NEXT: .reg .b64 %rd<9>;
-; PTX-EMPTY:
-; PTX-NEXT: // %bb.0:
-; PTX-NEXT: ld.param.b64 %rd1, [foo_param_0];
-; PTX-NEXT: ld.b32 %r1, [%rd1];
-; PTX-NEXT: ld.param.b64 %rd2, [foo_param_1];
-; PTX-NEXT: ld.param.b32 %r2, [foo_param_2];
-; PTX-NEXT: add.s32 %r3, %r1, %r2;
-; PTX-NEXT: mul.wide.s32 %rd3, %r3, 4;
-; PTX-NEXT: add.s64 %rd4, %rd2, %rd3;
-; PTX-NEXT: st.b32 [%rd4], 0;
-; PTX-NEXT: add.s64 %rd5, %rd4, %rd3;
-; PTX-NEXT: st.b32 [%rd5], 1;
-; PTX-NEXT: add.s64 %rd6, %rd5, 4;
-; PTX-NEXT: st.b32 [%rd5+4], 2;
-; PTX-NEXT: add.s64 %rd7, %rd6, %rd3;
-; PTX-NEXT: st.b32 [%rd7], 3;
-; PTX-NEXT: add.s64 %rd8, %rd7, %rd3;
-; PTX-NEXT: st.b32 [%rd8], 4;
-; PTX-NEXT: ret;
- %i.0 = load i32, ptr %a, align 8
- %i = add i32 %i.0, %j
- ; CHECK: [[L:%.*]] = load i32, ptr %a, align 8
- ; CHECK: [[I:%.*]] = add i32 [[L]], %j
- %gep.24 = getelementptr float, ptr %b, i32 %i
- ; CHECK: [[GEP0:%.*]] = getelementptr float, ptr %b, i32 [[I]]
- ; CHECK: store i32 0, ptr [[GEP0]]
- store i32 0, ptr %gep.24
- %gep.24.sum1 = add i32 %i, %i
- %gep.25 = getelementptr float, ptr %b, i32 %gep.24.sum1
- ; CHECK: [[EXT1:%.*]] = sext i32 [[I]] to i64
- ; CHECK: [[MUL1:%.*]] = shl i64 [[EXT1]], 2
- ; CHECK: [[GEP1:%.*]] = getelementptr i8, ptr [[GEP0]], i64 [[MUL1]]
- ; CHECK: store i32 1, ptr [[GEP1]]
- store i32 1, ptr %gep.25
- %gep.26.sum3 = add i32 1, %i
- %gep.27.sum = add i32 %gep.26.sum3, %i
- %gep.28 = getelementptr float, ptr %b, i32 %gep.27.sum
- ; CHECK: [[GEP2:%.*]] = getelementptr i8, ptr [[GEP1]], i64 4
- ; CHECK: store i32 2, ptr [[GEP2]]
- store i32 2, ptr %gep.28
- %gep.28.sum = add i32 %gep.27.sum, %i
- %gep.29 = getelementptr float, ptr %b, i32 %gep.28.sum
- ; CHECK: [[EXT2:%.*]] = sext i32 [[I]] to i64
- ; CHECK: [[MUL2:%.*]] = shl i64 [[EXT2]], 2
- ; CHECK: [[GEP3:%.*]] = getelementptr i8, ptr [[GEP2]], i64 [[MUL2]]
- ; CHECK: store i32 3, ptr [[GEP3]]
- store i32 3, ptr %gep.29
- %gep.29.sum = add i32 %gep.28.sum, %i
- %gep.30 = getelementptr float, ptr %b, i32 %gep.29.sum
- ; CHECK: [[EXT3:%.*]] = sext i32 [[I]] to i64
- ; CHECK: [[MUL3:%.*]] = shl i64 [[EXT3]], 2
- ; CHECK: [[GEP4:%.*]] = getelementptr i8, ptr [[GEP3]], i64 [[MUL3]]
- ; CHECK: store i32 4, ptr [[GEP4]]
- store i32 4, ptr %gep.30
- ret void
-}
diff --git a/llvm/test/Transforms/StraightLineStrengthReduce/path-compression.ll b/llvm/test/Transforms/StraightLineStrengthReduce/path-compression.ll
deleted file mode 100644
index 92676d88ae772..0000000000000
--- a/llvm/test/Transforms/StraightLineStrengthReduce/path-compression.ll
+++ /dev/null
@@ -1,35 +0,0 @@
-; RUN: opt < %s -passes="slsr" -S | FileCheck %s
-
-target datalayout = "e-i64:64-v16:16-v32:32-n16:32:64"
-
-%struct.B = type { i16 }
-%struct.A = type { %struct.B, %struct.B, %struct.B }
-
-define void @path_compression(i32 %a, ptr %base, i16 %r, i1 %cond) {
-; CHECK-LABEL: @path_compression(
-; CHECK: [[I:%.*]] = sext i32 %a to i64
-; CHECK: [[GEP1:%.*]] = getelementptr %struct.A, ptr %base, i64 [[I]]
-; CHECK: br
-; CHECK-LABEL: next
-; compress the path to use GEP1 as the Basis instead of GEP2
-; CHECK: [[GEP2:%.*]] = getelementptr inbounds i8, ptr [[GEP1]], i64 2
-; CHECK: [[GEP3:%.*]] = getelementptr inbounds i8, ptr [[GEP1]], i64 4
-
-
- %1 = sext i32 %a to i64
- %2 = add i64 %1, 1
- %getElem1 = getelementptr inbounds %struct.A, ptr %base, i64 %1
- br i1 %cond, label %next, label %ret
-
-next:
- %getElem2 = getelementptr inbounds %struct.A, ptr %base, i64 %1, i32 1
- %offset = sub i64 %2, 1
- %getElem3 = getelementptr inbounds %struct.A, ptr %base, i64 %offset, i32 2
- store i16 %r, ptr %getElem1, align 2
- store i16 %r, ptr %getElem2, align 2
- store i16 %r, ptr %getElem3, align 2
- br label %ret
-
-ret:
- ret void
-}
diff --git a/llvm/test/Transforms/StraightLineStrengthReduce/pick-candidate.ll b/llvm/test/Transforms/StraightLineStrengthReduce/pick-candidate.ll
deleted file mode 100644
index 7fa0e4f9474f1..0000000000000
--- a/llvm/test/Transforms/StraightLineStrengthReduce/pick-candidate.ll
+++ /dev/null
@@ -1,32 +0,0 @@
-; RUN: opt < %s -passes="slsr" -S | FileCheck %s
-
-target datalayout = "e-i64:64-v16:16-v32:32-n16:32:64"
-
-%struct.B = type { i16 }
-%struct.A = type { %struct.B, %struct.B }
-
-define i32 @pick(i32 %0, ptr %addr) {
-; `d` can be optimized by 2 approaches
-; 1. a = 1 + 1 * %0
-; d = 1 + 8 * %0
-; = a + 7 * %0
-; 2. c = (8 * %0) + 3
-; d = (8 * %0) + 1
-; = c - 2
-; Pick candidate (2) as it can save 1 instruction from (7 * %0)
-;
-; CHECK-LABEL: pick
-; CHECK: [[A:%.*]] = add i32 %0, 1
-; CHECK: [[B:%.*]] = shl i32 %0, 3
-; CHECK: [[C:%.*]] = add i32 [[B]], 3
-; CHECK: store i32 [[C]], ptr %addr
-; CHECK: [[D:%.*]] = add i32 [[C]], -2
-; CHECK: ret i32 %d
-
- %a = add i32 %0, 1
- %b = shl i32 %0, 3
- %c = add i32 %b, 3
- store i32 %c, ptr %addr
- %d = add i32 %b, 1
- ret i32 %d
-}
diff --git a/llvm/test/Transforms/StraightLineStrengthReduce/slsr-add.ll b/llvm/test/Transforms/StraightLineStrengthReduce/slsr-add.ll
index 1abf2fa331821..d85331f77b12f 100644
--- a/llvm/test/Transforms/StraightLineStrengthReduce/slsr-add.ll
+++ b/llvm/test/Transforms/StraightLineStrengthReduce/slsr-add.ll
@@ -4,8 +4,6 @@
target datalayout = "e-i64:64-v16:16-v32:32-n16:32:64"
-; Index Delta
-
define void @shl(i32 %b, i32 %s) {
; CHECK-LABEL: @shl(
; CHECK-NEXT: [[T1:%.*]] = add i32 [[B:%.*]], [[S:%.*]]
@@ -173,121 +171,3 @@ define void @slsr_strided_add_128bit(i128 %b, i128 %s) {
declare void @foo(i32)
declare void @voo(<2 x i32>)
declare void @bar(i128)
-
-; Stride Delta
-
-define void @stride_const(i32 %a, ptr %base, i16 %r) {
-; Reuse add1 to compute add2
-; CHECK-LABEL: @stride_const(
-; CHECK-NEXT: [[I1:%.*]] = sext i32 [[A:%.*]] to i64
-; CHECK-NEXT: [[I2:%.*]] = mul i64 [[I1]], 2
-; CHECK-NEXT: [[BI:%.*]] = ptrtoint ptr [[BASE:%.*]] to i64
-; CHECK-NEXT: [[ADD1:%.*]] = add i64 [[BI]], [[I2]]
-; CHECK-NEXT: [[ADD2:%.*]] = add i64 [[ADD1]], 8
-; CHECK-NEXT: [[ADDR1:%.*]] = inttoptr i64 [[ADD1]] to ptr
-; CHECK-NEXT: [[ADDR2:%.*]] = inttoptr i64 [[ADD2]] to ptr
-; CHECK-NEXT: store i16 [[R:%.*]], ptr [[ADDR1]], align 2
-; CHECK-NEXT: store i16 [[R]], ptr [[ADDR2]], align 2
-; CHECK-NEXT: ret void
-;
- %1 = sext i32 %a to i64
- %2 = mul i64 %1, 2
- %3 = add i64 %1, 4
- %4 = mul i64 %3, 2
- %baseInt = ptrtoint ptr %base to i64
- %add1 = add i64 %baseInt, %2
- %add2 = add i64 %baseInt, %4
- %addr1 = inttoptr i64 %add1 to ptr
- %addr2 = inttoptr i64 %add2 to ptr
- store i16 %r, ptr %addr1, align 2
- store i16 %r, ptr %addr2, align 2
- ret void
-}
-
-define void @stride_var(i32 %a, ptr %base, i16 %r, i64 %n) {
-; Reuse add1 to compute add2 to save a add.s64
-; CHECK-LABEL: @stride_var(
-; CHECK-NEXT: [[I1:%.*]] = sext i32 [[A:%.*]] to i64
-; CHECK-NEXT: [[I2:%.*]] = mul i64 [[I1]], 2
-; CHECK-NEXT: [[BI:%.*]] = ptrtoint ptr [[BASE:%.*]] to i64
-; CHECK-NEXT: [[ADD1:%.*]] = add i64 [[BI]], [[I2]]
-; CHECK-NEXT: [[TMP3:%.*]] = shl i64 [[N:%.*]], 1
-; CHECK-NEXT: [[ADD2:%.*]] = add i64 [[ADD1]], [[TMP3]]
-; CHECK-NEXT: [[ADDR1:%.*]] = inttoptr i64 [[ADD1]] to ptr
-; CHECK-NEXT: [[ADDR2:%.*]] = inttoptr i64 [[ADD2]] to ptr
-; CHECK-NEXT: store i16 [[R:%.*]], ptr [[ADDR1]], align 2
-; CHECK-NEXT: store i16 [[R]], ptr [[ADDR2]], align 2
-; CHECK-NEXT: ret void
-;
-
- %1 = sext i32 %a to i64
- %2 = mul i64 %1, 2
- %3 = add i64 %1, %n
- %4 = mul i64 %3, 2
- %baseInt = ptrtoint ptr %base to i64
- %add1 = add i64 %baseInt, %2
- %add2 = add i64 %baseInt, %4
- %addr1 = inttoptr i64 %add1 to ptr
- %addr2 = inttoptr i64 %add2 to ptr
- store i16 %r, ptr %addr1, align 2
- store i16 %r, ptr %addr2, align 2
- ret void
-}
-
-; Base Delta
-
-define void @base_const(i32 %a, ptr %base, i16 %r) {
-; Reuse add1 to compute add2
-; CHECK-LABEL: @base_const(
-; CHECK-NEXT: [[I1:%.*]] = sext i32 [[A:%.*]] to i64
-; CHECK-NEXT: [[I2:%.*]] = mul i64 [[I1]], 2
-; CHECK-NEXT: [[BI:%.*]] = ptrtoint ptr [[BASE:%.*]] to i64
-; CHECK-NEXT: [[ADD1:%.*]] = add i64 [[BI]], [[I2]]
-; CHECK-NEXT: [[ADD2:%.*]] = add i64 [[ADD1]], 5
-; CHECK-NEXT: [[ADDR1:%.*]] = inttoptr i64 [[ADD1]] to ptr
-; CHECK-NEXT: [[ADDR2:%.*]] = inttoptr i64 [[ADD2]] to ptr
-; CHECK-NEXT: store i16 [[R:%.*]], ptr [[ADDR1]], align 2
-; CHECK-NEXT: store i16 [[R]], ptr [[ADDR2]], align 2
-; CHECK-NEXT: ret void
-;
-
- %1 = sext i32 %a to i64
- %2 = mul i64 %1, 2
- %baseInt = ptrtoint ptr %base to i64
- %add1 = add i64 %baseInt, %2
- %add2.0 = add i64 %baseInt, 5
- %add2 = add i64 %add2.0, %2
- %addr1 = inttoptr i64 %add1 to ptr
- %addr2 = inttoptr i64 %add2 to ptr
- store i16 %r, ptr %addr1, align 2
- store i16 %r, ptr %addr2, align 2
- ret void
-}
-
-define void @base_var(i32 %a, ptr %base, i16 %r, i64 %n) {
-; Reuse add1 to compute add2
-; CHECK-LABEL: @base_var(
-; CHECK-NEXT: [[I1:%.*]] = sext i32 [[A:%.*]] to i64
-; CHECK-NEXT: [[I2:%.*]] = mul i64 [[I1]], 2
-; CHECK-NEXT: [[BI:%.*]] = ptrtoint ptr [[BASE:%.*]] to i64
-; CHECK-NEXT: [[ADD1:%.*]] = add i64 [[BI]], [[I2]]
-; CHECK-NEXT: [[ADD2:%.*]] = add i64 [[ADD1]], [[N:%.*]]
-; CHECK-NEXT: [[ADDR1:%.*]] = inttoptr i64 [[ADD1]] to ptr
-; CHECK-NEXT: [[ADDR2:%.*]] = inttoptr i64 [[ADD2]] to ptr
-; CHECK-NEXT: store i16 [[R:%.*]], ptr [[ADDR1]], align 2
-; CHECK-NEXT: store i16 [[R]], ptr [[ADDR2]], align 2
-; CHECK-NEXT: ret void
-;
-
- %1 = sext i32 %a to i64
- %2 = mul i64 %1, 2
- %baseInt = ptrtoint ptr %base to i64
- %add1 = add i64 %baseInt, %2
- %add2.0 = add i64 %baseInt, %n
- %add2 = add i64 %add2.0, %2
- %addr1 = inttoptr i64 %add1 to ptr
- %addr2 = inttoptr i64 %add2 to ptr
- store i16 %r, ptr %addr1, align 2
- store i16 %r, ptr %addr2, align 2
- ret void
-}
diff --git a/llvm/test/Transforms/StraightLineStrengthReduce/slsr-gep.ll b/llvm/test/Transforms/StraightLineStrengthReduce/slsr-gep.ll
index 8026b304c5957..7cd45329c24fe 100644
--- a/llvm/test/Transforms/StraightLineStrengthReduce/slsr-gep.ll
+++ b/llvm/test/Transforms/StraightLineStrengthReduce/slsr-gep.ll
@@ -3,43 +3,6 @@
target datalayout = "e-i64:64-v16:16-v32:32-n16:32:64-p:64:64:64-p1:32:32:32-p2:128:128:128:32"
-; Index Delta
-; Most of the original test cases in this file were optimized by Index-delta.
-; After adding Base-delta and Stride-delta, most of the GEP test cases
-; are optimized by Stride-delta now. The only case that GEP needs index-delta
-; SLSR is to reuse address computation from a GEP with different pointee type.
-; Once LLVM completely moves from typed GEP to PtrAdd, we can remove
-; index-delta for GEP/PtrAdd.
-
-define void @index_delta(ptr %input, i32 %c, i32 %b, i32 %n, float %r) {
-; CHECK-LABEL: define void @index_delta(
-; CHECK-SAME: ptr [[INPUT:%.*]], i32 [[C:%.*]], i32 [[B:%.*]], i32 [[N:%.*]], float [[R:%.*]]) {
-; CHECK-NEXT: [[ADD0:%.*]] = add nsw i32 [[B]], 1
-; CHECK-NEXT: [[MUL_1:%.*]] = mul nsw i32 [[ADD0]], [[N]]
-; CHECK-NEXT: [[ADD1:%.*]] = add i32 [[MUL_1]], [[C]]
-; CHECK-NEXT: [[OFFSET:%.*]] = sext i32 [[ADD1]] to i64
-; CHECK-NEXT: [[GETELEM:%.*]] = getelementptr i8, ptr [[INPUT]], i64 [[OFFSET]]
-; CHECK-NEXT: store float [[R]], ptr [[GETELEM]], align 4
-; CHECK-NEXT: [[TMP:%.*]] = mul i64 [[OFFSET]], 3
-; CHECK-NEXT: [[GETELEM_1:%.*]] = getelementptr inbounds i8, ptr [[GETELEM]], i64 [[TMP]]
-; CHECK-NEXT: store float [[R]], ptr [[GETELEM_1]], align 4
-; CHECK-NEXT: ret void
-;
-
- %add0 = add nsw i32 %b, 1
- %mul.1 = mul nsw i32 %add0, %n
- %add.1 = add i32 %mul.1, %c
- %offset = sext i32 %add.1 to i64
- %getElem = getelementptr i8, ptr %input, i64 %offset
- store float %r, ptr %getElem, align 4
- %getElem.1 = getelementptr inbounds float, ptr %input, i64 %offset
- store float %r, ptr %getElem.1, align 4
-
- ret void
-}
-
-; Stride Delta
-
; foo(input[0]);
; foo(input[s]);
; foo(input[s * 2]);
@@ -54,7 +17,7 @@ define void @slsr_gep(ptr %input, i64 %s) {
; CHECK-LABEL: define void @slsr_gep(
; CHECK-SAME: ptr [[INPUT:%.*]], i64 [[S:%.*]]) {
; CHECK-NEXT: call void @foo(ptr [[INPUT]])
-; CHECK-NEXT: [[P1:%.*]] = getelementptr i32, ptr [[INPUT]], i64 [[S]]
+; CHECK-NEXT: [[P1:%.*]] = getelementptr inbounds i32, ptr [[INPUT]], i64 [[S]]
; CHECK-NEXT: call void @foo(ptr [[P1]])
; CHECK-NEXT: [[TMP1:%.*]] = shl i64 [[S]], 2
; CHECK-NEXT: [[P2:%.*]] = getelementptr inbounds i8, ptr [[P1]], i64 [[TMP1]]
@@ -91,7 +54,7 @@ define void @slsr_gep_sext(ptr %input, i32 %s) {
; CHECK-SAME: ptr [[INPUT:%.*]], i32 [[S:%.*]]) {
; CHECK-NEXT: call void @foo(ptr [[INPUT]])
; CHECK-NEXT: [[T:%.*]] = sext i32 [[S]] to i64
-; CHECK-NEXT: [[P1:%.*]] = getelementptr i32, ptr [[INPUT]], i64 [[T]]
+; CHECK-NEXT: [[P1:%.*]] = getelementptr inbounds i32, ptr [[INPUT]], i64 [[T]]
; CHECK-NEXT: call void @foo(ptr [[P1]])
; CHECK-NEXT: [[TMP1:%.*]] = shl i64 [[T]], 2
; CHECK-NEXT: [[P2:%.*]] = getelementptr inbounds i8, ptr [[P1]], i64 [[TMP1]]
@@ -129,10 +92,10 @@ define void @slsr_gep_sext(ptr %input, i32 %s) {
define void @slsr_gep_2d(ptr %input, i64 %s, i64 %t) {
; CHECK-LABEL: define void @slsr_gep_2d(
; CHECK-SAME: ptr [[INPUT:%.*]], i64 [[S:%.*]], i64 [[T:%.*]]) {
-; CHECK-NEXT: [[P0:%.*]] = getelementptr [10 x [5 x i32]], ptr [[INPUT]], i64 0, i64 [[S]], i64 [[T]]
+; CHECK-NEXT: [[P0:%.*]] = getelementptr inbounds [10 x [5 x i32]], ptr [[INPUT]], i64 0, i64 [[S]], i64 [[T]]
; CHECK-NEXT: call void @foo(ptr [[P0]])
; CHECK-NEXT: [[TMP1:%.*]] = mul i64 [[S]], 20
-; CHECK-NEXT: [[P1:%.*]] = getelementptr i8, ptr [[P0]], i64 [[TMP1]]
+; CHECK-NEXT: [[P1:%.*]] = getelementptr inbounds i8, ptr [[P0]], i64 [[TMP1]]
; CHECK-NEXT: call void @foo(ptr [[P1]])
; CHECK-NEXT: [[P2:%.*]] = getelementptr inbounds i8, ptr [[P1]], i64 [[TMP1]]
; CHECK-NEXT: call void @foo(ptr [[P2]])
@@ -166,10 +129,10 @@ define void @slsr_gep_2d(ptr %input, i64 %s, i64 %t) {
define void @slsr_gep_uglygep(ptr %input, i64 %s, i64 %t) {
; CHECK-LABEL: define void @slsr_gep_uglygep(
; CHECK-SAME: ptr [[INPUT:%.*]], i64 [[S:%.*]], i64 [[T:%.*]]) {
-; CHECK-NEXT: [[P0:%.*]] = getelementptr [10 x [5 x [[STRUCT_S:%.*]]]], ptr [[INPUT]], i64 0, i64 [[S]], i64 [[T]], i32 0
+; CHECK-NEXT: [[P0:%.*]] = getelementptr inbounds [10 x [5 x %struct.S]], ptr [[INPUT]], i64 0, i64 [[S]], i64 [[T]], i32 0
; CHECK-NEXT: call void @bar(ptr [[P0]])
; CHECK-NEXT: [[TMP1:%.*]] = mul i64 [[S]], 60
-; CHECK-NEXT: [[P1:%.*]] = getelementptr i8, ptr [[P0]], i64 [[TMP1]]
+; CHECK-NEXT: [[P1:%.*]] = getelementptr inbounds i8, ptr [[P0]], i64 [[TMP1]]
; CHECK-NEXT: call void @bar(ptr [[P1]])
; CHECK-NEXT: [[P2:%.*]] = getelementptr inbounds i8, ptr [[P1]], i64 [[TMP1]]
; CHECK-NEXT: call void @bar(ptr [[P2]])
@@ -276,7 +239,7 @@ define void @slsr_gep_fat_pointer(ptr addrspace(2) %input, i32 %s) {
; p1 = &input[s]
; CHECK-LABEL: define void @slsr_gep_fat_pointer(
; CHECK-SAME: ptr addrspace(2) [[INPUT:%.*]], i32 [[S:%.*]]) {
-; CHECK-NEXT: [[P1:%.*]] = getelementptr i32, ptr addrspace(2) [[INPUT]], i32 [[S]]
+; CHECK-NEXT: [[P1:%.*]] = getelementptr inbounds i32, ptr addrspace(2) [[INPUT]], i32 [[S]]
; CHECK-NEXT: call void @baz2(ptr addrspace(2) [[P1]])
; CHECK-NEXT: [[TMP1:%.*]] = shl i32 [[S]], 2
; CHECK-NEXT: [[P2:%.*]] = getelementptr inbounds i8, ptr addrspace(2) [[P1]], i32 [[TMP1]]
@@ -300,115 +263,3 @@ declare void @foo(ptr)
declare void @bar(ptr)
declare void @baz(ptr addrspace(1))
declare void @baz2(ptr addrspace(2))
-
-define void @stride_const(ptr %input, i32 %c, i32 %b, i32 %n, float %r) {
-; CHECK-LABEL: define void @stride_const(
-; CHECK-SAME: ptr [[INPUT:%.*]], i32 [[C:%.*]], i32 [[B:%.*]], i32 [[N:%.*]], float [[R:%.*]]) {
-; CHECK-NEXT: [[MUL:%.*]] = mul nsw i32 [[B]], [[N]]
-; CHECK-NEXT: [[ADD:%.*]] = add i32 [[MUL]], [[C]]
-; CHECK-NEXT: [[ADD_1:%.*]] = add i32 [[ADD]], [[N]]
-; CHECK-NEXT: [[ADD_2:%.*]] = add i32 [[ADD_1]], [[N]]
-; CHECK-NEXT: [[OFFSET:%.*]] = sext i32 [[ADD_2]] to i64
-; CHECK-NEXT: [[GETELEM_1:%.*]] = getelementptr float, ptr [[INPUT]], i64 [[OFFSET]]
-; CHECK-NEXT: store float [[R]], ptr [[GETELEM_1]], align 4
-; CHECK-NEXT: [[GETELEM_2:%.*]] = getelementptr i8, ptr [[GETELEM_1]], i64 16
-; CHECK-NEXT: store float [[R]], ptr [[GETELEM_2]], align 4
-; CHECK-NEXT: ret void
-;
-
- %mul = mul nsw i32 %b, %n
- %add = add i32 %mul, %c
- %add.1 = add i32 %add, %n
- %add.2 = add i32 %add.1, %n
-
- %offset = sext i32 %add.2 to i64
- %1 = getelementptr float, ptr %input, i64 %offset
- store float %r, ptr %1, align 4
-
- %offset3 = add i64 %offset, 4
- %2 = getelementptr float, ptr %input, i64 %offset3
- store float %r, ptr %2, align 4
- ret void
-}
-
-
-define void @stride_var(ptr %input, i32 %c, i32 %b, i32 %n, float %r) {
-; CHECK-LABEL: define void @stride_var(
-; CHECK-SAME: ptr [[INPUT:%.*]], i32 [[C:%.*]], i32 [[B:%.*]], i32 [[N:%.*]], float [[R:%.*]]) {
-; CHECK-NEXT: [[ADD0:%.*]] = add nsw i32 [[B]], 1
-; CHECK-NEXT: [[MUL_1:%.*]] = mul nsw i32 [[ADD0]], [[N]]
-; CHECK-NEXT: [[ADD1:%.*]] = add i32 [[MUL_1]], [[C]]
-; CHECK-NEXT: [[I:%.*]] = sext i32 [[ADD1]] to i64
-; CHECK-NEXT: [[GETELEM:%.*]] = getelementptr float, ptr [[INPUT]], i64 [[I]]
-; CHECK-NEXT: store float [[R]], ptr [[GETELEM]], align 4
-; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[N]] to i64
-; CHECK-NEXT: [[TMP2:%.*]] = shl i64 [[TMP1]], 2
-; CHECK-NEXT: [[GETELEM_1:%.*]] = getelementptr inbounds i8, ptr [[GETELEM]], i64 [[TMP2]]
-; CHECK-NEXT: store float [[R]], ptr [[GETELEM_1]], align 4
-; CHECK-NEXT: ret void
-;
-; Reuse getElem to compute getElem.1 and getElem.2 with variable offset n extracted from Stride
-
- %add0 = add nsw i32 %b, 1
- %mul.1 = mul nsw i32 %add0, %n
- %add.1 = add i32 %mul.1, %c
- %offset = sext i32 %add.1 to i64
- %getElem = getelementptr float, ptr %input, i64 %offset
- store float %r, ptr %getElem, align 4
-
- %mul = mul nsw i32 %b, %n
- %add = add nsw i32 %mul, %c
- %add.11 = add nsw i32 %add, %n
- %add.2 = add nsw i32 %add.11, %n
- %offset1 = sext i32 %add.2 to i64
- %getElem.1 = getelementptr inbounds float, ptr %input, i64 %offset1
- store float %r, ptr %getElem.1, align 4
-
- ret void
-}
-
-; Base Delta
-
-%struct.B = type { i16 }
-%struct.A = type { %struct.B, %struct.B }
-
-define void @base_const(i32 %a, ptr %base, i16 %r) {
-; Reuse getElem1 to compute getElem2
-; CHECK-LABEL: define void @base_const(
-; CHECK-SAME: i32 [[A:%.*]], ptr [[BASE:%.*]], i16 [[R:%.*]]) {
-; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[A]] to i64
-; CHECK-NEXT: [[GEP1:%.*]] = getelementptr inbounds [[STRUCT_A:%.*]], ptr [[BASE]], i64 [[TMP1]]
-; CHECK-NEXT: store i16 [[R]], ptr [[GEP1]], align 2
-; CHECK-NEXT: [[GEP2:%.*]] = getelementptr inbounds i8, ptr [[GEP1]], i64 2
-; CHECK-NEXT: store i16 [[R]], ptr [[GEP2]], align 2
-; CHECK-NEXT: ret void
-;
-
- %1 = sext i32 %a to i64
- %getElem1 = getelementptr inbounds %struct.A, ptr %base, i64 %1
- store i16 %r, ptr %getElem1, align 2
- %getElem2 = getelementptr inbounds %struct.A, ptr %base, i64 %1, i32 1
- store i16 %r, ptr %getElem2, align 2
- ret void
-}
-
-define void @base_var(i32 %a, ptr %base, i16 %r, i64 %n) {
-; Reuse getElem1 to compute getElem2
-; CHECK-LABEL: define void @base_var(
-; CHECK-SAME: i32 [[A:%.*]], ptr [[BASE:%.*]], i16 [[R:%.*]], i64 [[N:%.*]]) {
-; CHECK-NEXT: [[TMP1:%.*]] = sext i32 [[A]] to i64
-; CHECK-NEXT: [[GETELEM1:%.*]] = getelementptr inbounds [[STRUCT_A:%.*]], ptr [[BASE]], i64 [[TMP1]]
-; CHECK-NEXT: store i16 [[R]], ptr [[GETELEM1]], align 2
-; CHECK-NEXT: [[GETELEM2:%.*]] = getelementptr inbounds i8, ptr [[GETELEM1]], i64 [[N]]
-; CHECK-NEXT: store i16 [[R]], ptr [[GETELEM2]], align 2
-; CHECK-NEXT: ret void
-;
-
- %1 = sext i32 %a to i64
- %base1 = getelementptr inbounds i8, ptr %base, i64 %n
- %getElem1 = getelementptr inbounds %struct.A, ptr %base, i64 %1
- store i16 %r, ptr %getElem1, align 2
- %getElem2 = getelementptr inbounds %struct.A, ptr %base1, i64 %1
- store i16 %r, ptr %getElem2, align 2
- ret void
-}
>From e7c1865b9efa9bcea1d28b7c27215e7a92f9dda0 Mon Sep 17 00:00:00 2001
From: Jakub Kuderski <jakub at nod-labs.com>
Date: Fri, 2 Jan 2026 11:29:12 -0500
Subject: [PATCH 02/10] Revert "[CMake] Fix variable name mistake"
This reverts commit 2b903df797f858ed5626bbb5aebd92872322298f.
---
llvm/cmake/modules/HandleLLVMOptions.cmake | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/llvm/cmake/modules/HandleLLVMOptions.cmake b/llvm/cmake/modules/HandleLLVMOptions.cmake
index 397aa0a6e0766..fa30ac8992a25 100644
--- a/llvm/cmake/modules/HandleLLVMOptions.cmake
+++ b/llvm/cmake/modules/HandleLLVMOptions.cmake
@@ -1172,8 +1172,8 @@ elseif(MSVC)
# Each target property or source property should be responsible to control
# them.
# CL.EXE complains to override flags like "/GR /GR-".
- string(REGEX REPLACE "(^| ) */EH[-cs]+ *( |$)" "\\1 \\2" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
- string(REGEX REPLACE "(^| ) */GR-? *( |$)" "\\1 \\2" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
+ string(REGEX REPLACE "(^| ) */EH[-cs]+ *( |$)" "\\1 \\2" CMAKE_CXXFLAGS "${CMAKE_CXX_FLAGS}")
+ string(REGEX REPLACE "(^| ) */GR-? *( |$)" "\\1 \\2" CMAKE_CXXFLAGS "${CMAKE_CXX_FLAGS}")
set(LLVM_CXXFLAGS_RTTI_DISABLE "/GR-")
set(LLVM_CXXFLAGS_RTTI_ENABLE "/GR")
elseif(CMAKE_CXXCOMPILER_ID MATCHES "XL")
>From 12d80f913c5a0f13f54610e9b0524137301665e3 Mon Sep 17 00:00:00 2001
From: Jakub Kuderski <jakub at nod-labs.com>
Date: Fri, 2 Jan 2026 11:29:22 -0500
Subject: [PATCH 03/10] Revert "[CMake] Re-add -fno-rtti to llvm-config
--cxxflags (#174084)"
This reverts commit d1b88ca8d4c5d849ae93111ae9e24d88dad4b762.
---
llvm/cmake/modules/AddLLVM.cmake | 16 +++++++++++-----
llvm/cmake/modules/HandleLLVMOptions.cmake | 15 +++------------
llvm/tools/llvm-config/CMakeLists.txt | 12 +++++-------
3 files changed, 19 insertions(+), 24 deletions(-)
diff --git a/llvm/cmake/modules/AddLLVM.cmake b/llvm/cmake/modules/AddLLVM.cmake
index d6acb4f984be9..18c0ac7015efa 100644
--- a/llvm/cmake/modules/AddLLVM.cmake
+++ b/llvm/cmake/modules/AddLLVM.cmake
@@ -51,13 +51,19 @@ function(llvm_update_compile_flags name)
# LLVM_REQUIRES_RTTI is an internal flag that individual
# targets can use to force RTTI
+ set(LLVM_CONFIG_HAS_RTTI YES CACHE INTERNAL "")
if(NOT (LLVM_REQUIRES_RTTI OR LLVM_ENABLE_RTTI))
- # TODO: GTEST_HAS_RTTI should be determined automatically, evaluate whether
- # the explicit definition is actually required.
+ set(LLVM_CONFIG_HAS_RTTI NO CACHE INTERNAL "")
list(APPEND LLVM_COMPILE_DEFINITIONS GTEST_HAS_RTTI=0)
- list(APPEND LLVM_COMPILE_CXXFLAGS ${LLVM_CXXFLAGS_RTTI_DISABLE})
- else()
- list(APPEND LLVM_COMPILE_CXXFLAGS ${LLVM_CXXFLAGS_RTTI_ENABLE})
+ if (LLVM_COMPILER_IS_GCC_COMPATIBLE)
+ list(APPEND LLVM_COMPILE_CXXFLAGS "-fno-rtti")
+ elseif (MSVC)
+ list(APPEND LLVM_COMPILE_CXXFLAGS "/GR-")
+ elseif (CMAKE_CXX_COMPILER_ID MATCHES "XL")
+ list(APPEND LLVM_COMPILE_CXXFLAGS "-qnortti")
+ endif ()
+ elseif(MSVC)
+ list(APPEND LLVM_COMPILE_CXXFLAGS "/GR")
endif()
target_compile_options(${name} PRIVATE ${LLVM_COMPILE_FLAGS} $<$<COMPILE_LANGUAGE:CXX>:${LLVM_COMPILE_CXXFLAGS}>)
diff --git a/llvm/cmake/modules/HandleLLVMOptions.cmake b/llvm/cmake/modules/HandleLLVMOptions.cmake
index fa30ac8992a25..575286e9987e1 100644
--- a/llvm/cmake/modules/HandleLLVMOptions.cmake
+++ b/llvm/cmake/modules/HandleLLVMOptions.cmake
@@ -1162,22 +1162,13 @@ elseif(MSVC)
endif()
endif()
-# llvm_update_compile_flags adds one of these to each target.
-set(LLVM_CXXFLAGS_RTTI_DISABLE "")
-set(LLVM_CXXFLAGS_RTTI_ENABLE "")
-if(LLVM_COMPILER_IS_GCC_COMPATIBLE)
- set(LLVM_CXXFLAGS_RTTI_DISABLE "-fno-rtti")
-elseif(MSVC)
+if(MSVC)
# Remove flags here, for exceptions and RTTI.
# Each target property or source property should be responsible to control
# them.
# CL.EXE complains to override flags like "/GR /GR-".
- string(REGEX REPLACE "(^| ) */EH[-cs]+ *( |$)" "\\1 \\2" CMAKE_CXXFLAGS "${CMAKE_CXX_FLAGS}")
- string(REGEX REPLACE "(^| ) */GR-? *( |$)" "\\1 \\2" CMAKE_CXXFLAGS "${CMAKE_CXX_FLAGS}")
- set(LLVM_CXXFLAGS_RTTI_DISABLE "/GR-")
- set(LLVM_CXXFLAGS_RTTI_ENABLE "/GR")
-elseif(CMAKE_CXXCOMPILER_ID MATCHES "XL")
- set(LLVM_CXXFLAGS_RTTI_DISABLE "-qnortti")
+ string(REGEX REPLACE "(^| ) */EH[-cs]+ *( |$)" "\\1 \\2" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
+ string(REGEX REPLACE "(^| ) */GR-? *( |$)" "\\1 \\2" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
endif()
# Provide public options to globally control RTTI and EH
diff --git a/llvm/tools/llvm-config/CMakeLists.txt b/llvm/tools/llvm-config/CMakeLists.txt
index bf9a012ef5302..3a8f40b9c6fea 100644
--- a/llvm/tools/llvm-config/CMakeLists.txt
+++ b/llvm/tools/llvm-config/CMakeLists.txt
@@ -45,6 +45,9 @@ endforeach()
string(REPLACE ";" " " SYSTEM_LIBS "${SYSTEM_LIBS}")
+# Fetch target specific compile options, e.g. RTTI option
+get_property(COMPILE_FLAGS TARGET llvm-config PROPERTY COMPILE_FLAGS)
+
# NOTE: We don't want to start extracting any random C/CXX flags that the
# user may add that could affect the ABI. We only want to extract flags
# that have been added by the LLVM build system.
@@ -58,14 +61,9 @@ set(LLVM_CPPFLAGS "${LLVM_DEFINITIONS}")
set(LLVM_CFLAGS "${LLVM_C_STD_FLAG} ${LLVM_DEFINITIONS}")
# The language standard potentially affects the ABI/API of LLVM, so we want
# to make sure it is reported by llvm-config.
-set(LLVM_CXXFLAGS "${CMAKE_CXX${CMAKE_CXX_STANDARD}_STANDARD_COMPILE_OPTION} ${LLVM_CXX_STDLIB_FLAG} ${LLVM_DEFINITIONS}")
-if(LLVM_ENABLE_RTTI)
- append(${LLVM_CXXFLAGS_RTTI_ENABLE} LLVM_CXXFLAGS)
-else()
- append(${LLVM_CXXFLAGS_RTTI_DISABLE} LLVM_CXXFLAGS)
-endif()
+set(LLVM_CXXFLAGS "${CMAKE_CXX${CMAKE_CXX_STANDARD}_STANDARD_COMPILE_OPTION} ${LLVM_CXX_STDLIB_FLAG} ${COMPILE_FLAGS} ${LLVM_DEFINITIONS}")
set(LLVM_BUILD_SYSTEM cmake)
-set(LLVM_HAS_RTTI ${LLVM_ENABLE_RTTI})
+set(LLVM_HAS_RTTI ${LLVM_CONFIG_HAS_RTTI})
set(LLVM_DYLIB_VERSION "${LLVM_VERSION_MAJOR}${LLVM_VERSION_SUFFIX}")
set(LLVM_SHARED_LIBRARY_PREFIX "${CMAKE_SHARED_LIBRARY_PREFIX}")
>From e4a8af3f40e639d9940ca82b9814b15f94c16c26 Mon Sep 17 00:00:00 2001
From: Shaked Flur <fshaked at gmail.com>
Date: Sun, 15 Feb 2026 11:59:03 +0100
Subject: [PATCH 04/10] Add peelForLoopLastIteration (#4)
* similar to peelForLoopFirstIteration.
* call rewriteAffineOpAfterPeeling to rewrite affine.min/max ops.
* compose affine map and operands when simplifying.
This is needed when previous calls optimized other ops.
---
.../mlir/Dialect/SCF/Transforms/Transforms.h | 3 ++
mlir/lib/Dialect/Affine/Analysis/Utils.cpp | 4 +-
.../SCF/Transforms/LoopSpecialization.cpp | 46 ++++++++++++++++++-
3 files changed, 51 insertions(+), 2 deletions(-)
diff --git a/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h b/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
index 00e8572307151..12b6fdfde411f 100644
--- a/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
@@ -106,6 +106,9 @@ LogicalResult peelForLoopAndSimplifyBounds(RewriterBase &rewriter, ForOp forOp,
LogicalResult peelForLoopFirstIteration(RewriterBase &rewriter, ForOp forOp,
scf::ForOp &partialIteration);
+LogicalResult peelForLoopLastIteration(RewriterBase &rewriter, ForOp forOp,
+ scf::ForOp &partialIteration);
+
/// Tile a parallel loop of the form
/// scf.parallel (%i0, %i1) = (%arg0, %arg1) to (%arg2, %arg3)
/// step (%arg4, %arg5)
diff --git a/mlir/lib/Dialect/Affine/Analysis/Utils.cpp b/mlir/lib/Dialect/Affine/Analysis/Utils.cpp
index f38493bc9a96e..277f41473b0b6 100644
--- a/mlir/lib/Dialect/Affine/Analysis/Utils.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/Utils.cpp
@@ -2291,7 +2291,9 @@ FailureOr<AffineValueMap> mlir::affine::simplifyConstrainedMinMaxOp(
Builder builder(ctx);
AffineMap map =
isMin ? cast<AffineMinOp>(op).getMap() : cast<AffineMaxOp>(op).getMap();
- ValueRange operands = op->getOperands();
+ SmallVector<Value> operands =
+ isMin ? cast<AffineMinOp>(op).getOperands() : cast<AffineMaxOp>(op).getOperands();
+ affine::fullyComposeAffineMapAndOperands(&map, &operands);
unsigned numResults = map.getNumResults();
// Add a few extra dimensions.
diff --git a/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp b/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
index c7588b433dee1..b33cd8358ea2f 100644
--- a/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
@@ -24,6 +24,8 @@
#include "mlir/IR/IRMapping.h"
#include "mlir/IR/PatternMatch.h"
#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/Support/MathExtras.h"
namespace mlir {
#define GEN_PASS_DEF_SCFFORLOOPPEELING
@@ -223,7 +225,7 @@ LogicalResult mlir::scf::peelForLoopFirstIteration(RewriterBase &b, ForOp forOp,
auto stepInt = getConstantIntValue(forOp.getStep());
// Peeling is not needed if there is one or less iteration.
- if (lbInt && ubInt && stepInt && ceil(float(*ubInt - *lbInt) / *stepInt) <= 1)
+ if (lbInt && ubInt && stepInt && llvm::divideCeil(*ubInt - *lbInt, *stepInt) <= 1)
return failure();
AffineExpr lbSymbol, stepSymbol;
@@ -253,6 +255,48 @@ LogicalResult mlir::scf::peelForLoopFirstIteration(RewriterBase &b, ForOp forOp,
return success();
}
+LogicalResult mlir::scf::peelForLoopLastIteration(RewriterBase &b, ForOp forOp,
+ ForOp &lastIteration) {
+ RewriterBase::InsertionGuard guard(b);
+ auto lbInt = getConstantIntValue(forOp.getLowerBound());
+ auto ubInt = getConstantIntValue(forOp.getUpperBound());
+ auto stepInt = getConstantIntValue(forOp.getStep());
+
+ // Peeling is not needed if there is one or less iteration.
+ if (lbInt && ubInt && stepInt && llvm::divideCeil(*ubInt - *lbInt, *stepInt) <= 1)
+ return failure();
+
+ AffineExpr ubSymbol, stepSymbol;
+ bindSymbols(b.getContext(), ubSymbol, stepSymbol);
+
+ // New upper bound for main loop: %ub - %step
+ auto ubMap = AffineMap::get(0, 2, {ubSymbol - stepSymbol});
+ b.setInsertionPoint(forOp);
+ auto loc = forOp.getLoc();
+ Value splitBound = b.createOrFold<AffineApplyOp>(
+ loc, ubMap, ValueRange{forOp.getUpperBound(), forOp.getStep()});
+
+ b.setInsertionPointAfter(forOp);
+
+ // Peel the last iteration.
+ IRMapping map;
+ map.map(forOp.getLowerBound(), splitBound);
+ lastIteration = cast<ForOp>(b.clone(*forOp.getOperation(), map));
+ b.replaceAllUsesWith(forOp.getResults(), lastIteration->getResults());
+ // This has to be done after the replace above, so that replace does not change it
+ lastIteration.getInitArgsMutable().assign(forOp->getResults());
+
+ // Update main loop with new upper bound.
+ b.modifyOpInPlace(forOp, [&]() {
+ forOp.getUpperBoundMutable().assign(splitBound);
+ });
+
+ // Rewrite affine.min and affine.max ops.
+ rewriteAffineOpAfterPeeling(b, forOp, lastIteration, lastIteration.getUpperBound());
+
+ return success();
+}
+
static constexpr char kPeeledLoopLabel[] = "__peeled_loop__";
static constexpr char kPartialIterationLabel[] = "__partial_iteration__";
>From ee19d113549020d9d86cba5a2d9834e995fb98f2 Mon Sep 17 00:00:00 2001
From: ayounes-synaptics <50144496+ayounes-synaptics at users.noreply.github.com>
Date: Sun, 15 Feb 2026 12:16:23 +0100
Subject: [PATCH 05/10] Do not use clone with map (#6)
The map argument is too aggressive, it changes every occurrence of the value,
not just the upper/lower-bound, as intended. For example, if the lower-bound is
%c0, and we want to change it to %c42, using the map, all the uses of %c0 in
the loop body will also change to %c10 (instead of just the lower-bound).
Co-authored-by: Shaked Flur <fshaked at gmail.com>
---
mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp b/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
index b33cd8358ea2f..61890ddd48d14 100644
--- a/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
@@ -279,12 +279,11 @@ LogicalResult mlir::scf::peelForLoopLastIteration(RewriterBase &b, ForOp forOp,
b.setInsertionPointAfter(forOp);
// Peel the last iteration.
- IRMapping map;
- map.map(forOp.getLowerBound(), splitBound);
- lastIteration = cast<ForOp>(b.clone(*forOp.getOperation(), map));
+ lastIteration = cast<ForOp>(b.clone(*forOp.getOperation()));
b.replaceAllUsesWith(forOp.getResults(), lastIteration->getResults());
// This has to be done after the replace above, so that replace does not change it
lastIteration.getInitArgsMutable().assign(forOp->getResults());
+ lastIteration.getLowerBoundMutable().assign(splitBound);
// Update main loop with new upper bound.
b.modifyOpInPlace(forOp, [&]() {
>From ea6e4677511a7e1280026c011c1bf770da055502 Mon Sep 17 00:00:00 2001
From: Lukas Sommer <lukas.sommer at amd.com>
Date: Tue, 27 Jan 2026 15:06:57 +0100
Subject: [PATCH 06/10] [mlir] Avoid segfault in 'MoveBlockRewrite' rollback
(#178148)
Prior to this change, rollback of the `MoveBlockRewrite` could result in
segfault if the block wasn't contained in a region anymore.
That situation could arise if the previous rollback of another rewrite
orphaned the block by removing it from its region, as demonstrated by
the new test pattern.
Signed-off-by: Lukas Sommer <lukas.sommer at amd.com>
---
.../Transforms/Utils/DialectConversion.cpp | 8 ++++-
.../Transforms/test-legalizer-rollback.mlir | 12 +++++++
mlir/test/lib/Dialect/Test/TestPatterns.cpp | 34 ++++++++++++++++++-
3 files changed, 52 insertions(+), 2 deletions(-)
diff --git a/mlir/lib/Transforms/Utils/DialectConversion.cpp b/mlir/lib/Transforms/Utils/DialectConversion.cpp
index bd972df271ed6..52067f1b95ad4 100644
--- a/mlir/lib/Transforms/Utils/DialectConversion.cpp
+++ b/mlir/lib/Transforms/Utils/DialectConversion.cpp
@@ -548,7 +548,13 @@ class MoveBlockRewrite : public BlockRewrite {
// Move the block back to its original position.
Region::iterator before =
insertBeforeBlock ? Region::iterator(insertBeforeBlock) : region->end();
- region->getBlocks().splice(before, block->getParent()->getBlocks(), block);
+ if (Region *currentParent = block->getParent()) {
+ // Block is still in a region, use cheap splice to move it back.
+ region->getBlocks().splice(before, currentParent->getBlocks(), block);
+ return;
+ }
+ // Block was orphaned by a prior rollback, can't splice.
+ region->getBlocks().insert(before, block);
}
private:
diff --git a/mlir/test/Transforms/test-legalizer-rollback.mlir b/mlir/test/Transforms/test-legalizer-rollback.mlir
index 4bcca6b7e5228..f6569201842b7 100644
--- a/mlir/test/Transforms/test-legalizer-rollback.mlir
+++ b/mlir/test/Transforms/test-legalizer-rollback.mlir
@@ -138,6 +138,18 @@ func.func @test_properties_rollback() {
// -----
+// CHECK-LABEL: func @test_undo_block_move_detached
+func.func @test_undo_block_move_detached() {
+ // expected-remark @below{{op 'test.undo_detached_block_move' is not legalizable}}
+ "test.undo_detached_block_move"() ({
+ ^bb0(%arg0: i64):
+ "test.return"() : () -> ()
+ }) : () -> ()
+ "test.return"() : () -> ()
+}
+
+// -----
+
// expected-remark at +1 {{applyPartialConversion failed}}
builtin.module {
// Test that region cloning can be properly undone.
diff --git a/mlir/test/lib/Dialect/Test/TestPatterns.cpp b/mlir/test/lib/Dialect/Test/TestPatterns.cpp
index 6c564a6592c11..6c44ace831e96 100644
--- a/mlir/test/lib/Dialect/Test/TestPatterns.cpp
+++ b/mlir/test/lib/Dialect/Test/TestPatterns.cpp
@@ -1042,6 +1042,38 @@ struct TestUndoPropertiesModification : public ConversionPattern {
}
};
+/// A pattern that tests the undo mechanism for a block move if the block was
+/// moved to a detached region. The block is first moved to a detached region
+/// and then a new operation is created with that region. During rollback, first
+/// the `CreateOperationRewrite` is rolled back, causing the block to be
+/// orphaned, i.e., removed from the region. Only then the `MoveBlockRewrite` is
+/// rolled back, which now can't access the region anymore. The test ensures
+/// that the rollback still works and doesn't try to access the orphaned block's
+/// containing region, leading to segfault.
+struct TestUndoMoveDetachedBlock : public ConversionPattern {
+ TestUndoMoveDetachedBlock(MLIRContext *ctx)
+ : ConversionPattern("test.undo_detached_block_move", /*benefit=*/1, ctx) {
+ }
+
+ LogicalResult
+ matchAndRewrite(Operation *op, ArrayRef<Value> operands,
+ ConversionPatternRewriter &rewriter) const final {
+ if (op->getNumRegions() != 1)
+ return failure();
+ // Create an illegal operation to trigger rollback.
+ OperationState state(op->getLoc(), "test.illegal_op_created_after_move",
+ operands, op->getResultTypes(), {}, BlockRange());
+ // Create detached region.
+ Region *newRegion = state.addRegion();
+ // Move blocks to the still detached region
+ rewriter.inlineRegionBefore(op->getRegion(0), *newRegion,
+ newRegion->begin());
+ Operation *newOp = rewriter.create(state);
+ rewriter.replaceOp(op, newOp->getResults());
+ return success();
+ }
+};
+
//===----------------------------------------------------------------------===//
// Type-Conversion Rewrite Testing
//===----------------------------------------------------------------------===//
@@ -1548,7 +1580,7 @@ struct TestLegalizePatternDriver
TestUpdateConsumerType, TestNonRootReplacement,
TestBoundedRecursiveRewrite, TestNestedOpCreationUndoRewrite,
TestReplaceEraseOp, TestCreateUnregisteredOp, TestUndoMoveOpBefore,
- TestUndoPropertiesModification, TestEraseOp,
+ TestUndoPropertiesModification, TestUndoMoveDetachedBlock, TestEraseOp,
TestReplaceWithValidProducer, TestReplaceWithValidConsumer,
TestRepetitive1ToNConsumer>(&getContext());
patterns.add<TestDropOpSignatureConversion, TestDropAndReplaceInvalidOp,
>From f8260b921e2b365a12af44a5fcb93e75ffd4e7f5 Mon Sep 17 00:00:00 2001
From: Shaked Flur <sflur at google.com>
Date: Wed, 6 May 2026 18:15:34 +0200
Subject: [PATCH 07/10] Fix the simplification after peeling last iteration
---
.../SCF/Transforms/LoopSpecialization.cpp | 40 +++++++++++++------
1 file changed, 27 insertions(+), 13 deletions(-)
diff --git a/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp b/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
index 61890ddd48d14..f0f1a95fe7ff1 100644
--- a/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
@@ -255,9 +255,9 @@ LogicalResult mlir::scf::peelForLoopFirstIteration(RewriterBase &b, ForOp forOp,
return success();
}
-LogicalResult mlir::scf::peelForLoopLastIteration(RewriterBase &b, ForOp forOp,
- ForOp &lastIteration) {
- RewriterBase::InsertionGuard guard(b);
+LogicalResult mlir::scf::peelForLoopLastIteration(RewriterBase &rewriter, ForOp forOp,
+ ForOp &partialIteration) {
+ RewriterBase::InsertionGuard guard(rewriter);
auto lbInt = getConstantIntValue(forOp.getLowerBound());
auto ubInt = getConstantIntValue(forOp.getUpperBound());
auto stepInt = getConstantIntValue(forOp.getStep());
@@ -267,31 +267,45 @@ LogicalResult mlir::scf::peelForLoopLastIteration(RewriterBase &b, ForOp forOp,
return failure();
AffineExpr ubSymbol, stepSymbol;
- bindSymbols(b.getContext(), ubSymbol, stepSymbol);
+ bindSymbols(rewriter.getContext(), ubSymbol, stepSymbol);
// New upper bound for main loop: %ub - %step
auto ubMap = AffineMap::get(0, 2, {ubSymbol - stepSymbol});
- b.setInsertionPoint(forOp);
+ rewriter.setInsertionPoint(forOp);
auto loc = forOp.getLoc();
- Value splitBound = b.createOrFold<AffineApplyOp>(
+ Value splitBound = rewriter.createOrFold<AffineApplyOp>(
loc, ubMap, ValueRange{forOp.getUpperBound(), forOp.getStep()});
- b.setInsertionPointAfter(forOp);
+ rewriter.setInsertionPointAfter(forOp);
// Peel the last iteration.
- lastIteration = cast<ForOp>(b.clone(*forOp.getOperation()));
- b.replaceAllUsesWith(forOp.getResults(), lastIteration->getResults());
+ partialIteration = cast<ForOp>(rewriter.clone(*forOp.getOperation()));
+ rewriter.replaceAllUsesWith(forOp.getResults(), partialIteration->getResults());
// This has to be done after the replace above, so that replace does not change it
- lastIteration.getInitArgsMutable().assign(forOp->getResults());
- lastIteration.getLowerBoundMutable().assign(splitBound);
+ rewriter.modifyOpInPlace(partialIteration, [&]() {
+ partialIteration.getInitArgsMutable().assign(forOp->getResults());
+ partialIteration.getLowerBoundMutable().assign(splitBound);
+ });
// Update main loop with new upper bound.
- b.modifyOpInPlace(forOp, [&]() {
+ rewriter.modifyOpInPlace(forOp, [&]() {
forOp.getUpperBoundMutable().assign(splitBound);
});
// Rewrite affine.min and affine.max ops.
- rewriteAffineOpAfterPeeling(b, forOp, lastIteration, lastIteration.getUpperBound());
+ forOp.walk([&](Operation *affineOp) {
+ if (!isa<AffineMinOp, AffineMaxOp>(affineOp))
+ return WalkResult::advance();
+ (void)canonicalizeMinMaxOpInLoop(rewriter, affineOp, matchForLikeLoop);
+ return WalkResult::advance();
+ });
+
+ partialIteration.walk([&](Operation *affineOp) {
+ if (!isa<AffineMinOp, AffineMaxOp>(affineOp))
+ return WalkResult::advance();
+ (void)canonicalizeMinMaxOpInLoop(rewriter, affineOp, matchForLikeLoop);
+ return WalkResult::advance();
+ });
return success();
}
>From 1af1d2031e82ce248685ab20dee5c6e77527a0f9 Mon Sep 17 00:00:00 2001
From: ayounes-synaptics <50144496+ayounes-synaptics at users.noreply.github.com>
Date: Mon, 13 Jul 2026 14:42:42 +0200
Subject: [PATCH 08/10] [mlir][tosa] Fix TransposeConv decomposition with i16
input (#12)
Patch is from @Livesh-kumar
** The upstream TransposeConvStridedConverter from TosaDecomposeTransposeConv.cpp converts:
tosa.conv + reshape + transpose + reshape + bias_add + rescale
For i16 input, the intermediate conv result is i48, and [TOSA reshape specification](https://www.mlplatform.org/tosa/tosa_spec_1_0_1.html#_reshape) does not support i48.
That causes a spec-level incompatibility.
** Implemented fix for transpose_conv + rescale path:
Replicated the core logic of TransposeConvStridedConverter, but changed ordering to:
transpose_conv + tosa.rescale = tosa.conv + bias_add + tosa.rescale + reshape + transpose + reshape
With this order, rescale converts i48 to i16 before reshape/transpose/reshape; as a result, reshape sees i16
input and proceeds correctly.
Advantage of this approach
* `reshape + transpose + reshape` will work on top rescaled data `(i64 -> i16/i32 -> i8)`, which is 4 times lesser data then the normal approach
When there is no rescale, it will lower through same pipeline as previous (**`TransposeConvStridedConverter`**)
Please refer this ticket for more information : synaptics-torq/torq-compiler-dev#1979
---
.../Transforms/TosaDecomposeTransposeConv.cpp | 647 +++++++++++-------
1 file changed, 415 insertions(+), 232 deletions(-)
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaDecomposeTransposeConv.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaDecomposeTransposeConv.cpp
index 8b23fd1341bc5..90a4442c2565e 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaDecomposeTransposeConv.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaDecomposeTransposeConv.cpp
@@ -24,6 +24,395 @@ using namespace mlir::tosa;
namespace {
+struct StridedTransposeConvPrep {
+ explicit StridedTransposeConvPrep(Location l) : loc(l) {}
+
+ Location loc;
+ Value input;
+ Value weight;
+ Value bias;
+
+ ShapedType inputTy;
+ ShapedType weightTy;
+ ShapedType biasTy;
+ ShapedType resultTy;
+
+ Type inputETy;
+ Type weightETy;
+ Type biasETy;
+ Type resultETy;
+
+ llvm::ArrayRef<int64_t> pad;
+ llvm::ArrayRef<int64_t> stride;
+
+ int64_t batch;
+ int64_t outputChannels;
+ int64_t inputChannels;
+ int64_t inputZpVal;
+ int64_t weightZpVal;
+};
+
+static LogicalResult prepareStridedTransposeConv(tosa::TransposeConv2DOp op,
+ PatternRewriter &rewriter,
+ StridedTransposeConvPrep &p) {
+ p.loc = op->getLoc();
+ p.input = op->getOperand(0);
+ p.weight = op->getOperand(1);
+ p.bias = op->getOperand(2);
+
+ p.inputTy = cast<ShapedType>(p.input.getType());
+ p.weightTy = cast<ShapedType>(p.weight.getType());
+ p.biasTy = cast<ShapedType>(p.bias.getType());
+ p.resultTy = cast<ShapedType>(op->getResult(0).getType());
+
+ p.inputETy = p.inputTy.getElementType();
+ p.weightETy = p.weightTy.getElementType();
+ p.biasETy = p.biasTy.getElementType();
+ p.resultETy = p.resultTy.getElementType();
+
+ p.pad = op.getOutPad();
+ p.stride = op.getStride();
+
+ if (llvm::all_of(p.stride, [](int64_t v) { return v == 1; }))
+ return rewriter.notifyMatchFailure(op, "non-one stride found.");
+
+ for (unsigned int i = 1; i < 4; ++i) {
+ if (p.inputTy.isDynamicDim(i) || p.resultTy.isDynamicDim(i))
+ return failure();
+ }
+
+ if (!p.weightTy.hasStaticShape() || !p.biasTy.hasStaticShape())
+ return failure();
+
+ FailureOr<int64_t> maybeIZp = op.getInputZeroPoint();
+ if (failed(maybeIZp))
+ return rewriter.notifyMatchFailure(
+ op, "input zero point cannot be statically determined");
+
+ FailureOr<int64_t> maybeWZp = op.getWeightZeroPoint();
+ if (failed(maybeWZp))
+ return rewriter.notifyMatchFailure(
+ op, "weight zero point cannot be statically determined");
+
+ p.inputZpVal = *maybeIZp;
+ p.weightZpVal = *maybeWZp;
+
+ if (op.verifyInputZeroPoint(p.inputZpVal).failed())
+ return rewriter.notifyMatchFailure(
+ op, "input zero point must be zero for non-int8 integer types");
+
+ if (op.verifyWeightZeroPoint(p.weightZpVal).failed())
+ return rewriter.notifyMatchFailure(
+ op, "weight zero point must be zero for non-int8 integer types");
+
+ p.batch = p.inputTy.getDimSize(0);
+ p.outputChannels = p.weightTy.getDimSize(0);
+ int64_t weightHeight = p.weightTy.getDimSize(1);
+ int64_t weightWidth = p.weightTy.getDimSize(2);
+ p.inputChannels = p.weightTy.getDimSize(3);
+
+ llvm::SmallVector<int64_t, 8> weightPadding = {0, 0, 0, 0, 0, 0, 0, 0};
+ weightPadding[3] = (weightHeight % p.stride[0])
+ ? (p.stride[0] - weightHeight % p.stride[0])
+ : 0;
+ weightPadding[5] = (weightWidth % p.stride[1])
+ ? (p.stride[1] - weightWidth % p.stride[1])
+ : 0;
+
+ Value weightPaddingVal =
+ getTosaConstShape(rewriter, op->getLoc(), weightPadding);
+
+ ImplicitLocOpBuilder builder(op->getLoc(), rewriter);
+ const Value inputPadConst =
+ createPadConstTensor(builder, op->getLoc(), p.input, p.inputZpVal);
+ const Value weightPadConst =
+ createPadConstTensor(builder, op->getLoc(), p.input, p.weightZpVal);
+
+ p.weight = CreateOpAndInferShape<tosa::PadOp>(
+ rewriter, p.loc, UnrankedTensorType::get(p.weightETy), p.weight,
+ weightPaddingVal, weightPadConst);
+
+ p.weightTy = cast<ShapedType>(p.weight.getType());
+ weightHeight = p.weightTy.getDimSize(1);
+ weightWidth = p.weightTy.getDimSize(2);
+
+ llvm::SmallVector<int64_t, 6> weightReshapeDims0 = {
+ p.outputChannels, weightHeight / p.stride[0],
+ p.stride[0], weightWidth / p.stride[1],
+ p.stride[1], p.inputChannels};
+
+ p.weight = CreateOpAndInferShape<tosa::ReshapeOp>(
+ builder, UnrankedTensorType::get(p.weightETy), p.weight,
+ getTosaConstShape(rewriter, p.loc, weightReshapeDims0));
+
+ p.weight = CreateOpAndInferShape<tosa::TransposeOp>(
+ rewriter, p.loc, UnrankedTensorType::get(p.weightETy), p.weight,
+ rewriter.getDenseI32ArrayAttr({2, 4, 0, 1, 3, 5}));
+
+ llvm::SmallVector<int64_t, 4> weightReshapeDims1 = {
+ p.outputChannels * p.stride[0] * p.stride[1], weightHeight / p.stride[0],
+ weightWidth / p.stride[1], p.inputChannels};
+
+ p.weight = CreateOpAndInferShape<tosa::ReshapeOp>(
+ rewriter, p.loc, UnrankedTensorType::get(p.weightETy), p.weight,
+ getTosaConstShape(rewriter, p.loc, weightReshapeDims1));
+ ShapedType restridedWeightTy = cast<ShapedType>(p.weight.getType());
+
+ p.weight = CreateOpAndInferShape<tosa::ReverseOp>(
+ rewriter, p.loc, UnrankedTensorType::get(p.weightETy), p.weight,
+ /* axis = */ rewriter.getI32IntegerAttr(1));
+ p.weight = CreateOpAndInferShape<tosa::ReverseOp>(
+ rewriter, p.loc, UnrankedTensorType::get(p.weightETy), p.weight,
+ /* axis = */ rewriter.getI32IntegerAttr(2));
+
+ llvm::SmallVector<int64_t, 8> inputPadding = {0, 0, 0, 0, 0, 0, 0, 0};
+ inputPadding[2] += restridedWeightTy.getDimSize(1) - 1;
+ inputPadding[3] += restridedWeightTy.getDimSize(1) - 1;
+ inputPadding[4] += restridedWeightTy.getDimSize(2) - 1;
+ inputPadding[5] += restridedWeightTy.getDimSize(2) - 1;
+
+ Value inputPaddingVal =
+ getTosaConstShape(rewriter, op->getLoc(), inputPadding);
+
+ p.input = CreateOpAndInferShape<tosa::PadOp>(
+ rewriter, p.loc, UnrankedTensorType::get(p.inputETy), p.input,
+ inputPaddingVal, inputPadConst);
+
+ return success();
+}
+
+static FailureOr<Value> buildConv2DFromPreparedInput(
+ tosa::TransposeConv2DOp op, PatternRewriter &rewriter,
+ const StridedTransposeConvPrep &p, Value convBias, Type convResultElemTy) {
+ auto inputZp =
+ createZeroPointTensor(rewriter, p.loc, p.input.getType(), p.inputZpVal);
+ auto weightZp =
+ createZeroPointTensor(rewriter, p.loc, p.weight.getType(), p.weightZpVal);
+
+ if (!inputZp.has_value() || !weightZp.has_value()) {
+ return failure();
+ }
+
+ Value conv2d =
+ CreateOpAndInferShape<tosa::Conv2DOp>(
+ rewriter, p.loc, UnrankedTensorType::get(convResultElemTy), p.input,
+ p.weight, convBias, inputZp.value(), weightZp.value(),
+ /*pad=*/rewriter.getDenseI64ArrayAttr({0, 0, 0, 0}),
+ /*stride=*/rewriter.getDenseI64ArrayAttr({1, 1}),
+ /*dilation=*/rewriter.getDenseI64ArrayAttr({1, 1}),
+ /* acc_type = */ op.getAccType())
+ .getResult();
+
+ return conv2d;
+}
+
+static Value shuffleSliceAndPadResult(PatternRewriter &rewriter, Location loc,
+ Value value, Type resultElemTy,
+ ShapedType resultTy, int64_t batch,
+ int64_t outputChannels,
+ llvm::ArrayRef<int64_t> stride,
+ llvm::ArrayRef<int64_t> pad) {
+ ShapedType convTy = cast<ShapedType>(value.getType());
+ Type convETy = convTy.getElementType();
+
+ int64_t convHeight = convTy.getDimSize(1);
+ int64_t convWidth = convTy.getDimSize(2);
+
+ llvm::SmallVector<int64_t, 6> convReshapeDims0 = {
+ batch, convHeight, convWidth, stride[0], stride[1], outputChannels};
+
+ auto convReshapeDims0Value =
+ getTosaConstShape(rewriter, loc, convReshapeDims0);
+
+ value = CreateOpAndInferShape<tosa::ReshapeOp>(
+ rewriter, loc, UnrankedTensorType::get(resultElemTy), value,
+ convReshapeDims0Value);
+
+ value = CreateOpAndInferShape<tosa::TransposeOp>(
+ rewriter, loc, UnrankedTensorType::get(convETy), value,
+ rewriter.getDenseI32ArrayAttr({0, 1, 3, 2, 4, 5}));
+
+ llvm::SmallVector<int64_t, 6> convReshapeDims1 = {
+ batch, convHeight * stride[0], convWidth * stride[1], outputChannels};
+
+ auto convReshapeDims1Value =
+ getTosaConstShape(rewriter, loc, convReshapeDims1);
+
+ value = CreateOpAndInferShape<tosa::ReshapeOp>(
+ rewriter, loc, UnrankedTensorType::get(resultElemTy), value,
+ convReshapeDims1Value);
+
+ int64_t resultSliceTop = std::max<int64_t>(0, -pad[0]);
+ int64_t resultSliceLeft = std::max<int64_t>(0, -pad[2]);
+ int64_t resultPadTop = std::max<int64_t>(0, pad[0]);
+ int64_t resultPadLeft = std::max<int64_t>(0, pad[2]);
+
+ int64_t resultSliceHeight =
+ std::min<int64_t>(convReshapeDims1[1] - resultSliceTop,
+ resultTy.getDimSize(1) - resultPadTop);
+ int64_t resultSliceWidth =
+ std::min<int64_t>(convReshapeDims1[2] - resultSliceLeft,
+ resultTy.getDimSize(2) - resultPadLeft);
+
+ llvm::SmallVector<int64_t, 4> sliceBegin = {0, resultSliceTop,
+ resultSliceLeft, 0};
+ llvm::SmallVector<int64_t, 4> sliceSize(convReshapeDims1.begin(),
+ convReshapeDims1.end());
+ sliceSize[1] = resultSliceHeight;
+ sliceSize[2] = resultSliceWidth;
+
+ auto slice = CreateOpAndInferShape<tosa::SliceOp>(
+ rewriter, loc, UnrankedTensorType::get(resultElemTy), value,
+ getTosaConstShape(rewriter, loc, sliceBegin),
+ getTosaConstShape(rewriter, loc, sliceSize))
+ .getResult();
+
+ llvm::SmallVector<int64_t, 8> resultPadding = {0, 0, 0, 0, 0, 0, 0, 0};
+ resultPadding[2] = resultPadTop;
+ resultPadding[3] = resultTy.getDimSize(1) - resultPadTop - sliceSize[1];
+ resultPadding[4] = resultPadLeft;
+ resultPadding[5] = resultTy.getDimSize(2) - resultPadLeft - sliceSize[2];
+
+ Value resultPaddingVal = getTosaConstShape(rewriter, loc, resultPadding);
+
+ return CreateOpAndInferShape<tosa::PadOp>(
+ rewriter, loc, UnrankedTensorType::get(resultElemTy), slice,
+ resultPaddingVal);
+}
+
+class TransposeConvRescaleDecompose : public OpRewritePattern<tosa::RescaleOp> {
+public:
+ explicit TransposeConvRescaleDecompose(MLIRContext *ctx)
+ : OpRewritePattern<tosa::RescaleOp>(ctx, PatternBenefit(2)) {}
+
+ LogicalResult matchAndRewrite(tosa::RescaleOp rescaleOp,
+ PatternRewriter &rewriter) const final {
+ // Match the pair: transpose_conv2d -> rescale.
+ // We intentionally anchor on rescale (instead of transpose_conv2d) so we
+ // can move quantization earlier in the decomposition flow.
+ //
+ // Why this matters:
+ // - transpose_conv2d may produce i48 accumulators (for example i16 input +
+ // i8 weight).
+ // - Some follow-up reshape/transpose paths are not legal/efficient on i48.
+ // - Running rescale early converts conv output to a supported narrow type
+ // (typically i16/i8), so later reshape/transpose/slice are cheaper and
+ // legal.
+ //
+ // This corresponds to the preferred strategy:
+ // conv2d + bias + rescale -> reshape + transpose + reshape + slice
+ // and avoids carrying wide accumulator types through depth-to-space style
+ // ops.
+ auto op = dyn_cast_or_null<tosa::TransposeConv2DOp>(
+ rescaleOp.getInput().getDefiningOp());
+ if (!op)
+ return rewriter.notifyMatchFailure(rescaleOp,
+ "input is not transpose_conv2d");
+
+ StridedTransposeConvPrep p(op.getLoc());
+ if (failed(prepareStridedTransposeConv(op, rewriter, p)))
+ return failure();
+
+ int64_t newOC = p.outputChannels * p.stride[0] * p.stride[1];
+
+ DenseElementsAttr biasAttr;
+ if (!matchPattern(p.bias, m_Constant(&biasAttr)))
+ return rewriter.notifyMatchFailure(op, "bias must be a static constant");
+
+ auto biasVals = llvm::to_vector(biasAttr.getValues<Attribute>());
+ if (static_cast<int64_t>(biasVals.size()) != p.outputChannels)
+ return rewriter.notifyMatchFailure(op, "bias size mismatch with OC");
+
+ SmallVector<Attribute> tiledBias(newOC);
+ for (int64_t s0 = 0; s0 < p.stride[0]; ++s0)
+ for (int64_t s1 = 0; s1 < p.stride[1]; ++s1)
+ for (int64_t oc = 0; oc < p.outputChannels; ++oc)
+ tiledBias[s0 * p.stride[1] * p.outputChannels +
+ s1 * p.outputChannels + oc] = biasVals[oc];
+
+ auto tiledBiasTy = RankedTensorType::get({newOC}, p.biasETy);
+ Value tiledBiasConst =
+ tosa::ConstOp::create(
+ rewriter, p.loc, tiledBiasTy,
+ DenseElementsAttr::get(tiledBiasTy, ArrayRef<Attribute>(tiledBias)))
+ .getResult();
+
+ FailureOr<Value> maybeConv = buildConv2DFromPreparedInput(
+ op, rewriter, p, tiledBiasConst,
+ cast<ShapedType>(op->getResult(0).getType()).getElementType());
+ if (failed(maybeConv))
+ return rewriter.notifyMatchFailure(
+ op, "fail to create transpose-conv decomposition conv2d");
+
+ Value multiplier = rescaleOp.getMultiplier();
+ Value shift = rescaleOp.getShift();
+ if (rescaleOp.getPerChannel()) {
+ DenseElementsAttr multAttr;
+ DenseElementsAttr shiftAttr;
+ if (!matchPattern(multiplier, m_Constant(&multAttr)) ||
+ !matchPattern(shift, m_Constant(&shiftAttr)))
+ return rewriter.notifyMatchFailure(
+ rescaleOp,
+ "per-channel rescale requires static multiplier/shift constants");
+
+ auto multVals = llvm::to_vector(multAttr.getValues<Attribute>());
+ auto shiftVals = llvm::to_vector(shiftAttr.getValues<Attribute>());
+ if (static_cast<int64_t>(multVals.size()) != p.outputChannels ||
+ static_cast<int64_t>(shiftVals.size()) != p.outputChannels)
+ return rewriter.notifyMatchFailure(
+ rescaleOp, "rescale constant size mismatch with OC");
+
+ SmallVector<Attribute> tiledMult(newOC);
+ SmallVector<Attribute> tiledShift(newOC);
+ for (int64_t s0 = 0; s0 < p.stride[0]; ++s0)
+ for (int64_t s1 = 0; s1 < p.stride[1]; ++s1)
+ for (int64_t oc = 0; oc < p.outputChannels; ++oc) {
+ int64_t ix = s0 * p.stride[1] * p.outputChannels +
+ s1 * p.outputChannels + oc;
+ tiledMult[ix] = multVals[oc];
+ tiledShift[ix] = shiftVals[oc];
+ }
+
+ auto multETy = cast<ShapedType>(multiplier.getType()).getElementType();
+ auto shiftETy = cast<ShapedType>(shift.getType()).getElementType();
+ auto tiledMultTy = RankedTensorType::get({newOC}, multETy);
+ auto tiledShiftTy = RankedTensorType::get({newOC}, shiftETy);
+
+ multiplier = tosa::ConstOp::create(
+ rewriter, p.loc, tiledMultTy,
+ DenseElementsAttr::get(tiledMultTy,
+ ArrayRef<Attribute>(tiledMult)))
+ .getResult();
+ shift = tosa::ConstOp::create(
+ rewriter, p.loc, tiledShiftTy,
+ DenseElementsAttr::get(tiledShiftTy,
+ ArrayRef<Attribute>(tiledShift)))
+ .getResult();
+ }
+
+ Type rescaleOutETy =
+ cast<ShapedType>(rescaleOp.getResult().getType()).getElementType();
+ Value rescaled =
+ CreateOpAndInferShape<tosa::RescaleOp>(
+ rewriter, p.loc, UnrankedTensorType::get(rescaleOutETy), *maybeConv,
+ multiplier, shift, rescaleOp.getInputZp(), rescaleOp.getOutputZp(),
+ rescaleOp.getScale32Attr(), rescaleOp.getRoundingModeAttr(),
+ rescaleOp.getPerChannelAttr(), rescaleOp.getInputUnsignedAttr(),
+ rescaleOp.getOutputUnsignedAttr())
+ .getResult();
+
+ ShapedType rescaleTy = cast<ShapedType>(rescaleOp.getResult().getType());
+ Value resultPad = shuffleSliceAndPadResult(
+ rewriter, p.loc, rescaled, rescaleOutETy, rescaleTy, p.batch,
+ p.outputChannels, p.stride, p.pad);
+
+ rewriter.replaceOp(rescaleOp, resultPad);
+ if (op->use_empty())
+ rewriter.eraseOp(op);
+ return success();
+ }
+};
+
class TransposeConvNonStridedConverter
: public OpRewritePattern<tosa::TransposeConv2DOp> {
public:
@@ -92,250 +481,37 @@ class TransposeConvStridedConverter
using OpRewritePattern<tosa::TransposeConv2DOp>::OpRewritePattern;
LogicalResult matchAndRewrite(tosa::TransposeConv2DOp op,
PatternRewriter &rewriter) const final {
- Location loc = op->getLoc();
- Value input = op->getOperand(0);
- Value weight = op->getOperand(1);
- Value bias = op->getOperand(2);
-
- ShapedType inputTy = cast<ShapedType>(input.getType());
- ShapedType weightTy = cast<ShapedType>(weight.getType());
- ShapedType biasTy = cast<ShapedType>(bias.getType());
- ShapedType resultTy = cast<ShapedType>(op->getResult(0).getType());
-
- Type inputETy = inputTy.getElementType();
- Type weightETy = weightTy.getElementType();
- Type biasETy = biasTy.getElementType();
- Type resultETy = resultTy.getElementType();
-
- llvm::ArrayRef<int64_t> pad = op.getOutPad();
- llvm::ArrayRef<int64_t> stride = op.getStride();
-
- // If striding is all 1 we can modify padding and reverse the kernel along
- // the x/y direction to make it a regular convolution. This is much simpler
- // then handling striding....
-
- // If strides are all 1 we dont need to use this one.
- if (llvm::all_of(stride, [](int64_t v) { return v == 1; }))
- return rewriter.notifyMatchFailure(op, "non-one stride found.");
-
- // Any dimensions other than batchSize cannot be dynamic for input/output
- for (unsigned int i = 1; i < 4; ++i) {
- if (inputTy.isDynamicDim(i) || resultTy.isDynamicDim(i))
- return failure();
- }
-
- if (!weightTy.hasStaticShape() || !biasTy.hasStaticShape())
+ StridedTransposeConvPrep p(op.getLoc());
+ if (failed(prepareStridedTransposeConv(op, rewriter, p)))
return failure();
- int64_t batch = inputTy.getDimSize(0);
-
- int64_t outputChannels = weightTy.getDimSize(0);
- int64_t weightHeight = weightTy.getDimSize(1);
- int64_t weightWidth = weightTy.getDimSize(2);
- int64_t inputChannels = weightTy.getDimSize(3);
-
- // Pad the weight so that it is modulo of the striding.
- llvm::SmallVector<int64_t, 8> weightPadding = {0, 0, 0, 0, 0, 0, 0, 0};
- weightPadding[3] =
- (weightHeight % stride[0]) ? (stride[0] - weightHeight % stride[0]) : 0;
- weightPadding[5] =
- (weightWidth % stride[1]) ? (stride[1] - weightWidth % stride[1]) : 0;
-
- Value weightPaddingVal =
- getTosaConstShape(rewriter, op->getLoc(), weightPadding);
-
- // Get and verify zero points.
- FailureOr<int64_t> maybeIZp = op.getInputZeroPoint();
- if (failed(maybeIZp))
- return rewriter.notifyMatchFailure(
- op, "input zero point cannot be statically determined");
-
- FailureOr<int64_t> maybeWZp = op.getWeightZeroPoint();
- if (failed(maybeWZp))
- return rewriter.notifyMatchFailure(
- op, "weight zero point cannot be statically determined");
-
- int64_t inputZpVal = *maybeIZp;
- int64_t weightZpVal = *maybeWZp;
-
- if (op.verifyInputZeroPoint(inputZpVal).failed())
- return rewriter.notifyMatchFailure(
- op, "input zero point must be zero for non-int8 integer types");
-
- if (op.verifyWeightZeroPoint(weightZpVal).failed())
- return rewriter.notifyMatchFailure(
- op, "weight zero point must be zero for non-int8 integer types");
-
- // construct pad_const values from zp values
- ImplicitLocOpBuilder builder(op->getLoc(), rewriter);
- const Value inputPadConst =
- createPadConstTensor(builder, op->getLoc(), input, inputZpVal);
- const Value weightPadConst =
- createPadConstTensor(builder, op->getLoc(), input, weightZpVal);
-
- weight = CreateOpAndInferShape<tosa::PadOp>(
- rewriter, loc, UnrankedTensorType::get(weightETy), weight,
- weightPaddingVal, weightPadConst);
-
- weightTy = cast<ShapedType>(weight.getType());
- weightHeight = weightTy.getDimSize(1);
- weightWidth = weightTy.getDimSize(2);
-
- // Split out the width / height by the stride dimensions.
- llvm::SmallVector<int64_t, 6> weightReshapeDims0 = {
- outputChannels, weightHeight / stride[0],
- stride[0], weightWidth / stride[1],
- stride[1], inputChannels};
-
- weight = CreateOpAndInferShape<tosa::ReshapeOp>(
- builder, UnrankedTensorType::get(weightETy), weight,
- getTosaConstShape(rewriter, loc, weightReshapeDims0));
-
- // Transpose the factored-out stride to the output channels.
- weight = CreateOpAndInferShape<tosa::TransposeOp>(
- rewriter, loc, UnrankedTensorType::get(weightETy), weight,
- rewriter.getDenseI32ArrayAttr({2, 4, 0, 1, 3, 5}));
-
- // Collapse the strides and output channels into a single dimension.
- llvm::SmallVector<int64_t, 4> weightReshapeDims1 = {
- outputChannels * stride[0] * stride[1], weightHeight / stride[0],
- weightWidth / stride[1], inputChannels};
-
- weight = CreateOpAndInferShape<tosa::ReshapeOp>(
- rewriter, loc, UnrankedTensorType::get(weightETy), weight,
- getTosaConstShape(rewriter, loc, weightReshapeDims1));
- ShapedType restridedWeightTy = cast<ShapedType>(weight.getType());
-
- weight = CreateOpAndInferShape<tosa::ReverseOp>(
- rewriter, loc, UnrankedTensorType::get(weightETy), weight,
- /* axis = */ rewriter.getI32IntegerAttr(1));
- weight = CreateOpAndInferShape<tosa::ReverseOp>(
- rewriter, loc, UnrankedTensorType::get(weightETy), weight,
- /* axis = */ rewriter.getI32IntegerAttr(2));
-
- // We need to pad the input far enough that we can pull all values.
- llvm::SmallVector<int64_t, 8> inputPadding = {0, 0, 0, 0, 0, 0, 0, 0};
- inputPadding[2] += restridedWeightTy.getDimSize(1) - 1;
- inputPadding[3] += restridedWeightTy.getDimSize(1) - 1;
- inputPadding[4] += restridedWeightTy.getDimSize(2) - 1;
- inputPadding[5] += restridedWeightTy.getDimSize(2) - 1;
-
- Value inputPaddingVal =
- getTosaConstShape(rewriter, op->getLoc(), inputPadding);
-
- input = CreateOpAndInferShape<tosa::PadOp>(
- rewriter, loc, UnrankedTensorType::get(inputETy), input,
- inputPaddingVal, inputPadConst);
-
// We use a zero bias as we need to broadcast the bias.
auto zeroBias = tosa::ConstOp::create(
- rewriter, loc,
- RankedTensorType::get({outputChannels * stride[0] * stride[1]},
- biasETy),
+ rewriter, p.loc,
+ RankedTensorType::get({p.outputChannels * p.stride[0] * p.stride[1]},
+ p.biasETy),
DenseElementsAttr::get(
- RankedTensorType::get({outputChannels * stride[0] * stride[1]},
- biasETy),
- rewriter.getZeroAttr(biasETy)));
-
- auto inputZp =
- createZeroPointTensor(rewriter, loc, input.getType(), inputZpVal);
- auto weightZp =
- createZeroPointTensor(rewriter, loc, weight.getType(), weightZpVal);
-
- if (!inputZp.has_value() || !weightZp.has_value()) {
+ RankedTensorType::get(
+ {p.outputChannels * p.stride[0] * p.stride[1]}, p.biasETy),
+ rewriter.getZeroAttr(p.biasETy)));
+
+ FailureOr<Value> maybeConv = buildConv2DFromPreparedInput(
+ op, rewriter, p, zeroBias,
+ cast<ShapedType>(op->getResult(0).getType()).getElementType());
+ if (failed(maybeConv))
return rewriter.notifyMatchFailure(
op, "fail to create a const zero point tensor");
- }
- // Perform the convolution using the zero bias.
- Value conv2d = CreateOpAndInferShape<tosa::Conv2DOp>(
- rewriter, loc, UnrankedTensorType::get(resultETy), input,
- weight, zeroBias, inputZp.value(), weightZp.value(),
- /*pad=*/rewriter.getDenseI64ArrayAttr({0, 0, 0, 0}),
- /*stride=*/rewriter.getDenseI64ArrayAttr({1, 1}),
- /*dilation=*/rewriter.getDenseI64ArrayAttr({1, 1}),
- /* acc_type = */ op.getAccType())
- .getResult();
+ Value resultPad = shuffleSliceAndPadResult(
+ rewriter, p.loc, *maybeConv, p.resultETy, p.resultTy, p.batch,
+ p.outputChannels, p.stride, p.pad);
- // Factor the resulting width / height.
- ShapedType convTy = cast<ShapedType>(conv2d.getType());
- Type convETy = convTy.getElementType();
-
- int64_t convHeight = convTy.getDimSize(1);
- int64_t convWidth = convTy.getDimSize(2);
-
- // Factor striding out of the convolution result.
- llvm::SmallVector<int64_t, 6> convReshapeDims0 = {
- batch, convHeight, convWidth, stride[0], stride[1], outputChannels};
-
- auto convReshapeDims0Value =
- getTosaConstShape(rewriter, loc, convReshapeDims0);
-
- conv2d = CreateOpAndInferShape<tosa::ReshapeOp>(
- rewriter, loc, UnrankedTensorType::get(resultETy), conv2d,
- convReshapeDims0Value);
-
- // Transpose the factored-out stride to the output channels.
- conv2d = CreateOpAndInferShape<tosa::TransposeOp>(
- rewriter, loc, UnrankedTensorType::get(convETy), conv2d,
- rewriter.getDenseI32ArrayAttr({0, 1, 3, 2, 4, 5}));
-
- // Fuse striding behavior back into width / height.
- llvm::SmallVector<int64_t, 6> convReshapeDims1 = {
- batch, convHeight * stride[0], convWidth * stride[1], outputChannels};
-
- auto convReshapeDims1Value =
- getTosaConstShape(rewriter, loc, convReshapeDims1);
-
- conv2d = CreateOpAndInferShape<tosa::ReshapeOp>(
- rewriter, loc, UnrankedTensorType::get(resultETy), conv2d,
- convReshapeDims1Value);
-
- // Determine the amount to slice / pad from the result start.
- int64_t resultSliceTop = std::max<int64_t>(0, -pad[0]);
- int64_t resultSliceLeft = std::max<int64_t>(0, -pad[2]);
- int64_t resultPadTop = std::max<int64_t>(0, pad[0]);
- int64_t resultPadLeft = std::max<int64_t>(0, pad[2]);
-
- // Try to slice the targetted result size, cap to the convolutions width.
- int64_t resultSliceHeight =
- std::min<int64_t>(convReshapeDims1[1] - resultSliceTop,
- resultTy.getDimSize(1) - resultPadTop);
- int64_t resultSliceWidth =
- std::min<int64_t>(convReshapeDims1[2] - resultSliceLeft,
- resultTy.getDimSize(2) - resultPadLeft);
-
- llvm::SmallVector<int64_t, 4> sliceBegin = {0, resultSliceTop,
- resultSliceLeft, 0};
- llvm::SmallVector<int64_t, 4> sliceSize(convReshapeDims1.begin(),
- convReshapeDims1.end());
- sliceSize[1] = resultSliceHeight;
- sliceSize[2] = resultSliceWidth;
-
- auto slice = CreateOpAndInferShape<tosa::SliceOp>(
- rewriter, loc, UnrankedTensorType::get(resultETy), conv2d,
- getTosaConstShape(rewriter, loc, sliceBegin),
- getTosaConstShape(rewriter, loc, sliceSize))
- .getResult();
-
- llvm::SmallVector<int64_t, 8> resultPadding = {0, 0, 0, 0, 0, 0, 0, 0};
- resultPadding[2] = resultPadTop;
- resultPadding[3] = resultTy.getDimSize(1) - resultPadTop - sliceSize[1];
- resultPadding[4] = resultPadLeft;
- resultPadding[5] = resultTy.getDimSize(2) - resultPadLeft - sliceSize[2];
-
- Value resultPaddingVal =
- getTosaConstShape(rewriter, op->getLoc(), resultPadding);
-
- Value resultPad = CreateOpAndInferShape<tosa::PadOp>(
- rewriter, loc, UnrankedTensorType::get(resultETy), slice,
- resultPaddingVal);
-
- if (EqualizeRanks(rewriter, op.getLoc(), resultPad, bias).failed()) {
+ if (EqualizeRanks(rewriter, op.getLoc(), resultPad, p.bias).failed()) {
return failure();
}
- rewriter.replaceOpWithNewOp<tosa::AddOp>(op, op.getType(), resultPad, bias);
+ rewriter.replaceOpWithNewOp<tosa::AddOp>(op, op.getType(), resultPad,
+ p.bias);
return success();
}
};
@@ -344,6 +520,13 @@ class TransposeConvStridedConverter
void mlir::tosa::populateTosaDecomposeTransposeConv(
MLIRContext *ctx, RewritePatternSet &patterns) {
+ // Preferred strided transpose-conv lowering with quantization:
+ // tosa.transpose_conv2d + tosa.rescale
+ // -> conv2d + (bias handling) + rescale + reshape/transpose/slice
+ // We keep rescale close to conv2d so later tensor reordering works on i8/i16
+ // instead of wide accumulator types. This reduces memory traffic on reshape &
+ // transpose
+ patterns.add<TransposeConvRescaleDecompose>(ctx);
patterns.add<TransposeConvNonStridedConverter>(ctx);
patterns.add<TransposeConvStridedConverter>(ctx);
}
>From 8def3b80bedec51c04129689e08275d8a96054fe Mon Sep 17 00:00:00 2001
From: liveshkumar-synaptics <lravi at synaptics.com>
Date: Tue, 14 Jul 2026 16:07:48 +0530
Subject: [PATCH 09/10] [mlir][tosa] Support scalar bias tensors in strided
TransposeConv2D + Rescale decomposition (#14)
* This commot extends `TransposeConvRescaleDecompose` to handle broadcast-form bias tensors (shape [1]),
which are common in quantized models.
* Previously, the pattern only accepted per-channel bias matching the output channel count.
* Now scalar bias values are correctly tiled across all stride-expanded channels during decomposition, enabling
patterns like tensor<1xi32> bias with tensor<4xi32> output channels to work correctly.
Explanation of the fix:
* Problem: Bias shape [1] with output channels 4 was rejected.
* Solution: The code now checks if bias is either:
* Scalar: size == 1 (broadcast to all channels)
* Per-channel: [size == OC (one value per channel)
---
.../Transforms/TosaDecomposeTransposeConv.cpp | 29 +++++++++++++------
1 file changed, 20 insertions(+), 9 deletions(-)
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaDecomposeTransposeConv.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaDecomposeTransposeConv.cpp
index 90a4442c2565e..ee4e22acbed35 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaDecomposeTransposeConv.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaDecomposeTransposeConv.cpp
@@ -320,15 +320,20 @@ class TransposeConvRescaleDecompose : public OpRewritePattern<tosa::RescaleOp> {
return rewriter.notifyMatchFailure(op, "bias must be a static constant");
auto biasVals = llvm::to_vector(biasAttr.getValues<Attribute>());
- if (static_cast<int64_t>(biasVals.size()) != p.outputChannels)
- return rewriter.notifyMatchFailure(op, "bias size mismatch with OC");
+ // Handle both scalar bias (shape [1]) and per-channel bias (shape [OC])
+ if (!(biasVals.size() == 1 ||
+ static_cast<int64_t>(biasVals.size()) == p.outputChannels))
+ return rewriter.notifyMatchFailure(
+ op, "bias must be scalar [1] or per-channel [OC]");
SmallVector<Attribute> tiledBias(newOC);
for (int64_t s0 = 0; s0 < p.stride[0]; ++s0)
for (int64_t s1 = 0; s1 < p.stride[1]; ++s1)
- for (int64_t oc = 0; oc < p.outputChannels; ++oc)
+ for (int64_t oc = 0; oc < p.outputChannels; ++oc) {
+ int64_t biasIdx = (biasVals.size() == 1) ? 0 : oc;
tiledBias[s0 * p.stride[1] * p.outputChannels +
- s1 * p.outputChannels + oc] = biasVals[oc];
+ s1 * p.outputChannels + oc] = biasVals[biasIdx];
+ }
auto tiledBiasTy = RankedTensorType::get({newOC}, p.biasETy);
Value tiledBiasConst =
@@ -357,10 +362,14 @@ class TransposeConvRescaleDecompose : public OpRewritePattern<tosa::RescaleOp> {
auto multVals = llvm::to_vector(multAttr.getValues<Attribute>());
auto shiftVals = llvm::to_vector(shiftAttr.getValues<Attribute>());
- if (static_cast<int64_t>(multVals.size()) != p.outputChannels ||
- static_cast<int64_t>(shiftVals.size()) != p.outputChannels)
+ // Allow both scalar and per-channel multiplier/shift
+ if (!((multVals.size() == 1 ||
+ static_cast<int64_t>(multVals.size()) == p.outputChannels) &&
+ (shiftVals.size() == 1 ||
+ static_cast<int64_t>(shiftVals.size()) == p.outputChannels)))
return rewriter.notifyMatchFailure(
- rescaleOp, "rescale constant size mismatch with OC");
+ rescaleOp,
+ "rescale multiplier/shift must be scalar [1] or per-channel [OC]");
SmallVector<Attribute> tiledMult(newOC);
SmallVector<Attribute> tiledShift(newOC);
@@ -369,8 +378,10 @@ class TransposeConvRescaleDecompose : public OpRewritePattern<tosa::RescaleOp> {
for (int64_t oc = 0; oc < p.outputChannels; ++oc) {
int64_t ix = s0 * p.stride[1] * p.outputChannels +
s1 * p.outputChannels + oc;
- tiledMult[ix] = multVals[oc];
- tiledShift[ix] = shiftVals[oc];
+ int64_t multIdx = (multVals.size() == 1) ? 0 : oc;
+ int64_t shiftIdx = (shiftVals.size() == 1) ? 0 : oc;
+ tiledMult[ix] = multVals[multIdx];
+ tiledShift[ix] = shiftVals[shiftIdx];
}
auto multETy = cast<ShapedType>(multiplier.getType()).getElementType();
>From c37ee8a92adf57c7a48a309a84c7fe24b7910968 Mon Sep 17 00:00:00 2001
From: Atabak Pouya <atabak.pouya at synaptics.com>
Date: Mon, 27 Jul 2026 01:11:40 -0700
Subject: [PATCH 10/10] [mlir][tosa] Limit shape op level checks to
MAX_SHAPE_LEN Cherry-pick of llvm/llvm-project at bc010989719d, adapted to the
shape ops present in this tree.
---
mlir/include/mlir/Dialect/Tosa/IR/TargetEnv.h | 9 ++--
.../Tosa/Transforms/TosaValidation.cpp | 52 +++++++++++++------
mlir/test/Dialect/Tosa/level_check.mlir | 26 +++++-----
.../Dialect/Tosa/tosa-validation-valid.mlir | 10 ++++
4 files changed, 65 insertions(+), 32 deletions(-)
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TargetEnv.h b/mlir/include/mlir/Dialect/Tosa/IR/TargetEnv.h
index e088eb31338dc..b80232f112b64 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TargetEnv.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TargetEnv.h
@@ -28,19 +28,22 @@ struct TosaLevel {
int32_t MAX_LOG2_SIZE = 0;
int32_t MAX_NESTING = 0;
int32_t MAX_TENSOR_LIST_SIZE = 0;
+ int32_t MAX_SHAPE_LEN = 0;
bool operator==(const TosaLevel &rhs) {
return MAX_RANK == rhs.MAX_RANK && MAX_KERNEL == rhs.MAX_KERNEL &&
MAX_STRIDE == rhs.MAX_STRIDE && MAX_SCALE == rhs.MAX_SCALE &&
MAX_LOG2_SIZE == rhs.MAX_LOG2_SIZE &&
MAX_NESTING == rhs.MAX_NESTING &&
- MAX_TENSOR_LIST_SIZE == rhs.MAX_TENSOR_LIST_SIZE;
+ MAX_TENSOR_LIST_SIZE == rhs.MAX_TENSOR_LIST_SIZE &&
+ MAX_SHAPE_LEN == rhs.MAX_SHAPE_LEN;
}
};
-static constexpr TosaLevel TOSA_LEVEL_EIGHTK = {6, 8192, 8192, 256, 31, 6, 64};
+static constexpr TosaLevel TOSA_LEVEL_EIGHTK = {6, 8192, 8192, 256,
+ 31, 6, 64, 16};
static constexpr TosaLevel TOSA_LEVEL_NONE = {32, 2147483647, 2147483647, 2048,
- 63, 256, 256};
+ 63, 256, 256, 64};
TargetEnvAttr lookupTargetEnv(Operation *op);
TargetEnvAttr getDefaultTargetEnv(MLIRContext *context);
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
index 897cc87529eca..dfe5ed05cfe14 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
@@ -218,12 +218,6 @@ struct TosaValidation : public tosa::impl::TosaValidationBase<TosaValidation> {
if (type.getRank() > highest_rank)
return op->emitOpError() << "failed level check: " << operandOrResult
<< " rank(shape) <= MAX_RANK";
- } else if (tosa::shapeType shapeType =
- dyn_cast<tosa::shapeType>(typeToCheck)) {
- if (shapeType.getRank() > highest_rank)
- return op->emitOpError()
- << "failed shape type level check: " << typeToCheck
- << " exceeds MAX_RANK";
}
return success();
}
@@ -245,6 +239,18 @@ struct TosaValidation : public tosa::impl::TosaValidationBase<TosaValidation> {
return levelCheckSize(op, v.getType(), operandOrResult);
}
+ // Perform the Level shape length check on a value.
+ LogicalResult levelCheckShapeLength(Operation *op, const Type typeToCheck,
+ const StringRef operandOrResult) {
+ if (tosa::shapeType shapeType = dyn_cast<tosa::shapeType>(typeToCheck)) {
+ if (shapeType.getRank() > targetEnv.getLevel().MAX_SHAPE_LEN)
+ return op->emitOpError()
+ << "failed shape type level check: " << typeToCheck
+ << " exceeds MAX_SHAPE_LEN";
+ }
+ return success();
+ }
+
// Level check sizes of all operands and results of the operation.
template <typename T>
LogicalResult levelCheckSizes(T tosaOp) {
@@ -278,6 +284,20 @@ struct TosaValidation : public tosa::impl::TosaValidationBase<TosaValidation> {
return success();
}
+ // Level check shape lengths of all operands and results of an operation that
+ // are tosa.shape type.
+ template <typename T>
+ LogicalResult levelCheckShapeLengths(T tosaOp) {
+ for (const auto &v : tosaOp->getOperands()) {
+ if (failed(levelCheckShapeLength(tosaOp, v.getType(), "operand")))
+ return failure();
+ }
+ if (failed(levelCheckShapeLength(tosaOp, tosaOp.getResult().getType(),
+ "result")))
+ return failure();
+ return success();
+ }
+
// Level check ranks and sizes.
LogicalResult levelCheckRanksAndSizes(Operation *op);
@@ -579,9 +599,9 @@ LogicalResult TosaValidation::levelCheckRanksAndSizes(Operation *op) {
return failure(); \
}
-#define CHECK_RANKS(tosaOp) \
+#define CHECK_SHAPE_LEN(tosaOp) \
if (isa<tosa::tosaOp##Op>(op)) { \
- if (failed(levelCheckRanks(cast<tosa::tosaOp##Op>(op)))) \
+ if (failed(levelCheckShapeLengths(cast<tosa::tosaOp##Op>(op)))) \
return failure(); \
}
@@ -688,19 +708,19 @@ LogicalResult TosaValidation::levelCheckRanksAndSizes(Operation *op) {
// Shape Operators
CHECK_SIZES(ConstShape);
- // For the following operations, check whether the rank of each operand
- // is valid given a level.
+ // For the following operations, check whether the shape length of each
+ // operand is valid given a level.
// Shape Operators
- CHECK_RANKS(AddShape);
- CHECK_RANKS(DivCeilShape);
- CHECK_RANKS(DivFloorShape);
- CHECK_RANKS(MulShape);
- CHECK_RANKS(SubShape);
+ CHECK_SHAPE_LEN(AddShape);
+ CHECK_SHAPE_LEN(DivCeilShape);
+ CHECK_SHAPE_LEN(DivFloorShape);
+ CHECK_SHAPE_LEN(MulShape);
+ CHECK_SHAPE_LEN(SubShape);
#undef CHECK_RANKS_AND_SIZES
#undef CHECK_SIZES
-#undef CHECK_RANKS
+#undef CHECK_SHAPE_LEN
return success();
}
diff --git a/mlir/test/Dialect/Tosa/level_check.mlir b/mlir/test/Dialect/Tosa/level_check.mlir
index da21c18e19783..f5caaa7d4fdd6 100644
--- a/mlir/test/Dialect/Tosa/level_check.mlir
+++ b/mlir/test/Dialect/Tosa/level_check.mlir
@@ -390,7 +390,7 @@ func.func @test_pad_rank_invalid(%arg0: tensor<1x1x1x1x13x21x3xf32>) -> tensor<1
func.func @test_reshape_rank_invalid(%arg0: tensor<13x21x3xf32>) -> tensor<1x1x1x1x1x1x819xf32> {
%1 = tosa.const_shape {values = dense<[1, 1, 1, 1, 1, 1, 819]> : tensor<7xindex>} : () -> !tosa.shape<7>
- // expected-error at +1 {{'tosa.reshape' op failed shape type level check: '!tosa.shape<7>' exceeds MAX_RANK}}
+ // expected-error at +1 {{'tosa.reshape' op failed level check: result rank(shape) <= MAX_RANK}}
%0 = "tosa.reshape"(%arg0, %1) : (tensor<13x21x3xf32>, !tosa.shape<7>) -> tensor<1x1x1x1x1x1x819xf32>
return %0 : tensor<1x1x1x1x1x1x819xf32>
}
@@ -1665,22 +1665,22 @@ func.func @test_cast_to_block_scaled_invalid_rank(%arg0: tensor<1x2x3x4x5x6x7x32
// -----
-func.func @test_add_shape_invalid_rank() -> !tosa.shape<13> {
- %a = tosa.const_shape {values = dense<[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]> : tensor<13xindex>} : () -> !tosa.shape<13>
- %b = tosa.const_shape {values = dense<[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]> : tensor<13xindex>} : () -> !tosa.shape<13>
- // expected-error at +1 {{'tosa.add_shape' op failed shape type level check: '!tosa.shape<13>' exceeds MAX_RANK}}
- %c = tosa.add_shape %a, %b : (!tosa.shape<13>, !tosa.shape<13>) -> !tosa.shape<13>
- return %c : !tosa.shape<13>
+func.func @test_add_shape_invalid_rank() -> !tosa.shape<17> {
+ %a = tosa.const_shape {values = dense<0> : tensor<17xindex>} : () -> !tosa.shape<17>
+ %b = tosa.const_shape {values = dense<0> : tensor<17xindex>} : () -> !tosa.shape<17>
+ // expected-error at +1 {{'tosa.add_shape' op failed shape type level check: '!tosa.shape<17>' exceeds MAX_SHAPE_LEN}}
+ %c = tosa.add_shape %a, %b : (!tosa.shape<17>, !tosa.shape<17>) -> !tosa.shape<17>
+ return %c : !tosa.shape<17>
}
// -----
-func.func @test_div_floor_shape_invalid_rank() -> !tosa.shape<7> {
- %a = tosa.const_shape {values = dense<[1, 2, 3, 4, 5, 6, 7]> : tensor<7xindex>} : () -> !tosa.shape<7>
- %b = tosa.const_shape {values = dense<[1, 2, 3, 4, 5, 6, 7]> : tensor<7xindex>} : () -> !tosa.shape<7>
- // expected-error at +1 {{'tosa.div_floor_shape' op failed shape type level check: '!tosa.shape<7>' exceeds MAX_RANK}}
- %c = tosa.div_floor_shape %a, %b : (!tosa.shape<7>, !tosa.shape<7>) -> !tosa.shape<7>
- return %c : !tosa.shape<7>
+func.func @test_div_floor_shape_invalid_rank() -> !tosa.shape<17> {
+ %a = tosa.const_shape {values = dense<0> : tensor<17xindex>} : () -> !tosa.shape<17>
+ %b = tosa.const_shape {values = dense<0> : tensor<17xindex>} : () -> !tosa.shape<17>
+ // expected-error at +1 {{'tosa.div_floor_shape' op failed shape type level check: '!tosa.shape<17>' exceeds MAX_SHAPE_LEN}}
+ %c = tosa.div_floor_shape %a, %b : (!tosa.shape<17>, !tosa.shape<17>) -> !tosa.shape<17>
+ return %c : !tosa.shape<17>
}
// -----
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-valid.mlir
index 663159e75d1a6..a6f18e765d57c 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-valid.mlir
@@ -29,3 +29,13 @@ func.func @test_rescale_output_unsigned(%arg0: tensor<1x1xi8>) -> (tensor<1x1xui
%r = tosa.rescale %arg0, %1, %0, %3, %2 {input_unsigned = false, output_unsigned = true, per_channel = false, rounding_mode = SINGLE_ROUND, scale32 = true} : (tensor<1x1xi8>, tensor<1xi32>, tensor<1xi8>, tensor<1xi8>, tensor<1xi8>) -> tensor<1x1xui8>
return %r : tensor<1x1xui8>
}
+
+// -----
+
+// CHECK-LABEL: test_pad_large_input_rank
+func.func @test_pad_large_input_rank(%arg0: tensor<13x21x3x1x1x1xf32>) -> tensor<13x21x3x1x1x1xf32> {
+ %0 = "tosa.const"() {values = dense<3.14> : tensor<1xf32>} : () -> tensor<1xf32>
+ %padding = tosa.const_shape {values = dense<0> : tensor<12xindex>} : () -> !tosa.shape<12>
+ %1 = tosa.pad %arg0, %padding, %0 : (tensor<13x21x3x1x1x1xf32>, !tosa.shape<12>, tensor<1xf32>) -> tensor<13x21x3x1x1x1xf32>
+ return %1 : tensor<13x21x3x1x1x1xf32>
+}
More information about the Mlir-commits
mailing list