[llvm-branch-commits] [llvm] [CodeGen] Bound region spill costs with measured wave counts (PR #236843)
Yaxun Liu via llvm-branch-commits
llvm-branch-commits at lists.llvm.org
Sun Nov 1 18:47:53 PST 2026
https://github.com/yxsamliu updated https://github.com/llvm/llvm-project/pull/236843
>From 91f2f6e803d40e2f29cc04c11f9d99be11c1b12b Mon Sep 17 00:00:00 2001
From: "Yaxun (Sam) Liu" <yaxun.liu at amd.com>
Date: Thu, 29 Oct 2026 13:56:55 +0000
Subject: [PATCH] [CodeGen] Bound region spill costs with measured wave counts
Greedy register allocation can keep a region that costs more than
splitting at each use block when it also finds a compact region. This
occurs in profiled sparse matrix multiplication, where extra spills in a
hot loop make the selected region expensive.
Use measured wave counts to reject such a choice when its modeled spill
cost is provably higher. Ordinary block frequencies and the positive
floor used by spill placement do not provide that proof.
Compare the two choices in original wave-count units, cancelling costs
in the same block first. For example, 1,000 extra measured spill
executions outweigh savings at two unmeasured acyclic blocks in a
function entered by 100 waves: each of those blocks can save at most 100
executions. An unmeasured block in a cycle has no finite upper bound.
Keep the existing choice when the counts are insufficient or the
arithmetic overflows.
Apply the comparison only when the selected physical region covers every
compact-region bundle, so its cost accounts for the complete proposed
split. Reuse the existing constraint and split-cost calculations to
build the difference without changing the candidate's spill-placement
solution.
The hidden option -enable-wave-profiled-region-cost=false disables this
comparison independently of -enable-wave-profiled-spill. With Clang,
pass these options through -mllvm.
---
llvm/include/llvm/CodeGen/SpillPlacement.h | 19 +-
llvm/lib/CodeGen/RegAllocGreedy.cpp | 119 +++++++----
llvm/lib/CodeGen/RegAllocGreedy.h | 12 +-
llvm/lib/CodeGen/SpillPlacement.cpp | 71 ++++++-
.../AMDGPU/wave-profile-region-cost.mir | 197 +++++++++++++++++
llvm/unittests/CodeGen/CMakeLists.txt | 1 +
llvm/unittests/CodeGen/WaveRegionCostTest.cpp | 199 ++++++++++++++++++
7 files changed, 575 insertions(+), 43 deletions(-)
create mode 100644 llvm/test/CodeGen/AMDGPU/wave-profile-region-cost.mir
create mode 100644 llvm/unittests/CodeGen/WaveRegionCostTest.cpp
diff --git a/llvm/include/llvm/CodeGen/SpillPlacement.h b/llvm/include/llvm/CodeGen/SpillPlacement.h
index 5c7994513c3e7d..2769f8add79d49 100644
--- a/llvm/include/llvm/CodeGen/SpillPlacement.h
+++ b/llvm/include/llvm/CodeGen/SpillPlacement.h
@@ -27,15 +27,16 @@
#define LLVM_LIB_CODEGEN_SPILLPLACEMENT_H
#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/BitVector.h"
#include "llvm/ADT/SmallVector.h"
#include "llvm/ADT/SparseSet.h"
#include "llvm/CodeGen/MachineFunctionAnalysis.h"
#include "llvm/CodeGen/MachineFunctionPass.h"
#include "llvm/Support/BlockFrequency.h"
+#include <cstdint>
namespace llvm {
-class BitVector;
class EdgeBundles;
class MachineBlockFrequencyInfo;
class MachineFunction;
@@ -64,6 +65,11 @@ class SpillPlacement {
// Block frequencies are computed once. Indexed by block number.
SmallVector<BlockFrequency, 8> BlockFrequencies;
+ // Empty when no block has an accepted wave count.
+ BitVector MeasuredWaveBlocks;
+ SmallVector<uint64_t> MeasuredWaveCounts;
+ uint64_t NormalizationWaveCount = 0;
+ BitVector AcyclicBlocks;
/// Decision threshold. A node gets the output value 0 if the weighted sum of
/// its inputs falls in the open interval (-Threshold;Threshold).
@@ -154,6 +160,17 @@ class SpillPlacement {
return BlockFrequencies[Number];
}
+ /// Raw counts for the optional wave-guided spill-cost comparison. Their
+ /// availability is independent of the spill-placement frequency policy.
+ bool hasMeasuredWaveBlocks() const { return !MeasuredWaveBlocks.empty(); }
+
+ /// Bound a signed difference of spill executions using original wave counts.
+ /// Missing acyclic blocks lie in [0, entry count]; missing cyclic blocks have
+ /// no finite upper bound. Spill-placement floors and scaling are excluded.
+ /// This orders modeled spill executions in the training run, not a runtime
+ /// profitability guarantee. Return false if arithmetic is not representable.
+ LLVM_ABI bool isWaveCostDifferencePositive(ArrayRef<int64_t> Coefficients);
+
LLVM_ABI bool invalidate(MachineFunction &MF, const PreservedAnalyses &PA,
MachineFunctionAnalysisManager::Invalidator &Inv);
diff --git a/llvm/lib/CodeGen/RegAllocGreedy.cpp b/llvm/lib/CodeGen/RegAllocGreedy.cpp
index f7655e97b99427..3ad66b46ba7e4b 100644
--- a/llvm/lib/CodeGen/RegAllocGreedy.cpp
+++ b/llvm/lib/CodeGen/RegAllocGreedy.cpp
@@ -574,7 +574,7 @@ MCRegister RAGreedy::tryAssign(const LiveInterval &VirtReg,
}
// We can also split the virtual register in cold blocks.
- if (trySplitAroundHintReg(PhysHint, VirtReg, NewVRegs, Order))
+ if (trySplitAroundHintReg(PhysHint, VirtReg, NewVRegs))
return MCRegister();
// Record the missed hint, we may be able to recover
@@ -762,20 +762,18 @@ MCRegister RAGreedy::tryEvict(const LiveInterval &VirtReg,
// Region Splitting
//===----------------------------------------------------------------------===//
-/// addSplitConstraints - Fill out the SplitConstraints vector based on the
-/// interference pattern in Physreg and its aliases. Add the constraints to
-/// SpillPlacement and return the static cost of this split in Cost, assuming
-/// that all preferences in SplitConstraints are met.
-/// Return false if there are no bundles with positive bias, or if the static
-/// cost reaches MaxCost.
-bool RAGreedy::addSplitConstraints(InterferenceCache::Cursor Intf,
- BlockFrequency &Cost,
- BlockFrequency MaxCost) {
+/// Fill SplitConstraints from the interference pattern. When requested, compute
+/// the static cost and spill-count differences. Return false if a required
+/// spill cannot be inserted or the static cost reaches MaxCost.
+bool RAGreedy::calcSplitConstraints(InterferenceCache::Cursor Intf,
+ BlockFrequency *Cost,
+ BlockFrequency MaxCost,
+ SmallVectorImpl<int64_t> *CostDifference) {
ArrayRef<SplitAnalysis::BlockInfo> UseBlocks = SA->getUseBlocks();
// Reset interference dependent info.
SplitConstraints.resize(UseBlocks.size());
- BlockFrequency StaticCost = BlockFrequency(0);
+ BlockFrequency StaticCost(0);
for (unsigned I = 0; I != UseBlocks.size(); ++I) {
const SplitAnalysis::BlockInfo &BI = UseBlocks[I];
SpillPlacement::BlockConstraint &BC = SplitConstraints[I];
@@ -828,14 +826,26 @@ bool RAGreedy::addSplitConstraints(InterferenceCache::Cursor Intf,
}
}
+ if (CostDifference)
+ (*CostDifference)[BC.Number] += Ins;
// Accumulate the total frequency of inserted spill code.
- while (Ins--)
- StaticCost += SpillPlacer->getBlockFrequency(BC.Number);
-
- if (StaticCost >= MaxCost)
- return false;
+ if (Cost) {
+ while (Ins--)
+ StaticCost += SpillPlacer->getBlockFrequency(BC.Number);
+ if (StaticCost >= MaxCost)
+ return false;
+ }
}
- Cost = StaticCost;
+ if (Cost)
+ *Cost = StaticCost;
+ return true;
+}
+
+bool RAGreedy::addSplitConstraints(InterferenceCache::Cursor Intf,
+ BlockFrequency &Cost,
+ BlockFrequency MaxCost) {
+ if (!calcSplitConstraints(Intf, &Cost, MaxCost))
+ return false;
// Add constraints for use-blocks. Note that these are the only constraints
// that may add a positive bias, it is downhill from here.
@@ -1042,13 +1052,13 @@ BlockFrequency RAGreedy::calcBlockSplitCost() {
return Cost;
}
-/// calcGlobalSplitCost - Return the global split cost of following the split
-/// pattern in LiveBundles. This cost should be added to the local cost of the
-/// interference pattern in SplitConstraints.
+/// addGlobalSplitCost - Add the global split cost of following LiveBundles
+/// to Cost and spill counts to CostDifference, when requested. Cost should
+/// initially contain the local cost of the interference in SplitConstraints.
///
-BlockFrequency RAGreedy::calcGlobalSplitCost(GlobalSplitCandidate &Cand,
- const AllocationOrder &Order) {
- BlockFrequency GlobalCost = BlockFrequency(0);
+void RAGreedy::addGlobalSplitCost(GlobalSplitCandidate &Cand,
+ BlockFrequency *Cost,
+ SmallVectorImpl<int64_t> *CostDifference) {
const BitVector &LiveBundles = Cand.LiveBundles;
ArrayRef<SplitAnalysis::BlockInfo> UseBlocks = SA->getUseBlocks();
for (unsigned I = 0; I != UseBlocks.size(); ++I) {
@@ -1064,8 +1074,12 @@ BlockFrequency RAGreedy::calcGlobalSplitCost(GlobalSplitCandidate &Cand,
Ins += RegIn != (BC.Entry == SpillPlacement::PrefReg);
if (BI.LiveOut)
Ins += RegOut != (BC.Exit == SpillPlacement::PrefReg);
- while (Ins--)
- GlobalCost += SpillPlacer->getBlockFrequency(BC.Number);
+ if (CostDifference)
+ (*CostDifference)[BC.Number] += Ins;
+ if (Cost) {
+ while (Ins--)
+ *Cost += SpillPlacer->getBlockFrequency(BC.Number);
+ }
}
for (unsigned Number : Cand.ActiveBlocks) {
@@ -1077,15 +1091,21 @@ BlockFrequency RAGreedy::calcGlobalSplitCost(GlobalSplitCandidate &Cand,
// We need double spill code if this block has interference.
Cand.Intf.moveToBlock(Number);
if (Cand.Intf.hasInterference()) {
- GlobalCost += SpillPlacer->getBlockFrequency(Number);
- GlobalCost += SpillPlacer->getBlockFrequency(Number);
+ if (CostDifference)
+ (*CostDifference)[Number] += 2;
+ if (Cost) {
+ *Cost += SpillPlacer->getBlockFrequency(Number);
+ *Cost += SpillPlacer->getBlockFrequency(Number);
+ }
}
continue;
}
// live-in / stack-out or stack-in live-out.
- GlobalCost += SpillPlacer->getBlockFrequency(Number);
+ if (CostDifference)
+ ++(*CostDifference)[Number];
+ if (Cost)
+ *Cost += SpillPlacer->getBlockFrequency(Number);
}
- return GlobalCost;
}
/// splitAroundRegion - Split the current live range around the regions
@@ -1265,6 +1285,18 @@ MCRegister RAGreedy::tryRegionSplit(const LiveInterval &VirtReg,
unsigned BestCand = calculateRegionSplitCost(VirtReg, Order, BestCost,
NumCands, false /*IgnoreCSR*/);
+ if (HasCompact && BestCand != NoCand && BestCost >= SpillCost &&
+ SpillPlacer->hasMeasuredWaveBlocks()) {
+ // doRegionSplit also assigns uncovered compact bundles. The physical
+ // candidate's cost is sufficient only when that union adds no bundles.
+ const BitVector &Physical = GlobalCand[BestCand].LiveBundles;
+ bool ExtraCompact = llvm::any_of(GlobalCand.front().LiveBundles.set_bits(),
+ [&](int I) { return !Physical.test(I); });
+ if (!ExtraCompact && isWaveRegionCostHigher(GlobalCand[BestCand])) {
+ LLVM_DEBUG(dbgs() << "Wave cost exceeds block splitting\n");
+ return MCRegister::NoRegister;
+ }
+ }
// No solutions found, fall back to single block splitting.
if (!HasCompact && BestCand == NoCand)
return MCRegister::NoRegister;
@@ -1272,8 +1304,25 @@ MCRegister RAGreedy::tryRegionSplit(const LiveInterval &VirtReg,
return doRegionSplit(VirtReg, BestCand, HasCompact, NewVRegs);
}
+bool RAGreedy::isWaveRegionCostHigher(GlobalSplitCandidate &Cand) {
+ // Only the selected physical candidate needs a wave-cost comparison. Other
+ // candidates overwrite SplitConstraints, so reconstruct its constraints
+ // without changing the spill placement solution or live bundles.
+ SmallVector<int64_t> CostDifference(MF->getNumBlockIDs(), 0);
+ if (!calcSplitConstraints(Cand.Intf, nullptr, BlockFrequency::max(),
+ &CostDifference))
+ return false;
+ addGlobalSplitCost(Cand, nullptr, &CostDifference);
+
+ // Use-block entries are not unique by block; accumulate every term.
+ for (const SplitAnalysis::BlockInfo &BI : SA->getUseBlocks()) {
+ CostDifference[BI.MBB->getNumber()] -=
+ 1 + unsigned(BI.LiveIn && BI.LiveOut && BI.FirstDef);
+ }
+ return SpillPlacer->isWaveCostDifferencePositive(CostDifference);
+}
+
unsigned RAGreedy::calculateRegionSplitCostAroundReg(MCRegister PhysReg,
- AllocationOrder &Order,
BlockFrequency &BestCost,
unsigned &NumCands,
unsigned &BestCand) {
@@ -1334,7 +1383,7 @@ unsigned RAGreedy::calculateRegionSplitCostAroundReg(MCRegister PhysReg,
return BestCand;
}
- Cost += calcGlobalSplitCost(Cand, Order);
+ addGlobalSplitCost(Cand, &Cost);
LLVM_DEBUG({
dbgs() << ", total = " << printBlockFreq(*MBFI, Cost) << " with bundles";
for (int I : Cand.LiveBundles.set_bits())
@@ -1361,8 +1410,7 @@ unsigned RAGreedy::calculateRegionSplitCost(const LiveInterval &VirtReg,
if (IgnoreCSR && EvictAdvisor->getUnusedCalleeSavedReg(VirtReg, PhysReg))
continue;
- calculateRegionSplitCostAroundReg(PhysReg, Order, BestCost, NumCands,
- BestCand);
+ calculateRegionSplitCostAroundReg(PhysReg, BestCost, NumCands, BestCand);
}
return BestCand;
@@ -1412,8 +1460,7 @@ MCRegister RAGreedy::doRegionSplit(const LiveInterval &VirtReg,
// Hint if we can place new COPY instructions in cold blocks.
bool RAGreedy::trySplitAroundHintReg(MCRegister Hint,
const LiveInterval &VirtReg,
- SmallVectorImpl<Register> &NewVRegs,
- AllocationOrder &Order) {
+ SmallVectorImpl<Register> &NewVRegs) {
// Split the VirtReg may generate COPY instructions in multiple cold basic
// blocks, and increase code size. So we avoid it when the function is
// optimized for size.
@@ -1488,7 +1535,7 @@ bool RAGreedy::trySplitAroundHintReg(MCRegister Hint,
unsigned NumCands = 0;
unsigned BestCand = NoCand;
SA->analyze(&VirtReg);
- calculateRegionSplitCostAroundReg(Hint, Order, Cost, NumCands, BestCand);
+ calculateRegionSplitCostAroundReg(Hint, Cost, NumCands, BestCand);
if (BestCand == NoCand)
return false;
diff --git a/llvm/lib/CodeGen/RegAllocGreedy.h b/llvm/lib/CodeGen/RegAllocGreedy.h
index 138da362a1e76f..baef31cbfd5286 100644
--- a/llvm/lib/CodeGen/RegAllocGreedy.h
+++ b/llvm/lib/CodeGen/RegAllocGreedy.h
@@ -313,12 +313,16 @@ class LLVM_LIBRARY_VISIBILITY RAGreedy : public RegAllocBase,
bool hasVirtRegAlloc();
BlockFrequency calcBlockSplitCost();
+ bool calcSplitConstraints(InterferenceCache::Cursor, BlockFrequency *,
+ BlockFrequency MaxCost,
+ SmallVectorImpl<int64_t> *CostDifference = nullptr);
bool addSplitConstraints(InterferenceCache::Cursor, BlockFrequency &,
BlockFrequency MaxCost);
+ bool isWaveRegionCostHigher(GlobalSplitCandidate &);
bool addThroughConstraints(InterferenceCache::Cursor, ArrayRef<unsigned>);
bool growRegion(GlobalSplitCandidate &Cand);
- BlockFrequency calcGlobalSplitCost(GlobalSplitCandidate &,
- const AllocationOrder &Order);
+ void addGlobalSplitCost(GlobalSplitCandidate &, BlockFrequency *,
+ SmallVectorImpl<int64_t> *CostDifference = nullptr);
bool calcCompactRegion(GlobalSplitCandidate &);
void splitAroundRegion(LiveRangeEdit &, ArrayRef<unsigned>);
void calcGapWeights(MCRegister, SmallVectorImpl<float> &);
@@ -338,7 +342,6 @@ class LLVM_LIBRARY_VISIBILITY RAGreedy : public RegAllocBase,
SmallVectorImpl<Register> &);
/// Calculate cost of region splitting around the specified register.
unsigned calculateRegionSplitCostAroundReg(MCRegister PhysReg,
- AllocationOrder &Order,
BlockFrequency &BestCost,
unsigned &NumCands,
unsigned &BestCand);
@@ -353,8 +356,7 @@ class LLVM_LIBRARY_VISIBILITY RAGreedy : public RegAllocBase,
SmallVectorImpl<Register> &NewVRegs);
/// Try to split VirtReg around physical Hint register.
bool trySplitAroundHintReg(MCRegister Hint, const LiveInterval &VirtReg,
- SmallVectorImpl<Register> &NewVRegs,
- AllocationOrder &Order);
+ SmallVectorImpl<Register> &NewVRegs);
/// Check other options before using a callee-saved register for the first
/// time.
MCRegister tryAssignCSRFirstTime(const LiveInterval &VirtReg,
diff --git a/llvm/lib/CodeGen/SpillPlacement.cpp b/llvm/lib/CodeGen/SpillPlacement.cpp
index 0c3c414cf6c847..5dec7cdf76fd92 100644
--- a/llvm/lib/CodeGen/SpillPlacement.cpp
+++ b/llvm/lib/CodeGen/SpillPlacement.cpp
@@ -29,6 +29,7 @@
#include "llvm/CodeGen/SpillPlacement.h"
#include "llvm/ADT/BitVector.h"
#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/SCCIterator.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
#include "llvm/CodeGen/EdgeBundles.h"
@@ -42,12 +43,14 @@
#include "llvm/IR/ProfDataUtils.h"
#include "llvm/InitializePasses.h"
#include "llvm/Pass.h"
+#include "llvm/Support/CheckedArithmetic.h"
#include "llvm/Support/CommandLine.h"
#include "llvm/Support/Debug.h"
#include "llvm/Support/ScaledNumber.h"
#include <algorithm>
#include <cassert>
#include <cstdint>
+#include <optional>
#include <utility>
using namespace llvm;
@@ -57,6 +60,9 @@ using namespace llvm;
static cl::opt<bool> EnableWaveProfiledSpill(
"enable-wave-profiled-spill", cl::Hidden, cl::init(true),
cl::desc("Use validated AMDGPU wave counts for spill placement"));
+static cl::opt<bool> EnableWaveProfiledRegionCost(
+ "enable-wave-profiled-region-cost", cl::Hidden, cl::init(true),
+ cl::desc("Use validated wave counts to bound region spill costs"));
static cl::opt<bool> ReportWaveProfiledSpill(
"report-wave-profiled-spill", cl::Hidden, cl::init(false),
cl::desc("Report wave-count mapping and spill-cost coverage"));
@@ -281,9 +287,10 @@ void SpillPlacement::run(MachineFunction &mf, EdgeBundles *Bundles,
BitVector HasWaveCount;
uint64_t EntryWaveCount = 0;
DenseMap<const BasicBlock *, BlockFrequency> WaveFrequencies;
+ DenseMap<const BasicBlock *, uint64_t> RawWaveCounts;
unsigned RejectedWaveBlocks = 0;
const bool HasWaveProfile =
- EnableWaveProfiledSpill &&
+ (EnableWaveProfiledSpill || EnableWaveProfiledRegionCost) &&
mf.getFunction().getParent()->getTargetTriple().isAMDGPU() &&
extractMappedBlockWaveCounts(mf.getFunction(), WaveCounts, HasWaveCount,
EntryWaveCount) &&
@@ -313,6 +320,11 @@ void SpillPlacement::run(MachineFunction &mf, EdgeBundles *Bundles,
continue;
}
uint64_t Count = WaveCounts[Index++];
+ // Keep the two consumers independent after validating the mapping.
+ if (EnableWaveProfiledRegionCost)
+ RawWaveCounts[&BB] = Count;
+ if (!EnableWaveProfiledSpill)
+ continue;
uint64_t Frequency =
ScaledNumber<uint64_t>::getFraction(Count, EntryWaveCount)
.scale(MBFI->getEntryFreq().getFrequency());
@@ -324,6 +336,14 @@ void SpillPlacement::run(MachineFunction &mf, EdgeBundles *Bundles,
// Compute total ingoing and outgoing block frequencies for all bundles.
BlockFrequencies.resize(mf.getNumBlockIDs());
+ MeasuredWaveBlocks.clear();
+ MeasuredWaveCounts.clear();
+ if (!RawWaveCounts.empty()) {
+ MeasuredWaveBlocks.resize(mf.getNumBlockIDs());
+ MeasuredWaveCounts.assign(mf.getNumBlockIDs(), 0);
+ }
+ AcyclicBlocks.clear();
+ NormalizationWaveCount = EntryWaveCount;
setThreshold(MBFI->getEntryFreq());
unsigned ChangedWaveCosts = 0;
for (auto &I : mf) {
@@ -331,6 +351,12 @@ void SpillPlacement::run(MachineFunction &mf, EdgeBundles *Bundles,
auto Wave = WaveFrequencies.find(I.getBasicBlock());
BlockFrequencies[Num] =
Wave == WaveFrequencies.end() ? MBFI->getBlockFreq(&I) : Wave->second;
+
+ if (auto Count = RawWaveCounts.find(I.getBasicBlock());
+ Count != RawWaveCounts.end()) {
+ MeasuredWaveBlocks.set(Num);
+ MeasuredWaveCounts[Num] = Count->second;
+ }
if (Wave != WaveFrequencies.end()) {
if (ReportWaveProfiledSpill)
ChangedWaveCosts += Wave->second != MBFI->getBlockFreq(&I);
@@ -346,6 +372,49 @@ void SpillPlacement::run(MachineFunction &mf, EdgeBundles *Bundles,
<< "\t" << ChangedWaveCosts << "\n";
}
+bool SpillPlacement::isWaveCostDifferencePositive(
+ ArrayRef<int64_t> Coefficients) {
+ if (!hasMeasuredWaveBlocks())
+ return false;
+ assert(Coefficients.size() == BlockFrequencies.size());
+ if (AcyclicBlocks.empty()) {
+ AcyclicBlocks.resize(MF->getNumBlockIDs());
+ // Unreachable blocks are left unknown. SCCs include irreducible cycles.
+ for (auto I = scc_begin(MF); !I.isAtEnd(); ++I) {
+ if (!I.hasCycle())
+ AcyclicBlocks.set(I->front()->getNumber());
+ }
+ }
+
+ uint64_t Positive = 0, Negative = 0;
+ for (unsigned I = 0; I != Coefficients.size(); ++I) {
+ int64_t C = Coefficients[I];
+ if (!C)
+ continue;
+ uint64_t Frequency = 0;
+ if (MeasuredWaveBlocks.test(I)) {
+ Frequency = MeasuredWaveCounts[I];
+ } else if (C < 0) {
+ if (!AcyclicBlocks.test(I))
+ return false;
+ Frequency = NormalizationWaveCount;
+ }
+ // Reject potentially saturated counters and unrepresentable arithmetic.
+ if (Frequency == UINT64_MAX || C == INT64_MIN)
+ return false;
+ std::optional<uint64_t> Product =
+ checkedMulUnsigned<uint64_t>(C < 0 ? -C : C, Frequency);
+ if (!Product)
+ return false;
+ uint64_t &Sum = C < 0 ? Negative : Positive;
+ std::optional<uint64_t> Total = checkedAddUnsigned<uint64_t>(Sum, *Product);
+ if (!Total)
+ return false;
+ Sum = *Total;
+ }
+ return Positive > Negative;
+}
+
/// activate - mark node n as active if it wasn't already.
void SpillPlacement::activate(unsigned n) {
TodoList.insert(n);
diff --git a/llvm/test/CodeGen/AMDGPU/wave-profile-region-cost.mir b/llvm/test/CodeGen/AMDGPU/wave-profile-region-cost.mir
new file mode 100644
index 00000000000000..1d9c99871c7712
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/wave-profile-region-cost.mir
@@ -0,0 +1,197 @@
+# NOTE: Do not autogenerate. Only spill positions matter; full MIR checks would obscure the cost-control comparison.
+# RUN: llc -mtriple=amdgpu9.00-amd-amdhsa -run-pass=greedy -stress-regalloc=2 -verify-regalloc -o - %s | FileCheck %s --check-prefixes=COST,COMMON
+# RUN: llc -mtriple=amdgpu9.00-amd-amdhsa -run-pass=greedy -stress-regalloc=2 -verify-regalloc -enable-wave-profiled-region-cost=false -o - %s | FileCheck %s --check-prefixes=NO-COST,COMMON
+# RUN: llc -mtriple=amdgpu9.00-amd-amdhsa -run-pass=greedy -stress-regalloc=2 -verify-regalloc -enable-wave-profiled-spill=false -o - %s | FileCheck %s --check-prefixes=COST,COMMON
+
+# Each block executes 100 waves. The through block permits a compact region,
+# while clobbers inside the four use blocks make the best physical region more
+# expensive than block splitting. No input split ancestry is needed: greedy
+# creates the splits itself. With costs enabled, reject that region and spill
+# at the definition in entry. Without the bound, keep entry in a register and
+# spill at the first clobber instead.
+# COMMON-LABEL: name: wave_cost
+# COMMON: bb.0.entry:
+# COMMON: S_NOP 0, implicit-def %[[DEF:[0-9]+]]
+# COST-NEXT: SI_SPILL_V32_SAVE %[[DEF]],
+# COMMON-NEXT: S_BRANCH %bb.1
+# COMMON: bb.1.a:
+# COST: SI_SPILL_V32_RESTORE
+# NO-COST: S_NOP 0, implicit %[[DEF]]
+# NO-COST-NEXT: SI_SPILL_V32_SAVE %[[DEF]],
+# COMMON: S_NOP 0, implicit-def dead $vgpr0, implicit-def dead $vgpr1
+# COMMON-NEXT: {{%[0-9]+}}:vgpr_32 = SI_SPILL_V32_RESTORE
+
+# Moving b's clobber before its first use leaves the incoming compact bundle
+# outside the physical region. The physical cost alone cannot bound the union
+# used by doRegionSplit, so retain region splitting with costs on or off.
+# COMMON-LABEL: name: extra_compact
+# COMMON: bb.0.entry:
+# COMMON: S_NOP 0, implicit-def %[[DEF:[0-9]+]]
+# COMMON-NEXT: S_BRANCH %bb.1
+# COMMON: bb.1.a:
+# COMMON: S_NOP 0, implicit %[[DEF]]
+# COMMON-NEXT: SI_SPILL_V32_SAVE %[[DEF]],
+# COMMON: bb.2.b:
+# COMMON: = lr-split COPY
+# COMMON: S_BRANCH %bb.3
+
+--- |
+ define void @wave_cost(ptr addrspace(1) %out) !wave.profile !0 {
+ entry:
+ store volatile i32 0, ptr addrspace(1) %out, align 4
+ br label %a, !wave.profile.block !1
+
+ a:
+ store volatile i32 1, ptr addrspace(1) %out, align 4
+ br label %b, !wave.profile.block !2
+
+ b:
+ store volatile i32 2, ptr addrspace(1) %out, align 4
+ br label %c, !wave.profile.block !3
+
+ c:
+ store volatile i32 3, ptr addrspace(1) %out, align 4
+ br label %d, !wave.profile.block !4
+
+ d:
+ store volatile i32 4, ptr addrspace(1) %out, align 4
+ br label %through, !wave.profile.block !5
+
+ through:
+ store volatile i32 5, ptr addrspace(1) %out, align 4
+ br label %exit, !wave.profile.block !6
+
+ exit:
+ ret void, !wave.profile.block !7
+ }
+
+ define void @extra_compact(ptr addrspace(1) %out) !wave.profile !8 {
+ entry:
+ store volatile i32 0, ptr addrspace(1) %out, align 4
+ br label %a, !wave.profile.block !9
+
+ a:
+ store volatile i32 1, ptr addrspace(1) %out, align 4
+ br label %b, !wave.profile.block !10
+
+ b:
+ store volatile i32 2, ptr addrspace(1) %out, align 4
+ br label %c, !wave.profile.block !11
+
+ c:
+ store volatile i32 3, ptr addrspace(1) %out, align 4
+ br label %d, !wave.profile.block !12
+
+ d:
+ store volatile i32 4, ptr addrspace(1) %out, align 4
+ br label %through, !wave.profile.block !13
+
+ through:
+ store volatile i32 5, ptr addrspace(1) %out, align 4
+ br label %exit, !wave.profile.block !14
+
+ exit:
+ ret void, !wave.profile.block !15
+ }
+
+ !0 = distinct !{i64 2, i64 14260561443891338516, i64 100, i64 100, i64 100, i64 100, i64 100, i64 100, i64 100}
+ !1 = !{i64 2, i64 14260561443891338516, i64 0, i64 1, i64 1}
+ !2 = !{i64 2, i64 14260561443891338516, i64 1, i64 1, i64 2}
+ !3 = !{i64 2, i64 14260561443891338516, i64 2, i64 1, i64 3}
+ !4 = !{i64 2, i64 14260561443891338516, i64 3, i64 1, i64 4}
+ !5 = !{i64 2, i64 14260561443891338516, i64 4, i64 1, i64 5}
+ !6 = !{i64 2, i64 14260561443891338516, i64 5, i64 1, i64 6}
+ !7 = !{i64 2, i64 14260561443891338516, i64 6, i64 1}
+ !8 = distinct !{i64 2, i64 2091616729901823961, i64 100, i64 100, i64 100, i64 100, i64 100, i64 100, i64 100}
+ !9 = !{i64 2, i64 2091616729901823961, i64 0, i64 1, i64 1}
+ !10 = !{i64 2, i64 2091616729901823961, i64 1, i64 1, i64 2}
+ !11 = !{i64 2, i64 2091616729901823961, i64 2, i64 1, i64 3}
+ !12 = !{i64 2, i64 2091616729901823961, i64 3, i64 1, i64 4}
+ !13 = !{i64 2, i64 2091616729901823961, i64 4, i64 1, i64 5}
+ !14 = !{i64 2, i64 2091616729901823961, i64 5, i64 1, i64 6}
+ !15 = !{i64 2, i64 2091616729901823961, i64 6, i64 1}
+...
+---
+name: wave_cost
+tracksRegLiveness: true
+machineFunctionInfo:
+ scratchRSrcReg: '$sgpr0_sgpr1_sgpr2_sgpr3'
+ stackPtrOffsetReg: '$sgpr32'
+body: |
+ bb.0.entry:
+ successors: %bb.1(0x80000000)
+ S_NOP 0, implicit-def %0:vgpr_32
+ S_BRANCH %bb.1
+ bb.1.a:
+ successors: %bb.2(0x80000000)
+ S_NOP 0, implicit %0
+ S_NOP 0, implicit-def dead $vgpr0, implicit-def dead $vgpr1
+ S_NOP 0, implicit %0
+ S_BRANCH %bb.2
+ bb.2.b:
+ successors: %bb.3(0x80000000)
+ S_NOP 0, implicit %0
+ S_NOP 0, implicit-def dead $vgpr0, implicit-def dead $vgpr1
+ S_NOP 0, implicit %0
+ S_BRANCH %bb.3
+ bb.3.c:
+ successors: %bb.4(0x80000000)
+ S_NOP 0, implicit %0
+ S_NOP 0, implicit-def dead $vgpr0, implicit-def dead $vgpr1
+ S_NOP 0, implicit %0
+ S_BRANCH %bb.4
+ bb.4.d:
+ successors: %bb.5(0x80000000)
+ S_NOP 0, implicit %0
+ S_NOP 0, implicit-def dead $vgpr0, implicit-def dead $vgpr1
+ S_NOP 0, implicit %0
+ S_BRANCH %bb.5
+ bb.5.through:
+ successors: %bb.6(0x80000000)
+ S_BRANCH %bb.6
+ bb.6.exit:
+ S_NOP 0, implicit %0
+ S_ENDPGM 0
+...
+---
+name: extra_compact
+tracksRegLiveness: true
+machineFunctionInfo:
+ scratchRSrcReg: '$sgpr0_sgpr1_sgpr2_sgpr3'
+ stackPtrOffsetReg: '$sgpr32'
+body: |
+ bb.0.entry:
+ successors: %bb.1(0x80000000)
+ S_NOP 0, implicit-def %0:vgpr_32
+ S_BRANCH %bb.1
+ bb.1.a:
+ successors: %bb.2(0x80000000)
+ S_NOP 0, implicit %0
+ S_NOP 0, implicit-def dead $vgpr0, implicit-def dead $vgpr1
+ S_NOP 0, implicit %0
+ S_BRANCH %bb.2
+ bb.2.b:
+ successors: %bb.3(0x80000000)
+ S_NOP 0, implicit-def dead $vgpr0, implicit-def dead $vgpr1
+ S_NOP 0, implicit %0
+ S_NOP 0, implicit %0
+ S_BRANCH %bb.3
+ bb.3.c:
+ successors: %bb.4(0x80000000)
+ S_NOP 0, implicit %0
+ S_NOP 0, implicit-def dead $vgpr0, implicit-def dead $vgpr1
+ S_NOP 0, implicit %0
+ S_BRANCH %bb.4
+ bb.4.d:
+ successors: %bb.5(0x80000000)
+ S_NOP 0, implicit %0
+ S_NOP 0, implicit-def dead $vgpr0, implicit-def dead $vgpr1
+ S_NOP 0, implicit %0
+ S_BRANCH %bb.5
+ bb.5.through:
+ successors: %bb.6(0x80000000)
+ S_BRANCH %bb.6
+ bb.6.exit:
+ S_NOP 0, implicit %0
+ S_ENDPGM 0
+...
diff --git a/llvm/unittests/CodeGen/CMakeLists.txt b/llvm/unittests/CodeGen/CMakeLists.txt
index cee089e335c9b6..d8f7fc1dd75326 100644
--- a/llvm/unittests/CodeGen/CMakeLists.txt
+++ b/llvm/unittests/CodeGen/CMakeLists.txt
@@ -56,6 +56,7 @@ add_llvm_unittest(CodeGenTests
TypeTraitsTest.cpp
TargetOptionsTest.cpp
TestAsmPrinter.cpp
+ WaveRegionCostTest.cpp
X86MCInstLowerTest.cpp
)
diff --git a/llvm/unittests/CodeGen/WaveRegionCostTest.cpp b/llvm/unittests/CodeGen/WaveRegionCostTest.cpp
new file mode 100644
index 00000000000000..51f4897bfb5c44
--- /dev/null
+++ b/llvm/unittests/CodeGen/WaveRegionCostTest.cpp
@@ -0,0 +1,199 @@
+//===- WaveRegionCostTest.cpp ---------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "CodeGenTestBase.h"
+#include "llvm/CodeGen/MachineBlockFrequencyInfo.h"
+#include "llvm/CodeGen/SpillPlacement.h"
+#include "llvm/Config/Targets.h"
+#include "llvm/IR/ProfDataUtils.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/SaveAndRestore.h"
+#include "llvm/Support/TargetSelect.h"
+#include <tuple>
+
+using namespace llvm;
+
+namespace {
+class WaveRegionCostTest : public CodeGenTestBase {
+public:
+ static void SetUpTestCase() {
+#if LLVM_HAS_AMDGPU_TARGET
+ LLVMInitializeAMDGPUTargetInfo();
+ LLVMInitializeAMDGPUTarget();
+ LLVMInitializeAMDGPUTargetMC();
+#endif
+ }
+ void SetUp() override {
+ setUpImpl("amdgpu9.00-amd-amdhsa", "", "");
+ if (!TM)
+ return;
+ ASSERT_TRUE(parseMIR(R"MIR(
+--- |
+ target triple = "amdgcn-amd-amdhsa"
+ define void @test(i1 %c) {
+ entry:
+ br label %once
+ once:
+ br i1 %c, label %a, label %b
+ a:
+ br i1 %c, label %b, label %exit
+ b:
+ br i1 %c, label %a, label %exit
+ exit:
+ ret void
+ }
+...
+---
+name: test
+body: |
+ bb.0.entry:
+ successors: %bb.1
+ S_BRANCH %bb.1
+ bb.1.once:
+ successors: %bb.2, %bb.3
+ S_CBRANCH_SCC1 %bb.2, implicit $scc
+ S_BRANCH %bb.3
+ bb.2.a:
+ successors: %bb.3, %bb.4
+ S_CBRANCH_SCC1 %bb.3, implicit $scc
+ S_BRANCH %bb.4
+ bb.3.b:
+ successors: %bb.2, %bb.4
+ S_CBRANCH_SCC1 %bb.2, implicit $scc
+ S_BRANCH %bb.4
+ bb.4.exit:
+ S_ENDPGM 0
+...
+)MIR"));
+ BitVector HasCounts(5);
+ HasCounts.set(0).set(2).set(4);
+ setBlockWaveCounts(*Mod->getFunction("test"), {100, 0, 1000, 0, 0},
+ HasCounts);
+ }
+ SpillPlacement &placement() {
+ return MFAM.getResult<SpillPlacementAnalysis>(getMF("test"));
+ }
+};
+
+TEST_F(WaveRegionCostTest, AcyclicUnknownBound) {
+ SpillPlacement &SP = placement();
+ ASSERT_TRUE(SP.hasMeasuredWaveBlocks());
+ EXPECT_TRUE(SP.isWaveCostDifferencePositive({0, -2, 1, 0, 0}));
+ EXPECT_FALSE(SP.isWaveCostDifferencePositive({0, -11, 1, 0, 0}));
+ EXPECT_FALSE(SP.isWaveCostDifferencePositive({1, -1, 0, 0, 0}));
+}
+
+TEST_F(WaveRegionCostTest, IrreducibleUnknownBound) {
+ SpillPlacement &SP = placement();
+ EXPECT_FALSE(SP.isWaveCostDifferencePositive({0, 0, 1, -1, 0}));
+ EXPECT_TRUE(SP.isWaveCostDifferencePositive({0, 0, 1, 1, 0}));
+ EXPECT_FALSE(SP.isWaveCostDifferencePositive({0, 0, 0, 1, 0}));
+}
+
+TEST_F(WaveRegionCostTest, MeasuredZeroKeepsFloor) {
+ SpillPlacement &SP = placement();
+ EXPECT_EQ(SP.getBlockFrequency(4).getFrequency(), 1u);
+ // A positive placement floor is not evidence of observed execution.
+ EXPECT_FALSE(SP.isWaveCostDifferencePositive({0, 0, 0, 0, 1}));
+ EXPECT_FALSE(SP.isWaveCostDifferencePositive({0, 0, 0, 0, -1}));
+}
+
+TEST_F(WaveRegionCostTest, CancellationAndOverflow) {
+ SpillPlacement &SP = placement();
+ EXPECT_FALSE(SP.isWaveCostDifferencePositive({0, 0, 0, 0, 0}));
+ EXPECT_FALSE(SP.isWaveCostDifferencePositive({0, 0, INT64_MAX, 0, 0}));
+ EXPECT_FALSE(SP.isWaveCostDifferencePositive({0, INT64_MIN, 1, 0, 0}));
+}
+
+TEST_F(WaveRegionCostTest, PositiveSumOverflow) {
+ setBlockWaveCounts(*Mod->getFunction("test"), {1, 0, UINT64_MAX - 1, 0, 100});
+ EXPECT_FALSE(placement().isWaveCostDifferencePositive({-1, 0, 1, 0, 1}));
+}
+
+TEST_F(WaveRegionCostTest, NegativeSumOverflow) {
+ setBlockWaveCounts(*Mod->getFunction("test"), {1, 0, UINT64_MAX - 1, 0, 2});
+ EXPECT_FALSE(placement().isWaveCostDifferencePositive({1, 0, -1, 0, -1}));
+}
+
+TEST_F(WaveRegionCostTest, MissingProfile) {
+ clearBlockWaveCounts(*Mod->getFunction("test"));
+ EXPECT_FALSE(placement().hasMeasuredWaveBlocks());
+ EXPECT_FALSE(placement().isWaveCostDifferencePositive({0, -1, 1, 0, 0}));
+}
+
+// A function-level uniformity marker supplies no execution frequency for
+// unmeasured blocks, including those in an irreducible cycle.
+TEST_F(WaveRegionCostTest, UniformityMarkerKeepsUnmeasuredFrequencies) {
+ MachineFunction &MF = getMF("test");
+ MF.getFunction().setMetadata(LLVMContext::MD_uniformity_profile,
+ MDNode::get(Mod->getContext(), {}));
+ MachineBlockFrequencyInfo &MBFI =
+ MFAM.getResult<MachineBlockFrequencyAnalysis>(MF);
+ SpillPlacement &SP = placement();
+ for (unsigned Number : {1u, 3u})
+ EXPECT_EQ(SP.getBlockFrequency(Number),
+ MBFI.getBlockFreq(MF.getBlockNumbered(Number)));
+ EXPECT_EQ(SP.getBlockFrequency(4).getFrequency(), 1u);
+}
+
+TEST_F(WaveRegionCostTest, UniformityMarkerWithoutWaveCounts) {
+ MachineFunction &MF = getMF("test");
+ clearBlockWaveCounts(MF.getFunction());
+ MF.getFunction().setMetadata(LLVMContext::MD_uniformity_profile,
+ MDNode::get(Mod->getContext(), {}));
+ MachineBlockFrequencyInfo &MBFI =
+ MFAM.getResult<MachineBlockFrequencyAnalysis>(MF);
+ SpillPlacement &SP = placement();
+ EXPECT_FALSE(SP.hasMeasuredWaveBlocks());
+ for (MachineBasicBlock &MBB : MF)
+ EXPECT_EQ(SP.getBlockFrequency(MBB.getNumber()), MBFI.getBlockFreq(&MBB));
+}
+
+TEST_F(WaveRegionCostTest, InvalidEntry) {
+ setBlockWaveCounts(*Mod->getFunction("test"), {0, 0, 1000, 0, 0});
+ EXPECT_FALSE(placement().hasMeasuredWaveBlocks());
+ EXPECT_FALSE(placement().isWaveCostDifferencePositive({0, -1, 1, 0, 0}));
+}
+
+TEST_F(WaveRegionCostTest, SaturatedCount) {
+ setBlockWaveCounts(*Mod->getFunction("test"), {1, 0, UINT64_MAX, 0, 0});
+ EXPECT_FALSE(placement().isWaveCostDifferencePositive({0, 0, 1, 0, 0}));
+}
+
+class WaveSpillOptionsTest
+ : public WaveRegionCostTest,
+ public testing::WithParamInterface<std::tuple<bool, bool>> {};
+
+TEST_P(WaveSpillOptionsTest, IndependentConsumers) {
+ auto [UsePlacement, UseCosts] = GetParam();
+ auto *Placement = static_cast<cl::opt<bool> *>(
+ cl::getRegisteredOptions().lookup("enable-wave-profiled-spill"));
+ auto *Costs = static_cast<cl::opt<bool> *>(
+ cl::getRegisteredOptions().lookup("enable-wave-profiled-region-cost"));
+ ASSERT_NE(Placement, nullptr);
+ ASSERT_NE(Costs, nullptr);
+ SaveAndRestore<bool> RestorePlacement(Placement->getValue(), UsePlacement);
+ SaveAndRestore<bool> RestoreCosts(Costs->getValue(), UseCosts);
+
+ MachineFunction &MF = getMF("test");
+ MachineBlockFrequencyInfo &MBFI =
+ MFAM.getResult<MachineBlockFrequencyAnalysis>(MF);
+ SpillPlacement &SP = placement();
+ EXPECT_EQ(SP.hasMeasuredWaveBlocks(), UseCosts);
+ // A positive raw-count bound survives disabling frequency reweighting.
+ EXPECT_EQ(SP.isWaveCostDifferencePositive({0, -2, 1, 0, 0}), UseCosts);
+ // Placement still uses the measured-zero floor with cost bounds disabled.
+ BlockFrequency Expected = UsePlacement
+ ? BlockFrequency(1)
+ : MBFI.getBlockFreq(MF.getBlockNumbered(4));
+ EXPECT_EQ(SP.getBlockFrequency(4), Expected);
+}
+
+INSTANTIATE_TEST_SUITE_P(IndependentOptions, WaveSpillOptionsTest,
+ testing::Combine(testing::Bool(), testing::Bool()));
+} // namespace
More information about the llvm-branch-commits
mailing list