[llvm] [AMDGPU] Introduce ABI occupancy for object linking (PR #199475)
Shilei Tian via llvm-commits
llvm-commits at lists.llvm.org
Tue Jun 30 17:47:40 PDT 2026
https://github.com/shiltian updated https://github.com/llvm/llvm-project/pull/199475
>From a5b0b1d4db6d66d77cda2cc07223c2ff60aa2c79 Mon Sep 17 00:00:00 2001
From: Shilei Tian <i at tianshilei.me>
Date: Sun, 24 May 2026 22:12:08 -0400
Subject: [PATCH] [AMDGPU] Introduce ABI occupancy for object linking
This PR introduces ABI occupancy as the contract used to compile functions under
object linking. The default is derived from the occupancy needed for a 1024
workitem workgroup, can be overridden with `-amdgpu-abi-waves-per-eu`, and can
be overridden per function by `amdgpu-flat-work-group-size` or tightened by an
accepted `amdgpu-waves-per-eu` hint.
The backend emits the selected occupancy in `.amdgpu.info` and uses it to
enforce the object linking register budget.
---
llvm/docs/AMDGPUUsage.rst | 63 +++++++++--
.../llvm/Support/AMDGPUObjLinkingInfo.h | 3 +
llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp | 102 ++++++++++++------
.../Target/AMDGPU/AMDGPUMCResourceInfo.cpp | 37 ++++++-
llvm/lib/Target/AMDGPU/AMDGPUSubtarget.cpp | 34 +++++-
llvm/lib/Target/AMDGPU/AMDGPUSubtarget.h | 8 ++
.../AMDGPU/AsmParser/AMDGPUAsmParser.cpp | 6 ++
llvm/lib/Target/AMDGPU/GCNSubtarget.cpp | 12 ++-
llvm/lib/Target/AMDGPU/GCNSubtarget.h | 17 +++
.../MCTargetDesc/AMDGPUTargetStreamer.cpp | 4 +
.../MCTargetDesc/AMDGPUTargetStreamer.h | 1 +
.../Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp | 8 ++
llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h | 3 +
...ct-linking-abi-occupancy-device-no-attr.ll | 27 +++++
...king-abi-occupancy-flat-workgroup-lower.ll | 24 +++++
...ccupancy-flat-workgroup-reject-override.ll | 28 +++++
...ct-linking-abi-occupancy-kernel-no-attr.ll | 31 ++++++
.../object-linking-abi-occupancy-override.ll | 30 ++++++
...ct-linking-abi-occupancy-preserves-attr.ll | 34 ++++++
.../amdgpu-abi-waves-per-eu-module-flag.ll | 16 +++
llvm/test/MC/AMDGPU/amdgpu-info-roundtrip.s | 6 ++
21 files changed, 446 insertions(+), 48 deletions(-)
create mode 100644 llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-device-no-attr.ll
create mode 100644 llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-flat-workgroup-lower.ll
create mode 100644 llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-flat-workgroup-reject-override.ll
create mode 100644 llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-kernel-no-attr.ll
create mode 100644 llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-override.ll
create mode 100644 llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-preserves-attr.ll
create mode 100644 llvm/test/Linker/amdgpu-abi-waves-per-eu-module-flag.ll
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 2b522395ee892..8c00b934853a0 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -2301,8 +2301,10 @@ The AMDGPU backend supports the following LLVM IR attributes.
CLANG attribute [CLANG-ATTR]_. This is an optimization hint,
and the backend may not be able to satisfy the request. If
the specified range is incompatible with the function's
- "amdgpu-flat-work-group-size" value, the implied occupancy
- bounds by the workgroup size takes precedence.
+ "amdgpu-flat-work-group-size" value, the occupancy bounds
+ implied by the workgroup size take precedence. Under object
+ linking, this hint cannot lower the ABI occupancy budget;
+ see :ref:`amdgpu-abi-occupancy`.
"amdgpu-ieee" true/false. GFX6-GFX11 (Except GFX11.7) Only
Specify whether the function expects the IEEE field of the
@@ -2713,6 +2715,51 @@ unit's worst case (i.e, maxima) ``num_vgpr``, ``num_agpr``, and
symbolic expressions. These three symbols are ``amdgcn.max_num_vgpr``,
``amdgcn.max_num_agpr``, and ``amdgcn.max_num_sgpr``.
+.. _amdgpu-abi-occupancy:
+
+ABI Occupancy (Object Linking)
+------------------------------
+
+When object linking is enabled, the AMDGPU backend no longer assumes
+whole-program visibility. Individual translation units must agree on an ABI that
+callers and callees can rely on without seeing each other's register usage. The
+compiler enforces this by using an *ABI occupancy* as a resource-budget floor
+for separately compiled functions. The default ABI occupancy is the number of
+waves per EU required to support a 1024-workitem workgroup:
+
+``ceil(ceil(1024 / wavefront-size) / EUs-per-CU)``.
+
+For example, on ``gfx900`` (wave64, 4 EUs/CU) the default ABI occupancy is
+``ceil(ceil(1024 / 64) / 4) = 4``, yielding a VGPR budget of
+``getMaxNumVGPRs(4) = 64``. On a gfx12 target such as ``gfx1200`` (wave32,
+4 EUs/CU), the default ABI occupancy is ``ceil(ceil(1024 / 32) / 4) = 8``,
+yielding a VGPR budget of ``getMaxNumVGPRs(8) = 192``.
+
+The ``amdgpu_abi_waves_per_eu`` module flag overrides the default ABI occupancy
+floor for a module. A value lower than the default accepts a looser ABI
+contract; a value higher than the default requests a stricter one. Any compiler
+driver option that exposes this control should set the module flag in generated
+IR rather than passing the value directly to the backend.
+
+``amdgpu-flat-work-group-size`` is ABI-significant. If a function carries this
+attribute, the occupancy implied by its maximum flat workgroup size replaces
+the default ABI occupancy for that function, so it can either raise or lower the
+register budget.
+
+``amdgpu-waves-per-eu`` remains a hint, and it does not define the ABI
+occupancy. It affects the ABI register budget only if the backend accepts it as
+an effective occupancy request. In that case, a minimum higher than the ABI
+occupancy makes the register budget stricter; a lower minimum cannot lower the
+ABI budget. The maximum side of an accepted ``amdgpu-waves-per-eu`` hint can
+still cap occupancy by inflating reported resource usage. Under object linking,
+the effective maximum used for this inflation is raised to at least the ABI
+occupancy, so the emitted resource usage does not describe a looser contract
+than the ABI budget.
+
+The backend records the register-budget occupancy in each function's
+``.amdgpu.info`` metadata so link-time object linking can reject incompatible
+occupancy contracts across translation units.
+
.. _amdgpu-elf-code-object:
ELF Code Object
@@ -3148,11 +3195,10 @@ if needed.
.. _amdgpu-info-section:
``.amdgpu.info``
- Per-function metadata for AMDGPU object linking, emitted only in relocatable
- code objects when object linking is enabled
- (``-amdgpu-enable-object-linking``). The linker uses this section to
- propagate resource usage (registers, stack, LDS) and resolve call graph
- dependencies across translation units.
+ Per-function metadata emitted only in relocatable code objects when object
+ linking is enabled. The linker uses this section to propagate resource usage
+ (registers, stack, LDS) and resolve call graph dependencies across
+ translation units.
Each entry uses a tagged, length-prefixed binary encoding:
@@ -3181,6 +3227,7 @@ if needed.
8 ``INFO_CALL`` 8B symbol ref; direct call edge
9 ``INFO_INDIRECT_CALL`` u32 strtab offset; indirect call type-ID
10 ``INFO_TYPEID`` u32 strtab offset; function type-ID
+ 11 ``INFO_OCCUPANCY`` u32; occupancy used to compile the function
===== ============================== ==========================================
.. table:: AMDGPU Info Function Flags (``INFO_FLAGS``)
@@ -21719,6 +21766,7 @@ The following sub-directives may appear inside the block:
``.amdgpu_num_vgpr`` *value* Architectural VGPRs used (u32)
``.amdgpu_num_agpr`` *value* Accumulator VGPRs used (u32)
``.amdgpu_private_segment_size`` *n* Private segment size in bytes (u32)
+ ``.amdgpu_occupancy`` *value* Occupancy used to compile the function (u32)
``.amdgpu_use`` *symbol* Resource dependency (LDS or barrier)
``.amdgpu_call`` *symbol* Direct call edge to *symbol*
``.amdgpu_indirect_call`` *"type-id"* Indirect call with given type-ID string
@@ -21735,6 +21783,7 @@ Example:
.amdgpu_num_vgpr 32
.amdgpu_num_agpr 0
.amdgpu_private_segment_size 0
+ .amdgpu_occupancy 4
.amdgpu_use lds_var
.amdgpu_call helper
.amdgpu_indirect_call "vi"
diff --git a/llvm/include/llvm/Support/AMDGPUObjLinkingInfo.h b/llvm/include/llvm/Support/AMDGPUObjLinkingInfo.h
index e65161e6545fc..ef71e836bfc40 100644
--- a/llvm/include/llvm/Support/AMDGPUObjLinkingInfo.h
+++ b/llvm/include/llvm/Support/AMDGPUObjLinkingInfo.h
@@ -58,6 +58,9 @@ enum class InfoKind : uint8_t {
/// string (at the given `.amdgpu.strtab` offset) so the linker can match
/// it against INFO_INDIRECT_CALL entries. [u32]
INFO_TYPEID = 10,
+ /// Occupancy used to compile the function.
+ /// [u32]
+ INFO_OCCUPANCY = 11,
};
/// Per-function flags packed into INFO_FLAGS entries.
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp
index 53d45d97ab527..e9c669d744a5c 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp
@@ -463,7 +463,7 @@ const AMDGPUMCExpr *createOccupancy(unsigned InitOcc, const MCExpr *NumSGPRs,
}
void AMDGPUAsmPrinter::validateMCResourceInfo(Function &F) {
- if (F.isDeclaration() || !AMDGPU::isModuleEntryFunctionCC(F.getCallingConv()))
+ if (F.isDeclaration())
return;
using RIK = MCResourceInfo::ResourceInfoKind;
@@ -479,6 +479,43 @@ void AMDGPUAsmPrinter::validateMCResourceInfo(Function &F) {
return false;
};
+ // Register allocation normally respects the ABI register budget. Final
+ // resource accounting can still exceed it when the IR names fixed physical
+ // registers, e.g. inline asm clobbers. Check the resolved per-function
+ // resource symbols here so object-linking metadata is not emitted for a
+ // function that cannot satisfy its ABI occupancy.
+ if (AMDGPUTargetMachine::EnableObjectLinking) {
+ const unsigned BudgetVGPR = STM.getMaxNumVGPRs(F);
+ const unsigned BudgetSGPR = STM.getMaxNumSGPRs(F);
+ const unsigned BudgetAGPR = STM.getMaxNumAGPRs(F);
+
+ auto CheckBudget = [&](RIK Kind, unsigned Budget, const char *What) {
+ MCSymbol *Sym = RI.getSymbol(FnSym->getName(), Kind, OutContext);
+ uint64_t Used;
+ if (!Sym->isVariable() ||
+ !TryGetMCExprValue(Sym->getVariableValue(), Used))
+ return false;
+ if (Used <= Budget)
+ return false;
+ F.getContext().diagnose(DiagnosticInfoResourceLimit(
+ F, What, Used, Budget, DS_Error, DK_ResourceLimit));
+ return true;
+ };
+
+ if (CheckBudget(RIK::RIK_NumVGPR, BudgetVGPR,
+ "VGPRs under object-linking ABI"))
+ return;
+ if (CheckBudget(RIK::RIK_NumAGPR, BudgetAGPR,
+ "AGPRs under object-linking ABI"))
+ return;
+ if (CheckBudget(RIK::RIK_NumSGPR, BudgetSGPR,
+ "SGPRs under object-linking ABI"))
+ return;
+ }
+
+ if (!AMDGPU::isModuleEntryFunctionCC(F.getCallingConv()))
+ return;
+
const uint64_t MaxScratchPerWorkitem =
STM.getMaxWaveScratchSize() / STM.getWavefrontSize();
MCSymbol *ScratchSizeSymbol =
@@ -551,20 +588,17 @@ void AMDGPUAsmPrinter::validateMCResourceInfo(Function &F) {
TryGetMCExprValue(NumVgprSymbol->getVariableValue(), NumVgpr) &&
TryGetMCExprValue(NumAgprSymbol->getVariableValue(), NumAgpr)) {
const SIMachineFunctionInfo &MFI = *MF->getInfo<SIMachineFunctionInfo>();
- unsigned MaxWaves = MFI.getMaxWavesPerEU();
+ unsigned MaxWaves = STM.getWavesPerEU(F).second;
uint64_t TotalNumVgpr =
getTotalNumVGPRs(STM.hasGFX90AInsts(), NumAgpr, NumVgpr);
- uint64_t NumVGPRsForWavesPerEU =
- std::max({TotalNumVgpr, (uint64_t)1,
- (uint64_t)STM.getMinNumVGPRs(
- MaxWaves, MFI.getDynamicVGPRBlockSize())});
- uint64_t NumSGPRsForWavesPerEU = std::max(
- {NumSgpr, (uint64_t)1, (uint64_t)STM.getMinNumSGPRs(MaxWaves)});
- const MCExpr *OccupancyExpr = createOccupancy(
- STM.getOccupancyWithWorkGroupSizes(*MF).second,
- MCConstantExpr::create(NumSGPRsForWavesPerEU, OutContext),
- MCConstantExpr::create(NumVGPRsForWavesPerEU, OutContext),
- MFI.getDynamicVGPRBlockSize(), STM, OutContext);
+ auto [EffectiveSGPRs, EffectiveVGPRs] =
+ STM.getEffectiveNumGPRsForWavesPerEU(MaxWaves, NumSgpr, TotalNumVgpr,
+ MFI.getDynamicVGPRBlockSize());
+ const MCExpr *OccupancyExpr =
+ createOccupancy(STM.getOccupancyWithWorkGroupSizes(*MF).second,
+ MCConstantExpr::create(EffectiveSGPRs, OutContext),
+ MCConstantExpr::create(EffectiveVGPRs, OutContext),
+ MFI.getDynamicVGPRBlockSize(), STM, OutContext);
uint64_t Occupancy;
const auto [MinWEU, MaxWEU] = AMDGPU::getIntegerPairAttribute(
@@ -735,30 +769,25 @@ bool AMDGPUAsmPrinter::doFinalization(Module &M) {
// LDS/named-barrier use edges, indirect calls, and address-taken type IDs).
emitAMDGPUInfo(M);
- // Assign expressions which can only be resolved when all other functions are
- // known.
- RI.finalize(OutContext);
-
- // Switch section and emit all GPR maximums within the processed module.
- OutStreamer->pushSection();
- MCSectionELF *MaxGPRSection =
- OutContext.getELFSection(".AMDGPU.gpr_maximums", ELF::SHT_PROGBITS, 0);
- OutStreamer->switchSection(MaxGPRSection);
- getTargetStreamer()->EmitMCResourceMaximums(
- RI.getMaxVGPRSymbol(OutContext), RI.getMaxAGPRSymbol(OutContext),
- RI.getMaxSGPRSymbol(OutContext), RI.getMaxNamedBarrierSymbol(OutContext));
- OutStreamer->popSection();
-
- // In the object-linking pipeline per-function resource MCExprs reference
- // external callee symbols that cannot be evaluated here, so cross-TU limit
- // checks would silently no-op for every non-leaf function. Defer resource
- // sanity checking to the linker, which re-validates against the aggregated
- // call graph in the combined .amdgpu.info metadata.
+ // Finalize non-object-linking resource propagation and emit the
+ // `amdgpu.max_num_*` fallback symbols.
if (!AMDGPUTargetMachine::EnableObjectLinking) {
- for (Function &F : M.functions())
- validateMCResourceInfo(F);
+ RI.finalize(OutContext);
+
+ OutStreamer->pushSection();
+ MCSectionELF *MaxGPRSection =
+ OutContext.getELFSection(".AMDGPU.gpr_maximums", ELF::SHT_PROGBITS, 0);
+ OutStreamer->switchSection(MaxGPRSection);
+ getTargetStreamer()->EmitMCResourceMaximums(
+ RI.getMaxVGPRSymbol(OutContext), RI.getMaxAGPRSymbol(OutContext),
+ RI.getMaxSGPRSymbol(OutContext),
+ RI.getMaxNamedBarrierSymbol(OutContext));
+ OutStreamer->popSection();
}
+ for (Function &F : M.functions())
+ validateMCResourceInfo(F);
+
RI.reset();
return AsmPrinter::doFinalization(M);
@@ -921,11 +950,14 @@ bool AMDGPUAsmPrinter::runOnMachineFunction(MachineFunction &MF) {
if (AMDGPUTargetMachine::EnableObjectLinking) {
const AMDGPUResourceUsageAnalysisWrapperPass::FunctionResourceInfo &RU =
*ResourceUsage;
+ uint32_t Occupancy =
+ static_cast<uint32_t>(STM.getWavesPerEU(MF.getFunction()).first);
FunctionInfos.push_back(
{/*NumSGPR=*/static_cast<uint32_t>(RU.NumExplicitSGPR),
/*NumArchVGPR=*/static_cast<uint32_t>(RU.NumVGPR),
/*NumAccVGPR=*/static_cast<uint32_t>(RU.NumAGPR),
/*PrivateSegmentSize=*/static_cast<uint32_t>(RU.PrivateSegmentSize),
+ /*Occupancy=*/Occupancy,
/*UsesVCC=*/RU.UsesVCC,
/*UsesFlatScratch=*/RU.UsesFlatScratch,
/*HasDynStack=*/RU.HasDynamicallySizedStack,
@@ -1341,7 +1373,7 @@ void AMDGPUAsmPrinter::getSIProgramInfo(SIProgramInfo &ProgInfo,
// Adjust number of registers used to meet default/requested minimum/maximum
// number of waves per execution unit request.
- unsigned MaxWaves = MFI->getMaxWavesPerEU();
+ unsigned MaxWaves = STM.getWavesPerEU(MF.getFunction()).second;
ProgInfo.NumSGPRsForWavesPerEU =
AMDGPUMCExpr::createMax({ProgInfo.NumSGPR, CreateExpr(1ul),
CreateExpr(STM.getMinNumSGPRs(MaxWaves))},
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUMCResourceInfo.cpp b/llvm/lib/Target/AMDGPU/AMDGPUMCResourceInfo.cpp
index 1731b0463a099..e8988cdb84d91 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUMCResourceInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUMCResourceInfo.cpp
@@ -13,6 +13,8 @@
//===----------------------------------------------------------------------===//
#include "AMDGPUMCResourceInfo.h"
+#include "AMDGPUTargetMachine.h"
+#include "GCNSubtarget.h"
#include "SIMachineFunctionInfo.h"
#include "Utils/AMDGPUBaseInfo.h"
#include "llvm/ADT/StringRef.h"
@@ -262,7 +264,8 @@ void MCResourceInfo::gatherResourceInfo(
MF.getInfo<SIMachineFunctionInfo>()->isDynamicVGPREnabled();
if (!AMDGPU::isEntryFunctionCC(CC)) {
- if (!IsDynamicVGPREnabled || !IsChainCC)
+ if ((!IsDynamicVGPREnabled || !IsChainCC) &&
+ !AMDGPUTargetMachine::EnableObjectLinking)
addMaxVGPRCandidate(FRI.NumVGPR);
addMaxAGPRCandidate(FRI.NumAGPR);
addMaxSGPRCandidate(FRI.NumExplicitSGPR);
@@ -285,9 +288,37 @@ void MCResourceInfo::gatherResourceInfo(
if (AMDGPUTargetMachine::EnableObjectLinking) {
LLVM_DEBUG(dbgs() << "MCResUse: object linking enabled, no call-graph "
"propagation; emitting local resource values only\n");
- SetToLocal(FRI.NumVGPR, RIK_NumVGPR);
+ const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
+ unsigned MaxWaves = ST.getWavesPerEU(MF.getFunction()).second;
+ unsigned DynamicVGPRBlockSize =
+ AMDGPU::getDynamicVGPRBlockSize(MF.getFunction());
+ if (DynamicVGPRBlockSize == 0 && ST.isDynamicVGPREnabled())
+ DynamicVGPRBlockSize = ST.getDynamicVGPRBlockSize();
+
+ uint64_t TotalVGPR =
+ AMDGPU::getTotalNumVGPRs(ST.hasGFX90AInsts(), FRI.NumAGPR, FRI.NumVGPR);
+ unsigned ExtraSGPRs =
+ AMDGPU::IsaInfo::getNumExtraSGPRs(ST, FRI.UsesVCC, FRI.UsesFlatScratch,
+ ST.getTargetID().isXnackOnOrAny());
+ uint64_t TotalSGPR = FRI.NumExplicitSGPR + ExtraSGPRs;
+ auto [EffectiveSGPRs, EffectiveVGPRs] = ST.getEffectiveNumGPRsForWavesPerEU(
+ MaxWaves, TotalSGPR, TotalVGPR, DynamicVGPRBlockSize);
+
+ int32_t EffectiveNumVGPR = FRI.NumVGPR;
+ if (EffectiveVGPRs > TotalVGPR) {
+ if (ST.hasGFX90AInsts())
+ EffectiveNumVGPR = static_cast<int32_t>(EffectiveVGPRs - FRI.NumAGPR);
+ else
+ EffectiveNumVGPR = static_cast<int32_t>(EffectiveVGPRs);
+ }
+
+ int32_t EffectiveNumSGPR = FRI.NumExplicitSGPR;
+ if (EffectiveSGPRs > TotalSGPR)
+ EffectiveNumSGPR += static_cast<int32_t>(EffectiveSGPRs - TotalSGPR);
+
+ SetToLocal(EffectiveNumVGPR, RIK_NumVGPR);
SetToLocal(FRI.NumAGPR, RIK_NumAGPR);
- SetToLocal(FRI.NumExplicitSGPR, RIK_NumSGPR);
+ SetToLocal(EffectiveNumSGPR, RIK_NumSGPR);
SetToLocal(FRI.NumNamedBarrier, RIK_NumNamedBarrier);
SetToLocal(FRI.PrivateSegmentSize, RIK_PrivateSegSize);
SetToLocal(FRI.UsesVCC, RIK_UsesVCC);
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUSubtarget.cpp b/llvm/lib/Target/AMDGPU/AMDGPUSubtarget.cpp
index 93bf957880074..74eaf6132a628 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUSubtarget.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUSubtarget.cpp
@@ -16,6 +16,7 @@
#include "AMDGPUInstructionSelector.h"
#include "AMDGPULegalizerInfo.h"
#include "AMDGPURegisterBankInfo.h"
+#include "AMDGPUTargetMachine.h"
#include "R600Subtarget.h"
#include "SIMachineFunctionInfo.h"
#include "Utils/AMDGPUBaseInfo.h"
@@ -26,6 +27,7 @@
#include "llvm/IR/IntrinsicsAMDGPU.h"
#include "llvm/IR/IntrinsicsR600.h"
#include "llvm/IR/MDBuilder.h"
+#include "llvm/Support/MathExtras.h"
#include <algorithm>
using namespace llvm;
@@ -41,7 +43,7 @@ AMDGPUSubtarget::getMaxLocalMemSizeWithWaveCount(unsigned NWaves,
const unsigned WaveSize = getWavefrontSize();
const unsigned WorkGroupSize = getFlatWorkGroupSizes(F).second;
const unsigned WavesPerWorkgroup =
- std::max(1u, (WorkGroupSize + WaveSize - 1) / WaveSize);
+ std::max(1u, divideCeil(WorkGroupSize, WaveSize));
const unsigned WorkGroupsPerCU =
std::max(1u, (NWaves * getEUsPerCU()) / WavesPerWorkgroup);
@@ -227,7 +229,35 @@ AMDGPUSubtarget::getWavesPerEU(std::pair<unsigned, unsigned> FlatWorkGroupSizes,
// Requested minimum/maximum number of waves per execution unit.
std::pair<unsigned, unsigned> Requested =
AMDGPU::getIntegerPairAttribute(F, "amdgpu-waves-per-eu", Default, true);
- return getEffectiveWavesPerEU(Requested, FlatWorkGroupSizes, LDSBytes);
+ std::pair<unsigned, unsigned> WavesPerEU =
+ getEffectiveWavesPerEU(Requested, FlatWorkGroupSizes, LDSBytes);
+ if (!AMDGPUTargetMachine::EnableObjectLinking)
+ return WavesPerEU;
+
+ unsigned FunctionABIOccupancy = getFunctionABIOccupancy(F);
+ unsigned EffectiveMinWaves =
+ F.hasFnAttribute("amdgpu-waves-per-eu")
+ ? std::max(WavesPerEU.first, FunctionABIOccupancy)
+ : FunctionABIOccupancy;
+ return {EffectiveMinWaves, std::max(WavesPerEU.second, EffectiveMinWaves)};
+}
+
+unsigned AMDGPUSubtarget::getDefaultABIOccupancy(const Module &M) const {
+ unsigned Override = AMDGPU::getAMDGPUABIWavesPerEU(M);
+ if (Override)
+ Override = std::clamp(Override, getMinWavesPerEU(), getMaxWavesPerEU());
+
+ constexpr unsigned DefaultMaxWorkGroupSize = 1024;
+ return Override ? Override
+ : getWavesPerEUForWorkGroup(DefaultMaxWorkGroupSize);
+}
+
+unsigned AMDGPUSubtarget::getFunctionABIOccupancy(const Function &F) const {
+ if (!F.hasFnAttribute("amdgpu-flat-work-group-size"))
+ return getDefaultABIOccupancy(*F.getParent());
+
+ std::pair<unsigned, unsigned> FlatWorkGroupSizes = getFlatWorkGroupSizes(F);
+ return getWavesPerEUForWorkGroup(FlatWorkGroupSizes.second);
}
std::optional<unsigned>
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUSubtarget.h b/llvm/lib/Target/AMDGPU/AMDGPUSubtarget.h
index af3facc0135f0..9b9d3773f289d 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUSubtarget.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPUSubtarget.h
@@ -25,6 +25,7 @@ enum AMDGPUDwarfFlavour : unsigned;
class Function;
class Instruction;
class MachineFunction;
+class Module;
class TargetMachine;
class AMDGPUSubtarget {
@@ -133,6 +134,13 @@ class AMDGPUSubtarget {
std::pair<unsigned, unsigned> FlatWorkGroupSizes,
unsigned LDSBytes) const;
+ /// \returns The default ABI occupancy for \p M.
+ unsigned getDefaultABIOccupancy(const Module &M) const;
+
+ /// \returns The ABI occupancy for \p F. A function-specific flat-workgroup
+ /// attribute overrides the default ABI occupancy.
+ unsigned getFunctionABIOccupancy(const Function &F) const;
+
/// Return the amount of LDS that can be used that will not restrict the
/// occupancy lower than WaveCount.
unsigned getMaxLocalMemSizeWithWaveCount(unsigned WaveCount,
diff --git a/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp b/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp
index 161395fffca77..deea1b4b8315b 100644
--- a/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp
+++ b/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp
@@ -6930,6 +6930,12 @@ bool AMDGPUAsmParser::ParseDirectiveAMDGPUInfo() {
return true;
FI.PrivateSegmentSize = static_cast<uint32_t>(Val);
HasScalarAttrs = true;
+ } else if (Dir == "occupancy") {
+ int64_t Val;
+ if (getParser().parseAbsoluteExpression(Val))
+ return true;
+ FI.Occupancy = static_cast<uint32_t>(Val);
+ HasScalarAttrs = true;
} else if (Dir == "use") {
StringRef ResName;
if (getParser().parseIdentifier(ResName))
diff --git a/llvm/lib/Target/AMDGPU/GCNSubtarget.cpp b/llvm/lib/Target/AMDGPU/GCNSubtarget.cpp
index 23c92b9095e36..1d62ff8653db4 100644
--- a/llvm/lib/Target/AMDGPU/GCNSubtarget.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNSubtarget.cpp
@@ -478,6 +478,16 @@ GCNSubtarget::computeOccupancy(const Function &F, unsigned LDSSize,
return {std::min(MinOcc, MaxOcc), MaxOcc};
}
+std::pair<uint64_t, uint64_t> GCNSubtarget::getEffectiveNumGPRsForWavesPerEU(
+ unsigned MaxWaves, uint64_t NumSGPRs, uint64_t NumVGPRs,
+ unsigned DynamicVGPRBlockSize) const {
+ // These are occupancy-reporting counts, not literal register usage. Clamp to
+ // at least one register because zero does not constrain occupancy.
+ return {std::max({NumSGPRs, uint64_t(1), uint64_t(getMinNumSGPRs(MaxWaves))}),
+ std::max({NumVGPRs, uint64_t(1),
+ uint64_t(getMinNumVGPRs(MaxWaves, DynamicVGPRBlockSize))})};
+}
+
unsigned GCNSubtarget::getBaseMaxNumSGPRs(
const Function &F, std::pair<unsigned, unsigned> WavesPerEU,
unsigned PreloadedSGPRs, unsigned ReservedNumSGPRs) const {
@@ -528,7 +538,7 @@ unsigned GCNSubtarget::getBaseMaxNumSGPRs(
unsigned GCNSubtarget::getMaxNumSGPRs(const MachineFunction &MF) const {
const Function &F = MF.getFunction();
const SIMachineFunctionInfo &MFI = *MF.getInfo<SIMachineFunctionInfo>();
- return getBaseMaxNumSGPRs(F, MFI.getWavesPerEU(), MFI.getNumPreloadedSGPRs(),
+ return getBaseMaxNumSGPRs(F, getWavesPerEU(F), MFI.getNumPreloadedSGPRs(),
getReservedNumSGPRs(MF));
}
diff --git a/llvm/lib/Target/AMDGPU/GCNSubtarget.h b/llvm/lib/Target/AMDGPU/GCNSubtarget.h
index 7e4d9bd20cb2f..2efb38b60d03b 100644
--- a/llvm/lib/Target/AMDGPU/GCNSubtarget.h
+++ b/llvm/lib/Target/AMDGPU/GCNSubtarget.h
@@ -23,6 +23,7 @@
#include "Utils/AMDGPUBaseInfo.h"
#include "llvm/Support/AMDHSAKernelDescriptor.h"
#include "llvm/Support/ErrorHandling.h"
+#include <cstdint>
#define GET_SUBTARGETINFO_HEADER
#include "AMDGPUGenSubtargetInfo.inc"
@@ -705,6 +706,22 @@ class GCNSubtarget final : public AMDGPUGenSubtargetInfo,
unsigned NumSGPRs = 0,
unsigned NumVGPRs = 0) const;
+ /// \returns {SGPRs, VGPRs} resource counts for occupancy-adjusted reporting.
+ ///
+ /// This is not the function's literal register usage. It is the count to use
+ /// when resource metadata or kernel descriptor fields need to encode the
+ /// effect of capping occupancy to at most \p MaxWaves waves per EU. If the
+ /// real usage is below the threshold needed to enforce that cap, the returned
+ /// count is inflated to the minimum count that does enforce it.
+ ///
+ /// The result is also clamped to at least one register in each bank. Zero
+ /// means "no register pressure" to the occupancy calculation, so it cannot
+ /// express an occupancy cap even for a function whose literal usage is zero.
+ std::pair<uint64_t, uint64_t>
+ getEffectiveNumGPRsForWavesPerEU(unsigned MaxWaves, uint64_t NumSGPRs,
+ uint64_t NumVGPRs,
+ unsigned DynamicVGPRBlockSize) const;
+
/// \returns true if the flat_scratch register should be initialized with the
/// pointer to the wave's scratch memory rather than a size and offset.
bool flatScratchIsPointer() const {
diff --git a/llvm/lib/Target/AMDGPU/MCTargetDesc/AMDGPUTargetStreamer.cpp b/llvm/lib/Target/AMDGPU/MCTargetDesc/AMDGPUTargetStreamer.cpp
index 97bcec66fe318..a72d38def748c 100644
--- a/llvm/lib/Target/AMDGPU/MCTargetDesc/AMDGPUTargetStreamer.cpp
+++ b/llvm/lib/Target/AMDGPU/MCTargetDesc/AMDGPUTargetStreamer.cpp
@@ -758,6 +758,8 @@ void AMDGPUTargetAsmStreamer::emitAMDGPUInfo(
OS << "\t\t.amdgpu_num_agpr " << Info->NumAccVGPR << '\n';
OS << "\t\t.amdgpu_private_segment_size " << Info->PrivateSegmentSize
<< '\n';
+ if (Info->Occupancy)
+ OS << "\t\t.amdgpu_occupancy " << Info->Occupancy << '\n';
}
for (MCSymbol *Res : Uses)
OS << "\t\t.amdgpu_use " << Res->getName() << '\n';
@@ -1232,6 +1234,8 @@ void AMDGPUTargetELFStreamer::emitAMDGPUInfo(
EmitU32Entry(AMDGPU::InfoKind::INFO_NUM_AGPR, Info->NumAccVGPR);
EmitU32Entry(AMDGPU::InfoKind::INFO_PRIVATE_SEGMENT_SIZE,
Info->PrivateSegmentSize);
+ if (Info->Occupancy)
+ EmitU32Entry(AMDGPU::InfoKind::INFO_OCCUPANCY, Info->Occupancy);
}
for (MCSymbol *Res : Uses)
diff --git a/llvm/lib/Target/AMDGPU/MCTargetDesc/AMDGPUTargetStreamer.h b/llvm/lib/Target/AMDGPU/MCTargetDesc/AMDGPUTargetStreamer.h
index c42ec5c2683cf..dd0759c573606 100644
--- a/llvm/lib/Target/AMDGPU/MCTargetDesc/AMDGPUTargetStreamer.h
+++ b/llvm/lib/Target/AMDGPU/MCTargetDesc/AMDGPUTargetStreamer.h
@@ -35,6 +35,7 @@ struct FuncInfo {
uint32_t NumArchVGPR = 0;
uint32_t NumAccVGPR = 0;
uint32_t PrivateSegmentSize = 0;
+ uint32_t Occupancy = 0;
bool UsesVCC = false;
bool UsesFlatScratch = false;
bool HasDynStack = false;
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
index 588ca588819cc..d2a9354597252 100644
--- a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
@@ -219,6 +219,14 @@ unsigned getAMDHSACodeObjectVersion(const Module &M) {
return getDefaultAMDHSACodeObjectVersion();
}
+unsigned getAMDGPUABIWavesPerEU(const Module &M) {
+ if (auto *Val = mdconst::extract_or_null<ConstantInt>(
+ M.getModuleFlag("amdgpu_abi_waves_per_eu")))
+ return Val->getZExtValue();
+
+ return 0;
+}
+
unsigned getDefaultAMDHSACodeObjectVersion() {
return DefaultAMDHSACodeObjectVersion;
}
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
index 716d6b54a4d20..a2a9ebae04ff1 100644
--- a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
+++ b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
@@ -75,6 +75,9 @@ bool isHsaAbi(const MCSubtargetInfo &STI);
/// \returns Code object version from the IR module flag.
unsigned getAMDHSACodeObjectVersion(const Module &M);
+/// \returns ABI occupancy override in waves per EU from the IR module flag.
+unsigned getAMDGPUABIWavesPerEU(const Module &M);
+
/// \returns Code object version from ELF's e_ident[EI_ABIVERSION].
unsigned getAMDHSACodeObjectVersion(unsigned ABIVersion);
diff --git a/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-device-no-attr.ll b/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-device-no-attr.ll
new file mode 100644
index 0000000000000..7a9d778fe36da
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-device-no-attr.ll
@@ -0,0 +1,27 @@
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -amdgpu-enable-object-linking < %s | FileCheck -check-prefix=ABI %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 < %s | FileCheck -check-prefix=NOABI %s
+
+; Device function with no amdgpu-waves-per-eu attribute. Under object linking
+; the ABI occupancy is used as a register-budget floor without changing the
+; source-level waves-per-EU range. Per-function resource reporting is local-only
+; under object linking, so only the function-level `.num_vgpr` / `.num_agpr`
+; symbols are emitted -- the module-level `amdgpu.max_num_*` symbols are
+; suppressed.
+
+; ABI-LABEL: {{^}}device_fn:
+; ABI: .set .Ldevice_fn.num_vgpr, 3
+; ABI: .set .Ldevice_fn.num_agpr, 0
+
+; ABI-NOT: amdgpu.max_num_
+
+; NOABI-LABEL: {{^}}device_fn:
+; NOABI: .set .Ldevice_fn.num_vgpr, 3
+; NOABI: .set .Ldevice_fn.num_agpr, 0
+
+; NOABI: .set amdgpu.max_num_vgpr, 3
+; NOABI: .set amdgpu.max_num_agpr, 0
+
+define void @device_fn(ptr addrspace(1) %p) {
+ store i32 0, ptr addrspace(1) %p
+ ret void
+}
diff --git a/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-flat-workgroup-lower.ll b/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-flat-workgroup-lower.ll
new file mode 100644
index 0000000000000..4ac1c35908935
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-flat-workgroup-lower.ll
@@ -0,0 +1,24 @@
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -mattr=+wavefrontsize64 -amdgpu-enable-object-linking < %s | FileCheck --check-prefixes=GCN,W64 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1010 -mattr=+wavefrontsize32 -amdgpu-enable-object-linking < %s | FileCheck --check-prefixes=GCN,W32 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1010 -mattr=+wavefrontsize64 -amdgpu-enable-object-linking < %s | FileCheck --check-prefixes=GCN,W64 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1100 -mattr=+wavefrontsize32 -amdgpu-enable-object-linking < %s | FileCheck --check-prefixes=GCN,W32 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1100 -mattr=+wavefrontsize64 -amdgpu-enable-object-linking < %s | FileCheck --check-prefixes=GCN,W64 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1200 -mattr=+wavefrontsize32 -amdgpu-enable-object-linking < %s | FileCheck --check-prefixes=GCN,W32 %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1200 -mattr=+wavefrontsize64 -amdgpu-enable-object-linking < %s | FileCheck --check-prefixes=GCN,W64 %s
+
+; amdgpu-flat-work-group-size is ABI-significant and can lower the ABI
+; occupancy implied by the default 1024-workitem workgroup.
+
+; GCN-LABEL: {{^}}fixed_vgpr_flat_lower:
+; GCN: .set .Lfixed_vgpr_flat_lower.num_vgpr, 71
+; W64: .amdgpu_occupancy 2
+; W32: .amdgpu_occupancy 4
+; GCN-NOT: amdgpu.max_num_
+
+define amdgpu_kernel void @fixed_vgpr_flat_lower(ptr addrspace(1) %p) #0 {
+ call void asm sideeffect "; clobber", "~{v70}"()
+ store i32 0, ptr addrspace(1) %p
+ ret void
+}
+
+attributes #0 = { "amdgpu-flat-work-group-size"="1,512" }
diff --git a/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-flat-workgroup-reject-override.ll b/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-flat-workgroup-reject-override.ll
new file mode 100644
index 0000000000000..6812e2b47fe95
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-flat-workgroup-reject-override.ll
@@ -0,0 +1,28 @@
+; RUN: not llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -mattr=+wavefrontsize64 -amdgpu-enable-object-linking -filetype=null < %s 2>&1 | FileCheck --check-prefix=GFX9 %s
+; RUN: not llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1010 -mattr=+wavefrontsize32 -amdgpu-enable-object-linking -filetype=null < %s 2>&1 | FileCheck --check-prefix=GFX10 %s
+; RUN: not llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1010 -mattr=+wavefrontsize64 -amdgpu-enable-object-linking -filetype=null < %s 2>&1 | FileCheck --check-prefix=GFX10 %s
+; RUN: not llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1100 -mattr=+wavefrontsize32 -amdgpu-enable-object-linking -filetype=null < %s 2>&1 | FileCheck --check-prefix=GFX11 %s
+; RUN: not llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1100 -mattr=+wavefrontsize64 -amdgpu-enable-object-linking -filetype=null < %s 2>&1 | FileCheck --check-prefix=GFX11 %s
+; RUN: not llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1200 -mattr=+wavefrontsize32 -amdgpu-enable-object-linking -filetype=null < %s 2>&1 | FileCheck --check-prefix=GFX12 %s
+; RUN: not llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1200 -mattr=+wavefrontsize64 -amdgpu-enable-object-linking -filetype=null < %s 2>&1 | FileCheck --check-prefix=GFX12 %s
+
+; amdgpu-flat-work-group-size is ABI-significant. Even when the default ABI
+; occupancy is overridden to 2 waves/EU by a module flag, a flat workgroup size
+; of 1024 requires the per-target occupancy implied by a 1024-workitem
+; workgroup.
+
+; GFX9: error: {{.*}}VGPRs under object-linking ABI (193) exceeds limit (64) in function 'fixed_vgpr_flat'
+; GFX10: error: {{.*}}VGPRs under object-linking ABI (193) exceeds limit (128) in function 'fixed_vgpr_flat'
+; GFX11: error: {{.*}}VGPRs under object-linking ABI (193) exceeds limit (192) in function 'fixed_vgpr_flat'
+; GFX12: error: {{.*}}VGPRs under object-linking ABI (193) exceeds limit (192) in function 'fixed_vgpr_flat'
+
+define amdgpu_kernel void @fixed_vgpr_flat(ptr addrspace(1) %p) #0 {
+ call void asm sideeffect "; clobber", "~{v192}"()
+ store i32 0, ptr addrspace(1) %p
+ ret void
+}
+
+attributes #0 = { "amdgpu-flat-work-group-size"="1,1024" }
+
+!llvm.module.flags = !{!0}
+!0 = !{i32 1, !"amdgpu_abi_waves_per_eu", i32 2}
diff --git a/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-kernel-no-attr.ll b/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-kernel-no-attr.ll
new file mode 100644
index 0000000000000..a4f4f2be85c7d
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-kernel-no-attr.ll
@@ -0,0 +1,31 @@
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -amdgpu-enable-object-linking < %s | FileCheck -check-prefix=ABI %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 < %s | FileCheck -check-prefix=NOABI %s
+
+; Kernel with no amdgpu-waves-per-eu attribute. Under object linking the ABI
+; occupancy is used as a register-budget floor, but it is not modeled as an
+; exact maximum occupancy request. The kernel descriptor therefore still reports
+; the local resource usage and resulting occupancy. Per-function resource
+; reporting is local-only under object linking, so no module-level
+; `amdgpu.max_num_*` symbols are emitted.
+
+; ABI-LABEL: {{^}}kernel_no_attr:
+; ABI: .set .Lkernel_no_attr.num_vgpr, 1
+; ABI: NumVGPRsForWavesPerEU: 1
+; ABI: Occupancy: 10
+; ABI: .amdgpu_occupancy 4
+
+; ABI-NOT: amdgpu.max_num_
+
+; NOABI-LABEL: {{^}}kernel_no_attr:
+; NOABI: .set .Lkernel_no_attr.num_vgpr, 1
+; NOABI: NumVGPRsForWavesPerEU: 1
+; NOABI: Occupancy: 10
+
+; NOABI: .set amdgpu.max_num_vgpr, 0
+; NOABI: .set amdgpu.max_num_agpr, 0
+; NOABI: .set amdgpu.max_num_sgpr, 0
+
+define amdgpu_kernel void @kernel_no_attr(ptr addrspace(1) %p) {
+ store i32 0, ptr addrspace(1) %p
+ ret void
+}
diff --git a/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-override.ll b/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-override.ll
new file mode 100644
index 0000000000000..5c71064e092f7
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-override.ll
@@ -0,0 +1,30 @@
+; RUN: split-file %s %t
+; RUN: not llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -amdgpu-enable-object-linking -filetype=null < %t/default.ll 2>&1 | FileCheck -check-prefix=DEFAULT %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -amdgpu-enable-object-linking < %t/override.ll | FileCheck -check-prefix=OVERRIDE %s
+
+; The default object-linking ABI occupancy on gfx900 is 4 waves/EU, which gives
+; a 64 VGPR budget. Overriding the ABI occupancy to 2 waves/EU with the
+; amdgpu_abi_waves_per_eu module flag raises the budget to 128 VGPRs.
+
+; DEFAULT: error: {{.*}}VGPRs under object-linking ABI (71) exceeds limit (64) in function 'fixed_vgpr'
+
+; OVERRIDE-LABEL: {{^}}fixed_vgpr:
+; OVERRIDE: .set .Lfixed_vgpr.num_vgpr, 71
+; OVERRIDE-NOT: amdgpu.max_num_
+
+;--- default.ll
+define amdgpu_kernel void @fixed_vgpr(ptr addrspace(1) %p) {
+ call void asm sideeffect "; clobber", "~{v70}"()
+ store i32 0, ptr addrspace(1) %p
+ ret void
+}
+
+;--- override.ll
+define amdgpu_kernel void @fixed_vgpr(ptr addrspace(1) %p) {
+ call void asm sideeffect "; clobber", "~{v70}"()
+ store i32 0, ptr addrspace(1) %p
+ ret void
+}
+
+!llvm.module.flags = !{!0}
+!0 = !{i32 1, !"amdgpu_abi_waves_per_eu", i32 2}
diff --git a/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-preserves-attr.ll b/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-preserves-attr.ll
new file mode 100644
index 0000000000000..8182ce77d3ed1
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/object-linking-abi-occupancy-preserves-attr.ll
@@ -0,0 +1,34 @@
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -amdgpu-enable-object-linking < %s | FileCheck -check-prefix=ABI %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 < %s | FileCheck -check-prefix=NOABI %s
+
+; Object-linking ABI occupancy is a budget floor, not a synthetic
+; amdgpu-waves-per-eu range. A max-occupancy hint that is weaker than the ABI
+; floor does not force resource inflation below the ABI floor.
+;
+; Here the kernel requests amdgpu-waves-per-eu="1,1", but the effective budget
+; remains compatible with the ABI. The kernel uses 30 VGPRs via inline asm, so
+; the final occupancy is resource-derived. No module-level `amdgpu.max_num_*`
+; symbols are emitted under object linking.
+
+; ABI-LABEL: {{^}}kernel_with_attr:
+; ABI: .set .Lkernel_with_attr.num_vgpr, 30
+; ABI: NumVGPRsForWavesPerEU: 30
+; ABI: Occupancy: 8
+; ABI: .amdgpu_occupancy 4
+
+; ABI-NOT: amdgpu.max_num_
+
+; NOABI-LABEL: {{^}}kernel_with_attr:
+; NOABI: .set .Lkernel_with_attr.num_vgpr, 30
+; NOABI: NumVGPRsForWavesPerEU: 30
+; NOABI: Occupancy: 8
+
+; NOABI: .set amdgpu.max_num_vgpr, 0
+
+define amdgpu_kernel void @kernel_with_attr(ptr addrspace(1) %p) #0 {
+ %r = call { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 } asm sideeffect "; clobber", "=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v,=v"()
+ store i32 0, ptr addrspace(1) %p
+ ret void
+}
+
+attributes #0 = { "amdgpu-waves-per-eu"="1,1" }
diff --git a/llvm/test/Linker/amdgpu-abi-waves-per-eu-module-flag.ll b/llvm/test/Linker/amdgpu-abi-waves-per-eu-module-flag.ll
new file mode 100644
index 0000000000000..3b7d98312611f
--- /dev/null
+++ b/llvm/test/Linker/amdgpu-abi-waves-per-eu-module-flag.ll
@@ -0,0 +1,16 @@
+; RUN: split-file %s %t
+; RUN: not llvm-link %t/a.ll %t/b.ll -S -o /dev/null 2>&1 | FileCheck %s
+
+; CHECK: error: linking module flags 'amdgpu_abi_waves_per_eu': IDs have conflicting values
+
+;--- a.ll
+target triple = "amdgcn-amd-amdhsa"
+
+!llvm.module.flags = !{!0}
+!0 = !{i32 1, !"amdgpu_abi_waves_per_eu", i32 2}
+
+;--- b.ll
+target triple = "amdgcn-amd-amdhsa"
+
+!llvm.module.flags = !{!0}
+!0 = !{i32 1, !"amdgpu_abi_waves_per_eu", i32 4}
diff --git a/llvm/test/MC/AMDGPU/amdgpu-info-roundtrip.s b/llvm/test/MC/AMDGPU/amdgpu-info-roundtrip.s
index d49890eb05174..72406e3ad60a6 100644
--- a/llvm/test/MC/AMDGPU/amdgpu-info-roundtrip.s
+++ b/llvm/test/MC/AMDGPU/amdgpu-info-roundtrip.s
@@ -40,6 +40,7 @@ addr_taken_func:
.amdgpu_num_vgpr 32
.amdgpu_num_agpr 4
.amdgpu_private_segment_size 0
+ .amdgpu_occupancy 4
.amdgpu_use lds_var
.amdgpu_call helper
.amdgpu_indirect_call "vi"
@@ -53,6 +54,7 @@ addr_taken_func:
.amdgpu_num_sgpr 8
.amdgpu_num_vgpr 10
.amdgpu_private_segment_size 16
+ .amdgpu_occupancy 8
.amdgpu_call extern_func
.end_amdgpu_info
@@ -62,6 +64,7 @@ addr_taken_func:
.amdgpu_num_sgpr 2
.amdgpu_num_vgpr 4
.amdgpu_private_segment_size 0
+ .amdgpu_occupancy 10
.amdgpu_typeid "vi"
.end_amdgpu_info
@@ -71,6 +74,7 @@ addr_taken_func:
// ASM: .amdgpu_num_vgpr 32
// ASM: .amdgpu_num_agpr 4
// ASM: .amdgpu_private_segment_size 0
+// ASM: .amdgpu_occupancy 4
// ASM: .amdgpu_use lds_var
// ASM: .amdgpu_call helper
// ASM: .amdgpu_indirect_call "vi"
@@ -82,6 +86,7 @@ addr_taken_func:
// ASM: .amdgpu_num_vgpr 10
// ASM-NOT: .amdgpu_num_agpr
// ASM: .amdgpu_private_segment_size 16
+// ASM: .amdgpu_occupancy 8
// ASM: .amdgpu_call extern_func
// ASM: .end_amdgpu_info
@@ -91,6 +96,7 @@ addr_taken_func:
// ASM: .amdgpu_num_vgpr 4
// ASM-NOT: .amdgpu_num_agpr
// ASM: .amdgpu_private_segment_size 0
+// ASM: .amdgpu_occupancy 10
// ASM: .amdgpu_typeid "vi"
// ASM: .end_amdgpu_info
More information about the llvm-commits
mailing list