[llvm-branch-commits] [llvm] [AMDGPU] Verify VGPR tuple alignment from the operand register class (PR #219230)
Valery Pykhtin via llvm-branch-commits
llvm-branch-commits at lists.llvm.org
Thu Aug 27 08:30:53 PDT 2026
https://github.com/vpykhtin created https://github.com/llvm/llvm-project/pull/219230
The machine verifier decided VGPR tuple alignment with isProperlyAlignedRC(),
which inspects only the register's own class. Alignment is not really a property
of the register in isolation: whether a 64-bit tuple must be even-aligned depends
on the operand it feeds, and on mixed-alignment targets the same register class
can be required to be aligned in one operand and exempt in another. Inspecting
only the register also conflates alignment with unrelated problems - a register
that is simply the wrong bank or size for the operand came out as "requires even
aligned vector registers" as well.
Make the operand's register class the source of truth instead: a register is
misaligned only when it does not satisfy the operand's class but its even-aligned
same-bank/width equivalent (SIRegisterInfo::getAlignedEquivalentRC) would. A
register that fits neither is a genuine class or bank mismatch and is left to the
illegal-register and sub-register checks. So an AGPR in a VGPR|SGPR (VS_64)
operand is now reported as an illegal register, and a wrong-size register (e.g. a
64-bit VGPR in a 128-bit MFMA source) or an invalid sub-register index is
reported by those checks alone, no longer doubled up as an "even aligned" error.
This drops the redundant diagnostics in tests.
Deriving the requirement from the operand class also lets several special cases
go away. The RegClass == -1 early-out is hoisted so the operand class is always
valid, and the V_MOV_B64_PSEUDO / AV_MOV_B64_IMM_PSEUDO / spill exemptions are
dropped: those operands use unaligned register classes (VReg_64, AV_64, and the
spill classes), which every register already satisfies, so the comparison never
flags them. Inline-asm operands (RegClass == -1) are no longer alignment-checked,
matching the prior FIXME that they were never meaningfully verified.
The same reasoning removes the DS_GWS-specific alignment check: on subtargets
that require aligned VGPRs the DS_GWS data0 operand has the AV_64_Align2 register
class, so the operand-class check above already diagnoses its alignment and the
separate check is redundant. The image vaddr operand is a plain VGPR_32 whose
class cannot encode even-alignment, so its dedicated position check is kept.
Co-Authored-By: Claude <noreply at anthropic.com>
>From 423d6bdbad69ba7927df59cc475134fe14e9fc97 Mon Sep 17 00:00:00 2001
From: Valery Pykhtin <valery.pykhtin at amd.com>
Date: Wed, 26 Aug 2026 17:57:52 +0000
Subject: [PATCH] [AMDGPU] Verify VGPR tuple alignment from the operand
register class
The machine verifier decided VGPR tuple alignment with isProperlyAlignedRC(),
which inspects only the register's own class. Alignment is not really a property
of the register in isolation: whether a 64-bit tuple must be even-aligned depends
on the operand it feeds, and on mixed-alignment targets the same register class
can be required to be aligned in one operand and exempt in another. Inspecting
only the register also conflates alignment with unrelated problems - a register
that is simply the wrong bank or size for the operand came out as "requires even
aligned vector registers" as well.
Make the operand's register class the source of truth instead: a register is
misaligned only when it does not satisfy the operand's class but its even-aligned
same-bank/width equivalent (SIRegisterInfo::getAlignedEquivalentRC) would. A
register that fits neither is a genuine class or bank mismatch and is left to the
illegal-register and sub-register checks. So an AGPR in a VGPR|SGPR (VS_64)
operand is now reported as an illegal register, and a wrong-size register (e.g. a
64-bit VGPR in a 128-bit MFMA source) or an invalid sub-register index is
reported by those checks alone, no longer doubled up as an "even aligned" error.
This drops the redundant diagnostics in tests.
Deriving the requirement from the operand class also lets several special cases
go away. The RegClass == -1 early-out is hoisted so the operand class is always
valid, and the V_MOV_B64_PSEUDO / AV_MOV_B64_IMM_PSEUDO / spill exemptions are
dropped: those operands use unaligned register classes (VReg_64, AV_64, and the
spill classes), which every register already satisfies, so the comparison never
flags them. Inline-asm operands (RegClass == -1) are no longer alignment-checked,
matching the prior FIXME that they were never meaningfully verified.
The same reasoning removes the DS_GWS-specific alignment check: on subtargets
that require aligned VGPRs the DS_GWS data0 operand has the AV_64_Align2 register
class, so the operand-class check above already diagnoses its alignment and the
separate check is redundant. The image vaddr operand is a plain VGPR_32 whose
class cannot encode even-alignment, so its dedicated position check is kept.
Co-Authored-By: Claude <noreply at anthropic.com>
---
llvm/lib/Target/AMDGPU/SIInstrInfo.cpp | 70 +++++++++----------
llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp | 15 ++++
llvm/lib/Target/AMDGPU/SIRegisterInfo.h | 10 +++
.../CodeGen/AMDGPU/verify-ds-gws-align.mir | 10 ---
.../AMDGPU/verify-gfx90a-aligned-vgprs.mir | 10 ---
...ported-subreg-index-aligned-vgpr-check.mir | 4 --
...rted-unaligned-vgpr-check-vsrc-operand.mir | 2 -
7 files changed, 57 insertions(+), 64 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp b/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
index bc419d813f171..8ba0d47572afa 100644
--- a/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
@@ -5464,44 +5464,48 @@ bool SIInstrInfo::verifyInstruction(const MachineInstr &MI,
break;
}
+ // Operands without a fixed register class (RegClass == -1), such as inline
+ // asm operands, are not verified here.
+ if (RegClass == -1)
+ continue;
+
if (!MO.isReg())
continue;
Register Reg = MO.getReg();
if (!Reg)
continue;
- // FIXME: Ideally we would have separate instruction definitions with the
- // aligned register constraint.
- // FIXME: We do not verify inline asm operands, but custom inline asm
- // verification is broken anyway
- if (ST.needsAlignedVGPRs() && Opcode != AMDGPU::AV_MOV_B64_IMM_PSEUDO &&
- Opcode != AMDGPU::V_MOV_B64_PSEUDO && !isSpill(MI)) {
- const TargetRegisterClass *RC = RI.getRegClassForReg(MRI, Reg);
- if (RI.hasVectorRegisters(RC) && MO.getSubReg()) {
- if (const TargetRegisterClass *SubRC =
- RI.getSubRegisterClass(RC, MO.getSubReg())) {
- RC = RI.getCompatibleSubRegClass(RC, SubRC, MO.getSubReg());
- if (RC)
- RC = SubRC;
+ const TargetRegisterClass *OpRC = RI.getRegClass(RegClass);
+
+ if (ST.needsAlignedVGPRs()) {
+ const TargetRegisterClass *RegRC = RI.getRegClassForReg(MRI, Reg);
+ if (RI.hasVectorRegisters(RegRC)) {
+ if (MO.getSubReg()) {
+ // Narrow to the sub-register's class. getSubRegisterClass already
+ // accounts for the sub-register index's alignment within the tuple
+ // (an odd-aligned slice yields an unaligned class, caught below); a
+ // null result means an invalid sub-register index and is left to the
+ // sub-register check.
+ RegRC = RI.getSubRegisterClass(RegRC, MO.getSubReg());
+ }
+ // Flag an alignment-only mismatch: the register does not satisfy the
+ // operand's class, but its even-aligned same-bank/width equivalent
+ // would. A bank or size mismatch fails even when aligned, so it is left
+ // to the illegal-register / sub-register checks.
+ if (RegRC && !OpRC->hasSubClassEq(RegRC)) {
+ const TargetRegisterClass *AlignedRegRC =
+ RI.getAlignedEquivalentRC(RegRC);
+ if (AlignedRegRC && OpRC->hasSubClassEq(AlignedRegRC)) {
+ ErrInfo = "Subtarget requires even aligned vector registers";
+ return false;
+ }
}
- }
-
- // Check that this is the aligned version of the class.
- if (!RC || !RI.isProperlyAlignedRC(*RC)) {
- ErrInfo = "Subtarget requires even aligned vector registers";
- return false;
}
}
- if (RegClass != -1) {
- if (Reg.isVirtual())
- continue;
-
- const TargetRegisterClass *RC = RI.getRegClass(RegClass);
- if (!RC->contains(Reg)) {
- ErrInfo = "Operand has incorrect register class.";
- return false;
- }
+ if (Reg.isPhysical() && !OpRC->contains(Reg)) {
+ ErrInfo = "Operand has incorrect register class.";
+ return false;
}
}
@@ -6048,16 +6052,6 @@ bool SIInstrInfo::verifyInstruction(const MachineInstr &MI,
!(RI.getChannelFromSubReg(Op->getSubReg()) & 1);
};
- if (Opcode == AMDGPU::DS_GWS_INIT || Opcode == AMDGPU::DS_GWS_SEMA_BR ||
- Opcode == AMDGPU::DS_GWS_BARRIER) {
-
- if (!isAlignedReg(AMDGPU::OpName::data0)) {
- ErrInfo = "Subtarget requires even aligned vector registers "
- "for DS_GWS instructions";
- return false;
- }
- }
-
if (isMIMG(MI)) {
if (!isAlignedReg(AMDGPU::OpName::vaddr)) {
ErrInfo = "Subtarget requires even aligned vector registers "
diff --git a/llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp b/llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp
index 2b0d9fd380793..62a32808bb3f9 100644
--- a/llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp
@@ -3874,6 +3874,21 @@ SIRegisterInfo::getDefaultVectorSuperClassForBitWidth(unsigned BitWidth) const {
: getVGPRClassForBitWidth(BitWidth);
}
+const TargetRegisterClass *
+SIRegisterInfo::getAlignedEquivalentRC(const TargetRegisterClass *RC) const {
+ assert(RC->isAllocatable() &&
+ "expected an allocatable register class; non-allocatable VS_* operand "
+ "classes cannot be a register's class");
+ unsigned Width = getRegSizeInBits(*RC);
+ if (isVGPRClass(RC))
+ return getVGPRClassForBitWidth(Width);
+ if (isAGPRClass(RC))
+ return getAGPRClassForBitWidth(Width);
+ if (isVectorSuperClass(RC))
+ return getVectorSuperClassForBitWidth(Width);
+ return nullptr;
+}
+
const TargetRegisterClass *
SIRegisterInfo::getSGPRClassForBitWidth(unsigned BitWidth) {
if (BitWidth == 16 || BitWidth == 32)
diff --git a/llvm/lib/Target/AMDGPU/SIRegisterInfo.h b/llvm/lib/Target/AMDGPU/SIRegisterInfo.h
index e464c9334ffea..b96a01404d3bc 100644
--- a/llvm/lib/Target/AMDGPU/SIRegisterInfo.h
+++ b/llvm/lib/Target/AMDGPU/SIRegisterInfo.h
@@ -226,6 +226,16 @@ class SIRegisterInfo final : public AMDGPUGenRegisterInfo {
const TargetRegisterClass *
getDefaultVectorSuperClassForBitWidth(unsigned BitWidth) const;
+ /// \returns the even-aligned register class with the same register bank and
+ /// width as \p RC (its aligned "canonical" class), or null if \p RC has no
+ /// vector registers. Only meaningful on subtargets that require aligned
+ /// VGPRs, where the per-bitwidth getters return the aligned variant. \p RC
+ /// must be an allocatable register's class; the non-allocatable VS_* operand
+ /// classes are intentionally not handled.
+ LLVM_READONLY
+ const TargetRegisterClass *
+ getAlignedEquivalentRC(const TargetRegisterClass *RC) const;
+
LLVM_READONLY
static const TargetRegisterClass *getSGPRClassForBitWidth(unsigned BitWidth);
diff --git a/llvm/test/CodeGen/AMDGPU/verify-ds-gws-align.mir b/llvm/test/CodeGen/AMDGPU/verify-ds-gws-align.mir
index 8a7fc2a3af057..feffe3d27e451 100644
--- a/llvm/test/CodeGen/AMDGPU/verify-ds-gws-align.mir
+++ b/llvm/test/CodeGen/AMDGPU/verify-ds-gws-align.mir
@@ -10,29 +10,21 @@ body: |
bb.0:
; A 32-bit sub-register does not fit the 64-bit AV_64_Align2 data0 operand.
%0:areg_128_align2 = IMPLICIT_DEF
- ; GFX90A-ERR: *** Bad machine code: Subtarget requires even aligned vector registers for DS_GWS instructions ***
- ; GFX90A-ERR: - instruction: DS_GWS_INIT killed %0.sub1:areg_128_align2
; GFX90A-ERR: *** Bad machine code: Illegal virtual register for instruction ***
; GFX90A-ERR: - instruction: DS_GWS_INIT killed %0.sub1:areg_128_align2
; GFX90A-ERR: AReg_128_Align2.sub1 cannot be used for AV_64_Align2 operands.
DS_GWS_INIT killed %0.sub1, 0, implicit $m0, implicit $exec :: (store (s32) into custom "GWSResource")
%0:areg_128_align2 = IMPLICIT_DEF
- ; GFX90A-ERR: *** Bad machine code: Subtarget requires even aligned vector registers for DS_GWS instructions ***
- ; GFX90A-ERR: - instruction: DS_GWS_INIT killed %0.sub3:areg_128_align2
; GFX90A-ERR: *** Bad machine code: Illegal virtual register for instruction ***
; GFX90A-ERR: - instruction: DS_GWS_INIT killed %0.sub3:areg_128_align2
; GFX90A-ERR: AReg_128_Align2.sub3 cannot be used for AV_64_Align2 operands.
DS_GWS_INIT killed %0.sub3, 0, implicit $m0, implicit $exec :: (store (s32) into custom "GWSResource")
%1:vreg_64_align2 = IMPLICIT_DEF
- ; GFX90A-ERR: *** Bad machine code: Subtarget requires even aligned vector registers for DS_GWS instructions ***
- ; GFX90A-ERR: - instruction: DS_GWS_SEMA_BR killed %1.sub1:vreg_64_align2
; GFX90A-ERR: *** Bad machine code: Illegal virtual register for instruction ***
; GFX90A-ERR: - instruction: DS_GWS_SEMA_BR killed %1.sub1:vreg_64_align2
; GFX90A-ERR: VReg_64_Align2.sub1 cannot be used for AV_64_Align2 operands.
DS_GWS_SEMA_BR killed %1.sub1, 0, implicit $m0, implicit $exec :: (store (s32) into custom "GWSResource")
%2:vreg_64 = IMPLICIT_DEF
- ; GFX90A-ERR: *** Bad machine code: Subtarget requires even aligned vector registers for DS_GWS instructions ***
- ; GFX90A-ERR: - instruction: DS_GWS_BARRIER killed %2.sub0:vreg_64
; GFX90A-ERR: *** Bad machine code: Illegal virtual register for instruction ***
; GFX90A-ERR: - instruction: DS_GWS_BARRIER killed %2.sub0:vreg_64
; GFX90A-ERR: VReg_64.sub0 cannot be used for AV_64_Align2 operands.
@@ -40,8 +32,6 @@ body: |
; A whole 32-bit register is the wrong size for the 64-bit operand.
%3:vgpr_32 = IMPLICIT_DEF
- ; GFX90A-ERR: *** Bad machine code: Subtarget requires even aligned vector registers for DS_GWS instructions ***
- ; GFX90A-ERR: - instruction: DS_GWS_INIT killed %3:vgpr_32
; GFX90A-ERR: *** Bad machine code: Illegal virtual register for instruction ***
; GFX90A-ERR: - instruction: DS_GWS_INIT killed %3:vgpr_32
; GFX90A-ERR: Expected a AV_64_Align2 register, but got a VGPR_32 register
diff --git a/llvm/test/CodeGen/AMDGPU/verify-gfx90a-aligned-vgprs.mir b/llvm/test/CodeGen/AMDGPU/verify-gfx90a-aligned-vgprs.mir
index f8f8197cbc005..8cec088873a94 100644
--- a/llvm/test/CodeGen/AMDGPU/verify-gfx90a-aligned-vgprs.mir
+++ b/llvm/test/CodeGen/AMDGPU/verify-gfx90a-aligned-vgprs.mir
@@ -193,16 +193,12 @@ body: |
$vgpr0_vgpr1 = V_PK_ADD_F32 0, %13, 11, %12, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
; src %14 is an AGPR - a bank mismatch, not an alignment problem
- ; CHECK: *** Bad machine code: Subtarget requires even aligned vector registers ***
- ; CHECK: - instruction: $vgpr0_vgpr1 = V_PK_ADD_F32 0, %13:vreg_64_align2, 11, %14.sub1_sub2:areg_96_align2
; CHECK: *** Bad machine code: Illegal virtual register for instruction ***
; CHECK: - instruction: $vgpr0_vgpr1 = V_PK_ADD_F32 0, %13:vreg_64_align2, 11, %14.sub1_sub2:areg_96_align2
; CHECK: AReg_96_Align2.sub1_sub2 cannot be used for VS_64_Align2 operands.
$vgpr0_vgpr1 = V_PK_ADD_F32 0, %13, 11, %14.sub1_sub2, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
; src %14 is an AGPR - a bank mismatch, not an alignment problem
- ; CHECK: *** Bad machine code: Subtarget requires even aligned vector registers ***
- ; CHECK: - instruction: $vgpr0_vgpr1 = V_PK_ADD_F32 0, %14.sub1_sub2:areg_96_align2, 11, %13:vreg_64_align2
; CHECK: *** Bad machine code: Illegal virtual register for instruction ***
; CHECK: - instruction: $vgpr0_vgpr1 = V_PK_ADD_F32 0, %14.sub1_sub2:areg_96_align2, 11, %13:vreg_64_align2
; CHECK: AReg_96_Align2.sub1_sub2 cannot be used for VS_64_Align2 operands.
@@ -219,16 +215,12 @@ body: |
$vgpr0_vgpr1 = V_PK_MUL_F32 0, %13, 11, %12, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
; src %14 is an AGPR - a bank mismatch, not an alignment problem
- ; CHECK: *** Bad machine code: Subtarget requires even aligned vector registers ***
- ; CHECK: - instruction: $vgpr0_vgpr1 = V_PK_MUL_F32 0, %13:vreg_64_align2, 11, %14.sub1_sub2:areg_96_align2
; CHECK: *** Bad machine code: Illegal virtual register for instruction ***
; CHECK: - instruction: $vgpr0_vgpr1 = V_PK_MUL_F32 0, %13:vreg_64_align2, 11, %14.sub1_sub2:areg_96_align2
; CHECK: AReg_96_Align2.sub1_sub2 cannot be used for VS_64_Align2 operands.
$vgpr0_vgpr1 = V_PK_MUL_F32 0, %13, 11, %14.sub1_sub2, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
; src %14 is an AGPR - a bank mismatch, not an alignment problem
- ; CHECK: *** Bad machine code: Subtarget requires even aligned vector registers ***
- ; CHECK: - instruction: $vgpr0_vgpr1 = V_PK_MUL_F32 0, %14.sub1_sub2:areg_96_align2, 11, %13:vreg_64_align2
; CHECK: *** Bad machine code: Illegal virtual register for instruction ***
; CHECK: - instruction: $vgpr0_vgpr1 = V_PK_MUL_F32 0, %14.sub1_sub2:areg_96_align2, 11, %13:vreg_64_align2
; CHECK: AReg_96_Align2.sub1_sub2 cannot be used for VS_64_Align2 operands.
@@ -249,8 +241,6 @@ body: |
$vgpr0_vgpr1 = nofpexcept V_PK_FMA_F32 8, %13, 8, %12, 11, %14.sub0_sub1, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
; src %14 is an AGPR - a bank mismatch (both %13 sources are aligned)
- ; CHECK: *** Bad machine code: Subtarget requires even aligned vector registers ***
- ; CHECK: - instruction: $vgpr0_vgpr1 = nofpexcept V_PK_FMA_F32 8, %13:vreg_64_align2, 8, %13:vreg_64_align2, 11, %14.sub1_sub2:areg_96_align2
; CHECK: *** Bad machine code: Illegal virtual register for instruction ***
; CHECK: - instruction: $vgpr0_vgpr1 = nofpexcept V_PK_FMA_F32 8, %13:vreg_64_align2, 8, %13:vreg_64_align2, 11, %14.sub1_sub2:areg_96_align2
; CHECK: AReg_96_Align2.sub1_sub2 cannot be used for VS_64_Align2 operands.
diff --git a/llvm/test/MachineVerifier/AMDGPU/unsupported-subreg-index-aligned-vgpr-check.mir b/llvm/test/MachineVerifier/AMDGPU/unsupported-subreg-index-aligned-vgpr-check.mir
index 5b1aaa6337261..6ddf0f62a6f1c 100644
--- a/llvm/test/MachineVerifier/AMDGPU/unsupported-subreg-index-aligned-vgpr-check.mir
+++ b/llvm/test/MachineVerifier/AMDGPU/unsupported-subreg-index-aligned-vgpr-check.mir
@@ -21,10 +21,6 @@ body: |
; CHECK-NEXT: - operand 1: %1.sub16_sub17_sub18_sub19:vreg_512_align2
; CHECK-NEXT: Register class VReg_512_Align2 does not support subreg index sub16_sub17_sub18_sub19
- ; CHECK: *** Bad machine code: Subtarget requires even aligned vector registers ***
- ; CHECK-NEXT: - function: uses_invalid_subregister_for_regclass
- ; CHECK-NEXT: - basic block: %bb.0
- ; CHECK-NEXT: - instruction: GLOBAL_STORE_DWORDX4_SADDR %0:vgpr_32, %2.sub16_sub17_sub18_sub19:vreg_512, undef $sgpr8_sgpr9, 80, 0, implicit $exec :: (store (s128), addrspace 1)
GLOBAL_STORE_DWORDX4_SADDR %0, %1.sub16_sub17_sub18_sub19, undef $sgpr8_sgpr9, 80, 0, implicit $exec :: (store (s128), addrspace 1)
; Test with unaligned class
diff --git a/llvm/test/MachineVerifier/AMDGPU/unsupported-unaligned-vgpr-check-vsrc-operand.mir b/llvm/test/MachineVerifier/AMDGPU/unsupported-unaligned-vgpr-check-vsrc-operand.mir
index c982b1a35d44a..3df4ce13fc15d 100644
--- a/llvm/test/MachineVerifier/AMDGPU/unsupported-unaligned-vgpr-check-vsrc-operand.mir
+++ b/llvm/test/MachineVerifier/AMDGPU/unsupported-unaligned-vgpr-check-vsrc-operand.mir
@@ -23,12 +23,10 @@ body: |
%1:vgpr_32 = V_MOV_B32_e32 1, implicit $exec
%2:vreg_64 = IMPLICIT_DEF
- ; CHECK: *** Bad machine code: Subtarget requires even aligned vector registers ***
; CHECK: *** Bad machine code: Illegal virtual register for instruction ***
%3:areg_128_align2 = V_MFMA_F32_4X4X1F32_e64 %0, %1, %2, 0, 0, 0, implicit $mode, implicit $exec
%4:vreg_64 = IMPLICIT_DEF
- ; CHECK: *** Bad machine code: Subtarget requires even aligned vector registers ***
; CHECK: *** Bad machine code: Illegal virtual register for instruction ***
%5:vreg_128_align2 = V_MFMA_F32_4X4X1F32_vgprcd_e64 %0, %1, %4, 0, 0, 0, implicit $mode, implicit $exec
...
More information about the llvm-branch-commits
mailing list