[Mlir-commits] [llvm] [mlir] [AMDGPU][mlir] Fix unscaled MFMA fold to require E8M0 1.0 scales (PR #232261)
Keshav Vinayak Jha
llvmlistbot at llvm.org
Fri Oct 16 02:20:39 PDT 2026
https://github.com/keshavvinayak01 updated https://github.com/llvm/llvm-project/pull/232261
>From de86d457215371d9e3333d85f50661c8d14266e6 Mon Sep 17 00:00:00 2001
From: Keshav Vinayak Jha <keshavvinayakjha at gmail.com>
Date: Thu, 15 Oct 2026 10:21:47 +0000
Subject: [PATCH 1/4] [AMDGPU][mlir] Fix unscaled MFMA fold to require E8M0 1.0
scales
`UnscaledMFMAOptimizationPat` folds
`llvm.amdgcn.mfma.scale.f32.{16x16x128,32x32x64}.f8f6f4` to the
unscaled `v_mfma_f32_*_f8f6f4` when both scale operands are constant 0.
The scales are E8M0 (`2^(s-127)`), so 0 is `2^-127`, not 1.0, and the
fold drops that factor. On gfx950, a scaled MFMA of FP8 1.0 inputs with
runtime scales of 0 returns 0, while the same call with constant scales
of 0 is folded and returns 128. A scale of 127, which is the real 1.0,
is never folded.
The pattern now matches `opsel = 0, scale = 127` for both A and B, so
it only fires when the selected scale byte is 1.0. It stays a plain
TableGen pattern, so SelectionDAG and GlobalISel both keep the fold.
Other encodings of 1.0 (for example `opsel = 1` with `0x7f00`) are left
scaled, which is correct but not folded.
The AMDGPUToROCDL lowering of unscaled f8/f6/f4 `amdgpu.mfma` on gfx950
emits `rocdl.mfma.scale.*` with scales of 0 and relied on this fold to
get unscaled results. It now emits 127. This is in the same PR so there
is no commit where MLIR-generated MFMAs are scaled by `2^-254`.
Out-of-tree producers that pass constant 0 expecting no scaling will
now get the scaled instruction.
Tests: tests that used constant 0 scales to select the unscaled
instruction now use 127. `___constant_scale_0_0_a/_b` keep 0 and now
check the scaled instruction. Added 127/127 with `opsel = 1` and 127/0
cases to the 16x16x128 test; both intrinsics share the same pattern.
Ran `check-llvm-codegen-amdgpu` and
`check-mlir-conversion-amdgputorocdl`.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
---
llvm/lib/Target/AMDGPU/SIInstrInfo.td | 5 +-
....amdgcn.mfma.scale.f32.16x16x128.f8f6f4.ll | 424 ++++++++++++------
...m.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.ll | 242 +++++-----
.../rewrite-vgpr-mfma-to-agpr.gfx950.ll | 12 +-
.../AMDGPUToROCDL/AMDGPUToROCDL.cpp | 5 +-
.../Conversion/AMDGPUToROCDL/mfma-gfx950.mlir | 26 +-
6 files changed, 433 insertions(+), 281 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/SIInstrInfo.td b/llvm/lib/Target/AMDGPU/SIInstrInfo.td
index c68da27dcf255c..01e700e1a52875 100644
--- a/llvm/lib/Target/AMDGPU/SIInstrInfo.td
+++ b/llvm/lib/Target/AMDGPU/SIInstrInfo.td
@@ -318,12 +318,13 @@ def SIdenorm_mode : SDNode<"AMDGPUISD::DENORM_MODE",
// Optimize v_mfma_scale* instructions to avoid the scale if the
-// scales are known 0.
+// scales are known 1.0. Scales are E8M0 (2^(s-127)), so 1.0 is 127 in the
+// byte selected by op_sel.
class UnscaledMFMAOptimizationPat<SDPatternOperator intrin> : PatFrag<
(ops node:$srca, node:$srcb, node:$srcc,
node:$cbsz, node:$blgp),
(intrin $srca, $srcb, $srcc, $cbsz, $blgp,
- srcvalue, 0, srcvalue, 0)
+ 0, 127, 0, 127)
>;
def mfma_f32_16x16x128_f8f6f4 : UnscaledMFMAOptimizationPat<int_amdgcn_mfma_scale_f32_16x16x128_f8f6f4>;
diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.ll
index aeeb5958e9ca30..7cce811d8e9362 100644
--- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.ll
+++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.ll
@@ -248,14 +248,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_3_2__cbsz1__blgp1(<8 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp0__constant_scale_0_0(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp0__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp0__constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp0__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19]
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp0__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp0__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
@@ -273,7 +273,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp0__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 0, ; cbsz
i32 0, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -308,14 +308,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp1(<8 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp1__constant_scale_0_0(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp1__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp1__constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp1__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19] blgp:1
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp1__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp1__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
@@ -333,7 +333,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp1__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 0, ; cbsz
i32 1, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -368,14 +368,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp2(<8 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp2__constant_scale_0_0(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp2__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp2__constant_scale_127_127(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp2__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:13], v[14:17] blgp:2
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp2__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp2__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -393,7 +393,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp2__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v6i32(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 0, ; cbsz
i32 2, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -428,14 +428,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp3(<8 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp3__constant_scale_0_0(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp3__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp3__constant_scale_127_127(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp3__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:13], v[14:17] blgp:3
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp3__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp3__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -453,7 +453,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp3__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v6i32(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 0, ; cbsz
i32 3, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -488,14 +488,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp4(<8 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp4__constant_scale_0_0(<8 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp4__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp4__constant_scale_127_127(<8 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp4__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:11], v[12:15] blgp:4
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp4__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp4__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -513,7 +513,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp4__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v4i32(<8 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2,
i32 0, ; cbsz
i32 4, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -548,14 +548,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp0(<8 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp0__constant_scale_0_0(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp0__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp0__constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp0__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19] cbsz:1
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp0__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp0__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
@@ -573,7 +573,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp0__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 1, ; cbsz
i32 0, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -609,14 +609,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp1(<8 x
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp1__constant_scale_0_0(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp1__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp1__constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp1__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19] cbsz:1 blgp:1
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp1__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp1__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
@@ -634,7 +634,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp1__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 1, ; cbsz
i32 1, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -668,14 +668,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp2(<8 x
ret <4 x float> %result
}
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp2__constant_scale_0(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp2__constant_scale_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp2__constant_scale_127_127(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp2__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:13], v[14:17] cbsz:1 blgp:2
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp2__constant_scale_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp2__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -693,7 +693,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp2__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v6i32(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 1, ; cbsz
i32 2, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -728,14 +728,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp3(<8 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp3__constant_scale_0_0(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp3__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp3__constant_scale_127_127(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp3__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:13], v[14:17] cbsz:1 blgp:3
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp3__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp3__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -753,7 +753,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp3__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v6i32(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 1, ; cbsz
i32 3, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -788,14 +788,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp4(<8 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp4__constant_scale_0_0(<8 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp4__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp4__constant_scale_127_127(<8 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp4__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:11], v[12:15] cbsz:1 blgp:4
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp4__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp4__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -813,7 +813,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp4__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v4i32(<8 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2,
i32 1, ; cbsz
i32 4, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -848,14 +848,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp0(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp0__constant_scale_0_0(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp0__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp0__constant_scale_127_127(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp0__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:13], v[14:17] cbsz:2
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp0__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp0__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -873,7 +873,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp0__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v8i32(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 2, ; cbsz
i32 0, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -908,14 +908,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp1(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp1__constant_scale_0_0(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp1__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp1__constant_scale_127_127(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp1__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:13], v[14:17] cbsz:2 blgp:1
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp1__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp1__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -933,7 +933,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp1__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v8i32(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 2, ; cbsz
i32 1, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -968,14 +968,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2__constant_scale_0_0(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2__constant_scale_127_127(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:11], v[12:15] cbsz:2 blgp:2
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -993,7 +993,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 2, ; cbsz
i32 2, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -1028,14 +1028,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp3(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp3__constant_scale_0_0(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp3__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp3__constant_scale_127_127(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp3__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:11], v[12:15] cbsz:2 blgp:3
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp3__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp3__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -1053,7 +1053,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp3__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 2, ; cbsz
i32 3, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -1089,14 +1089,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp0(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp0__constant_scale_0_0(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp0__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp0__constant_scale_127_127(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp0__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:13], v[14:17] cbsz:3
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp0__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp0__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -1114,7 +1114,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp0__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v8i32(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 3, ; cbsz
i32 0, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -1149,14 +1149,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp1(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp1__constant_scale_0_0(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp1__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp1__constant_scale_127_127(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp1__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:13], v[14:17] cbsz:3 blgp:1
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp1__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp1__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -1174,7 +1174,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp1__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v8i32(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 3, ; cbsz
i32 1, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -1209,14 +1209,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp2(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp2__constant_scale_0_0(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp2__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp2__constant_scale_127_127(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp2__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:11], v[12:15] cbsz:3 blgp:2
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp2__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp2__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -1234,7 +1234,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp2__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 3, ; cbsz
i32 2, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -1269,14 +1269,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp4(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp4__constant_scale_0_0(<6 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp4__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp4__constant_scale_127_127(<6 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp4__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:9], v[10:13] cbsz:3 blgp:4
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp4__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp4__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v10
@@ -1294,7 +1294,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp4__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v4i32(<6 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2,
i32 3, ; cbsz
i32 4, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -1329,14 +1329,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp3(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp3__constant_scale_0_0(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp3__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp3__constant_scale_127_127(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp3__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:11], v[12:15] cbsz:3 blgp:3
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp3__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp3__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -1354,7 +1354,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp3__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 3, ; cbsz
i32 3, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -1389,14 +1389,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp4(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp4__constant_scale_0_0(<6 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp4__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp4__constant_scale_127_127(<6 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp4__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:9], v[10:13] cbsz:2 blgp:4
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp4__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp4__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v10
@@ -1414,7 +1414,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp4__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v4i32(<6 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2,
i32 2, ; cbsz
i32 4, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -1449,14 +1449,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp0(<4 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp0__constant_scale_0_0(<4 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp0__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp0__constant_scale_127_127(<4 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp0__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:3], v[4:11], v[12:15] cbsz:4
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp0__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp0__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -1474,7 +1474,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp0__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v4i32.v8i32(<4 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 4, ; cbsz
i32 0, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -1509,14 +1509,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp1(<4 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp1__constant_scale_0_0(<4 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp1__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp1__constant_scale_127_127(<4 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp1__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:3], v[4:11], v[12:15] cbsz:4 blgp:1
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp1__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp1__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -1534,7 +1534,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp1__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v4i32.v8i32(<4 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 4, ; cbsz
i32 1, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -1569,14 +1569,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp2(<4 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp2__constant_scale_0_0(<4 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp2__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp2__constant_scale_127_127(<4 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp2__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:3], v[4:9], v[10:13] cbsz:4 blgp:2
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp2__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp2__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v10
@@ -1594,7 +1594,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp2__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v4i32.v6i32(<4 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 4, ; cbsz
i32 2, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -1629,14 +1629,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp3(<4 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp3__constant_scale_0_0(<4 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp3__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp3__constant_scale_127_127(<4 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp3__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:3], v[4:9], v[10:13] cbsz:4 blgp:3
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp3__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp3__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v10
@@ -1654,7 +1654,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp3__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v4i32.v6i32(<4 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 4, ; cbsz
i32 3, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -1689,14 +1689,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4(<4 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4__constant_scale_0_0(<4 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4__constant_scale_0_0:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4__constant_scale_127_127(<4 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:3], v[4:7], v[8:11] cbsz:4 blgp:4
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v8
@@ -1714,7 +1714,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v4i32.v4i32(<4 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2,
i32 4, ; cbsz
i32 4, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -3121,24 +3121,44 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_a(
; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_a:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19]
+; GCN-NEXT: v_mov_b32_e32 v20, 0
+; GCN-NEXT: s_nop 1
+; GCN-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19], v20, v20 op_sel_hi:[0,0,0]
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_a:
-; AGPR: ; %bb.0:
-; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
-; AGPR-NEXT: v_accvgpr_write_b32 a1, v17
-; AGPR-NEXT: v_accvgpr_write_b32 a2, v18
-; AGPR-NEXT: v_accvgpr_write_b32 a3, v19
-; AGPR-NEXT: s_nop 1
-; AGPR-NEXT: v_mfma_f32_16x16x128_f8f6f4 a[0:3], v[0:7], v[8:15], a[0:3]
-; AGPR-NEXT: s_nop 11
-; AGPR-NEXT: v_accvgpr_read_b32 v0, a0
-; AGPR-NEXT: v_accvgpr_read_b32 v1, a1
-; AGPR-NEXT: v_accvgpr_read_b32 v2, a2
-; AGPR-NEXT: v_accvgpr_read_b32 v3, a3
-; AGPR-NEXT: s_setpc_b64 s[30:31]
+; AGPR-SDAG-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_a:
+; AGPR-SDAG: ; %bb.0:
+; AGPR-SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; AGPR-SDAG-NEXT: v_mov_b32_e32 v20, 0
+; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a0, v16
+; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a1, v17
+; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a2, v18
+; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a3, v19
+; AGPR-SDAG-NEXT: s_nop 1
+; AGPR-SDAG-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 a[0:3], v[0:7], v[8:15], a[0:3], v20, v20 op_sel_hi:[0,0,0]
+; AGPR-SDAG-NEXT: s_nop 11
+; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v0, a0
+; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v1, a1
+; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v2, a2
+; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v3, a3
+; AGPR-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; AGPR-GISEL-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_a:
+; AGPR-GISEL: ; %bb.0:
+; AGPR-GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a0, v16
+; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a1, v17
+; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a2, v18
+; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a3, v19
+; AGPR-GISEL-NEXT: v_mov_b32_e32 v16, 0
+; AGPR-GISEL-NEXT: s_nop 1
+; AGPR-GISEL-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 a[0:3], v[0:7], v[8:15], a[0:3], v16, v16 op_sel_hi:[0,0,0]
+; AGPR-GISEL-NEXT: s_nop 11
+; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v0, a0
+; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v1, a1
+; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v2, a2
+; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v3, a3
+; AGPR-GISEL-NEXT: s_setpc_b64 s[30:31]
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -3148,24 +3168,44 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_b(
; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_b:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19]
+; GCN-NEXT: v_mov_b32_e32 v20, 0
+; GCN-NEXT: s_nop 1
+; GCN-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19], v20, v20 op_sel:[1,1,0] op_sel_hi:[1,0,0]
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_b:
-; AGPR: ; %bb.0:
-; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
-; AGPR-NEXT: v_accvgpr_write_b32 a1, v17
-; AGPR-NEXT: v_accvgpr_write_b32 a2, v18
-; AGPR-NEXT: v_accvgpr_write_b32 a3, v19
-; AGPR-NEXT: s_nop 1
-; AGPR-NEXT: v_mfma_f32_16x16x128_f8f6f4 a[0:3], v[0:7], v[8:15], a[0:3]
-; AGPR-NEXT: s_nop 11
-; AGPR-NEXT: v_accvgpr_read_b32 v0, a0
-; AGPR-NEXT: v_accvgpr_read_b32 v1, a1
-; AGPR-NEXT: v_accvgpr_read_b32 v2, a2
-; AGPR-NEXT: v_accvgpr_read_b32 v3, a3
-; AGPR-NEXT: s_setpc_b64 s[30:31]
+; AGPR-SDAG-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_b:
+; AGPR-SDAG: ; %bb.0:
+; AGPR-SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; AGPR-SDAG-NEXT: v_mov_b32_e32 v20, 0
+; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a0, v16
+; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a1, v17
+; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a2, v18
+; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a3, v19
+; AGPR-SDAG-NEXT: s_nop 1
+; AGPR-SDAG-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 a[0:3], v[0:7], v[8:15], a[0:3], v20, v20 op_sel:[1,1,0] op_sel_hi:[1,0,0]
+; AGPR-SDAG-NEXT: s_nop 11
+; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v0, a0
+; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v1, a1
+; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v2, a2
+; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v3, a3
+; AGPR-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; AGPR-GISEL-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_b:
+; AGPR-GISEL: ; %bb.0:
+; AGPR-GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a0, v16
+; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a1, v17
+; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a2, v18
+; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a3, v19
+; AGPR-GISEL-NEXT: v_mov_b32_e32 v16, 0
+; AGPR-GISEL-NEXT: s_nop 1
+; AGPR-GISEL-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 a[0:3], v[0:7], v[8:15], a[0:3], v16, v16 op_sel:[1,1,0] op_sel_hi:[1,0,0]
+; AGPR-GISEL-NEXT: s_nop 11
+; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v0, a0
+; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v1, a1
+; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v2, a2
+; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v3, a3
+; AGPR-GISEL-NEXT: s_setpc_b64 s[30:31]
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2, i32 0, i32 0, i32 3, i32 0, i32 1, i32 0)
ret <4 x float> %result
}
@@ -3286,6 +3326,110 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_1_0_a(
ret <4 x float> %result
}
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_127_127__opsel1(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_127_127__opsel1:
+; GCN: ; %bb.0:
+; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GCN-NEXT: v_mov_b32_e32 v20, 0x7f
+; GCN-NEXT: s_nop 1
+; GCN-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19], v20, v20 op_sel:[1,1,0] op_sel_hi:[0,0,0]
+; GCN-NEXT: s_setpc_b64 s[30:31]
+;
+; AGPR-SDAG-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_127_127__opsel1:
+; AGPR-SDAG: ; %bb.0:
+; AGPR-SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; AGPR-SDAG-NEXT: v_mov_b32_e32 v20, 0x7f
+; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a0, v16
+; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a1, v17
+; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a2, v18
+; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a3, v19
+; AGPR-SDAG-NEXT: s_nop 1
+; AGPR-SDAG-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 a[0:3], v[0:7], v[8:15], a[0:3], v20, v20 op_sel:[1,1,0] op_sel_hi:[0,0,0]
+; AGPR-SDAG-NEXT: s_nop 11
+; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v0, a0
+; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v1, a1
+; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v2, a2
+; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v3, a3
+; AGPR-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; AGPR-GISEL-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_127_127__opsel1:
+; AGPR-GISEL: ; %bb.0:
+; AGPR-GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a0, v16
+; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a1, v17
+; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a2, v18
+; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a3, v19
+; AGPR-GISEL-NEXT: v_mov_b32_e32 v16, 0x7f
+; AGPR-GISEL-NEXT: s_nop 1
+; AGPR-GISEL-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 a[0:3], v[0:7], v[8:15], a[0:3], v16, v16 op_sel:[1,1,0] op_sel_hi:[0,0,0]
+; AGPR-GISEL-NEXT: s_nop 11
+; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v0, a0
+; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v1, a1
+; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v2, a2
+; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v3, a3
+; AGPR-GISEL-NEXT: s_setpc_b64 s[30:31]
+ %result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2, i32 0, i32 0, i32 1, i32 127, i32 1, i32 127)
+ ret <4 x float> %result
+}
+
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_127_0(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; SDAG-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_127_0:
+; SDAG: ; %bb.0:
+; SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; SDAG-NEXT: v_mov_b32_e32 v20, 0
+; SDAG-NEXT: v_mov_b32_e32 v21, 0x7f
+; SDAG-NEXT: s_nop 1
+; SDAG-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19], v21, v20 op_sel_hi:[0,0,0]
+; SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; GISEL-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_127_0:
+; GISEL: ; %bb.0:
+; GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GISEL-NEXT: v_mov_b32_e32 v20, 0x7f
+; GISEL-NEXT: v_mov_b32_e32 v21, 0
+; GISEL-NEXT: s_nop 1
+; GISEL-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19], v20, v21 op_sel_hi:[0,0,0]
+; GISEL-NEXT: s_setpc_b64 s[30:31]
+;
+; AGPR-SDAG-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_127_0:
+; AGPR-SDAG: ; %bb.0:
+; AGPR-SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; AGPR-SDAG-NEXT: v_mov_b32_e32 v20, 0
+; AGPR-SDAG-NEXT: v_mov_b32_e32 v21, 0x7f
+; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a0, v16
+; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a1, v17
+; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a2, v18
+; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a3, v19
+; AGPR-SDAG-NEXT: s_nop 1
+; AGPR-SDAG-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 a[0:3], v[0:7], v[8:15], a[0:3], v21, v20 op_sel_hi:[0,0,0]
+; AGPR-SDAG-NEXT: s_nop 11
+; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v0, a0
+; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v1, a1
+; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v2, a2
+; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v3, a3
+; AGPR-SDAG-NEXT: s_setpc_b64 s[30:31]
+;
+; AGPR-GISEL-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_127_0:
+; AGPR-GISEL: ; %bb.0:
+; AGPR-GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a0, v16
+; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a1, v17
+; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a2, v18
+; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a3, v19
+; AGPR-GISEL-NEXT: v_mov_b32_e32 v16, 0x7f
+; AGPR-GISEL-NEXT: v_mov_b32_e32 v17, 0
+; AGPR-GISEL-NEXT: s_nop 1
+; AGPR-GISEL-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 a[0:3], v[0:7], v[8:15], a[0:3], v16, v17 op_sel_hi:[0,0,0]
+; AGPR-GISEL-NEXT: s_nop 11
+; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v0, a0
+; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v1, a1
+; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v2, a2
+; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v3, a3
+; AGPR-GISEL-NEXT: s_setpc_b64 s[30:31]
+ %result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2, i32 0, i32 0, i32 0, i32 127, i32 0, i32 0)
+ ret <4 x float> %result
+}
+
; --------------------------------------------------------------------
; Incorrect signature for format cases (IR vector too large)
; --------------------------------------------------------------------
@@ -3377,14 +3521,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp6__v8i32_fp6(
ret <4 x float> %result
}
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp6__v8i32_fp6__0_scale(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp6__v8i32_fp6__0_scale:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp6__v8i32_fp6__127_scale(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp6__v8i32_fp6__127_scale:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19] cbsz:2 blgp:2
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp6__v8i32_fp6__0_scale:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp6__v8i32_fp6__127_scale:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
@@ -3402,7 +3546,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp6__v8i32_fp6_
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 2, ; cbsz
i32 2, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
@@ -3551,14 +3695,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp4__v8i32_fp4(
ret <4 x float> %result
}
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp4__v8i32_fp4__0_scale(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp4__v8i32_fp4__0_scale:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp4__v8i32_fp4__127_scale(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp4__v8i32_fp4__127_scale:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19] cbsz:4 blgp:4
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp4__v8i32_fp4__0_scale:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp4__v8i32_fp4__127_scale:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
@@ -3576,7 +3720,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp4__v8i32_fp4_
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 4, ; cbsz
i32 4, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <4 x float> %result
}
diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.ll
index 6d19ba5fff3093..6e74a96fcc1ae1 100644
--- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.ll
+++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.ll
@@ -1224,8 +1224,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_3_2__cbsz1__blgp1(<8 x
}
; This should be optimized to avoid the scale
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0__constant_scale_0_0(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0__constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: scratch_load_dword v31, off, s32
@@ -1251,7 +1251,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0__cons
; GCN-NEXT: v_mov_b32_e32 v15, v31
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: scratch_load_dword a15, off, s32
@@ -1295,7 +1295,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 0, ; cbsz
i32 0, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -1451,8 +1451,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1(<8 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1__constant_scale_0_0(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1__constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: scratch_load_dword v31, off, s32
@@ -1478,7 +1478,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1__cons
; GCN-NEXT: v_mov_b32_e32 v15, v31
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: scratch_load_dword a15, off, s32
@@ -1522,7 +1522,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 0, ; cbsz
i32 1, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -1603,8 +1603,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2(<8 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2__constant_scale_0_0(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2__constant_scale_127_127(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[14:29], v[0:7], v[8:13], v[14:29] blgp:2
@@ -1628,7 +1628,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2__cons
; GCN-NEXT: v_mov_b32_e32 v15, v29
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -1671,7 +1671,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v6i32(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 0, ; cbsz
i32 2, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -1752,8 +1752,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3(<8 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3__constant_scale_0_0(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3__constant_scale_127_127(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[14:29], v[0:7], v[8:13], v[14:29] blgp:3
@@ -1777,7 +1777,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3__cons
; GCN-NEXT: v_mov_b32_e32 v15, v29
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -1820,7 +1820,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v6i32(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 0, ; cbsz
i32 3, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -1897,8 +1897,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4(<8 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4__constant_scale_0_0(<8 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4__constant_scale_127_127(<8 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[12:27], v[0:7], v[8:11], v[12:27] blgp:4
@@ -1922,7 +1922,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4__cons
; GCN-NEXT: v_mov_b32_e32 v15, v27
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -1965,7 +1965,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v4i32(<8 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2,
i32 0, ; cbsz
i32 4, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -2121,8 +2121,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0(<8 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0__constant_scale_0_0(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0__constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: scratch_load_dword v31, off, s32
@@ -2148,7 +2148,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0__cons
; GCN-NEXT: v_mov_b32_e32 v15, v31
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: scratch_load_dword a15, off, s32
@@ -2192,7 +2192,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 1, ; cbsz
i32 0, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -2349,8 +2349,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1(<8 x
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1__constant_scale_0_0(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1__constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: scratch_load_dword v31, off, s32
@@ -2376,7 +2376,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1__cons
; GCN-NEXT: v_mov_b32_e32 v15, v31
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: scratch_load_dword a15, off, s32
@@ -2420,7 +2420,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 1, ; cbsz
i32 1, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -2501,8 +2501,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2(<8 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2__constant_scale_0(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2__constant_scale_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2__constant_scale_127_127(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[14:29], v[0:7], v[8:13], v[14:29] cbsz:1 blgp:2
@@ -2526,7 +2526,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2__cons
; GCN-NEXT: v_mov_b32_e32 v15, v29
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2__constant_scale_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -2569,7 +2569,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v6i32(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 1, ; cbsz
i32 2, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -2650,8 +2650,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3(<8 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3__constant_scale_0_0(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3__constant_scale_127_127(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[14:29], v[0:7], v[8:13], v[14:29] cbsz:1 blgp:3
@@ -2675,7 +2675,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3__cons
; GCN-NEXT: v_mov_b32_e32 v15, v29
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -2718,7 +2718,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v6i32(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 1, ; cbsz
i32 3, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -2795,8 +2795,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4(<8 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4__constant_scale_0_0(<8 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4__constant_scale_127_127(<8 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[12:27], v[0:7], v[8:11], v[12:27] cbsz:1 blgp:4
@@ -2820,7 +2820,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4__cons
; GCN-NEXT: v_mov_b32_e32 v15, v27
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -2863,7 +2863,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v4i32(<8 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2,
i32 1, ; cbsz
i32 4, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -2944,8 +2944,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0__constant_scale_0_0(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0__constant_scale_127_127(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[14:29], v[0:5], v[6:13], v[14:29] cbsz:2
@@ -2969,7 +2969,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0__cons
; GCN-NEXT: v_mov_b32_e32 v15, v29
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -3012,7 +3012,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v8i32(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 2, ; cbsz
i32 0, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -3093,8 +3093,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1__constant_scale_0_0(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1__constant_scale_127_127(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[14:29], v[0:5], v[6:13], v[14:29] cbsz:2 blgp:1
@@ -3118,7 +3118,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1__cons
; GCN-NEXT: v_mov_b32_e32 v15, v29
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -3161,7 +3161,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v8i32(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 2, ; cbsz
i32 1, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -3236,8 +3236,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2__constant_scale_0_0(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2__constant_scale_127_127(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[12:27], v[0:5], v[6:11], v[12:27] cbsz:2 blgp:2
@@ -3260,7 +3260,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2__cons
; GCN-NEXT: v_mov_b32_e32 v15, v27
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -3302,7 +3302,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 2, ; cbsz
i32 2, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -3377,8 +3377,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3__constant_scale_0_0(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3__constant_scale_127_127(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[12:27], v[0:5], v[6:11], v[12:27] cbsz:2 blgp:3
@@ -3401,7 +3401,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3__cons
; GCN-NEXT: v_mov_b32_e32 v15, v27
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -3443,7 +3443,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 2, ; cbsz
i32 3, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -3525,8 +3525,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0__constant_scale_0_0(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0__constant_scale_127_127(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[14:29], v[0:5], v[6:13], v[14:29] cbsz:3
@@ -3550,7 +3550,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0__cons
; GCN-NEXT: v_mov_b32_e32 v15, v29
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -3593,7 +3593,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v8i32(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 3, ; cbsz
i32 0, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -3674,8 +3674,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1__constant_scale_0_0(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1__constant_scale_127_127(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[14:29], v[0:5], v[6:13], v[14:29] cbsz:3 blgp:1
@@ -3699,7 +3699,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1__cons
; GCN-NEXT: v_mov_b32_e32 v15, v29
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -3742,7 +3742,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v8i32(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 3, ; cbsz
i32 1, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -3817,8 +3817,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2__constant_scale_0_0(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2__constant_scale_127_127(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[12:27], v[0:5], v[6:11], v[12:27] cbsz:3 blgp:2
@@ -3841,7 +3841,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2__cons
; GCN-NEXT: v_mov_b32_e32 v15, v27
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -3883,7 +3883,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 3, ; cbsz
i32 2, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -3958,8 +3958,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4__constant_scale_0_0(<6 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4__constant_scale_127_127(<6 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[10:25], v[0:5], v[6:9], v[10:25] cbsz:3 blgp:4
@@ -3982,7 +3982,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4__cons
; GCN-NEXT: v_mov_b32_e32 v15, v25
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v10
@@ -4024,7 +4024,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v4i32(<6 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2,
i32 3, ; cbsz
i32 4, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -4099,8 +4099,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3__constant_scale_0_0(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3__constant_scale_127_127(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[12:27], v[0:5], v[6:11], v[12:27] cbsz:3 blgp:3
@@ -4123,7 +4123,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3__cons
; GCN-NEXT: v_mov_b32_e32 v15, v27
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -4165,7 +4165,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 3, ; cbsz
i32 3, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -4240,8 +4240,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4__constant_scale_0_0(<6 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4__constant_scale_127_127(<6 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[10:25], v[0:5], v[6:9], v[10:25] cbsz:2 blgp:4
@@ -4264,7 +4264,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4__cons
; GCN-NEXT: v_mov_b32_e32 v15, v25
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v10
@@ -4306,7 +4306,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v4i32(<6 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2,
i32 2, ; cbsz
i32 4, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -4383,8 +4383,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0(<4 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0__constant_scale_0_0(<4 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0__constant_scale_127_127(<4 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[12:27], v[0:3], v[4:11], v[12:27] cbsz:4
@@ -4408,7 +4408,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0__cons
; GCN-NEXT: v_mov_b32_e32 v15, v27
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -4451,7 +4451,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v4i32.v8i32(<4 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 4, ; cbsz
i32 0, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -4528,8 +4528,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1(<4 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1__constant_scale_0_0(<4 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1__constant_scale_127_127(<4 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[12:27], v[0:3], v[4:11], v[12:27] cbsz:4 blgp:1
@@ -4553,7 +4553,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1__cons
; GCN-NEXT: v_mov_b32_e32 v15, v27
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -4596,7 +4596,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v4i32.v8i32(<4 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 4, ; cbsz
i32 1, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -4671,8 +4671,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2(<4 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2__constant_scale_0_0(<4 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2__constant_scale_127_127(<4 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[10:25], v[0:3], v[4:9], v[10:25] cbsz:4 blgp:2
@@ -4695,7 +4695,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2__cons
; GCN-NEXT: v_mov_b32_e32 v15, v25
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v10
@@ -4737,7 +4737,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v4i32.v6i32(<4 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 4, ; cbsz
i32 2, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -4812,8 +4812,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3(<4 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3__constant_scale_0_0(<4 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3__constant_scale_127_127(<4 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[10:25], v[0:3], v[4:9], v[10:25] cbsz:4 blgp:3
@@ -4836,7 +4836,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3__cons
; GCN-NEXT: v_mov_b32_e32 v15, v25
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v10
@@ -4878,7 +4878,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v4i32.v6i32(<4 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 4, ; cbsz
i32 3, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -4953,8 +4953,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4(<4 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4__constant_scale_0_0(<4 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4__constant_scale_0_0:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4__constant_scale_127_127(<4 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4__constant_scale_127_127:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[8:23], v[0:3], v[4:7], v[8:23] cbsz:4 blgp:4
@@ -4977,7 +4977,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4__cons
; GCN-NEXT: v_mov_b32_e32 v15, v23
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4__constant_scale_0_0:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4__constant_scale_127_127:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v8
@@ -5019,7 +5019,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v4i32.v4i32(<4 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2,
i32 4, ; cbsz
i32 4, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -8211,8 +8211,8 @@ define amdgpu_kernel void @test_mfma_scale_f32_32x32x64_f8f6f4_25_42__nonmac(<8
ret void
}
-define amdgpu_kernel void @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__vgprcd_nonmac(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) #0 {
-; SDAG-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__vgprcd_nonmac:
+define amdgpu_kernel void @test_mfma_scale_f32_32x32x64_f8f6f4_127_127__vgprcd_nonmac(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) #0 {
+; SDAG-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_127_127__vgprcd_nonmac:
; SDAG: ; %bb.0:
; SDAG-NEXT: s_load_dwordx16 s[8:23], s[4:5], 0x0
; SDAG-NEXT: s_load_dwordx16 s[36:51], s[4:5], 0x40
@@ -8282,7 +8282,7 @@ define amdgpu_kernel void @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__vgprcd_nonma
; SDAG-NEXT: s_waitcnt vmcnt(0)
; SDAG-NEXT: s_endpgm
;
-; GISEL-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__vgprcd_nonmac:
+; GISEL-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_127_127__vgprcd_nonmac:
; GISEL: ; %bb.0:
; GISEL-NEXT: s_load_dwordx16 s[36:51], s[4:5], 0x0
; GISEL-NEXT: s_load_dwordx16 s[8:23], s[4:5], 0x40
@@ -8336,7 +8336,7 @@ define amdgpu_kernel void @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__vgprcd_nonma
; GISEL-NEXT: s_waitcnt vmcnt(0)
; GISEL-NEXT: s_endpgm
;
-; AGPR-SDAG-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__vgprcd_nonmac:
+; AGPR-SDAG-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_127_127__vgprcd_nonmac:
; AGPR-SDAG: ; %bb.0:
; AGPR-SDAG-NEXT: s_load_dwordx16 s[8:23], s[4:5], 0x0
; AGPR-SDAG-NEXT: s_load_dwordx16 s[36:51], s[4:5], 0x40
@@ -8406,7 +8406,7 @@ define amdgpu_kernel void @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__vgprcd_nonma
; AGPR-SDAG-NEXT: s_waitcnt vmcnt(0)
; AGPR-SDAG-NEXT: s_endpgm
;
-; AGPR-GISEL-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__vgprcd_nonmac:
+; AGPR-GISEL-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_127_127__vgprcd_nonmac:
; AGPR-GISEL: ; %bb.0:
; AGPR-GISEL-NEXT: s_load_dwordx16 s[36:51], s[4:5], 0x0
; AGPR-GISEL-NEXT: s_load_dwordx16 s[8:23], s[4:5], 0x40
@@ -8459,7 +8459,7 @@ define amdgpu_kernel void @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__vgprcd_nonma
; AGPR-GISEL-NEXT: global_store_dwordx4 v[18:19], v[12:15], off sc0 sc1
; AGPR-GISEL-NEXT: s_waitcnt vmcnt(0)
; AGPR-GISEL-NEXT: s_endpgm
- %result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2, i32 0, i32 2, i32 0, i32 0, i32 0, i32 0)
+ %result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2, i32 0, i32 2, i32 0, i32 127, i32 0, i32 127)
store volatile <16 x float> %arg2, ptr addrspace(1) null, align 64
store volatile <16 x float> %result, ptr addrspace(1) null, align 64
ret void
@@ -8730,8 +8730,10 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___constant_scale_0_0_a(
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: scratch_load_dword v31, off, s32
+; GCN-NEXT: v_mov_b32_e32 v32, 0
; GCN-NEXT: s_waitcnt vmcnt(0)
-; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[16:31], v[0:7], v[8:15], v[16:31]
+; GCN-NEXT: s_nop 0
+; GCN-NEXT: v_mfma_scale_f32_32x32x64_f8f6f4 v[16:31], v[0:7], v[8:15], v[16:31], v32, v32 op_sel_hi:[0,0,0]
; GCN-NEXT: s_nop 15
; GCN-NEXT: s_nop 3
; GCN-NEXT: v_mov_b32_e32 v0, v16
@@ -8756,6 +8758,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___constant_scale_0_0_a(
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: scratch_load_dword a15, off, s32
+; AGPR-NEXT: v_mov_b32_e32 v31, 0
; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
; AGPR-NEXT: v_accvgpr_write_b32 a1, v17
; AGPR-NEXT: v_accvgpr_write_b32 a2, v18
@@ -8773,7 +8776,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___constant_scale_0_0_a(
; AGPR-NEXT: v_accvgpr_write_b32 a14, v30
; AGPR-NEXT: s_waitcnt vmcnt(0)
; AGPR-NEXT: s_nop 0
-; AGPR-NEXT: v_mfma_f32_32x32x64_f8f6f4 a[0:15], v[0:7], v[8:15], a[0:15]
+; AGPR-NEXT: v_mfma_scale_f32_32x32x64_f8f6f4 a[0:15], v[0:7], v[8:15], a[0:15], v31, v31 op_sel_hi:[0,0,0]
; AGPR-NEXT: s_nop 15
; AGPR-NEXT: s_nop 3
; AGPR-NEXT: v_accvgpr_read_b32 v0, a0
@@ -8802,8 +8805,10 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___constant_scale_0_0_b(
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: scratch_load_dword v31, off, s32
+; GCN-NEXT: v_mov_b32_e32 v32, 0
; GCN-NEXT: s_waitcnt vmcnt(0)
-; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[16:31], v[0:7], v[8:15], v[16:31]
+; GCN-NEXT: s_nop 0
+; GCN-NEXT: v_mfma_scale_f32_32x32x64_f8f6f4 v[16:31], v[0:7], v[8:15], v[16:31], v32, v32 op_sel:[1,1,0] op_sel_hi:[1,0,0]
; GCN-NEXT: s_nop 15
; GCN-NEXT: s_nop 3
; GCN-NEXT: v_mov_b32_e32 v0, v16
@@ -8828,6 +8833,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___constant_scale_0_0_b(
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: scratch_load_dword a15, off, s32
+; AGPR-NEXT: v_mov_b32_e32 v31, 0
; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
; AGPR-NEXT: v_accvgpr_write_b32 a1, v17
; AGPR-NEXT: v_accvgpr_write_b32 a2, v18
@@ -8845,7 +8851,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___constant_scale_0_0_b(
; AGPR-NEXT: v_accvgpr_write_b32 a14, v30
; AGPR-NEXT: s_waitcnt vmcnt(0)
; AGPR-NEXT: s_nop 0
-; AGPR-NEXT: v_mfma_f32_32x32x64_f8f6f4 a[0:15], v[0:7], v[8:15], a[0:15]
+; AGPR-NEXT: v_mfma_scale_f32_32x32x64_f8f6f4 a[0:15], v[0:7], v[8:15], a[0:15], v31, v31 op_sel:[1,1,0] op_sel_hi:[1,0,0]
; AGPR-NEXT: s_nop 15
; AGPR-NEXT: s_nop 3
; AGPR-NEXT: v_accvgpr_read_b32 v0, a0
@@ -9622,8 +9628,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6(
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6__0_scale(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6__0_scale:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6__127_scale(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6__127_scale:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: scratch_load_dword v31, off, s32
@@ -9648,7 +9654,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6_
; GCN-NEXT: v_mov_b32_e32 v15, v31
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6__0_scale:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6__127_scale:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: scratch_load_dword a15, off, s32
@@ -9691,7 +9697,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6_
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 2, ; cbsz
i32 2, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
@@ -10296,8 +10302,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4(
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4__0_scale(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4__0_scale:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4__127_scale(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4__127_scale:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: scratch_load_dword v31, off, s32
@@ -10322,7 +10328,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4_
; GCN-NEXT: v_mov_b32_e32 v15, v31
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4__0_scale:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4__127_scale:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: scratch_load_dword a15, off, s32
@@ -10365,7 +10371,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4_
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 4, ; cbsz
i32 4, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
ret <16 x float> %result
}
diff --git a/llvm/test/CodeGen/AMDGPU/rewrite-vgpr-mfma-to-agpr.gfx950.ll b/llvm/test/CodeGen/AMDGPU/rewrite-vgpr-mfma-to-agpr.gfx950.ll
index b163b57883481f..248822c1d50c95 100644
--- a/llvm/test/CodeGen/AMDGPU/rewrite-vgpr-mfma-to-agpr.gfx950.ll
+++ b/llvm/test/CodeGen/AMDGPU/rewrite-vgpr-mfma-to-agpr.gfx950.ll
@@ -149,7 +149,7 @@ define void @test_rewrite_mfma_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp0(<8 x i32>
%mai = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %src2,
i32 0, ; cbsz
i32 0, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
call void asm sideeffect "; use $0", "a"(<4 x float> %mai)
ret void
}
@@ -189,7 +189,7 @@ define void @test_rewrite_mfma_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2(<6 x i32>
%mai = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %src2,
i32 2, ; cbsz
i32 2, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
call void asm sideeffect "; use $0", "a"(<4 x float> %mai)
ret void
}
@@ -229,7 +229,7 @@ define void @test_rewrite_mfma_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4(<4 x i32>
%mai = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v4i32.v4i32(<4 x i32> %arg0, <4 x i32> %arg1, <4 x float> %src2,
i32 4, ; cbsz
i32 4, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
call void asm sideeffect "; use $0", "a"(<4 x float> %mai)
ret void
}
@@ -276,7 +276,7 @@ define void @test_rewrite_mfma_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0(<8 x i32> %
%mai = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %src2,
i32 0, ; cbsz
i32 0, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
call void asm sideeffect "; use $0", "a"(<16 x float> %mai)
ret void
}
@@ -322,7 +322,7 @@ define void @test_rewrite_mfma_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2(<6 x i32> %
%mai = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %src2,
i32 2, ; cbsz
i32 2, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
call void asm sideeffect "; use $0", "a"(<16 x float> %mai)
ret void
}
@@ -368,7 +368,7 @@ define void @test_rewrite_mfma_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4(<4 x i32> %
%mai = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v4i32.v4i32(<4 x i32> %arg0, <4 x i32> %arg1, <16 x float> %src2,
i32 4, ; cbsz
i32 4, ; blgp
- i32 0, i32 0, i32 0, i32 0)
+ i32 0, i32 127, i32 0, i32 127)
call void asm sideeffect "; use $0", "a"(<16 x float> %mai)
ret void
}
diff --git a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
index 92a47dab42f019..1fcbfe84aa9f7a 100644
--- a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
+++ b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
@@ -1593,9 +1593,10 @@ struct MFMAOpLowering : public ConvertOpToLLVMPattern<MFMAOp> {
rewriter, loc, adaptor.getSourceB(), allowBf16),
adaptor.getDestC()});
if (isScaled) {
- Value zero = createI32Constant(rewriter, loc, 0);
+ // Scales are E8M0 (2^(s-127)), so 127 is a scale of 1.0.
+ Value one = createI32Constant(rewriter, loc, 127);
auto [_scaledName, aTypeCode, bTypeCode] = *maybeScaledIntrinsic;
- loweredOp.addOperands({/*scale A=*/zero, /*scale B=*/zero});
+ loweredOp.addOperands({/*scale A=*/one, /*scale B=*/one});
loweredOp.addAttributes(
{{"cbsz",
ROCDL::MatrixFormatAttr::get(rewriter.getContext(), aTypeCode)},
diff --git a/mlir/test/Conversion/AMDGPUToROCDL/mfma-gfx950.mlir b/mlir/test/Conversion/AMDGPUToROCDL/mfma-gfx950.mlir
index 8e5bd123121bec..5edc852c92cf4d 100644
--- a/mlir/test/Conversion/AMDGPUToROCDL/mfma-gfx950.mlir
+++ b/mlir/test/Conversion/AMDGPUToROCDL/mfma-gfx950.mlir
@@ -19,31 +19,31 @@ func.func @mfma_to_rocdl(%arg0 : vector<8xf16>, %arg1 : vector<16xf32>,
// CHECK: rocdl.mfma.i32.16x16x64.i8{{.*}}: (vector<4xi32>, vector<4xi32>, vector<4xi32>) -> vector<4xi32>
amdgpu.mfma 16x16x64 %arg4 * %arg4 + %arg6 : vector<16xi8>, vector<16xi8>, vector<4xi32>
- // CHECK: %[[c0:.+]] = llvm.mlir.constant(0 : i32) : i32
- // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp8_e4m3, fp8_e4m3, 0, %[[c0]], 0, %[[c0]] : (vector<8xi32>, vector<8xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
+ // CHECK: %[[c127:.+]] = llvm.mlir.constant(127 : i32) : i32
+ // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp8_e4m3, fp8_e4m3, 0, %[[c127]], 0, %[[c127]] : (vector<8xi32>, vector<8xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
amdgpu.mfma 32x32x64 %arg7 * %arg7 + %arg1 : vector<32xf8E4M3FN>, vector<32xf8E4M3FN>, vector<16xf32>
- // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp8_e4m3, fp8_e4m3, 0, %[[c0]], 0, %[[c0]] : (vector<8xi32>, vector<8xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
+ // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp8_e4m3, fp8_e4m3, 0, %[[c127]], 0, %[[c127]] : (vector<8xi32>, vector<8xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
amdgpu.mfma 16x16x128 %arg7 * %arg7 + %arg2 : vector<32xf8E4M3FN>, vector<32xf8E4M3FN>, vector<4xf32>
- // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp8_e5m2, fp8_e5m2, 0, %[[c0]], 0, %[[c0]] : (vector<8xi32>, vector<8xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
+ // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp8_e5m2, fp8_e5m2, 0, %[[c127]], 0, %[[c127]] : (vector<8xi32>, vector<8xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
amdgpu.mfma 32x32x64 %arg8 * %arg8 + %arg1 : vector<32xf8E5M2>, vector<32xf8E5M2>, vector<16xf32>
- // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp8_e5m2, fp8_e5m2, 0, %[[c0]], 0, %[[c0]] : (vector<8xi32>, vector<8xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
+ // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp8_e5m2, fp8_e5m2, 0, %[[c127]], 0, %[[c127]] : (vector<8xi32>, vector<8xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
amdgpu.mfma 16x16x128 %arg8 * %arg8 + %arg2 : vector<32xf8E5M2>, vector<32xf8E5M2>, vector<4xf32>
- // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e2m3, fp6_e2m3, 0, %[[c0]], 0, %[[c0]] : (vector<6xi32>, vector<6xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
+ // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e2m3, fp6_e2m3, 0, %[[c127]], 0, %[[c127]] : (vector<6xi32>, vector<6xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
amdgpu.mfma 32x32x64 %arg9 * %arg9 + %arg1 : vector<32xf6E2M3FN>, vector<32xf6E2M3FN>, vector<16xf32>
- // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e2m3, fp6_e2m3, 0, %[[c0]], 0, %[[c0]] : (vector<6xi32>, vector<6xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
+ // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e2m3, fp6_e2m3, 0, %[[c127]], 0, %[[c127]] : (vector<6xi32>, vector<6xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
amdgpu.mfma 16x16x128 %arg9 * %arg9 + %arg2 : vector<32xf6E2M3FN>, vector<32xf6E2M3FN>, vector<4xf32>
- // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e3m2, fp6_e3m2, 0, %[[c0]], 0, %[[c0]] : (vector<6xi32>, vector<6xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
+ // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e3m2, fp6_e3m2, 0, %[[c127]], 0, %[[c127]] : (vector<6xi32>, vector<6xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
amdgpu.mfma 32x32x64 %arg10 * %arg10 + %arg1 : vector<32xf6E3M2FN>, vector<32xf6E3M2FN>, vector<16xf32>
- // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e3m2, fp6_e3m2, 0, %[[c0]], 0, %[[c0]] : (vector<6xi32>, vector<6xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
+ // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e3m2, fp6_e3m2, 0, %[[c127]], 0, %[[c127]] : (vector<6xi32>, vector<6xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
amdgpu.mfma 16x16x128 %arg10 * %arg10 + %arg2 : vector<32xf6E3M2FN>, vector<32xf6E3M2FN>, vector<4xf32>
- // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp4_e2m1, fp4_e2m1, 0, %[[c0]], 0, %[[c0]] : (vector<4xi32>, vector<4xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
+ // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp4_e2m1, fp4_e2m1, 0, %[[c127]], 0, %[[c127]] : (vector<4xi32>, vector<4xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
amdgpu.mfma 32x32x64 %arg11 * %arg11 + %arg1 : vector<32xf4E2M1FN>, vector<32xf4E2M1FN>, vector<16xf32>
- // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp4_e2m1, fp4_e2m1, 0, %[[c0]], 0, %[[c0]] : (vector<4xi32>, vector<4xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
+ // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp4_e2m1, fp4_e2m1, 0, %[[c127]], 0, %[[c127]] : (vector<4xi32>, vector<4xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
amdgpu.mfma 16x16x128 %arg11 * %arg11 + %arg2 : vector<32xf4E2M1FN>, vector<32xf4E2M1FN>, vector<4xf32>
- // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e2m3, fp4_e2m1, 0, %[[c0]], 0, %[[c0]] : (vector<6xi32>, vector<4xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
+ // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e2m3, fp4_e2m1, 0, %[[c127]], 0, %[[c127]] : (vector<6xi32>, vector<4xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
amdgpu.mfma 32x32x64 %arg9 * %arg11 + %arg1 : vector<32xf6E2M3FN>, vector<32xf4E2M1FN>, vector<16xf32>
- // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e2m3, fp4_e2m1, 0, %[[c0]], 0, %[[c0]] : (vector<6xi32>, vector<4xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
+ // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e2m3, fp4_e2m1, 0, %[[c127]], 0, %[[c127]] : (vector<6xi32>, vector<4xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
amdgpu.mfma 16x16x128 %arg9 * %arg11 + %arg2 : vector<32xf6E2M3FN>, vector<32xf4E2M1FN>, vector<4xf32>
func.return
>From e72754ea754ea951e5aa28a058f2bc8796bbc8a0 Mon Sep 17 00:00:00 2001
From: Keshav Vinayak Jha <keshavvinayakjha at gmail.com>
Date: Fri, 16 Oct 2026 08:09:14 +0000
Subject: [PATCH 2/4] [AMDGPU] Fold mfma.scale to unscaled MFMA for E8M0 1.0
scales
Keep the existing fold for constant 0 scales, since Composable Kernel
and the MLIR AMDGPUToROCDL lowering use it as the way to get the
unscaled gfx950 f8f6f4 MFMA, and document that behavior on the
intrinsic. Add a second alternative to UnscaledMFMAOptimizationPat that
folds when both scales are 127 with op_sel 0, which is E8M0 1.0.
This drops the previous approach of only folding 127, which would have
broken frontends relying on constant 0 scales. The MLIR lowering change
and the existing test updates are reverted. New tests cover 127/127,
127/127 with op_sel = 1, and 127/0.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
Signed-off-by: Keshav Vinayak Jha <keshavvinayakjha at gmail.com>
---
llvm/include/llvm/IR/IntrinsicsAMDGPU.td | 6 +
llvm/lib/Target/AMDGPU/SIInstrInfo.td | 14 +-
....amdgcn.mfma.scale.f32.16x16x128.f8f6f4.ll | 346 +++++++++---------
...m.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.ll | 314 +++++++++-------
.../rewrite-vgpr-mfma-to-agpr.gfx950.ll | 12 +-
.../AMDGPUToROCDL/AMDGPUToROCDL.cpp | 5 +-
.../Conversion/AMDGPUToROCDL/mfma-gfx950.mlir | 26 +-
7 files changed, 391 insertions(+), 332 deletions(-)
diff --git a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
index 9004f4afed7964..95cc36af4b5230 100644
--- a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
+++ b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
@@ -3689,6 +3689,12 @@ class AMDGPUMfmaIntrinsic<LLVMType DestTy, LLVMType SrcABTy> :
// requires more registers than the corresponding vector type (e.g. it
// is illegal to use <6 x i32> in operand 0 if cbsz specifies an f8
// format that requires 8 registers).
+//
+// The scales are E8M0 (2^(s-127)) values, taken from the byte of each scale
+// operand selected by its op_sel. There is no separate intrinsic for the
+// unscaled instruction: if both scale operands are the constant 0, the
+// unscaled instruction is used and no scaling is applied, instead of the
+// 2^-127 that a non-constant 0 scale gives.
class AMDGPUMfmaScaleIntrinsic<LLVMType DestTy> :
DefaultAttrsIntrinsic<[DestTy],
[llvm_anyvector_ty, llvm_anyvector_ty, DestTy,
diff --git a/llvm/lib/Target/AMDGPU/SIInstrInfo.td b/llvm/lib/Target/AMDGPU/SIInstrInfo.td
index 01e700e1a52875..7e11bc5c0bc539 100644
--- a/llvm/lib/Target/AMDGPU/SIInstrInfo.td
+++ b/llvm/lib/Target/AMDGPU/SIInstrInfo.td
@@ -317,14 +317,16 @@ def SIdenorm_mode : SDNode<"AMDGPUISD::DENORM_MODE",
>;
-// Optimize v_mfma_scale* instructions to avoid the scale if the
-// scales are known 1.0. Scales are E8M0 (2^(s-127)), so 1.0 is 127 in the
-// byte selected by op_sel.
-class UnscaledMFMAOptimizationPat<SDPatternOperator intrin> : PatFrag<
+// Optimize v_mfma_scale* instructions to avoid the scale. Constant 0 scales
+// are the documented way to request the unscaled instruction (see the
+// intrinsic definition). Scales of 127 with op_sel 0 are E8M0 1.0.
+class UnscaledMFMAOptimizationPat<SDPatternOperator intrin> : PatFrags<
(ops node:$srca, node:$srcb, node:$srcc,
node:$cbsz, node:$blgp),
- (intrin $srca, $srcb, $srcc, $cbsz, $blgp,
- 0, 127, 0, 127)
+ [(intrin $srca, $srcb, $srcc, $cbsz, $blgp,
+ srcvalue, 0, srcvalue, 0),
+ (intrin $srca, $srcb, $srcc, $cbsz, $blgp,
+ 0, 127, 0, 127)]
>;
def mfma_f32_16x16x128_f8f6f4 : UnscaledMFMAOptimizationPat<int_amdgcn_mfma_scale_f32_16x16x128_f8f6f4>;
diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.ll
index 7cce811d8e9362..dab2b8bdeb6390 100644
--- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.ll
+++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.ll
@@ -248,14 +248,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_3_2__cbsz1__blgp1(<8 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp0__constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp0__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp0__constant_scale_0_0(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp0__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19]
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp0__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp0__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
@@ -273,7 +273,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp0__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 0, ; cbsz
i32 0, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -308,14 +308,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp1(<8 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp1__constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp1__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp1__constant_scale_0_0(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp1__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19] blgp:1
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp1__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp1__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
@@ -333,7 +333,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp1__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 0, ; cbsz
i32 1, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -368,14 +368,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp2(<8 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp2__constant_scale_127_127(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp2__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp2__constant_scale_0_0(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp2__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:13], v[14:17] blgp:2
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp2__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp2__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -393,7 +393,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp2__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v6i32(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 0, ; cbsz
i32 2, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -428,14 +428,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp3(<8 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp3__constant_scale_127_127(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp3__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp3__constant_scale_0_0(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp3__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:13], v[14:17] blgp:3
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp3__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp3__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -453,7 +453,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp3__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v6i32(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 0, ; cbsz
i32 3, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -488,14 +488,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp4(<8 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp4__constant_scale_127_127(<8 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp4__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp4__constant_scale_0_0(<8 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp4__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:11], v[12:15] blgp:4
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp4__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp4__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -513,7 +513,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp4__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v4i32(<8 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2,
i32 0, ; cbsz
i32 4, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -548,14 +548,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp0(<8 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp0__constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp0__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp0__constant_scale_0_0(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp0__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19] cbsz:1
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp0__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp0__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
@@ -573,7 +573,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp0__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 1, ; cbsz
i32 0, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -609,14 +609,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp1(<8 x
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp1__constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp1__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp1__constant_scale_0_0(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp1__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19] cbsz:1 blgp:1
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp1__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp1__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
@@ -634,7 +634,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp1__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 1, ; cbsz
i32 1, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -668,14 +668,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp2(<8 x
ret <4 x float> %result
}
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp2__constant_scale_127_127(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp2__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp2__constant_scale_0(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp2__constant_scale_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:13], v[14:17] cbsz:1 blgp:2
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp2__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp2__constant_scale_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -693,7 +693,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp2__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v6i32(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 1, ; cbsz
i32 2, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -728,14 +728,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp3(<8 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp3__constant_scale_127_127(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp3__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp3__constant_scale_0_0(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp3__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:13], v[14:17] cbsz:1 blgp:3
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp3__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp3__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -753,7 +753,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp3__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v6i32(<8 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 1, ; cbsz
i32 3, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -788,14 +788,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp4(<8 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp4__constant_scale_127_127(<8 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp4__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp4__constant_scale_0_0(<8 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp4__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:11], v[12:15] cbsz:1 blgp:4
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp4__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp4__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -813,7 +813,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz1__blgp4__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v4i32(<8 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2,
i32 1, ; cbsz
i32 4, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -848,14 +848,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp0(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp0__constant_scale_127_127(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp0__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp0__constant_scale_0_0(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp0__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:13], v[14:17] cbsz:2
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp0__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp0__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -873,7 +873,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp0__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v8i32(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 2, ; cbsz
i32 0, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -908,14 +908,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp1(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp1__constant_scale_127_127(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp1__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp1__constant_scale_0_0(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp1__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:13], v[14:17] cbsz:2 blgp:1
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp1__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp1__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -933,7 +933,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp1__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v8i32(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 2, ; cbsz
i32 1, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -968,14 +968,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2__constant_scale_127_127(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2__constant_scale_0_0(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:11], v[12:15] cbsz:2 blgp:2
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -993,7 +993,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 2, ; cbsz
i32 2, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -1028,14 +1028,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp3(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp3__constant_scale_127_127(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp3__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp3__constant_scale_0_0(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp3__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:11], v[12:15] cbsz:2 blgp:3
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp3__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp3__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -1053,7 +1053,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp3__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 2, ; cbsz
i32 3, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -1089,14 +1089,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp0(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp0__constant_scale_127_127(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp0__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp0__constant_scale_0_0(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp0__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:13], v[14:17] cbsz:3
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp0__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp0__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -1114,7 +1114,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp0__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v8i32(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 3, ; cbsz
i32 0, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -1149,14 +1149,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp1(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp1__constant_scale_127_127(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp1__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp1__constant_scale_0_0(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp1__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:13], v[14:17] cbsz:3 blgp:1
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp1__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp1__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -1174,7 +1174,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp1__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v8i32(<6 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 3, ; cbsz
i32 1, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -1209,14 +1209,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp2(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp2__constant_scale_127_127(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp2__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp2__constant_scale_0_0(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp2__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:11], v[12:15] cbsz:3 blgp:2
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp2__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp2__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -1234,7 +1234,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp2__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 3, ; cbsz
i32 2, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -1269,14 +1269,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp4(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp4__constant_scale_127_127(<6 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp4__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp4__constant_scale_0_0(<6 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp4__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:9], v[10:13] cbsz:3 blgp:4
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp4__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp4__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v10
@@ -1294,7 +1294,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp4__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v4i32(<6 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2,
i32 3, ; cbsz
i32 4, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -1329,14 +1329,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp3(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp3__constant_scale_127_127(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp3__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp3__constant_scale_0_0(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp3__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:11], v[12:15] cbsz:3 blgp:3
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp3__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp3__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -1354,7 +1354,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz3__blgp3__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 3, ; cbsz
i32 3, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -1389,14 +1389,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp4(<6 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp4__constant_scale_127_127(<6 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp4__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp4__constant_scale_0_0(<6 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp4__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:5], v[6:9], v[10:13] cbsz:2 blgp:4
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp4__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp4__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v10
@@ -1414,7 +1414,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp4__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v4i32(<6 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2,
i32 2, ; cbsz
i32 4, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -1449,14 +1449,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp0(<4 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp0__constant_scale_127_127(<4 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp0__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp0__constant_scale_0_0(<4 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp0__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:3], v[4:11], v[12:15] cbsz:4
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp0__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp0__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -1474,7 +1474,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp0__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v4i32.v8i32(<4 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 4, ; cbsz
i32 0, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -1509,14 +1509,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp1(<4 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp1__constant_scale_127_127(<4 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp1__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp1__constant_scale_0_0(<4 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp1__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:3], v[4:11], v[12:15] cbsz:4 blgp:1
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp1__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp1__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -1534,7 +1534,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp1__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v4i32.v8i32(<4 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 4, ; cbsz
i32 1, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -1569,14 +1569,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp2(<4 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp2__constant_scale_127_127(<4 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp2__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp2__constant_scale_0_0(<4 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp2__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:3], v[4:9], v[10:13] cbsz:4 blgp:2
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp2__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp2__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v10
@@ -1594,7 +1594,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp2__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v4i32.v6i32(<4 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 4, ; cbsz
i32 2, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -1629,14 +1629,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp3(<4 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp3__constant_scale_127_127(<4 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp3__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp3__constant_scale_0_0(<4 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp3__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:3], v[4:9], v[10:13] cbsz:4 blgp:3
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp3__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp3__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v10
@@ -1654,7 +1654,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp3__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v4i32.v6i32(<4 x i32> %arg0, <6 x i32> %arg1, <4 x float> %arg2,
i32 4, ; cbsz
i32 3, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -1689,14 +1689,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4(<4 x
}
; This should be optimized to avoid the scale
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4__constant_scale_127_127(<4 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4__constant_scale_127_127:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4__constant_scale_0_0(<4 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:3], v[4:7], v[8:11] cbsz:4 blgp:4
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v8
@@ -1714,7 +1714,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4__cons
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v4i32.v4i32(<4 x i32> %arg0, <4 x i32> %arg1, <4 x float> %arg2,
i32 4, ; cbsz
i32 4, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -3121,44 +3121,24 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_a(
; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_a:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GCN-NEXT: v_mov_b32_e32 v20, 0
-; GCN-NEXT: s_nop 1
-; GCN-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19], v20, v20 op_sel_hi:[0,0,0]
+; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19]
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-SDAG-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_a:
-; AGPR-SDAG: ; %bb.0:
-; AGPR-SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; AGPR-SDAG-NEXT: v_mov_b32_e32 v20, 0
-; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a0, v16
-; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a1, v17
-; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a2, v18
-; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a3, v19
-; AGPR-SDAG-NEXT: s_nop 1
-; AGPR-SDAG-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 a[0:3], v[0:7], v[8:15], a[0:3], v20, v20 op_sel_hi:[0,0,0]
-; AGPR-SDAG-NEXT: s_nop 11
-; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v0, a0
-; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v1, a1
-; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v2, a2
-; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v3, a3
-; AGPR-SDAG-NEXT: s_setpc_b64 s[30:31]
-;
-; AGPR-GISEL-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_a:
-; AGPR-GISEL: ; %bb.0:
-; AGPR-GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a0, v16
-; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a1, v17
-; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a2, v18
-; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a3, v19
-; AGPR-GISEL-NEXT: v_mov_b32_e32 v16, 0
-; AGPR-GISEL-NEXT: s_nop 1
-; AGPR-GISEL-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 a[0:3], v[0:7], v[8:15], a[0:3], v16, v16 op_sel_hi:[0,0,0]
-; AGPR-GISEL-NEXT: s_nop 11
-; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v0, a0
-; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v1, a1
-; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v2, a2
-; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v3, a3
-; AGPR-GISEL-NEXT: s_setpc_b64 s[30:31]
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_a:
+; AGPR: ; %bb.0:
+; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
+; AGPR-NEXT: v_accvgpr_write_b32 a1, v17
+; AGPR-NEXT: v_accvgpr_write_b32 a2, v18
+; AGPR-NEXT: v_accvgpr_write_b32 a3, v19
+; AGPR-NEXT: s_nop 1
+; AGPR-NEXT: v_mfma_f32_16x16x128_f8f6f4 a[0:3], v[0:7], v[8:15], a[0:3]
+; AGPR-NEXT: s_nop 11
+; AGPR-NEXT: v_accvgpr_read_b32 v0, a0
+; AGPR-NEXT: v_accvgpr_read_b32 v1, a1
+; AGPR-NEXT: v_accvgpr_read_b32 v2, a2
+; AGPR-NEXT: v_accvgpr_read_b32 v3, a3
+; AGPR-NEXT: s_setpc_b64 s[30:31]
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -3168,44 +3148,24 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_b(
; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_b:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GCN-NEXT: v_mov_b32_e32 v20, 0
-; GCN-NEXT: s_nop 1
-; GCN-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19], v20, v20 op_sel:[1,1,0] op_sel_hi:[1,0,0]
+; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19]
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-SDAG-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_b:
-; AGPR-SDAG: ; %bb.0:
-; AGPR-SDAG-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; AGPR-SDAG-NEXT: v_mov_b32_e32 v20, 0
-; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a0, v16
-; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a1, v17
-; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a2, v18
-; AGPR-SDAG-NEXT: v_accvgpr_write_b32 a3, v19
-; AGPR-SDAG-NEXT: s_nop 1
-; AGPR-SDAG-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 a[0:3], v[0:7], v[8:15], a[0:3], v20, v20 op_sel:[1,1,0] op_sel_hi:[1,0,0]
-; AGPR-SDAG-NEXT: s_nop 11
-; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v0, a0
-; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v1, a1
-; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v2, a2
-; AGPR-SDAG-NEXT: v_accvgpr_read_b32 v3, a3
-; AGPR-SDAG-NEXT: s_setpc_b64 s[30:31]
-;
-; AGPR-GISEL-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_b:
-; AGPR-GISEL: ; %bb.0:
-; AGPR-GISEL-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a0, v16
-; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a1, v17
-; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a2, v18
-; AGPR-GISEL-NEXT: v_accvgpr_write_b32 a3, v19
-; AGPR-GISEL-NEXT: v_mov_b32_e32 v16, 0
-; AGPR-GISEL-NEXT: s_nop 1
-; AGPR-GISEL-NEXT: v_mfma_scale_f32_16x16x128_f8f6f4 a[0:3], v[0:7], v[8:15], a[0:3], v16, v16 op_sel:[1,1,0] op_sel_hi:[1,0,0]
-; AGPR-GISEL-NEXT: s_nop 11
-; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v0, a0
-; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v1, a1
-; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v2, a2
-; AGPR-GISEL-NEXT: v_accvgpr_read_b32 v3, a3
-; AGPR-GISEL-NEXT: s_setpc_b64 s[30:31]
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_0_0_b:
+; AGPR: ; %bb.0:
+; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
+; AGPR-NEXT: v_accvgpr_write_b32 a1, v17
+; AGPR-NEXT: v_accvgpr_write_b32 a2, v18
+; AGPR-NEXT: v_accvgpr_write_b32 a3, v19
+; AGPR-NEXT: s_nop 1
+; AGPR-NEXT: v_mfma_f32_16x16x128_f8f6f4 a[0:3], v[0:7], v[8:15], a[0:3]
+; AGPR-NEXT: s_nop 11
+; AGPR-NEXT: v_accvgpr_read_b32 v0, a0
+; AGPR-NEXT: v_accvgpr_read_b32 v1, a1
+; AGPR-NEXT: v_accvgpr_read_b32 v2, a2
+; AGPR-NEXT: v_accvgpr_read_b32 v3, a3
+; AGPR-NEXT: s_setpc_b64 s[30:31]
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2, i32 0, i32 0, i32 3, i32 0, i32 1, i32 0)
ret <4 x float> %result
}
@@ -3326,6 +3286,32 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_1_0_a(
ret <4 x float> %result
}
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_127_127:
+; GCN: ; %bb.0:
+; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19]
+; GCN-NEXT: s_setpc_b64 s[30:31]
+;
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_127_127:
+; AGPR: ; %bb.0:
+; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
+; AGPR-NEXT: v_accvgpr_write_b32 a1, v17
+; AGPR-NEXT: v_accvgpr_write_b32 a2, v18
+; AGPR-NEXT: v_accvgpr_write_b32 a3, v19
+; AGPR-NEXT: s_nop 1
+; AGPR-NEXT: v_mfma_f32_16x16x128_f8f6f4 a[0:3], v[0:7], v[8:15], a[0:3]
+; AGPR-NEXT: s_nop 11
+; AGPR-NEXT: v_accvgpr_read_b32 v0, a0
+; AGPR-NEXT: v_accvgpr_read_b32 v1, a1
+; AGPR-NEXT: v_accvgpr_read_b32 v2, a2
+; AGPR-NEXT: v_accvgpr_read_b32 v3, a3
+; AGPR-NEXT: s_setpc_b64 s[30:31]
+ %result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2, i32 0, i32 0, i32 0, i32 127, i32 0, i32 127)
+ ret <4 x float> %result
+}
+
define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_127_127__opsel1(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___constant_scale_127_127__opsel1:
; GCN: ; %bb.0:
@@ -3521,14 +3507,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp6__v8i32_fp6(
ret <4 x float> %result
}
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp6__v8i32_fp6__127_scale(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp6__v8i32_fp6__127_scale:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp6__v8i32_fp6__0_scale(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp6__v8i32_fp6__0_scale:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19] cbsz:2 blgp:2
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp6__v8i32_fp6__127_scale:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp6__v8i32_fp6__0_scale:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
@@ -3546,7 +3532,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp6__v8i32_fp6_
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 2, ; cbsz
i32 2, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
@@ -3695,14 +3681,14 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp4__v8i32_fp4(
ret <4 x float> %result
}
-define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp4__v8i32_fp4__127_scale(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
-; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp4__v8i32_fp4__127_scale:
+define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp4__v8i32_fp4__0_scale(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2) #2 {
+; GCN-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp4__v8i32_fp4__0_scale:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_16x16x128_f8f6f4 v[0:3], v[0:7], v[8:15], v[16:19] cbsz:4 blgp:4
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp4__v8i32_fp4__127_scale:
+; AGPR-LABEL: test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp4__v8i32_fp4__0_scale:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
@@ -3720,7 +3706,7 @@ define <4 x float> @test_mfma_scale_f32_16x16x128_f8f6f4___v8i32_fp4__v8i32_fp4_
%result = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %arg2,
i32 4, ; cbsz
i32 4, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <4 x float> %result
}
diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.ll
index 6e74a96fcc1ae1..3cebfb43691171 100644
--- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.ll
+++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.ll
@@ -1224,8 +1224,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_3_2__cbsz1__blgp1(<8 x
}
; This should be optimized to avoid the scale
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0__constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0__constant_scale_0_0(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: scratch_load_dword v31, off, s32
@@ -1251,7 +1251,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0__cons
; GCN-NEXT: v_mov_b32_e32 v15, v31
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: scratch_load_dword a15, off, s32
@@ -1295,7 +1295,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 0, ; cbsz
i32 0, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -1451,8 +1451,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1(<8 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1__constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1__constant_scale_0_0(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: scratch_load_dword v31, off, s32
@@ -1478,7 +1478,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1__cons
; GCN-NEXT: v_mov_b32_e32 v15, v31
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: scratch_load_dword a15, off, s32
@@ -1522,7 +1522,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp1__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 0, ; cbsz
i32 1, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -1603,8 +1603,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2(<8 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2__constant_scale_127_127(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2__constant_scale_0_0(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[14:29], v[0:7], v[8:13], v[14:29] blgp:2
@@ -1628,7 +1628,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2__cons
; GCN-NEXT: v_mov_b32_e32 v15, v29
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -1671,7 +1671,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp2__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v6i32(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 0, ; cbsz
i32 2, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -1752,8 +1752,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3(<8 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3__constant_scale_127_127(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3__constant_scale_0_0(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[14:29], v[0:7], v[8:13], v[14:29] blgp:3
@@ -1777,7 +1777,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3__cons
; GCN-NEXT: v_mov_b32_e32 v15, v29
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -1820,7 +1820,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp3__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v6i32(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 0, ; cbsz
i32 3, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -1897,8 +1897,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4(<8 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4__constant_scale_127_127(<8 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4__constant_scale_0_0(<8 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[12:27], v[0:7], v[8:11], v[12:27] blgp:4
@@ -1922,7 +1922,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4__cons
; GCN-NEXT: v_mov_b32_e32 v15, v27
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -1965,7 +1965,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp4__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v4i32(<8 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2,
i32 0, ; cbsz
i32 4, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -2121,8 +2121,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0(<8 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0__constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0__constant_scale_0_0(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: scratch_load_dword v31, off, s32
@@ -2148,7 +2148,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0__cons
; GCN-NEXT: v_mov_b32_e32 v15, v31
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: scratch_load_dword a15, off, s32
@@ -2192,7 +2192,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp0__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 1, ; cbsz
i32 0, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -2349,8 +2349,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1(<8 x
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1__constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1__constant_scale_0_0(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: scratch_load_dword v31, off, s32
@@ -2376,7 +2376,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1__cons
; GCN-NEXT: v_mov_b32_e32 v15, v31
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: scratch_load_dword a15, off, s32
@@ -2420,7 +2420,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp1__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 1, ; cbsz
i32 1, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -2501,8 +2501,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2(<8 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2__constant_scale_127_127(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2__constant_scale_0(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2__constant_scale_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[14:29], v[0:7], v[8:13], v[14:29] cbsz:1 blgp:2
@@ -2526,7 +2526,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2__cons
; GCN-NEXT: v_mov_b32_e32 v15, v29
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2__constant_scale_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -2569,7 +2569,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp2__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v6i32(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 1, ; cbsz
i32 2, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -2650,8 +2650,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3(<8 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3__constant_scale_127_127(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3__constant_scale_0_0(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[14:29], v[0:7], v[8:13], v[14:29] cbsz:1 blgp:3
@@ -2675,7 +2675,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3__cons
; GCN-NEXT: v_mov_b32_e32 v15, v29
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -2718,7 +2718,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp3__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v6i32(<8 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 1, ; cbsz
i32 3, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -2795,8 +2795,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4(<8 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4__constant_scale_127_127(<8 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4__constant_scale_0_0(<8 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[12:27], v[0:7], v[8:11], v[12:27] cbsz:1 blgp:4
@@ -2820,7 +2820,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4__cons
; GCN-NEXT: v_mov_b32_e32 v15, v27
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -2863,7 +2863,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz1__blgp4__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v4i32(<8 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2,
i32 1, ; cbsz
i32 4, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -2944,8 +2944,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0__constant_scale_127_127(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0__constant_scale_0_0(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[14:29], v[0:5], v[6:13], v[14:29] cbsz:2
@@ -2969,7 +2969,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0__cons
; GCN-NEXT: v_mov_b32_e32 v15, v29
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -3012,7 +3012,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp0__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v8i32(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 2, ; cbsz
i32 0, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -3093,8 +3093,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1__constant_scale_127_127(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1__constant_scale_0_0(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[14:29], v[0:5], v[6:13], v[14:29] cbsz:2 blgp:1
@@ -3118,7 +3118,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1__cons
; GCN-NEXT: v_mov_b32_e32 v15, v29
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -3161,7 +3161,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp1__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v8i32(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 2, ; cbsz
i32 1, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -3236,8 +3236,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2__constant_scale_127_127(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2__constant_scale_0_0(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[12:27], v[0:5], v[6:11], v[12:27] cbsz:2 blgp:2
@@ -3260,7 +3260,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2__cons
; GCN-NEXT: v_mov_b32_e32 v15, v27
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -3302,7 +3302,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 2, ; cbsz
i32 2, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -3377,8 +3377,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3__constant_scale_127_127(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3__constant_scale_0_0(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[12:27], v[0:5], v[6:11], v[12:27] cbsz:2 blgp:3
@@ -3401,7 +3401,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3__cons
; GCN-NEXT: v_mov_b32_e32 v15, v27
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -3443,7 +3443,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp3__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 2, ; cbsz
i32 3, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -3525,8 +3525,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0__constant_scale_127_127(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0__constant_scale_0_0(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[14:29], v[0:5], v[6:13], v[14:29] cbsz:3
@@ -3550,7 +3550,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0__cons
; GCN-NEXT: v_mov_b32_e32 v15, v29
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -3593,7 +3593,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp0__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v8i32(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 3, ; cbsz
i32 0, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -3674,8 +3674,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1__constant_scale_127_127(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1__constant_scale_0_0(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[14:29], v[0:5], v[6:13], v[14:29] cbsz:3 blgp:1
@@ -3699,7 +3699,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1__cons
; GCN-NEXT: v_mov_b32_e32 v15, v29
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v14
@@ -3742,7 +3742,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp1__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v8i32(<6 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 3, ; cbsz
i32 1, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -3817,8 +3817,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2__constant_scale_127_127(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2__constant_scale_0_0(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[12:27], v[0:5], v[6:11], v[12:27] cbsz:3 blgp:2
@@ -3841,7 +3841,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2__cons
; GCN-NEXT: v_mov_b32_e32 v15, v27
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -3883,7 +3883,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp2__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 3, ; cbsz
i32 2, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -3958,8 +3958,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4__constant_scale_127_127(<6 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4__constant_scale_0_0(<6 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[10:25], v[0:5], v[6:9], v[10:25] cbsz:3 blgp:4
@@ -3982,7 +3982,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4__cons
; GCN-NEXT: v_mov_b32_e32 v15, v25
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v10
@@ -4024,7 +4024,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp4__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v4i32(<6 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2,
i32 3, ; cbsz
i32 4, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -4099,8 +4099,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3__constant_scale_127_127(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3__constant_scale_0_0(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[12:27], v[0:5], v[6:11], v[12:27] cbsz:3 blgp:3
@@ -4123,7 +4123,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3__cons
; GCN-NEXT: v_mov_b32_e32 v15, v27
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -4165,7 +4165,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz3__blgp3__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 3, ; cbsz
i32 3, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -4240,8 +4240,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4(<6 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4__constant_scale_127_127(<6 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4__constant_scale_0_0(<6 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[10:25], v[0:5], v[6:9], v[10:25] cbsz:2 blgp:4
@@ -4264,7 +4264,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4__cons
; GCN-NEXT: v_mov_b32_e32 v15, v25
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v10
@@ -4306,7 +4306,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp4__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v4i32(<6 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2,
i32 2, ; cbsz
i32 4, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -4383,8 +4383,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0(<4 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0__constant_scale_127_127(<4 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0__constant_scale_0_0(<4 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[12:27], v[0:3], v[4:11], v[12:27] cbsz:4
@@ -4408,7 +4408,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0__cons
; GCN-NEXT: v_mov_b32_e32 v15, v27
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -4451,7 +4451,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp0__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v4i32.v8i32(<4 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 4, ; cbsz
i32 0, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -4528,8 +4528,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1(<4 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1__constant_scale_127_127(<4 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1__constant_scale_0_0(<4 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[12:27], v[0:3], v[4:11], v[12:27] cbsz:4 blgp:1
@@ -4553,7 +4553,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1__cons
; GCN-NEXT: v_mov_b32_e32 v15, v27
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v12
@@ -4596,7 +4596,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp1__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v4i32.v8i32(<4 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 4, ; cbsz
i32 1, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -4671,8 +4671,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2(<4 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2__constant_scale_127_127(<4 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2__constant_scale_0_0(<4 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[10:25], v[0:3], v[4:9], v[10:25] cbsz:4 blgp:2
@@ -4695,7 +4695,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2__cons
; GCN-NEXT: v_mov_b32_e32 v15, v25
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v10
@@ -4737,7 +4737,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp2__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v4i32.v6i32(<4 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 4, ; cbsz
i32 2, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -4812,8 +4812,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3(<4 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3__constant_scale_127_127(<4 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3__constant_scale_0_0(<4 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[10:25], v[0:3], v[4:9], v[10:25] cbsz:4 blgp:3
@@ -4836,7 +4836,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3__cons
; GCN-NEXT: v_mov_b32_e32 v15, v25
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v10
@@ -4878,7 +4878,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp3__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v4i32.v6i32(<4 x i32> %arg0, <6 x i32> %arg1, <16 x float> %arg2,
i32 4, ; cbsz
i32 3, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -4953,8 +4953,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4(<4 x
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4__constant_scale_127_127(<4 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4__constant_scale_127_127:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4__constant_scale_0_0(<4 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4__constant_scale_0_0:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[8:23], v[0:3], v[4:7], v[8:23] cbsz:4 blgp:4
@@ -4977,7 +4977,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4__cons
; GCN-NEXT: v_mov_b32_e32 v15, v23
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4__constant_scale_127_127:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4__constant_scale_0_0:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: v_accvgpr_write_b32 a0, v8
@@ -5019,7 +5019,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4__cons
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v4i32.v4i32(<4 x i32> %arg0, <4 x i32> %arg1, <16 x float> %arg2,
i32 4, ; cbsz
i32 4, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -8211,8 +8211,8 @@ define amdgpu_kernel void @test_mfma_scale_f32_32x32x64_f8f6f4_25_42__nonmac(<8
ret void
}
-define amdgpu_kernel void @test_mfma_scale_f32_32x32x64_f8f6f4_127_127__vgprcd_nonmac(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) #0 {
-; SDAG-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_127_127__vgprcd_nonmac:
+define amdgpu_kernel void @test_mfma_scale_f32_32x32x64_f8f6f4_0_0__vgprcd_nonmac(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) #0 {
+; SDAG-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__vgprcd_nonmac:
; SDAG: ; %bb.0:
; SDAG-NEXT: s_load_dwordx16 s[8:23], s[4:5], 0x0
; SDAG-NEXT: s_load_dwordx16 s[36:51], s[4:5], 0x40
@@ -8282,7 +8282,7 @@ define amdgpu_kernel void @test_mfma_scale_f32_32x32x64_f8f6f4_127_127__vgprcd_n
; SDAG-NEXT: s_waitcnt vmcnt(0)
; SDAG-NEXT: s_endpgm
;
-; GISEL-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_127_127__vgprcd_nonmac:
+; GISEL-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__vgprcd_nonmac:
; GISEL: ; %bb.0:
; GISEL-NEXT: s_load_dwordx16 s[36:51], s[4:5], 0x0
; GISEL-NEXT: s_load_dwordx16 s[8:23], s[4:5], 0x40
@@ -8336,7 +8336,7 @@ define amdgpu_kernel void @test_mfma_scale_f32_32x32x64_f8f6f4_127_127__vgprcd_n
; GISEL-NEXT: s_waitcnt vmcnt(0)
; GISEL-NEXT: s_endpgm
;
-; AGPR-SDAG-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_127_127__vgprcd_nonmac:
+; AGPR-SDAG-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__vgprcd_nonmac:
; AGPR-SDAG: ; %bb.0:
; AGPR-SDAG-NEXT: s_load_dwordx16 s[8:23], s[4:5], 0x0
; AGPR-SDAG-NEXT: s_load_dwordx16 s[36:51], s[4:5], 0x40
@@ -8406,7 +8406,7 @@ define amdgpu_kernel void @test_mfma_scale_f32_32x32x64_f8f6f4_127_127__vgprcd_n
; AGPR-SDAG-NEXT: s_waitcnt vmcnt(0)
; AGPR-SDAG-NEXT: s_endpgm
;
-; AGPR-GISEL-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_127_127__vgprcd_nonmac:
+; AGPR-GISEL-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4_0_0__vgprcd_nonmac:
; AGPR-GISEL: ; %bb.0:
; AGPR-GISEL-NEXT: s_load_dwordx16 s[36:51], s[4:5], 0x0
; AGPR-GISEL-NEXT: s_load_dwordx16 s[8:23], s[4:5], 0x40
@@ -8459,7 +8459,7 @@ define amdgpu_kernel void @test_mfma_scale_f32_32x32x64_f8f6f4_127_127__vgprcd_n
; AGPR-GISEL-NEXT: global_store_dwordx4 v[18:19], v[12:15], off sc0 sc1
; AGPR-GISEL-NEXT: s_waitcnt vmcnt(0)
; AGPR-GISEL-NEXT: s_endpgm
- %result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2, i32 0, i32 2, i32 0, i32 127, i32 0, i32 127)
+ %result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2, i32 0, i32 2, i32 0, i32 0, i32 0, i32 0)
store volatile <16 x float> %arg2, ptr addrspace(1) null, align 64
store volatile <16 x float> %result, ptr addrspace(1) null, align 64
ret void
@@ -8730,10 +8730,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___constant_scale_0_0_a(
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: scratch_load_dword v31, off, s32
-; GCN-NEXT: v_mov_b32_e32 v32, 0
; GCN-NEXT: s_waitcnt vmcnt(0)
-; GCN-NEXT: s_nop 0
-; GCN-NEXT: v_mfma_scale_f32_32x32x64_f8f6f4 v[16:31], v[0:7], v[8:15], v[16:31], v32, v32 op_sel_hi:[0,0,0]
+; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[16:31], v[0:7], v[8:15], v[16:31]
; GCN-NEXT: s_nop 15
; GCN-NEXT: s_nop 3
; GCN-NEXT: v_mov_b32_e32 v0, v16
@@ -8758,7 +8756,6 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___constant_scale_0_0_a(
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: scratch_load_dword a15, off, s32
-; AGPR-NEXT: v_mov_b32_e32 v31, 0
; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
; AGPR-NEXT: v_accvgpr_write_b32 a1, v17
; AGPR-NEXT: v_accvgpr_write_b32 a2, v18
@@ -8776,7 +8773,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___constant_scale_0_0_a(
; AGPR-NEXT: v_accvgpr_write_b32 a14, v30
; AGPR-NEXT: s_waitcnt vmcnt(0)
; AGPR-NEXT: s_nop 0
-; AGPR-NEXT: v_mfma_scale_f32_32x32x64_f8f6f4 a[0:15], v[0:7], v[8:15], a[0:15], v31, v31 op_sel_hi:[0,0,0]
+; AGPR-NEXT: v_mfma_f32_32x32x64_f8f6f4 a[0:15], v[0:7], v[8:15], a[0:15]
; AGPR-NEXT: s_nop 15
; AGPR-NEXT: s_nop 3
; AGPR-NEXT: v_accvgpr_read_b32 v0, a0
@@ -8805,10 +8802,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___constant_scale_0_0_b(
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: scratch_load_dword v31, off, s32
-; GCN-NEXT: v_mov_b32_e32 v32, 0
; GCN-NEXT: s_waitcnt vmcnt(0)
-; GCN-NEXT: s_nop 0
-; GCN-NEXT: v_mfma_scale_f32_32x32x64_f8f6f4 v[16:31], v[0:7], v[8:15], v[16:31], v32, v32 op_sel:[1,1,0] op_sel_hi:[1,0,0]
+; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[16:31], v[0:7], v[8:15], v[16:31]
; GCN-NEXT: s_nop 15
; GCN-NEXT: s_nop 3
; GCN-NEXT: v_mov_b32_e32 v0, v16
@@ -8833,7 +8828,6 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___constant_scale_0_0_b(
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: scratch_load_dword a15, off, s32
-; AGPR-NEXT: v_mov_b32_e32 v31, 0
; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
; AGPR-NEXT: v_accvgpr_write_b32 a1, v17
; AGPR-NEXT: v_accvgpr_write_b32 a2, v18
@@ -8851,7 +8845,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___constant_scale_0_0_b(
; AGPR-NEXT: v_accvgpr_write_b32 a14, v30
; AGPR-NEXT: s_waitcnt vmcnt(0)
; AGPR-NEXT: s_nop 0
-; AGPR-NEXT: v_mfma_scale_f32_32x32x64_f8f6f4 a[0:15], v[0:7], v[8:15], a[0:15], v31, v31 op_sel:[1,1,0] op_sel_hi:[1,0,0]
+; AGPR-NEXT: v_mfma_f32_32x32x64_f8f6f4 a[0:15], v[0:7], v[8:15], a[0:15]
; AGPR-NEXT: s_nop 15
; AGPR-NEXT: s_nop 3
; AGPR-NEXT: v_accvgpr_read_b32 v0, a0
@@ -9175,6 +9169,78 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___constant_scale_1_0_a(
ret <16 x float> %result
}
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___constant_scale_127_127(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___constant_scale_127_127:
+; GCN: ; %bb.0:
+; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GCN-NEXT: scratch_load_dword v31, off, s32
+; GCN-NEXT: s_waitcnt vmcnt(0)
+; GCN-NEXT: v_mfma_f32_32x32x64_f8f6f4 v[16:31], v[0:7], v[8:15], v[16:31]
+; GCN-NEXT: s_nop 15
+; GCN-NEXT: s_nop 3
+; GCN-NEXT: v_mov_b32_e32 v0, v16
+; GCN-NEXT: v_mov_b32_e32 v1, v17
+; GCN-NEXT: v_mov_b32_e32 v2, v18
+; GCN-NEXT: v_mov_b32_e32 v3, v19
+; GCN-NEXT: v_mov_b32_e32 v4, v20
+; GCN-NEXT: v_mov_b32_e32 v5, v21
+; GCN-NEXT: v_mov_b32_e32 v6, v22
+; GCN-NEXT: v_mov_b32_e32 v7, v23
+; GCN-NEXT: v_mov_b32_e32 v8, v24
+; GCN-NEXT: v_mov_b32_e32 v9, v25
+; GCN-NEXT: v_mov_b32_e32 v10, v26
+; GCN-NEXT: v_mov_b32_e32 v11, v27
+; GCN-NEXT: v_mov_b32_e32 v12, v28
+; GCN-NEXT: v_mov_b32_e32 v13, v29
+; GCN-NEXT: v_mov_b32_e32 v14, v30
+; GCN-NEXT: v_mov_b32_e32 v15, v31
+; GCN-NEXT: s_setpc_b64 s[30:31]
+;
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___constant_scale_127_127:
+; AGPR: ; %bb.0:
+; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; AGPR-NEXT: scratch_load_dword a15, off, s32
+; AGPR-NEXT: v_accvgpr_write_b32 a0, v16
+; AGPR-NEXT: v_accvgpr_write_b32 a1, v17
+; AGPR-NEXT: v_accvgpr_write_b32 a2, v18
+; AGPR-NEXT: v_accvgpr_write_b32 a3, v19
+; AGPR-NEXT: v_accvgpr_write_b32 a4, v20
+; AGPR-NEXT: v_accvgpr_write_b32 a5, v21
+; AGPR-NEXT: v_accvgpr_write_b32 a6, v22
+; AGPR-NEXT: v_accvgpr_write_b32 a7, v23
+; AGPR-NEXT: v_accvgpr_write_b32 a8, v24
+; AGPR-NEXT: v_accvgpr_write_b32 a9, v25
+; AGPR-NEXT: v_accvgpr_write_b32 a10, v26
+; AGPR-NEXT: v_accvgpr_write_b32 a11, v27
+; AGPR-NEXT: v_accvgpr_write_b32 a12, v28
+; AGPR-NEXT: v_accvgpr_write_b32 a13, v29
+; AGPR-NEXT: v_accvgpr_write_b32 a14, v30
+; AGPR-NEXT: s_waitcnt vmcnt(0)
+; AGPR-NEXT: s_nop 0
+; AGPR-NEXT: v_mfma_f32_32x32x64_f8f6f4 a[0:15], v[0:7], v[8:15], a[0:15]
+; AGPR-NEXT: s_nop 15
+; AGPR-NEXT: s_nop 3
+; AGPR-NEXT: v_accvgpr_read_b32 v0, a0
+; AGPR-NEXT: v_accvgpr_read_b32 v1, a1
+; AGPR-NEXT: v_accvgpr_read_b32 v2, a2
+; AGPR-NEXT: v_accvgpr_read_b32 v3, a3
+; AGPR-NEXT: v_accvgpr_read_b32 v4, a4
+; AGPR-NEXT: v_accvgpr_read_b32 v5, a5
+; AGPR-NEXT: v_accvgpr_read_b32 v6, a6
+; AGPR-NEXT: v_accvgpr_read_b32 v7, a7
+; AGPR-NEXT: v_accvgpr_read_b32 v8, a8
+; AGPR-NEXT: v_accvgpr_read_b32 v9, a9
+; AGPR-NEXT: v_accvgpr_read_b32 v10, a10
+; AGPR-NEXT: v_accvgpr_read_b32 v11, a11
+; AGPR-NEXT: v_accvgpr_read_b32 v12, a12
+; AGPR-NEXT: v_accvgpr_read_b32 v13, a13
+; AGPR-NEXT: v_accvgpr_read_b32 v14, a14
+; AGPR-NEXT: v_accvgpr_read_b32 v15, a15
+; AGPR-NEXT: s_setpc_b64 s[30:31]
+ %result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2, i32 0, i32 0, i32 0, i32 127, i32 0, i32 127)
+ ret <16 x float> %result
+}
+
; --------------------------------------------------------------------
; Incorrect signature for format cases (IR vector too large)
; --------------------------------------------------------------------
@@ -9628,8 +9694,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6(
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6__127_scale(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6__127_scale:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6__0_scale(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6__0_scale:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: scratch_load_dword v31, off, s32
@@ -9654,7 +9720,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6_
; GCN-NEXT: v_mov_b32_e32 v15, v31
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6__127_scale:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6__0_scale:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: scratch_load_dword a15, off, s32
@@ -9697,7 +9763,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp6__v8i32_fp6_
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 2, ; cbsz
i32 2, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
@@ -10302,8 +10368,8 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4(
ret <16 x float> %result
}
-define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4__127_scale(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
-; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4__127_scale:
+define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4__0_scale(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2) {
+; GCN-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4__0_scale:
; GCN: ; %bb.0:
; GCN-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; GCN-NEXT: scratch_load_dword v31, off, s32
@@ -10328,7 +10394,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4_
; GCN-NEXT: v_mov_b32_e32 v15, v31
; GCN-NEXT: s_setpc_b64 s[30:31]
;
-; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4__127_scale:
+; AGPR-LABEL: test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4__0_scale:
; AGPR: ; %bb.0:
; AGPR-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
; AGPR-NEXT: scratch_load_dword a15, off, s32
@@ -10371,7 +10437,7 @@ define <16 x float> @test_mfma_scale_f32_32x32x64_f8f6f4___v8i32_fp4__v8i32_fp4_
%result = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %arg2,
i32 4, ; cbsz
i32 4, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
ret <16 x float> %result
}
diff --git a/llvm/test/CodeGen/AMDGPU/rewrite-vgpr-mfma-to-agpr.gfx950.ll b/llvm/test/CodeGen/AMDGPU/rewrite-vgpr-mfma-to-agpr.gfx950.ll
index 248822c1d50c95..b163b57883481f 100644
--- a/llvm/test/CodeGen/AMDGPU/rewrite-vgpr-mfma-to-agpr.gfx950.ll
+++ b/llvm/test/CodeGen/AMDGPU/rewrite-vgpr-mfma-to-agpr.gfx950.ll
@@ -149,7 +149,7 @@ define void @test_rewrite_mfma_f32_16x16x128_f8f6f4_0_0__cbsz0__blgp0(<8 x i32>
%mai = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <4 x float> %src2,
i32 0, ; cbsz
i32 0, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
call void asm sideeffect "; use $0", "a"(<4 x float> %mai)
ret void
}
@@ -189,7 +189,7 @@ define void @test_rewrite_mfma_f32_16x16x128_f8f6f4_0_0__cbsz2__blgp2(<6 x i32>
%mai = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <4 x float> %src2,
i32 2, ; cbsz
i32 2, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
call void asm sideeffect "; use $0", "a"(<4 x float> %mai)
ret void
}
@@ -229,7 +229,7 @@ define void @test_rewrite_mfma_f32_16x16x128_f8f6f4_0_0__cbsz4__blgp4(<4 x i32>
%mai = call <4 x float> @llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4.v4i32.v4i32(<4 x i32> %arg0, <4 x i32> %arg1, <4 x float> %src2,
i32 4, ; cbsz
i32 4, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
call void asm sideeffect "; use $0", "a"(<4 x float> %mai)
ret void
}
@@ -276,7 +276,7 @@ define void @test_rewrite_mfma_f32_32x32x64_f8f6f4_0_0__cbsz0__blgp0(<8 x i32> %
%mai = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32(<8 x i32> %arg0, <8 x i32> %arg1, <16 x float> %src2,
i32 0, ; cbsz
i32 0, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
call void asm sideeffect "; use $0", "a"(<16 x float> %mai)
ret void
}
@@ -322,7 +322,7 @@ define void @test_rewrite_mfma_f32_32x32x64_f8f6f4_0_0__cbsz2__blgp2(<6 x i32> %
%mai = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v6i32.v6i32(<6 x i32> %arg0, <6 x i32> %arg1, <16 x float> %src2,
i32 2, ; cbsz
i32 2, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
call void asm sideeffect "; use $0", "a"(<16 x float> %mai)
ret void
}
@@ -368,7 +368,7 @@ define void @test_rewrite_mfma_f32_32x32x64_f8f6f4_0_0__cbsz4__blgp4(<4 x i32> %
%mai = call <16 x float> @llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4.v4i32.v4i32(<4 x i32> %arg0, <4 x i32> %arg1, <16 x float> %src2,
i32 4, ; cbsz
i32 4, ; blgp
- i32 0, i32 127, i32 0, i32 127)
+ i32 0, i32 0, i32 0, i32 0)
call void asm sideeffect "; use $0", "a"(<16 x float> %mai)
ret void
}
diff --git a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
index 1fcbfe84aa9f7a..92a47dab42f019 100644
--- a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
+++ b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
@@ -1593,10 +1593,9 @@ struct MFMAOpLowering : public ConvertOpToLLVMPattern<MFMAOp> {
rewriter, loc, adaptor.getSourceB(), allowBf16),
adaptor.getDestC()});
if (isScaled) {
- // Scales are E8M0 (2^(s-127)), so 127 is a scale of 1.0.
- Value one = createI32Constant(rewriter, loc, 127);
+ Value zero = createI32Constant(rewriter, loc, 0);
auto [_scaledName, aTypeCode, bTypeCode] = *maybeScaledIntrinsic;
- loweredOp.addOperands({/*scale A=*/one, /*scale B=*/one});
+ loweredOp.addOperands({/*scale A=*/zero, /*scale B=*/zero});
loweredOp.addAttributes(
{{"cbsz",
ROCDL::MatrixFormatAttr::get(rewriter.getContext(), aTypeCode)},
diff --git a/mlir/test/Conversion/AMDGPUToROCDL/mfma-gfx950.mlir b/mlir/test/Conversion/AMDGPUToROCDL/mfma-gfx950.mlir
index 5edc852c92cf4d..8e5bd123121bec 100644
--- a/mlir/test/Conversion/AMDGPUToROCDL/mfma-gfx950.mlir
+++ b/mlir/test/Conversion/AMDGPUToROCDL/mfma-gfx950.mlir
@@ -19,31 +19,31 @@ func.func @mfma_to_rocdl(%arg0 : vector<8xf16>, %arg1 : vector<16xf32>,
// CHECK: rocdl.mfma.i32.16x16x64.i8{{.*}}: (vector<4xi32>, vector<4xi32>, vector<4xi32>) -> vector<4xi32>
amdgpu.mfma 16x16x64 %arg4 * %arg4 + %arg6 : vector<16xi8>, vector<16xi8>, vector<4xi32>
- // CHECK: %[[c127:.+]] = llvm.mlir.constant(127 : i32) : i32
- // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp8_e4m3, fp8_e4m3, 0, %[[c127]], 0, %[[c127]] : (vector<8xi32>, vector<8xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
+ // CHECK: %[[c0:.+]] = llvm.mlir.constant(0 : i32) : i32
+ // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp8_e4m3, fp8_e4m3, 0, %[[c0]], 0, %[[c0]] : (vector<8xi32>, vector<8xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
amdgpu.mfma 32x32x64 %arg7 * %arg7 + %arg1 : vector<32xf8E4M3FN>, vector<32xf8E4M3FN>, vector<16xf32>
- // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp8_e4m3, fp8_e4m3, 0, %[[c127]], 0, %[[c127]] : (vector<8xi32>, vector<8xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
+ // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp8_e4m3, fp8_e4m3, 0, %[[c0]], 0, %[[c0]] : (vector<8xi32>, vector<8xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
amdgpu.mfma 16x16x128 %arg7 * %arg7 + %arg2 : vector<32xf8E4M3FN>, vector<32xf8E4M3FN>, vector<4xf32>
- // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp8_e5m2, fp8_e5m2, 0, %[[c127]], 0, %[[c127]] : (vector<8xi32>, vector<8xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
+ // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp8_e5m2, fp8_e5m2, 0, %[[c0]], 0, %[[c0]] : (vector<8xi32>, vector<8xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
amdgpu.mfma 32x32x64 %arg8 * %arg8 + %arg1 : vector<32xf8E5M2>, vector<32xf8E5M2>, vector<16xf32>
- // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp8_e5m2, fp8_e5m2, 0, %[[c127]], 0, %[[c127]] : (vector<8xi32>, vector<8xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
+ // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp8_e5m2, fp8_e5m2, 0, %[[c0]], 0, %[[c0]] : (vector<8xi32>, vector<8xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
amdgpu.mfma 16x16x128 %arg8 * %arg8 + %arg2 : vector<32xf8E5M2>, vector<32xf8E5M2>, vector<4xf32>
- // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e2m3, fp6_e2m3, 0, %[[c127]], 0, %[[c127]] : (vector<6xi32>, vector<6xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
+ // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e2m3, fp6_e2m3, 0, %[[c0]], 0, %[[c0]] : (vector<6xi32>, vector<6xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
amdgpu.mfma 32x32x64 %arg9 * %arg9 + %arg1 : vector<32xf6E2M3FN>, vector<32xf6E2M3FN>, vector<16xf32>
- // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e2m3, fp6_e2m3, 0, %[[c127]], 0, %[[c127]] : (vector<6xi32>, vector<6xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
+ // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e2m3, fp6_e2m3, 0, %[[c0]], 0, %[[c0]] : (vector<6xi32>, vector<6xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
amdgpu.mfma 16x16x128 %arg9 * %arg9 + %arg2 : vector<32xf6E2M3FN>, vector<32xf6E2M3FN>, vector<4xf32>
- // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e3m2, fp6_e3m2, 0, %[[c127]], 0, %[[c127]] : (vector<6xi32>, vector<6xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
+ // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e3m2, fp6_e3m2, 0, %[[c0]], 0, %[[c0]] : (vector<6xi32>, vector<6xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
amdgpu.mfma 32x32x64 %arg10 * %arg10 + %arg1 : vector<32xf6E3M2FN>, vector<32xf6E3M2FN>, vector<16xf32>
- // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e3m2, fp6_e3m2, 0, %[[c127]], 0, %[[c127]] : (vector<6xi32>, vector<6xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
+ // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e3m2, fp6_e3m2, 0, %[[c0]], 0, %[[c0]] : (vector<6xi32>, vector<6xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
amdgpu.mfma 16x16x128 %arg10 * %arg10 + %arg2 : vector<32xf6E3M2FN>, vector<32xf6E3M2FN>, vector<4xf32>
- // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp4_e2m1, fp4_e2m1, 0, %[[c127]], 0, %[[c127]] : (vector<4xi32>, vector<4xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
+ // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp4_e2m1, fp4_e2m1, 0, %[[c0]], 0, %[[c0]] : (vector<4xi32>, vector<4xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
amdgpu.mfma 32x32x64 %arg11 * %arg11 + %arg1 : vector<32xf4E2M1FN>, vector<32xf4E2M1FN>, vector<16xf32>
- // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp4_e2m1, fp4_e2m1, 0, %[[c127]], 0, %[[c127]] : (vector<4xi32>, vector<4xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
+ // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp4_e2m1, fp4_e2m1, 0, %[[c0]], 0, %[[c0]] : (vector<4xi32>, vector<4xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
amdgpu.mfma 16x16x128 %arg11 * %arg11 + %arg2 : vector<32xf4E2M1FN>, vector<32xf4E2M1FN>, vector<4xf32>
- // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e2m3, fp4_e2m1, 0, %[[c127]], 0, %[[c127]] : (vector<6xi32>, vector<4xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
+ // CHECK: rocdl.mfma.scale.f32.32x32x64.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e2m3, fp4_e2m1, 0, %[[c0]], 0, %[[c0]] : (vector<6xi32>, vector<4xi32>, vector<16xf32>, i32, i32) -> vector<16xf32>
amdgpu.mfma 32x32x64 %arg9 * %arg11 + %arg1 : vector<32xf6E2M3FN>, vector<32xf4E2M1FN>, vector<16xf32>
- // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e2m3, fp4_e2m1, 0, %[[c127]], 0, %[[c127]] : (vector<6xi32>, vector<4xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
+ // CHECK: rocdl.mfma.scale.f32.16x16x128.f8f6f4 %{{.*}}, %{{.*}}, %{{.*}}, fp6_e2m3, fp4_e2m1, 0, %[[c0]], 0, %[[c0]] : (vector<6xi32>, vector<4xi32>, vector<4xf32>, i32, i32) -> vector<4xf32>
amdgpu.mfma 16x16x128 %arg9 * %arg11 + %arg2 : vector<32xf6E2M3FN>, vector<32xf4E2M1FN>, vector<4xf32>
func.return
>From bea5f6e225994c1262172b95bff7626eec5cac0b Mon Sep 17 00:00:00 2001
From: Keshav Vinayak Jha <keshavvinayakjha at gmail.com>
Date: Fri, 16 Oct 2026 09:19:23 +0000
Subject: [PATCH 3/4] Add simpler comment to folder; remove comment from
intrinisic
Signed-off-by: Keshav Vinayak Jha <keshavvinayakjha at gmail.com>
---
llvm/include/llvm/IR/IntrinsicsAMDGPU.td | 6 ------
llvm/lib/Target/AMDGPU/SIInstrInfo.td | 14 +++++++++-----
2 files changed, 9 insertions(+), 11 deletions(-)
diff --git a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
index 95cc36af4b5230..9004f4afed7964 100644
--- a/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
+++ b/llvm/include/llvm/IR/IntrinsicsAMDGPU.td
@@ -3689,12 +3689,6 @@ class AMDGPUMfmaIntrinsic<LLVMType DestTy, LLVMType SrcABTy> :
// requires more registers than the corresponding vector type (e.g. it
// is illegal to use <6 x i32> in operand 0 if cbsz specifies an f8
// format that requires 8 registers).
-//
-// The scales are E8M0 (2^(s-127)) values, taken from the byte of each scale
-// operand selected by its op_sel. There is no separate intrinsic for the
-// unscaled instruction: if both scale operands are the constant 0, the
-// unscaled instruction is used and no scaling is applied, instead of the
-// 2^-127 that a non-constant 0 scale gives.
class AMDGPUMfmaScaleIntrinsic<LLVMType DestTy> :
DefaultAttrsIntrinsic<[DestTy],
[llvm_anyvector_ty, llvm_anyvector_ty, DestTy,
diff --git a/llvm/lib/Target/AMDGPU/SIInstrInfo.td b/llvm/lib/Target/AMDGPU/SIInstrInfo.td
index 7e11bc5c0bc539..eb6cff7d0fc597 100644
--- a/llvm/lib/Target/AMDGPU/SIInstrInfo.td
+++ b/llvm/lib/Target/AMDGPU/SIInstrInfo.td
@@ -317,16 +317,20 @@ def SIdenorm_mode : SDNode<"AMDGPUISD::DENORM_MODE",
>;
-// Optimize v_mfma_scale* instructions to avoid the scale. Constant 0 scales
-// are the documented way to request the unscaled instruction (see the
-// intrinsic definition). Scales of 127 with op_sel 0 are E8M0 1.0.
+// Select the unscaled v_mfma instruction instead of v_mfma_scale:
+// - Both scales are 127 with op_sel 0. This is E8M0 1.0, so the result is
+// unchanged.
+// - Both scales are the constant 0. E8M0 0 is 2^-127, so this is NOT unscaled.
+// It is kept because there is no unscaled intrinsic, and frontends
+// pass constant 0 scales expecting an unscaled instruction.
+// TODO: Remove this folder or add add a supported unscaled intrinisic.
class UnscaledMFMAOptimizationPat<SDPatternOperator intrin> : PatFrags<
(ops node:$srca, node:$srcb, node:$srcc,
node:$cbsz, node:$blgp),
[(intrin $srca, $srcb, $srcc, $cbsz, $blgp,
- srcvalue, 0, srcvalue, 0),
+ 0, 127, 0, 127),
(intrin $srca, $srcb, $srcc, $cbsz, $blgp,
- 0, 127, 0, 127)]
+ srcvalue, 0, srcvalue, 0)]
>;
def mfma_f32_16x16x128_f8f6f4 : UnscaledMFMAOptimizationPat<int_amdgcn_mfma_scale_f32_16x16x128_f8f6f4>;
>From 9a9930ec721e601537ee7432e85802f0df3a16e9 Mon Sep 17 00:00:00 2001
From: Keshav Vinayak Jha <31160700+keshavvinayak01 at users.noreply.github.com>
Date: Fri, 16 Oct 2026 14:50:24 +0530
Subject: [PATCH 4/4] Fix typo in TODO comment for UnscaledMFMAOptimizationPat
---
llvm/lib/Target/AMDGPU/SIInstrInfo.td | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/llvm/lib/Target/AMDGPU/SIInstrInfo.td b/llvm/lib/Target/AMDGPU/SIInstrInfo.td
index eb6cff7d0fc597..bc0b5f30d9b93e 100644
--- a/llvm/lib/Target/AMDGPU/SIInstrInfo.td
+++ b/llvm/lib/Target/AMDGPU/SIInstrInfo.td
@@ -323,7 +323,7 @@ def SIdenorm_mode : SDNode<"AMDGPUISD::DENORM_MODE",
// - Both scales are the constant 0. E8M0 0 is 2^-127, so this is NOT unscaled.
// It is kept because there is no unscaled intrinsic, and frontends
// pass constant 0 scales expecting an unscaled instruction.
-// TODO: Remove this folder or add add a supported unscaled intrinisic.
+// TODO: Remove this folder or add a supported unscaled intrinisic.
class UnscaledMFMAOptimizationPat<SDPatternOperator intrin> : PatFrags<
(ops node:$srca, node:$srcb, node:$srcc,
node:$cbsz, node:$blgp),
More information about the Mlir-commits
mailing list