[Mlir-commits] [mlir] [MLIR][XeVM] Use fp conversion spirv builtins (PR #232463)
Sang Ik Lee
llvmlistbot at llvm.org
Thu Oct 15 12:02:06 PDT 2026
https://github.com/silee2 created https://github.com/llvm/llvm-project/pull/232463
IGC's __builtin_IB API is not stable so move to __builtin_spirv if an alternative is available.
For fp4, this update results in reducing some scalarized instructions as __builtin_spirv API support vector operand directly.
>From 44d6e8d2dbbfe738a9a3789021c78173928e9dbb Mon Sep 17 00:00:00 2001
From: "Lee, Sang Ik" <sang.ik.lee at intel.com>
Date: Wed, 14 Oct 2026 20:57:27 +0000
Subject: [PATCH 1/3] [mlir][XeVM] Use IGC's vector fp4 conversion builtin
instead of scalarising
`TruncfToOCLPattern` lowered a truncation to `f4E2M1FN` by emitting
`__builtin_IB_dnscl_{hf16,bf16}` directly: split the source into i32 lanes, call
the builtin once per lane pair with complementary byte modes, OR the pairs
together, and reassemble the dwords with `insertelement`.
IGC's BiFModule already exposes that packing as an overloaded SPIR-V friendly
builtin, `__builtin_spirv_Convert{BF16,FP16}ToE2M1INTEL`, declared for every
SPIR-V vector length - and its implementation *is* the sequence above, byte modes
and ORs included. Call the vector form and let IGC do the breakdown.
The source is passed as a whole vector, so nothing has to be split: a bf16 source
is bitcast to i16, which is what the builtin spells `short`, and an f16 source is
passed as-is. A destination narrower than one byte comes back as the one element
vector `uchar1`, which is bitcast down to the op's scalar type.
This removes the hand-written lane splitting, the mode bookkeeping and the dword
reassembly - 136 lines of the pattern become 40 - and with them the
`extractelement` per call. On a downstream fp4 GEMM that quantizes A in-kernel,
the final LLVM-dialect IR goes from 2616 to 2356 lines, `llvm.extractelement`
from 141 to 13, `llvm.insertelement` from 251 to 219, and the 64 scalar
`dnscl` calls become 16 vector calls.
Runtime is unchanged: 804,231 clk before against 802,315 / 801,634 after, two
runs each on one workgroup at M = N = 256, K = 4096, with `mov` down only 32 of
117,080. IGC was evidently already coalescing the scalarised sequence, so this is
a simplification and an IR-size win rather than a performance fix.
Verified on the CRI simulator, since the fp4 execution tests are XFAIL without a
device: the bf16 and f16 length-16 round trips reproduce their inputs exactly,
the lengths-2/3/4/8 round trip reproduces its expected
`[0, 0.5, -1, ...]` pattern, and the GEMM above returns `mismatches: 0`.
Compile-only coverage for every length is unchanged, and the full `mlir/test`
run matches the `origin/main` baseline.
Note the existing comment above these patterns says they target IGC builtins
because no official OpenCL API exists yet. That still holds - these
`__builtin_spirv_*` entry points are the experimental SPIR-V extension
implementation, not a released OpenCL API - so the TODO to move to an official
API when one appears is left in place.
---
mlir/lib/Conversion/XeVMToLLVM/XeVMToLLVM.cpp | 136 +++++-------------
.../xevm_fp_conversion_lengths.mlir | 64 +++------
.../XeVMToLLVM/xevm_mx-to-llvm.mlir | 71 +--------
3 files changed, 58 insertions(+), 213 deletions(-)
diff --git a/mlir/lib/Conversion/XeVMToLLVM/XeVMToLLVM.cpp b/mlir/lib/Conversion/XeVMToLLVM/XeVMToLLVM.cpp
index f3bf1a5fc7866c..f872fa1874fb66 100644
--- a/mlir/lib/Conversion/XeVMToLLVM/XeVMToLLVM.cpp
+++ b/mlir/lib/Conversion/XeVMToLLVM/XeVMToLLVM.cpp
@@ -1239,113 +1239,41 @@ class TruncfToOCLPattern : public OpConversionPattern<TruncfOp> {
// Handle the case where dst type is fp4 first.
if (dstEtype == TruncfDstElemTypes::E2M1) {
- // `__builtin_IB_dnscl_{hf16,bf16}(uint a, uint b, convert_to, mode)`
- // takes two dwords, each holding two source elements, and packs each pair
- // into one byte of the result dword. `mode` picks which bytes of that
- // dword are written: mode 0 writes bytes 0 and 2, mode 2 writes bytes 1
- // and 3. Two calls with complementary modes therefore OR together into
- // one fully packed dword covering eight source elements.
+ // IGC's BiFModule exposes the fp4 packing as an overloaded SPIR-V
+ // friendly builtin, `__builtin_spirv_Convert{BF16,FP16}ToE2M1INTEL`,
+ // declared for every SPIR-V vector length. Its implementation *is* the
+ // `__builtin_IB_dnscl_*` sequence this used to emit by hand - pairs of
+ // calls with complementary byte modes, ORed together - so calling the
+ // vector form hands that breakdown to IGC. Doing it here required
+ // splitting the source into dword lanes, which cost two
+ // `llvm.extractelement` per `dnscl` call and left the backend a
+ // scalarised sequence to re-vectorise.
//
- // A pair of elements is the conversion granularity, so an odd length is
- // padded up and the spare nibble left undefined.
- constexpr int kDnsclConvertToE2M1 = 1;
- constexpr int kDnsclModeBytes02 = 0;
- constexpr int kDnsclModeBytes13 = 2;
- // One dword lane per element pair, and one result byte per lane. The op
- // verifier has already checked that the destination is exactly that wide.
- int64_t numLanes = llvm::divideCeil(numElements, 2);
-
- Type i32Ty = rewriter.getI32Type();
- Type i8Ty = rewriter.getI8Type();
- // Pad an odd length up to a whole number of pairs, then view the source
- // as dword lanes.
- Value padded = src;
- if (numElements != numLanes * 2) {
- SmallVector<int32_t> mask = llvm::to_vector(
- llvm::seq<int32_t>(0, static_cast<int32_t>(numElements)));
- // The padding element is never read back, so any valid index will do.
- mask.append(static_cast<size_t>(numLanes * 2 - numElements), 0);
- padded = LLVM::ShuffleVectorOp::create(rewriter, loc, src, src, mask);
- }
- // A single lane is passed as a bare i32 rather than a one element vector,
- // which SPIR-V has no type for.
- Value laneVec;
- if (numLanes > 1)
- laneVec = LLVM::BitcastOp::create(
- rewriter, loc, VectorType::get(numLanes, i32Ty), padded);
- else
- laneVec = LLVM::BitcastOp::create(rewriter, loc, i32Ty, padded);
- auto getLane = [&](int64_t idx) -> Value {
- if (numLanes == 1)
- return laneVec;
- Value pos =
- LLVM::ConstantOp::create(rewriter, loc, rewriter.getI32Type(), idx);
- return LLVM::ExtractElementOp::create(rewriter, loc, laneVec, pos)
- ->getResult(0);
- };
-
- std::string fnName = "__builtin_IB_dnscl_";
- fnName += (srcEtype == TruncfSrcElemTypes::F16) ? "hf16" : "bf16";
- Value convertTo =
- LLVM::ConstantOp::create(rewriter, loc, i32Ty, kDnsclConvertToE2M1);
- auto genDnscl = [&](Value lo, Value hi, int mode) -> Value {
- Value modeVal = LLVM::ConstantOp::create(rewriter, loc, i32Ty, mode);
- SmallVector<Type> argTypes{lo.getType(), hi.getType(),
- convertTo.getType(), modeVal.getType()};
- SmallVector<Value> args{lo, hi, convertTo, modeVal};
- return createDeviceFunctionCall(rewriter, fnName, i32Ty, argTypes, args,
- {}, funcAttrs, op.getOperation())
- ->getResult(0);
- };
-
- Value result;
- if (numLanes <= 2) {
- // Fewer than four lanes cannot fill a dword, so a single call is made
- // and the written bytes, 0 and 2, are compacted afterwards.
- Value lo = getLane(0);
- Value hi =
- numLanes == 2
- ? getLane(1)
- : LLVM::UndefOp::create(rewriter, loc, i32Ty)->getResult(0);
- Value dword = genDnscl(lo, hi, kDnsclModeBytes02);
- if (numLanes == 1) {
- // A single byte, so the low one, is all that is kept.
- result = LLVM::TruncOp::create(rewriter, loc, i8Ty, dword);
- } else {
- Value bytes = LLVM::BitcastOp::create(
- rewriter, loc, VectorType::get(4, i8Ty), dword);
- result = LLVM::ShuffleVectorOp::create(rewriter, loc, bytes, bytes,
- ArrayRef<int32_t>{0, 2});
- }
+ // See IGC/BiFModule/Implementation/SPV_INTEL_fp_conversions/
+ // downconversions_fp4.cl for the declarations and their semantics.
+ Value arg = src;
+ StringRef baseName;
+ if (srcEtype == TruncfSrcElemTypes::F16) {
+ baseName = "__builtin_spirv_ConvertFP16ToE2M1INTEL";
} else {
- // Four lanes, eight source elements, per fully packed dword.
- SmallVector<Value> dwords;
- for (int64_t base = 0; base < numLanes; base += 4) {
- // Each lane is bound to a name first: `getLane` builds ops, and the
- // order of evaluation within an argument list is unspecified, which
- // would otherwise leave the order of the emitted ops up to the host
- // compiler.
- Value lane0 = getLane(base);
- Value lane2 = getLane(base + 2);
- Value even = genDnscl(lane0, lane2, kDnsclModeBytes02);
- Value lane1 = getLane(base + 1);
- Value lane3 = getLane(base + 3);
- Value odd = genDnscl(lane1, lane3, kDnsclModeBytes13);
- dwords.push_back(LLVM::OrOp::create(rewriter, loc, even, odd));
- }
- if (dwords.size() == 1) {
- result = dwords.front();
- } else {
- Type packedTy = VectorType::get(dwords.size(), i32Ty);
- result = LLVM::UndefOp::create(rewriter, loc, packedTy);
- for (auto [idx, dword] : llvm::enumerate(dwords)) {
- Value pos = LLVM::ConstantOp::create(rewriter, loc, i32Ty, idx);
- result =
- LLVM::InsertElementOp::create(rewriter, loc, result, dword, pos)
- ->getResult(0);
- }
- }
+ baseName = "__builtin_spirv_ConvertBF16ToE2M1INTEL";
+ // bf16 is passed as its bit pattern, which the builtin spells `short`.
+ arg = LLVM::BitcastOp::create(
+ rewriter, loc,
+ VectorType::get(vecSrcTy.getShape(), rewriter.getI16Type()), src);
}
+ // A pair of elements is the conversion granularity and packs into one
+ // byte, so an odd length rounds up and leaves the spare nibble undefined.
+ // The builtin returns that as a vector even when it is a single byte
+ // (`uchar1`), which is what the op's destination type already is.
+ int64_t numBytes = llvm::divideCeil(numElements, 2);
+ Type retTy = VectorType::get(numBytes, rewriter.getI8Type());
+ SmallVector<Type> argTypes{arg.getType()};
+ SmallVector<Value> args{arg};
+ Value result = createDeviceFunctionCall(
+ rewriter, mangle(baseName, argTypes), retTy, argTypes,
+ args, {}, funcAttrs, op.getOperation())
+ ->getResult(0);
rewriter.replaceOp(op, castIfNeeded(rewriter, loc, dstTy, result));
return success();
}
diff --git a/mlir/test/Conversion/XeVMToLLVM/xevm_fp_conversion_lengths.mlir b/mlir/test/Conversion/XeVMToLLVM/xevm_fp_conversion_lengths.mlir
index fa2183f358798e..1de67e2a8e07b7 100644
--- a/mlir/test/Conversion/XeVMToLLVM/xevm_fp_conversion_lengths.mlir
+++ b/mlir/test/Conversion/XeVMToLLVM/xevm_fp_conversion_lengths.mlir
@@ -75,20 +75,16 @@ llvm.func @extf_bf8_to_bf16_v4(%src: vector<4xi8>) -> vector<4xbf16> {
// -----
-// Two fp4 values pack into a single byte. SPIR-V has no vector of length one,
-// so the packed side is a scalar, and only the low byte of the dnscl result is
-// kept.
-// CHECK: llvm.func spir_funccc @__builtin_IB_dnscl_hf16(i32, i32, i32, i32) -> i32
+// Two fp4 values pack into a single byte, which the builtin returns as the one
+// element vector it spells `uchar1`. The op's destination is the scalar SPIR-V
+// has no vector type for, so the result is bitcast down to it.
+// CHECK: llvm.func spir_funccc @_Z38__builtin_spirv_ConvertFP16ToE2M1INTELDv2_Dh(vector<2xf16>) -> vector<1xi8>
// CHECK-LABEL: llvm.func @truncf_f16_to_e2m1_v2
// CHECK-SAME: %[[ARG0:.*]]: vector<2xf16>
llvm.func @truncf_f16_to_e2m1_v2(%src: vector<2xf16>) -> i8 {
- // CHECK-DAG: %[[C0:.*]] = llvm.mlir.constant(0 : i32) : i32
- // CHECK-DAG: %[[UNDEF:.*]] = llvm.mlir.undef : i32
- // CHECK-DAG: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
- // CHECK: %[[BC:.*]] = llvm.bitcast %[[ARG0]] : vector<2xf16> to i32
- // CHECK: %[[CALL:.*]] = llvm.call spir_funccc @__builtin_IB_dnscl_hf16(%[[BC]], %[[UNDEF]], %[[C1]], %[[C0]])
- // CHECK-SAME: : (i32, i32, i32, i32) -> i32
- // CHECK: %[[RES:.*]] = llvm.trunc %[[CALL]] : i32 to i8
+ // CHECK: %[[CALL:.*]] = llvm.call spir_funccc @_Z38__builtin_spirv_ConvertFP16ToE2M1INTELDv2_Dh(%[[ARG0]])
+ // CHECK-SAME: : (vector<2xf16>) -> vector<1xi8>
+ // CHECK: llvm.bitcast %[[CALL]] : vector<1xi8> to i8
%dst = xevm.truncf %src { src_etype = f16, dst_etype = e2m1 } : (vector<2xf16>) -> i8
llvm.return %dst : i8
}
@@ -98,61 +94,39 @@ llvm.func @truncf_f16_to_e2m1_v2(%src: vector<2xf16>) -> i8 {
// An element pair is the conversion granularity, so 3 elements are padded up to
// 4 and the spare nibble is left undefined. The two bytes the call writes, 0
// and 2, are compacted into the result.
-// CHECK: llvm.func spir_funccc @__builtin_IB_dnscl_hf16(i32, i32, i32, i32) -> i32
+// CHECK: llvm.func spir_funccc @_Z38__builtin_spirv_ConvertFP16ToE2M1INTELDv3_Dh(vector<3xf16>) -> vector<2xi8>
// CHECK-LABEL: llvm.func @truncf_f16_to_e2m1_v3
// CHECK-SAME: %[[ARG0:.*]]: vector<3xf16>
llvm.func @truncf_f16_to_e2m1_v3(%src: vector<3xf16>) -> vector<2xi8> {
- // CHECK-DAG: %[[C0:.*]] = llvm.mlir.constant(0 : i32) : i32
- // CHECK-DAG: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
- // CHECK: %[[PAD:.*]] = llvm.shufflevector %[[ARG0]], %[[ARG0]] [0, 1, 2, 0] : vector<3xf16>
- // CHECK: %[[LANES:.*]] = llvm.bitcast %[[PAD]] : vector<4xf16> to vector<2xi32>
- // CHECK-DAG: %[[L1:.*]] = llvm.extractelement %[[LANES]][%[[C1]] : i32] : vector<2xi32>
- // CHECK-DAG: %[[L0:.*]] = llvm.extractelement %[[LANES]][%[[C0]] : i32] : vector<2xi32>
- // CHECK: %[[CALL:.*]] = llvm.call spir_funccc @__builtin_IB_dnscl_hf16(%[[L0]], %[[L1]], %[[C1]], %[[C0]])
- // CHECK-SAME: : (i32, i32, i32, i32) -> i32
- // CHECK: %[[BYTES:.*]] = llvm.bitcast %[[CALL]] : i32 to vector<4xi8>
- // CHECK: %[[RES:.*]] = llvm.shufflevector %[[BYTES]], %[[BYTES]] [0, 2] : vector<4xi8>
+ // CHECK: %[[CALL:.*]] = llvm.call spir_funccc @_Z38__builtin_spirv_ConvertFP16ToE2M1INTELDv3_Dh(%[[ARG0]])
+ // CHECK-SAME: : (vector<3xf16>) -> vector<2xi8>
%dst = xevm.truncf %src { src_etype = f16, dst_etype = e2m1 } : (vector<3xf16>) -> vector<2xi8>
llvm.return %dst : vector<2xi8>
}
// -----
-// CHECK: llvm.func spir_funccc @__builtin_IB_dnscl_hf16(i32, i32, i32, i32) -> i32
+// CHECK: llvm.func spir_funccc @_Z38__builtin_spirv_ConvertFP16ToE2M1INTELDv4_Dh(vector<4xf16>) -> vector<2xi8>
// CHECK-LABEL: llvm.func @truncf_f16_to_e2m1_v4
// CHECK-SAME: %[[ARG0:.*]]: vector<4xf16>
llvm.func @truncf_f16_to_e2m1_v4(%src: vector<4xf16>) -> vector<2xi8> {
- // CHECK: %[[LANES:.*]] = llvm.bitcast %[[ARG0]] : vector<4xf16> to vector<2xi32>
- // CHECK: llvm.call spir_funccc @__builtin_IB_dnscl_hf16
- // CHECK: %[[BYTES:.*]] = llvm.bitcast {{.*}} : i32 to vector<4xi8>
- // CHECK: %[[RES:.*]] = llvm.shufflevector %[[BYTES]], %[[BYTES]] [0, 2] : vector<4xi8>
+ // CHECK: %[[CALL:.*]] = llvm.call spir_funccc @_Z38__builtin_spirv_ConvertFP16ToE2M1INTELDv4_Dh(%[[ARG0]])
+ // CHECK-SAME: : (vector<4xf16>) -> vector<2xi8>
%dst = xevm.truncf %src { src_etype = f16, dst_etype = e2m1 } : (vector<4xf16>) -> vector<2xi8>
llvm.return %dst : vector<2xi8>
}
// -----
-// Eight elements fill a dword: two calls with complementary nibble modes, 0 for
-// bytes 0 and 2 and 2 for bytes 1 and 3, are OR-ed together.
-// CHECK: llvm.func spir_funccc @__builtin_IB_dnscl_bf16(i32, i32, i32, i32) -> i32
+// Eight elements pack into four bytes. bf16 is passed as its bit pattern, which
+// the builtin spells `short`, so the source is bitcast to i16 first.
+// CHECK: llvm.func spir_funccc @_Z38__builtin_spirv_ConvertBF16ToE2M1INTELDv8_s(vector<8xi16>) -> vector<4xi8>
// CHECK-LABEL: llvm.func @truncf_bf16_to_e2m1_v8
// CHECK-SAME: %[[ARG0:.*]]: vector<8xbf16>
llvm.func @truncf_bf16_to_e2m1_v8(%src: vector<8xbf16>) -> vector<4xi8> {
- // CHECK-DAG: %[[C0:.*]] = llvm.mlir.constant(0 : i32) : i32
- // CHECK-DAG: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
- // CHECK-DAG: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
- // CHECK-DAG: %[[C3:.*]] = llvm.mlir.constant(3 : i32) : i32
- // CHECK: %[[LANES:.*]] = llvm.bitcast %[[ARG0]] : vector<8xbf16> to vector<4xi32>
- // CHECK: %[[L0:.*]] = llvm.extractelement %[[LANES]][%[[C0]] : i32] : vector<4xi32>
- // CHECK: %[[L2:.*]] = llvm.extractelement %[[LANES]][%[[C2]] : i32] : vector<4xi32>
- // CHECK: %[[EVEN:.*]] = llvm.call spir_funccc @__builtin_IB_dnscl_bf16(%[[L0]], %[[L2]], %[[C1]], %[[C0]])
- // CHECK-SAME: : (i32, i32, i32, i32) -> i32
- // CHECK: %[[L1:.*]] = llvm.extractelement %[[LANES]][%[[C1]] : i32] : vector<4xi32>
- // CHECK: %[[L3:.*]] = llvm.extractelement %[[LANES]][%[[C3]] : i32] : vector<4xi32>
- // CHECK: %[[ODD:.*]] = llvm.call spir_funccc @__builtin_IB_dnscl_bf16(%[[L1]], %[[L3]], %[[C1]], %[[C2]])
- // CHECK-SAME: : (i32, i32, i32, i32) -> i32
- // CHECK: %[[OR:.*]] = llvm.or %[[EVEN]], %[[ODD]] : i32
- // CHECK: %[[RES:.*]] = llvm.bitcast %[[OR]] : i32 to vector<4xi8>
+ // CHECK: %[[SRC:.*]] = llvm.bitcast %[[ARG0]] : vector<8xbf16> to vector<8xi16>
+ // CHECK: %[[CALL:.*]] = llvm.call spir_funccc @_Z38__builtin_spirv_ConvertBF16ToE2M1INTELDv8_s(%[[SRC]])
+ // CHECK-SAME: : (vector<8xi16>) -> vector<4xi8>
%dst = xevm.truncf %src { src_etype = bf16, dst_etype = e2m1 } : (vector<8xbf16>) -> vector<4xi8>
llvm.return %dst : vector<4xi8>
}
diff --git a/mlir/test/Conversion/XeVMToLLVM/xevm_mx-to-llvm.mlir b/mlir/test/Conversion/XeVMToLLVM/xevm_mx-to-llvm.mlir
index bc920092929592..54da0238e4f90d 100644
--- a/mlir/test/Conversion/XeVMToLLVM/xevm_mx-to-llvm.mlir
+++ b/mlir/test/Conversion/XeVMToLLVM/xevm_mx-to-llvm.mlir
@@ -80,82 +80,25 @@ llvm.func @truncf_bf16_to_hf8(%src: vector<16xbf16>) -> vector<16xi8> {
// -----
-// CHECK: llvm.func spir_funccc @__builtin_IB_dnscl_hf16(i32, i32, i32, i32) -> i32
+// CHECK: llvm.func spir_funccc @_Z38__builtin_spirv_ConvertFP16ToE2M1INTELDv16_Dh(vector<16xf16>) -> vector<8xi8>
// CHECK-LABEL: llvm.func @truncf_f16_to_e2m1
// CHECK-SAME: %[[ARG0:.*]]: vector<16xf16>
llvm.func @truncf_f16_to_e2m1(%src: vector<16xf16>) -> vector<8xi8> {
- // CHECK-DAG: %[[UNDEF:.*]] = llvm.mlir.undef : vector<2xi32>
- // CHECK-DAG: %[[C7:.*]] = llvm.mlir.constant(7 : i32) : i32
- // CHECK-DAG: %[[C6:.*]] = llvm.mlir.constant(6 : i32) : i32
- // CHECK-DAG: %[[C5:.*]] = llvm.mlir.constant(5 : i32) : i32
- // CHECK-DAG: %[[C4:.*]] = llvm.mlir.constant(4 : i32) : i32
- // CHECK-DAG: %[[C3:.*]] = llvm.mlir.constant(3 : i32) : i32
- // CHECK-DAG: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
- // CHECK-DAG: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
- // CHECK-DAG: %[[C0:.*]] = llvm.mlir.constant(0 : i32) : i32
- // CHECK: %[[BC:.*]] = llvm.bitcast %[[ARG0]] : vector<16xf16> to vector<8xi32>
- // CHECK: %[[E0:.*]] = llvm.extractelement %[[BC]][%[[C0]] : i32] : vector<8xi32>
- // CHECK: %[[E2:.*]] = llvm.extractelement %[[BC]][%[[C2]] : i32] : vector<8xi32>
- // CHECK: %[[CALL0:.*]] = llvm.call spir_funccc @__builtin_IB_dnscl_hf16(%[[E0]], %[[E2]], %[[C1]], %[[C0]])
- // CHECK-SAME: : (i32, i32, i32, i32) -> i32
- // CHECK: %[[E1:.*]] = llvm.extractelement %[[BC]][%[[C1]] : i32] : vector<8xi32>
- // CHECK: %[[E3:.*]] = llvm.extractelement %[[BC]][%[[C3]] : i32] : vector<8xi32>
- // CHECK: %[[CALL1:.*]] = llvm.call spir_funccc @__builtin_IB_dnscl_hf16(%[[E1]], %[[E3]], %[[C1]], %[[C2]])
- // CHECK-SAME: : (i32, i32, i32, i32) -> i32
- // CHECK: %[[OR0:.*]] = llvm.or %[[CALL0]], %[[CALL1]] : i32
- // CHECK: %[[E4:.*]] = llvm.extractelement %[[BC]][%[[C4]] : i32] : vector<8xi32>
- // CHECK: %[[E6:.*]] = llvm.extractelement %[[BC]][%[[C6]] : i32] : vector<8xi32>
- // CHECK: %[[CALL2:.*]] = llvm.call spir_funccc @__builtin_IB_dnscl_hf16(%[[E4]], %[[E6]], %[[C1]], %[[C0]])
- // CHECK-SAME: : (i32, i32, i32, i32) -> i32
- // CHECK: %[[E5:.*]] = llvm.extractelement %[[BC]][%[[C5]] : i32] : vector<8xi32>
- // CHECK: %[[E7:.*]] = llvm.extractelement %[[BC]][%[[C7]] : i32] : vector<8xi32>
- // CHECK: %[[CALL3:.*]] = llvm.call spir_funccc @__builtin_IB_dnscl_hf16(%[[E5]], %[[E7]], %[[C1]], %[[C2]])
- // CHECK-SAME: : (i32, i32, i32, i32) -> i32
- // CHECK: %[[OR1:.*]] = llvm.or %[[CALL2]], %[[CALL3]] : i32
- // CHECK: %[[INS0:.*]] = llvm.insertelement %[[OR0]], %[[UNDEF]][%[[C0]] : i32] : vector<2xi32>
- // CHECK: %[[INS1:.*]] = llvm.insertelement %[[OR1]], %[[INS0]][%[[C1]] : i32] : vector<2xi32>
- // CHECK: %[[RES:.*]] = llvm.bitcast %[[INS1]] : vector<2xi32> to vector<8xi8>
+ // CHECK: llvm.call spir_funccc @_Z38__builtin_spirv_ConvertFP16ToE2M1INTELDv16_Dh(%[[ARG0]])
+ // CHECK-SAME: : (vector<16xf16>) -> vector<8xi8>
%dst = xevm.truncf %src { src_etype = f16, dst_etype = e2m1 } : (vector<16xf16>) -> vector<8xi8>
llvm.return %dst : vector<8xi8>
}
// -----
-// CHECK: llvm.func spir_funccc @__builtin_IB_dnscl_bf16(i32, i32, i32, i32) -> i32
+// CHECK: llvm.func spir_funccc @_Z38__builtin_spirv_ConvertBF16ToE2M1INTELDv16_s(vector<16xi16>) -> vector<8xi8>
// CHECK-LABEL: llvm.func @truncf_bf16_to_e2m1
// CHECK-SAME: %[[ARG0:.*]]: vector<16xbf16>
llvm.func @truncf_bf16_to_e2m1(%src: vector<16xbf16>) -> vector<8xi8> {
- // CHECK-DAG: %[[UNDEF:.*]] = llvm.mlir.undef : vector<2xi32>
- // CHECK-DAG: %[[C7:.*]] = llvm.mlir.constant(7 : i32) : i32
- // CHECK-DAG: %[[C6:.*]] = llvm.mlir.constant(6 : i32) : i32
- // CHECK-DAG: %[[C5:.*]] = llvm.mlir.constant(5 : i32) : i32
- // CHECK-DAG: %[[C4:.*]] = llvm.mlir.constant(4 : i32) : i32
- // CHECK-DAG: %[[C3:.*]] = llvm.mlir.constant(3 : i32) : i32
- // CHECK-DAG: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
- // CHECK-DAG: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
- // CHECK-DAG: %[[C0:.*]] = llvm.mlir.constant(0 : i32) : i32
- // CHECK: %[[BC:.*]] = llvm.bitcast %[[ARG0]] : vector<16xbf16> to vector<8xi32>
- // CHECK: %[[E0:.*]] = llvm.extractelement %[[BC]][%[[C0]] : i32] : vector<8xi32>
- // CHECK: %[[E2:.*]] = llvm.extractelement %[[BC]][%[[C2]] : i32] : vector<8xi32>
- // CHECK: %[[CALL0:.*]] = llvm.call spir_funccc @__builtin_IB_dnscl_bf16(%[[E0]], %[[E2]], %[[C1]], %[[C0]])
- // CHECK-SAME: : (i32, i32, i32, i32) -> i32
- // CHECK: %[[E1:.*]] = llvm.extractelement %[[BC]][%[[C1]] : i32] : vector<8xi32>
- // CHECK: %[[E3:.*]] = llvm.extractelement %[[BC]][%[[C3]] : i32] : vector<8xi32>
- // CHECK: %[[CALL1:.*]] = llvm.call spir_funccc @__builtin_IB_dnscl_bf16(%[[E1]], %[[E3]], %[[C1]], %[[C2]])
- // CHECK-SAME: : (i32, i32, i32, i32) -> i32
- // CHECK: %[[OR0:.*]] = llvm.or %[[CALL0]], %[[CALL1]] : i32
- // CHECK: %[[E4:.*]] = llvm.extractelement %[[BC]][%[[C4]] : i32] : vector<8xi32>
- // CHECK: %[[E6:.*]] = llvm.extractelement %[[BC]][%[[C6]] : i32] : vector<8xi32>
- // CHECK: %[[CALL2:.*]] = llvm.call spir_funccc @__builtin_IB_dnscl_bf16(%[[E4]], %[[E6]], %[[C1]], %[[C0]])
- // CHECK-SAME: : (i32, i32, i32, i32) -> i32
- // CHECK: %[[E5:.*]] = llvm.extractelement %[[BC]][%[[C5]] : i32] : vector<8xi32>
- // CHECK: %[[E7:.*]] = llvm.extractelement %[[BC]][%[[C7]] : i32] : vector<8xi32>
- // CHECK: %[[CALL3:.*]] = llvm.call spir_funccc @__builtin_IB_dnscl_bf16(%[[E5]], %[[E7]], %[[C1]], %[[C2]])
- // CHECK-SAME: : (i32, i32, i32, i32) -> i32
- // CHECK: %[[OR1:.*]] = llvm.or %[[CALL2]], %[[CALL3]] : i32
- // CHECK: %[[INS0:.*]] = llvm.insertelement %[[OR0]], %[[UNDEF]][%[[C0]] : i32] : vector<2xi32>
- // CHECK: %[[INS1:.*]] = llvm.insertelement %[[OR1]], %[[INS0]][%[[C1]] : i32] : vector<2xi32>
- // CHECK: %[[RES:.*]] = llvm.bitcast %[[INS1]] : vector<2xi32> to vector<8xi8>
+ // CHECK: %[[SRC:.*]] = llvm.bitcast %[[ARG0]] : vector<16xbf16> to vector<16xi16>
+ // CHECK: llvm.call spir_funccc @_Z38__builtin_spirv_ConvertBF16ToE2M1INTELDv16_s(%[[SRC]])
+ // CHECK-SAME: : (vector<16xi16>) -> vector<8xi8>
%dst = xevm.truncf %src { src_etype = bf16, dst_etype = e2m1 } : (vector<16xbf16>) -> vector<8xi8>
llvm.return %dst : vector<8xi8>
}
>From a9ee583e7581ce7638174ff526a5626aeab9603b Mon Sep 17 00:00:00 2001
From: "Lee, Sang Ik" <sang.ik.lee at intel.com>
Date: Thu, 15 Oct 2026 18:16:26 +0000
Subject: [PATCH 2/3] [mlir][XeVM] Lower the fp8 conversions through the SPIR-V
extension wrappers
TruncfToOCLPattern and ExtfToOCLPattern named IGC's internal fp8 builtins directly, pasting the vector length onto the symbol. Two of those names do not exist: E4M3 is spelled e4m3 in the conversion builtins, not hf8, so f16 -> f8 asked for __builtin_IB_hftohf8_N where IGC defines __builtin_IB_hftoe4m3_N, and f8 -> f16 asked for __builtin_IB_hf8tohf_N where it defines __builtin_IB_e4m3tohf_N. Only E5M2 uses the bf8 spelling. The asymmetry is easy to miss because the scaled DPAS builtin does spell E4M3 as hf8 (bdpas_f_f_hf8_hf8_8_8), which is correct and unchanged here.
Since createDeviceFunctionCall emits these names unmangled, the nonexistent symbol linked silently and the call returned undef: A reached the DPAS as 0xCC fill bytes, which decode in E4M3 as -1.5 x 2^2 = -6, so a K=32 dot product of ones produced -192 instead of 32.
Call the SPV_INTEL_fp_conversions wrappers instead, as the fp4 lowering now does:
charN __builtin_spirv_ConvertFP16ToE5M2EXT(halfN)
charN __builtin_spirv_ConvertFP16ToE4M3EXT(halfN)
charN __builtin_spirv_ConvertBF16ToE5M2EXT(shortN)
charN __builtin_spirv_ConvertBF16ToE4M3EXT(shortN)
halfN __builtin_spirv_ConvertE5M2ToFP16EXT(charN)
halfN __builtin_spirv_ConvertE4M3ToFP16EXT(charN)
shortN __builtin_spirv_ConvertE5M2ToBF16EXT(charN)
shortN __builtin_spirv_ConvertE4M3ToBF16EXT(charN)
They are overloaded, so the existing mangle() helper derives the length from the argument type and a name or signature mismatch fails to link instead of binding silently to nothing. As the note above these patterns says, they are the experimental SPIR-V extension implementation rather than an official OpenCL API, so the TODO to move once one exists still stands.
The bf16 forms take and return the bits as shortN, matching the bitcast the lowering already did, and route through f32 and f16 internally. That retires the explicit bftof / convert_half and convert_float / ftobf round trips, so each bf16 conversion is one call rather than three and lenSuffix is gone from both patterns.
Verified on the CRI simulator. The two block-scaled DPAS f8 tests that returned -192 now return the expected 32, the bf8, bf16 and e2m1 variants and the plain DPAS test still pass, and mlir/test/Conversion/{XeVMToLLVM,XeGPUToXeVM}, mlir/test/Dialect/{XeVM,XeGPU} and mlir/test/Target/LLVM are clean.
---
mlir/lib/Conversion/XeVMToLLVM/XeVMToLLVM.cpp | 158 +++++++-----------
.../xevm_fp_conversion_lengths.mlir | 43 ++---
.../XeVMToLLVM/xevm_mx-to-llvm.mlir | 76 +++------
3 files changed, 107 insertions(+), 170 deletions(-)
diff --git a/mlir/lib/Conversion/XeVMToLLVM/XeVMToLLVM.cpp b/mlir/lib/Conversion/XeVMToLLVM/XeVMToLLVM.cpp
index f872fa1874fb66..68e7f59335c1be 100644
--- a/mlir/lib/Conversion/XeVMToLLVM/XeVMToLLVM.cpp
+++ b/mlir/lib/Conversion/XeVMToLLVM/XeVMToLLVM.cpp
@@ -1279,62 +1279,52 @@ class TruncfToOCLPattern : public OpConversionPattern<TruncfOp> {
}
// Handle the case where dst type is fp8.
- // The fp8 conversions come as one builtin per vector length, so the length
- // is simply appended to the builtin name.
- std::string lenSuffix = std::to_string(numElements);
- // BF16 type needs some preprocessing before conversion,
- // First extended to F32 and then truncated to F16.
- if (srcEtype == TruncfSrcElemTypes::BF16) {
- // Step 1: Extend to F32
- // Use floatN __builtin_IB_bftof_N(shortN)
+ //
+ // The conversions come from the SPV_INTEL_fp_conversions half of the IGC
+ // BiFModule, which overloads one wrapper per vector length:
+ // charN __builtin_spirv_ConvertFP16ToE5M2EXT(halfN)
+ // charN __builtin_spirv_ConvertFP16ToE4M3EXT(halfN)
+ // charN __builtin_spirv_ConvertBF16ToE5M2EXT(shortN)
+ // charN __builtin_spirv_ConvertBF16ToE4M3EXT(shortN)
+ //
+ // These wrap the __builtin_IB_* builtins that used to be called here. As
+ // the note above says, they are the experimental SPIR-V extension
+ // implementation rather than an official OpenCL API, so the TODO to move
+ // once one exists still stands. They are still preferable to naming the
+ // IGC-internal builtins directly: being declared `overloadable` they are
+ // mangled, so the vector length comes from the argument type instead of a
+ // suffix pasted onto the name, and a name or signature mismatch fails to
+ // link rather than binding silently to nothing and yielding undef. Note
+ // too that IGC spells E4M3 as `e4m3` in the conversion builtins rather
+ // than `hf8`, even though the scaled DPAS builtin does use `hf8`; going
+ // through the wrappers avoids having to track that asymmetry here.
+ //
+ // A bf16 source takes the bf16 form, which accepts the bits as shortN and
+ // routes through f32 and f16 internally, so no preprocessing is needed
+ // beyond the bitcast to shortN.
+ bool srcIsBF16 = srcEtype == TruncfSrcElemTypes::BF16;
+ if (srcIsBF16)
src = LLVM::BitcastOp::create(
rewriter, op.getLoc(),
VectorType::get(vecSrcTy.getShape(), rewriter.getI16Type()), src);
- std::string fnName = "__builtin_IB_bftof_" + lenSuffix;
- SmallVector<Type> argTypes{src.getType()};
- SmallVector<Value> args{src};
- Type resTy = VectorType::get(vecSrcTy.getShape(), rewriter.getF32Type());
- src = createDeviceFunctionCall(rewriter, fnName, resTy, argTypes, args,
- {}, funcAttrs, op.getOperation())
- ->getResult(0);
- // Step 2: Truncf to F16
- // Use halfN convert_halfN(floatN)
- std::string truncFnName = "convert_half" + lenSuffix;
- SmallVector<Type> truncArgTypes{src.getType()};
- SmallVector<Value> truncArgs{src};
- truncFnName = mangle(truncFnName, truncArgTypes);
- resTy = VectorType::get(vecSrcTy.getShape(), rewriter.getF16Type());
- src =
- createDeviceFunctionCall(rewriter, truncFnName, resTy, truncArgTypes,
- truncArgs, {}, funcAttrs, op.getOperation())
- ->getResult(0);
- }
+ StringRef convertName;
if (dstEtype == TruncfDstElemTypes::BF8) { // Float8E5M2Type
- // Use charN __builtin_IB_hftobf8_N(halfN)
- std::string fnName = "__builtin_IB_hftobf8_" + lenSuffix;
- SmallVector<Type> argTypes{src.getType()};
- SmallVector<Value> args{src};
- Value result =
- createDeviceFunctionCall(rewriter, fnName, dstTy, argTypes, args, {},
- funcAttrs, op.getOperation())
- ->getResult(0);
-
- rewriter.replaceOp(op, result);
+ convertName = srcIsBF16 ? "__builtin_spirv_ConvertBF16ToE5M2EXT"
+ : "__builtin_spirv_ConvertFP16ToE5M2EXT";
} else if (dstEtype == TruncfDstElemTypes::F8) { // Float8E4M3FNType
- // Use charN __builtin_IB_hftohf8_N(halfN)
- std::string fnName = "__builtin_IB_hftohf8_" + lenSuffix;
- SmallVector<Type> argTypes{src.getType()};
- SmallVector<Value> args{src};
- Value result =
- createDeviceFunctionCall(rewriter, fnName, dstTy, argTypes, args, {},
- funcAttrs, op.getOperation())
- ->getResult(0);
-
- rewriter.replaceOp(op, result);
+ convertName = srcIsBF16 ? "__builtin_spirv_ConvertBF16ToE4M3EXT"
+ : "__builtin_spirv_ConvertFP16ToE4M3EXT";
} else {
return rewriter.notifyMatchFailure(
op, "Unsupported src, dst element type pair.");
}
+ SmallVector<Type> argTypes{src.getType()};
+ SmallVector<Value> args{src};
+ Value result = createDeviceFunctionCall(
+ rewriter, mangle(convertName, argTypes), dstTy, argTypes,
+ args, {}, funcAttrs, op.getOperation())
+ ->getResult(0);
+ rewriter.replaceOp(op, result);
return success();
}
};
@@ -1435,54 +1425,36 @@ class ExtfToOCLPattern : public OpConversionPattern<ExtfOp> {
if (!vecSrcTy || vecSrcTy.getNumElements() != numElements)
return rewriter.notifyMatchFailure(
op, "fp8 src and dst must have the same number of elements");
- std::string lenSuffix = std::to_string(numElements);
-
- // Step 1: Extend fp8 (bf8/hf8) to F16.
- // bf8 -> half: halfN __builtin_IB_bf8tohf_N(charN)
- // hf8 -> half: halfN __builtin_IB_hf8tohf_N(charN)
- std::string fnName = (srcEtype == ExtfSrcElemTypes::BF8)
- ? "__builtin_IB_bf8tohf_"
- : "__builtin_IB_hf8tohf_";
- fnName += lenSuffix;
- Type f16Ty = VectorType::get(vecSrcTy.getShape(), rewriter.getF16Type());
+ // Mangled SPV_INTEL_fp_conversions wrappers, for the reasons given in the
+ // truncf lowering above:
+ // halfN __builtin_spirv_ConvertE5M2ToFP16EXT(charN)
+ // halfN __builtin_spirv_ConvertE4M3ToFP16EXT(charN)
+ // shortN __builtin_spirv_ConvertE5M2ToBF16EXT(charN)
+ // shortN __builtin_spirv_ConvertE4M3ToBF16EXT(charN)
+ //
+ // A bf16 destination takes the bf16 form, which returns the bits as
+ // shortN, so no postprocessing is needed beyond the bitcast back.
+ bool dstIsBF16 = dstEtype == ExtfDstElemTypes::BF16;
+ StringRef convertName;
+ if (srcEtype == ExtfSrcElemTypes::BF8) { // Float8E5M2Type
+ convertName = dstIsBF16 ? "__builtin_spirv_ConvertE5M2ToBF16EXT"
+ : "__builtin_spirv_ConvertE5M2ToFP16EXT";
+ } else { // Float8E4M3FNType
+ convertName = dstIsBF16 ? "__builtin_spirv_ConvertE4M3ToBF16EXT"
+ : "__builtin_spirv_ConvertE4M3ToFP16EXT";
+ }
+ Type resElemTy = dstIsBF16 ? cast<Type>(rewriter.getI16Type())
+ : cast<Type>(rewriter.getF16Type());
+ Type resTy = VectorType::get(vecSrcTy.getShape(), resElemTy);
SmallVector<Type> argTypes{src.getType()};
SmallVector<Value> args{src};
- Value result =
- createDeviceFunctionCall(rewriter, fnName, f16Ty, argTypes, args, {},
- funcAttrs, op.getOperation())
- ->getResult(0);
-
- // When the destination is F16, we are done.
- if (dstEtype == ExtfDstElemTypes::F16) {
- rewriter.replaceOp(op, result);
- return success();
- }
-
- // BF16 destination needs some postprocessing.
- // First extend F16 to F32 and then truncate to BF16.
- // Step 2: Extend to F32.
- // Use floatN convert_floatN(halfN)
- std::string convFnName = "convert_float" + lenSuffix;
- SmallVector<Type> convArgTypes{result.getType()};
- SmallVector<Value> convArgs{result};
- convFnName = mangle(convFnName, convArgTypes);
- Type f32Ty = VectorType::get(vecSrcTy.getShape(), rewriter.getF32Type());
- result =
- createDeviceFunctionCall(rewriter, convFnName, f32Ty, convArgTypes,
- convArgs, {}, funcAttrs, op.getOperation())
- ->getResult(0);
- // Step 3: Truncate F32 to BF16.
- // Use shortN __builtin_IB_ftobf_N(floatN)
- std::string ftobfFnName = "__builtin_IB_ftobf_" + lenSuffix;
- SmallVector<Type> ftobfArgTypes{result.getType()};
- SmallVector<Value> ftobfArgs{result};
- Type i16Ty = VectorType::get(vecSrcTy.getShape(), rewriter.getI16Type());
- result =
- createDeviceFunctionCall(rewriter, ftobfFnName, i16Ty, ftobfArgTypes,
- ftobfArgs, {}, funcAttrs, op.getOperation())
- ->getResult(0);
- // The builtin returns the bf16 bits as i16, bitcast to the bf16 dst type.
- result = LLVM::BitcastOp::create(rewriter, op.getLoc(), vecDstTy, result);
+ Value result = createDeviceFunctionCall(
+ rewriter, mangle(convertName, argTypes), resTy, argTypes,
+ args, {}, funcAttrs, op.getOperation())
+ ->getResult(0);
+ // The bf16 wrapper returns the bf16 bits as i16; bitcast to the dst type.
+ if (dstIsBF16)
+ result = LLVM::BitcastOp::create(rewriter, op.getLoc(), vecDstTy, result);
rewriter.replaceOp(op, result);
return success();
}
diff --git a/mlir/test/Conversion/XeVMToLLVM/xevm_fp_conversion_lengths.mlir b/mlir/test/Conversion/XeVMToLLVM/xevm_fp_conversion_lengths.mlir
index 1de67e2a8e07b7..6d878b154448a4 100644
--- a/mlir/test/Conversion/XeVMToLLVM/xevm_fp_conversion_lengths.mlir
+++ b/mlir/test/Conversion/XeVMToLLVM/xevm_fp_conversion_lengths.mlir
@@ -1,14 +1,15 @@
// RUN: mlir-opt --convert-xevm-to-llvm --split-input-file %s | FileCheck %s
-// The fp8 and fp4 conversions are provided as one builtin per vector length.
+// The fp4 conversions are provided as one builtin per vector length, and the
+// fp8 conversions as one overload per vector length.
// This file covers the SPIR-V vector lengths other than 16, which
// xevm_mx-to-llvm.mlir covers.
-// CHECK: llvm.func spir_funccc @__builtin_IB_hftobf8_2(vector<2xf16>) -> vector<2xi8>
+// CHECK: llvm.func spir_funccc @_Z36__builtin_spirv_ConvertFP16ToE5M2EXTDv2_Dh(vector<2xf16>) -> vector<2xi8>
// CHECK-LABEL: llvm.func @truncf_f16_to_bf8_v2
// CHECK-SAME: %[[ARG0:.*]]: vector<2xf16>
llvm.func @truncf_f16_to_bf8_v2(%src: vector<2xf16>) -> vector<2xi8> {
- // CHECK: %[[RES:.*]] = llvm.call spir_funccc @__builtin_IB_hftobf8_2(%[[ARG0]])
+ // CHECK: %[[RES:.*]] = llvm.call spir_funccc @_Z36__builtin_spirv_ConvertFP16ToE5M2EXTDv2_Dh(%[[ARG0]])
// CHECK-SAME: : (vector<2xf16>) -> vector<2xi8>
%dst = xevm.truncf %src { src_etype = f16, dst_etype = bf8 } : (vector<2xf16>) -> vector<2xi8>
llvm.return %dst : vector<2xi8>
@@ -16,59 +17,47 @@ llvm.func @truncf_f16_to_bf8_v2(%src: vector<2xf16>) -> vector<2xi8> {
// -----
-// A 3 element vector is a SPIR-V vector length, and the builtins provide it.
-// CHECK: llvm.func spir_funccc @__builtin_IB_hftohf8_3(vector<3xf16>) -> vector<3xi8>
-// CHECK: llvm.func spir_funccc @_Z13convert_half3Dv3_f(vector<3xf32>) -> vector<3xf16>
-// CHECK: llvm.func spir_funccc @__builtin_IB_bftof_3(vector<3xi16>) -> vector<3xf32>
+// A 3 element vector is a SPIR-V vector length, and the wrappers overload it.
+// CHECK: llvm.func spir_funccc @_Z36__builtin_spirv_ConvertBF16ToE4M3EXTDv3_s(vector<3xi16>) -> vector<3xi8>
// CHECK-LABEL: llvm.func @truncf_bf16_to_hf8_v3
// CHECK-SAME: %[[ARG0:.*]]: vector<3xbf16>
llvm.func @truncf_bf16_to_hf8_v3(%src: vector<3xbf16>) -> vector<3xi8> {
// CHECK: %[[BC:.*]] = llvm.bitcast %[[ARG0]] : vector<3xbf16> to vector<3xi16>
- // CHECK: %[[F32:.*]] = llvm.call spir_funccc @__builtin_IB_bftof_3(%[[BC]])
- // CHECK-SAME: : (vector<3xi16>) -> vector<3xf32>
- // CHECK: %[[F16:.*]] = llvm.call spir_funccc @_Z13convert_half3Dv3_f(%[[F32]])
- // CHECK-SAME: : (vector<3xf32>) -> vector<3xf16>
- // CHECK: %[[RES:.*]] = llvm.call spir_funccc @__builtin_IB_hftohf8_3(%[[F16]])
- // CHECK-SAME: : (vector<3xf16>) -> vector<3xi8>
+ // CHECK: %[[RES:.*]] = llvm.call spir_funccc @_Z36__builtin_spirv_ConvertBF16ToE4M3EXTDv3_s(%[[BC]])
+ // CHECK-SAME: : (vector<3xi16>) -> vector<3xi8>
%dst = xevm.truncf %src { src_etype = bf16, dst_etype = f8 } : (vector<3xbf16>) -> vector<3xi8>
llvm.return %dst : vector<3xi8>
}
// -----
-// CHECK: llvm.func spir_funccc @__builtin_IB_hftobf8_8(vector<8xf16>) -> vector<8xi8>
+// CHECK: llvm.func spir_funccc @_Z36__builtin_spirv_ConvertFP16ToE5M2EXTDv8_Dh(vector<8xf16>) -> vector<8xi8>
// CHECK-LABEL: llvm.func @truncf_f16_to_bf8_v8
llvm.func @truncf_f16_to_bf8_v8(%src: vector<8xf16>) -> vector<8xi8> {
- // CHECK: llvm.call spir_funccc @__builtin_IB_hftobf8_8({{.*}}) {{.*}} : (vector<8xf16>) -> vector<8xi8>
+ // CHECK: llvm.call spir_funccc @_Z36__builtin_spirv_ConvertFP16ToE5M2EXTDv8_Dh({{.*}}) {{.*}} : (vector<8xf16>) -> vector<8xi8>
%dst = xevm.truncf %src { src_etype = f16, dst_etype = bf8 } : (vector<8xf16>) -> vector<8xi8>
llvm.return %dst : vector<8xi8>
}
// -----
-// CHECK: llvm.func spir_funccc @__builtin_IB_bf8tohf_2(vector<2xi8>) -> vector<2xf16>
+// CHECK: llvm.func spir_funccc @_Z36__builtin_spirv_ConvertE5M2ToFP16EXTDv2_c(vector<2xi8>) -> vector<2xf16>
// CHECK-LABEL: llvm.func @extf_bf8_to_f16_v2
llvm.func @extf_bf8_to_f16_v2(%src: vector<2xi8>) -> vector<2xf16> {
- // CHECK: llvm.call spir_funccc @__builtin_IB_bf8tohf_2({{.*}}) {{.*}} : (vector<2xi8>) -> vector<2xf16>
+ // CHECK: llvm.call spir_funccc @_Z36__builtin_spirv_ConvertE5M2ToFP16EXTDv2_c({{.*}}) {{.*}} : (vector<2xi8>) -> vector<2xf16>
%dst = xevm.extf %src { src_etype = bf8, dst_etype = f16 } : (vector<2xi8>) -> vector<2xf16>
llvm.return %dst : vector<2xf16>
}
// -----
-// CHECK: llvm.func spir_funccc @__builtin_IB_ftobf_4(vector<4xf32>) -> vector<4xi16>
-// CHECK: llvm.func spir_funccc @_Z14convert_float4Dv4_Dh(vector<4xf16>) -> vector<4xf32>
-// CHECK: llvm.func spir_funccc @__builtin_IB_bf8tohf_4(vector<4xi8>) -> vector<4xf16>
+// CHECK: llvm.func spir_funccc @_Z36__builtin_spirv_ConvertE5M2ToBF16EXTDv4_c(vector<4xi8>) -> vector<4xi16>
// CHECK-LABEL: llvm.func @extf_bf8_to_bf16_v4
// CHECK-SAME: %[[ARG0:.*]]: vector<4xi8>
llvm.func @extf_bf8_to_bf16_v4(%src: vector<4xi8>) -> vector<4xbf16> {
- // CHECK: %[[F16:.*]] = llvm.call spir_funccc @__builtin_IB_bf8tohf_4(%[[ARG0]])
- // CHECK-SAME: : (vector<4xi8>) -> vector<4xf16>
- // CHECK: %[[F32:.*]] = llvm.call spir_funccc @_Z14convert_float4Dv4_Dh(%[[F16]])
- // CHECK-SAME: : (vector<4xf16>) -> vector<4xf32>
- // CHECK: %[[BF:.*]] = llvm.call spir_funccc @__builtin_IB_ftobf_4(%[[F32]])
- // CHECK-SAME: : (vector<4xf32>) -> vector<4xi16>
- // CHECK: %[[RES:.*]] = llvm.bitcast %[[BF]] : vector<4xi16> to vector<4xbf16>
+ // CHECK: %[[F16:.*]] = llvm.call spir_funccc @_Z36__builtin_spirv_ConvertE5M2ToBF16EXTDv4_c(%[[ARG0]])
+ // CHECK-SAME: : (vector<4xi8>) -> vector<4xi16>
+ // CHECK: %[[RES:.*]] = llvm.bitcast %[[F16]] : vector<4xi16> to vector<4xbf16>
%dst = xevm.extf %src { src_etype = bf8, dst_etype = bf16 } : (vector<4xi8>) -> vector<4xbf16>
llvm.return %dst : vector<4xbf16>
}
diff --git a/mlir/test/Conversion/XeVMToLLVM/xevm_mx-to-llvm.mlir b/mlir/test/Conversion/XeVMToLLVM/xevm_mx-to-llvm.mlir
index 54da0238e4f90d..132b7dae077d2d 100644
--- a/mlir/test/Conversion/XeVMToLLVM/xevm_mx-to-llvm.mlir
+++ b/mlir/test/Conversion/XeVMToLLVM/xevm_mx-to-llvm.mlir
@@ -1,18 +1,18 @@
// RUN: mlir-opt --convert-xevm-to-llvm --split-input-file %s | FileCheck %s
-// CHECK: llvm.func spir_funccc @__builtin_IB_hftobf8_16(vector<16xf16>) -> vector<16xi8>
+// CHECK: llvm.func spir_funccc @_Z36__builtin_spirv_ConvertFP16ToE5M2EXTDv16_Dh(vector<16xf16>) -> vector<16xi8>
// CHECK-SAME: attributes {convergent, memory_effects = #llvm.memory_effects<other = none,
// CHECK-SAME: argMem = none, inaccessibleMem = none, errnoMem = none,
// CHECK-SAME: targetMem0 = none, targetMem1 = none>, no_unwind, will_return}
// CHECK-LABEL: llvm.func @truncf_f16_to_bf8
// CHECK-SAME: %[[ARG0:.*]]: vector<16xf16>
llvm.func @truncf_f16_to_bf8(%src: vector<16xf16>) -> vector<16xi8> {
- // CHECK: %[[VAR0:.*]] = llvm.call spir_funccc @__builtin_IB_hftobf8_16(%[[ARG0]])
+ // CHECK: %[[VAR0:.*]] = llvm.call spir_funccc @_Z36__builtin_spirv_ConvertFP16ToE5M2EXTDv16_Dh(%[[ARG0]])
// CHECK-SAME: {convergent, function_type = !llvm.func<vector<16xi8> (vector<16xf16>)>,
// CHECK-SAME: linkage = #llvm.linkage<external>, memory_effects = #llvm.memory_effects<other = none,
// CHECK-SAME: argMem = none, inaccessibleMem = none, errnoMem = none,
// CHECK-SAME: targetMem0 = none, targetMem1 = none>,
- // CHECK-SAME: no_unwind, sym_name = "__builtin_IB_hftobf8_16",
+ // CHECK-SAME: no_unwind, sym_name = "_Z36__builtin_spirv_ConvertFP16ToE5M2EXTDv16_Dh",
// CHECK-SAME: visibility_ = 0 : i64, will_return} :
// CHECK-SAME: (vector<16xf16>) -> vector<16xi8>
%dst = xevm.truncf %src { src_etype = f16, dst_etype = bf8 } : (vector<16xf16>) -> vector<16xi8>
@@ -21,19 +21,19 @@ llvm.func @truncf_f16_to_bf8(%src: vector<16xf16>) -> vector<16xi8> {
// -----
-// CHECK: llvm.func spir_funccc @__builtin_IB_hftohf8_16(vector<16xf16>) -> vector<16xi8>
+// CHECK: llvm.func spir_funccc @_Z36__builtin_spirv_ConvertFP16ToE4M3EXTDv16_Dh(vector<16xf16>) -> vector<16xi8>
// CHECK-SAME: attributes {convergent, memory_effects = #llvm.memory_effects<other = none,
// CHECK-SAME: argMem = none, inaccessibleMem = none, errnoMem = none,
// CHECK-SAME: targetMem0 = none, targetMem1 = none>, no_unwind, will_return}
// CHECK-LABEL: llvm.func @truncf_f16_to_hf8
// CHECK-SAME: %[[ARG0:.*]]: vector<16xf16>
llvm.func @truncf_f16_to_hf8(%src: vector<16xf16>) -> vector<16xi8> {
- // CHECK: %[[VAR0:.*]] = llvm.call spir_funccc @__builtin_IB_hftohf8_16(%[[ARG0]])
+ // CHECK: %[[VAR0:.*]] = llvm.call spir_funccc @_Z36__builtin_spirv_ConvertFP16ToE4M3EXTDv16_Dh(%[[ARG0]])
// CHECK-SAME: {convergent, function_type = !llvm.func<vector<16xi8> (vector<16xf16>)>,
// CHECK-SAME: linkage = #llvm.linkage<external>, memory_effects = #llvm.memory_effects<other = none,
// CHECK-SAME: argMem = none, inaccessibleMem = none, errnoMem = none,
// CHECK-SAME: targetMem0 = none, targetMem1 = none>,
- // CHECK-SAME: no_unwind, sym_name = "__builtin_IB_hftohf8_16",
+ // CHECK-SAME: no_unwind, sym_name = "_Z36__builtin_spirv_ConvertFP16ToE4M3EXTDv16_Dh",
// CHECK-SAME: visibility_ = 0 : i64, will_return} :
// CHECK-SAME: (vector<16xf16>) -> vector<16xi8>
%dst = xevm.truncf %src { src_etype = f16, dst_etype = f8 } : (vector<16xf16>) -> vector<16xi8>
@@ -42,38 +42,26 @@ llvm.func @truncf_f16_to_hf8(%src: vector<16xf16>) -> vector<16xi8> {
// -----
-// CHECK: llvm.func spir_funccc @__builtin_IB_hftobf8_16(vector<16xf16>) -> vector<16xi8>
-// CHECK: llvm.func spir_funccc @_Z14convert_half16Dv16_f(vector<16xf32>) -> vector<16xf16>
-// CHECK: llvm.func spir_funccc @__builtin_IB_bftof_16(vector<16xi16>) -> vector<16xf32>
+// CHECK: llvm.func spir_funccc @_Z36__builtin_spirv_ConvertBF16ToE5M2EXTDv16_s(vector<16xi16>) -> vector<16xi8>
// CHECK-LABEL: llvm.func @truncf_bf16_to_bf8
// CHECK-SAME: %[[ARG0:.*]]: vector<16xbf16>
llvm.func @truncf_bf16_to_bf8(%src: vector<16xbf16>) -> vector<16xi8> {
// CHECK: %[[VAR0:.*]] = llvm.bitcast %[[ARG0]] : vector<16xbf16> to vector<16xi16>
- // CHECK: %[[VAR1:.*]] = llvm.call spir_funccc @__builtin_IB_bftof_16(%[[VAR0]])
- // CHECK-SAME: : (vector<16xi16>) -> vector<16xf32>
- // CHECK: %[[VAR2:.*]] = llvm.call spir_funccc @_Z14convert_half16Dv16_f(%[[VAR1]])
- // CHECK-SAME: : (vector<16xf32>) -> vector<16xf16>
- // CHECK: %[[VAR3:.*]] = llvm.call spir_funccc @__builtin_IB_hftobf8_16(%[[VAR2]])
- // CHECK-SAME: : (vector<16xf16>) -> vector<16xi8>
+ // CHECK: %[[VAR3:.*]] = llvm.call spir_funccc @_Z36__builtin_spirv_ConvertBF16ToE5M2EXTDv16_s(%[[VAR0]])
+ // CHECK-SAME: : (vector<16xi16>) -> vector<16xi8>
%dst = xevm.truncf %src { src_etype = bf16, dst_etype = bf8 } : (vector<16xbf16>) -> vector<16xi8>
llvm.return %dst : vector<16xi8>
}
// -----
-// CHECK: llvm.func spir_funccc @__builtin_IB_hftohf8_16(vector<16xf16>) -> vector<16xi8>
-// CHECK: llvm.func spir_funccc @_Z14convert_half16Dv16_f(vector<16xf32>) -> vector<16xf16>
-// CHECK: llvm.func spir_funccc @__builtin_IB_bftof_16(vector<16xi16>) -> vector<16xf32>
+// CHECK: llvm.func spir_funccc @_Z36__builtin_spirv_ConvertBF16ToE4M3EXTDv16_s(vector<16xi16>) -> vector<16xi8>
// CHECK-LABEL: llvm.func @truncf_bf16_to_hf8
// CHECK-SAME: %[[ARG0:.*]]: vector<16xbf16>
llvm.func @truncf_bf16_to_hf8(%src: vector<16xbf16>) -> vector<16xi8> {
// CHECK: %[[VAR0:.*]] = llvm.bitcast %[[ARG0]] : vector<16xbf16> to vector<16xi16>
- // CHECK: %[[VAR1:.*]] = llvm.call spir_funccc @__builtin_IB_bftof_16(%[[VAR0]])
- // CHECK-SAME: : (vector<16xi16>) -> vector<16xf32>
- // CHECK: %[[VAR2:.*]] = llvm.call spir_funccc @_Z14convert_half16Dv16_f(%[[VAR1]])
- // CHECK-SAME: : (vector<16xf32>) -> vector<16xf16>
- // CHECK: %[[VAR3:.*]] = llvm.call spir_funccc @__builtin_IB_hftohf8_16(%[[VAR2]])
- // CHECK-SAME: : (vector<16xf16>) -> vector<16xi8>
+ // CHECK: %[[VAR3:.*]] = llvm.call spir_funccc @_Z36__builtin_spirv_ConvertBF16ToE4M3EXTDv16_s(%[[VAR0]])
+ // CHECK-SAME: : (vector<16xi16>) -> vector<16xi8>
%dst = xevm.truncf %src { src_etype = bf16, dst_etype = f8 } : (vector<16xbf16>) -> vector<16xi8>
llvm.return %dst : vector<16xi8>
}
@@ -105,19 +93,19 @@ llvm.func @truncf_bf16_to_e2m1(%src: vector<16xbf16>) -> vector<8xi8> {
// -----
-// CHECK: llvm.func spir_funccc @__builtin_IB_bf8tohf_16(vector<16xi8>) -> vector<16xf16>
+// CHECK: llvm.func spir_funccc @_Z36__builtin_spirv_ConvertE5M2ToFP16EXTDv16_c(vector<16xi8>) -> vector<16xf16>
// CHECK-SAME: attributes {convergent, memory_effects = #llvm.memory_effects<other = none,
// CHECK-SAME: argMem = none, inaccessibleMem = none, errnoMem = none,
// CHECK-SAME: targetMem0 = none, targetMem1 = none>, no_unwind, will_return}
// CHECK-LABEL: llvm.func @extf_bf8_to_f16
// CHECK-SAME: %[[ARG0:.*]]: vector<16xi8>
llvm.func @extf_bf8_to_f16(%src: vector<16xi8>) -> vector<16xf16> {
- // CHECK: %[[VAR0:.*]] = llvm.call spir_funccc @__builtin_IB_bf8tohf_16(%[[ARG0]])
+ // CHECK: %[[VAR0:.*]] = llvm.call spir_funccc @_Z36__builtin_spirv_ConvertE5M2ToFP16EXTDv16_c(%[[ARG0]])
// CHECK-SAME: {convergent, function_type = !llvm.func<vector<16xf16> (vector<16xi8>)>,
// CHECK-SAME: linkage = #llvm.linkage<external>, memory_effects = #llvm.memory_effects<other = none,
// CHECK-SAME: argMem = none, inaccessibleMem = none, errnoMem = none,
// CHECK-SAME: targetMem0 = none, targetMem1 = none>,
- // CHECK-SAME: no_unwind, sym_name = "__builtin_IB_bf8tohf_16",
+ // CHECK-SAME: no_unwind, sym_name = "_Z36__builtin_spirv_ConvertE5M2ToFP16EXTDv16_c",
// CHECK-SAME: visibility_ = 0 : i64, will_return} :
// CHECK-SAME: (vector<16xi8>) -> vector<16xf16>
%dst = xevm.extf %src { src_etype = bf8, dst_etype = f16 } : (vector<16xi8>) -> vector<16xf16>
@@ -126,19 +114,19 @@ llvm.func @extf_bf8_to_f16(%src: vector<16xi8>) -> vector<16xf16> {
// -----
-// CHECK: llvm.func spir_funccc @__builtin_IB_hf8tohf_16(vector<16xi8>) -> vector<16xf16>
+// CHECK: llvm.func spir_funccc @_Z36__builtin_spirv_ConvertE4M3ToFP16EXTDv16_c(vector<16xi8>) -> vector<16xf16>
// CHECK-SAME: attributes {convergent, memory_effects = #llvm.memory_effects<other = none,
// CHECK-SAME: argMem = none, inaccessibleMem = none, errnoMem = none,
// CHECK-SAME: targetMem0 = none, targetMem1 = none>, no_unwind, will_return}
// CHECK-LABEL: llvm.func @extf_f8_to_f16
// CHECK-SAME: %[[ARG0:.*]]: vector<16xi8>
llvm.func @extf_f8_to_f16(%src: vector<16xi8>) -> vector<16xf16> {
- // CHECK: %[[VAR0:.*]] = llvm.call spir_funccc @__builtin_IB_hf8tohf_16(%[[ARG0]])
+ // CHECK: %[[VAR0:.*]] = llvm.call spir_funccc @_Z36__builtin_spirv_ConvertE4M3ToFP16EXTDv16_c(%[[ARG0]])
// CHECK-SAME: {convergent, function_type = !llvm.func<vector<16xf16> (vector<16xi8>)>,
// CHECK-SAME: linkage = #llvm.linkage<external>, memory_effects = #llvm.memory_effects<other = none,
// CHECK-SAME: argMem = none, inaccessibleMem = none, errnoMem = none,
// CHECK-SAME: targetMem0 = none, targetMem1 = none>,
- // CHECK-SAME: no_unwind, sym_name = "__builtin_IB_hf8tohf_16",
+ // CHECK-SAME: no_unwind, sym_name = "_Z36__builtin_spirv_ConvertE4M3ToFP16EXTDv16_c",
// CHECK-SAME: visibility_ = 0 : i64, will_return} :
// CHECK-SAME: (vector<16xi8>) -> vector<16xf16>
%dst = xevm.extf %src { src_etype = f8, dst_etype = f16 } : (vector<16xi8>) -> vector<16xf16>
@@ -147,38 +135,26 @@ llvm.func @extf_f8_to_f16(%src: vector<16xi8>) -> vector<16xf16> {
// -----
-// CHECK: llvm.func spir_funccc @__builtin_IB_ftobf_16(vector<16xf32>) -> vector<16xi16>
-// CHECK: llvm.func spir_funccc @_Z15convert_float16Dv16_Dh(vector<16xf16>) -> vector<16xf32>
-// CHECK: llvm.func spir_funccc @__builtin_IB_bf8tohf_16(vector<16xi8>) -> vector<16xf16>
+// CHECK: llvm.func spir_funccc @_Z36__builtin_spirv_ConvertE5M2ToBF16EXTDv16_c(vector<16xi8>) -> vector<16xi16>
// CHECK-LABEL: llvm.func @extf_bf8_to_bf16
// CHECK-SAME: %[[ARG0:.*]]: vector<16xi8>
llvm.func @extf_bf8_to_bf16(%src: vector<16xi8>) -> vector<16xbf16> {
- // CHECK: %[[VAR0:.*]] = llvm.call spir_funccc @__builtin_IB_bf8tohf_16(%[[ARG0]])
- // CHECK-SAME: : (vector<16xi8>) -> vector<16xf16>
- // CHECK: %[[VAR1:.*]] = llvm.call spir_funccc @_Z15convert_float16Dv16_Dh(%[[VAR0]])
- // CHECK-SAME: : (vector<16xf16>) -> vector<16xf32>
- // CHECK: %[[VAR2:.*]] = llvm.call spir_funccc @__builtin_IB_ftobf_16(%[[VAR1]])
- // CHECK-SAME: : (vector<16xf32>) -> vector<16xi16>
- // CHECK: %[[VAR3:.*]] = llvm.bitcast %[[VAR2]] : vector<16xi16> to vector<16xbf16>
+ // CHECK: %[[VAR0:.*]] = llvm.call spir_funccc @_Z36__builtin_spirv_ConvertE5M2ToBF16EXTDv16_c(%[[ARG0]])
+ // CHECK-SAME: : (vector<16xi8>) -> vector<16xi16>
+ // CHECK: %[[VAR3:.*]] = llvm.bitcast %[[VAR0]] : vector<16xi16> to vector<16xbf16>
%dst = xevm.extf %src { src_etype = bf8, dst_etype = bf16 } : (vector<16xi8>) -> vector<16xbf16>
llvm.return %dst : vector<16xbf16>
}
// -----
-// CHECK: llvm.func spir_funccc @__builtin_IB_ftobf_16(vector<16xf32>) -> vector<16xi16>
-// CHECK: llvm.func spir_funccc @_Z15convert_float16Dv16_Dh(vector<16xf16>) -> vector<16xf32>
-// CHECK: llvm.func spir_funccc @__builtin_IB_hf8tohf_16(vector<16xi8>) -> vector<16xf16>
+// CHECK: llvm.func spir_funccc @_Z36__builtin_spirv_ConvertE4M3ToBF16EXTDv16_c(vector<16xi8>) -> vector<16xi16>
// CHECK-LABEL: llvm.func @extf_f8_to_bf16
// CHECK-SAME: %[[ARG0:.*]]: vector<16xi8>
llvm.func @extf_f8_to_bf16(%src: vector<16xi8>) -> vector<16xbf16> {
- // CHECK: %[[VAR0:.*]] = llvm.call spir_funccc @__builtin_IB_hf8tohf_16(%[[ARG0]])
- // CHECK-SAME: : (vector<16xi8>) -> vector<16xf16>
- // CHECK: %[[VAR1:.*]] = llvm.call spir_funccc @_Z15convert_float16Dv16_Dh(%[[VAR0]])
- // CHECK-SAME: : (vector<16xf16>) -> vector<16xf32>
- // CHECK: %[[VAR2:.*]] = llvm.call spir_funccc @__builtin_IB_ftobf_16(%[[VAR1]])
- // CHECK-SAME: : (vector<16xf32>) -> vector<16xi16>
- // CHECK: %[[VAR3:.*]] = llvm.bitcast %[[VAR2]] : vector<16xi16> to vector<16xbf16>
+ // CHECK: %[[VAR0:.*]] = llvm.call spir_funccc @_Z36__builtin_spirv_ConvertE4M3ToBF16EXTDv16_c(%[[ARG0]])
+ // CHECK-SAME: : (vector<16xi8>) -> vector<16xi16>
+ // CHECK: %[[VAR3:.*]] = llvm.bitcast %[[VAR0]] : vector<16xi16> to vector<16xbf16>
%dst = xevm.extf %src { src_etype = f8, dst_etype = bf16 } : (vector<16xi8>) -> vector<16xbf16>
llvm.return %dst : vector<16xbf16>
}
>From 9130ed8288280301d65f60e5e4a68ab5649ae071 Mon Sep 17 00:00:00 2001
From: "Lee, Sang Ik" <sang.ik.lee at intel.com>
Date: Thu, 15 Oct 2026 18:47:50 +0000
Subject: [PATCH 3/3] [mlir][XeVM] Use IGC's vector fp4 up-conversion builtin
instead of hand packing
ExtfToOCLPattern unpacked e2m1 by emitting the __builtin_IB_shfl_idx4_* pair directly: fetch the format lookup table, call the per-source-byte packed conversion, bitcast the dword-per-byte result to the destination element type, then shuffle away what an odd length rounded up to. The byte count was pasted onto the builtin name, and a single byte needed a scalar rather than a one element vector.
IGC's BiFModule already exposes that as __builtin_spirv_ConvertE2M1To{FP16,BF16}INTEL, declared for every SPIR-V vector length, and its implementation is exactly that sequence - lookup table, packed conversion and the trailing trim included. Call it and let IGC do the breakdown, mirroring what the fp4 down-conversion now does.
The name cannot come from mangle(). These builtins take a vector of i4, which the SPIR-V translator mangles as 'i', the same code as 'int', and IGC's PromoteSubByte pass then widens the parameter to whole bytes, so Dv<N>_i is declared taking charN/2. upconversions_fp4.cl mangles them by hand for that reason, since overloadable would mangle on char instead, so the suffix is built from the element count here too. The 2D and 1D block IO lowerings in this file already build their _Z names the same way.
takeLeadingElements had no other caller and goes with it.
Verified on the CRI simulator. A new integration test round trips f16 through e2m1 and back over values e2m1 represents exactly, so both directions have to reproduce their input bit for bit; the existing block-scaled e2m1 DPAS test and the fp8 and bf16 variants still pass. Separately probed the two awkward lengths before committing: Dv2_i with its one element vector parameter and Dv3_i with its odd width both link and convert correctly, as does Dv16_i. mlir/test/Conversion/{XeVMToLLVM,XeGPUToXeVM}, mlir/test/Dialect/{XeVM,XeGPU} and mlir/test/Target/LLVM are clean.
---
mlir/lib/Conversion/XeVMToLLVM/XeVMToLLVM.cpp | 101 +++++++-----------
.../xevm_fp_conversion_lengths.mlir | 47 ++++----
.../XeVMToLLVM/xevm_mx-to-llvm.mlir | 23 ++--
3 files changed, 68 insertions(+), 103 deletions(-)
diff --git a/mlir/lib/Conversion/XeVMToLLVM/XeVMToLLVM.cpp b/mlir/lib/Conversion/XeVMToLLVM/XeVMToLLVM.cpp
index 68e7f59335c1be..d9989d74dcbdca 100644
--- a/mlir/lib/Conversion/XeVMToLLVM/XeVMToLLVM.cpp
+++ b/mlir/lib/Conversion/XeVMToLLVM/XeVMToLLVM.cpp
@@ -1177,19 +1177,6 @@ static Value castIfNeeded(ConversionPatternRewriter &rewriter, Location loc,
return LLVM::BitcastOp::create(rewriter, loc, ty, val);
}
-/// Selects `numElements` leading elements of the vector `val`. Used to drop the
-/// padding the 3 element case needs, as the hardware conversions work on whole
-/// pairs of elements.
-static Value takeLeadingElements(ConversionPatternRewriter &rewriter,
- Location loc, Value val, int64_t numElements) {
- auto vecTy = cast<VectorType>(val.getType());
- if (vecTy.getNumElements() == numElements)
- return val;
- SmallVector<int32_t> mask =
- llvm::to_vector(llvm::seq<int32_t>(0, static_cast<int32_t>(numElements)));
- return LLVM::ShuffleVectorOp::create(rewriter, loc, val, val, mask);
-}
-
//
// Note: TruncfToOCLPattern and ExtfToOCLPattern does not lower to OpenCL API
// calls as there are not official ones yet. They are lowered directly to Intel
@@ -1364,58 +1351,50 @@ class ExtfToOCLPattern : public OpConversionPattern<ExtfOp> {
// Handle the case where src type is fp4 (e2m1) first.
if (srcEtype == ExtfSrcElemTypes::E2M1) {
- // Two fp4 values are packed per source byte, and one builtin exists per
- // source byte count:
- // uint16 __builtin_IB_shfl_idx4_lut(int lut_index)
- // uint __builtin_IB_shfl_idx4_to_fp16_packed(uint16 lut, char src)
- // uintN __builtin_IB_shfl_idx4_to_fp16_N_packed(uint16 lut, charN src)
- // Each returns one dword, holding two f16/bf16 values, per source byte.
- // The lookup table selects the target format:
- // 7 = e2m1 -> f16, 5 = e2m1 -> bf16.
+ // IGC's BiFModule exposes the fp4 unpacking as
+ // `__builtin_spirv_ConvertE2M1To{FP16,BF16}INTEL`, declared for every
+ // SPIR-V vector length. Its implementation *is* the
+ // `__builtin_IB_shfl_idx4_*` pair this used to emit by hand - the format
+ // lookup table, the per-byte packed conversion, and the trailing trim an
+ // odd length needs - so calling it hands all of that to IGC.
//
- // A byte is the conversion granularity, so an odd length reads one spare
- // value that is dropped afterwards. The op verifier has already checked
- // that the source is exactly as wide as those bytes.
+ // Unlike the other conversions here the name cannot come from mangle().
+ // These builtins take a vector of i4, which the SPIR-V translator
+ // mangles as `i`, the same code as `int`, and IGC's PromoteSubByte pass
+ // then widens the parameter to whole bytes. So the declared parameter is
+ // half as wide as the name says: `Dv<N>_i` takes `charN/2`.
+ // upconversions_fp4.cl mangles these by hand for the same reason, as
+ // `overloadable` would mangle them on `char` instead. Build the suffix
+ // from the element count to match.
+ //
+ // See IGC/BiFModule/Implementation/SPV_INTEL_fp_conversions/
+ // upconversions_fp4.cl for the declarations and their semantics.
+ StringRef baseName = (dstEtype == ExtfDstElemTypes::F16)
+ ? "__builtin_spirv_ConvertE2M1ToFP16INTEL"
+ : "__builtin_spirv_ConvertE2M1ToBF16INTEL";
+ // Two fp4 values pack into one byte, so the source is half as wide,
+ // rounded up. A scalar source is spelled as such for a single byte, and
+ // the builtin wants the one element vector, hence the cast.
int64_t numBytes = llvm::divideCeil(numElements, 2);
- constexpr int kLutE2M1ToF16 = 7;
- constexpr int kLutE2M1ToBF16 = 5;
- int lutIndex =
- (dstEtype == ExtfDstElemTypes::F16) ? kLutE2M1ToF16 : kLutE2M1ToBF16;
- Value lutIdx = LLVM::ConstantOp::create(rewriter, loc,
- rewriter.getI32Type(), lutIndex);
- Type lutTy = VectorType::get(16, rewriter.getI32Type());
- Value lut =
- createDeviceFunctionCall(rewriter, "__builtin_IB_shfl_idx4_lut",
- lutTy, {lutIdx.getType()}, {lutIdx}, {},
- funcAttrs, op.getOperation())
- ->getResult(0);
- // A single byte is passed as a bare i8, and one dword returned as a bare
- // i32, rather than as one element vectors SPIR-V has no type for.
- Type i8Ty = rewriter.getI8Type();
- Type i32Ty = rewriter.getI32Type();
- std::string fnName = "__builtin_IB_shfl_idx4_to_fp16_";
- Type argTy, packedResTy;
- if (numBytes == 1) {
- argTy = i8Ty;
- packedResTy = i32Ty;
- } else {
- fnName += std::to_string(numBytes) + "_";
- argTy = VectorType::get(numBytes, i8Ty);
- packedResTy = VectorType::get(numBytes, i32Ty);
- }
- fnName += "packed";
- SmallVector<Type> convArgTypes{lut.getType(), argTy};
- SmallVector<Value> convArgs{lut, castIfNeeded(rewriter, loc, argTy, src)};
+ Value arg = castIfNeeded(
+ rewriter, loc, VectorType::get(numBytes, rewriter.getI8Type()), src);
+ // A bf16 result comes back as its bit pattern, which the builtin spells
+ // `short`. The builtin returns exactly `numElements` values, having
+ // already dropped what an odd length rounded up to.
+ Type resElemTy = (dstEtype == ExtfDstElemTypes::F16)
+ ? cast<Type>(rewriter.getF16Type())
+ : cast<Type>(rewriter.getI16Type());
+ Type resTy = VectorType::get(numElements, resElemTy);
+ std::string fnName = llvm::formatv("_Z{0}{1}Dv{2}_i", baseName.size(),
+ baseName, numElements)
+ .str();
+ SmallVector<Type> argTypes{arg.getType()};
+ SmallVector<Value> args{arg};
Value result =
- createDeviceFunctionCall(rewriter, fnName, packedResTy, convArgTypes,
- convArgs, {}, funcAttrs, op.getOperation())
+ createDeviceFunctionCall(rewriter, fnName, resTy, argTypes, args, {},
+ funcAttrs, op.getOperation())
->getResult(0);
- // The builtin returns the f16/bf16 bits packed as i32, bitcast to the
- // f16/bf16 dst type and drop the padding an odd length produced.
- Type wideTy = VectorType::get(numBytes * 2, vecDstTy.getElementType());
- result = LLVM::BitcastOp::create(rewriter, loc, wideTy, result);
- result = takeLeadingElements(rewriter, loc, result, numElements);
- rewriter.replaceOp(op, result);
+ rewriter.replaceOp(op, castIfNeeded(rewriter, loc, vecDstTy, result));
return success();
}
diff --git a/mlir/test/Conversion/XeVMToLLVM/xevm_fp_conversion_lengths.mlir b/mlir/test/Conversion/XeVMToLLVM/xevm_fp_conversion_lengths.mlir
index 6d878b154448a4..4f8bfc3fe6764e 100644
--- a/mlir/test/Conversion/XeVMToLLVM/xevm_fp_conversion_lengths.mlir
+++ b/mlir/test/Conversion/XeVMToLLVM/xevm_fp_conversion_lengths.mlir
@@ -122,61 +122,56 @@ llvm.func @truncf_bf16_to_e2m1_v8(%src: vector<8xbf16>) -> vector<4xi8> {
// -----
-// The fp4 up-conversion takes a scalar source byte for the shortest length.
-// CHECK: llvm.func spir_funccc @__builtin_IB_shfl_idx4_to_fp16_packed(vector<16xi32>, i8) -> i32
-// CHECK: llvm.func spir_funccc @__builtin_IB_shfl_idx4_lut(i32) -> vector<16xi32>
+// The fp4 up-conversion takes a scalar source byte for the shortest length. It
+// is cast to the one element vector the builtin's promoted parameter wants.
+// CHECK: llvm.func spir_funccc @_Z38__builtin_spirv_ConvertE2M1ToFP16INTELDv2_i(vector<1xi8>) -> vector<2xf16>
// CHECK-LABEL: llvm.func @extf_e2m1_to_f16_v2
// CHECK-SAME: %[[ARG0:.*]]: i8
llvm.func @extf_e2m1_to_f16_v2(%src: i8) -> vector<2xf16> {
- // CHECK: %[[LUTIDX:.*]] = llvm.mlir.constant(7 : i32) : i32
- // CHECK: %[[LUT:.*]] = llvm.call spir_funccc @__builtin_IB_shfl_idx4_lut(%[[LUTIDX]])
- // CHECK-SAME: : (i32) -> vector<16xi32>
- // CHECK: %[[CONV:.*]] = llvm.call spir_funccc @__builtin_IB_shfl_idx4_to_fp16_packed(%[[LUT]], %[[ARG0]])
- // CHECK-SAME: : (vector<16xi32>, i8) -> i32
- // CHECK: %[[RES:.*]] = llvm.bitcast %[[CONV]] : i32 to vector<2xf16>
+ // CHECK: %[[BC:.*]] = llvm.bitcast %[[ARG0]] : i8 to vector<1xi8>
+ // CHECK: %[[RES:.*]] = llvm.call spir_funccc @_Z38__builtin_spirv_ConvertE2M1ToFP16INTELDv2_i(%[[BC]])
+ // CHECK-SAME: : (vector<1xi8>) -> vector<2xf16>
%dst = xevm.extf %src { src_etype = e2m1, dst_etype = f16 } : (i8) -> vector<2xf16>
llvm.return %dst : vector<2xf16>
}
// -----
-// Three values are read as two whole bytes, and the spare value dropped.
-// CHECK: llvm.func spir_funccc @__builtin_IB_shfl_idx4_to_fp16_2_packed(vector<16xi32>, vector<2xi8>) -> vector<2xi32>
-// CHECK: llvm.func spir_funccc @__builtin_IB_shfl_idx4_lut(i32) -> vector<16xi32>
+// Three values are read as two whole bytes; the builtin returns three, having
+// already dropped the spare one.
+// CHECK: llvm.func spir_funccc @_Z38__builtin_spirv_ConvertE2M1ToFP16INTELDv3_i(vector<2xi8>) -> vector<3xf16>
// CHECK-LABEL: llvm.func @extf_e2m1_to_f16_v3
// CHECK-SAME: %[[ARG0:.*]]: vector<2xi8>
llvm.func @extf_e2m1_to_f16_v3(%src: vector<2xi8>) -> vector<3xf16> {
- // CHECK: %[[CONV:.*]] = llvm.call spir_funccc @__builtin_IB_shfl_idx4_to_fp16_2_packed({{.*}}, %[[ARG0]])
- // CHECK-SAME: : (vector<16xi32>, vector<2xi8>) -> vector<2xi32>
- // CHECK: %[[WIDE:.*]] = llvm.bitcast %[[CONV]] : vector<2xi32> to vector<4xf16>
- // CHECK: %[[RES:.*]] = llvm.shufflevector %[[WIDE]], %[[WIDE]] [0, 1, 2] : vector<4xf16>
+ // CHECK: %[[RES:.*]] = llvm.call spir_funccc @_Z38__builtin_spirv_ConvertE2M1ToFP16INTELDv3_i(%[[ARG0]])
+ // CHECK-SAME: : (vector<2xi8>) -> vector<3xf16>
%dst = xevm.extf %src { src_etype = e2m1, dst_etype = f16 } : (vector<2xi8>) -> vector<3xf16>
llvm.return %dst : vector<3xf16>
}
// -----
-// The lookup table index selects the destination format: 5 for bf16, 7 for f16.
-// CHECK: llvm.func spir_funccc @__builtin_IB_shfl_idx4_to_fp16_2_packed(vector<16xi32>, vector<2xi8>) -> vector<2xi32>
+// The destination format picks the builtin, and a bf16 result arrives as its
+// bit pattern.
+// CHECK: llvm.func spir_funccc @_Z38__builtin_spirv_ConvertE2M1ToBF16INTELDv4_i(vector<2xi8>) -> vector<4xi16>
// CHECK-LABEL: llvm.func @extf_e2m1_to_bf16_v4
+// CHECK-SAME: %[[ARG0:.*]]: vector<2xi8>
llvm.func @extf_e2m1_to_bf16_v4(%src: vector<2xi8>) -> vector<4xbf16> {
- // CHECK: %[[LUTIDX:.*]] = llvm.mlir.constant(5 : i32) : i32
- // CHECK: llvm.call spir_funccc @__builtin_IB_shfl_idx4_lut(%[[LUTIDX]])
- // CHECK: %[[CONV:.*]] = llvm.call spir_funccc @__builtin_IB_shfl_idx4_to_fp16_2_packed
- // CHECK: %[[RES:.*]] = llvm.bitcast %[[CONV]] : vector<2xi32> to vector<4xbf16>
+ // CHECK: %[[CONV:.*]] = llvm.call spir_funccc @_Z38__builtin_spirv_ConvertE2M1ToBF16INTELDv4_i(%[[ARG0]])
+ // CHECK-SAME: : (vector<2xi8>) -> vector<4xi16>
+ // CHECK: %[[RES:.*]] = llvm.bitcast %[[CONV]] : vector<4xi16> to vector<4xbf16>
%dst = xevm.extf %src { src_etype = e2m1, dst_etype = bf16 } : (vector<2xi8>) -> vector<4xbf16>
llvm.return %dst : vector<4xbf16>
}
// -----
-// CHECK: llvm.func spir_funccc @__builtin_IB_shfl_idx4_to_fp16_4_packed(vector<16xi32>, vector<4xi8>) -> vector<4xi32>
+// CHECK: llvm.func spir_funccc @_Z38__builtin_spirv_ConvertE2M1ToFP16INTELDv8_i(vector<4xi8>) -> vector<8xf16>
// CHECK-LABEL: llvm.func @extf_e2m1_to_f16_v8
// CHECK-SAME: %[[ARG0:.*]]: vector<4xi8>
llvm.func @extf_e2m1_to_f16_v8(%src: vector<4xi8>) -> vector<8xf16> {
- // CHECK: %[[CONV:.*]] = llvm.call spir_funccc @__builtin_IB_shfl_idx4_to_fp16_4_packed({{.*}}, %[[ARG0]])
- // CHECK-SAME: : (vector<16xi32>, vector<4xi8>) -> vector<4xi32>
- // CHECK: %[[RES:.*]] = llvm.bitcast %[[CONV]] : vector<4xi32> to vector<8xf16>
+ // CHECK: %[[RES:.*]] = llvm.call spir_funccc @_Z38__builtin_spirv_ConvertE2M1ToFP16INTELDv8_i(%[[ARG0]])
+ // CHECK-SAME: : (vector<4xi8>) -> vector<8xf16>
%dst = xevm.extf %src { src_etype = e2m1, dst_etype = f16 } : (vector<4xi8>) -> vector<8xf16>
llvm.return %dst : vector<8xf16>
}
diff --git a/mlir/test/Conversion/XeVMToLLVM/xevm_mx-to-llvm.mlir b/mlir/test/Conversion/XeVMToLLVM/xevm_mx-to-llvm.mlir
index 132b7dae077d2d..f4717909bc2298 100644
--- a/mlir/test/Conversion/XeVMToLLVM/xevm_mx-to-llvm.mlir
+++ b/mlir/test/Conversion/XeVMToLLVM/xevm_mx-to-llvm.mlir
@@ -161,34 +161,25 @@ llvm.func @extf_f8_to_bf16(%src: vector<16xi8>) -> vector<16xbf16> {
// -----
-// CHECK: llvm.func spir_funccc @__builtin_IB_shfl_idx4_to_fp16_8_packed(vector<16xi32>, vector<8xi8>) -> vector<8xi32>
-// CHECK: llvm.func spir_funccc @__builtin_IB_shfl_idx4_lut(i32) -> vector<16xi32>
+// CHECK: llvm.func spir_funccc @_Z38__builtin_spirv_ConvertE2M1ToFP16INTELDv16_i(vector<8xi8>) -> vector<16xf16>
// CHECK-LABEL: llvm.func @extf_e2m1_to_f16
// CHECK-SAME: %[[ARG0:.*]]: vector<8xi8>
llvm.func @extf_e2m1_to_f16(%src: vector<8xi8>) -> vector<16xf16> {
- // CHECK: %[[LUTIDX:.*]] = llvm.mlir.constant(7 : i32) : i32
- // CHECK: %[[LUT:.*]] = llvm.call spir_funccc @__builtin_IB_shfl_idx4_lut(%[[LUTIDX]])
- // CHECK-SAME: : (i32) -> vector<16xi32>
- // CHECK: %[[CONV:.*]] = llvm.call spir_funccc @__builtin_IB_shfl_idx4_to_fp16_8_packed(%[[LUT]], %[[ARG0]])
- // CHECK-SAME: : (vector<16xi32>, vector<8xi8>) -> vector<8xi32>
- // CHECK: %[[RES:.*]] = llvm.bitcast %[[CONV]] : vector<8xi32> to vector<16xf16>
+ // CHECK: %[[RES:.*]] = llvm.call spir_funccc @_Z38__builtin_spirv_ConvertE2M1ToFP16INTELDv16_i(%[[ARG0]])
+ // CHECK-SAME: : (vector<8xi8>) -> vector<16xf16>
%dst = xevm.extf %src { src_etype = e2m1, dst_etype = f16 } : (vector<8xi8>) -> vector<16xf16>
llvm.return %dst : vector<16xf16>
}
// -----
-// CHECK: llvm.func spir_funccc @__builtin_IB_shfl_idx4_to_fp16_8_packed(vector<16xi32>, vector<8xi8>) -> vector<8xi32>
-// CHECK: llvm.func spir_funccc @__builtin_IB_shfl_idx4_lut(i32) -> vector<16xi32>
+// CHECK: llvm.func spir_funccc @_Z38__builtin_spirv_ConvertE2M1ToBF16INTELDv16_i(vector<8xi8>) -> vector<16xi16>
// CHECK-LABEL: llvm.func @extf_e2m1_to_bf16
// CHECK-SAME: %[[ARG0:.*]]: vector<8xi8>
llvm.func @extf_e2m1_to_bf16(%src: vector<8xi8>) -> vector<16xbf16> {
- // CHECK: %[[LUTIDX:.*]] = llvm.mlir.constant(5 : i32) : i32
- // CHECK: %[[LUT:.*]] = llvm.call spir_funccc @__builtin_IB_shfl_idx4_lut(%[[LUTIDX]])
- // CHECK-SAME: : (i32) -> vector<16xi32>
- // CHECK: %[[CONV:.*]] = llvm.call spir_funccc @__builtin_IB_shfl_idx4_to_fp16_8_packed(%[[LUT]], %[[ARG0]])
- // CHECK-SAME: : (vector<16xi32>, vector<8xi8>) -> vector<8xi32>
- // CHECK: %[[RES:.*]] = llvm.bitcast %[[CONV]] : vector<8xi32> to vector<16xbf16>
+ // CHECK: %[[CONV:.*]] = llvm.call spir_funccc @_Z38__builtin_spirv_ConvertE2M1ToBF16INTELDv16_i(%[[ARG0]])
+ // CHECK-SAME: : (vector<8xi8>) -> vector<16xi16>
+ // CHECK: %[[RES:.*]] = llvm.bitcast %[[CONV]] : vector<16xi16> to vector<16xbf16>
%dst = xevm.extf %src { src_etype = e2m1, dst_etype = bf16 } : (vector<8xi8>) -> vector<16xbf16>
llvm.return %dst : vector<16xbf16>
}
More information about the Mlir-commits
mailing list