[clang] [llvm] [X86] Add ACE v1 (AI Compute Extensions) support (PR #208408)
via cfe-commits
cfe-commits at lists.llvm.org
Thu Jul 9 02:53:05 PDT 2026
llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-llvm-ir
Author: Ganesh (ganeshgit)
<details>
<summary>Changes</summary>
ACE v1 introduces new tile-based matrix operations with fixed 16x64 tile dimensions (Palette 2), unlike AMX's configurable dimensions.
Key features:
- New __acetile type with fixed 16x64 dimensions
- __tile_ace_* intrinsics for ACE tile operations
- Outer product instructions: TOP4BUUD, TOP4BUSD, TOP4BSSD, TOP4BSUD, TOP2BF16PS
- Mixed precision FP8 instructions: TOP4MX variants
- Tile movement: TILEMOVROW, TILEMOVCOL for ZMM<->tile transfers
- BSR (Block Scale Register) operations for scaling factors
- ACE tile spill/reload using TILEMOVROW + VMOVUPS row-by-row (ACE doesn't have TILELOADD/TILESTORED instructions)
Backend changes:
- X86LowerTileCopy handles ACE tile copies
- X86PreTileConfig/X86FastPreTileConfig for ACE register allocation
- Palette 2 tile configuration support
Reused from AMX:
- The x86_amx type
- AMX-AVX512f definitions and TILE instruction definitions
- Passes for lowering and handling the x86_amx type
- Wrappers internally use rows and cols to use amx definitions
This PR builds on PR #<!-- -->206888 and should be reviewed after/with it.
---
Patch is 228.20 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/208408.diff
58 Files Affected:
- (modified) clang/include/clang/Basic/BuiltinsX86_64.td (+68-3)
- (modified) clang/include/clang/Options/Options.td (+2)
- (modified) clang/lib/Basic/Targets/X86.cpp (+6)
- (modified) clang/lib/Basic/Targets/X86.h (+1)
- (modified) clang/lib/Headers/CMakeLists.txt (+1)
- (added) clang/lib/Headers/acev1intrin.h (+766)
- (modified) clang/lib/Headers/cpuid.h (+2)
- (modified) clang/lib/Headers/immintrin.h (+2)
- (modified) clang/lib/Sema/SemaX86.cpp (+5)
- (added) clang/test/CodeGen/X86/ace-api.c (+145)
- (added) clang/test/CodeGen/X86/ace-builtins.c (+119)
- (added) clang/test/CodeGen/X86/ace-inline-asm.c (+85)
- (added) clang/test/CodeGen/X86/acev1.c (+38)
- (added) clang/test/CodeGen/X86/acev1_api.c (+138)
- (added) clang/test/CodeGen/X86/acev1_bsr.c (+63)
- (added) clang/test/CodeGen/X86/acev1_errors.c (+17)
- (added) clang/test/CodeGen/X86/acev1_inline_asm.c (+76)
- (added) clang/test/CodeGen/X86/acev1_mxfp8.c (+51)
- (added) clang/test/CodeGen/X86/acev1_tile_movement.c (+113)
- (modified) clang/test/CodeGen/attr-target-x86.c (+2-2)
- (modified) clang/test/Preprocessor/x86_target_features.c (+7)
- (modified) llvm/include/llvm/IR/IntrinsicsX86.td (+198)
- (modified) llvm/include/llvm/TargetParser/X86TargetParser.def (+1)
- (modified) llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp (+50)
- (modified) llvm/lib/Target/X86/MCTargetDesc/X86ATTInstPrinter.cpp (+90-2)
- (modified) llvm/lib/Target/X86/MCTargetDesc/X86ATTInstPrinter.h (+1)
- (modified) llvm/lib/Target/X86/MCTargetDesc/X86IntelInstPrinter.cpp (+87-1)
- (modified) llvm/lib/Target/X86/MCTargetDesc/X86IntelInstPrinter.h (+1)
- (modified) llvm/lib/Target/X86/X86.td (+3)
- (modified) llvm/lib/Target/X86/X86ExpandPseudo.cpp (+89)
- (modified) llvm/lib/Target/X86/X86FastPreTileConfig.cpp (+80-25)
- (modified) llvm/lib/Target/X86/X86FastTileConfig.cpp (+13-4)
- (modified) llvm/lib/Target/X86/X86FrameLowering.cpp (+3-2)
- (modified) llvm/lib/Target/X86/X86ISelDAGToDAG.cpp (+19)
- (modified) llvm/lib/Target/X86/X86ISelLowering.cpp (+282-8)
- (added) llvm/lib/Target/X86/X86InstrACE.td (+417)
- (modified) llvm/lib/Target/X86/X86InstrAMX.td (+44-23)
- (modified) llvm/lib/Target/X86/X86InstrInfo.cpp (+7-1)
- (modified) llvm/lib/Target/X86/X86InstrInfo.td (+3)
- (modified) llvm/lib/Target/X86/X86InstrPredicates.td (+3)
- (modified) llvm/lib/Target/X86/X86LowerAMXType.cpp (+143-2)
- (modified) llvm/lib/Target/X86/X86LowerTileCopy.cpp (+152-48)
- (modified) llvm/lib/Target/X86/X86MachineFunctionInfo.cpp (+3-1)
- (modified) llvm/lib/Target/X86/X86MachineFunctionInfo.h (+22)
- (modified) llvm/lib/Target/X86/X86PreTileConfig.cpp (+7-3)
- (modified) llvm/lib/Target/X86/X86RegisterInfo.cpp (+15-2)
- (modified) llvm/lib/Target/X86/X86RegisterInfo.td (+9)
- (modified) llvm/lib/Target/X86/X86TileConfig.cpp (+8-2)
- (modified) llvm/lib/TargetParser/X86TargetParser.cpp (+1)
- (added) llvm/test/CodeGen/X86/ACE/ace-intrinsics.ll (+244)
- (added) llvm/test/CodeGen/X86/ACE/acev1-greedy-ra.ll (+99)
- (added) llvm/test/CodeGen/X86/ACE/acev1-outer-product.ll (+204)
- (added) llvm/test/CodeGen/X86/ACE/acev1-tile-basic.ll (+94)
- (modified) llvm/test/CodeGen/X86/ipra-reg-usage.ll (+2-2)
- (added) llvm/test/MC/X86/ACE/ace-att.s (+109)
- (added) llvm/test/MC/X86/ACE/ace-error.s (+39)
- (added) llvm/test/MC/X86/ACE/ace-intel.s (+109)
- (modified) llvm/test/TableGen/x86-fold-tables.inc (+5)
``````````diff
diff --git a/clang/include/clang/Basic/BuiltinsX86_64.td b/clang/include/clang/Basic/BuiltinsX86_64.td
index 0a98b3049df34..6f34256139144 100644
--- a/clang/include/clang/Basic/BuiltinsX86_64.td
+++ b/clang/include/clang/Basic/BuiltinsX86_64.td
@@ -223,6 +223,9 @@ let Features = "amx-int8", Attributes = [NoThrow] in {
let Features = "amx-tile", Attributes = [NoThrow] in {
def tilestored64_internal : X86Builtin<"void(unsigned short, unsigned short, void *, size_t, _Vector<256, int>)">;
+}
+
+let Features = "amx-tile|acev1", Attributes = [NoThrow] in {
def tilezero_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short)">;
}
@@ -239,7 +242,7 @@ let Features = "amx-complex", Attributes = [NoThrow] in {
def tcmmrlfp16ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Vector<256, int>, _Vector<256, int>, _Vector<256, int>)">;
}
-let Features = "amx-avx512,avx10.2", Attributes = [NoThrow, RequiredVectorWidth<512>] in {
+let Features = "amx-avx512,avx10.2|acev1", Attributes = [NoThrow, RequiredVectorWidth<512>] in {
def tcvtrowd2ps_internal : X86Builtin<"_Vector<16, float>(unsigned short, unsigned short, _Vector<256, int>, unsigned int)">;
def tcvtrowps2bf16h_internal : X86Builtin<"_Vector<32, __bf16>(unsigned short, unsigned short, _Vector<256, int>, unsigned int)">;
def tcvtrowps2bf16l_internal : X86Builtin<"_Vector<32, __bf16>(unsigned short, unsigned short, _Vector<256, int>, unsigned int)">;
@@ -255,7 +258,7 @@ let Features = "amx-fp8", Attributes = [NoThrow] in {
def tdphf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Vector<256, int>, _Vector<256, int>, _Vector<256, int>)">;
}
-let Features = "amx-tile", Attributes = [NoThrow] in {
+let Features = "amx-tile|acev1", Attributes = [NoThrow] in {
def tile_loadconfig : X86Builtin<"void(void const *)">;
def tile_storeconfig : X86Builtin<"void(void const *)">;
def tilerelease : X86Builtin<"void()">;
@@ -293,7 +296,7 @@ let Features = "amx-complex", Attributes = [NoThrow] in {
def tcmmrlfp16ps : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">;
}
-let Features = "amx-avx512,avx10.2", Attributes = [NoThrow, RequiredVectorWidth<512>] in {
+let Features = "amx-avx512,avx10.2|acev1", Attributes = [NoThrow, RequiredVectorWidth<512>] in {
def tcvtrowd2ps : X86Builtin<"_Vector<16, float>(_Constant unsigned char, unsigned int)">;
def tcvtrowps2bf16h : X86Builtin<"_Vector<32, __bf16>(_Constant unsigned char, unsigned int)">;
def tcvtrowps2bf16l : X86Builtin<"_Vector<32, __bf16>(_Constant unsigned char, unsigned int)">;
@@ -389,3 +392,65 @@ let Features = "movrs,avx10.2", Attributes = [NoThrow, RequiredVectorWidth<256>]
let Features = "movrs,avx10.2", Attributes = [NoThrow, RequiredVectorWidth<512>] in {
def vmovrsw512 : X86Builtin<"_Vector<32, short>(_Vector<32, short const *>)">;
}
+
+// ACE (AI Compute Extensions) Builtins
+// Tile movement - single builtin handles both immediate and register index forms
+// The lowering code checks if index is constant and selects appropriate instruction
+let Features = "acev1", Attributes = [NoThrow] in {
+ def tilesetcol : X86Builtin<"void(_Constant unsigned char, _Vector<16, int>, unsigned int)">;
+ // ACE tilemovrow writes TO tile (AMX tilemovrow reads FROM tile)
+ def tilesetrow : X86Builtin<"void(_Constant unsigned char, _Vector<16, int>, unsigned int)">;
+}
+
+// Tile movement internal (for IR-level operations)
+let Features = "acev1", Attributes = [NoThrow] in {
+ def tilesetcol_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, _Vector<16, int>, unsigned int)">;
+ // ACE tilemovrow_to_tile writes TO tile (AMX tilemovrow reads FROM tile)
+ def tilesetrow_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, _Vector<16, int>, unsigned int)">;
+}
+
+// BSR management
+let Features = "acev1", Attributes = [NoThrow] in {
+ def bsrinit : X86Builtin<"void()">;
+ def bsrmovf : X86Builtin<"void(_Vector<16, int>, _Vector<16, int>)">;
+ def bsrmovh_set : X86Builtin<"void(_Vector<16, int>)">;
+ def bsrmovh_get : X86Builtin<"_Vector<16, int>()">;
+ def bsrmovl_set : X86Builtin<"void(_Vector<16, int>)">;
+ def bsrmovl_get : X86Builtin<"_Vector<16, int>()">;
+}
+
+// Outer products - user-facing (immediate tile IDs)
+let Features = "acev1", Attributes = [NoThrow] in {
+ def top2bf16ps : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">;
+ def top4buud : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">;
+ def top4busd : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">;
+ def top4bssd : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">;
+ def top4bsud : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">;
+}
+
+// Mixed precision outer products - user-facing
+let Features = "acev1", Attributes = [NoThrow] in {
+ def top4mxhf8ps : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">;
+ def top4mxbhf8ps : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">;
+ def top4mxhbf8ps : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">;
+ def top4mxbf8ps : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">;
+ def top4mxbssps : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">;
+}
+
+// Outer products - internal (for IR-level operations with tile values)
+let Features = "acev1", Attributes = [NoThrow] in {
+ def top2bf16ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Vector<256, int>, _Vector<32, __bf16>, _Vector<32, __bf16>)">;
+ def top4buud_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Vector<256, int>, _Vector<64, signed char>, _Vector<64, signed char>)">;
+ def top4busd_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Vector<256, int>, _Vector<64, signed char>, _Vector<64, signed char>)">;
+ def top4bssd_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Vector<256, int>, _Vector<64, signed char>, _Vector<64, signed char>)">;
+ def top4bsud_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Vector<256, int>, _Vector<64, signed char>, _Vector<64, signed char>)">;
+}
+
+// Mixed precision internal
+let Features = "acev1", Attributes = [NoThrow] in {
+ def top4mxhf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">;
+ def top4mxbhf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">;
+ def top4mxhbf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">;
+ def top4mxbf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">;
+ def top4mxbssps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">;
+}
diff --git a/clang/include/clang/Options/Options.td b/clang/include/clang/Options/Options.td
index 59bfd621a5994..c6ab7b6468802 100644
--- a/clang/include/clang/Options/Options.td
+++ b/clang/include/clang/Options/Options.td
@@ -7178,6 +7178,8 @@ def mamx_tile : Flag<["-"], "mamx-tile">, Group<m_x86_Features_Group>;
def mno_amx_tile : Flag<["-"], "mno-amx-tile">, Group<m_x86_Features_Group>;
def mamx_movrs: Flag<["-"], "mamx-movrs">, Group<m_x86_Features_Group>;
def mno_amx_movrs: Flag<["-"], "mno-amx-movrs">, Group<m_x86_Features_Group>;
+def macev1 : Flag<["-"], "macev1">, Group<m_x86_Features_Group>;
+def mno_acev1 : Flag<["-"], "mno-acev1">, Group<m_x86_Features_Group>;
def mcmpccxadd : Flag<["-"], "mcmpccxadd">, Group<m_x86_Features_Group>;
def mno_cmpccxadd : Flag<["-"], "mno-cmpccxadd">, Group<m_x86_Features_Group>;
def msse : Flag<["-"], "msse">, Group<m_x86_Features_Group>;
diff --git a/clang/lib/Basic/Targets/X86.cpp b/clang/lib/Basic/Targets/X86.cpp
index f77b2d4e0815d..c694458b1d1ff 100644
--- a/clang/lib/Basic/Targets/X86.cpp
+++ b/clang/lib/Basic/Targets/X86.cpp
@@ -400,6 +400,8 @@ bool X86TargetInfo::handleTargetFeatures(std::vector<std::string> &Features,
HasAMXMOVRS = true;
} else if (Feature == "+amx-avx512") {
HasAMXAVX512 = true;
+ } else if (Feature == "+acev1") {
+ HasACEV1 = true;
} else if (Feature == "+cmpccxadd") {
HasCMPCCXADD = true;
} else if (Feature == "+raoint") {
@@ -948,6 +950,8 @@ void X86TargetInfo::getTargetDefines(const LangOptions &Opts,
Builder.defineMacro("__AMX_MOVRS__");
if (HasAMXAVX512)
Builder.defineMacro("__AMX_AVX512__");
+ if (HasACEV1)
+ Builder.defineMacro("__ACEV1__");
if (HasCMPCCXADD)
Builder.defineMacro("__CMPCCXADD__");
if (HasRAOINT)
@@ -1087,6 +1091,7 @@ bool X86TargetInfo::isValidFeatureName(StringRef Name) const {
.Case("amx-int8", true)
.Case("amx-movrs", true)
.Case("amx-tile", true)
+ .Case("acev1", true)
.Case("avx", true)
.Case("avx10.1", true)
.Case("avx10.2", true)
@@ -1208,6 +1213,7 @@ bool X86TargetInfo::hasFeature(StringRef Feature) const {
.Case("amx-int8", HasAMXINT8)
.Case("amx-movrs", HasAMXMOVRS)
.Case("amx-tile", HasAMXTILE)
+ .Case("acev1", HasACEV1)
.Case("avx", SSELevel >= AVX)
.Case("avx10.1", HasAVX10_1)
.Case("avx10.2", HasAVX10_2)
diff --git a/clang/lib/Basic/Targets/X86.h b/clang/lib/Basic/Targets/X86.h
index e305d9017d897..c787f8e77b185 100644
--- a/clang/lib/Basic/Targets/X86.h
+++ b/clang/lib/Basic/Targets/X86.h
@@ -164,6 +164,7 @@ class LLVM_LIBRARY_VISIBILITY X86TargetInfo : public TargetInfo {
bool HasAMXFP8 = false;
bool HasAMXMOVRS = false;
bool HasAMXAVX512 = false;
+ bool HasACEV1 = false;
bool HasSERIALIZE = false;
bool HasTSXLDTRK = false;
bool HasUSERMSR = false;
diff --git a/clang/lib/Headers/CMakeLists.txt b/clang/lib/Headers/CMakeLists.txt
index 4bbfb9ba1662a..a64952e4d87e3 100644
--- a/clang/lib/Headers/CMakeLists.txt
+++ b/clang/lib/Headers/CMakeLists.txt
@@ -167,6 +167,7 @@ set(webassembly_files
set(x86_files
# Intrinsics
+ acev1intrin.h
adcintrin.h
adxintrin.h
ammintrin.h
diff --git a/clang/lib/Headers/acev1intrin.h b/clang/lib/Headers/acev1intrin.h
new file mode 100644
index 0000000000000..0491c7f0b383d
--- /dev/null
+++ b/clang/lib/Headers/acev1intrin.h
@@ -0,0 +1,766 @@
+/*===-------------- acev1intrin.h - ACEV1 intrinsics -*- C/C++ -*------------===
+ *
+ * Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+ * See https://llvm.org/LICENSE.txt for license information.
+ * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+ *
+ *===------------------------------------------------------------------------===
+ */
+
+#ifndef __IMMINTRIN_H
+#error "Never use <acev1intrin.h> directly; include <immintrin.h> instead."
+#endif /* __IMMINTRIN_H */
+
+#ifndef __ACEV1INTRIN_H
+#define __ACEV1INTRIN_H
+#ifdef __x86_64__
+
+/* Define the default attributes for the functions in this file. */
+#define __DEFAULT_FN_ATTRS_ACE \
+ __attribute__((__always_inline__, __nodebug__, __target__("acev1")))
+
+/// Load tile configuration from a 64-byte memory location. For ACE
+/// (Palette 2), the palette_id byte must be 2. Unlike AMX (Palette 1),
+/// ACE tiles have fixed dimensions of 16 rows × 64 bytes, so per-tile
+/// row/column configuration bytes are ignored. If palette_id is zero,
+/// tiles return to init state and are zeroed. Invalid configurations
+/// result in #GP fault.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> LDTILECFG </c> instruction.
+///
+/// \param __config
+/// A pointer to 64-byte tile configuration (use __ace_tile_config).
+static __inline__ void __DEFAULT_FN_ATTRS_ACE
+_tile_ace_loadconfig(const void *__config) {
+ __builtin_ia32_tile_loadconfig(__config);
+}
+
+/// Store the current tile configuration to a 64-byte memory location.
+/// For ACE (Palette 2), the stored palette_id will be 2 and per-tile
+/// configuration bytes reflect the fixed 16×64 dimensions. If tiles
+/// are not configured, all zeroes are stored.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> STTILECFG </c> instruction.
+///
+/// \param __config
+/// A pointer to 64-byte tile configuration buffer.
+static __inline__ void __DEFAULT_FN_ATTRS_ACE
+_tile_ace_storeconfig(void *__config) {
+ __builtin_ia32_tile_storeconfig(__config);
+}
+
+/// Release the tile configuration to return to init state, releasing
+/// all tile storage. After this, tiles must be reconfigured with
+/// _tile_ace_loadconfig before use.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> TILERELEASE </c> instruction.
+static __inline__ void __DEFAULT_FN_ATTRS_ACE _tile_ace_release(void) {
+ __builtin_ia32_tilerelease();
+}
+
+/// Zero the ACE tile specified by "tile". Sets all 1024 bytes
+/// (16 rows × 64 bytes) of the tile register to zero.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> TILEZERO </c> instruction.
+///
+/// \param tile
+/// Destination tile register ID (0-7).
+#define _tile_ace_zero(tile) __builtin_ia32_tilezero((tile))
+
+/// Move a 64-byte ZMM vector to a tile column. The 16 doublewords from
+/// the ZMM are written as a vertical column in the tile at the specified
+/// index.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> TILEMOVCOL </c> instruction.
+///
+/// \param dst
+/// Destination tile register ID (0-7).
+/// \param src
+/// Source ZMM register ID (0-31).
+/// \param idx
+/// Column index (0-15). Immediate or register form selected automatically.
+#define _tile_setcol(dst, src, idx) \
+ __builtin_ia32_tilesetcol((dst), (src), (idx))
+
+/// Move a 64-byte ZMM vector to a tile row. The ZMM contents are written
+/// as a horizontal row in the tile at the specified index.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> TILEMOVROW </c> instruction.
+///
+/// \param dst
+/// Destination tile register ID (0-7).
+/// \param src
+/// Source ZMM register ID (0-31).
+/// \param idx
+/// Row index (0-15). Immediate or register form selected automatically.
+#define _tile_setrow(dst, src, idx) \
+ __builtin_ia32_tilesetrow((dst), (src), (idx))
+
+/// Initialize the Block Scale Register (BSR) to zero. The BSR holds
+/// 32 scaling factors used by mixed-precision outer product instructions
+/// (TOP4MX* variants). Must be called before using BSR-scaled operations.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> BSRINIT </c> instruction.
+static __inline__ void __DEFAULT_FN_ATTRS_ACE _bsr0_init(void) {
+ __builtin_ia32_bsrinit();
+}
+
+/// Load the full BSR (32 scale factors) from two ZMM registers. The low
+/// half (16 factors) comes from __src1, high half from __src2. Each
+/// doubleword contains one scale factor.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> BSRMOVF </c> instruction.
+///
+/// \param __src1
+/// ZMM with scale factors for BSR low half (factors 0-15).
+/// \param __src2
+/// ZMM with scale factors for BSR high half (factors 16-31).
+static __inline__ void __DEFAULT_FN_ATTRS_ACE _bsr0_movf(__m512i __src1,
+ __m512i __src2) {
+ __builtin_ia32_bsrmovf((__v16si)__src1, (__v16si)__src2);
+}
+
+/// Load the high half of BSR (scale factors 16-31) from a ZMM register.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> BSRMOVH </c> instruction.
+///
+/// \param __src
+/// ZMM with 16 scale factors to write to BSR high half.
+static __inline__ void __DEFAULT_FN_ATTRS_ACE _bsr0_movh_set(__m512i __src) {
+ __builtin_ia32_bsrmovh_set((__v16si)__src);
+}
+
+/// Read the high half of BSR (scale factors 16-31) to a ZMM register.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> BSRMOVH </c> instruction.
+///
+/// \returns
+/// ZMM containing 16 scale factors from BSR high half.
+static __inline__ __m512i __DEFAULT_FN_ATTRS_ACE _bsr0_movh_get(void) {
+ return (__m512i)__builtin_ia32_bsrmovh_get();
+}
+
+/// Load the low half of BSR (scale factors 0-15) from a ZMM register.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> BSRMOVL </c> instruction.
+///
+/// \param __src
+/// ZMM with 16 scale factors to write to BSR low half.
+static __inline__ void __DEFAULT_FN_ATTRS_ACE _bsr0_movl_set(__m512i __src) {
+ __builtin_ia32_bsrmovl_set((__v16si)__src);
+}
+
+/// Read the low half of BSR (scale factors 0-15) to a ZMM register.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> BSRMOVL </c> instruction.
+///
+/// \returns
+/// ZMM containing 16 scale factors from BSR low half.
+static __inline__ __m512i __DEFAULT_FN_ATTRS_ACE _bsr0_movl_get(void) {
+ return (__m512i)__builtin_ia32_bsrmovl_get();
+}
+
+/// Compute 2-way outer product of BF16 pairs, accumulating to FP32.
+/// Each BF16 pair from src1 and src2 produces two FP32 products that
+/// are accumulated into the destination tile.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> TOP2BF16PS </c> instruction.
+///
+/// \param dst
+/// Destination/accumulator tile register ID (0-7).
+/// \param src1
+/// First source ZMM register ID (0-31) containing 32 BF16 values.
+/// \param src2
+/// Second source ZMM register ID (0-31) containing 32 BF16 values.
+#define _tile_top2bf16ps(dst, src1, src2) \
+ __builtin_ia32_top2bf16ps((dst), (src1), (src2))
+
+/// Compute 4-way outer product of unsigned×unsigned bytes to INT32.
+/// Each group of 4 unsigned byte pairs from src1 and src2 produces
+/// 4 products accumulated into the destination tile as 32-bit integers.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> TOP4BUUD </c> instruction.
+///
+/// \param dst
+/// Destination/accumulator tile register ID (0-7).
+/// \param src1
+/// First source ZMM register ID (0-31) containing 64 unsigned bytes.
+/// \param src2
+/// Second source ZMM register ID (0-31) containing 64 unsigned bytes.
+#define _tile_top4buud(dst, src1, src2) \
+ __builtin_ia32_top4buud((dst), (src1), (src2))
+
+/// Compute 4-way outer product of unsigned×signed bytes to INT32.
+/// Each group of 4 byte pairs (unsigned from src1, signed from src2)
+/// produces 4 products accumulated into the destination tile.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> TOP4BUSD </c> instruction.
+///
+/// \param dst
+/// Destination/accumulator tile register ID (0-7).
+/// \param src1
+/// First source ZMM register ID (0-31) containing 64 unsigned bytes.
+/// \param src2
+/// Second source ZMM register ID (0-31) containing 64 signed bytes.
+#define _tile_top4busd(dst, src1, src2) \
+ __builtin_ia32_top4busd((dst), (src1), (src2))
+
+/// Compute 4-way outer product of signed×signed bytes to INT32.
+/// Each group of 4 signed byte pairs produces 4 products accumulated
+/// into the destination tile as 32-bit integers.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsi...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/208408
More information about the cfe-commits
mailing list