[llvm-branch-commits] [clang] [llvm] [X86][ACE] Add x86_bsr type and Block Scale Register support (PR #208706)
via llvm-branch-commits
llvm-branch-commits at lists.llvm.org
Fri Jul 10 05:01:17 PDT 2026
https://github.com/ganeshgit created https://github.com/llvm/llvm-project/pull/208706
This patch adds comprehensive support for the Block Scale Register (BSR), a 1024-bit architectural register used by ACE (AI Compute Extensions) for storing scale factors in mixed-precision matrix operations.
- Add x86_bsr as an IR type and update similar to x86_amx
- Make x86_bsr explicit in ACE operations.
Intrinsics Infrastructure:
- Add x86_bsr intrinsics in IntrinsicsX86.td:
- bsrmovf: Store both halves to hardware BSR
- bsrmovh_set/bsrmovl_set: Store individual halves
- bsrmovh_get/bsrmovl_get: Read individual halves
- bsr_create: Create x86_bsr from two 512-bit vectors
- Conversion intrinsics between x86_bsr and vector types
X86 Backend:
- Add X86LowerBSRType pass to lower x86_bsr SSA values to implicit hardware register operations (similar to X86LowerAMXType)
- Handle BSR in register allocation and frame lowering
- Reserve BSR during register allocation to prevent conflicts
Header and API:
- Add __bsr struct type in acev1intrin.h bundling lo/hi 512-bit halves
- Add struct-based API: __bsr_make, __bsr_load, __bsr_store, __bsr_get_lo, __bsr_get_hi, __bsr_set_lo, __bsr_set_hi
This PR builds on PR https://github.com/llvm/llvm-project/pull/208408 and should be reviewed after/with it.
Co-authored-by: Umesh Kalvakuntla
>From 2ad788bb7549824b8640c6808d05087362f668ec Mon Sep 17 00:00:00 2001
From: Ganesh Gopalasubramanian <Ganesh.Gopalasubramanian at amd.com>
Date: Thu, 9 Jul 2026 21:59:53 +0530
Subject: [PATCH] [X86][ACE] Add x86_bsr type and Block Scale Register support
This patch adds comprehensive support for the Block Scale Register (BSR),
a 1024-bit architectural register used by ACE (AI Compute Extensions) for
storing scale factors in mixed-precision matrix operations.
- Add x86_bsr as an IR type and update similar to x86_amx
Intrinsics Infrastructure:
- Add x86_bsr intrinsics in IntrinsicsX86.td:
- bsrmovf: Store both halves to hardware BSR
- bsrmovh_set/bsrmovl_set: Store individual halves
- bsrmovh_get/bsrmovl_get: Read individual halves
- bsr_create: Create x86_bsr from two 512-bit vectors
- Conversion intrinsics between x86_bsr and vector types
X86 Backend:
- Add X86LowerBSRType pass to lower x86_bsr SSA values to implicit
hardware register operations (similar to X86LowerAMXType)
- Handle BSR in register allocation and frame lowering
- Reserve BSR during register allocation to prevent conflicts
Header and API:
- Add __bsr struct type in acev1intrin.h bundling lo/hi 512-bit halves
- Add struct-based API: __bsr_make, __bsr_load, __bsr_store,
__bsr_get_lo, __bsr_get_hi, __bsr_set_lo, __bsr_set_hi
---
clang/include/clang/Basic/BuiltinsX86_64.td | 19 +-
clang/lib/CodeGen/CGBuiltin.cpp | 26 +-
clang/lib/Headers/acev1intrin.h | 244 +++++++++++-
clang/test/CodeGen/X86/ace-api.c | 87 ++++
llvm/include/llvm-c/Core.h | 10 +
llvm/include/llvm/Analysis/IR2Vec.h | 1 +
llvm/include/llvm/Bitcode/LLVMBitCodes.h | 2 +
llvm/include/llvm/CodeGen/ValueTypes.td | 1 +
llvm/include/llvm/IR/DataLayout.h | 2 +
llvm/include/llvm/IR/Intrinsics.h | 1 +
llvm/include/llvm/IR/Intrinsics.td | 3 +
llvm/include/llvm/IR/IntrinsicsX86.td | 114 +++++-
llvm/include/llvm/IR/Type.h | 10 +-
llvm/include/llvm/SandboxIR/Type.h | 3 +
llvm/lib/Analysis/ConstantFolding.cpp | 8 +-
llvm/lib/AsmParser/LLLexer.cpp | 1 +
llvm/lib/Bitcode/Reader/BitcodeReader.cpp | 3 +
llvm/lib/Bitcode/Writer/BitcodeWriter.cpp | 3 +
llvm/lib/CodeGen/ValueTypes.cpp | 5 +
llvm/lib/IR/AsmWriter.cpp | 3 +
llvm/lib/IR/ConstantFold.cpp | 2 +-
llvm/lib/IR/Core.cpp | 8 +
llvm/lib/IR/DataLayout.cpp | 2 +
llvm/lib/IR/Intrinsics.cpp | 10 +
llvm/lib/IR/LLVMContextImpl.cpp | 7 +-
llvm/lib/IR/LLVMContextImpl.h | 2 +-
llvm/lib/IR/Type.cpp | 18 +-
llvm/lib/IR/TypedPointerType.cpp | 2 +-
llvm/lib/IR/Verifier.cpp | 11 +-
.../DirectX/DXILWriter/DXILBitcodeWriter.cpp | 1 +
.../Hexagon/HexagonTargetObjectFile.cpp | 1 +
llvm/lib/Target/X86/CMakeLists.txt | 1 +
llvm/lib/Target/X86/X86.h | 12 +
llvm/lib/Target/X86/X86ISelLowering.cpp | 4 +
llvm/lib/Target/X86/X86LowerAMXType.cpp | 9 +-
llvm/lib/Target/X86/X86LowerBSRType.cpp | 373 ++++++++++++++++++
llvm/lib/Target/X86/X86RegisterInfo.cpp | 3 +
llvm/lib/Target/X86/X86RegisterInfo.td | 2 +-
llvm/lib/Target/X86/X86TargetMachine.cpp | 4 +-
.../InstCombine/InstCombineCasts.cpp | 2 +-
.../InstCombineLoadStoreAlloca.cpp | 11 +-
llvm/test/Assembler/x86_bsr.ll | 40 ++
llvm/test/CodeGen/X86/ACE/ace-bsr-ordering.ll | 32 ++
.../X86/ACE/ace-internal-intrinsics.ll | 176 +++++++++
llvm/test/CodeGen/X86/O0-pipeline.ll | 1 +
llvm/test/CodeGen/X86/opt-pipeline.ll | 1 +
.../Instrumentor/alloca_and_function.ll | 10 +-
.../test/Instrumentation/Instrumentor/cast.ll | 60 +--
.../Instrumentor/cast_crash.ll | 4 +-
.../Instrumentation/Instrumentor/compare.ll | 22 +-
.../Instrumentor/load_store.ll | 72 ++--
.../Instrumentor/load_store_args.ll | 72 ++--
.../Instrumentor/load_store_noreplace.ll | 72 ++--
.../Instrumentor/module_and_globals.ll | 20 +-
.../Instrumentation/Instrumentor/numeric.ll | 64 +--
.../Instrumentor/numeric_subtypeid.ll | 2 +-
llvm/test/TableGen/CPtrWildcard.td | 4 +-
llvm/test/TableGen/x86-fold-tables.inc | 5 -
llvm/test/Verifier/x86_bsr.ll | 6 +
.../TableGen/Basic/CodeGenIntrinsics.cpp | 4 +-
llvm/utils/TableGen/X86ManualFoldTables.def | 9 +
61 files changed, 1439 insertions(+), 268 deletions(-)
create mode 100644 llvm/lib/Target/X86/X86LowerBSRType.cpp
create mode 100644 llvm/test/Assembler/x86_bsr.ll
create mode 100644 llvm/test/CodeGen/X86/ACE/ace-bsr-ordering.ll
create mode 100644 llvm/test/CodeGen/X86/ACE/ace-internal-intrinsics.ll
create mode 100644 llvm/test/Verifier/x86_bsr.ll
diff --git a/clang/include/clang/Basic/BuiltinsX86_64.td b/clang/include/clang/Basic/BuiltinsX86_64.td
index 6f34256139144..983d4c2764862 100644
--- a/clang/include/clang/Basic/BuiltinsX86_64.td
+++ b/clang/include/clang/Basic/BuiltinsX86_64.td
@@ -446,11 +446,18 @@ let Features = "acev1", Attributes = [NoThrow] in {
def top4bsud_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Vector<256, int>, _Vector<64, signed char>, _Vector<64, signed char>)">;
}
-// Mixed precision internal
+// Mixed precision internal (last arg is BSR as v32i32, converted to x86_bsr by CGBuiltin)
let Features = "acev1", Attributes = [NoThrow] in {
- def top4mxhf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">;
- def top4mxbhf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">;
- def top4mxhbf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">;
- def top4mxbf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">;
- def top4mxbssps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">;
+ // BSR versions (8 args, with BSR) - arg 3 is immediate (i8)
+ def top4mxhf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>, _Vector<32, int>)">;
+ def top4mxbhf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>, _Vector<32, int>)">;
+ def top4mxhbf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>, _Vector<32, int>)">;
+ def top4mxbf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>, _Vector<32, int>)">;
+ def top4mxbssps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>, _Vector<32, int>)">;
+ // Non-BSR versions (7 args, no BSR) - used by __tile_ace_* macros in acev1intrin.h
+ def top4mxhf8ps_nobsr_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Constant unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">;
+ def top4mxbhf8ps_nobsr_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Constant unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">;
+ def top4mxhbf8ps_nobsr_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Constant unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">;
+ def top4mxbf8ps_nobsr_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Constant unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">;
+ def top4mxbssps_nobsr_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Constant unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">;
}
diff --git a/clang/lib/CodeGen/CGBuiltin.cpp b/clang/lib/CodeGen/CGBuiltin.cpp
index 475bfec6199fc..7ac5f34c09ee2 100644
--- a/clang/lib/CodeGen/CGBuiltin.cpp
+++ b/clang/lib/CodeGen/CGBuiltin.cpp
@@ -7047,11 +7047,17 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
}
}
- // Cast vector type (e.g., v256i32) to x86_amx, this only happen
- // in amx intrinsics.
+ // Cast vector type to x86_amx (v256i32) or x86_bsr (v32i32).
+ // Use CreateIntrinsicWithoutFolding to avoid constant folding issues
+ // with these special types that cannot have constant values.
if (PTy->isX86_AMXTy())
- ArgValue = Builder.CreateIntrinsic(Intrinsic::x86_cast_vector_to_tile,
- {ArgValue->getType()}, {ArgValue});
+ ArgValue = Builder.CreateIntrinsicWithoutFolding(
+ Intrinsic::x86_cast_vector_to_tile, {ArgValue->getType()},
+ {ArgValue});
+ else if (PTy->isX86_BSRTy())
+ ArgValue = Builder.CreateIntrinsicWithoutFolding(
+ Intrinsic::x86_cast_vector_to_bsr, {ArgValue->getType()},
+ {ArgValue});
else
ArgValue = Builder.CreateBitCast(ArgValue, PTy);
}
@@ -7076,11 +7082,15 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
}
}
- // Cast x86_amx to vector type (e.g., v256i32), this only happen
- // in amx intrinsics.
+ // Cast x86_amx (v256i32) or x86_bsr (v32i32) to vector type.
+ // Use CreateIntrinsicWithoutFolding to avoid constant folding issues
+ // with these special types that cannot have constant values.
if (V->getType()->isX86_AMXTy())
- V = Builder.CreateIntrinsic(Intrinsic::x86_cast_tile_to_vector, {RetTy},
- {V});
+ V = Builder.CreateIntrinsicWithoutFolding(
+ Intrinsic::x86_cast_tile_to_vector, {RetTy}, {V});
+ else if (V->getType()->isX86_BSRTy())
+ V = Builder.CreateIntrinsicWithoutFolding(
+ Intrinsic::x86_cast_bsr_to_vector, {RetTy}, {V});
else
V = Builder.CreateBitCast(V, RetTy);
}
diff --git a/clang/lib/Headers/acev1intrin.h b/clang/lib/Headers/acev1intrin.h
index 0491c7f0b383d..44cbd2295a307 100644
--- a/clang/lib/Headers/acev1intrin.h
+++ b/clang/lib/Headers/acev1intrin.h
@@ -19,6 +19,144 @@
#define __DEFAULT_FN_ATTRS_ACE \
__attribute__((__always_inline__, __nodebug__, __target__("acev1")))
+/// Vector type for combining two 512-bit halves into 1024-bit BSR value.
+typedef int __v32si __attribute__((__vector_size__(128)));
+
+/// Combine two 512-bit vector halves into a single 1024-bit vector.
+/// This is an internal helper for BSR intrinsics.
+///
+/// \param __lo
+/// The low 512-bit half (B-scales, BSR bits [511:0]).
+/// \param __hi
+/// The high 512-bit half (A-scales, BSR bits [1023:512]).
+/// \returns A 1024-bit vector with __lo in elements [0:15] and __hi in [16:31].
+static __inline__ __v32si __DEFAULT_FN_ATTRS_ACE
+__bsr_combine_v32(__v16si __lo, __v16si __hi) {
+ return __builtin_shufflevector(__lo, __hi, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
+ 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
+ 23, 24, 25, 26, 27, 28, 29, 30, 31);
+}
+
+/// \struct __bsr
+/// \brief BSR (Block Scale Register) struct type.
+///
+/// The __bsr struct bundles the low and high 512-bit halves of the 1024-bit
+/// Block Scale Register into a single value type. This mirrors the __tile1024i
+/// pattern from AMX, but without shape metadata (BSR has fixed 1024-bit size).
+///
+/// BSR layout per ACE spec:
+/// - BSR[511:0] = B-scales (lo) - column/B-input scales
+/// - BSR[1023:512] = A-scales (hi) - row/A-input scales
+///
+/// Usage:
+/// \code
+/// __bsr scales = __bsr_make(lo_zmm, hi_zmm);
+/// __bsr_store(scales); // write to hardware BSR before compute
+/// \endcode
+typedef struct __bsr_str {
+ __m512i lo; ///< Low 512-bit half (B-scales, BSR bits [511:0])
+ __m512i hi; ///< High 512-bit half (A-scales, BSR bits [1023:512])
+} __bsr;
+
+/// Construct a BSR value from low and high 512-bit halves.
+///
+/// \headerfile <immintrin.h>
+///
+/// \param __lo
+/// The low 512-bit half (B-scales, BSR bits [511:0]).
+/// \param __hi
+/// The high 512-bit half (A-scales, BSR bits [1023:512]).
+/// \returns A __bsr struct containing both halves.
+static __inline__ __bsr __DEFAULT_FN_ATTRS_ACE __bsr_make(__m512i __lo,
+ __m512i __hi) {
+ __bsr __b;
+ __b.lo = __lo;
+ __b.hi = __hi;
+ return __b;
+}
+
+/// Extract the low 512-bit half (B-scales) from a BSR value.
+///
+/// \headerfile <immintrin.h>
+///
+/// \param __b
+/// The BSR value to extract from.
+/// \returns The low 512-bit half (B-scales, BSR bits [511:0]).
+static __inline__ __m512i __DEFAULT_FN_ATTRS_ACE __bsr_get_lo(__bsr __b) {
+ return __b.lo;
+}
+
+/// Extract the high 512-bit half (A-scales) from a BSR value.
+///
+/// \headerfile <immintrin.h>
+///
+/// \param __b
+/// The BSR value to extract from.
+/// \returns The high 512-bit half (A-scales, BSR bits [1023:512]).
+static __inline__ __m512i __DEFAULT_FN_ATTRS_ACE __bsr_get_hi(__bsr __b) {
+ return __b.hi;
+}
+
+/// Return a new BSR value with the low half (B-scales) replaced.
+///
+/// \headerfile <immintrin.h>
+///
+/// \param __b
+/// The original BSR value.
+/// \param __lo
+/// The new low 512-bit half (B-scales).
+/// \returns A new __bsr with the low half replaced.
+static __inline__ __bsr __DEFAULT_FN_ATTRS_ACE __bsr_set_lo(__bsr __b,
+ __m512i __lo) {
+ __b.lo = __lo;
+ return __b;
+}
+
+/// Return a new BSR value with the high half (A-scales) replaced.
+///
+/// \headerfile <immintrin.h>
+///
+/// \param __b
+/// The original BSR value.
+/// \param __hi
+/// The new high 512-bit half (A-scales).
+/// \returns A new __bsr with the high half replaced.
+static __inline__ __bsr __DEFAULT_FN_ATTRS_ACE __bsr_set_hi(__bsr __b,
+ __m512i __hi) {
+ __b.hi = __hi;
+ return __b;
+}
+
+/// Store a BSR value to the hardware Block Scale Register.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> BSRMOVF </c> instruction.
+///
+/// Per ACE spec: BSRMOVF writes A-scales (hi) to BSR[1023:512] and
+/// B-scales (lo) to BSR[511:0].
+///
+/// \param __b
+/// The BSR value to store to the hardware register.
+static __inline__ void __DEFAULT_FN_ATTRS_ACE __bsr_store(__bsr __b) {
+ __builtin_ia32_bsrmovf((__v16si)__b.hi, (__v16si)__b.lo);
+}
+
+/// Load the current hardware BSR state into a __bsr struct.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> BSRMOVH </c> and <c> BSRMOVL </c>
+/// instructions (read forms).
+///
+/// \returns A __bsr struct containing the current hardware BSR state.
+static __inline__ __bsr __DEFAULT_FN_ATTRS_ACE __bsr_load(void) {
+ __bsr __b;
+ __b.lo = (__m512i)__builtin_ia32_bsrmovl_get();
+ __b.hi = (__m512i)__builtin_ia32_bsrmovh_get();
+ return __b;
+}
+
/// Load tile configuration from a 64-byte memory location. For ACE
/// (Palette 2), the palette_id byte must be 2. Unlike AMX (Palette 1),
/// ACE tiles have fixed dimensions of 16 rows × 64 bytes, so per-tile
@@ -521,7 +659,7 @@ static __inline__ void __tile_ace_top2bf16ps(__acetile *dst, __m512bh src1,
/// \param imm
/// 8-bit immediate selecting BSR scale factors to apply.
#define __tile_ace_top4mxhf8ps(dst, src1, src2, imm) \
- (*(dst) = __builtin_ia32_top4mxhf8ps_internal( \
+ (*(dst) = __builtin_ia32_top4mxhf8ps_nobsr_internal( \
16, 64, 64, (imm), *(dst), (__v16si)(src1), (__v16si)(src2)))
/// Compute 4-way mixed precision outer product with BF8/HF8 format.
@@ -541,7 +679,7 @@ static __inline__ void __tile_ace_top2bf16ps(__acetile *dst, __m512bh src1,
/// \param imm
/// 8-bit immediate selecting BSR scale factors to apply.
#define __tile_ace_top4mxbhf8ps(dst, src1, src2, imm) \
- (*(dst) = __builtin_ia32_top4mxbhf8ps_internal( \
+ (*(dst) = __builtin_ia32_top4mxbhf8ps_nobsr_internal( \
16, 64, 64, (imm), *(dst), (__v16si)(src1), (__v16si)(src2)))
/// Compute 4-way mixed precision outer product with HF8/BF8 format.
@@ -561,7 +699,7 @@ static __inline__ void __tile_ace_top2bf16ps(__acetile *dst, __m512bh src1,
/// \param imm
/// 8-bit immediate selecting BSR scale factors to apply.
#define __tile_ace_top4mxhbf8ps(dst, src1, src2, imm) \
- (*(dst) = __builtin_ia32_top4mxhbf8ps_internal( \
+ (*(dst) = __builtin_ia32_top4mxhbf8ps_nobsr_internal( \
16, 64, 64, (imm), *(dst), (__v16si)(src1), (__v16si)(src2)))
/// Compute 4-way mixed precision outer product with BF8 (E5M2) format.
@@ -581,7 +719,7 @@ static __inline__ void __tile_ace_top2bf16ps(__acetile *dst, __m512bh src1,
/// \param imm
/// 8-bit immediate selecting BSR scale factors to apply.
#define __tile_ace_top4mxbf8ps(dst, src1, src2, imm) \
- (*(dst) = __builtin_ia32_top4mxbf8ps_internal( \
+ (*(dst) = __builtin_ia32_top4mxbf8ps_nobsr_internal( \
16, 64, 64, (imm), *(dst), (__v16si)(src1), (__v16si)(src2)))
/// Compute 4-way mixed precision outer product of MX INT8 with BSR scaling.
@@ -601,7 +739,7 @@ static __inline__ void __tile_ace_top2bf16ps(__acetile *dst, __m512bh src1,
/// \param imm
/// 8-bit immediate selecting BSR scale factors to apply.
#define __tile_ace_top4mxbssps(dst, src1, src2, imm) \
- (*(dst) = __builtin_ia32_top4mxbssps_internal( \
+ (*(dst) = __builtin_ia32_top4mxbssps_nobsr_internal( \
16, 64, 64, (imm), *(dst), (__v16si)(src1), (__v16si)(src2)))
/// Write a ZMM vector as a column in an ACE tile.
@@ -760,6 +898,102 @@ static __inline__ __m512h __tile_ace_cvtrowps2phl(__acetile *src,
return __builtin_ia32_tcvtrowps2phl_internal(16, 64, *src, idx);
}
+/// Compute 4-way mixed precision outer product with HF8 (E4M3) format
+/// using explicit BSR scales. Multiplies HF8 values from src1 with HF8
+/// values from src2, applies scales from the __bsr struct, converts to
+/// FP32 and accumulates into the ACE tile.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> TOP4MXHF8PS </c> instruction.
+///
+/// \param dst
+/// Pointer to destination/accumulator __acetile.
+/// \param src1
+/// First source ZMM vector containing HF8 (E4M3) values.
+/// \param src2
+/// Second source ZMM vector containing HF8 (E4M3) values.
+/// \param scales
+/// __bsr struct containing A-scales (hi) and B-scales (lo).
+/// \param imm
+/// 8-bit immediate selecting BSR scale factors to apply.
+#define __tile_ace_top4mxhf8ps_bsr(dst, src1, src2, scales, imm) \
+ (*(dst) = __builtin_ia32_top4mxhf8ps_internal( \
+ 16, 64, 64, (imm), *(dst), (__v16si)(src1), (__v16si)(src2), \
+ __bsr_combine_v32((__v16si)((scales).lo), (__v16si)((scales).hi))))
+
+/// Compute 4-way mixed precision outer product with BF8/HF8 format
+/// using explicit BSR scales. Multiplies BF8 (E5M2) values from src1
+/// with HF8 (E4M3) values from src2, applies scales from the __bsr struct,
+/// converts to FP32 and accumulates into the ACE tile.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> TOP4MXBHF8PS </c> instruction.
+///
+/// \param dst
+/// Pointer to destination/accumulator __acetile.
+/// \param src1
+/// First source ZMM vector containing BF8 (E5M2) values.
+/// \param src2
+/// Second source ZMM vector containing HF8 (E4M3) values.
+/// \param scales
+/// __bsr struct containing A-scales (hi) and B-scales (lo).
+/// \param imm
+/// 8-bit immediate selecting BSR scale factors to apply.
+#define __tile_ace_top4mxbhf8ps_bsr(dst, src1, src2, scales, imm) \
+ (*(dst) = __builtin_ia32_top4mxbhf8ps_internal( \
+ 16, 64, 64, (imm), *(dst), (__v16si)(src1), (__v16si)(src2), \
+ __bsr_combine_v32((__v16si)((scales).lo), (__v16si)((scales).hi))))
+
+/// Compute 4-way mixed precision outer product with HF8/BF8 format
+/// using explicit BSR scales. Multiplies HF8 (E4M3) values from src1
+/// with BF8 (E5M2) values from src2, applies scales from the __bsr struct,
+/// converts to FP32 and accumulates into the ACE tile.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> TOP4MXHBF8PS </c> instruction.
+///
+/// \param dst
+/// Pointer to destination/accumulator __acetile.
+/// \param src1
+/// First source ZMM vector containing HF8 (E4M3) values.
+/// \param src2
+/// Second source ZMM vector containing BF8 (E5M2) values.
+/// \param scales
+/// __bsr struct containing A-scales (hi) and B-scales (lo).
+/// \param imm
+/// 8-bit immediate selecting BSR scale factors to apply.
+#define __tile_ace_top4mxhbf8ps_bsr(dst, src1, src2, scales, imm) \
+ (*(dst) = __builtin_ia32_top4mxhbf8ps_internal( \
+ 16, 64, 64, (imm), *(dst), (__v16si)(src1), (__v16si)(src2), \
+ __bsr_combine_v32((__v16si)((scales).lo), (__v16si)((scales).hi))))
+
+/// Compute 4-way mixed precision outer product with BF8 (E5M2) format
+/// using explicit BSR scales. Multiplies BF8 values from both sources,
+/// applies scales from the __bsr struct, converts to FP32 and accumulates
+/// into the ACE tile.
+///
+/// \headerfile <immintrin.h>
+///
+/// This intrinsic corresponds to the <c> TOP4MXBF8PS </c> instruction.
+///
+/// \param dst
+/// Pointer to destination/accumulator __acetile.
+/// \param src1
+/// First source ZMM vector containing BF8 (E5M2) values.
+/// \param src2
+/// Second source ZMM vector containing BF8 (E5M2) values.
+/// \param scales
+/// __bsr struct containing A-scales (hi) and B-scales (lo).
+/// \param imm
+/// 8-bit immediate selecting BSR scale factors to apply.
+#define __tile_ace_top4mxbf8ps_bsr(dst, src1, src2, scales, imm) \
+ (*(dst) = __builtin_ia32_top4mxbf8ps_internal( \
+ 16, 64, 64, (imm), *(dst), (__v16si)(src1), (__v16si)(src2), \
+ __bsr_combine_v32((__v16si)((scales).lo), (__v16si)((scales).hi))))
+
#undef __DEFAULT_FN_ATTRS_ACE
#endif /* __x86_64__ */
diff --git a/clang/test/CodeGen/X86/ace-api.c b/clang/test/CodeGen/X86/ace-api.c
index 8f82fbf66e39a..bc4298289bf9e 100644
--- a/clang/test/CodeGen/X86/ace-api.c
+++ b/clang/test/CodeGen/X86/ace-api.c
@@ -143,3 +143,90 @@ void test_ace_workflow(__m512i *input, __m512i *output) {
// Extract result via getrow
output[0] = __tile_ace_getrow(&acc, 0);
}
+
+// Test BSR struct-based API functions
+
+// CHECK-LABEL: @test_bsr_make
+// CHECK: ret void
+void test_bsr_make(__m512i lo, __m512i hi) {
+ __bsr b = __bsr_make(lo, hi);
+ (void)b;
+}
+
+// CHECK-LABEL: @test_bsr_get_lo
+// CHECK: ret <8 x i64>
+__m512i test_bsr_get_lo(__m512i lo, __m512i hi) {
+ __bsr b = __bsr_make(lo, hi);
+ return __bsr_get_lo(b);
+}
+
+// CHECK-LABEL: @test_bsr_get_hi
+// CHECK: ret <8 x i64>
+__m512i test_bsr_get_hi(__m512i lo, __m512i hi) {
+ __bsr b = __bsr_make(lo, hi);
+ return __bsr_get_hi(b);
+}
+
+// CHECK-LABEL: @test_bsr_set_lo
+// CHECK: ret void
+void test_bsr_set_lo(__m512i lo, __m512i hi, __m512i new_lo) {
+ __bsr b = __bsr_make(lo, hi);
+ b = __bsr_set_lo(b, new_lo);
+ (void)b;
+}
+
+// CHECK-LABEL: @test_bsr_set_hi
+// CHECK: ret void
+void test_bsr_set_hi(__m512i lo, __m512i hi, __m512i new_hi) {
+ __bsr b = __bsr_make(lo, hi);
+ b = __bsr_set_hi(b, new_hi);
+ (void)b;
+}
+
+// CHECK-LABEL: @test_bsr_store
+// CHECK: call void @llvm.x86.bsrmovf(<16 x i32> %{{.*}}, <16 x i32> %{{.*}})
+void test_bsr_store(__m512i lo, __m512i hi) {
+ __bsr b = __bsr_make(lo, hi);
+ __bsr_store(b);
+}
+
+// CHECK-LABEL: @test_bsr_load
+// CHECK: call <16 x i32> @llvm.x86.bsrmovl.get()
+// CHECK: call <16 x i32> @llvm.x86.bsrmovh.get()
+__bsr test_bsr_load(void) {
+ return __bsr_load();
+}
+
+// Test BSR-based mixed-precision outer product macros with __acetile
+
+// CHECK-LABEL: @test_ace_bsr_top4mxhf8ps
+// CHECK: call x86_bsr @llvm.x86.cast.vector.to.bsr.v32i32(<32 x i32>
+// CHECK: call x86_amx @llvm.x86.top4mxhf8ps.bsr.internal(i16 16, i16 64, i16 64, i8 5, x86_amx %{{.*}}, <16 x i32> %{{.*}}, <16 x i32> %{{.*}}, x86_bsr %{{.*}})
+void test_ace_bsr_top4mxhf8ps(__acetile *dst, __m512i a, __m512i b, __m512i lo, __m512i hi) {
+ __bsr scales = __bsr_make(lo, hi);
+ __tile_ace_top4mxhf8ps_bsr(dst, a, b, scales, 5);
+}
+
+// CHECK-LABEL: @test_ace_bsr_top4mxbhf8ps
+// CHECK: call x86_bsr @llvm.x86.cast.vector.to.bsr.v32i32(<32 x i32>
+// CHECK: call x86_amx @llvm.x86.top4mxbhf8ps.bsr.internal(i16 16, i16 64, i16 64, i8 3, x86_amx %{{.*}}, <16 x i32> %{{.*}}, <16 x i32> %{{.*}}, x86_bsr %{{.*}})
+void test_ace_bsr_top4mxbhf8ps(__acetile *dst, __m512i a, __m512i b, __m512i lo, __m512i hi) {
+ __bsr scales = __bsr_make(lo, hi);
+ __tile_ace_top4mxbhf8ps_bsr(dst, a, b, scales, 3);
+}
+
+// CHECK-LABEL: @test_ace_bsr_top4mxhbf8ps
+// CHECK: call x86_bsr @llvm.x86.cast.vector.to.bsr.v32i32(<32 x i32>
+// CHECK: call x86_amx @llvm.x86.top4mxhbf8ps.bsr.internal(i16 16, i16 64, i16 64, i8 7, x86_amx %{{.*}}, <16 x i32> %{{.*}}, <16 x i32> %{{.*}}, x86_bsr %{{.*}})
+void test_ace_bsr_top4mxhbf8ps(__acetile *dst, __m512i a, __m512i b, __m512i lo, __m512i hi) {
+ __bsr scales = __bsr_make(lo, hi);
+ __tile_ace_top4mxhbf8ps_bsr(dst, a, b, scales, 7);
+}
+
+// CHECK-LABEL: @test_ace_bsr_top4mxbf8ps
+// CHECK: call x86_bsr @llvm.x86.cast.vector.to.bsr.v32i32(<32 x i32>
+// CHECK: call x86_amx @llvm.x86.top4mxbf8ps.bsr.internal(i16 16, i16 64, i16 64, i8 2, x86_amx %{{.*}}, <16 x i32> %{{.*}}, <16 x i32> %{{.*}}, x86_bsr %{{.*}})
+void test_ace_bsr_top4mxbf8ps(__acetile *dst, __m512i a, __m512i b, __m512i lo, __m512i hi) {
+ __bsr scales = __bsr_make(lo, hi);
+ __tile_ace_top4mxbf8ps_bsr(dst, a, b, scales, 2);
+}
diff --git a/llvm/include/llvm-c/Core.h b/llvm/include/llvm-c/Core.h
index f3b98a31c5bd7..8f2402c00a176 100644
--- a/llvm/include/llvm-c/Core.h
+++ b/llvm/include/llvm-c/Core.h
@@ -172,6 +172,7 @@ typedef enum {
LLVMX86_AMXTypeKind = 19, /**< X86 AMX */
LLVMTargetExtTypeKind = 20, /**< Target extension type */
LLVMByteTypeKind = 21, /**< Arbitrary bit width bytes */
+ LLVMX86_BSRTypeKind = 22, /**< X86 BSR */
} LLVMTypeKind;
typedef enum {
@@ -1923,6 +1924,11 @@ LLVM_C_ABI LLVMTypeRef LLVMLabelTypeInContext(LLVMContextRef C);
*/
LLVM_C_ABI LLVMTypeRef LLVMX86AMXTypeInContext(LLVMContextRef C);
+/**
+ * Create a X86 BSR type in a context.
+ */
+LLVM_C_ABI LLVMTypeRef LLVMX86BSRTypeInContext(LLVMContextRef C);
+
/**
* Create a token type in a context.
*/
@@ -1949,6 +1955,10 @@ LLVM_C_ABI
LLVM_ATTRIBUTE_C_DEPRECATED(LLVMTypeRef LLVMX86AMXType(void),
"Use of the global context is deprecated, use "
"LLVMX86AMXTypeInContext instead");
+LLVM_C_ABI
+LLVM_ATTRIBUTE_C_DEPRECATED(LLVMTypeRef LLVMX86BSRType(void),
+ "Use of the global context is deprecated, use "
+ "LLVMX86BSRTypeInContext instead");
/**
* Create a target extension type in LLVM context.
diff --git a/llvm/include/llvm/Analysis/IR2Vec.h b/llvm/include/llvm/Analysis/IR2Vec.h
index b89d52a002445..8718f783383fe 100644
--- a/llvm/include/llvm/Analysis/IR2Vec.h
+++ b/llvm/include/llvm/Analysis/IR2Vec.h
@@ -493,6 +493,7 @@ class Vocabulary {
CanonicalTypeID::LabelTy, // LabelTyID
CanonicalTypeID::MetadataTy, // MetadataTyID
CanonicalTypeID::VectorTy, // X86_AMXTyID
+ CanonicalTypeID::VectorTy, // X86_BSRTyID
CanonicalTypeID::TokenTy, // TokenTyID
CanonicalTypeID::IntegerTy, // IntegerTyID
CanonicalTypeID::ByteTy, // ByteTyID
diff --git a/llvm/include/llvm/Bitcode/LLVMBitCodes.h b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
index 358f9a65a80af..7783b5d66cfbd 100644
--- a/llvm/include/llvm/Bitcode/LLVMBitCodes.h
+++ b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
@@ -184,6 +184,8 @@ enum TypeCodes {
TYPE_CODE_TARGET_TYPE = 26, // TARGET_TYPE
TYPE_CODE_BYTE = 27, // BYTE: [width]
+
+ TYPE_CODE_X86_BSR = 28, // X86 BSR
};
enum OperandBundleTagCode {
diff --git a/llvm/include/llvm/CodeGen/ValueTypes.td b/llvm/include/llvm/CodeGen/ValueTypes.td
index 4709a7d72ba8f..bb16769e15d13 100644
--- a/llvm/include/llvm/CodeGen/ValueTypes.td
+++ b/llvm/include/llvm/CodeGen/ValueTypes.td
@@ -352,6 +352,7 @@ def funcref : ValueType<0>; // WebAssembly's funcref type
def externref : ValueType<0>; // WebAssembly's externref type
def exnref : ValueType<0>; // WebAssembly's exnref type
def x86amx : ValueType<8192>; // X86 AMX value
+def x86bsr : ValueType<1024>; // X86 BSR value
def i64x8 : ValueType<512>; // 8 Consecutive GPRs (AArch64)
def aarch64svcount
: ValueType<16>; // AArch64 predicate-as-counter
diff --git a/llvm/include/llvm/IR/DataLayout.h b/llvm/include/llvm/IR/DataLayout.h
index 934c838782417..1d6f83ed3f77c 100644
--- a/llvm/include/llvm/IR/DataLayout.h
+++ b/llvm/include/llvm/IR/DataLayout.h
@@ -820,6 +820,8 @@ inline TypeSize DataLayout::getTypeSizeInBits(Type *Ty) const {
return TypeSize::getFixed(128);
case Type::X86_AMXTyID:
return TypeSize::getFixed(8192);
+ case Type::X86_BSRTyID:
+ return TypeSize::getFixed(1024);
// In memory objects this is always aligned to a higher boundary, but
// only 80 bits contain information.
case Type::X86_FP80TyID:
diff --git a/llvm/include/llvm/IR/Intrinsics.h b/llvm/include/llvm/IR/Intrinsics.h
index 2f9239aa7b77a..2408748725078 100644
--- a/llvm/include/llvm/IR/Intrinsics.h
+++ b/llvm/include/llvm/IR/Intrinsics.h
@@ -188,6 +188,7 @@ struct IITDescriptor {
Pointer, // Address space of the pointer in PointerAddressSpace.
Struct, // Number of elements in StructNumElements.
AMX,
+ BSR,
PPCQuad,
AArch64Svcount,
WasmExternref,
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index f75335cd2c60a..6600a56bc6a5a 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -380,6 +380,7 @@ def IIT_V6 : IIT_Vec<6, 50>;
def IIT_V10 : IIT_Vec<10, 51>;
def IIT_V2048 : IIT_Vec<2048, 52>;
def IIT_V4096 : IIT_Vec<4096, 53>;
+def IIT_BSR : IIT_VT<x86bsr, 54>;
}
defvar IIT_all_FixedTypes = !filter(iit, IIT_all,
@@ -635,6 +636,8 @@ def llvm_aarch64_svcount_ty : LLVMType<aarch64svcount>;
def llvm_x86amx_ty : LLVMType<x86amx>;
+def llvm_x86bsr_ty : LLVMType<x86bsr>;
+
def llvm_v2i1_ty : LLVMType<v2i1>; // 2 x i1
def llvm_v4i1_ty : LLVMType<v4i1>; // 4 x i1
def llvm_v8i1_ty : LLVMType<v8i1>; // 8 x i1
diff --git a/llvm/include/llvm/IR/IntrinsicsX86.td b/llvm/include/llvm/IR/IntrinsicsX86.td
index 1f3a8160e606e..bd6b314a1bb6c 100644
--- a/llvm/include/llvm/IR/IntrinsicsX86.td
+++ b/llvm/include/llvm/IR/IntrinsicsX86.td
@@ -5684,28 +5684,60 @@ let TargetPrefix = "x86" in {
let TargetPrefix = "x86" in {
// BSRINIT - Initialize Block Scale Register
def int_x86_bsrinit : ClangBuiltin<"__builtin_ia32_bsrinit">,
- Intrinsic<[], [], []>;
+ Intrinsic<[], [], [IntrHasSideEffects]>;
- // BSRMOVF - Move Full to BSR
+ // BSRMOVF - Move Full to BSR (legacy, implicit BSR)
// Takes two ZMM registers as input (doubleword integers), writes to BSR (implicit)
def int_x86_bsrmovf : ClangBuiltin<"__builtin_ia32_bsrmovf">,
- Intrinsic<[], [llvm_v16i32_ty, llvm_v16i32_ty], []>;
+ Intrinsic<[], [llvm_v16i32_ty, llvm_v16i32_ty], [IntrHasSideEffects]>;
- // BSRMOVH - Move Half to/from BSR
+ // BSRMOVH - Move Half to/from BSR (legacy, implicit BSR)
// Load variant: read from ZMM (doubleword integers), update BSR
def int_x86_bsrmovh_set : ClangBuiltin<"__builtin_ia32_bsrmovh_set">,
- Intrinsic<[], [llvm_v16i32_ty], []>;
+ Intrinsic<[], [llvm_v16i32_ty], [IntrHasSideEffects]>;
// Store variant: read from BSR, return ZMM (doubleword integers)
def int_x86_bsrmovh_get : ClangBuiltin<"__builtin_ia32_bsrmovh_get">,
- Intrinsic<[llvm_v16i32_ty], [], []>;
+ Intrinsic<[llvm_v16i32_ty], [], [IntrHasSideEffects]>;
- // BSRMOVL - Move Low to/from BSR
+ // BSRMOVL - Move Low to/from BSR (legacy, implicit BSR)
// Load variant: read from ZMM (doubleword integers), write to BSR
def int_x86_bsrmovl_set : ClangBuiltin<"__builtin_ia32_bsrmovl_set">,
- Intrinsic<[], [llvm_v16i32_ty], []>;
+ Intrinsic<[], [llvm_v16i32_ty], [IntrHasSideEffects]>;
// Store variant: read from BSR, return ZMM (doubleword integers)
def int_x86_bsrmovl_get : ClangBuiltin<"__builtin_ia32_bsrmovl_get">,
- Intrinsic<[llvm_v16i32_ty], [], []>;
+ Intrinsic<[llvm_v16i32_ty], [], [IntrHasSideEffects]>;
+
+ // --- New x86_bsr-typed BSR intrinsics ---
+
+ // BSR create: construct a full x86_bsr value from two ZMM halves
+ def int_x86_bsr_create :
+ Intrinsic<[llvm_x86bsr_ty], [llvm_v16i32_ty, llvm_v16i32_ty],
+ [IntrNoMem]>;
+
+ // BSR get high half: extract high 512-bit half from x86_bsr
+ def int_x86_bsr_get_hi :
+ Intrinsic<[llvm_v16i32_ty], [llvm_x86bsr_ty], [IntrNoMem]>;
+
+ // BSR get low half: extract low 512-bit half from x86_bsr
+ def int_x86_bsr_get_lo :
+ Intrinsic<[llvm_v16i32_ty], [llvm_x86bsr_ty], [IntrNoMem]>;
+
+ // BSR set high half: update high 512-bit half of x86_bsr
+ def int_x86_bsr_set_hi :
+ Intrinsic<[llvm_x86bsr_ty], [llvm_x86bsr_ty, llvm_v16i32_ty],
+ [IntrNoMem]>;
+
+ // BSR set low half: update low 512-bit half of x86_bsr
+ def int_x86_bsr_set_lo :
+ Intrinsic<[llvm_x86bsr_ty], [llvm_x86bsr_ty, llvm_v16i32_ty],
+ [IntrNoMem]>;
+
+ // Bridging intrinsics: cast between vector and x86_bsr
+ // (parallel to cast_vector_to_tile / cast_tile_to_vector)
+ def int_x86_cast_vector_to_bsr :
+ DefaultAttrsIntrinsic<[llvm_x86bsr_ty], [llvm_anyvector_ty], [IntrNoMem]>;
+ def int_x86_cast_bsr_to_vector :
+ DefaultAttrsIntrinsic<[llvm_anyvector_ty], [llvm_x86bsr_ty], [IntrNoMem]>;
// TILEMOVCOL - Move Column from Vector to Tile
// Single intrinsic handles both immediate and register index forms.
@@ -5839,41 +5871,83 @@ let TargetPrefix = "x86" in {
[llvm_i16_ty, llvm_i16_ty, llvm_i16_ty,
llvm_x86amx_ty, llvm_v64i8_ty, llvm_v64i8_ty], []>;
- // TOP4MXHFBPS internal - 4th arg is BSR index (i8 immediate)
- // FP8 data packed in dwords (16 dwords per ZMM)
+ // TOP4MX internal intrinsics: LOWERED form (no BSR arg, matches DAG patterns)
+ // X86LowerBSRType rewrites *_bsr_internal -> these + preceding bsrmovf.
+
+ // TOP4MXHFBPS internal (lowered, no BSR) - used by __tile_ace_* macros
def int_x86_top4mxhf8ps_internal :
- ClangBuiltin<"__builtin_ia32_top4mxhf8ps_internal">,
+ ClangBuiltin<"__builtin_ia32_top4mxhf8ps_nobsr_internal">,
Intrinsic<[llvm_x86amx_ty],
[llvm_i16_ty, llvm_i16_ty, llvm_i16_ty, llvm_i8_ty,
llvm_x86amx_ty, llvm_v16i32_ty, llvm_v16i32_ty], [ImmArg<ArgIndex<3>>]>;
- // TOP4MXBHFBPS internal - FP8 data packed in dwords
+ // TOP4MXBHFBPS internal (lowered, no BSR) - used by __tile_ace_* macros
def int_x86_top4mxbhf8ps_internal :
- ClangBuiltin<"__builtin_ia32_top4mxbhf8ps_internal">,
+ ClangBuiltin<"__builtin_ia32_top4mxbhf8ps_nobsr_internal">,
Intrinsic<[llvm_x86amx_ty],
[llvm_i16_ty, llvm_i16_ty, llvm_i16_ty, llvm_i8_ty,
llvm_x86amx_ty, llvm_v16i32_ty, llvm_v16i32_ty], [ImmArg<ArgIndex<3>>]>;
- // TOP4MXHBFBPS internal - FP8 data packed in dwords
+ // TOP4MXHBFBPS internal (lowered, no BSR) - used by __tile_ace_* macros
def int_x86_top4mxhbf8ps_internal :
- ClangBuiltin<"__builtin_ia32_top4mxhbf8ps_internal">,
+ ClangBuiltin<"__builtin_ia32_top4mxhbf8ps_nobsr_internal">,
Intrinsic<[llvm_x86amx_ty],
[llvm_i16_ty, llvm_i16_ty, llvm_i16_ty, llvm_i8_ty,
llvm_x86amx_ty, llvm_v16i32_ty, llvm_v16i32_ty], [ImmArg<ArgIndex<3>>]>;
- // TOP4MXBF8PS internal - MX FP8 BF8xBF8 data packed in dwords
+ // TOP4MXBF8PS internal (lowered, no BSR) - used by __tile_ace_* macros
def int_x86_top4mxbf8ps_internal :
- ClangBuiltin<"__builtin_ia32_top4mxbf8ps_internal">,
+ ClangBuiltin<"__builtin_ia32_top4mxbf8ps_nobsr_internal">,
Intrinsic<[llvm_x86amx_ty],
[llvm_i16_ty, llvm_i16_ty, llvm_i16_ty, llvm_i8_ty,
llvm_x86amx_ty, llvm_v16i32_ty, llvm_v16i32_ty], [ImmArg<ArgIndex<3>>]>;
- // TOP4MXBSSPS internal - MX INT8 SxS to FP32 (BSR scaled)
+ // TOP4MXBSSPS internal (lowered, no BSR) - used by __tile_ace_* macros
def int_x86_top4mxbssps_internal :
- ClangBuiltin<"__builtin_ia32_top4mxbssps_internal">,
+ ClangBuiltin<"__builtin_ia32_top4mxbssps_nobsr_internal">,
Intrinsic<[llvm_x86amx_ty],
[llvm_i16_ty, llvm_i16_ty, llvm_i16_ty, llvm_i8_ty,
llvm_x86amx_ty, llvm_v16i32_ty, llvm_v16i32_ty], [ImmArg<ArgIndex<3>>]>;
+
+ // TOP4MX internal intrinsics: BSR form (with explicit x86_bsr SSA arg)
+ // Emitted by Clang via ClangBuiltin annotation. X86LowerBSRType lowers these
+ // to the non-BSR form above + bsrmovf, before ISel.
+
+ def int_x86_top4mxhf8ps_bsr_internal :
+ ClangBuiltin<"__builtin_ia32_top4mxhf8ps_internal">,
+ Intrinsic<[llvm_x86amx_ty],
+ [llvm_i16_ty, llvm_i16_ty, llvm_i16_ty, llvm_i8_ty,
+ llvm_x86amx_ty, llvm_v16i32_ty, llvm_v16i32_ty,
+ llvm_x86bsr_ty], [ImmArg<ArgIndex<3>>]>;
+
+ def int_x86_top4mxbhf8ps_bsr_internal :
+ ClangBuiltin<"__builtin_ia32_top4mxbhf8ps_internal">,
+ Intrinsic<[llvm_x86amx_ty],
+ [llvm_i16_ty, llvm_i16_ty, llvm_i16_ty, llvm_i8_ty,
+ llvm_x86amx_ty, llvm_v16i32_ty, llvm_v16i32_ty,
+ llvm_x86bsr_ty], [ImmArg<ArgIndex<3>>]>;
+
+ def int_x86_top4mxhbf8ps_bsr_internal :
+ ClangBuiltin<"__builtin_ia32_top4mxhbf8ps_internal">,
+ Intrinsic<[llvm_x86amx_ty],
+ [llvm_i16_ty, llvm_i16_ty, llvm_i16_ty, llvm_i8_ty,
+ llvm_x86amx_ty, llvm_v16i32_ty, llvm_v16i32_ty,
+ llvm_x86bsr_ty], [ImmArg<ArgIndex<3>>]>;
+
+ def int_x86_top4mxbf8ps_bsr_internal :
+ ClangBuiltin<"__builtin_ia32_top4mxbf8ps_internal">,
+ Intrinsic<[llvm_x86amx_ty],
+ [llvm_i16_ty, llvm_i16_ty, llvm_i16_ty, llvm_i8_ty,
+ llvm_x86amx_ty, llvm_v16i32_ty, llvm_v16i32_ty,
+ llvm_x86bsr_ty], [ImmArg<ArgIndex<3>>]>;
+
+ // TOP4MXBSSPS internal - MX INT8 SxS to FP32 (BSR scaled)
+ def int_x86_top4mxbssps_bsr_internal :
+ ClangBuiltin<"__builtin_ia32_top4mxbssps_internal">,
+ Intrinsic<[llvm_x86amx_ty],
+ [llvm_i16_ty, llvm_i16_ty, llvm_i16_ty, llvm_i8_ty,
+ llvm_x86amx_ty, llvm_v16i32_ty, llvm_v16i32_ty,
+ llvm_x86bsr_ty], [ImmArg<ArgIndex<3>>]>;
}
//===----------------------------------------------------------------------===//
diff --git a/llvm/include/llvm/IR/Type.h b/llvm/include/llvm/IR/Type.h
index eadadea56dc21..e66ca59a026f8 100644
--- a/llvm/include/llvm/IR/Type.h
+++ b/llvm/include/llvm/IR/Type.h
@@ -65,6 +65,7 @@ class Type {
LabelTyID, ///< Labels
MetadataTyID, ///< Metadata
X86_AMXTyID, ///< AMX vectors (8192 bits, X86 specific)
+ X86_BSRTyID, ///< BSR register (1024 bits, X86 specific)
TokenTyID, ///< Tokens
// Derived types... see DerivedTypes.h file.
@@ -201,6 +202,9 @@ class Type {
/// Return true if this is X86 AMX.
bool isX86_AMXTy() const { return getTypeID() == X86_AMXTyID; }
+ /// Return true if this is X86 BSR.
+ bool isX86_BSRTy() const { return getTypeID() == X86_BSRTyID; }
+
/// Return true if this is a target extension type.
bool isTargetExtTy() const { return getTypeID() == TargetExtTyID; }
@@ -310,7 +314,8 @@ class Type {
/// includes all first-class types except struct and array types.
bool isSingleValueType() const {
return isFloatingPointTy() || isIntegerTy() || isPointerTy() ||
- isVectorTy() || isX86_AMXTy() || isTargetExtTy() || isByteTy();
+ isVectorTy() || isX86_AMXTy() || isX86_BSRTy() || isTargetExtTy() ||
+ isByteTy();
}
/// Return true if the type is an aggregate type. This means it is valid as
@@ -327,7 +332,7 @@ class Type {
// If it's a primitive, it is always sized.
if (getTypeID() == IntegerTyID || isFloatingPointTy() ||
getTypeID() == PointerTyID || getTypeID() == X86_AMXTyID ||
- getTypeID() == ByteTyID)
+ getTypeID() == X86_BSRTyID || getTypeID() == ByteTyID)
return true;
// If it is not something that can have a size (e.g. a function or label),
// it doesn't have a size.
@@ -471,6 +476,7 @@ class Type {
LLVM_ABI static Type *getFP128Ty(LLVMContext &C);
LLVM_ABI static Type *getPPC_FP128Ty(LLVMContext &C);
LLVM_ABI static Type *getX86_AMXTy(LLVMContext &C);
+ LLVM_ABI static Type *getX86_BSRTy(LLVMContext &C);
LLVM_ABI static Type *getTokenTy(LLVMContext &C);
LLVM_ABI static ByteType *getByteNTy(LLVMContext &C, unsigned N);
LLVM_ABI static ByteType *getByte1Ty(LLVMContext &C);
diff --git a/llvm/include/llvm/SandboxIR/Type.h b/llvm/include/llvm/SandboxIR/Type.h
index bc7e04f41bd17..fce16865fe383 100644
--- a/llvm/include/llvm/SandboxIR/Type.h
+++ b/llvm/include/llvm/SandboxIR/Type.h
@@ -148,6 +148,9 @@ class Type {
/// Return true if this is X86 AMX.
bool isX86_AMXTy() const { return LLVMTy->isX86_AMXTy(); }
+ /// Return true if this is X86 BSR.
+ bool isX86_BSRTy() const { return LLVMTy->isX86_BSRTy(); }
+
/// Return true if this is a target extension type.
bool isTargetExtTy() const { return LLVMTy->isTargetExtTy(); }
diff --git a/llvm/lib/Analysis/ConstantFolding.cpp b/llvm/lib/Analysis/ConstantFolding.cpp
index 1d17f6a6cc2c7..2e49a54480dfa 100644
--- a/llvm/lib/Analysis/ConstantFolding.cpp
+++ b/llvm/lib/Analysis/ConstantFolding.cpp
@@ -707,14 +707,16 @@ Constant *FoldReinterpretLoadFromConst(Constant *C, Type *LoadTy,
DL.getTypeSizeInBits(LoadTy).getFixedValue());
if (Constant *Res =
FoldReinterpretLoadFromConst(C, MapTy, OrigLoadTy, Offset, DL)) {
- if (Res->isNullValue() && !LoadTy->isX86_AMXTy())
+ if (Res->isNullValue() && !LoadTy->isX86_AMXTy() &&
+ !LoadTy->isX86_BSRTy())
// Materializing a zero can be done trivially without a bitcast
return Constant::getNullValue(LoadTy);
Type *CastTy = LoadTy->isPtrOrPtrVectorTy() ? DL.getIntPtrType(LoadTy) : LoadTy;
Res = FoldBitCast(Res, CastTy, DL);
if (LoadTy->isPtrOrPtrVectorTy()) {
// For vector of pointer, we needed to first convert to a vector of integer, then do vector inttoptr
- if (Res->isNullValue() && !LoadTy->isX86_AMXTy())
+ if (Res->isNullValue() && !LoadTy->isX86_AMXTy() &&
+ !LoadTy->isX86_BSRTy())
return Constant::getNullValue(LoadTy);
if (DL.isNonIntegralPointerType(LoadTy->getScalarType()))
// Be careful not to replace a load of an addrspace value with an inttoptr here
@@ -912,7 +914,7 @@ Constant *llvm::ConstantFoldLoadFromUniformValue(Constant *C, Type *Ty,
// uniform.
if (!DL.typeSizeEqualsStoreSize(C->getType()))
return nullptr;
- if (C->isNullValue() && !Ty->isX86_AMXTy())
+ if (C->isNullValue() && !Ty->isX86_AMXTy() && !Ty->isX86_BSRTy())
return Constant::getNullValue(Ty);
if (C->isAllOnesValue() &&
(Ty->isIntOrIntVectorTy() || Ty->isByteOrByteVectorTy() ||
diff --git a/llvm/lib/AsmParser/LLLexer.cpp b/llvm/lib/AsmParser/LLLexer.cpp
index 069a180056488..ca2c4fd34f2ca 100644
--- a/llvm/lib/AsmParser/LLLexer.cpp
+++ b/llvm/lib/AsmParser/LLLexer.cpp
@@ -923,6 +923,7 @@ lltok::Kind LLLexer::LexIdentifier() {
TYPEKEYWORD("label", Type::getLabelTy(Context));
TYPEKEYWORD("metadata", Type::getMetadataTy(Context));
TYPEKEYWORD("x86_amx", Type::getX86_AMXTy(Context));
+ TYPEKEYWORD("x86_bsr", Type::getX86_BSRTy(Context));
TYPEKEYWORD("token", Type::getTokenTy(Context));
TYPEKEYWORD("ptr", PointerType::getUnqual(Context));
diff --git a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
index a5a11ab6221d4..c76f1bd116398 100644
--- a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
+++ b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
@@ -2692,6 +2692,9 @@ Error BitcodeReader::parseTypeTableBody() {
case bitc::TYPE_CODE_X86_AMX: // X86_AMX
ResultTy = Type::getX86_AMXTy(Context);
break;
+ case bitc::TYPE_CODE_X86_BSR: // X86_BSR
+ ResultTy = Type::getX86_BSRTy(Context);
+ break;
case bitc::TYPE_CODE_TOKEN: // TOKEN
ResultTy = Type::getTokenTy(Context);
break;
diff --git a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
index a6e7941a06699..65e264318b1fd 100644
--- a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -1228,6 +1228,9 @@ void ModuleBitcodeWriter::writeTypeTable() {
Code = bitc::TYPE_CODE_METADATA;
break;
case Type::X86_AMXTyID: Code = bitc::TYPE_CODE_X86_AMX; break;
+ case Type::X86_BSRTyID:
+ Code = bitc::TYPE_CODE_X86_BSR;
+ break;
case Type::TokenTyID: Code = bitc::TYPE_CODE_TOKEN; break;
case Type::ByteTyID:
// BYTE: [width]
diff --git a/llvm/lib/CodeGen/ValueTypes.cpp b/llvm/lib/CodeGen/ValueTypes.cpp
index a8eb5f801a280..5f60f5f4a86d6 100644
--- a/llvm/lib/CodeGen/ValueTypes.cpp
+++ b/llvm/lib/CodeGen/ValueTypes.cpp
@@ -186,6 +186,8 @@ std::string EVT::getEVTString() const {
case MVT::Glue: return "glue";
case MVT::x86mmx: return "x86mmx";
case MVT::x86amx: return "x86amx";
+ case MVT::x86bsr:
+ return "x86bsr";
case MVT::i64x8: return "i64x8";
case MVT::Metadata: return "Metadata";
case MVT::Untyped: return "Untyped";
@@ -228,6 +230,7 @@ Type *EVT::getTypeForEVT(LLVMContext &Context) const {
case MVT::aarch64mfp8:
return FixedVectorType::get(IntegerType::get(Context, 8), 1);
case MVT::x86amx: return Type::getX86_AMXTy(Context);
+ case MVT::x86bsr: return Type::getX86_BSRTy(Context);
case MVT::i64x8: return IntegerType::get(Context, 512);
case MVT::amdgpuBufferFatPointer: return IntegerType::get(Context, 160);
case MVT::amdgpuBufferStridedPointer: return IntegerType::get(Context, 192);
@@ -287,6 +290,8 @@ MVT MVT::getVT(Type *Ty, bool HandleUnknown){
llvm_unreachable("Unknown target ext type!");
}
case Type::X86_AMXTyID: return MVT(MVT::x86amx);
+ case Type::X86_BSRTyID:
+ return MVT(MVT::x86bsr);
case Type::FP128TyID: return MVT(MVT::f128);
case Type::PPC_FP128TyID: return MVT(MVT::ppcf128);
case Type::FixedVectorTyID:
diff --git a/llvm/lib/IR/AsmWriter.cpp b/llvm/lib/IR/AsmWriter.cpp
index e90630a8cae59..72de3ddba7738 100644
--- a/llvm/lib/IR/AsmWriter.cpp
+++ b/llvm/lib/IR/AsmWriter.cpp
@@ -668,6 +668,9 @@ void TypePrinting::print(Type *Ty, raw_ostream &OS) {
OS << "metadata";
return;
case Type::X86_AMXTyID: OS << "x86_amx"; return;
+ case Type::X86_BSRTyID:
+ OS << "x86_bsr";
+ return;
case Type::TokenTyID: OS << "token"; return;
case Type::ByteTyID:
OS << 'b' << Ty->getByteBitWidth();
diff --git a/llvm/lib/IR/ConstantFold.cpp b/llvm/lib/IR/ConstantFold.cpp
index f8abd576f93c4..30eecda242370 100644
--- a/llvm/lib/IR/ConstantFold.cpp
+++ b/llvm/lib/IR/ConstantFold.cpp
@@ -175,7 +175,7 @@ Constant *llvm::ConstantFoldCastInstruction(unsigned opc, Constant *V,
return UndefValue::get(DestTy);
}
- if (V->isNullValue() && !DestTy->isX86_AMXTy() &&
+ if (V->isNullValue() && !DestTy->isX86_AMXTy() && !DestTy->isX86_BSRTy() &&
opc != Instruction::AddrSpaceCast)
return Constant::getNullValue(DestTy);
diff --git a/llvm/lib/IR/Core.cpp b/llvm/lib/IR/Core.cpp
index a7abd3eed31c3..24fcc40f1a307 100644
--- a/llvm/lib/IR/Core.cpp
+++ b/llvm/lib/IR/Core.cpp
@@ -648,6 +648,8 @@ LLVMTypeKind LLVMGetTypeKind(LLVMTypeRef Ty) {
return LLVMVectorTypeKind;
case Type::X86_AMXTyID:
return LLVMX86_AMXTypeKind;
+ case Type::X86_BSRTyID:
+ return LLVMX86_BSRTypeKind;
case Type::TokenTyID:
return LLVMTokenTypeKind;
case Type::ScalableVectorTyID:
@@ -771,6 +773,9 @@ LLVMTypeRef LLVMPPCFP128TypeInContext(LLVMContextRef C) {
LLVMTypeRef LLVMX86AMXTypeInContext(LLVMContextRef C) {
return (LLVMTypeRef) Type::getX86_AMXTy(*unwrap(C));
}
+LLVMTypeRef LLVMX86BSRTypeInContext(LLVMContextRef C) {
+ return (LLVMTypeRef)Type::getX86_BSRTy(*unwrap(C));
+}
LLVMTypeRef LLVMHalfType(void) {
return LLVMHalfTypeInContext(getGlobalContextForCAPI());
@@ -796,6 +801,9 @@ LLVMTypeRef LLVMPPCFP128Type(void) {
LLVMTypeRef LLVMX86AMXType(void) {
return LLVMX86AMXTypeInContext(getGlobalContextForCAPI());
}
+LLVMTypeRef LLVMX86BSRType(void) {
+ return LLVMX86BSRTypeInContext(getGlobalContextForCAPI());
+}
/*--.. Operations on function types ........................................--*/
diff --git a/llvm/lib/IR/DataLayout.cpp b/llvm/lib/IR/DataLayout.cpp
index 82b33887b81f2..80f521e1b78d5 100644
--- a/llvm/lib/IR/DataLayout.cpp
+++ b/llvm/lib/IR/DataLayout.cpp
@@ -939,6 +939,8 @@ Align DataLayout::getAlignment(Type *Ty, bool abi_or_pref) const {
}
case Type::X86_AMXTyID:
return Align(64);
+ case Type::X86_BSRTyID:
+ return Align(64);
case Type::TargetExtTyID: {
Type *LayoutTy = cast<TargetExtType>(Ty)->getLayoutType();
return getAlignment(LayoutTy, abi_or_pref);
diff --git a/llvm/lib/IR/Intrinsics.cpp b/llvm/lib/IR/Intrinsics.cpp
index 6ddfc18c56501..94725d1ff05ff 100644
--- a/llvm/lib/IR/Intrinsics.cpp
+++ b/llvm/lib/IR/Intrinsics.cpp
@@ -161,6 +161,9 @@ static std::string getMangledTypeStr(Type *Ty, bool &HasUnnamedType) {
case Type::X86_AMXTyID:
Result += "x86amx";
break;
+ case Type::X86_BSRTyID:
+ Result += "x86bsr";
+ break;
case Type::IntegerTyID:
Result += "i" + utostr(cast<IntegerType>(Ty)->getBitWidth());
break;
@@ -251,6 +254,9 @@ DecodeIITType(unsigned &NextElt, ArrayRef<unsigned char> Infos,
case IIT_AMX:
OutputTable.push_back(IITDescriptor::get(IITDescriptor::AMX, 0));
return;
+ case IIT_BSR:
+ OutputTable.push_back(IITDescriptor::get(IITDescriptor::BSR, 0));
+ return;
case IIT_TOKEN:
OutputTable.push_back(IITDescriptor::get(IITDescriptor::Token, 0));
return;
@@ -549,6 +555,8 @@ static Type *DecodeFixedType(ArrayRef<Intrinsic::IITDescriptor> &Infos,
return llvm::FixedVectorType::get(llvm::IntegerType::get(Context, 64), 1);
case IITDescriptor::AMX:
return Type::getX86_AMXTy(Context);
+ case IITDescriptor::BSR:
+ return Type::getX86_BSRTy(Context);
case IITDescriptor::Token:
return Type::getTokenTy(Context);
case IITDescriptor::Metadata:
@@ -1020,6 +1028,8 @@ matchIntrinsicType(Type *Ty, ArrayRef<Intrinsic::IITDescriptor> &Infos,
}
case IITDescriptor::AMX:
return PrintMsg(Ty->isX86_AMXTy(), "x86_amx");
+ case IITDescriptor::BSR:
+ return PrintMsg(Ty->isX86_BSRTy(), "x86_bsr");
case IITDescriptor::Token:
return PrintMsg(Ty->isTokenTy(), "token");
case IITDescriptor::Metadata:
diff --git a/llvm/lib/IR/LLVMContextImpl.cpp b/llvm/lib/IR/LLVMContextImpl.cpp
index 90afa09f73abe..f161ff32f972d 100644
--- a/llvm/lib/IR/LLVMContextImpl.cpp
+++ b/llvm/lib/IR/LLVMContextImpl.cpp
@@ -36,9 +36,10 @@ LLVMContextImpl::LLVMContextImpl(LLVMContext &C)
MetadataTy(C, Type::MetadataTyID), TokenTy(C, Type::TokenTyID),
X86_FP80Ty(C, Type::X86_FP80TyID), FP128Ty(C, Type::FP128TyID),
PPC_FP128Ty(C, Type::PPC_FP128TyID), X86_AMXTy(C, Type::X86_AMXTyID),
- Int1Ty(C, 1), Int8Ty(C, 8), Int16Ty(C, 16), Int32Ty(C, 32),
- Int64Ty(C, 64), Int128Ty(C, 128), Byte1Ty(C, 1), Byte8Ty(C, 8),
- Byte16Ty(C, 16), Byte32Ty(C, 32), Byte64Ty(C, 64), Byte128Ty(C, 128) {}
+ X86_BSRTy(C, Type::X86_BSRTyID), Int1Ty(C, 1), Int8Ty(C, 8),
+ Int16Ty(C, 16), Int32Ty(C, 32), Int64Ty(C, 64), Int128Ty(C, 128),
+ Byte1Ty(C, 1), Byte8Ty(C, 8), Byte16Ty(C, 16), Byte32Ty(C, 32),
+ Byte64Ty(C, 64), Byte128Ty(C, 128) {}
LLVMContextImpl::~LLVMContextImpl() {
#ifndef NDEBUG
diff --git a/llvm/lib/IR/LLVMContextImpl.h b/llvm/lib/IR/LLVMContextImpl.h
index 41c8a92c56eda..80b12b14a024e 100644
--- a/llvm/lib/IR/LLVMContextImpl.h
+++ b/llvm/lib/IR/LLVMContextImpl.h
@@ -1673,7 +1673,7 @@ class LLVMContextImpl {
// Basic type instances.
Type VoidTy, LabelTy, HalfTy, BFloatTy, FloatTy, DoubleTy, MetadataTy,
TokenTy;
- Type X86_FP80Ty, FP128Ty, PPC_FP128Ty, X86_AMXTy;
+ Type X86_FP80Ty, FP128Ty, PPC_FP128Ty, X86_AMXTy, X86_BSRTy;
IntegerType Int1Ty, Int8Ty, Int16Ty, Int32Ty, Int64Ty, Int128Ty;
ByteType Byte1Ty, Byte8Ty, Byte16Ty, Byte32Ty, Byte64Ty, Byte128Ty;
diff --git a/llvm/lib/IR/Type.cpp b/llvm/lib/IR/Type.cpp
index 8c108a4d1f275..7778eac3b7f76 100644
--- a/llvm/lib/IR/Type.cpp
+++ b/llvm/lib/IR/Type.cpp
@@ -48,6 +48,8 @@ Type *Type::getPrimitiveType(LLVMContext &C, TypeID IDNumber) {
case LabelTyID : return getLabelTy(C);
case MetadataTyID : return getMetadataTy(C);
case X86_AMXTyID : return getX86_AMXTy(C);
+ case X86_BSRTyID:
+ return getX86_BSRTy(C);
case TokenTyID : return getTokenTy(C);
default:
return nullptr;
@@ -172,6 +174,14 @@ bool Type::canLosslesslyBitCastTo(Type *Ty) const {
Ty->getPrimitiveSizeInBits().getFixedValue() == 8192)
return true;
+ // 1024-bit fixed width vector types can be losslessly converted to x86bsr.
+ if (((isa<FixedVectorType>(this)) && Ty->isX86_BSRTy()) &&
+ getPrimitiveSizeInBits().getFixedValue() == 1024)
+ return true;
+ if ((isX86_BSRTy() && isa<FixedVectorType>(Ty)) &&
+ Ty->getPrimitiveSizeInBits().getFixedValue() == 1024)
+ return true;
+
// Conservatively assume we can't losslessly convert between pointers with
// different address spaces.
return false;
@@ -212,6 +222,8 @@ TypeSize Type::getPrimitiveSizeInBits() const {
return TypeSize::getFixed(128);
case Type::X86_AMXTyID:
return TypeSize::getFixed(8192);
+ case Type::X86_BSRTyID:
+ return TypeSize::getFixed(1024);
case Type::ByteTyID:
return TypeSize::getFixed(cast<ByteType>(this)->getBitWidth());
case Type::IntegerTyID:
@@ -291,6 +303,7 @@ Type *Type::getX86_FP80Ty(LLVMContext &C) { return &C.pImpl->X86_FP80Ty; }
Type *Type::getFP128Ty(LLVMContext &C) { return &C.pImpl->FP128Ty; }
Type *Type::getPPC_FP128Ty(LLVMContext &C) { return &C.pImpl->PPC_FP128Ty; }
Type *Type::getX86_AMXTy(LLVMContext &C) { return &C.pImpl->X86_AMXTy; }
+Type *Type::getX86_BSRTy(LLVMContext &C) { return &C.pImpl->X86_BSRTy; }
ByteType *Type::getByte1Ty(LLVMContext &C) { return &C.pImpl->Byte1Ty; }
ByteType *Type::getByte8Ty(LLVMContext &C) { return &C.pImpl->Byte8Ty; }
@@ -829,7 +842,8 @@ ArrayType *ArrayType::get(Type *ElementType, uint64_t NumElements) {
bool ArrayType::isValidElementType(Type *ElemTy) {
return !ElemTy->isVoidTy() && !ElemTy->isLabelTy() &&
!ElemTy->isMetadataTy() && !ElemTy->isFunctionTy() &&
- !ElemTy->isTokenTy() && !ElemTy->isX86_AMXTy();
+ !ElemTy->isTokenTy() && !ElemTy->isX86_AMXTy() &&
+ !ElemTy->isX86_BSRTy();
}
//===----------------------------------------------------------------------===//
@@ -940,7 +954,7 @@ PointerType *Type::getPointerTo(unsigned AddrSpace) const {
bool PointerType::isValidElementType(Type *ElemTy) {
return !ElemTy->isVoidTy() && !ElemTy->isLabelTy() &&
!ElemTy->isMetadataTy() && !ElemTy->isTokenTy() &&
- !ElemTy->isX86_AMXTy();
+ !ElemTy->isX86_AMXTy() && !ElemTy->isX86_BSRTy();
}
bool PointerType::isLoadableOrStorableType(Type *ElemTy) {
diff --git a/llvm/lib/IR/TypedPointerType.cpp b/llvm/lib/IR/TypedPointerType.cpp
index 85f7a5a2ae4c3..4bc96f3573c23 100644
--- a/llvm/lib/IR/TypedPointerType.cpp
+++ b/llvm/lib/IR/TypedPointerType.cpp
@@ -38,5 +38,5 @@ TypedPointerType::TypedPointerType(Type *E, unsigned AddrSpace)
bool TypedPointerType::isValidElementType(Type *ElemTy) {
return !ElemTy->isVoidTy() && !ElemTy->isLabelTy() &&
!ElemTy->isMetadataTy() && !ElemTy->isTokenTy() &&
- !ElemTy->isX86_AMXTy();
+ !ElemTy->isX86_AMXTy() && !ElemTy->isX86_BSRTy();
}
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index f605af1d0a31b..aa11a5b44096c 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3112,6 +3112,8 @@ void Verifier::visitFunction(const Function &F) {
"Function takes token but isn't an intrinsic", &Arg, &F);
Check(!Arg.getType()->isX86_AMXTy(),
"Function takes x86_amx but isn't an intrinsic", &Arg, &F);
+ Check(!Arg.getType()->isX86_BSRTy(),
+ "Function takes x86_bsr but isn't an intrinsic", &Arg, &F);
}
// Check that swifterror argument is only used by loads and stores.
@@ -3126,6 +3128,8 @@ void Verifier::visitFunction(const Function &F) {
"Function returns a token but isn't an intrinsic", &F);
Check(!F.getReturnType()->isX86_AMXTy(),
"Function returns a x86_amx but isn't an intrinsic", &F);
+ Check(!F.getReturnType()->isX86_BSRTy(),
+ "Function returns a x86_bsr but isn't an intrinsic", &F);
}
// Get the function metadata attachments.
@@ -3980,6 +3984,8 @@ void Verifier::visitCallBase(CallBase &Call) {
"Return type cannot be token for indirect call!");
Check(!FTy->getReturnType()->isX86_AMXTy(),
"Return type cannot be x86_amx for indirect call!");
+ Check(!FTy->getReturnType()->isX86_BSRTy(),
+ "Return type cannot be x86_bsr for indirect call!");
}
if (Intrinsic::ID ID = Call.getIntrinsicID())
@@ -5901,9 +5907,12 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
for (Value *V : Call.args()) {
if (auto *MD = dyn_cast<MetadataAsValue>(V))
visitMetadataAsValue(*MD, Call.getCaller());
- if (auto *Const = dyn_cast<Constant>(V))
+ if (auto *Const = dyn_cast<Constant>(V)) {
Check(!Const->getType()->isX86_AMXTy(),
"const x86_amx is not allowed in argument!");
+ Check(!Const->getType()->isX86_BSRTy(),
+ "const x86_bsr is not allowed in argument!");
+ }
}
switch (ID) {
diff --git a/llvm/lib/Target/DirectX/DXILWriter/DXILBitcodeWriter.cpp b/llvm/lib/Target/DirectX/DXILWriter/DXILBitcodeWriter.cpp
index 6f14f70014b01..22be0fd1e7af7 100644
--- a/llvm/lib/Target/DirectX/DXILWriter/DXILBitcodeWriter.cpp
+++ b/llvm/lib/Target/DirectX/DXILWriter/DXILBitcodeWriter.cpp
@@ -1026,6 +1026,7 @@ void DXILBitcodeWriter::writeTypeTable() {
switch (T->getTypeID()) {
case Type::BFloatTyID:
case Type::X86_AMXTyID:
+ case Type::X86_BSRTyID:
case Type::TokenTyID:
case Type::TargetExtTyID:
llvm_unreachable("These should never be used!!!");
diff --git a/llvm/lib/Target/Hexagon/HexagonTargetObjectFile.cpp b/llvm/lib/Target/Hexagon/HexagonTargetObjectFile.cpp
index 0212dffa38645..4c4c1f2feefb8 100644
--- a/llvm/lib/Target/Hexagon/HexagonTargetObjectFile.cpp
+++ b/llvm/lib/Target/Hexagon/HexagonTargetObjectFile.cpp
@@ -331,6 +331,7 @@ unsigned HexagonTargetObjectFile::getSmallestAddressableSize(const Type *Ty,
case Type::LabelTyID:
case Type::MetadataTyID:
case Type::X86_AMXTyID:
+ case Type::X86_BSRTyID:
case Type::TokenTyID:
case Type::TypedPointerTyID:
case Type::TargetExtTyID:
diff --git a/llvm/lib/Target/X86/CMakeLists.txt b/llvm/lib/Target/X86/CMakeLists.txt
index 62987bdbd1c2b..01aa3a72ceba3 100644
--- a/llvm/lib/Target/X86/CMakeLists.txt
+++ b/llvm/lib/Target/X86/CMakeLists.txt
@@ -40,6 +40,7 @@ set(sources
X86LowerTileCopy.cpp
X86LowerAMXType.cpp
X86LowerAMXIntrinsics.cpp
+ X86LowerBSRType.cpp
X86TileConfig.cpp
X86FastPreTileConfig.cpp
X86FastTileConfig.cpp
diff --git a/llvm/lib/Target/X86/X86.h b/llvm/lib/Target/X86/X86.h
index 48dedd9d2a758..a7f7613c89192 100644
--- a/llvm/lib/Target/X86/X86.h
+++ b/llvm/lib/Target/X86/X86.h
@@ -381,6 +381,17 @@ class X86LowerAMXTypePass : public RequiredPassInfoMixin<X86LowerAMXTypePass> {
FunctionPass *createX86LowerAMXTypeLegacyPass();
+/// The pass lowers x86_bsr type operations to the legacy implicit-BSR
+/// intrinsics (bsrmovf, bsrmovh, bsrmovl) and handles cast_vector_to_bsr /
+/// cast_bsr_to_vector bridging.
+class X86LowerBSRTypePass : public PassInfoMixin<X86LowerBSRTypePass> {
+public:
+ PreservedAnalyses run(Function &F, FunctionAnalysisManager &FAM);
+ static bool isRequired() { return true; }
+};
+
+FunctionPass *createX86LowerBSRTypeLegacyPass();
+
// Suppresses APX features for relocations for supporting older linkers.
class X86SuppressAPXForRelocationPass
: public OptionalPassInfoMixin<X86SuppressAPXForRelocationPass> {
@@ -503,6 +514,7 @@ void initializeX86LoadValueInjectionLoadHardeningLegacyPass(PassRegistry &);
void initializeX86LoadValueInjectionRetHardeningLegacyPass(PassRegistry &);
void initializeX86LowerAMXIntrinsicsLegacyPassPass(PassRegistry &);
void initializeX86LowerAMXTypeLegacyPassPass(PassRegistry &);
+void initializeX86LowerBSRTypeLegacyPassPass(PassRegistry &);
void initializeX86LowerTileCopyLegacyPass(PassRegistry &);
void initializeX86OptimizeLEAsLegacyPass(PassRegistry &);
void initializeX86PartialReductionLegacyPass(PassRegistry &);
diff --git a/llvm/lib/Target/X86/X86ISelLowering.cpp b/llvm/lib/Target/X86/X86ISelLowering.cpp
index a9f779c09132c..0db95f9561908 100644
--- a/llvm/lib/Target/X86/X86ISelLowering.cpp
+++ b/llvm/lib/Target/X86/X86ISelLowering.cpp
@@ -2666,6 +2666,10 @@ X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
addRegisterClass(MVT::x86amx, &X86::TILERegClass);
}
+ if (!Subtarget.useSoftFloat() && Subtarget.hasACEV1()) {
+ addRegisterClass(MVT::x86bsr, &X86::BSRRegClass);
+ }
+
// We want to custom lower some of our intrinsics.
setOperationAction(ISD::INTRINSIC_WO_CHAIN, MVT::Other, Custom);
setOperationAction(ISD::INTRINSIC_W_CHAIN, MVT::Other, Custom);
diff --git a/llvm/lib/Target/X86/X86LowerAMXType.cpp b/llvm/lib/Target/X86/X86LowerAMXType.cpp
index 9b97025d8532c..97b750d772914 100644
--- a/llvm/lib/Target/X86/X86LowerAMXType.cpp
+++ b/llvm/lib/Target/X86/X86LowerAMXType.cpp
@@ -276,12 +276,17 @@ std::pair<Value *, Value *> getShape(IntrinsicInst *II, unsigned OpNo) {
break;
}
// ACE TOP4MX intrinsics - mixed precision with BSR index
- // Pattern: (m, n, k, bsr_idx, acc_tile, zmm1, zmm2)
+ // Pattern: (m, n, k, bsr_idx, acc_tile, zmm1, zmm2[, bsr])
case Intrinsic::x86_top4mxhf8ps_internal:
case Intrinsic::x86_top4mxbhf8ps_internal:
case Intrinsic::x86_top4mxhbf8ps_internal:
case Intrinsic::x86_top4mxbf8ps_internal:
- case Intrinsic::x86_top4mxbssps_internal: {
+ case Intrinsic::x86_top4mxbssps_internal:
+ case Intrinsic::x86_top4mxhf8ps_bsr_internal:
+ case Intrinsic::x86_top4mxbhf8ps_bsr_internal:
+ case Intrinsic::x86_top4mxhbf8ps_bsr_internal:
+ case Intrinsic::x86_top4mxbf8ps_bsr_internal:
+ case Intrinsic::x86_top4mxbssps_bsr_internal: {
switch (OpNo) {
case 4: // Accumulator tile
Row = II->getArgOperand(0);
diff --git a/llvm/lib/Target/X86/X86LowerBSRType.cpp b/llvm/lib/Target/X86/X86LowerBSRType.cpp
new file mode 100644
index 0000000000000..8342d3a681740
--- /dev/null
+++ b/llvm/lib/Target/X86/X86LowerBSRType.cpp
@@ -0,0 +1,373 @@
+//===- Target/X86/X86LowerBSRType.cpp - -------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+/// \file Pass to lower x86_bsr type operations.
+///
+/// The x86_bsr type represents a 1024-bit BSR (Block Scale Register) value.
+/// This pass lowers cast_vector_to_bsr / cast_bsr_to_vector bridging
+/// intrinsics and converts bsr_create / bsr_get_hi / bsr_get_lo / bsr_set_hi
+/// / bsr_set_lo intrinsics into the legacy implicit-BSR intrinsics
+/// (bsrmovf, bsrmovh_set, bsrmovh_get, bsrmovl_set, bsrmovl_get).
+///
+/// This pass runs before instruction selection, parallel to X86LowerAMXType
+/// for AMX tiles.
+///
+//===----------------------------------------------------------------------===//
+//
+#include "X86.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/CodeGen/Passes.h"
+#include "llvm/IR/Function.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/IntrinsicsX86.h"
+#include "llvm/IR/PassManager.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Pass.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "x86-lower-bsr-type"
+
+namespace {
+
+static bool isBSRIntrinsic(IntrinsicInst *II) {
+ switch (II->getIntrinsicID()) {
+ default:
+ return false;
+ case Intrinsic::x86_bsr_create:
+ case Intrinsic::x86_bsr_get_hi:
+ case Intrinsic::x86_bsr_get_lo:
+ case Intrinsic::x86_bsr_set_hi:
+ case Intrinsic::x86_bsr_set_lo:
+ case Intrinsic::x86_cast_vector_to_bsr:
+ case Intrinsic::x86_cast_bsr_to_vector:
+ case Intrinsic::x86_top4mxhf8ps_bsr_internal:
+ case Intrinsic::x86_top4mxbhf8ps_bsr_internal:
+ case Intrinsic::x86_top4mxhbf8ps_bsr_internal:
+ case Intrinsic::x86_top4mxbf8ps_bsr_internal:
+ return true;
+ }
+}
+
+/// Map a BSR-carrying TOP4MX intrinsic to its non-BSR legacy equivalent.
+static Intrinsic::ID getBSRToLegacyID(Intrinsic::ID ID) {
+ switch (ID) {
+ case Intrinsic::x86_top4mxhf8ps_bsr_internal:
+ return Intrinsic::x86_top4mxhf8ps_internal;
+ case Intrinsic::x86_top4mxbhf8ps_bsr_internal:
+ return Intrinsic::x86_top4mxbhf8ps_internal;
+ case Intrinsic::x86_top4mxhbf8ps_bsr_internal:
+ return Intrinsic::x86_top4mxhbf8ps_internal;
+ case Intrinsic::x86_top4mxbf8ps_bsr_internal:
+ return Intrinsic::x86_top4mxbf8ps_internal;
+ default:
+ return Intrinsic::not_intrinsic;
+ }
+}
+
+/// Lower a single BSR intrinsic call. Returns true if lowered.
+static bool lowerBSRIntrinsic(IntrinsicInst *II, IRBuilder<> &Builder) {
+ Builder.SetInsertPoint(II);
+
+ switch (II->getIntrinsicID()) {
+ default:
+ return false;
+
+ case Intrinsic::x86_bsr_create: {
+ // bsr_create(lo, hi) -> bsrmovf(hi, lo) + implicit BSR
+ // Per ACE spec: BSRMOVF(src1, src2) where src1→A-scales (upper=hi),
+ // src2→B-scales (lower=lo). The legacy intrinsic writes to BSR implicitly.
+ Value *Lo = II->getArgOperand(0);
+ Value *Hi = II->getArgOperand(1);
+ Builder.CreateCall(Intrinsic::getOrInsertDeclaration(
+ II->getModule(), Intrinsic::x86_bsrmovf),
+ {Hi, Lo});
+ // The result (x86_bsr SSA value) is only used by get_hi/get_lo/set_hi/
+ // set_lo which will also be lowered. For now, replace uses with poison.
+ II->replaceAllUsesWith(PoisonValue::get(II->getType()));
+ II->eraseFromParent();
+ return true;
+ }
+
+ case Intrinsic::x86_bsr_get_hi: {
+ // bsr_get_hi(bsr) -> bsrmovh_get()
+ Value *Result = Builder.CreateCall(Intrinsic::getOrInsertDeclaration(
+ II->getModule(), Intrinsic::x86_bsrmovh_get));
+ II->replaceAllUsesWith(Result);
+ II->eraseFromParent();
+ return true;
+ }
+
+ case Intrinsic::x86_bsr_get_lo: {
+ // bsr_get_lo(bsr) -> bsrmovl_get()
+ Value *Result = Builder.CreateCall(Intrinsic::getOrInsertDeclaration(
+ II->getModule(), Intrinsic::x86_bsrmovl_get));
+ II->replaceAllUsesWith(Result);
+ II->eraseFromParent();
+ return true;
+ }
+
+ case Intrinsic::x86_bsr_set_hi: {
+ // bsr_set_hi(bsr, val) -> bsrmovh_set(val)
+ Value *Val = II->getArgOperand(1);
+ Builder.CreateCall(Intrinsic::getOrInsertDeclaration(
+ II->getModule(), Intrinsic::x86_bsrmovh_set),
+ {Val});
+ II->replaceAllUsesWith(PoisonValue::get(II->getType()));
+ II->eraseFromParent();
+ return true;
+ }
+
+ case Intrinsic::x86_bsr_set_lo: {
+ // bsr_set_lo(bsr, val) -> bsrmovl_set(val)
+ Value *Val = II->getArgOperand(1);
+ Builder.CreateCall(Intrinsic::getOrInsertDeclaration(
+ II->getModule(), Intrinsic::x86_bsrmovl_set),
+ {Val});
+ II->replaceAllUsesWith(PoisonValue::get(II->getType()));
+ II->eraseFromParent();
+ return true;
+ }
+
+ case Intrinsic::x86_cast_vector_to_bsr: {
+ // cast_vector_to_bsr(vec) -> split vector and call bsrmovf
+ // For v32i32 (1024 bits), split into two v16i32 halves.
+ // Input vector layout: [lo=B-scales, hi=A-scales] from __bsr_combine_v32.
+ //
+ // If this cast's result has no uses (e.g., because all consumers already
+ // extracted the operands directly during their lowering), skip emitting
+ // bsrmovf to avoid duplicate instructions.
+ if (II->use_empty()) {
+ II->eraseFromParent();
+ return true;
+ }
+
+ Value *Vec = II->getArgOperand(0);
+ auto *VecTy = cast<FixedVectorType>(Vec->getType());
+ unsigned NumElts = VecTy->getNumElements();
+ Type *EltTy = VecTy->getElementType();
+ unsigned HalfElts = NumElts / 2;
+ auto *HalfVecTy = FixedVectorType::get(EltTy, HalfElts);
+
+ // Extract low and high halves using shufflevector.
+ // Lo = Vec[0:15] = B-scales (lower), Hi = Vec[16:31] = A-scales (upper).
+ SmallVector<int, 32> LoMask(HalfElts), HiMask(HalfElts);
+ for (unsigned i = 0; i < HalfElts; ++i) {
+ LoMask[i] = i;
+ HiMask[i] = i + HalfElts;
+ }
+ Value *Lo = Builder.CreateShuffleVector(Vec, LoMask);
+ Value *Hi = Builder.CreateShuffleVector(Vec, HiMask);
+
+ // Bitcast to v16i32 if needed.
+ auto *V16I32Ty = FixedVectorType::get(Builder.getInt32Ty(), 16);
+ if (HalfVecTy != V16I32Ty) {
+ Lo = Builder.CreateBitCast(Lo, V16I32Ty);
+ Hi = Builder.CreateBitCast(Hi, V16I32Ty);
+ }
+
+ // Per ACE spec: BSRMOVF(src1, src2) where src1→A-scales (upper=Hi),
+ // src2→B-scales (lower=Lo).
+ Builder.CreateCall(Intrinsic::getOrInsertDeclaration(
+ II->getModule(), Intrinsic::x86_bsrmovf),
+ {Hi, Lo});
+ II->replaceAllUsesWith(PoisonValue::get(II->getType()));
+ II->eraseFromParent();
+ return true;
+ }
+
+ case Intrinsic::x86_cast_bsr_to_vector: {
+ // cast_bsr_to_vector(bsr) -> get low + get high, combine
+ auto *V16I32Ty = FixedVectorType::get(Builder.getInt32Ty(), 16);
+ Value *Lo = Builder.CreateCall(Intrinsic::getOrInsertDeclaration(
+ II->getModule(), Intrinsic::x86_bsrmovl_get));
+ Value *Hi = Builder.CreateCall(Intrinsic::getOrInsertDeclaration(
+ II->getModule(), Intrinsic::x86_bsrmovh_get));
+
+ auto *RetTy = cast<FixedVectorType>(II->getType());
+ unsigned NumElts = RetTy->getNumElements();
+ unsigned HalfElts = NumElts / 2;
+
+ // Bitcast from v16i32 to the half-type if needed.
+ auto *HalfTy = FixedVectorType::get(RetTy->getElementType(), HalfElts);
+ if (V16I32Ty != HalfTy) {
+ Lo = Builder.CreateBitCast(Lo, HalfTy);
+ Hi = Builder.CreateBitCast(Hi, HalfTy);
+ }
+
+ // Combine via shufflevector.
+ SmallVector<int, 32> CombineMask(NumElts);
+ for (unsigned i = 0; i < HalfElts; ++i) {
+ CombineMask[i] = i;
+ CombineMask[i + HalfElts] = i + HalfElts;
+ }
+ Value *Result = Builder.CreateShuffleVector(Lo, Hi, CombineMask);
+ II->replaceAllUsesWith(Result);
+ II->eraseFromParent();
+ return true;
+ }
+
+ case Intrinsic::x86_top4mxhf8ps_bsr_internal:
+ case Intrinsic::x86_top4mxbhf8ps_bsr_internal:
+ case Intrinsic::x86_top4mxhbf8ps_bsr_internal:
+ case Intrinsic::x86_top4mxbf8ps_bsr_internal: {
+ // Lower BSR-carrying TOP4MX: emit bsrmovf before the call, replace with
+ // legacy (non-BSR) intrinsic call.
+ //
+ // Input: %result = call x86_amx @llvm.x86.top4mxhf8ps.bsr.internal(
+ // i16 %m, i16 %n, i16 %k, i8 %imm, x86_amx %acc,
+ // <16 x i32> %src1, <16 x i32> %src2, x86_bsr %bsr)
+ //
+ // Output: ; extract lo/hi from BSR source
+ // call void @llvm.x86.bsrmovf(<16 x i32> %lo, <16 x i32> %hi)
+ // %result = call x86_amx @llvm.x86.top4mxhf8ps.internal(
+ // i16 %m, i16 %n, i16 %k, i8 %imm, x86_amx %acc,
+ // <16 x i32> %src1, <16 x i32> %src2)
+
+ Value *BSRVal = II->getArgOperand(7); // x86_bsr argument
+
+ // Extract lo and hi vectors from the x86_bsr value.
+ Value *Lo = nullptr;
+ Value *Hi = nullptr;
+ auto *V16I32Ty = FixedVectorType::get(Builder.getInt32Ty(), 16);
+
+ if (auto *SrcII = dyn_cast<IntrinsicInst>(BSRVal)) {
+ if (SrcII->getIntrinsicID() == Intrinsic::x86_bsr_create) {
+ Lo = SrcII->getArgOperand(0);
+ Hi = SrcII->getArgOperand(1);
+ } else if (SrcII->getIntrinsicID() == Intrinsic::x86_cast_vector_to_bsr) {
+ // The cast_vector_to_bsr takes a single large vector and splits it.
+ // We need to do the same split here.
+ Value *Vec = SrcII->getArgOperand(0);
+ auto *VecTy = cast<FixedVectorType>(Vec->getType());
+ unsigned NumElts = VecTy->getNumElements();
+ unsigned HalfElts = NumElts / 2;
+
+ SmallVector<int, 32> LoMask(HalfElts), HiMask(HalfElts);
+ for (unsigned i = 0; i < HalfElts; ++i) {
+ LoMask[i] = i;
+ HiMask[i] = i + HalfElts;
+ }
+ Lo = Builder.CreateShuffleVector(Vec, LoMask);
+ Hi = Builder.CreateShuffleVector(Vec, HiMask);
+
+ auto *HalfVecTy =
+ FixedVectorType::get(VecTy->getElementType(), HalfElts);
+ if (HalfVecTy != V16I32Ty) {
+ Lo = Builder.CreateBitCast(Lo, V16I32Ty);
+ Hi = Builder.CreateBitCast(Hi, V16I32Ty);
+ }
+ }
+ }
+
+ if (!Lo || !Hi) {
+ // Fallback: emit bsr_get_lo/bsr_get_hi calls.
+ // These will be lowered by this same pass (they're also in
+ // BSRIntrinsics).
+ Lo = Builder.CreateCall(Intrinsic::getOrInsertDeclaration(
+ II->getModule(), Intrinsic::x86_bsr_get_lo),
+ {BSRVal});
+ Hi = Builder.CreateCall(Intrinsic::getOrInsertDeclaration(
+ II->getModule(), Intrinsic::x86_bsr_get_hi),
+ {BSRVal});
+ }
+
+ // Emit bsrmovf(hi, lo) immediately before the TOP4MX call.
+ // Per ACE spec: BSRMOVF(src1, src2) where src1→A-scales (upper=Hi),
+ // src2→B-scales (lower=Lo).
+ Builder.CreateCall(Intrinsic::getOrInsertDeclaration(
+ II->getModule(), Intrinsic::x86_bsrmovf),
+ {Hi, Lo});
+
+ // Create the legacy (non-BSR) TOP4MX intrinsic call with args 0-6.
+ Intrinsic::ID LegacyID = getBSRToLegacyID(II->getIntrinsicID());
+ SmallVector<Value *, 7> LegacyArgs;
+ for (unsigned i = 0; i < 7; ++i)
+ LegacyArgs.push_back(II->getArgOperand(i));
+
+ // The legacy intrinsic has an ImmArg at index 3; we need to use the
+ // overloaded declaration getter with no type overloads (non-overloaded).
+ Value *NewCall = Builder.CreateCall(
+ Intrinsic::getOrInsertDeclaration(II->getModule(), LegacyID),
+ LegacyArgs);
+
+ II->replaceAllUsesWith(NewCall);
+ II->eraseFromParent();
+ return true;
+ }
+ }
+}
+
+static bool runOnFunction(Function &F) {
+ bool Changed = false;
+ SmallVector<IntrinsicInst *, 8> BSRConsumers; // TOP4MX *_bsr_internal
+ SmallVector<IntrinsicInst *, 8> BSRProducers; // bsr_create, bsr_set_*, casts
+
+ for (auto &BB : F) {
+ for (auto &I : BB) {
+ if (auto *II = dyn_cast<IntrinsicInst>(&I)) {
+ if (!isBSRIntrinsic(II))
+ continue;
+ // Separate consumers (TOP4MX BSR) from producers (bsr_create etc.)
+ // so we lower consumers first (they need to read producer operands).
+ if (getBSRToLegacyID(II->getIntrinsicID()) != Intrinsic::not_intrinsic)
+ BSRConsumers.push_back(II);
+ else
+ BSRProducers.push_back(II);
+ }
+ }
+ }
+
+ if (BSRConsumers.empty() && BSRProducers.empty())
+ return false;
+
+ IRBuilder<> Builder(F.getContext());
+
+ // Lower consumers first: they may reference bsr_create operands that would
+ // be replaced with poison if we lowered producers first.
+ for (auto *II : BSRConsumers)
+ Changed |= lowerBSRIntrinsic(II, Builder);
+ for (auto *II : BSRProducers)
+ Changed |= lowerBSRIntrinsic(II, Builder);
+
+ return Changed;
+}
+
+class X86LowerBSRTypeLegacyPass : public FunctionPass {
+public:
+ static char ID;
+
+ X86LowerBSRTypeLegacyPass() : FunctionPass(ID) {
+ initializeX86LowerBSRTypeLegacyPassPass(*PassRegistry::getPassRegistry());
+ }
+
+ bool runOnFunction(Function &F) override { return ::runOnFunction(F); }
+
+ StringRef getPassName() const override { return "X86 Lower BSR Type"; }
+};
+
+} // anonymous namespace
+
+char X86LowerBSRTypeLegacyPass::ID = 0;
+
+INITIALIZE_PASS_BEGIN(X86LowerBSRTypeLegacyPass, DEBUG_TYPE,
+ "X86 Lower BSR Type", false, false)
+INITIALIZE_PASS_END(X86LowerBSRTypeLegacyPass, DEBUG_TYPE, "X86 Lower BSR Type",
+ false, false)
+
+FunctionPass *llvm::createX86LowerBSRTypeLegacyPass() {
+ return new X86LowerBSRTypeLegacyPass();
+}
+
+PreservedAnalyses X86LowerBSRTypePass::run(Function &F,
+ FunctionAnalysisManager &FAM) {
+ if (!runOnFunction(F))
+ return PreservedAnalyses::all();
+ return PreservedAnalyses::none();
+}
diff --git a/llvm/lib/Target/X86/X86RegisterInfo.cpp b/llvm/lib/Target/X86/X86RegisterInfo.cpp
index 6915a52708f7f..2fd97599dae7b 100644
--- a/llvm/lib/Target/X86/X86RegisterInfo.cpp
+++ b/llvm/lib/Target/X86/X86RegisterInfo.cpp
@@ -543,6 +543,9 @@ BitVector X86RegisterInfo::getReservedRegs(const MachineFunction &MF) const {
// Set the SIMD floating point control register as reserved.
Reserved.set(X86::MXCSR);
+ // Set the Block Scale Register as reserved (singleton hardware register).
+ Reserved.set(X86::BSR0);
+
// Set the stack-pointer register and its aliases as reserved.
for (const MCPhysReg &SubReg : subregs_inclusive(X86::RSP))
Reserved.set(SubReg);
diff --git a/llvm/lib/Target/X86/X86RegisterInfo.td b/llvm/lib/Target/X86/X86RegisterInfo.td
index 17c003d660d3c..abede7c21516d 100644
--- a/llvm/lib/Target/X86/X86RegisterInfo.td
+++ b/llvm/lib/Target/X86/X86RegisterInfo.td
@@ -861,7 +861,7 @@ def TILE : RegisterClass<"X86", [x86amx], 8192,
// Block Scale Registers (ACE)
let CopyCost = -1 in // Don't allow copying of BSR register
-def BSR : RegisterClass<"X86", [untyped], 0, (add BSR0)> {
+def BSR : RegisterClass<"X86", [x86bsr], 1024, (add BSR0)> {
let Size = 1024;
let isAllocatable = 0;
}
diff --git a/llvm/lib/Target/X86/X86TargetMachine.cpp b/llvm/lib/Target/X86/X86TargetMachine.cpp
index 932669b5cbac6..df20adafaffd1 100644
--- a/llvm/lib/Target/X86/X86TargetMachine.cpp
+++ b/llvm/lib/Target/X86/X86TargetMachine.cpp
@@ -71,6 +71,7 @@ extern "C" LLVM_C_ABI void LLVMInitializeX86Target() {
PassRegistry &PR = *PassRegistry::getPassRegistry();
initializeX86LowerAMXIntrinsicsLegacyPassPass(PR);
initializeX86LowerAMXTypeLegacyPassPass(PR);
+ initializeX86LowerBSRTypeLegacyPassPass(PR);
initializeX86PreTileConfigLegacyPass(PR);
initializeGlobalISel(PR);
initializeWinEHStateLegacyPass(PR);
@@ -423,7 +424,8 @@ void X86PassConfig::addIRPasses() {
// We add both pass anyway and when these two passes run, we skip the pass
// based on the option level and option attribute.
addPass(createX86LowerAMXIntrinsicsLegacyPass());
- addPass(createX86LowerAMXTypeLegacyPass());
+ addPass(createX86LowerBSRTypeLegacyPass()); // BSR lowering FIRST
+ addPass(createX86LowerAMXTypeLegacyPass()); // Then AMX type lowering
TargetPassConfig::addIRPasses();
diff --git a/llvm/lib/Transforms/InstCombine/InstCombineCasts.cpp b/llvm/lib/Transforms/InstCombine/InstCombineCasts.cpp
index f630d1efbd7e7..bbe60dbe6c007 100644
--- a/llvm/lib/Transforms/InstCombine/InstCombineCasts.cpp
+++ b/llvm/lib/Transforms/InstCombine/InstCombineCasts.cpp
@@ -3225,7 +3225,7 @@ Instruction *InstCombinerImpl::optimizeBitCastFromPhi(CastInst &CI,
// "load x86_amx, x86_amx*", because x86_amx* is invalid.
// TODO: Remove this check when bitcast between vector and x86_amx
// is replaced with a specific intrinsic.
- if (DestTy->isX86_AMXTy())
+ if (DestTy->isX86_AMXTy() || DestTy->isX86_BSRTy())
return nullptr;
if (LI->hasOneUse() && LI->isSimple())
continue;
diff --git a/llvm/lib/Transforms/InstCombine/InstCombineLoadStoreAlloca.cpp b/llvm/lib/Transforms/InstCombine/InstCombineLoadStoreAlloca.cpp
index 99ed22c97c3a8..fc94d972288c0 100644
--- a/llvm/lib/Transforms/InstCombine/InstCombineLoadStoreAlloca.cpp
+++ b/llvm/lib/Transforms/InstCombine/InstCombineLoadStoreAlloca.cpp
@@ -700,7 +700,8 @@ static Instruction *combineLoadToOperationType(InstCombinerImpl &IC,
Type *LoadTy = Load.getType();
if (auto *BC = dyn_cast<BitCastInst>(Load.user_back())) {
assert(!LoadTy->isX86_AMXTy() && "Load from x86_amx* should not happen!");
- if (BC->getType()->isX86_AMXTy())
+ assert(!LoadTy->isX86_BSRTy() && "Load from x86_bsr* should not happen!");
+ if (BC->getType()->isX86_AMXTy() || BC->getType()->isX86_BSRTy())
return nullptr;
}
@@ -1301,10 +1302,12 @@ static bool combineStoreToValueType(InstCombinerImpl &IC, StoreInst &SI) {
if (auto *BC = dyn_cast<BitCastInst>(V)) {
assert(!BC->getType()->isX86_AMXTy() &&
"store to x86_amx* should not happen!");
+ assert(!BC->getType()->isX86_BSRTy() &&
+ "store to x86_bsr* should not happen!");
V = BC->getOperand(0);
- // Don't transform when the type is x86_amx, it makes the pass that lower
- // x86_amx type happy.
- if (V->getType()->isX86_AMXTy())
+ // Don't transform when the type is x86_amx/x86_bsr, it makes the pass
+ // that lowers these types happy.
+ if (V->getType()->isX86_AMXTy() || V->getType()->isX86_BSRTy())
return false;
if (!SI.isAtomic() || isSupportedAtomicType(V->getType())) {
combineStoreToNewValue(IC, SI, V);
diff --git a/llvm/test/Assembler/x86_bsr.ll b/llvm/test/Assembler/x86_bsr.ll
new file mode 100644
index 0000000000000..6df9f6b217128
--- /dev/null
+++ b/llvm/test/Assembler/x86_bsr.ll
@@ -0,0 +1,40 @@
+; RUN: llvm-as < %s | llvm-dis | FileCheck %s
+
+; Test that x86_bsr type round-trips through bitcode correctly.
+; x86_bsr can only appear in intrinsic signatures, so we test it via
+; intrinsic calls inside regular functions that use normal types in
+; their own signatures.
+
+; CHECK-LABEL: define void @test_bsr_round_trip(<16 x i32> %lo, <16 x i32> %hi)
+define void @test_bsr_round_trip(<16 x i32> %lo, <16 x i32> %hi) {
+; CHECK: %bsr = call x86_bsr @llvm.x86.bsr.create(<16 x i32> %lo, <16 x i32> %hi)
+ %bsr = call x86_bsr @llvm.x86.bsr.create(<16 x i32> %lo, <16 x i32> %hi)
+; CHECK: %lo.out = call <16 x i32> @llvm.x86.bsr.get.lo(x86_bsr %bsr)
+ %lo.out = call <16 x i32> @llvm.x86.bsr.get.lo(x86_bsr %bsr)
+; CHECK: %hi.out = call <16 x i32> @llvm.x86.bsr.get.hi(x86_bsr %bsr)
+ %hi.out = call <16 x i32> @llvm.x86.bsr.get.hi(x86_bsr %bsr)
+ ret void
+}
+
+; CHECK-LABEL: define void @test_bsr_set(<16 x i32> %new_lo, <16 x i32> %new_hi)
+define void @test_bsr_set(<16 x i32> %new_lo, <16 x i32> %new_hi) {
+; CHECK: %bsr = call x86_bsr @llvm.x86.bsr.create(<16 x i32> %new_lo, <16 x i32> %new_hi)
+ %bsr = call x86_bsr @llvm.x86.bsr.create(<16 x i32> %new_lo, <16 x i32> %new_hi)
+; CHECK: %bsr2 = call x86_bsr @llvm.x86.bsr.set.lo(x86_bsr %bsr, <16 x i32> %new_lo)
+ %bsr2 = call x86_bsr @llvm.x86.bsr.set.lo(x86_bsr %bsr, <16 x i32> %new_lo)
+; CHECK: %bsr3 = call x86_bsr @llvm.x86.bsr.set.hi(x86_bsr %bsr2, <16 x i32> %new_hi)
+ %bsr3 = call x86_bsr @llvm.x86.bsr.set.hi(x86_bsr %bsr2, <16 x i32> %new_hi)
+ ret void
+}
+
+; CHECK: declare x86_bsr @llvm.x86.bsr.create(<16 x i32>, <16 x i32>)
+; CHECK: declare <16 x i32> @llvm.x86.bsr.get.lo(x86_bsr)
+; CHECK: declare <16 x i32> @llvm.x86.bsr.get.hi(x86_bsr)
+; CHECK: declare x86_bsr @llvm.x86.bsr.set.lo(x86_bsr, <16 x i32>)
+; CHECK: declare x86_bsr @llvm.x86.bsr.set.hi(x86_bsr, <16 x i32>)
+
+declare x86_bsr @llvm.x86.bsr.create(<16 x i32>, <16 x i32>)
+declare <16 x i32> @llvm.x86.bsr.get.lo(x86_bsr)
+declare <16 x i32> @llvm.x86.bsr.get.hi(x86_bsr)
+declare x86_bsr @llvm.x86.bsr.set.lo(x86_bsr, <16 x i32>)
+declare x86_bsr @llvm.x86.bsr.set.hi(x86_bsr, <16 x i32>)
diff --git a/llvm/test/CodeGen/X86/ACE/ace-bsr-ordering.ll b/llvm/test/CodeGen/X86/ACE/ace-bsr-ordering.ll
new file mode 100644
index 0000000000000..fa43d00a25417
--- /dev/null
+++ b/llvm/test/CodeGen/X86/ACE/ace-bsr-ordering.ll
@@ -0,0 +1,32 @@
+; RUN: opt -O2 -S < %s | FileCheck %s
+; Verify that BSR SSA threading prevents reordering of BSR write relative to TOP4MX compute.
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+; The x86_bsr argument creates a data dependency: the bsr_create result flows into
+; the TOP4MX call, so the optimizer cannot move bsr_create past the compute.
+
+define void @bsr_ordering_preserved(ptr %tile_ptr, ptr %zmm_ptr, ptr %bsr_ptr, i16 %m, i16 %n, i16 %k) #0 {
+; CHECK-LABEL: @bsr_ordering_preserved
+; Verify that bsr.create dominates top4mxhf8ps.bsr.internal (SSA guarantees this)
+; CHECK: %bsr = {{.*}}call x86_bsr @llvm.x86.bsr.create(
+; CHECK: {{.*}}call x86_amx @llvm.x86.top4mxhf8ps.bsr.internal({{.*}}, x86_bsr %bsr)
+entry:
+ %acc = call x86_amx @llvm.x86.tileloadd64.internal(i16 %m, i16 %n, ptr %tile_ptr, i64 64)
+ %zmm1 = load <16 x i32>, ptr %zmm_ptr, align 64
+ %bsr_lo = load <16 x i32>, ptr %bsr_ptr, align 64
+ %bsr_hi_ptr = getelementptr inbounds <16 x i32>, ptr %bsr_ptr, i64 1
+ %bsr_hi = load <16 x i32>, ptr %bsr_hi_ptr, align 64
+ %bsr = call x86_bsr @llvm.x86.bsr.create(<16 x i32> %bsr_lo, <16 x i32> %bsr_hi)
+ %result = call x86_amx @llvm.x86.top4mxhf8ps.bsr.internal(i16 %m, i16 %n, i16 %k, i8 0, x86_amx %acc, <16 x i32> %zmm1, <16 x i32> %zmm1, x86_bsr %bsr)
+ call void @llvm.x86.tilestored64.internal(i16 %m, i16 %n, ptr %tile_ptr, i64 64, x86_amx %result)
+ ret void
+}
+
+attributes #0 = { nounwind "target-features"="+ace,+avx512f,+amx-tile" }
+
+declare x86_amx @llvm.x86.tileloadd64.internal(i16, i16, ptr, i64)
+declare void @llvm.x86.tilestored64.internal(i16, i16, ptr, i64, x86_amx)
+declare x86_bsr @llvm.x86.bsr.create(<16 x i32>, <16 x i32>)
+declare x86_amx @llvm.x86.top4mxhf8ps.bsr.internal(i16, i16, i16, i8, x86_amx, <16 x i32>, <16 x i32>, x86_bsr)
diff --git a/llvm/test/CodeGen/X86/ACE/ace-internal-intrinsics.ll b/llvm/test/CodeGen/X86/ACE/ace-internal-intrinsics.ll
new file mode 100644
index 0000000000000..734d9335d80e1
--- /dev/null
+++ b/llvm/test/CodeGen/X86/ACE/ace-internal-intrinsics.ll
@@ -0,0 +1,176 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+acev1,+avx512f,+amx-tile | FileCheck %s
+
+;
+; ACE Internal Intrinsic Tests
+;
+; These test the internal intrinsics that go through X86LowerAMXType pass.
+; Internal intrinsics take explicit dimension parameters and x86_amx tile values.
+;
+
+; Test TOP4BUUD internal intrinsic
+define void @test_top4buud_internal(ptr %tile_ptr, ptr %zmm_ptr, i16 %m, i16 %n, i16 %k) {
+; CHECK-LABEL: test_top4buud_internal:
+; CHECK: # %bb.0:
+; CHECK: ldtilecfg
+; CHECK: tileloadd (%rdi,%rax), %tmm0
+; CHECK: vmovaps (%rsi), %zmm0
+; CHECK: top4buud %zmm0, %zmm0, %tmm0
+; CHECK: tilestored %tmm0, (%rdi,%rax)
+; CHECK: tilerelease
+; CHECK: retq
+ %acc = call x86_amx @llvm.x86.tileloadd64.internal(i16 %m, i16 %n, ptr %tile_ptr, i64 64)
+ %zmm1 = load <64 x i8>, ptr %zmm_ptr, align 64
+ %zmm2 = load <64 x i8>, ptr %zmm_ptr, align 64
+ %result = call x86_amx @llvm.x86.top4buud.internal(i16 %m, i16 %n, i16 %k, x86_amx %acc, <64 x i8> %zmm1, <64 x i8> %zmm2)
+ call void @llvm.x86.tilestored64.internal(i16 %m, i16 %n, ptr %tile_ptr, i64 64, x86_amx %result)
+ ret void
+}
+
+; Test TOP4BUSD internal intrinsic
+define void @test_top4busd_internal(ptr %tile_ptr, ptr %zmm_ptr, i16 %m, i16 %n, i16 %k) {
+; CHECK-LABEL: test_top4busd_internal:
+; CHECK: # %bb.0:
+; CHECK: ldtilecfg
+; CHECK: tileloadd (%rdi,%rax), %tmm0
+; CHECK: vmovaps (%rsi), %zmm0
+; CHECK: top4busd %zmm0, %zmm0, %tmm0
+; CHECK: tilestored %tmm0, (%rdi,%rax)
+; CHECK: tilerelease
+; CHECK: retq
+ %acc = call x86_amx @llvm.x86.tileloadd64.internal(i16 %m, i16 %n, ptr %tile_ptr, i64 64)
+ %zmm1 = load <64 x i8>, ptr %zmm_ptr, align 64
+ %zmm2 = load <64 x i8>, ptr %zmm_ptr, align 64
+ %result = call x86_amx @llvm.x86.top4busd.internal(i16 %m, i16 %n, i16 %k, x86_amx %acc, <64 x i8> %zmm1, <64 x i8> %zmm2)
+ call void @llvm.x86.tilestored64.internal(i16 %m, i16 %n, ptr %tile_ptr, i64 64, x86_amx %result)
+ ret void
+}
+
+; Test TOP4BSSD internal intrinsic
+define void @test_top4bssd_internal(ptr %tile_ptr, ptr %zmm_ptr, i16 %m, i16 %n, i16 %k) {
+; CHECK-LABEL: test_top4bssd_internal:
+; CHECK: # %bb.0:
+; CHECK: ldtilecfg
+; CHECK: tileloadd (%rdi,%rax), %tmm0
+; CHECK: vmovaps (%rsi), %zmm0
+; CHECK: top4bssd %zmm0, %zmm0, %tmm0
+; CHECK: tilestored %tmm0, (%rdi,%rax)
+; CHECK: tilerelease
+; CHECK: retq
+ %acc = call x86_amx @llvm.x86.tileloadd64.internal(i16 %m, i16 %n, ptr %tile_ptr, i64 64)
+ %zmm1 = load <64 x i8>, ptr %zmm_ptr, align 64
+ %zmm2 = load <64 x i8>, ptr %zmm_ptr, align 64
+ %result = call x86_amx @llvm.x86.top4bssd.internal(i16 %m, i16 %n, i16 %k, x86_amx %acc, <64 x i8> %zmm1, <64 x i8> %zmm2)
+ call void @llvm.x86.tilestored64.internal(i16 %m, i16 %n, ptr %tile_ptr, i64 64, x86_amx %result)
+ ret void
+}
+
+; Test TOP4BSUD internal intrinsic
+define void @test_top4bsud_internal(ptr %tile_ptr, ptr %zmm_ptr, i16 %m, i16 %n, i16 %k) {
+; CHECK-LABEL: test_top4bsud_internal:
+; CHECK: # %bb.0:
+; CHECK: ldtilecfg
+; CHECK: tileloadd (%rdi,%rax), %tmm0
+; CHECK: vmovaps (%rsi), %zmm0
+; CHECK: top4bsud %zmm0, %zmm0, %tmm0
+; CHECK: tilestored %tmm0, (%rdi,%rax)
+; CHECK: tilerelease
+; CHECK: retq
+ %acc = call x86_amx @llvm.x86.tileloadd64.internal(i16 %m, i16 %n, ptr %tile_ptr, i64 64)
+ %zmm1 = load <64 x i8>, ptr %zmm_ptr, align 64
+ %zmm2 = load <64 x i8>, ptr %zmm_ptr, align 64
+ %result = call x86_amx @llvm.x86.top4bsud.internal(i16 %m, i16 %n, i16 %k, x86_amx %acc, <64 x i8> %zmm1, <64 x i8> %zmm2)
+ call void @llvm.x86.tilestored64.internal(i16 %m, i16 %n, ptr %tile_ptr, i64 64, x86_amx %result)
+ ret void
+}
+
+; Test TOP4MXHF8PS internal intrinsic (with BSR index immediate)
+define void @test_top4mxhf8ps_internal(ptr %tile_ptr, ptr %zmm_ptr, i16 %m, i16 %n, i16 %k) {
+; CHECK-LABEL: test_top4mxhf8ps_internal:
+; CHECK: # %bb.0:
+; CHECK: ldtilecfg
+; CHECK: tileloadd (%rdi,%rax), %tmm0
+; CHECK: vmovaps (%rsi), %zmm0
+; CHECK: top4mxhf8ps $5, %zmm0, %zmm0, %tmm0
+; CHECK: tilestored %tmm0, (%rdi,%rax)
+; CHECK: tilerelease
+; CHECK: retq
+ %acc = call x86_amx @llvm.x86.tileloadd64.internal(i16 %m, i16 %n, ptr %tile_ptr, i64 64)
+ %zmm1 = load <16 x i32>, ptr %zmm_ptr, align 64
+ %zmm2 = load <16 x i32>, ptr %zmm_ptr, align 64
+ %result = call x86_amx @llvm.x86.top4mxhf8ps.internal(i16 %m, i16 %n, i16 %k, i8 5, x86_amx %acc, <16 x i32> %zmm1, <16 x i32> %zmm2)
+ call void @llvm.x86.tilestored64.internal(i16 %m, i16 %n, ptr %tile_ptr, i64 64, x86_amx %result)
+ ret void
+}
+
+; Test TOP4MXHF8PS BSR internal intrinsic (with explicit x86_bsr SSA arg)
+define void @test_top4mxhf8ps_bsr_internal(ptr %tile_ptr, ptr %zmm_ptr, ptr %bsr_ptr, i16 %m, i16 %n, i16 %k) {
+; CHECK-LABEL: test_top4mxhf8ps_bsr_internal:
+; CHECK: # %bb.0:
+; CHECK: ldtilecfg
+; CHECK: tileloadd (%rdi,%rax), %tmm0
+; CHECK: vmovaps (%rsi), %zmm0
+; CHECK-DAG: vmovaps (%rdx), %zmm
+; CHECK-DAG: vmovaps 64(%rdx), %zmm
+; CHECK: bsrmovf %zmm{{[0-9]+}}, %zmm{{[0-9]+}}
+; CHECK: top4mxhf8ps $5, %zmm0, %zmm0, %tmm0
+; CHECK: tilestored %tmm0, (%rdi,%rax)
+; CHECK: tilerelease
+; CHECK: retq
+ %acc = call x86_amx @llvm.x86.tileloadd64.internal(i16 %m, i16 %n, ptr %tile_ptr, i64 64)
+ %zmm1 = load <16 x i32>, ptr %zmm_ptr, align 64
+ %zmm2 = load <16 x i32>, ptr %zmm_ptr, align 64
+ %bsr_lo_ptr = getelementptr inbounds <16 x i32>, ptr %bsr_ptr, i64 0
+ %bsr_hi_ptr = getelementptr inbounds <16 x i32>, ptr %bsr_ptr, i64 1
+ %bsr_lo = load <16 x i32>, ptr %bsr_lo_ptr, align 64
+ %bsr_hi = load <16 x i32>, ptr %bsr_hi_ptr, align 64
+ %bsr = call x86_bsr @llvm.x86.bsr.create(<16 x i32> %bsr_lo, <16 x i32> %bsr_hi)
+ %result = call x86_amx @llvm.x86.top4mxhf8ps.bsr.internal(i16 %m, i16 %n, i16 %k, i8 5, x86_amx %acc, <16 x i32> %zmm1, <16 x i32> %zmm2, x86_bsr %bsr)
+ call void @llvm.x86.tilestored64.internal(i16 %m, i16 %n, ptr %tile_ptr, i64 64, x86_amx %result)
+ ret void
+}
+
+; Test TILEMOVCOL internal intrinsic
+define void @test_tilemovcol_internal(ptr %tile_ptr, ptr %zmm_ptr, i16 %m, i16 %n) {
+; CHECK-LABEL: test_tilemovcol_internal:
+; CHECK: # %bb.0:
+; CHECK: ldtilecfg
+; CHECK: vmovaps (%rsi), %zmm0
+; CHECK: tilemovcol %eax, %zmm0, %tmm0
+; CHECK: tilestored %tmm0, (%rdi,%rax)
+; CHECK: tilerelease
+; CHECK: retq
+ %zmm = load <16 x i32>, ptr %zmm_ptr, align 64
+ %result = call x86_amx @llvm.x86.tilesetcol.internal(i16 %m, i16 %n, <16 x i32> %zmm, i32 3)
+ call void @llvm.x86.tilestored64.internal(i16 %m, i16 %n, ptr %tile_ptr, i64 64, x86_amx %result)
+ ret void
+}
+
+; Test TILEMOVROW internal intrinsic (ACE direction: ZMM to tile)
+define void @test_tilemovrow_internal(ptr %tile_ptr, ptr %zmm_ptr, i16 %m, i16 %n) {
+; CHECK-LABEL: test_tilemovrow_internal:
+; CHECK: # %bb.0:
+; CHECK: ldtilecfg
+; CHECK: vmovaps (%rsi), %zmm0
+; CHECK: tilemovrow %eax, %zmm0, %tmm0
+; CHECK: tilestored %tmm0, (%rdi,%rax)
+; CHECK: tilerelease
+; CHECK: retq
+ %zmm = load <16 x i32>, ptr %zmm_ptr, align 64
+ %result = call x86_amx @llvm.x86.tilesetrow.internal(i16 %m, i16 %n, <16 x i32> %zmm, i32 5)
+ call void @llvm.x86.tilestored64.internal(i16 %m, i16 %n, ptr %tile_ptr, i64 64, x86_amx %result)
+ ret void
+}
+
+; Intrinsic declarations
+declare x86_amx @llvm.x86.tileloadd64.internal(i16, i16, ptr, i64)
+declare void @llvm.x86.tilestored64.internal(i16, i16, ptr, i64, x86_amx)
+declare x86_amx @llvm.x86.top4buud.internal(i16, i16, i16, x86_amx, <64 x i8>, <64 x i8>)
+declare x86_amx @llvm.x86.top4busd.internal(i16, i16, i16, x86_amx, <64 x i8>, <64 x i8>)
+declare x86_amx @llvm.x86.top4bssd.internal(i16, i16, i16, x86_amx, <64 x i8>, <64 x i8>)
+declare x86_amx @llvm.x86.top4bsud.internal(i16, i16, i16, x86_amx, <64 x i8>, <64 x i8>)
+declare x86_amx @llvm.x86.top4mxhf8ps.internal(i16, i16, i16, i8, x86_amx, <16 x i32>, <16 x i32>)
+declare x86_bsr @llvm.x86.bsr.create(<16 x i32>, <16 x i32>)
+declare x86_amx @llvm.x86.top4mxhf8ps.bsr.internal(i16, i16, i16, i8, x86_amx, <16 x i32>, <16 x i32>, x86_bsr)
+declare x86_amx @llvm.x86.tilesetcol.internal(i16, i16, <16 x i32>, i32)
+declare x86_amx @llvm.x86.tilesetrow.internal(i16, i16, <16 x i32>, i32)
diff --git a/llvm/test/CodeGen/X86/O0-pipeline.ll b/llvm/test/CodeGen/X86/O0-pipeline.ll
index e8a3084563573..79a22466c959d 100644
--- a/llvm/test/CodeGen/X86/O0-pipeline.ll
+++ b/llvm/test/CodeGen/X86/O0-pipeline.ll
@@ -22,6 +22,7 @@
; CHECK-NEXT: Expand IR instructions
; CHECK-NEXT: Expand Atomic instructions
; CHECK-NEXT: Lower AMX intrinsics
+; CHECK-NEXT: X86 Lower BSR Type
; CHECK-NEXT: Lower AMX type for load/store
; CHECK-NEXT: Module Verifier
; CHECK-NEXT: Lower Garbage Collection Instructions
diff --git a/llvm/test/CodeGen/X86/opt-pipeline.ll b/llvm/test/CodeGen/X86/opt-pipeline.ll
index 24390f2d852d3..b9b0feabd6191 100644
--- a/llvm/test/CodeGen/X86/opt-pipeline.ll
+++ b/llvm/test/CodeGen/X86/opt-pipeline.ll
@@ -37,6 +37,7 @@
; CHECK-NEXT: Expand IR instructions
; CHECK-NEXT: Expand Atomic instructions
; CHECK-NEXT: Lower AMX intrinsics
+; CHECK-NEXT: X86 Lower BSR Type
; CHECK-NEXT: Lower AMX type for load/store
; CHECK-NEXT: Module Verifier
; CHECK-NEXT: Dominator Tree Construction
diff --git a/llvm/test/Instrumentation/Instrumentor/alloca_and_function.ll b/llvm/test/Instrumentation/Instrumentor/alloca_and_function.ll
index a2afecda64ebf..5cc4ac2364bec 100644
--- a/llvm/test/Instrumentation/Instrumentor/alloca_and_function.ll
+++ b/llvm/test/Instrumentation/Instrumentor/alloca_and_function.ll
@@ -9,12 +9,14 @@ target datalayout = "e-m:o-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16
declare void @use(ptr)
+;.
; CHECK: @llvm.global_ctors = appending global [1 x { i32, ptr, ptr }] [{ i32, ptr, ptr } { i32 1000, ptr @__instrumentor_ctor, ptr null }]
; CHECK: @__instrumentor_.str = private unnamed_addr constant [8 x i8] c"<stdin>\00", align 1
; CHECK: @__instrumentor_.str.1 = private unnamed_addr constant [1 x i8] zeroinitializer, align 1
; CHECK: @llvm.global_dtors = appending global [1 x { i32, ptr, ptr }] [{ i32, ptr, ptr } { i32 1000, ptr @__instrumentor_dtor, ptr null }]
; CHECK: @__instrumentor_.str.2 = private unnamed_addr constant [4 x i8] c"foo\00", align 1
-; CHECK: @__instrumentor_value_pack = internal global <{ i32, i32, [6 x i8], i16, i32, i32, [4 x i8], float }> <{ i32 2, i32 12, [6 x i8] zeroinitializer, i16 0, i32 4, i32 2, [4 x i8] zeroinitializer, float 0.000000e+00 }>
+; CHECK: @__instrumentor_value_pack = internal global <{ i32, i32, [6 x i8], i16, i32, i32, [4 x i8], float }> <{ i32 2, i32 13, [6 x i8] zeroinitializer, i16 0, i32 4, i32 2, [4 x i8] zeroinitializer, float 0.000000e+00 }>
+;.
define float @foo(i16 %a, float %b) {
; CHECK-LABEL: define float @foo(
; CHECK-SAME: i16 [[A:%.*]], float [[B:%.*]]) {
@@ -35,9 +37,9 @@ define float @foo(i16 %a, float %b) {
; CHECK-NEXT: [[TMP9:%.*]] = call ptr @__instrumentor_post_alloca(ptr [[TMP8]], i64 2, i64 16, i32 -3) #[[ATTR1]]
; CHECK-NEXT: [[TMP15:%.*]] = call ptr @__instrumentor_post_base_pointer_info(ptr [[TMP9]], i32 2, i32 -4) #[[ATTR1]]
; CHECK-NEXT: [[TMP10:%.*]] = zext i16 [[TMP4]] to i64
-; CHECK-NEXT: [[TMP11:%.*]] = call ptr @__instrumentor_pre_store(ptr [[TMP9]], i32 0, ptr [[TMP15]], i64 [[TMP10]], i64 2, i64 2, i32 12, i32 -1, i32 0, i8 1, i8 0, i32 4) #[[ATTR1]]
+; CHECK-NEXT: [[TMP11:%.*]] = call ptr @__instrumentor_pre_store(ptr [[TMP9]], i32 0, ptr [[TMP15]], i64 [[TMP10]], i64 2, i64 2, i32 13, i32 -1, i32 0, i8 1, i8 0, i32 4) #[[ATTR1]]
; CHECK-NEXT: store i16 [[TMP4]], ptr [[TMP11]], align 2
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[TMP9]], i32 0, ptr [[TMP15]], i64 [[TMP10]], i64 2, i64 2, i32 12, i32 -1, i32 0, i8 1, i8 0, i32 -4) #[[ATTR1]]
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[TMP9]], i32 0, ptr [[TMP15]], i64 [[TMP10]], i64 2, i64 2, i32 13, i32 -1, i32 0, i8 1, i8 0, i32 -4) #[[ATTR1]]
; CHECK-NEXT: call void @use(ptr [[TMP9]])
; CHECK-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[TMP0]], ptr @__instrumentor_value_pack, i64 32, i1 false)
; CHECK-NEXT: [[TMP12:%.*]] = getelementptr inbounds nuw <{ i32, i32, [6 x i8], i16, i32, i32, [4 x i8], float }>, ptr [[TMP0]], i32 0, i32 3
@@ -53,5 +55,7 @@ entry:
call void @use(ptr %0)
ret float %b
}
+;.
; CHECK: attributes #[[ATTR0:[0-9]+]] = { nocallback nofree nosync nounwind willreturn memory(argmem: readwrite) }
; CHECK: attributes #[[ATTR1]] = { willreturn }
+;.
diff --git a/llvm/test/Instrumentation/Instrumentor/cast.ll b/llvm/test/Instrumentation/Instrumentor/cast.ll
index 77d9e0452784d..5b28c8c6f7d1a 100644
--- a/llvm/test/Instrumentation/Instrumentor/cast.ll
+++ b/llvm/test/Instrumentation/Instrumentor/cast.ll
@@ -13,9 +13,9 @@ define noundef i64 @test_ptrtoint_basic(ptr noundef %ptr) {
; CHECK-SAME: ptr noundef [[PTR:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = ptrtoint ptr [[PTR]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP0]], i32 15, i32 8, i32 12, i32 8, i32 48) #[[ATTR0:[0-9]+]]
+; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP0]], i32 16, i32 8, i32 13, i32 8, i32 48) #[[ATTR0:[0-9]+]]
; CHECK-NEXT: [[ADDR:%.*]] = ptrtoint ptr [[PTR]] to i64
-; CHECK-NEXT: [[TMP1:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP0]], i32 15, i32 8, i64 [[ADDR]], i32 12, i32 8, i32 48) #[[ATTR0]]
+; CHECK-NEXT: [[TMP1:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP0]], i32 16, i32 8, i64 [[ADDR]], i32 13, i32 8, i32 48) #[[ATTR0]]
; CHECK-NEXT: ret i64 [[TMP1]]
;
entry:
@@ -28,10 +28,10 @@ define noundef ptr @test_inttoptr_basic(i64 noundef %addr) {
; CHECK-LABEL: define noundef ptr @test_inttoptr_basic(
; CHECK-SAME: i64 noundef [[ADDR:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[ADDR]], i32 12, i32 8, i32 15, i32 8, i32 50) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[ADDR]], i32 13, i32 8, i32 16, i32 8, i32 50) #[[ATTR0]]
; CHECK-NEXT: [[PTR:%.*]] = inttoptr i64 [[ADDR]] to ptr
; CHECK-NEXT: [[TMP0:%.*]] = ptrtoint ptr [[PTR]] to i64
-; CHECK-NEXT: [[TMP1:%.*]] = call i64 @__instrumentor_post_cast(i64 [[ADDR]], i32 12, i32 8, i64 [[TMP0]], i32 15, i32 8, i32 50) #[[ATTR0]]
+; CHECK-NEXT: [[TMP1:%.*]] = call i64 @__instrumentor_post_cast(i64 [[ADDR]], i32 13, i32 8, i64 [[TMP0]], i32 16, i32 8, i32 50) #[[ATTR0]]
; CHECK-NEXT: [[TMP2:%.*]] = inttoptr i64 [[TMP1]] to ptr
; CHECK-NEXT: ret ptr [[TMP2]]
;
@@ -45,10 +45,10 @@ define noundef i32 @test_trunc(i64 noundef %val) {
; CHECK-LABEL: define noundef i32 @test_trunc(
; CHECK-SAME: i64 noundef [[VAL:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[VAL]], i32 12, i32 8, i32 12, i32 4, i32 39) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[VAL]], i32 13, i32 8, i32 13, i32 4, i32 39) #[[ATTR0]]
; CHECK-NEXT: [[RESULT:%.*]] = trunc i64 [[VAL]] to i32
; CHECK-NEXT: [[TMP0:%.*]] = zext i32 [[RESULT]] to i64
-; CHECK-NEXT: [[TMP1:%.*]] = call i64 @__instrumentor_post_cast(i64 [[VAL]], i32 12, i32 8, i64 [[TMP0]], i32 12, i32 4, i32 39) #[[ATTR0]]
+; CHECK-NEXT: [[TMP1:%.*]] = call i64 @__instrumentor_post_cast(i64 [[VAL]], i32 13, i32 8, i64 [[TMP0]], i32 13, i32 4, i32 39) #[[ATTR0]]
; CHECK-NEXT: [[TMP2:%.*]] = trunc i64 [[TMP1]] to i32
; CHECK-NEXT: ret i32 [[TMP2]]
;
@@ -63,9 +63,9 @@ define noundef i64 @test_zext(i32 noundef %val) {
; CHECK-SAME: i32 noundef [[VAL:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = zext i32 [[VAL]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP0]], i32 12, i32 4, i32 12, i32 8, i32 40) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP0]], i32 13, i32 4, i32 13, i32 8, i32 40) #[[ATTR0]]
; CHECK-NEXT: [[RESULT:%.*]] = zext i32 [[VAL]] to i64
-; CHECK-NEXT: [[TMP1:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP0]], i32 12, i32 4, i64 [[RESULT]], i32 12, i32 8, i32 40) #[[ATTR0]]
+; CHECK-NEXT: [[TMP1:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP0]], i32 13, i32 4, i64 [[RESULT]], i32 13, i32 8, i32 40) #[[ATTR0]]
; CHECK-NEXT: ret i64 [[TMP1]]
;
entry:
@@ -79,9 +79,9 @@ define noundef i64 @test_sext(i32 noundef %val) {
; CHECK-SAME: i32 noundef [[VAL:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = zext i32 [[VAL]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP0]], i32 12, i32 4, i32 12, i32 8, i32 41) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP0]], i32 13, i32 4, i32 13, i32 8, i32 41) #[[ATTR0]]
; CHECK-NEXT: [[RESULT:%.*]] = sext i32 [[VAL]] to i64
-; CHECK-NEXT: [[TMP1:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP0]], i32 12, i32 4, i64 [[RESULT]], i32 12, i32 8, i32 41) #[[ATTR0]]
+; CHECK-NEXT: [[TMP1:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP0]], i32 13, i32 4, i64 [[RESULT]], i32 13, i32 8, i32 41) #[[ATTR0]]
; CHECK-NEXT: ret i64 [[TMP1]]
;
entry:
@@ -96,10 +96,10 @@ define noundef i32 @test_fptoui(float noundef %val) {
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = bitcast float [[VAL]] to i32
; CHECK-NEXT: [[TMP1:%.*]] = zext i32 [[TMP0]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP1]], i32 2, i32 4, i32 12, i32 4, i32 42) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP1]], i32 2, i32 4, i32 13, i32 4, i32 42) #[[ATTR0]]
; CHECK-NEXT: [[RESULT:%.*]] = fptoui float [[VAL]] to i32
; CHECK-NEXT: [[TMP2:%.*]] = zext i32 [[RESULT]] to i64
-; CHECK-NEXT: [[TMP3:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP1]], i32 2, i32 4, i64 [[TMP2]], i32 12, i32 4, i32 42) #[[ATTR0]]
+; CHECK-NEXT: [[TMP3:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP1]], i32 2, i32 4, i64 [[TMP2]], i32 13, i32 4, i32 42) #[[ATTR0]]
; CHECK-NEXT: [[TMP4:%.*]] = trunc i64 [[TMP3]] to i32
; CHECK-NEXT: ret i32 [[TMP4]]
;
@@ -115,10 +115,10 @@ define noundef i32 @test_fptosi(float noundef %val) {
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = bitcast float [[VAL]] to i32
; CHECK-NEXT: [[TMP1:%.*]] = zext i32 [[TMP0]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP1]], i32 2, i32 4, i32 12, i32 4, i32 43) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP1]], i32 2, i32 4, i32 13, i32 4, i32 43) #[[ATTR0]]
; CHECK-NEXT: [[RESULT:%.*]] = fptosi float [[VAL]] to i32
; CHECK-NEXT: [[TMP2:%.*]] = zext i32 [[RESULT]] to i64
-; CHECK-NEXT: [[TMP3:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP1]], i32 2, i32 4, i64 [[TMP2]], i32 12, i32 4, i32 43) #[[ATTR0]]
+; CHECK-NEXT: [[TMP3:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP1]], i32 2, i32 4, i64 [[TMP2]], i32 13, i32 4, i32 43) #[[ATTR0]]
; CHECK-NEXT: [[TMP4:%.*]] = trunc i64 [[TMP3]] to i32
; CHECK-NEXT: ret i32 [[TMP4]]
;
@@ -133,11 +133,11 @@ define noundef float @test_uitofp(i32 noundef %val) {
; CHECK-SAME: i32 noundef [[VAL:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = zext i32 [[VAL]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP0]], i32 12, i32 4, i32 2, i32 4, i32 44) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP0]], i32 13, i32 4, i32 2, i32 4, i32 44) #[[ATTR0]]
; CHECK-NEXT: [[RESULT:%.*]] = uitofp i32 [[VAL]] to float
; CHECK-NEXT: [[TMP1:%.*]] = bitcast float [[RESULT]] to i32
; CHECK-NEXT: [[TMP2:%.*]] = zext i32 [[TMP1]] to i64
-; CHECK-NEXT: [[TMP3:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP0]], i32 12, i32 4, i64 [[TMP2]], i32 2, i32 4, i32 44) #[[ATTR0]]
+; CHECK-NEXT: [[TMP3:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP0]], i32 13, i32 4, i64 [[TMP2]], i32 2, i32 4, i32 44) #[[ATTR0]]
; CHECK-NEXT: [[TMP4:%.*]] = trunc i64 [[TMP3]] to i32
; CHECK-NEXT: [[TMP5:%.*]] = bitcast i32 [[TMP4]] to float
; CHECK-NEXT: ret float [[TMP5]]
@@ -153,11 +153,11 @@ define noundef float @test_sitofp(i32 noundef %val) {
; CHECK-SAME: i32 noundef [[VAL:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = zext i32 [[VAL]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP0]], i32 12, i32 4, i32 2, i32 4, i32 45) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP0]], i32 13, i32 4, i32 2, i32 4, i32 45) #[[ATTR0]]
; CHECK-NEXT: [[RESULT:%.*]] = sitofp i32 [[VAL]] to float
; CHECK-NEXT: [[TMP1:%.*]] = bitcast float [[RESULT]] to i32
; CHECK-NEXT: [[TMP2:%.*]] = zext i32 [[TMP1]] to i64
-; CHECK-NEXT: [[TMP3:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP0]], i32 12, i32 4, i64 [[TMP2]], i32 2, i32 4, i32 45) #[[ATTR0]]
+; CHECK-NEXT: [[TMP3:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP0]], i32 13, i32 4, i64 [[TMP2]], i32 2, i32 4, i32 45) #[[ATTR0]]
; CHECK-NEXT: [[TMP4:%.*]] = trunc i64 [[TMP3]] to i32
; CHECK-NEXT: [[TMP5:%.*]] = bitcast i32 [[TMP4]] to float
; CHECK-NEXT: ret float [[TMP5]]
@@ -212,10 +212,10 @@ define noundef ptr addrspace(1) @test_addrspacecast(ptr noundef %ptr) {
; CHECK-SAME: ptr noundef [[PTR:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = ptrtoint ptr [[PTR]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP0]], i32 15, i32 8, i32 15, i32 8, i32 52) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP0]], i32 16, i32 8, i32 16, i32 8, i32 52) #[[ATTR0]]
; CHECK-NEXT: [[RESULT:%.*]] = addrspacecast ptr [[PTR]] to ptr addrspace(1)
; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr addrspace(1) [[RESULT]] to i64
-; CHECK-NEXT: [[TMP2:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP0]], i32 15, i32 8, i64 [[TMP1]], i32 15, i32 8, i32 52) #[[ATTR0]]
+; CHECK-NEXT: [[TMP2:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP0]], i32 16, i32 8, i64 [[TMP1]], i32 16, i32 8, i32 52) #[[ATTR0]]
; CHECK-NEXT: [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr addrspace(1)
; CHECK-NEXT: ret ptr addrspace(1) [[TMP3]]
;
@@ -230,9 +230,9 @@ define noundef i64 @test_bitcast(double noundef %val) {
; CHECK-SAME: double noundef [[VAL:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = bitcast double [[VAL]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP0]], i32 3, i32 8, i32 12, i32 8, i32 51) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP0]], i32 3, i32 8, i32 13, i32 8, i32 51) #[[ATTR0]]
; CHECK-NEXT: [[RESULT:%.*]] = bitcast double [[VAL]] to i64
-; CHECK-NEXT: [[TMP1:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP0]], i32 3, i32 8, i64 [[RESULT]], i32 12, i32 8, i32 51) #[[ATTR0]]
+; CHECK-NEXT: [[TMP1:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP0]], i32 3, i32 8, i64 [[RESULT]], i32 13, i32 8, i32 51) #[[ATTR0]]
; CHECK-NEXT: ret i64 [[TMP1]]
;
entry:
@@ -246,20 +246,20 @@ define void @test_multiple_conversions(ptr noundef %p1, i64 noundef %i1, float n
; CHECK-SAME: ptr noundef [[P1:%.*]], i64 noundef [[I1:%.*]], float noundef [[F1:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = ptrtoint ptr [[P1]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP0]], i32 15, i32 8, i32 12, i32 8, i32 48) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP0]], i32 16, i32 8, i32 13, i32 8, i32 48) #[[ATTR0]]
; CHECK-NEXT: [[A1:%.*]] = ptrtoint ptr [[P1]] to i64
-; CHECK-NEXT: [[TMP1:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP0]], i32 15, i32 8, i64 [[A1]], i32 12, i32 8, i32 48) #[[ATTR0]]
-; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[I1]], i32 12, i32 8, i32 15, i32 8, i32 50) #[[ATTR0]]
+; CHECK-NEXT: [[TMP1:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP0]], i32 16, i32 8, i64 [[A1]], i32 13, i32 8, i32 48) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[I1]], i32 13, i32 8, i32 16, i32 8, i32 50) #[[ATTR0]]
; CHECK-NEXT: [[P2:%.*]] = inttoptr i64 [[I1]] to ptr
; CHECK-NEXT: [[TMP2:%.*]] = ptrtoint ptr [[P2]] to i64
-; CHECK-NEXT: [[TMP3:%.*]] = call i64 @__instrumentor_post_cast(i64 [[I1]], i32 12, i32 8, i64 [[TMP2]], i32 15, i32 8, i32 50) #[[ATTR0]]
+; CHECK-NEXT: [[TMP3:%.*]] = call i64 @__instrumentor_post_cast(i64 [[I1]], i32 13, i32 8, i64 [[TMP2]], i32 16, i32 8, i32 50) #[[ATTR0]]
; CHECK-NEXT: [[TMP4:%.*]] = inttoptr i64 [[TMP3]] to ptr
; CHECK-NEXT: [[TMP5:%.*]] = bitcast float [[F1]] to i32
; CHECK-NEXT: [[TMP6:%.*]] = zext i32 [[TMP5]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP6]], i32 2, i32 4, i32 12, i32 4, i32 42) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP6]], i32 2, i32 4, i32 13, i32 4, i32 42) #[[ATTR0]]
; CHECK-NEXT: [[I2:%.*]] = fptoui float [[F1]] to i32
; CHECK-NEXT: [[TMP7:%.*]] = zext i32 [[I2]] to i64
-; CHECK-NEXT: [[TMP8:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP6]], i32 2, i32 4, i64 [[TMP7]], i32 12, i32 4, i32 42) #[[ATTR0]]
+; CHECK-NEXT: [[TMP8:%.*]] = call i64 @__instrumentor_post_cast(i64 [[TMP6]], i32 2, i32 4, i64 [[TMP7]], i32 13, i32 4, i32 42) #[[ATTR0]]
; CHECK-NEXT: [[TMP9:%.*]] = trunc i64 [[TMP8]] to i32
; CHECK-NEXT: call void @use_values(i64 [[TMP1]], ptr [[TMP4]], i32 [[TMP9]])
; CHECK-NEXT: ret void
@@ -278,11 +278,11 @@ define i128 @test_ext(i32 %p1) {
; CHECK-NEXT: [[TMP1:%.*]] = alloca i128, align 8
; CHECK-NEXT: [[TMP2:%.*]] = alloca i64, align 8
; CHECK-NEXT: [[TMP3:%.*]] = zext i32 [[P1]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP3]], i32 12, i32 4, i32 12, i32 16, i32 40) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP3]], i32 13, i32 4, i32 13, i32 16, i32 40) #[[ATTR0]]
; CHECK-NEXT: [[I1:%.*]] = zext i32 [[P1]] to i128
; CHECK-NEXT: store i64 [[TMP3]], ptr [[TMP2]], align 4
; CHECK-NEXT: store i128 [[I1]], ptr [[TMP1]], align 4
-; CHECK-NEXT: call void @__instrumentor_post_cast_ind(ptr [[TMP2]], i32 12, i32 4, ptr [[TMP1]], i32 12, i32 16, i32 40) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_cast_ind(ptr [[TMP2]], i32 13, i32 4, ptr [[TMP1]], i32 13, i32 16, i32 40) #[[ATTR0]]
; CHECK-NEXT: [[TMP4:%.*]] = load i128, ptr [[TMP1]], align 4
; CHECK-NEXT: ret i128 [[TMP4]]
;
diff --git a/llvm/test/Instrumentation/Instrumentor/cast_crash.ll b/llvm/test/Instrumentation/Instrumentor/cast_crash.ll
index f9bf05c9c0cca..a83ed859e7fe4 100644
--- a/llvm/test/Instrumentation/Instrumentor/cast_crash.ll
+++ b/llvm/test/Instrumentation/Instrumentor/cast_crash.ll
@@ -14,11 +14,11 @@ define i128 @test_ext(i32 %p1) {
; CHECK-NEXT: [[TMP4:%.*]] = getelementptr inbounds i8, ptr [[TMP1]], i32 12
; CHECK-NEXT: [[TMP5:%.*]] = load i32, ptr [[TMP4]], align 4
; CHECK-NEXT: [[TMP6:%.*]] = zext i32 [[TMP5]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP6]], i32 12, i32 -1, i32 4, i32 12, i32 -1, i32 16, i32 40, i32 3) #[[ATTR1]]
+; CHECK-NEXT: call void @__instrumentor_pre_cast(i64 [[TMP6]], i32 13, i32 -1, i32 4, i32 13, i32 -1, i32 16, i32 40, i32 3) #[[ATTR1]]
; CHECK-NEXT: [[I1:%.*]] = zext i32 [[TMP5]] to i128
; CHECK-NEXT: store i64 [[TMP6]], ptr [[TMP2]], align 4
; CHECK-NEXT: store i128 [[I1]], ptr [[TMP1]], align 4
-; CHECK-NEXT: call void @__instrumentor_post_cast_ind(ptr [[TMP2]], i32 12, i32 -1, i32 4, ptr [[TMP1]], i32 12, i32 -1, i32 16, i32 40, i32 -3) #[[ATTR1]]
+; CHECK-NEXT: call void @__instrumentor_post_cast_ind(ptr [[TMP2]], i32 13, i32 -1, i32 4, ptr [[TMP1]], i32 13, i32 -1, i32 16, i32 40, i32 -3) #[[ATTR1]]
; CHECK-NEXT: [[TMP7:%.*]] = load i128, ptr [[TMP1]], align 4
; CHECK-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[TMP1]], ptr @__instrumentor_value_pack, i64 16, i1 false)
; CHECK-NEXT: [[TMP8:%.*]] = getelementptr inbounds nuw <{ i32, i32, [4 x i8], i32 }>, ptr [[TMP1]], i32 0, i32 3
diff --git a/llvm/test/Instrumentation/Instrumentor/compare.ll b/llvm/test/Instrumentation/Instrumentor/compare.ll
index 2eba6da3c6b68..be1aa45f3cafd 100644
--- a/llvm/test/Instrumentation/Instrumentor/compare.ll
+++ b/llvm/test/Instrumentation/Instrumentor/compare.ll
@@ -9,10 +9,10 @@ define <128 x i1> @test_ivec_128(<128 x i32> %0, <128 x i32> %1) {
; CHECK-NEXT: [[TMP5:%.*]] = alloca <128 x i32>, align 512
; CHECK-NEXT: store <128 x i32> [[TMP0]], ptr [[TMP5]], align 512
; CHECK-NEXT: store <128 x i32> [[TMP1]], ptr [[TMP4]], align 512
-; CHECK-NEXT: call void @__instrumentor_pre_compare_ind(i32 18, i32 512, i32 55, ptr [[TMP5]], ptr [[TMP4]], i64 0, i32 1) #[[ATTR0:[0-9]+]]
+; CHECK-NEXT: call void @__instrumentor_pre_compare_ind(i32 19, i32 512, i32 55, ptr [[TMP5]], ptr [[TMP4]], i64 0, i32 1) #[[ATTR0:[0-9]+]]
; CHECK-NEXT: [[TMP6:%.*]] = icmp eq <128 x i32> [[TMP0]], [[TMP1]]
; CHECK-NEXT: store <128 x i1> [[TMP6]], ptr [[TMP3]], align 16
-; CHECK-NEXT: call void @__instrumentor_post_compare_ind(i32 18, i32 512, i32 55, ptr [[TMP5]], ptr [[TMP4]], i32 18, i32 16, ptr [[TMP3]], i64 0, i32 -1) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_compare_ind(i32 19, i32 512, i32 55, ptr [[TMP5]], ptr [[TMP4]], i32 19, i32 16, ptr [[TMP3]], i64 0, i32 -1) #[[ATTR0]]
; CHECK-NEXT: [[TMP7:%.*]] = load <128 x i1>, ptr [[TMP3]], align 16
; CHECK-NEXT: ret <128 x i1> [[TMP7]]
;
@@ -28,10 +28,10 @@ define <32 x i1> @test_ivec_32(<32 x i32> %0, <32 x i32> %1) {
; CHECK-NEXT: [[TMP5:%.*]] = alloca <32 x i32>, align 128
; CHECK-NEXT: store <32 x i32> [[TMP0]], ptr [[TMP5]], align 128
; CHECK-NEXT: store <32 x i32> [[TMP1]], ptr [[TMP4]], align 128
-; CHECK-NEXT: call void @__instrumentor_pre_compare_ind(i32 18, i32 128, i32 55, ptr [[TMP5]], ptr [[TMP4]], i64 0, i32 2) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_compare_ind(i32 19, i32 128, i32 55, ptr [[TMP5]], ptr [[TMP4]], i64 0, i32 2) #[[ATTR0]]
; CHECK-NEXT: [[TMP6:%.*]] = icmp eq <32 x i32> [[TMP0]], [[TMP1]]
; CHECK-NEXT: store <32 x i1> [[TMP6]], ptr [[TMP3]], align 4
-; CHECK-NEXT: call void @__instrumentor_post_compare_ind(i32 18, i32 128, i32 55, ptr [[TMP5]], ptr [[TMP4]], i32 18, i32 4, ptr [[TMP3]], i64 0, i32 -2) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_compare_ind(i32 19, i32 128, i32 55, ptr [[TMP5]], ptr [[TMP4]], i32 19, i32 4, ptr [[TMP3]], i64 0, i32 -2) #[[ATTR0]]
; CHECK-NEXT: [[TMP7:%.*]] = load <32 x i1>, ptr [[TMP3]], align 4
; CHECK-NEXT: ret <32 x i1> [[TMP7]]
;
@@ -44,10 +44,10 @@ define i1 @test_int_i1(i32 %0, i32 %1) {
; CHECK-SAME: i32 [[TMP0:%.*]], i32 [[TMP1:%.*]]) {
; CHECK-NEXT: [[TMP3:%.*]] = zext i32 [[TMP0]] to i64
; CHECK-NEXT: [[TMP4:%.*]] = zext i32 [[TMP1]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_compare(i32 12, i32 4, i32 55, i64 [[TMP3]], i64 [[TMP4]], i64 0, i32 3) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_compare(i32 13, i32 4, i32 55, i64 [[TMP3]], i64 [[TMP4]], i64 0, i32 3) #[[ATTR0]]
; CHECK-NEXT: [[TMP5:%.*]] = icmp ult i32 [[TMP0]], [[TMP1]]
; CHECK-NEXT: [[TMP6:%.*]] = zext i1 [[TMP5]] to i64
-; CHECK-NEXT: [[TMP7:%.*]] = call i64 @__instrumentor_post_compare(i32 12, i32 4, i32 55, i64 [[TMP3]], i64 [[TMP4]], i32 12, i32 1, i64 [[TMP6]], i64 0, i32 -3) #[[ATTR0]]
+; CHECK-NEXT: [[TMP7:%.*]] = call i64 @__instrumentor_post_compare(i32 13, i32 4, i32 55, i64 [[TMP3]], i64 [[TMP4]], i32 13, i32 1, i64 [[TMP6]], i64 0, i32 -3) #[[ATTR0]]
; CHECK-NEXT: [[TMP8:%.*]] = trunc i64 [[TMP7]] to i1
; CHECK-NEXT: ret i1 [[TMP8]]
;
@@ -63,10 +63,10 @@ define <128 x i1> @test_fvec_128(<128 x float> %0, <128 x float> %1) {
; CHECK-NEXT: [[TMP5:%.*]] = alloca <128 x float>, align 512
; CHECK-NEXT: store <128 x float> [[TMP0]], ptr [[TMP5]], align 512
; CHECK-NEXT: store <128 x float> [[TMP1]], ptr [[TMP4]], align 512
-; CHECK-NEXT: call void @__instrumentor_pre_compare_ind(i32 18, i32 512, i32 56, ptr [[TMP5]], ptr [[TMP4]], i64 0, i32 4) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_compare_ind(i32 19, i32 512, i32 56, ptr [[TMP5]], ptr [[TMP4]], i64 0, i32 4) #[[ATTR0]]
; CHECK-NEXT: [[TMP6:%.*]] = fcmp olt <128 x float> [[TMP0]], [[TMP1]]
; CHECK-NEXT: store <128 x i1> [[TMP6]], ptr [[TMP3]], align 16
-; CHECK-NEXT: call void @__instrumentor_post_compare_ind(i32 18, i32 512, i32 56, ptr [[TMP5]], ptr [[TMP4]], i32 18, i32 16, ptr [[TMP3]], i64 0, i32 -4) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_compare_ind(i32 19, i32 512, i32 56, ptr [[TMP5]], ptr [[TMP4]], i32 19, i32 16, ptr [[TMP3]], i64 0, i32 -4) #[[ATTR0]]
; CHECK-NEXT: [[TMP7:%.*]] = load <128 x i1>, ptr [[TMP3]], align 16
; CHECK-NEXT: ret <128 x i1> [[TMP7]]
;
@@ -82,10 +82,10 @@ define <32 x i1> @test_fvec_32(<32 x float> %0, <32 x float> %1) {
; CHECK-NEXT: [[TMP5:%.*]] = alloca <32 x float>, align 128
; CHECK-NEXT: store <32 x float> [[TMP0]], ptr [[TMP5]], align 128
; CHECK-NEXT: store <32 x float> [[TMP1]], ptr [[TMP4]], align 128
-; CHECK-NEXT: call void @__instrumentor_pre_compare_ind(i32 18, i32 128, i32 56, ptr [[TMP5]], ptr [[TMP4]], i64 2, i32 5) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_compare_ind(i32 19, i32 128, i32 56, ptr [[TMP5]], ptr [[TMP4]], i64 2, i32 5) #[[ATTR0]]
; CHECK-NEXT: [[TMP6:%.*]] = fcmp nnan oeq <32 x float> [[TMP0]], [[TMP1]]
; CHECK-NEXT: store <32 x i1> [[TMP6]], ptr [[TMP3]], align 4
-; CHECK-NEXT: call void @__instrumentor_post_compare_ind(i32 18, i32 128, i32 56, ptr [[TMP5]], ptr [[TMP4]], i32 18, i32 4, ptr [[TMP3]], i64 2, i32 -5) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_compare_ind(i32 19, i32 128, i32 56, ptr [[TMP5]], ptr [[TMP4]], i32 19, i32 4, ptr [[TMP3]], i64 2, i32 -5) #[[ATTR0]]
; CHECK-NEXT: [[TMP7:%.*]] = load <32 x i1>, ptr [[TMP3]], align 4
; CHECK-NEXT: ret <32 x i1> [[TMP7]]
;
@@ -103,7 +103,7 @@ define i1 @test_float_i1(float %0, float %1) {
; CHECK-NEXT: call void @__instrumentor_pre_compare(i32 2, i32 4, i32 56, i64 [[TMP4]], i64 [[TMP6]], i64 14, i32 6) #[[ATTR0]]
; CHECK-NEXT: [[TMP7:%.*]] = fcmp fast ueq float [[TMP0]], [[TMP1]]
; CHECK-NEXT: [[TMP8:%.*]] = zext i1 [[TMP7]] to i64
-; CHECK-NEXT: [[TMP9:%.*]] = call i64 @__instrumentor_post_compare(i32 2, i32 4, i32 56, i64 [[TMP4]], i64 [[TMP6]], i32 12, i32 1, i64 [[TMP8]], i64 14, i32 -6) #[[ATTR0]]
+; CHECK-NEXT: [[TMP9:%.*]] = call i64 @__instrumentor_post_compare(i32 2, i32 4, i32 56, i64 [[TMP4]], i64 [[TMP6]], i32 13, i32 1, i64 [[TMP8]], i64 14, i32 -6) #[[ATTR0]]
; CHECK-NEXT: [[TMP10:%.*]] = trunc i64 [[TMP9]] to i1
; CHECK-NEXT: ret i1 [[TMP10]]
;
diff --git a/llvm/test/Instrumentation/Instrumentor/load_store.ll b/llvm/test/Instrumentation/Instrumentor/load_store.ll
index 6cfdaaaa85cb5..9528b9122dc3a 100644
--- a/llvm/test/Instrumentation/Instrumentor/load_store.ll
+++ b/llvm/test/Instrumentation/Instrumentor/load_store.ll
@@ -8,14 +8,14 @@ define noundef zeroext i1 @_Z15store_load_boolPb(ptr captures(none) noundef init
; CHECK-LABEL: define noundef zeroext i1 @_Z15store_load_boolPb(
; CHECK-SAME: ptr noundef captures(none) initializes((0, 1)) [[A:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: [[TMP0:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 1, i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP0:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 1, i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0:[0-9]+]]
; CHECK-NEXT: store i8 1, ptr [[TMP0]], align 1
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 1, i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 1, i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 1
-; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP2:%.*]] = load i8, ptr [[TMP1]], align 1
; CHECK-NEXT: [[TMP3:%.*]] = zext i8 [[TMP2]] to i64
-; CHECK-NEXT: [[TMP4:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP3]], i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP4:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP3]], i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP5:%.*]] = trunc i64 [[TMP4]] to i8
; CHECK-NEXT: [[LOADEDV:%.*]] = trunc nuw i8 [[TMP5]] to i1
; CHECK-NEXT: ret i1 [[LOADEDV]]
@@ -33,14 +33,14 @@ define noundef signext i8 @_Z15store_load_charPc(ptr captures(none) noundef init
; CHECK-LABEL: define noundef signext i8 @_Z15store_load_charPc(
; CHECK-SAME: ptr noundef captures(none) initializes((0, 1)) [[A:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: [[TMP0:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 1, i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP0:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 1, i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store i8 1, ptr [[TMP0]], align 1
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 1, i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 1, i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 1
-; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP2:%.*]] = load i8, ptr [[TMP1]], align 1
; CHECK-NEXT: [[TMP3:%.*]] = zext i8 [[TMP2]] to i64
-; CHECK-NEXT: [[TMP4:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP3]], i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP4:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP3]], i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP5:%.*]] = trunc i64 [[TMP4]] to i8
; CHECK-NEXT: ret i8 [[TMP5]]
;
@@ -56,14 +56,14 @@ define noundef signext i16 @_Z16store_load_shortPs(ptr captures(none) noundef in
; CHECK-LABEL: define noundef signext i16 @_Z16store_load_shortPs(
; CHECK-SAME: ptr noundef captures(none) initializes((0, 2)) [[A:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: [[TMP0:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 2, i64 2, i64 2, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP0:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 2, i64 2, i64 2, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store i16 2, ptr [[TMP0]], align 2
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 2, i64 2, i64 2, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 2, i64 2, i64 2, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 2
-; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 2, i64 2, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 2, i64 2, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP2:%.*]] = load i16, ptr [[TMP1]], align 2
; CHECK-NEXT: [[TMP3:%.*]] = zext i16 [[TMP2]] to i64
-; CHECK-NEXT: [[TMP4:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP3]], i64 2, i64 2, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP4:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP3]], i64 2, i64 2, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP5:%.*]] = trunc i64 [[TMP4]] to i16
; CHECK-NEXT: ret i16 [[TMP5]]
;
@@ -79,14 +79,14 @@ define noundef i32 @_Z14store_load_intPi(ptr captures(none) noundef initializes(
; CHECK-LABEL: define noundef i32 @_Z14store_load_intPi(
; CHECK-SAME: ptr noundef captures(none) initializes((0, 4)) [[A:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: [[TMP0:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 3, i64 4, i64 4, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP0:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 3, i64 4, i64 4, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store i32 3, ptr [[TMP0]], align 4
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 3, i64 4, i64 4, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 3, i64 4, i64 4, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 4
-; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 4, i64 4, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 4, i64 4, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP2:%.*]] = load i32, ptr [[TMP1]], align 4
; CHECK-NEXT: [[TMP3:%.*]] = zext i32 [[TMP2]] to i64
-; CHECK-NEXT: [[TMP4:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP3]], i64 4, i64 4, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP4:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP3]], i64 4, i64 4, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP5:%.*]] = trunc i64 [[TMP4]] to i32
; CHECK-NEXT: ret i32 [[TMP5]]
;
@@ -102,13 +102,13 @@ define noundef i64 @_Z15store_load_longPl(ptr captures(none) noundef initializes
; CHECK-LABEL: define noundef i64 @_Z15store_load_longPl(
; CHECK-SAME: ptr noundef captures(none) initializes((0, 8)) [[A:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: [[TMP0:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 4, i64 8, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP0:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 4, i64 8, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store i64 4, ptr [[TMP0]], align 8
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 4, i64 8, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 4, i64 8, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 8
-; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 8, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 8, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP2:%.*]] = load i64, ptr [[TMP1]], align 8
-; CHECK-NEXT: [[TMP3:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP2]], i64 8, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP3:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP2]], i64 8, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: ret i64 [[TMP3]]
;
entry:
@@ -125,14 +125,14 @@ define noundef i128 @_Z20store_load_long_longPx(ptr captures(none) noundef initi
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = alloca i128, align 16
; CHECK-NEXT: store i128 5, ptr [[TMP0]], align 16
-; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store i128 5, ptr [[TMP1]], align 8
-; CHECK-NEXT: call void @__instrumentor_post_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 16
-; CHECK-NEXT: [[TMP2:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 16, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP2:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 16, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP3:%.*]] = load i128, ptr [[TMP2]], align 8
; CHECK-NEXT: store i128 [[TMP3]], ptr [[TMP0]], align 16
-; CHECK-NEXT: call void @__instrumentor_post_load_ind(ptr [[ARRAYIDX]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_load_ind(ptr [[ARRAYIDX]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP4:%.*]] = load i128, ptr [[TMP0]], align 16
; CHECK-NEXT: ret i128 [[TMP4]]
;
@@ -148,15 +148,15 @@ define noundef float @_Z16store_load_floatPf(ptr captures(none) noundef initiali
; CHECK-LABEL: define noundef float @_Z16store_load_floatPf(
; CHECK-SAME: ptr noundef captures(none) initializes((0, 4)) [[A:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: [[TMP0:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 1086324736, i64 4, i64 4, i32 2, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP0:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 1086324736, i64 4, i64 4, i32 2, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store float 6.000000e+00, ptr [[TMP0]], align 4
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 1086324736, i64 4, i64 4, i32 2, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 1086324736, i64 4, i64 4, i32 2, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 4
-; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 4, i64 4, i32 2, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 4, i64 4, i32 2, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP2:%.*]] = load float, ptr [[TMP1]], align 4
; CHECK-NEXT: [[TMP3:%.*]] = bitcast float [[TMP2]] to i32
; CHECK-NEXT: [[TMP4:%.*]] = zext i32 [[TMP3]] to i64
-; CHECK-NEXT: [[TMP5:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP4]], i64 4, i64 4, i32 2, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP5:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP4]], i64 4, i64 4, i32 2, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP6:%.*]] = trunc i64 [[TMP5]] to i32
; CHECK-NEXT: [[TMP7:%.*]] = bitcast i32 [[TMP6]] to float
; CHECK-NEXT: ret float [[TMP7]]
@@ -173,14 +173,14 @@ define noundef double @_Z17store_load_doublePd(ptr captures(none) noundef initia
; CHECK-LABEL: define noundef double @_Z17store_load_doublePd(
; CHECK-SAME: ptr noundef captures(none) initializes((0, 8)) [[A:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: [[TMP0:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 4619567317775286272, i64 8, i64 8, i32 3, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP0:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 4619567317775286272, i64 8, i64 8, i32 3, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store double 7.000000e+00, ptr [[TMP0]], align 8
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 4619567317775286272, i64 8, i64 8, i32 3, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 4619567317775286272, i64 8, i64 8, i32 3, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 8
-; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 8, i64 8, i32 3, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 8, i64 8, i32 3, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP2:%.*]] = load double, ptr [[TMP1]], align 8
; CHECK-NEXT: [[TMP3:%.*]] = bitcast double [[TMP2]] to i64
-; CHECK-NEXT: [[TMP4:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP3]], i64 8, i64 8, i32 3, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP4:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP3]], i64 8, i64 8, i32 3, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP5:%.*]] = bitcast i64 [[TMP4]] to double
; CHECK-NEXT: ret double [[TMP5]]
;
@@ -198,14 +198,14 @@ define noundef x86_fp80 @_Z22store_load_long_doublePe(ptr captures(none) noundef
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = alloca x86_fp80, align 16
; CHECK-NEXT: store x86_fp80 8.000000e+00, ptr [[TMP0]], align 16
-; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store x86_fp80 8.000000e+00, ptr [[TMP1]], align 16
-; CHECK-NEXT: call void @__instrumentor_post_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 16
-; CHECK-NEXT: [[TMP2:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 10, i64 16, i32 4, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP2:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 10, i64 16, i32 4, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP3:%.*]] = load x86_fp80, ptr [[TMP2]], align 16
; CHECK-NEXT: store x86_fp80 [[TMP3]], ptr [[TMP0]], align 16
-; CHECK-NEXT: call void @__instrumentor_post_load_ind(ptr [[ARRAYIDX]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_load_ind(ptr [[ARRAYIDX]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP4:%.*]] = load x86_fp80, ptr [[TMP0]], align 16
; CHECK-NEXT: ret x86_fp80 [[TMP4]]
;
diff --git a/llvm/test/Instrumentation/Instrumentor/load_store_args.ll b/llvm/test/Instrumentation/Instrumentor/load_store_args.ll
index aeda9bad5c30f..cb7bf92ff5ec7 100644
--- a/llvm/test/Instrumentation/Instrumentor/load_store_args.ll
+++ b/llvm/test/Instrumentation/Instrumentor/load_store_args.ll
@@ -10,14 +10,14 @@ define noundef zeroext i1 @_Z15store_load_boolPbb(ptr captures(none) noundef ini
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[STOREDV:%.*]] = zext i1 [[VAL]] to i8
; CHECK-NEXT: [[TMP0:%.*]] = zext i8 [[STOREDV]] to i64
-; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0:[0-9]+]]
; CHECK-NEXT: store i8 [[STOREDV]], ptr [[TMP1]], align 1
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 1
-; CHECK-NEXT: [[TMP2:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP2:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP3:%.*]] = load i8, ptr [[TMP2]], align 1
; CHECK-NEXT: [[TMP4:%.*]] = zext i8 [[TMP3]] to i64
-; CHECK-NEXT: [[TMP5:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP4]], i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP5:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP4]], i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP7:%.*]] = trunc i64 [[TMP5]] to i8
; CHECK-NEXT: [[LOADEDV2:%.*]] = trunc nuw i8 [[TMP7]] to i1
; CHECK-NEXT: ret i1 [[LOADEDV2]]
@@ -37,14 +37,14 @@ define noundef signext i8 @_Z15store_load_charPcc(ptr captures(none) noundef ini
; CHECK-SAME: ptr noundef captures(none) initializes((0, 1)) [[A:%.*]], i8 noundef signext [[VAL:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = zext i8 [[VAL]] to i64
-; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store i8 [[VAL]], ptr [[TMP1]], align 1
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 1
-; CHECK-NEXT: [[TMP2:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP2:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP3:%.*]] = load i8, ptr [[TMP2]], align 1
; CHECK-NEXT: [[TMP4:%.*]] = zext i8 [[TMP3]] to i64
-; CHECK-NEXT: [[TMP5:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP4]], i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP5:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP4]], i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP7:%.*]] = trunc i64 [[TMP5]] to i8
; CHECK-NEXT: ret i8 [[TMP7]]
;
@@ -61,14 +61,14 @@ define noundef signext i16 @_Z16store_load_shortPss(ptr captures(none) noundef i
; CHECK-SAME: ptr noundef captures(none) initializes((0, 2)) [[A:%.*]], i16 noundef signext [[VAL:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = zext i16 [[VAL]] to i64
-; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 2, i64 2, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 2, i64 2, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store i16 [[VAL]], ptr [[TMP1]], align 2
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 2, i64 2, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 2, i64 2, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 2
-; CHECK-NEXT: [[TMP2:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 2, i64 2, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP2:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 2, i64 2, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP3:%.*]] = load i16, ptr [[TMP2]], align 2
; CHECK-NEXT: [[TMP4:%.*]] = zext i16 [[TMP3]] to i64
-; CHECK-NEXT: [[TMP5:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP4]], i64 2, i64 2, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP5:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP4]], i64 2, i64 2, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP7:%.*]] = trunc i64 [[TMP5]] to i16
; CHECK-NEXT: ret i16 [[TMP7]]
;
@@ -85,14 +85,14 @@ define noundef i32 @_Z14store_load_intPii(ptr captures(none) noundef initializes
; CHECK-SAME: ptr noundef captures(none) initializes((0, 4)) [[A:%.*]], i32 noundef [[VAL:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = zext i32 [[VAL]] to i64
-; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 4, i64 4, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 4, i64 4, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store i32 [[VAL]], ptr [[TMP1]], align 4
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 4, i64 4, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 4, i64 4, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 4
-; CHECK-NEXT: [[TMP2:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 4, i64 4, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP2:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 4, i64 4, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP3:%.*]] = load i32, ptr [[TMP2]], align 4
; CHECK-NEXT: [[TMP4:%.*]] = zext i32 [[TMP3]] to i64
-; CHECK-NEXT: [[TMP5:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP4]], i64 4, i64 4, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP5:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP4]], i64 4, i64 4, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP7:%.*]] = trunc i64 [[TMP5]] to i32
; CHECK-NEXT: ret i32 [[TMP7]]
;
@@ -108,13 +108,13 @@ define noundef i64 @_Z15store_load_longPll(ptr captures(none) noundef initialize
; CHECK-LABEL: define noundef i64 @_Z15store_load_longPll(
; CHECK-SAME: ptr noundef captures(none) initializes((0, 8)) [[A:%.*]], i64 noundef [[VAL:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: [[TMP0:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 [[VAL]], i64 8, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP0:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 [[VAL]], i64 8, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store i64 [[VAL]], ptr [[TMP0]], align 8
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 [[VAL]], i64 8, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 [[VAL]], i64 8, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 8
-; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 8, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 8, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP2:%.*]] = load i64, ptr [[TMP1]], align 8
-; CHECK-NEXT: [[TMP4:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP2]], i64 8, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP4:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP2]], i64 8, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: ret i64 [[TMP4]]
;
entry:
@@ -131,14 +131,14 @@ define noundef i128 @_Z20store_load_long_longPxx(ptr captures(none) noundef init
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = alloca i128, align 16
; CHECK-NEXT: store i128 [[VAL]], ptr [[TMP0]], align 16
-; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store i128 [[VAL]], ptr [[TMP1]], align 8
-; CHECK-NEXT: call void @__instrumentor_post_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 16
-; CHECK-NEXT: [[TMP4:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 16, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP4:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 16, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP3:%.*]] = load i128, ptr [[TMP4]], align 8
; CHECK-NEXT: store i128 [[TMP3]], ptr [[TMP0]], align 16
-; CHECK-NEXT: call void @__instrumentor_post_load_ind(ptr [[ARRAYIDX]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_load_ind(ptr [[ARRAYIDX]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP2:%.*]] = load i128, ptr [[TMP0]], align 16
; CHECK-NEXT: ret i128 [[TMP2]]
;
@@ -156,15 +156,15 @@ define noundef float @_Z16store_load_floatPff(ptr captures(none) noundef initial
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = bitcast float [[VAL]] to i32
; CHECK-NEXT: [[TMP1:%.*]] = zext i32 [[TMP0]] to i64
-; CHECK-NEXT: [[TMP2:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 [[TMP1]], i64 4, i64 4, i32 2, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP2:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 [[TMP1]], i64 4, i64 4, i32 2, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store float [[VAL]], ptr [[TMP2]], align 4
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 [[TMP1]], i64 4, i64 4, i32 2, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 [[TMP1]], i64 4, i64 4, i32 2, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 4
-; CHECK-NEXT: [[TMP3:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 4, i64 4, i32 2, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP3:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 4, i64 4, i32 2, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP4:%.*]] = load float, ptr [[TMP3]], align 4
; CHECK-NEXT: [[TMP5:%.*]] = bitcast float [[TMP4]] to i32
; CHECK-NEXT: [[TMP6:%.*]] = zext i32 [[TMP5]] to i64
-; CHECK-NEXT: [[TMP7:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP6]], i64 4, i64 4, i32 2, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP7:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP6]], i64 4, i64 4, i32 2, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP8:%.*]] = trunc i64 [[TMP7]] to i32
; CHECK-NEXT: [[TMP10:%.*]] = bitcast i32 [[TMP8]] to float
; CHECK-NEXT: ret float [[TMP10]]
@@ -182,14 +182,14 @@ define noundef double @_Z17store_load_doublePdd(ptr captures(none) noundef initi
; CHECK-SAME: ptr noundef captures(none) initializes((0, 8)) [[A:%.*]], double noundef [[VAL:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = bitcast double [[VAL]] to i64
-; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 8, i64 8, i32 3, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 8, i64 8, i32 3, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store double [[VAL]], ptr [[TMP1]], align 8
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 8, i64 8, i32 3, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 [[TMP0]], i64 8, i64 8, i32 3, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 8
-; CHECK-NEXT: [[TMP2:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 8, i64 8, i32 3, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP2:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 8, i64 8, i32 3, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP3:%.*]] = load double, ptr [[TMP2]], align 8
; CHECK-NEXT: [[TMP4:%.*]] = bitcast double [[TMP3]] to i64
-; CHECK-NEXT: [[TMP5:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP4]], i64 8, i64 8, i32 3, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP5:%.*]] = call i64 @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP4]], i64 8, i64 8, i32 3, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP7:%.*]] = bitcast i64 [[TMP5]] to double
; CHECK-NEXT: ret double [[TMP7]]
;
@@ -207,14 +207,14 @@ define noundef x86_fp80 @_Z22store_load_long_doublePee(ptr captures(none) nounde
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = alloca x86_fp80, align 16
; CHECK-NEXT: store x86_fp80 [[VAL]], ptr [[TMP0]], align 16
-; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_pre_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store x86_fp80 [[VAL]], ptr [[TMP1]], align 16
-; CHECK-NEXT: call void @__instrumentor_post_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 16
-; CHECK-NEXT: [[TMP4:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 10, i64 16, i32 4, i32 0, i8 1, i8 0)
+; CHECK-NEXT: [[TMP4:%.*]] = call ptr @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 10, i64 16, i32 4, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP3:%.*]] = load x86_fp80, ptr [[TMP4]], align 16
; CHECK-NEXT: store x86_fp80 [[TMP3]], ptr [[TMP0]], align 16
-; CHECK-NEXT: call void @__instrumentor_post_load_ind(ptr [[ARRAYIDX]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_load_ind(ptr [[ARRAYIDX]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP2:%.*]] = load x86_fp80, ptr [[TMP0]], align 16
; CHECK-NEXT: ret x86_fp80 [[TMP2]]
;
diff --git a/llvm/test/Instrumentation/Instrumentor/load_store_noreplace.ll b/llvm/test/Instrumentation/Instrumentor/load_store_noreplace.ll
index e00233b0cfcd0..793d4ec8f100b 100644
--- a/llvm/test/Instrumentation/Instrumentor/load_store_noreplace.ll
+++ b/llvm/test/Instrumentation/Instrumentor/load_store_noreplace.ll
@@ -8,14 +8,14 @@ define noundef zeroext i1 @_Z15store_load_boolPb(ptr captures(none) noundef init
; CHECK-LABEL: define noundef zeroext i1 @_Z15store_load_boolPb(
; CHECK-SAME: ptr noundef captures(none) initializes((0, 1)) [[A:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: call void @__instrumentor_pre_store(ptr [[A]], i32 0, i64 1, i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_store(ptr [[A]], i32 0, i64 1, i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0:[0-9]+]]
; CHECK-NEXT: store i8 1, ptr [[A]], align 1
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 1, i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 1, i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 1
-; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP5:%.*]] = load i8, ptr [[ARRAYIDX]], align 1
; CHECK-NEXT: [[TMP1:%.*]] = zext i8 [[TMP5]] to i64
-; CHECK-NEXT: call void @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP1]], i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP1]], i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[LOADEDV:%.*]] = trunc nuw i8 [[TMP5]] to i1
; CHECK-NEXT: ret i1 [[LOADEDV]]
;
@@ -32,14 +32,14 @@ define noundef signext i8 @_Z15store_load_charPc(ptr captures(none) noundef init
; CHECK-LABEL: define noundef signext i8 @_Z15store_load_charPc(
; CHECK-SAME: ptr noundef captures(none) initializes((0, 1)) [[A:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: call void @__instrumentor_pre_store(ptr [[A]], i32 0, i64 1, i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_store(ptr [[A]], i32 0, i64 1, i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store i8 1, ptr [[A]], align 1
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 1, i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 1, i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 1
-; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP5:%.*]] = load i8, ptr [[ARRAYIDX]], align 1
; CHECK-NEXT: [[TMP1:%.*]] = zext i8 [[TMP5]] to i64
-; CHECK-NEXT: call void @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP1]], i64 1, i64 1, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP1]], i64 1, i64 1, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: ret i8 [[TMP5]]
;
entry:
@@ -54,14 +54,14 @@ define noundef signext i16 @_Z16store_load_shortPs(ptr captures(none) noundef in
; CHECK-LABEL: define noundef signext i16 @_Z16store_load_shortPs(
; CHECK-SAME: ptr noundef captures(none) initializes((0, 2)) [[A:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: call void @__instrumentor_pre_store(ptr [[A]], i32 0, i64 2, i64 2, i64 2, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_store(ptr [[A]], i32 0, i64 2, i64 2, i64 2, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store i16 2, ptr [[A]], align 2
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 2, i64 2, i64 2, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 2, i64 2, i64 2, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 2
-; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 2, i64 2, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 2, i64 2, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP5:%.*]] = load i16, ptr [[ARRAYIDX]], align 2
; CHECK-NEXT: [[TMP1:%.*]] = zext i16 [[TMP5]] to i64
-; CHECK-NEXT: call void @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP1]], i64 2, i64 2, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP1]], i64 2, i64 2, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: ret i16 [[TMP5]]
;
entry:
@@ -76,14 +76,14 @@ define noundef i32 @_Z14store_load_intPi(ptr captures(none) noundef initializes(
; CHECK-LABEL: define noundef i32 @_Z14store_load_intPi(
; CHECK-SAME: ptr noundef captures(none) initializes((0, 4)) [[A:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: call void @__instrumentor_pre_store(ptr [[A]], i32 0, i64 3, i64 4, i64 4, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_store(ptr [[A]], i32 0, i64 3, i64 4, i64 4, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store i32 3, ptr [[A]], align 4
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 3, i64 4, i64 4, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 3, i64 4, i64 4, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 4
-; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 4, i64 4, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 4, i64 4, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP5:%.*]] = load i32, ptr [[ARRAYIDX]], align 4
; CHECK-NEXT: [[TMP1:%.*]] = zext i32 [[TMP5]] to i64
-; CHECK-NEXT: call void @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP1]], i64 4, i64 4, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP1]], i64 4, i64 4, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: ret i32 [[TMP5]]
;
entry:
@@ -98,13 +98,13 @@ define noundef i64 @_Z15store_load_longPl(ptr captures(none) noundef initializes
; CHECK-LABEL: define noundef i64 @_Z15store_load_longPl(
; CHECK-SAME: ptr noundef captures(none) initializes((0, 8)) [[A:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: call void @__instrumentor_pre_store(ptr [[A]], i32 0, i64 4, i64 8, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_store(ptr [[A]], i32 0, i64 4, i64 8, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store i64 4, ptr [[A]], align 8
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 4, i64 8, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 4, i64 8, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 8
-; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 8, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 8, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP3:%.*]] = load i64, ptr [[ARRAYIDX]], align 8
-; CHECK-NEXT: call void @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP3]], i64 8, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP3]], i64 8, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: ret i64 [[TMP3]]
;
entry:
@@ -121,14 +121,14 @@ define noundef i128 @_Z20store_load_long_longPx(ptr captures(none) noundef initi
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = alloca i128, align 16
; CHECK-NEXT: store i128 5, ptr [[TMP0]], align 16
-; CHECK-NEXT: call void @__instrumentor_pre_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store i128 5, ptr [[A]], align 8
-; CHECK-NEXT: call void @__instrumentor_post_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 16
-; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 16, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 16, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP4:%.*]] = load i128, ptr [[ARRAYIDX]], align 8
; CHECK-NEXT: store i128 [[TMP4]], ptr [[TMP0]], align 16
-; CHECK-NEXT: call void @__instrumentor_post_load_ind(ptr [[ARRAYIDX]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 12, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_load_ind(ptr [[ARRAYIDX]], i32 0, ptr [[TMP0]], i64 16, i64 8, i32 13, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: ret i128 [[TMP4]]
;
entry:
@@ -143,15 +143,15 @@ define noundef float @_Z16store_load_floatPf(ptr captures(none) noundef initiali
; CHECK-LABEL: define noundef float @_Z16store_load_floatPf(
; CHECK-SAME: ptr noundef captures(none) initializes((0, 4)) [[A:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: call void @__instrumentor_pre_store(ptr [[A]], i32 0, i64 1086324736, i64 4, i64 4, i32 2, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_store(ptr [[A]], i32 0, i64 1086324736, i64 4, i64 4, i32 2, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store float 6.000000e+00, ptr [[A]], align 4
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 1086324736, i64 4, i64 4, i32 2, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 1086324736, i64 4, i64 4, i32 2, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 4
-; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 4, i64 4, i32 2, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 4, i64 4, i32 2, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP7:%.*]] = load float, ptr [[ARRAYIDX]], align 4
; CHECK-NEXT: [[TMP1:%.*]] = bitcast float [[TMP7]] to i32
; CHECK-NEXT: [[TMP2:%.*]] = zext i32 [[TMP1]] to i64
-; CHECK-NEXT: call void @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP2]], i64 4, i64 4, i32 2, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP2]], i64 4, i64 4, i32 2, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: ret float [[TMP7]]
;
entry:
@@ -166,14 +166,14 @@ define noundef double @_Z17store_load_doublePd(ptr captures(none) noundef initia
; CHECK-LABEL: define noundef double @_Z17store_load_doublePd(
; CHECK-SAME: ptr noundef captures(none) initializes((0, 8)) [[A:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: call void @__instrumentor_pre_store(ptr [[A]], i32 0, i64 4619567317775286272, i64 8, i64 8, i32 3, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_store(ptr [[A]], i32 0, i64 4619567317775286272, i64 8, i64 8, i32 3, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store double 7.000000e+00, ptr [[A]], align 8
-; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 4619567317775286272, i64 8, i64 8, i32 3, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store(ptr [[A]], i32 0, i64 4619567317775286272, i64 8, i64 8, i32 3, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 8
-; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 8, i64 8, i32 3, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 8, i64 8, i32 3, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP5:%.*]] = load double, ptr [[ARRAYIDX]], align 8
; CHECK-NEXT: [[TMP1:%.*]] = bitcast double [[TMP5]] to i64
-; CHECK-NEXT: call void @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP1]], i64 8, i64 8, i32 3, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_load(ptr [[ARRAYIDX]], i32 0, i64 [[TMP1]], i64 8, i64 8, i32 3, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: ret double [[TMP5]]
;
entry:
@@ -190,14 +190,14 @@ define noundef x86_fp80 @_Z22store_load_long_doublePe(ptr captures(none) noundef
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = alloca x86_fp80, align 16
; CHECK-NEXT: store x86_fp80 8.000000e+00, ptr [[TMP0]], align 16
-; CHECK-NEXT: call void @__instrumentor_pre_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: store x86_fp80 8.000000e+00, ptr [[A]], align 16
-; CHECK-NEXT: call void @__instrumentor_post_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_store_ind(ptr [[A]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[ARRAYIDX:%.*]] = getelementptr inbounds nuw i8, ptr [[A]], i64 16
-; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 10, i64 16, i32 4, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_pre_load(ptr [[ARRAYIDX]], i32 0, i64 10, i64 16, i32 4, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: [[TMP4:%.*]] = load x86_fp80, ptr [[ARRAYIDX]], align 16
; CHECK-NEXT: store x86_fp80 [[TMP4]], ptr [[TMP0]], align 16
-; CHECK-NEXT: call void @__instrumentor_post_load_ind(ptr [[ARRAYIDX]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0)
+; CHECK-NEXT: call void @__instrumentor_post_load_ind(ptr [[ARRAYIDX]], i32 0, ptr [[TMP0]], i64 10, i64 16, i32 4, i32 0, i8 1, i8 0) #[[ATTR0]]
; CHECK-NEXT: ret x86_fp80 [[TMP4]]
;
entry:
diff --git a/llvm/test/Instrumentation/Instrumentor/module_and_globals.ll b/llvm/test/Instrumentation/Instrumentor/module_and_globals.ll
index 64cb0737d0863..d30f7c4bd803c 100644
--- a/llvm/test/Instrumentation/Instrumentor/module_and_globals.ll
+++ b/llvm/test/Instrumentation/Instrumentor/module_and_globals.ll
@@ -39,34 +39,34 @@ entry:
; CHECK-NEXT: [[TMP1:%.*]] = call ptr @__instrumentor_post_base_pointer_info(ptr [[Y_SHADOW_LOAD]], i32 2, i32 -10) #[[ATTR0]]
; CHECK-NEXT: [[X_SHADOW_LOAD:%.*]] = load ptr, ptr @__instrumentor_shadow.X, align 8
; CHECK-NEXT: [[TMP2:%.*]] = call ptr @__instrumentor_post_base_pointer_info(ptr [[X_SHADOW_LOAD]], i32 2, i32 -9) #[[ATTR0]]
-; CHECK-NEXT: [[TMP10:%.*]] = call ptr @__instrumentor_pre_load(ptr [[X_SHADOW_LOAD]], i32 0, ptr [[TMP2]], i64 4, i64 4, i32 12, i32 -1, i32 0, i8 1, i8 0, i32 9) #[[ATTR0]]
+; CHECK-NEXT: [[TMP10:%.*]] = call ptr @__instrumentor_pre_load(ptr [[X_SHADOW_LOAD]], i32 0, ptr [[TMP2]], i64 4, i64 4, i32 13, i32 -1, i32 0, i8 1, i8 0, i32 9) #[[ATTR0]]
; CHECK-NEXT: [[TMP11:%.*]] = load i32, ptr [[TMP10]], align 4
; CHECK-NEXT: [[TMP12:%.*]] = zext i32 [[TMP11]] to i64
-; CHECK-NEXT: [[TMP13:%.*]] = call i64 @__instrumentor_post_load(ptr [[X_SHADOW_LOAD]], i32 0, ptr [[TMP2]], i64 [[TMP12]], i64 4, i64 4, i32 12, i32 -1, i32 0, i8 1, i8 0, i32 -9) #[[ATTR0]]
+; CHECK-NEXT: [[TMP13:%.*]] = call i64 @__instrumentor_post_load(ptr [[X_SHADOW_LOAD]], i32 0, ptr [[TMP2]], i64 [[TMP12]], i64 4, i64 4, i32 13, i32 -1, i32 0, i8 1, i8 0, i32 -9) #[[ATTR0]]
; CHECK-NEXT: [[TMP14:%.*]] = trunc i64 [[TMP13]] to i32
-; CHECK-NEXT: [[TMP8:%.*]] = call ptr @__instrumentor_pre_load(ptr [[Y_SHADOW_LOAD]], i32 0, ptr [[TMP1]], i64 4, i64 4, i32 12, i32 -1, i32 0, i8 1, i8 0, i32 10) #[[ATTR0]]
+; CHECK-NEXT: [[TMP8:%.*]] = call ptr @__instrumentor_pre_load(ptr [[Y_SHADOW_LOAD]], i32 0, ptr [[TMP1]], i64 4, i64 4, i32 13, i32 -1, i32 0, i8 1, i8 0, i32 10) #[[ATTR0]]
; CHECK-NEXT: [[TMP4:%.*]] = load i32, ptr [[TMP8]], align 4
; CHECK-NEXT: [[TMP15:%.*]] = zext i32 [[TMP4]] to i64
-; CHECK-NEXT: [[TMP21:%.*]] = call i64 @__instrumentor_post_load(ptr [[Y_SHADOW_LOAD]], i32 0, ptr [[TMP1]], i64 [[TMP15]], i64 4, i64 4, i32 12, i32 -1, i32 0, i8 1, i8 0, i32 -10) #[[ATTR0]]
+; CHECK-NEXT: [[TMP21:%.*]] = call i64 @__instrumentor_post_load(ptr [[Y_SHADOW_LOAD]], i32 0, ptr [[TMP1]], i64 [[TMP15]], i64 4, i64 4, i32 13, i32 -1, i32 0, i8 1, i8 0, i32 -10) #[[ATTR0]]
; CHECK-NEXT: [[TMP25:%.*]] = trunc i64 [[TMP21]] to i32
-; CHECK-NEXT: [[TMP26:%.*]] = call ptr @__instrumentor_pre_load(ptr [[Z_SHADOW_LOAD]], i32 0, ptr [[TMP0]], i64 4, i64 4, i32 12, i32 -1, i32 0, i8 1, i8 0, i32 11) #[[ATTR0]]
+; CHECK-NEXT: [[TMP26:%.*]] = call ptr @__instrumentor_pre_load(ptr [[Z_SHADOW_LOAD]], i32 0, ptr [[TMP0]], i64 4, i64 4, i32 13, i32 -1, i32 0, i8 1, i8 0, i32 11) #[[ATTR0]]
; CHECK-NEXT: [[TMP9:%.*]] = load i32, ptr [[TMP26]], align 4
; CHECK-NEXT: [[TMP16:%.*]] = zext i32 [[TMP9]] to i64
-; CHECK-NEXT: [[TMP27:%.*]] = call i64 @__instrumentor_post_load(ptr [[Z_SHADOW_LOAD]], i32 0, ptr [[TMP0]], i64 [[TMP16]], i64 4, i64 4, i32 12, i32 -1, i32 0, i8 1, i8 0, i32 -11) #[[ATTR0]]
+; CHECK-NEXT: [[TMP27:%.*]] = call i64 @__instrumentor_post_load(ptr [[Z_SHADOW_LOAD]], i32 0, ptr [[TMP0]], i64 [[TMP16]], i64 4, i64 4, i32 13, i32 -1, i32 0, i8 1, i8 0, i32 -11) #[[ATTR0]]
; CHECK-NEXT: [[TMP28:%.*]] = trunc i64 [[TMP27]] to i32
; CHECK-NEXT: [[TMP29:%.*]] = zext i32 [[TMP14]] to i64
; CHECK-NEXT: [[TMP30:%.*]] = zext i32 [[TMP25]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 12, i32 -1, i32 4, i32 14, i64 [[TMP29]], i64 [[TMP30]], i64 1, i32 12) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 13, i32 -1, i32 4, i32 14, i64 [[TMP29]], i64 [[TMP30]], i64 1, i32 12) #[[ATTR0]]
; CHECK-NEXT: [[ADD:%.*]] = add nsw i32 [[TMP14]], [[TMP25]]
; CHECK-NEXT: [[TMP17:%.*]] = zext i32 [[ADD]] to i64
-; CHECK-NEXT: [[TMP18:%.*]] = call i64 @__instrumentor_post_numeric(i32 12, i32 -1, i32 4, i32 14, i64 [[TMP29]], i64 [[TMP30]], i64 [[TMP17]], i64 1, i32 -12) #[[ATTR0]]
+; CHECK-NEXT: [[TMP18:%.*]] = call i64 @__instrumentor_post_numeric(i32 13, i32 -1, i32 4, i32 14, i64 [[TMP29]], i64 [[TMP30]], i64 [[TMP17]], i64 1, i32 -12) #[[ATTR0]]
; CHECK-NEXT: [[TMP19:%.*]] = trunc i64 [[TMP18]] to i32
; CHECK-NEXT: [[TMP20:%.*]] = zext i32 [[TMP19]] to i64
; CHECK-NEXT: [[TMP24:%.*]] = zext i32 [[TMP28]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 12, i32 -1, i32 4, i32 14, i64 [[TMP20]], i64 [[TMP24]], i64 1, i32 13) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 13, i32 -1, i32 4, i32 14, i64 [[TMP20]], i64 [[TMP24]], i64 1, i32 13) #[[ATTR0]]
; CHECK-NEXT: [[ADD3:%.*]] = add nsw i32 [[TMP19]], [[TMP28]]
; CHECK-NEXT: [[TMP22:%.*]] = zext i32 [[ADD3]] to i64
-; CHECK-NEXT: [[TMP23:%.*]] = call i64 @__instrumentor_post_numeric(i32 12, i32 -1, i32 4, i32 14, i64 [[TMP20]], i64 [[TMP24]], i64 [[TMP22]], i64 1, i32 -13) #[[ATTR0]]
+; CHECK-NEXT: [[TMP23:%.*]] = call i64 @__instrumentor_post_numeric(i32 13, i32 -1, i32 4, i32 14, i64 [[TMP20]], i64 [[TMP24]], i64 [[TMP22]], i64 1, i32 -13) #[[ATTR0]]
; CHECK-NEXT: [[ADD2:%.*]] = trunc i64 [[TMP23]] to i32
; CHECK-NEXT: call void @__instrumentor_post_function(ptr @foo, ptr @__instrumentor_.str.5, i32 0, ptr null, i8 0, i32 -15) #[[ATTR0]]
; CHECK-NEXT: ret i32 [[ADD2]]
diff --git a/llvm/test/Instrumentation/Instrumentor/numeric.ll b/llvm/test/Instrumentation/Instrumentor/numeric.ll
index b101b0c860bb0..473df0c7b5d2d 100644
--- a/llvm/test/Instrumentation/Instrumentor/numeric.ll
+++ b/llvm/test/Instrumentation/Instrumentor/numeric.ll
@@ -158,28 +158,28 @@ define i32 @test_i32(i32 %p1, i32 %p2) {
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[TMP0:%.*]] = zext i32 [[P1]] to i64
; CHECK-NEXT: [[TMP1:%.*]] = zext i32 [[P2]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 12, i32 4, i32 14, i64 [[TMP0]], i64 [[TMP1]], i64 0, i32 16) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 13, i32 4, i32 14, i64 [[TMP0]], i64 [[TMP1]], i64 0, i32 16) #[[ATTR0]]
; CHECK-NEXT: [[A1:%.*]] = add i32 [[P1]], [[P2]]
; CHECK-NEXT: [[TMP2:%.*]] = zext i32 [[A1]] to i64
-; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 12, i32 4, i32 14, i64 [[TMP0]], i64 [[TMP1]], i64 [[TMP2]], i64 0, i32 -16) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 13, i32 4, i32 14, i64 [[TMP0]], i64 [[TMP1]], i64 [[TMP2]], i64 0, i32 -16) #[[ATTR0]]
; CHECK-NEXT: [[TMP3:%.*]] = zext i32 [[P1]] to i64
; CHECK-NEXT: [[TMP4:%.*]] = zext i32 [[A1]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 12, i32 4, i32 18, i64 [[TMP3]], i64 [[TMP4]], i64 2, i32 17) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 13, i32 4, i32 18, i64 [[TMP3]], i64 [[TMP4]], i64 2, i32 17) #[[ATTR0]]
; CHECK-NEXT: [[A2:%.*]] = mul nuw i32 [[P1]], [[A1]]
; CHECK-NEXT: [[TMP5:%.*]] = zext i32 [[A2]] to i64
-; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 12, i32 4, i32 18, i64 [[TMP3]], i64 [[TMP4]], i64 [[TMP5]], i64 2, i32 -17) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 13, i32 4, i32 18, i64 [[TMP3]], i64 [[TMP4]], i64 [[TMP5]], i64 2, i32 -17) #[[ATTR0]]
; CHECK-NEXT: [[TMP6:%.*]] = zext i32 [[A2]] to i64
; CHECK-NEXT: [[TMP7:%.*]] = zext i32 [[P2]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 12, i32 4, i32 21, i64 [[TMP6]], i64 [[TMP7]], i64 64, i32 18) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 13, i32 4, i32 21, i64 [[TMP6]], i64 [[TMP7]], i64 64, i32 18) #[[ATTR0]]
; CHECK-NEXT: [[A3:%.*]] = sdiv exact i32 [[A2]], [[P2]]
; CHECK-NEXT: [[TMP8:%.*]] = zext i32 [[A3]] to i64
-; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 12, i32 4, i32 21, i64 [[TMP6]], i64 [[TMP7]], i64 [[TMP8]], i64 64, i32 -18) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 13, i32 4, i32 21, i64 [[TMP6]], i64 [[TMP7]], i64 [[TMP8]], i64 64, i32 -18) #[[ATTR0]]
; CHECK-NEXT: [[TMP9:%.*]] = zext i32 [[A3]] to i64
; CHECK-NEXT: [[TMP10:%.*]] = zext i32 [[A1]] to i64
-; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 12, i32 4, i32 16, i64 [[TMP9]], i64 [[TMP10]], i64 0, i32 19) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 13, i32 4, i32 16, i64 [[TMP9]], i64 [[TMP10]], i64 0, i32 19) #[[ATTR0]]
; CHECK-NEXT: [[A4:%.*]] = sub i32 [[A3]], [[A1]]
; CHECK-NEXT: [[TMP11:%.*]] = zext i32 [[A4]] to i64
-; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 12, i32 4, i32 16, i64 [[TMP9]], i64 [[TMP10]], i64 [[TMP11]], i64 0, i32 -19) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 13, i32 4, i32 16, i64 [[TMP9]], i64 [[TMP10]], i64 [[TMP11]], i64 0, i32 -19) #[[ATTR0]]
; CHECK-NEXT: ret i32 [[A4]]
;
entry:
@@ -194,24 +194,24 @@ define i64 @test_i64(i64 %p1, i64 %p2) {
; CHECK-LABEL: define i64 @test_i64(
; CHECK-SAME: i64 [[P1:%.*]], i64 [[P2:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*:]]
-; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 12, i32 8, i32 14, i64 [[P1]], i64 [[P2]], i64 0, i32 20) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 13, i32 8, i32 14, i64 [[P1]], i64 [[P2]], i64 0, i32 20) #[[ATTR0]]
; CHECK-NEXT: [[A1:%.*]] = add i64 [[P1]], [[P2]]
-; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 12, i32 8, i32 14, i64 [[P1]], i64 [[P2]], i64 [[A1]], i64 0, i32 -20) #[[ATTR0]]
-; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 12, i32 8, i32 18, i64 [[P1]], i64 [[A1]], i64 0, i32 21) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 13, i32 8, i32 14, i64 [[P1]], i64 [[P2]], i64 [[A1]], i64 0, i32 -20) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 13, i32 8, i32 18, i64 [[P1]], i64 [[A1]], i64 0, i32 21) #[[ATTR0]]
; CHECK-NEXT: [[A2:%.*]] = mul i64 [[P1]], [[A1]]
-; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 12, i32 8, i32 18, i64 [[P1]], i64 [[A1]], i64 [[A2]], i64 0, i32 -21) #[[ATTR0]]
-; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 12, i32 8, i32 20, i64 [[A2]], i64 [[P2]], i64 0, i32 22) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 13, i32 8, i32 18, i64 [[P1]], i64 [[A1]], i64 [[A2]], i64 0, i32 -21) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 13, i32 8, i32 20, i64 [[A2]], i64 [[P2]], i64 0, i32 22) #[[ATTR0]]
; CHECK-NEXT: [[A3:%.*]] = udiv i64 [[A2]], [[P2]]
-; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 12, i32 8, i32 20, i64 [[A2]], i64 [[P2]], i64 [[A3]], i64 0, i32 -22) #[[ATTR0]]
-; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 12, i32 8, i32 16, i64 [[A3]], i64 [[A1]], i64 0, i32 23) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 13, i32 8, i32 20, i64 [[A2]], i64 [[P2]], i64 [[A3]], i64 0, i32 -22) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 13, i32 8, i32 16, i64 [[A3]], i64 [[A1]], i64 0, i32 23) #[[ATTR0]]
; CHECK-NEXT: [[A4:%.*]] = sub i64 [[A3]], [[A1]]
-; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 12, i32 8, i32 16, i64 [[A3]], i64 [[A1]], i64 [[A4]], i64 0, i32 -23) #[[ATTR0]]
-; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 12, i32 8, i32 29, i64 [[A4]], i64 [[A2]], i64 0, i32 24) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 13, i32 8, i32 16, i64 [[A3]], i64 [[A1]], i64 [[A4]], i64 0, i32 -23) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 13, i32 8, i32 29, i64 [[A4]], i64 [[A2]], i64 0, i32 24) #[[ATTR0]]
; CHECK-NEXT: [[A5:%.*]] = and i64 [[A4]], [[A2]]
-; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 12, i32 8, i32 29, i64 [[A4]], i64 [[A2]], i64 [[A5]], i64 0, i32 -24) #[[ATTR0]]
-; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 12, i32 8, i32 31, i64 [[A5]], i64 [[A3]], i64 0, i32 25) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 13, i32 8, i32 29, i64 [[A4]], i64 [[A2]], i64 [[A5]], i64 0, i32 -24) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric(i32 13, i32 8, i32 31, i64 [[A5]], i64 [[A3]], i64 0, i32 25) #[[ATTR0]]
; CHECK-NEXT: [[A6:%.*]] = xor i64 [[A5]], [[A3]]
-; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 12, i32 8, i32 31, i64 [[A5]], i64 [[A3]], i64 [[A6]], i64 0, i32 -25) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_numeric(i32 13, i32 8, i32 31, i64 [[A5]], i64 [[A3]], i64 [[A6]], i64 0, i32 -25) #[[ATTR0]]
; CHECK-NEXT: ret i64 [[A6]]
;
entry:
@@ -233,40 +233,40 @@ define i128 @test_i128(i128 %p1, i128 %p2) {
; CHECK-NEXT: [[TMP2:%.*]] = alloca i128, align 8
; CHECK-NEXT: store i128 [[P1]], ptr [[TMP2]], align 4
; CHECK-NEXT: store i128 [[P2]], ptr [[TMP1]], align 4
-; CHECK-NEXT: call void @__instrumentor_pre_numeric_ind(i32 12, i32 16, i32 14, ptr [[TMP2]], ptr [[TMP1]], i64 0, i32 26) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric_ind(i32 13, i32 16, i32 14, ptr [[TMP2]], ptr [[TMP1]], i64 0, i32 26) #[[ATTR0]]
; CHECK-NEXT: [[A1:%.*]] = add i128 [[P1]], [[P2]]
; CHECK-NEXT: store i128 [[A1]], ptr [[TMP0]], align 4
-; CHECK-NEXT: call void @__instrumentor_post_numeric_ind(i32 12, i32 16, i32 14, ptr [[TMP2]], ptr [[TMP1]], ptr [[TMP0]], i64 0, i32 -26) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_numeric_ind(i32 13, i32 16, i32 14, ptr [[TMP2]], ptr [[TMP1]], ptr [[TMP0]], i64 0, i32 -26) #[[ATTR0]]
; CHECK-NEXT: store i128 [[P1]], ptr [[TMP2]], align 4
; CHECK-NEXT: store i128 [[A1]], ptr [[TMP0]], align 4
-; CHECK-NEXT: call void @__instrumentor_pre_numeric_ind(i32 12, i32 16, i32 18, ptr [[TMP2]], ptr [[TMP0]], i64 0, i32 27) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric_ind(i32 13, i32 16, i32 18, ptr [[TMP2]], ptr [[TMP0]], i64 0, i32 27) #[[ATTR0]]
; CHECK-NEXT: [[A2:%.*]] = mul i128 [[P1]], [[A1]]
; CHECK-NEXT: store i128 [[A2]], ptr [[TMP1]], align 4
-; CHECK-NEXT: call void @__instrumentor_post_numeric_ind(i32 12, i32 16, i32 18, ptr [[TMP2]], ptr [[TMP0]], ptr [[TMP1]], i64 0, i32 -27) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_numeric_ind(i32 13, i32 16, i32 18, ptr [[TMP2]], ptr [[TMP0]], ptr [[TMP1]], i64 0, i32 -27) #[[ATTR0]]
; CHECK-NEXT: store i128 [[A2]], ptr [[TMP2]], align 4
; CHECK-NEXT: store i128 [[P2]], ptr [[TMP1]], align 4
-; CHECK-NEXT: call void @__instrumentor_pre_numeric_ind(i32 12, i32 16, i32 24, ptr [[TMP2]], ptr [[TMP1]], i64 0, i32 28) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric_ind(i32 13, i32 16, i32 24, ptr [[TMP2]], ptr [[TMP1]], i64 0, i32 28) #[[ATTR0]]
; CHECK-NEXT: [[A3:%.*]] = srem i128 [[A2]], [[P2]]
; CHECK-NEXT: store i128 [[A3]], ptr [[TMP0]], align 4
-; CHECK-NEXT: call void @__instrumentor_post_numeric_ind(i32 12, i32 16, i32 24, ptr [[TMP2]], ptr [[TMP1]], ptr [[TMP0]], i64 0, i32 -28) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_numeric_ind(i32 13, i32 16, i32 24, ptr [[TMP2]], ptr [[TMP1]], ptr [[TMP0]], i64 0, i32 -28) #[[ATTR0]]
; CHECK-NEXT: store i128 [[A3]], ptr [[TMP2]], align 4
; CHECK-NEXT: store i128 [[A1]], ptr [[TMP0]], align 4
-; CHECK-NEXT: call void @__instrumentor_pre_numeric_ind(i32 12, i32 16, i32 16, ptr [[TMP2]], ptr [[TMP0]], i64 1, i32 29) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric_ind(i32 13, i32 16, i32 16, ptr [[TMP2]], ptr [[TMP0]], i64 1, i32 29) #[[ATTR0]]
; CHECK-NEXT: [[A4:%.*]] = sub nsw i128 [[A3]], [[A1]]
; CHECK-NEXT: store i128 [[A4]], ptr [[TMP1]], align 4
-; CHECK-NEXT: call void @__instrumentor_post_numeric_ind(i32 12, i32 16, i32 16, ptr [[TMP2]], ptr [[TMP0]], ptr [[TMP1]], i64 1, i32 -29) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_numeric_ind(i32 13, i32 16, i32 16, ptr [[TMP2]], ptr [[TMP0]], ptr [[TMP1]], i64 1, i32 -29) #[[ATTR0]]
; CHECK-NEXT: store i128 [[A4]], ptr [[TMP2]], align 4
; CHECK-NEXT: store i128 [[A2]], ptr [[TMP1]], align 4
-; CHECK-NEXT: call void @__instrumentor_pre_numeric_ind(i32 12, i32 16, i32 30, ptr [[TMP2]], ptr [[TMP1]], i64 32, i32 30) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric_ind(i32 13, i32 16, i32 30, ptr [[TMP2]], ptr [[TMP1]], i64 32, i32 30) #[[ATTR0]]
; CHECK-NEXT: [[A5:%.*]] = or disjoint i128 [[A4]], [[A2]]
; CHECK-NEXT: store i128 [[A5]], ptr [[TMP0]], align 4
-; CHECK-NEXT: call void @__instrumentor_post_numeric_ind(i32 12, i32 16, i32 30, ptr [[TMP2]], ptr [[TMP1]], ptr [[TMP0]], i64 32, i32 -30) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_numeric_ind(i32 13, i32 16, i32 30, ptr [[TMP2]], ptr [[TMP1]], ptr [[TMP0]], i64 32, i32 -30) #[[ATTR0]]
; CHECK-NEXT: store i128 [[A5]], ptr [[TMP2]], align 4
; CHECK-NEXT: store i128 [[A3]], ptr [[TMP0]], align 4
-; CHECK-NEXT: call void @__instrumentor_pre_numeric_ind(i32 12, i32 16, i32 26, ptr [[TMP2]], ptr [[TMP0]], i64 0, i32 31) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_pre_numeric_ind(i32 13, i32 16, i32 26, ptr [[TMP2]], ptr [[TMP0]], i64 0, i32 31) #[[ATTR0]]
; CHECK-NEXT: [[A6:%.*]] = shl i128 [[A5]], [[A3]]
; CHECK-NEXT: store i128 [[A6]], ptr [[TMP1]], align 4
-; CHECK-NEXT: call void @__instrumentor_post_numeric_ind(i32 12, i32 16, i32 26, ptr [[TMP2]], ptr [[TMP0]], ptr [[TMP1]], i64 0, i32 -31) #[[ATTR0]]
+; CHECK-NEXT: call void @__instrumentor_post_numeric_ind(i32 13, i32 16, i32 26, ptr [[TMP2]], ptr [[TMP0]], ptr [[TMP1]], i64 0, i32 -31) #[[ATTR0]]
; CHECK-NEXT: ret i128 [[A6]]
;
entry:
diff --git a/llvm/test/Instrumentation/Instrumentor/numeric_subtypeid.ll b/llvm/test/Instrumentation/Instrumentor/numeric_subtypeid.ll
index 4f6b40f19c995..71bc2a4db5d55 100644
--- a/llvm/test/Instrumentation/Instrumentor/numeric_subtypeid.ll
+++ b/llvm/test/Instrumentation/Instrumentor/numeric_subtypeid.ll
@@ -13,7 +13,7 @@ define <4 x float> @test_vector_fadd(<4 x float> %a, <4 x float> %b) {
; CHECK-NEXT: store <4 x float> [[A]], ptr [[TMP2]], align 16
; CHECK-NEXT: store <4 x float> [[B]], ptr [[TMP1]], align 16
; CHECK-NEXT: store <4 x float> [[RES]], ptr [[TMP0]], align 16
-; CHECK-NEXT: call void @__instrumentor_post_numeric_ind(i32 18, i32 2, i32 16, i32 15, ptr [[TMP2]], ptr [[TMP1]], ptr [[TMP0]], i64 0, i32 -1) #[[ATTR0:[0-9]+]]
+; CHECK-NEXT: call void @__instrumentor_post_numeric_ind(i32 19, i32 2, i32 16, i32 15, ptr [[TMP2]], ptr [[TMP1]], ptr [[TMP0]], i64 0, i32 -1) #[[ATTR0:[0-9]+]]
; CHECK-NEXT: ret <4 x float> [[RES]]
;
entry:
diff --git a/llvm/test/TableGen/CPtrWildcard.td b/llvm/test/TableGen/CPtrWildcard.td
index 082e510f5a595..bf71ff7f9453a 100644
--- a/llvm/test/TableGen/CPtrWildcard.td
+++ b/llvm/test/TableGen/CPtrWildcard.td
@@ -8,13 +8,13 @@
// CHECK-NEXT:/* 3*/ OPC_CheckChild0Integer, [[#]],
// CHECK-NEXT:/* 5*/ OPC_RecordChild1, // #0 = $src
// CHECK-NEXT:/* 6*/ OPC_Scope /*2 children */, 9, // ->17
-// CHECK-NEXT:/* 8*/ OPC_CheckChild1Type, /*MVT::c64*/6|128,2/*262*/,
+// CHECK-NEXT:/* 8*/ OPC_CheckChild1Type, /*MVT::c64*/7|128,2/*263*/,
// CHECK-NEXT:/* 11*/ OPC_MorphNodeTo1None, TARGET_VAL(MyTarget::C64_TO_I64),
// CHECK-NEXT: MVT::i64, 1/*#Ops*/, /*OperandList*/0, // Ops = #0
// CHECK-NEXT: // Src: (intrinsic_wo_chain:{ *:[i64] } [[#]]:{ *:[iPTR] }, c64:{ *:[c64] }:$src) - Complexity = 8
// CHECK-NEXT: // Dst: (C64_TO_I64:{ *:[i64] } ?:{ *:[c64] }:$src)
// CHECK-NEXT:/* 17*/ /*Scope*/ 9, // ->27
-// CHECK-NEXT:/* 18*/ OPC_CheckChild1Type, /*MVT::c128*/7|128,2/*263*/,
+// CHECK-NEXT:/* 18*/ OPC_CheckChild1Type, /*MVT::c128*/8|128,2/*264*/,
// CHECK-NEXT:/* 21*/ OPC_MorphNodeTo1None, TARGET_VAL(MyTarget::C128_TO_I64),
// CHECK-NEXT: MVT::i64, 1/*#Ops*/, /*OperandList*/0, // Ops = #0
// CHECK-NEXT: // Src: (intrinsic_wo_chain:{ *:[i64] } [[#]]:{ *:[iPTR] }, c128:{ *:[c128] }:$src) - Complexity = 8
diff --git a/llvm/test/TableGen/x86-fold-tables.inc b/llvm/test/TableGen/x86-fold-tables.inc
index 2d2e02f1b9968..07466b31e7bb7 100644
--- a/llvm/test/TableGen/x86-fold-tables.inc
+++ b/llvm/test/TableGen/x86-fold-tables.inc
@@ -354,10 +354,6 @@ static const X86FoldTableEntry Table2Addr[] = {
};
static const X86FoldTableEntry Table0[] = {
- {X86::BSRMOVHrr_get, X86::BSRMOVHmr_get, TB_FOLDED_STORE},
- {X86::BSRMOVHrr_set, X86::BSRMOVHrm_set, TB_FOLDED_LOAD},
- {X86::BSRMOVLrr_get, X86::BSRMOVLmr_get, TB_FOLDED_STORE},
- {X86::BSRMOVLrr_set, X86::BSRMOVLrm_set, TB_FOLDED_LOAD},
{X86::BT16ri8, X86::BT16mi8, TB_FOLDED_LOAD},
{X86::BT32ri8, X86::BT32mi8, TB_FOLDED_LOAD},
{X86::BT64ri8, X86::BT64mi8, TB_FOLDED_LOAD},
@@ -688,7 +684,6 @@ static const X86FoldTableEntry Table1[] = {
{X86::BLSR64rr, X86::BLSR64rm, 0},
{X86::BLSR64rr_EVEX, X86::BLSR64rm_EVEX, 0},
{X86::BLSR64rr_NF, X86::BLSR64rm_NF, 0},
- {X86::BSRMOVFrr, X86::BSRMOVFrm, 0},
{X86::BZHI32rr, X86::BZHI32rm, 0},
{X86::BZHI32rr_EVEX, X86::BZHI32rm_EVEX, 0},
{X86::BZHI32rr_NF, X86::BZHI32rm_NF, 0},
diff --git a/llvm/test/Verifier/x86_bsr.ll b/llvm/test/Verifier/x86_bsr.ll
new file mode 100644
index 0000000000000..2912aee119bfa
--- /dev/null
+++ b/llvm/test/Verifier/x86_bsr.ll
@@ -0,0 +1,6 @@
+; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
+
+; CHECK: Function takes x86_bsr but isn't an intrinsic
+define void @takes_bsr(x86_bsr %b) {
+ ret void
+}
diff --git a/llvm/utils/TableGen/Basic/CodeGenIntrinsics.cpp b/llvm/utils/TableGen/Basic/CodeGenIntrinsics.cpp
index 76fc2a4f38811..1da76b2230ed3 100644
--- a/llvm/utils/TableGen/Basic/CodeGenIntrinsics.cpp
+++ b/llvm/utils/TableGen/Basic/CodeGenIntrinsics.cpp
@@ -164,8 +164,8 @@ static bool doesSuffixLookLikeMangledType(StringRef Suffix) {
// Match one of the named types.
static constexpr StringLiteral NamedTypes[] = {
- "isVoid", "Metadata", "f16", "f32", "f64",
- "f80", "f128", "bf16", "ppcf128", "x86amx"};
+ "isVoid", "Metadata", "f16", "f32", "f64", "f80",
+ "f128", "bf16", "ppcf128", "x86amx", "x86bsr"};
return is_contained(NamedTypes, Suffix);
}
diff --git a/llvm/utils/TableGen/X86ManualFoldTables.def b/llvm/utils/TableGen/X86ManualFoldTables.def
index 693b3de69ae46..66ec513c1bf45 100644
--- a/llvm/utils/TableGen/X86ManualFoldTables.def
+++ b/llvm/utils/TableGen/X86ManualFoldTables.def
@@ -13,6 +13,15 @@
#ifndef NOFOLD
#define NOFOLD(INSN)
#endif
+// BSR instructions: SDE does not correctly handle the memory-operand forms
+// of bsrmovl (F3 prefix). Memory-destination bsrmovl_get writes zeros instead
+// of BSR.lo contents, and memory-source bsrmovl_set reads the wrong BSR half.
+// Prevent all BSR memory folding to avoid silent miscompilation under SDE.
+NOFOLD(BSRMOVFrr)
+NOFOLD(BSRMOVHrr_get)
+NOFOLD(BSRMOVHrr_set)
+NOFOLD(BSRMOVLrr_get)
+NOFOLD(BSRMOVLrr_set)
NOFOLD(BTC16rr)
NOFOLD(BTC32rr)
NOFOLD(BTC64rr)
More information about the llvm-branch-commits
mailing list