[llvm] 0fc5d0a - [IR] Introduce `mem.cache_hint` metadata for composable memory cache control hints (#181612)
via llvm-commits
llvm-commits at lists.llvm.org
Fri May 29 11:58:03 PDT 2026
Author: Fei Peng
Date: 2026-05-29T11:57:58-07:00
New Revision: 0fc5d0ab0cea3afe32592f24af17b5d7e02c9dfe
URL: https://github.com/llvm/llvm-project/commit/0fc5d0ab0cea3afe32592f24af17b5d7e02c9dfe
DIFF: https://github.com/llvm/llvm-project/commit/0fc5d0ab0cea3afe32592f24af17b5d7e02c9dfe.diff
LOG: [IR] Introduce `mem.cache_hint` metadata for composable memory cache control hints (#181612)
Add target-agnostic infrastructure for the !mem.cache_hint metadata
kind,
https://discourse.llvm.org/t/rfc-composable-and-extensible-memory-cache-control-hints-in-llvm-ir/89443
This patch includes:
- Registration of mem.cache_hint in FixedMetadataKinds
- IR Verifier validation of structural constraints
- Metadata helper support in combineMetadata(), copyMetadataForLoad(),
and dropUBImplyingAttrsAndMetadata()
- LangRef documentation for the metadata format and semantics
- Verifier and transform pass test coverage (GVN, InstCombine,
SimplifyCFG)
Co-authored-by: Yonah Goldberg <ygoldberg at nvidia.com>
Assisted-by: Claude Code
---------
Co-authored-by: Yonah Goldberg <ygoldberg at nvidia.com>
Added:
llvm/test/Verifier/mem-cache-hint.ll
Modified:
llvm/docs/LangRef.rst
llvm/include/llvm/IR/FixedMetadataKinds.def
llvm/lib/IR/Instruction.cpp
llvm/lib/IR/Verifier.cpp
llvm/lib/Transforms/Utils/Local.cpp
llvm/test/Bitcode/compatibility.ll
llvm/test/Transforms/GVN/metadata.ll
llvm/test/Transforms/InstCombine/loadstore-metadata.ll
llvm/test/Transforms/SimplifyCFG/hoist-with-metadata.ll
Removed:
################################################################################
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 7ce73cd64c462..00cfcb20da74c 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -7869,6 +7869,81 @@ The ``!captures`` attribute makes no statement about other uses of ``%x``, or
uses of the stored-to memory location after it has been overwritten with a
diff erent value.
+'``mem.cache_hint``' Metadata
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``!mem.cache_hint`` metadata may be attached to any instruction that reads
+or writes memory, except non-intrinsic calls. Cache hints are not permitted on
+function calls because their memory behavior depends on attributes, which may
+change independently.
+
+The ``!mem.cache_hint`` metadata provides target-specific cache control hints
+for the memory operation. This metadata is purely a performance hint: dropping
+or ignoring it must not change the observable behavior of the program
+(particularly, cache hints do not affect memory model semantics).
+
+The ``!mem.cache_hint`` node must contain an even number of entries,
+alternating between ``i32`` operand numbers and metadata hint nodes.
+
+Each operand number identifies the pointer operand to which the following hint
+node applies. For non-call instructions, the operand number is the IR operand
+index. For intrinsic calls, the operand number is the call argument index.
+Operand numbers must be unique and appear in increasing order within a
+``!mem.cache_hint`` node and must refer to a pointer-typed operand.
+
+For example, for a ``load``, the pointer is operand 0. For a ``store``, the
+value is operand 0 and the pointer is operand 1. For intrinsic calls such as
+``llvm.memcpy``, operand numbers correspond to argument positions: e.g.,
+destination is argument 0 and source is argument 1.
+
+Each hint node is a metadata node containing target-prefixed key/value pairs.
+Keys must be strings, and values must be either strings or integer constants.
+Keys within a single hint node must be unique.
+
+The hint node keys are prefixed with a target identifier (e.g., ``nvvm.``) and
+their interpretation is target-dependent. Each hint must describe a property of
+the individual memory access through the corresponding pointer operand. The IR
+verifier enforces only the structural rules above; validation of target-specific
+keys and values is performed by the corresponding backend. Unsupported
+properties may be silently ignored during code generation.
+
+The following examples use ``nvvm.`` prefixed keys for NVIDIA GPU targets.
+Other targets may define their own prefixed keys.
+
+Example: load with cache hints:
+
+.. code-block:: llvm
+
+ %v = load i32, ptr addrspace(1) %p, align 4, !mem.cache_hint !0
+
+ !0 = !{ i32 0, !1 }
+ !1 = !{ !"nvvm.l1_eviction", !"first",
+ !"nvvm.l2_eviction", !"first",
+ !"nvvm.l2_prefetch_size", !"128B" }
+
+Example: store with cache hints:
+
+.. code-block:: llvm
+
+ store i32 %v, ptr addrspace(1) %p, align 4, !mem.cache_hint !0
+
+ !0 = !{ i32 1, !1 }
+ !1 = !{ !"nvvm.l1_eviction", !"last",
+ !"nvvm.l2_eviction", !"last" }
+
+Example: memcpy with per-operand hints:
+
+.. code-block:: llvm
+
+ call void @llvm.memcpy.p1.p1.i64(ptr addrspace(1) %d,
+ ptr addrspace(1) %s, i64 16, i1 false),
+ !mem.cache_hint !0
+
+ !0 = !{ i32 0, !1, i32 1, !2 }
+ !1 = !{ !"nvvm.l1_eviction", !"last" }
+ !2 = !{ !"nvvm.l1_eviction", !"first",
+ !"nvvm.l2_prefetch_size", !"128B" }
+
.. _llvm.loop:
'``llvm.loop``'
diff --git a/llvm/include/llvm/IR/FixedMetadataKinds.def b/llvm/include/llvm/IR/FixedMetadataKinds.def
index 5c8eb2079f857..552c1f60f2d38 100644
--- a/llvm/include/llvm/IR/FixedMetadataKinds.def
+++ b/llvm/include/llvm/IR/FixedMetadataKinds.def
@@ -63,3 +63,4 @@ LLVM_FIXED_MD_KIND(MD_call_target, "call_target", 48)
LLVM_FIXED_MD_KIND(MD_inline_history, "inline_history", 49)
LLVM_FIXED_MD_KIND(MD_elf_section_properties, "elf_section_properties", 50)
LLVM_FIXED_MD_KIND(MD_unique_id, "guid", 51)
+LLVM_FIXED_MD_KIND(MD_mem_cache_hint, "mem.cache_hint", 52)
diff --git a/llvm/lib/IR/Instruction.cpp b/llvm/lib/IR/Instruction.cpp
index 71229c4c44a4a..3b6f33870f622 100644
--- a/llvm/lib/IR/Instruction.cpp
+++ b/llvm/lib/IR/Instruction.cpp
@@ -587,12 +587,14 @@ void Instruction::dropUBImplyingAttrsAndMetadata(ArrayRef<unsigned> Keep) {
// !annotation and !prof metadata does not impact semantics.
// !range, !nonnull and !align produce poison, so they are safe to speculate.
// !fpmath specifies floating-point precision and does not imply UB.
+ // !mem.cache_hint is a performance hint and does not imply UB.
// !noundef and various AA metadata must be dropped, as it generally produces
// immediate undefined behavior.
static const unsigned KnownIDs[] = {
- LLVMContext::MD_annotation, LLVMContext::MD_range,
- LLVMContext::MD_nonnull, LLVMContext::MD_align,
- LLVMContext::MD_fpmath, LLVMContext::MD_prof};
+ LLVMContext::MD_annotation, LLVMContext::MD_range,
+ LLVMContext::MD_nonnull, LLVMContext::MD_align,
+ LLVMContext::MD_fpmath, LLVMContext::MD_prof,
+ LLVMContext::MD_mem_cache_hint};
SmallVector<unsigned> KeepIDs;
KeepIDs.reserve(Keep.size() + std::size(KnownIDs));
append_range(KeepIDs, (!ProfcheckDisableMetadataFixes ? KnownIDs
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 86ec5dd522146..89a0bf6f4de5a 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -552,6 +552,7 @@ class Verifier : public InstVisitor<Verifier>, VerifierSupport {
void visitCapturesMetadata(Instruction &I, const MDNode *Captures);
void visitAllocTokenMetadata(Instruction &I, MDNode *MD);
void visitInlineHistoryMetadata(Instruction &I, MDNode *MD);
+ void visitMemCacheHintMetadata(Instruction &I, MDNode *MD);
template <class Ty> bool isValidMetadataArray(const MDTuple &N);
#define HANDLE_SPECIALIZED_MDNODE_LEAF(CLASS) void visit##CLASS(const CLASS &N);
@@ -5697,6 +5698,80 @@ void Verifier::visitInlineHistoryMetadata(Instruction &I, MDNode *MD) {
}
}
+void Verifier::visitMemCacheHintMetadata(Instruction &I, MDNode *MD) {
+ Check(I.mayReadOrWriteMemory(),
+ "!mem.cache_hint is only valid on memory operations", &I);
+
+ Check(MD->getNumOperands() % 2 == 0,
+ "!mem.cache_hint must have even number of operands "
+ "(operand_no, hint_node pairs)",
+ MD);
+
+ const auto *CB = dyn_cast<CallBase>(&I);
+ if (CB)
+ Check(CB->getIntrinsicID() != Intrinsic::not_intrinsic,
+ "!mem.cache_hint is not supported on non-intrinsic calls", &I);
+
+ unsigned NumOperands = CB ? CB->arg_size() : I.getNumOperands();
+
+ SmallDenseSet<unsigned, 4> SeenOperandNos;
+ std::optional<uint64_t> LastOperandNo;
+
+ // Top-level metadata alternates: i32 operand_no, MDNode hint_node.
+ for (unsigned J = 0; J + 1 < MD->getNumOperands(); J += 2) {
+ auto *OpNoCI = mdconst::dyn_extract<ConstantInt>(MD->getOperand(J));
+ Check(OpNoCI,
+ "!mem.cache_hint must alternate between i32 operand numbers and "
+ "metadata hint nodes",
+ MD);
+
+ Check(OpNoCI->getValue().isNonNegative(),
+ "!mem.cache_hint operand number must be non-negative", MD);
+
+ uint64_t OperandNo = OpNoCI->getZExtValue();
+ Check(OperandNo < NumOperands,
+ "!mem.cache_hint operand number is out of range", &I);
+
+ Value *Operand =
+ CB ? CB->getArgOperand(OperandNo) : I.getOperand(OperandNo);
+ Check(Operand->getType()->isPtrOrPtrVectorTy(),
+ "!mem.cache_hint operand number must refer to a pointer operand", &I);
+
+ bool Inserted = SeenOperandNos.insert(OperandNo).second;
+ Check(Inserted, "!mem.cache_hint contains duplicate operand number", MD);
+
+ Check(!Inserted || !LastOperandNo || OperandNo > *LastOperandNo,
+ "!mem.cache_hint operand numbers must be in increasing order", MD);
+ LastOperandNo = OperandNo;
+
+ const auto *Node = dyn_cast<MDNode>(MD->getOperand(J + 1));
+ Check(Node,
+ "!mem.cache_hint must alternate between i32 operand numbers and "
+ "metadata hint nodes",
+ MD);
+
+ Check(Node->getNumOperands() % 2 == 0,
+ "!mem.cache_hint hint node must have even number of operands "
+ "(key-value pairs)",
+ Node);
+
+ StringSet<> SeenKeys;
+ for (unsigned K = 0; K + 1 < Node->getNumOperands(); K += 2) {
+ const auto *Key = dyn_cast<MDString>(Node->getOperand(K));
+ Check(Key, "!mem.cache_hint key must be a string", Node);
+
+ StringRef KeyStr = Key->getString();
+ Check(SeenKeys.insert(KeyStr).second,
+ "!mem.cache_hint hint node contains duplicate key", Node);
+
+ const Metadata *Value = Node->getOperand(K + 1).get();
+ Check(isa_and_nonnull<MDString>(Value) ||
+ mdconst::dyn_extract<ConstantInt>(Value),
+ "!mem.cache_hint value must be a string or integer", Node);
+ }
+ }
+}
+
/// verifyInstruction - Verify that an instruction is well formed.
///
void Verifier::visitInstruction(Instruction &I) {
@@ -5932,6 +6007,9 @@ void Verifier::visitInstruction(Instruction &I) {
if (MDNode *MD = I.getMetadata(LLVMContext::MD_inline_history))
visitInlineHistoryMetadata(I, MD);
+ if (MDNode *MD = I.getMetadata(LLVMContext::MD_mem_cache_hint))
+ visitMemCacheHintMetadata(I, MD);
+
if (MDNode *N = I.getDebugLoc().getAsMDNode()) {
CheckDI(isa<DILocation>(N), "invalid !dbg metadata attachment", &I, N);
visitMDNode(*N, AreDebugLocsAllowed::Yes);
diff --git a/llvm/lib/Transforms/Utils/Local.cpp b/llvm/lib/Transforms/Utils/Local.cpp
index e812c00936097..711130c5c34f0 100644
--- a/llvm/lib/Transforms/Utils/Local.cpp
+++ b/llvm/lib/Transforms/Utils/Local.cpp
@@ -3058,6 +3058,12 @@ static void combineMetadata(Instruction *K, const Instruction *J,
if (!AAOnly)
K->setMetadata(Kind, JMD);
break;
+ case LLVMContext::MD_mem_cache_hint:
+ // Preserve !mem.cache_hint only if it is present and equivalent on both
+ // instructions.
+ if (!AAOnly && KMD != JMD)
+ K->setMetadata(Kind, nullptr);
+ break;
case LLVMContext::MD_noalias_addrspace:
if (DoesKMove)
K->setMetadata(Kind,
@@ -3075,9 +3081,7 @@ static void combineMetadata(Instruction *K, const Instruction *J,
break;
case LLVMContext::MD_alloc_token:
// Preserve !alloc_token if both K and J have it, and they are equal.
- if (KMD == JMD)
- K->setMetadata(Kind, JMD);
- else
+ if (KMD != JMD)
K->setMetadata(Kind, nullptr);
break;
}
@@ -3168,6 +3172,7 @@ void llvm::copyMetadataForLoad(LoadInst &Dest, const LoadInst &Source) {
case LLVMContext::MD_alias_scope:
case LLVMContext::MD_noalias:
case LLVMContext::MD_nontemporal:
+ case LLVMContext::MD_mem_cache_hint:
case LLVMContext::MD_mem_parallel_loop_access:
case LLVMContext::MD_access_group:
case LLVMContext::MD_noundef:
diff --git a/llvm/test/Bitcode/compatibility.ll b/llvm/test/Bitcode/compatibility.ll
index adb50a704fb02..22e60cbfa3d10 100644
--- a/llvm/test/Bitcode/compatibility.ll
+++ b/llvm/test/Bitcode/compatibility.ll
@@ -1625,6 +1625,10 @@ define void @instructions.aggregateops({ i8, i32 } %up, <{ i8, i32 }> %p,
!7 = !{i32 1}
!8 = !{}
!9 = !{i64 4}
+!10 = !{i32 0, !11}
+!11 = !{!"nvvm.l1_eviction", !"first", !"nvvm.l2_prefetch_size", i32 128}
+!12 = !{i32 1, !13}
+!13 = !{!"nvvm.l1_eviction", !"last"}
define void @instructions.memops(ptr %base) {
alloca i32, i8 4, align 4
; CHECK: alloca i32, i8 4, align 4
@@ -1635,11 +1639,15 @@ define void @instructions.memops(ptr %base) {
; CHECK: load ptr, ptr %base, align 8, !invariant.load !7, !nontemporal !8, !nonnull !8, !dereferenceable !9, !dereferenceable_or_null !9
load volatile ptr, ptr %base, align 8, !invariant.load !7, !nontemporal !8, !nonnull !8, !dereferenceable !9, !dereferenceable_or_null !9
; CHECK: load volatile ptr, ptr %base, align 8, !invariant.load !7, !nontemporal !8, !nonnull !8, !dereferenceable !9, !dereferenceable_or_null !9
+ load ptr, ptr %base, align 8, !mem.cache_hint !10
+ ; CHECK: load ptr, ptr %base, align 8, !mem.cache_hint !10
store ptr null, ptr %base, align 4, !nontemporal !8
; CHECK: store ptr null, ptr %base, align 4, !nontemporal !8
store volatile ptr null, ptr %base, align 4, !nontemporal !8
; CHECK: store volatile ptr null, ptr %base, align 4, !nontemporal !8
+ store ptr null, ptr %base, align 4, !mem.cache_hint !12
+ ; CHECK: store ptr null, ptr %base, align 4, !mem.cache_hint !12
ret void
}
@@ -1894,21 +1902,21 @@ declare i64 @llvm.readcyclecounter()
declare void @llvm.clear_cache(ptr, ptr)
declare void @llvm.instrprof_increment(ptr, i64, i32, i32)
-!10 = !{!"rax"}
+!14 = !{!"rax"}
define void @intrinsics.codegen() {
call ptr @llvm.returnaddress(i32 1)
; CHECK: call ptr @llvm.returnaddress.p0(i32 1)
call ptr @llvm.frameaddress(i32 1)
; CHECK: call ptr @llvm.frameaddress.p0(i32 1)
- call i32 @llvm.read_register.i32(metadata !10)
- ; CHECK: call i32 @llvm.read_register.i32(metadata !10)
- call i64 @llvm.read_register.i64(metadata !10)
- ; CHECK: call i64 @llvm.read_register.i64(metadata !10)
- call void @llvm.write_register.i32(metadata !10, i32 0)
- ; CHECK: call void @llvm.write_register.i32(metadata !10, i32 0)
- call void @llvm.write_register.i64(metadata !10, i64 0)
- ; CHECK: call void @llvm.write_register.i64(metadata !10, i64 0)
+ call i32 @llvm.read_register.i32(metadata !14)
+ ; CHECK: call i32 @llvm.read_register.i32(metadata !14)
+ call i64 @llvm.read_register.i64(metadata !14)
+ ; CHECK: call i64 @llvm.read_register.i64(metadata !14)
+ call void @llvm.write_register.i32(metadata !14, i32 0)
+ ; CHECK: call void @llvm.write_register.i32(metadata !14, i32 0)
+ call void @llvm.write_register.i64(metadata !14, i64 0)
+ ; CHECK: call void @llvm.write_register.i64(metadata !14, i64 0)
%stack = call ptr @llvm.stacksave()
; CHECK: %stack = call ptr @llvm.stacksave.p0()
@@ -1954,10 +1962,10 @@ define void @intrinsics.localrecover() {
; We need this function to provide `uses' for some metadata tests.
define void @misc.metadata() {
- call void @f1(), !srcloc !11
- call void @f1(), !srcloc !12
- call void @f1(), !srcloc !13
- call void @f1(), !srcloc !14
+ call void @f1(), !srcloc !15
+ call void @f1(), !srcloc !16
+ call void @f1(), !srcloc !17
+ call void @f1(), !srcloc !18
ret void
}
@@ -2578,11 +2586,11 @@ define void @denormal_fpenv__preservesign_preservesign_float_dynamic_dynamic() d
; CHECK: !6 = !{i32 6, !"mod6", !0}
; Metadata -- Check `distinct'
-!11 = distinct !{}
-; CHECK: !11 = distinct !{}
-!12 = distinct !{}
-; CHECK: !12 = distinct !{}
-!13 = !{!11}
-; CHECK: !13 = !{!11}
-!14 = !{!12}
-; CHECK: !14 = !{!12}
+!15 = distinct !{}
+; CHECK: !15 = distinct !{}
+!16 = distinct !{}
+; CHECK: !16 = distinct !{}
+!17 = !{!15}
+; CHECK: !17 = !{!15}
+!18 = !{!16}
+; CHECK: !18 = !{!16}
diff --git a/llvm/test/Transforms/GVN/metadata.ll b/llvm/test/Transforms/GVN/metadata.ll
index ff055d889eac2..c2725e99570c6 100644
--- a/llvm/test/Transforms/GVN/metadata.ll
+++ b/llvm/test/Transforms/GVN/metadata.ll
@@ -485,6 +485,63 @@ join:
ret void
}
+; Both loads have the same !mem.cache_hint node so we preserve it after GVN deduplication.
+define i64 @test_mem_cache_hint_both(ptr %p) {
+; CHECK-LABEL: define i64 @test_mem_cache_hint_both
+; CHECK-SAME: (ptr [[P:%.*]]) {
+; CHECK-NEXT: [[A:%.*]] = load i64, ptr [[P]], align 4, !mem.cache_hint [[META11:![0-9]+]]
+; CHECK-NEXT: [[C:%.*]] = add i64 [[A]], [[A]]
+; CHECK-NEXT: ret i64 [[C]]
+;
+ %a = load i64, ptr %p, !mem.cache_hint !12
+ %b = load i64, ptr %p, !mem.cache_hint !12
+ %c = add i64 %a, %b
+ ret i64 %c
+}
+
+; Cache hint values can be integer constants.
+define i64 @test_mem_cache_hint_integer_value(ptr %p) {
+; CHECK-LABEL: define i64 @test_mem_cache_hint_integer_value
+; CHECK-SAME: (ptr [[P:%.*]]) {
+; CHECK-NEXT: [[A:%.*]] = load i64, ptr [[P]], align 4, !mem.cache_hint [[META13:![0-9]+]]
+; CHECK-NEXT: [[C:%.*]] = add i64 [[A]], [[A]]
+; CHECK-NEXT: ret i64 [[C]]
+;
+ %a = load i64, ptr %p, !mem.cache_hint !14
+ %b = load i64, ptr %p, !mem.cache_hint !14
+ %c = add i64 %a, %b
+ ret i64 %c
+}
+
+; Only one load has !mem.cache_hint so we drop the metadata after GVN deduplication.
+define i64 @test_mem_cache_hint_one(ptr %p) {
+; CHECK-LABEL: define i64 @test_mem_cache_hint_one
+; CHECK-SAME: (ptr [[P:%.*]]) {
+; CHECK-NEXT: [[A:%.*]] = load i64, ptr [[P]], align 4
+; CHECK-NEXT: [[C:%.*]] = add i64 [[A]], [[A]]
+; CHECK-NEXT: ret i64 [[C]]
+;
+ %a = load i64, ptr %p, !mem.cache_hint !12
+ %b = load i64, ptr %p
+ %c = add i64 %a, %b
+ ret i64 %c
+}
+
+; Both loads have
diff erent !mem.cache_hint metadata so we drop the metadata.
+; TODO: delegate to TTI to let targets decide how to merge
diff ering payloads.
+define i64 @test_mem_cache_hint_
diff (ptr %p) {
+; CHECK-LABEL: define i64 @test_mem_cache_hint_
diff
+; CHECK-SAME: (ptr [[P:%.*]]) {
+; CHECK-NEXT: [[A:%.*]] = load i64, ptr [[P]], align 4
+; CHECK-NEXT: [[C:%.*]] = add i64 [[A]], [[A]]
+; CHECK-NEXT: ret i64 [[C]]
+;
+ %a = load i64, ptr %p, !mem.cache_hint !12
+ %b = load i64, ptr %p, !mem.cache_hint !16
+ %c = add i64 %a, %b
+ ret i64 %c
+}
+
!0 = !{i32 0, i32 2}
!1 = !{i32 3, i32 5}
!2 = !{i32 2, i32 5}
@@ -497,6 +554,12 @@ join:
!9 = !{i32 1, i32 5}
!10 = !{i32 5, i32 1}
!11 = !{}
+!12 = !{ i32 0, !13 }
+!13 = !{ !"nvvm.l1_eviction", !"first" }
+!14 = !{ i32 0, !15 }
+!15 = !{ !"nvvm.prefetch_bytes", i32 128 }
+!16 = !{ i32 0, !17 }
+!17 = !{ !"nvvm.l1_eviction", !"last" }
;.
; CHECK: attributes #[[ATTR0:[0-9]+]] = { memory(none) }
;.
@@ -511,4 +574,8 @@ join:
; CHECK: [[RNG8]] = !{i64 0, i64 10}
; CHECK: [[RNG9]] = !{i64 0, i64 10, i64 20, i64 30}
; CHECK: [[RNG10]] = !{i64 10, i64 30}
+; CHECK: [[META11]] = !{i32 0, [[META12:![0-9]+]]}
+; CHECK: [[META12]] = !{!"nvvm.l1_eviction", !"first"}
+; CHECK: [[META13]] = !{i32 0, [[META14:![0-9]+]]}
+; CHECK: [[META14]] = !{!"nvvm.prefetch_bytes", i32 128}
;.
diff --git a/llvm/test/Transforms/InstCombine/loadstore-metadata.ll b/llvm/test/Transforms/InstCombine/loadstore-metadata.ll
index ec485edbb709b..ab270b2ee366f 100644
--- a/llvm/test/Transforms/InstCombine/loadstore-metadata.ll
+++ b/llvm/test/Transforms/InstCombine/loadstore-metadata.ll
@@ -396,6 +396,19 @@ entry:
ret <1 x float> %c
}
+define i32 @test_load_cast_combine_mem_cache_hint(ptr %ptr) {
+; CHECK-LABEL: define i32 @test_load_cast_combine_mem_cache_hint(
+; CHECK-SAME: ptr [[PTR:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*:]]
+; CHECK-NEXT: [[L1:%.*]] = load i32, ptr [[PTR]], align 4, !mem.cache_hint [[META17:![0-9]+]]
+; CHECK-NEXT: ret i32 [[L1]]
+;
+entry:
+ %l = load float, ptr %ptr, !mem.cache_hint !17
+ %c = bitcast float %l to i32
+ ret i32 %c
+}
+
!0 = !{!1, !1, i64 0}
!1 = !{!"scalar type", !2}
!2 = !{!"root"}
@@ -413,6 +426,8 @@ entry:
!14 = !{!15}
!15 = distinct !{!15, !16}
!16 = distinct !{!16}
+!17 = !{ i32 0, !18 }
+!18 = !{ !"nvvm.l1_eviction", !"first" }
;.
; CHECK: [[SCALAR_TYPE_TBAA0]] = !{[[LOOP1]], [[LOOP1]], i64 0}
@@ -432,4 +447,6 @@ entry:
; CHECK: [[META14]] = distinct !{[[META14]]}
; CHECK: [[ACC_GRP15]] = distinct !{}
; CHECK: [[META16]] = !{i32 3}
+; CHECK: [[META17]] = !{i32 0, [[META18:![0-9]+]]}
+; CHECK: [[META18]] = !{!"nvvm.l1_eviction", !"first"}
;.
diff --git a/llvm/test/Transforms/SimplifyCFG/hoist-with-metadata.ll b/llvm/test/Transforms/SimplifyCFG/hoist-with-metadata.ll
index 26e87393e2e7b..47aca1ae1cbe5 100644
--- a/llvm/test/Transforms/SimplifyCFG/hoist-with-metadata.ll
+++ b/llvm/test/Transforms/SimplifyCFG/hoist-with-metadata.ll
@@ -635,6 +635,60 @@ out:
}
+define void @hoist_mem_cache_hint_drop(i1 %c, ptr %p) {
+; CHECK-LABEL: @hoist_mem_cache_hint_drop(
+; CHECK-NEXT: if:
+; CHECK-NEXT: [[T:%.*]] = load i32, ptr [[P:%.*]], align 4
+; CHECK-NEXT: ret void
+;
+if:
+ br i1 %c, label %then, label %else
+then:
+ %t = load i32, ptr %p, !mem.cache_hint !10
+ br label %out
+else:
+ %e = load i32, ptr %p, !mem.cache_hint !13
+ br label %out
+out:
+ ret void
+}
+
+define void @hoist_mem_cache_hint_drop2(i1 %c, ptr %p) {
+; CHECK-LABEL: @hoist_mem_cache_hint_drop2(
+; CHECK-NEXT: if:
+; CHECK-NEXT: [[T:%.*]] = load i32, ptr [[P:%.*]], align 4
+; CHECK-NEXT: ret void
+;
+if:
+ br i1 %c, label %then, label %else
+then:
+ %t = load i32, ptr %p, !mem.cache_hint !10
+ br label %out
+else:
+ %e = load i32, ptr %p
+ br label %out
+out:
+ ret void
+}
+
+define void @hoist_mem_cache_hint_keep(i1 %c, ptr %p) {
+; CHECK-LABEL: @hoist_mem_cache_hint_keep(
+; CHECK-NEXT: if:
+; CHECK-NEXT: store i32 0, ptr [[P:%.*]], align 4, !mem.cache_hint [[META13:![0-9]+]]
+; CHECK-NEXT: ret void
+;
+if:
+ br i1 %c, label %then, label %else
+then:
+ store i32 0, ptr %p, !mem.cache_hint !11
+ br label %out
+else:
+ store i32 0, ptr %p, !mem.cache_hint !11
+ br label %out
+out:
+ ret void
+}
+
!0 = !{ i8 0, i8 1 }
!1 = !{ i8 3, i8 5 }
!2 = !{}
@@ -645,7 +699,11 @@ out:
!7 = !{i32 4, i32 8, i32 20, i32 31}
!8 = !{i32 2, i32 5}
!9 = !{i32 2, i32 5, i32 22, i32 42, i32 45, i32 50}
-
+!10 = !{ i32 0, !12 }
+!11 = !{ i32 1, !12 }
+!12 = !{ !"nvvm.l1_eviction", !"first" }
+!13 = !{ i32 0, !14 }
+!14 = !{ !"nvvm.l1_eviction", !"last" }
;.
; CHECK: [[RNG0]] = !{i8 0, i8 1, i8 3, i8 5}
; CHECK: [[RNG1]] = !{i8 0, i8 1, i8 3, i8 5, i8 7, i8 9}
@@ -660,4 +718,6 @@ out:
; CHECK: [[META10]] = !{!"address", !"read_provenance"}
; CHECK: [[META11]] = !{!"provenance"}
; CHECK: [[META12]] = !{i32 3}
+; CHECK: [[META13]] = !{i32 1, [[META14:![0-9]+]]}
+; CHECK: [[META14]] = !{!"nvvm.l1_eviction", !"first"}
;.
diff --git a/llvm/test/Verifier/mem-cache-hint.ll b/llvm/test/Verifier/mem-cache-hint.ll
new file mode 100644
index 0000000000000..110eadedd4d6e
--- /dev/null
+++ b/llvm/test/Verifier/mem-cache-hint.ll
@@ -0,0 +1,100 @@
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
+
+declare void @foo(ptr)
+declare void @llvm.memcpy.p0.p0.i64(ptr noalias writeonly, ptr noalias readonly, i64, i1 immarg)
+
+; CHECK: !mem.cache_hint is only valid on memory operations
+define void @non_memory_op(i32 %x, i32 %y) {
+ %z = add i32 %x, %y, !mem.cache_hint !{i32 0, !{!"nvvm.l1_eviction", !"first"}}
+ ret void
+}
+
+; CHECK: !mem.cache_hint must have even number of operands
+define void @odd_top_level_operands(ptr %p) {
+ %v = load i32, ptr %p, !mem.cache_hint !{i32 0}
+ ret void
+}
+
+; CHECK: !mem.cache_hint must alternate between i32 operand numbers and metadata hint nodes
+define void @operand_no_not_integer(ptr %p) {
+ %v = load i32, ptr %p, !mem.cache_hint !{!"zero", !{!"nvvm.l1_eviction", !"first"}}
+ ret void
+}
+
+; CHECK: !mem.cache_hint operand number must be non-negative
+define void @operand_no_negative(ptr %p) {
+ %v = load i32, ptr %p, !mem.cache_hint !{i32 -1, !{!"nvvm.l1_eviction", !"first"}}
+ ret void
+}
+
+; CHECK: !mem.cache_hint is not supported on non-intrinsic calls
+define void @non_intrinsic_call(ptr %p) {
+ call void @foo(ptr %p), !mem.cache_hint !{i32 0, !{!"nvvm.l1_eviction", !"first"}}
+ ret void
+}
+
+; CHECK: !mem.cache_hint operand number must refer to a pointer operand
+define void @operand_no_not_pointer(ptr %d, ptr %s) {
+ call void @llvm.memcpy.p0.p0.i64(ptr %d, ptr %s, i64 8, i1 false), !mem.cache_hint !{i32 2, !{!"nvvm.l1_eviction", !"first"}}
+ ret void
+}
+
+; CHECK: !mem.cache_hint operand number must refer to a pointer operand
+define void @store_operand_no_is_value(ptr %p) {
+ store i32 0, ptr %p, !mem.cache_hint !{i32 0, !{!"nvvm.l1_eviction", !"first"}}
+ ret void
+}
+
+; CHECK: !mem.cache_hint operand number is out of range
+define void @operand_no_out_of_range(ptr %p) {
+ %v = load i32, ptr %p, !mem.cache_hint !{i32 1, !{!"nvvm.l1_eviction", !"first"}}
+ ret void
+}
+
+; CHECK: !mem.cache_hint contains duplicate operand number
+define void @duplicate_operand_no(ptr %p) {
+ call void @llvm.memcpy.p0.p0.i64(ptr %p, ptr %p, i64 8, i1 false), !mem.cache_hint !{
+ i32 0, !{!"nvvm.l1_eviction", !"first"},
+ i32 0, !{!"nvvm.l1_eviction", !"last"}}
+ ret void
+}
+
+; CHECK: !mem.cache_hint operand numbers must be in increasing order
+define void @operand_no_not_increasing(ptr %d, ptr %s) {
+ call void @llvm.memcpy.p0.p0.i64(ptr %d, ptr %s, i64 8, i1 false), !mem.cache_hint !{
+ i32 1, !{!"nvvm.l1_eviction", !"first"},
+ i32 0, !{!"nvvm.l1_eviction", !"last"}}
+ ret void
+}
+
+; CHECK: !mem.cache_hint must alternate between i32 operand numbers and metadata hint nodes
+define void @hint_node_not_mdnode(ptr %p) {
+ %v = load i32, ptr %p, !mem.cache_hint !{i32 0, !"not_a_node"}
+ ret void
+}
+
+; CHECK: !mem.cache_hint hint node must have even number of operands
+define void @hint_node_odd_operands(ptr %p) {
+ %v = load i32, ptr %p, !mem.cache_hint !{i32 0, !{!"nvvm.l1_eviction"}}
+ ret void
+}
+
+; CHECK: !mem.cache_hint key must be a string
+define void @key_not_string(ptr %p) {
+ %v = load i32, ptr %p, !mem.cache_hint !{i32 0, !{i32 0, !"first"}}
+ ret void
+}
+
+; CHECK: !mem.cache_hint value must be a string or integer
+define void @value_not_string_or_integer(ptr %p) {
+ %v = load i32, ptr %p, !mem.cache_hint !{i32 0, !{!"nvvm.l1_eviction", !{}}}
+ ret void
+}
+
+; CHECK: !mem.cache_hint hint node contains duplicate key
+define void @duplicate_key(ptr %p) {
+ %v = load i32, ptr %p, !mem.cache_hint !{i32 0, !{
+ !"nvvm.l1_eviction", !"first",
+ !"nvvm.l1_eviction", !"last"}}
+ ret void
+}
More information about the llvm-commits
mailing list