[llvm] Provide intrinsics for speculative loads (PR #179642)
Florian Hahn via llvm-commits
llvm-commits at lists.llvm.org
Tue May 5 06:47:27 PDT 2026
https://github.com/fhahn updated https://github.com/llvm/llvm-project/pull/179642
>From 02708e4a06bff205d32641c96ccf884a32897be0 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Wed, 4 Feb 2026 11:06:38 +0000
Subject: [PATCH 1/9] Add speculative load intrinsics
Introduce two new intrinsics to enable vectorization of loops with early
exits that have potentially faulting loads.
This has previously been discussed in
https://github.com/llvm/llvm-project/pull/120603 and is similar to
@nikic's https://hackmd.io/@nikic/S1O4QWYZkx, with the major difference
being that there is no `%defined_size` argument and instead the load
returns the stored values for the bytes within bounds and undef
otherwise. I don't think we can easily compute the defined size because
it may depend on the loaded values (i.e. at what lane the early exit has
been taken).
1. `@llvm.speculative.load` (name subject to change) - perform a load that
may access memory beyond the allocated object. It must be used in
combination with `@llvm.can.load.speculatively` to ensure the load is
guaranteed to not trap.
2. `@llvm.can.load.speculatively` - Returns true if it's safe to speculatively
load a given number of bytes from a pointer. The semantics are
target-dependent. On some targets, this may check that the access
does not cross page boundaries, or stricter checks for example on
AArch64 with MTE, which limits the access size to 16 bytes.
`@llvm.speculative.load` is lowered to a regular load in SelectionDAG
without MODereferenceable. I am not sure if we need to be more careful
than this, i.e. if we could still reason about SelectionDAG loads to
infer dereferencability for the pointer.
`@llvm.can.load.speculatively` is lowered to regular IR in PreISel
lowering, using a target-lowering hook. By default, it conservatively
expands to false.
These intrinsics should allow the loop vectorizer to vectorize early-exit
loops with potentially non-dereferenceable loads.
---
llvm/docs/LangRef.rst | 113 ++++++++++++++
llvm/include/llvm/CodeGen/TargetLowering.h | 13 ++
llvm/include/llvm/IR/Intrinsics.td | 14 ++
llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp | 39 +++++
.../SelectionDAG/SelectionDAGBuilder.cpp | 30 ++++
.../SelectionDAG/SelectionDAGBuilder.h | 1 +
llvm/lib/IR/Verifier.cpp | 18 +++
.../Target/AArch64/AArch64ISelLowering.cpp | 49 ++++++
llvm/lib/Target/AArch64/AArch64ISelLowering.h | 2 +
.../CodeGen/AArch64/can-load-speculatively.ll | 78 ++++++++++
.../AArch64/speculative-load-intrinsic-sve.ll | 66 ++++++++
.../AArch64/speculative-load-intrinsic.ll | 117 ++++++++++++++
.../CodeGen/X86/can-load-speculatively.ll | 32 ++++
.../CodeGen/X86/speculative-load-intrinsic.ll | 146 ++++++++++++++++++
llvm/test/Verifier/can-load-speculatively.ll | 19 +++
llvm/test/Verifier/speculative-load.ll | 18 +++
16 files changed, 755 insertions(+)
create mode 100644 llvm/test/CodeGen/AArch64/can-load-speculatively.ll
create mode 100644 llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
create mode 100644 llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
create mode 100644 llvm/test/CodeGen/X86/can-load-speculatively.ll
create mode 100644 llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
create mode 100644 llvm/test/Verifier/can-load-speculatively.ll
create mode 100644 llvm/test/Verifier/speculative-load.ll
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 83bd6c7e95ac5..981034d5b43eb 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28184,6 +28184,119 @@ Semantics:
Follows the same semantics as :ref:`srem <i_srem>` with the exception that disabled lanes cannot produce undefined behaviour and always result in poison.
+Speculative Load Intrinsics
+---------------------------
+
+LLVM provides intrinsics for speculatively loading memory that may be
+out-of-bounds. These intrinsics enable optimizations like early-exit loop
+vectorization where the vectorized loop may read beyond the end of an array,
+provided the access is guaranteed to not trap by target-specific checks.
+
+.. _int_speculative_load:
+
+'``llvm.speculative.load``' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic.
+
+::
+
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>)
+ declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>)
+ declare i64 @llvm.speculative.load.i64.p0(ptr <ptr>)
+
+Overview:
+"""""""""
+
+The '``llvm.speculative.load``' intrinsic loads a value from memory. Unlike a
+regular load, the memory access may
+extend beyond the bounds of the allocated object, provided the pointer has been
+verified by :ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to
+ensure the access cannot fault.
+
+Arguments:
+""""""""""
+
+The argument is a pointer to the memory location to load from. The return type
+must have a power-of-2 size in bytes.
+
+Semantics:
+""""""""""
+
+The '``llvm.speculative.load``' intrinsic performs a load that may access
+memory beyond the allocated object. It must be used in combination with
+:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure
+the access cannot fault.
+
+For bytes that are within the bounds of the allocated object, the intrinsic
+returns the stored value. For bytes that are beyond the bounds of the
+allocated object, the intrinsic returns ``undef`` for those bytes. At least the
+first accessed byte must be within the bounds of an allocated object the pointer is
+based on.
+
+The behavior is undefined if this intrinsic is used to load from a pointer
+for which ``llvm.can.load.speculatively`` returns false.
+
+.. _int_can_load_speculatively:
+
+'``llvm.can.load.speculatively``' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic.
+
+::
+
+ declare i1 @llvm.can.load.speculatively.p0(ptr <ptr>, i64 <num_bytes>)
+ declare i1 @llvm.can.load.speculatively.p1(ptr addrspace(1) <ptr>, i64 <num_bytes>)
+
+Overview:
+"""""""""
+
+The '``llvm.can.load.speculatively``' intrinsic returns true if it is safe
+to speculatively load ``num_bytes`` bytes starting from ``ptr``,
+even if the memory may be beyond the bounds of an allocated object.
+
+Arguments:
+""""""""""
+
+The first argument is a pointer to the memory location.
+
+The second argument is an i64 specifying the size in bytes of the load.
+The size must be a positive power of 2. If the size is not a power-of-2, the
+result is ``poison``.
+
+Semantics:
+""""""""""
+
+This intrinsic has **target-dependent** semantics. It returns ``true`` if
+loading ``num_bytes`` bytes from ``ptr`` is guaranteed not to trap,
+even if the memory is beyond the bounds of an allocated object. It returns
+``false`` otherwise.
+
+The specific conditions under which this intrinsic returns ``true`` are
+determined by the target. For example, a target may check whether the pointer
+alignment guarantees the load cannot cross a page boundary.
+
+.. code-block:: llvm
+
+ ; Check if we can safely load 16 bytes from %ptr
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ br i1 %can_load, label %speculative_path, label %safe_path
+
+ speculative_path:
+ ; Safe to speculatively load from %ptr
+ %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr)
+ ...
+
+ safe_path:
+ ; Fall back to masked load or scalar operations
+ ...
+
+
Memory Use Markers
------------------
diff --git a/llvm/include/llvm/CodeGen/TargetLowering.h b/llvm/include/llvm/CodeGen/TargetLowering.h
index 9b0bfa111f2d5..199fcbc5e7da5 100644
--- a/llvm/include/llvm/CodeGen/TargetLowering.h
+++ b/llvm/include/llvm/CodeGen/TargetLowering.h
@@ -2325,6 +2325,19 @@ class LLVM_ABI TargetLoweringBase {
llvm_unreachable("Store conditional unimplemented on this target");
}
+ /// Emit code to check if a speculative load of the given size from Ptr is
+ /// safe. Returns a Value* representing the check result (i1), or nullptr
+ /// to use the default lowering (which returns false). Targets can override
+ /// to provide their own safety check (e.g., alignment-based page boundary
+ /// check).
+ /// \param Builder IRBuilder positioned at the intrinsic call site
+ /// \param Ptr the pointer operand
+ /// \param Size the size in bytes (constant or runtime value for scalable)
+ virtual Value *emitCanLoadSpeculatively(IRBuilderBase &Builder, Value *Ptr,
+ Value *Size) const {
+ return nullptr;
+ }
+
/// Perform a masked atomicrmw using a target-specific intrinsic. This
/// represents the core LL/SC loop which will be lowered at a late stage by
/// the backend. The target-specific intrinsic returns the loaded value and
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 993ddd7e33701..49cc434809360 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2652,6 +2652,20 @@ def int_masked_srem:
LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>],
[IntrNoMem]>;
+// Speculatively load a value from memory; lowers to a regular aligned load.
+// The loaded type must have a power-of-2 size.
+def int_speculative_load:
+ DefaultAttrsIntrinsic<[llvm_any_ty],
+ [llvm_anyptr_ty],
+ [IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
+
+// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
+// The size can be a runtime value to support scalable vectors.
+def int_can_load_speculatively:
+ DefaultAttrsIntrinsic<[llvm_i1_ty],
+ [llvm_anyptr_ty, llvm_i64_ty],
+ [IntrNoMem, IntrSpeculatable, IntrWillReturn]>;
+
// Test whether a pointer is associated with a type metadata identifier.
def int_type_test : DefaultAttrsIntrinsic<[llvm_i1_ty], [llvm_ptr_ty, llvm_metadata_ty],
[IntrNoMem, IntrSpeculatable]>;
diff --git a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
index 2fa92b61bb297..06a65e7375f00 100644
--- a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
+++ b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
@@ -136,6 +136,42 @@ static bool lowerLoadRelative(Function &F) {
return Changed;
}
+/// Lower @llvm.can.load.speculatively using target-specific expansion.
+/// Each target provides its own expansion via
+/// TargetLowering::emitCanLoadSpeculatively.
+/// The default expansion returns false (conservative).
+static bool lowerCanLoadSpeculatively(Function &F, const TargetMachine *TM) {
+ if (F.use_empty())
+ return false;
+
+ bool Changed = false;
+
+ for (Use &U : llvm::make_early_inc_range(F.uses())) {
+ auto *CI = dyn_cast<CallInst>(U.getUser());
+ if (!CI || CI->getCalledOperand() != &F)
+ continue;
+
+ Function *ParentFunc = CI->getFunction();
+ const TargetLowering *TLI =
+ TM->getSubtargetImpl(*ParentFunc)->getTargetLowering();
+
+ IRBuilder<> Builder(CI);
+ Value *Ptr = CI->getArgOperand(0);
+ Value *Size = CI->getArgOperand(1);
+
+ // Ask target for expansion; nullptr means use default (return false)
+ Value *Result = TLI->emitCanLoadSpeculatively(Builder, Ptr, Size);
+ if (!Result)
+ Result = Builder.getFalse();
+
+ CI->replaceAllUsesWith(Result);
+ CI->eraseFromParent();
+ Changed = true;
+ }
+
+ return Changed;
+}
+
// ObjCARC has knowledge about whether an obj-c runtime function needs to be
// always tail-called or never tail-called.
static CallInst::TailCallKind getOverridingTailCallKind(const Function &F) {
@@ -674,6 +710,9 @@ bool PreISelIntrinsicLowering::lowerIntrinsics(Module &M) const {
case Intrinsic::load_relative:
Changed |= lowerLoadRelative(F);
break;
+ case Intrinsic::can_load_speculatively:
+ Changed |= lowerCanLoadSpeculatively(F, TM);
+ break;
case Intrinsic::is_constant:
case Intrinsic::objectsize:
Changed |= forEachCall(F, [&](CallInst *CI) {
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 894f04bb4668d..91ddc8d53357e 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5145,6 +5145,33 @@ void SelectionDAGBuilder::visitMaskedLoad(const CallInst &I, bool IsExpanding) {
setValue(&I, Res);
}
+void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
+ SDLoc sdl = getCurSDLoc();
+ Value *PtrOperand = I.getArgOperand(0);
+ SDValue Ptr = getValue(PtrOperand);
+
+ const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+ EVT VT = TLI.getValueType(DAG.getDataLayout(), I.getType());
+ Align Alignment = I.getParamAlign(0).valueOrOne();
+ AAMDNodes AAInfo = I.getAAMetadata();
+ TypeSize StoreSize = VT.getStoreSize();
+
+ SDValue InChain = DAG.getRoot();
+
+ // Use MOLoad but NOT MODereferenceable - the memory may not be
+ // fully dereferenceable.
+ MachineMemOperand::Flags MMOFlags = MachineMemOperand::MOLoad;
+ LocationSize LocSize = StoreSize.isScalable()
+ ? LocationSize::beforeOrAfterPointer()
+ : LocationSize::precise(StoreSize);
+ MachineMemOperand *MMO = DAG.getMachineFunction().getMachineMemOperand(
+ MachinePointerInfo(PtrOperand), MMOFlags, LocSize, Alignment, AAInfo);
+
+ SDValue Load = DAG.getLoad(VT, sdl, InChain, Ptr, MMO);
+ PendingLoads.push_back(Load.getValue(1));
+ setValue(&I, Load);
+}
+
void SelectionDAGBuilder::visitMaskedGather(const CallInst &I) {
SDLoc sdl = getCurSDLoc();
@@ -6906,6 +6933,9 @@ void SelectionDAGBuilder::visitIntrinsicCall(const CallInst &I,
case Intrinsic::masked_compressstore:
visitMaskedStore(I, true /* IsCompressing */);
return;
+ case Intrinsic::speculative_load:
+ visitSpeculativeLoad(I);
+ return;
case Intrinsic::powi:
setValue(&I, ExpandPowI(sdl, getValue(I.getArgOperand(0)),
getValue(I.getArgOperand(1)), DAG));
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
index 21aac333a73cd..1f5405acd7045 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
@@ -620,6 +620,7 @@ class SelectionDAGBuilder {
void visitStore(const StoreInst &I);
void visitMaskedLoad(const CallInst &I, bool IsExpanding = false);
void visitMaskedStore(const CallInst &I, bool IsCompressing = false);
+ void visitSpeculativeLoad(const CallInst &I);
void visitMaskedGather(const CallInst &I);
void visitMaskedScatter(const CallInst &I);
void visitAtomicCmpXchg(const AtomicCmpXchgInst &I);
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 2ea113fe665d9..1c24473f91481 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -6854,6 +6854,24 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
&Call);
break;
}
+ case Intrinsic::speculative_load: {
+ Type *LoadTy = Call.getType();
+ TypeSize Size = DL.getTypeStoreSize(LoadTy);
+ // For scalable vectors, check the known minimum size is a power of 2.
+ Check(Size.getKnownMinValue() > 0 && isPowerOf2_64(Size.getKnownMinValue()),
+ "llvm.speculative.load type must have a power-of-2 size", &Call);
+ break;
+ }
+ case Intrinsic::can_load_speculatively: {
+ // If size is a constant, verify it's a positive power of 2.
+ if (auto *SizeCI = dyn_cast<ConstantInt>(Call.getArgOperand(1))) {
+ uint64_t Size = SizeCI->getZExtValue();
+ Check(Size > 0 && isPowerOf2_64(Size),
+ "llvm.can.load.speculatively size must be a positive power of 2",
+ &Call);
+ }
+ break;
+ }
case Intrinsic::vector_insert: {
Value *Vec = Call.getArgOperand(0);
Value *SubVec = Call.getArgOperand(1);
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
index 356ebdf407cff..f33f5ab242c04 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
@@ -30949,6 +30949,55 @@ Value *AArch64TargetLowering::emitStoreConditional(IRBuilderBase &Builder,
return CI;
}
+Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
+ Value *Ptr,
+ Value *Size) const {
+ // For power-of-2 sizes <= 16, emit alignment check: (ptr & (size - 1)) == 0.
+ // If the pointer is aligned to at least 'size' bytes, loading 'size' bytes
+ // cannot cross a page boundary, so it's safe to speculate.
+ // The 16-byte limit ensures correctness with MTE (memory tagging), since
+ // MTE uses 16-byte tag granules.
+ //
+ // The alignment check only works for power-of-2 sizes. For non-power-of-2
+ // sizes, we conservatively return false.
+ const DataLayout &DL =
+ Builder.GetInsertBlock()->getModule()->getDataLayout();
+
+ if (auto *CI = dyn_cast<ConstantInt>(Size)) {
+ uint64_t SizeVal = CI->getZExtValue();
+ assert(isPowerOf2_64(SizeVal) && "size must be power-of-two");
+ // For constant sizes > 16, return nullptr (default false).
+ if (SizeVal > 16)
+ return nullptr;
+
+ // Power-of-2 constant size <= 16: use fast alignment check.
+ unsigned PtrBits = DL.getPointerSizeInBits();
+ Type *IntPtrTy = Builder.getIntNTy(PtrBits);
+ Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
+ Value *Mask = ConstantInt::get(IntPtrTy, SizeVal - 1);
+ Value *Masked = Builder.CreateAnd(PtrInt, Mask);
+ return Builder.CreateICmpEQ(Masked, ConstantInt::get(IntPtrTy, 0));
+ }
+
+ // Check power-of-2 size <= 16 and alignment.
+ unsigned PtrBits = DL.getPointerSizeInBits();
+ Type *IntPtrTy = Builder.getIntNTy(PtrBits);
+ Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
+ Value *SizeExt = Builder.CreateZExtOrTrunc(Size, IntPtrTy);
+
+ Value *SizeLE16 =
+ Builder.CreateICmpULE(SizeExt, ConstantInt::get(IntPtrTy, 16));
+
+ // alignment check: (ptr & (size - 1)) == 0
+ Value *SizeMinusOne =
+ Builder.CreateSub(SizeExt, ConstantInt::get(IntPtrTy, 1));
+ Value *Masked = Builder.CreateAnd(PtrInt, SizeMinusOne);
+ Value *AlignCheck =
+ Builder.CreateICmpEQ(Masked, ConstantInt::get(IntPtrTy, 0));
+
+ return Builder.CreateAnd(SizeLE16, AlignCheck);
+}
+
bool AArch64TargetLowering::functionArgumentNeedsConsecutiveRegisters(
Type *Ty, CallingConv::ID CallConv, bool isVarArg,
const DataLayout &DL) const {
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.h b/llvm/lib/Target/AArch64/AArch64ISelLowering.h
index 80d49bb2ed318..9ef516354790c 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.h
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.h
@@ -340,6 +340,8 @@ class AArch64TargetLowering : public TargetLowering {
AtomicOrdering Ord) const override;
Value *emitStoreConditional(IRBuilderBase &Builder, Value *Val, Value *Addr,
AtomicOrdering Ord) const override;
+ Value *emitCanLoadSpeculatively(IRBuilderBase &Builder, Value *Ptr,
+ Value *Size) const override;
void emitAtomicCmpXchgNoStoreLLBalance(IRBuilderBase &Builder) const override;
diff --git a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
new file mode 100644
index 0000000000000..7916f2e4d340f
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
@@ -0,0 +1,78 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -mtriple=aarch64-unknown-linux-gnu -passes=pre-isel-intrinsic-lowering -S < %s | FileCheck %s
+
+; Test that @llvm.can.load.speculatively is lowered to an alignment check
+; for power-of-2 sizes <= 16 bytes on AArch64, and returns false for larger sizes.
+; The 16-byte limit ensures correctness with MTE (memory tagging).
+; Note: non-power-of-2 constant sizes are rejected by the verifier.
+
+define i1 @can_load_speculatively_16(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_16(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
+; Size > 16 - returns false (may cross MTE tag granule boundary)
+define i1 @can_load_speculatively_32(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_32(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 32)
+ ret i1 %can_load
+}
+
+; Size > 16 - returns false (may cross MTE tag granule boundary)
+define i1 @can_load_speculatively_64(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_64(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 64)
+ ret i1 %can_load
+}
+
+; Test with address space
+define i1 @can_load_speculatively_addrspace1(ptr addrspace(1) %ptr) {
+; CHECK-LABEL: @can_load_speculatively_addrspace1(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr addrspace(1) [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p1(ptr addrspace(1) %ptr, i64 16)
+ ret i1 %can_load
+}
+
+; Test size 8 (within limit, power-of-2)
+define i1 @can_load_speculatively_8(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_8(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 7
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 8)
+ ret i1 %can_load
+}
+
+; Test with runtime size - checks size <= 16 and alignment
+define i1 @can_load_speculatively_runtime(ptr %ptr, i64 %size) {
+; CHECK-LABEL: @can_load_speculatively_runtime(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = icmp ule i64 [[SIZE:%.*]], 16
+; CHECK-NEXT: [[TMP3:%.*]] = sub i64 [[SIZE]], 1
+; CHECK-NEXT: [[TMP4:%.*]] = and i64 [[TMP1]], [[TMP3]]
+; CHECK-NEXT: [[TMP5:%.*]] = icmp eq i64 [[TMP4]], 0
+; CHECK-NEXT: [[TMP6:%.*]] = and i1 [[TMP2]], [[TMP5]]
+; CHECK-NEXT: ret i1 [[TMP6]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 %size)
+ ret i1 %can_load
+}
+
+declare i1 @llvm.can.load.speculatively.p0(ptr, i64)
+declare i1 @llvm.can.load.speculatively.p1(ptr addrspace(1), i64)
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
new file mode 100644
index 0000000000000..78a56f3539d11
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
@@ -0,0 +1,66 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-unknown-linux-gnu -mattr=+sve < %s | FileCheck %s
+
+; Test that @llvm.speculative.load with scalable vectors is lowered to a
+; regular load in SelectionDAG.
+
+define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv4i32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr)
+ ret <vscale x 4 x i32> %load
+}
+
+define <vscale x 2 x i64> @speculative_load_nxv2i64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv2i64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr %ptr)
+ ret <vscale x 2 x i64> %load
+}
+
+define <vscale x 8 x i16> @speculative_load_nxv8i16(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv8i16:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr align 8 %ptr)
+ ret <vscale x 8 x i16> %load
+}
+
+define <vscale x 16 x i8> @speculative_load_nxv16i8(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv16i8:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr %ptr)
+ ret <vscale x 16 x i8> %load
+}
+
+define <vscale x 4 x float> @speculative_load_nxv4f32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv4f32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr align 4 %ptr)
+ ret <vscale x 4 x float> %load
+}
+
+define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv2f64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr)
+ ret <vscale x 2 x double> %load
+}
+
+declare <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr)
+declare <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr)
+declare <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr)
+declare <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr)
+declare <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr)
+declare <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr)
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
new file mode 100644
index 0000000000000..8f9a17414ff05
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -0,0 +1,117 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-unknown-linux-gnu < %s | FileCheck %s
+
+; Test that @llvm.speculative.load is lowered to a regular load
+; in SelectionDAG, respecting the alignment attribute.
+
+define <4 x i32> @speculative_load_v4i32_align16(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_align16:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr)
+ ret <4 x i32> %load
+}
+
+define <4 x i32> @speculative_load_v4i32_align4(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_align4:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 4 %ptr)
+ ret <4 x i32> %load
+}
+
+define <4 x i32> @speculative_load_v4i32_noalign(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_noalign:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr)
+ ret <4 x i32> %load
+}
+
+define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v8i32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldp q0, q1, [x0]
+; CHECK-NEXT: ret
+ %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr)
+ ret <8 x i32> %load
+}
+
+define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v2i64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
+ ret <2 x i64> %load
+}
+
+define <4 x float> @speculative_load_v4f32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4f32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
+ ret <4 x float> %load
+}
+
+define <2 x double> @speculative_load_v2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v2f64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr)
+ ret <2 x double> %load
+}
+
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr)
+declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr)
+declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr)
+declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr)
+declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr)
+
+; Scalar type tests
+
+define i32 @speculative_load_i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_i32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr w0, [x0]
+; CHECK-NEXT: ret
+ %load = call i32 @llvm.speculative.load.i32.p0(ptr align 4 %ptr)
+ ret i32 %load
+}
+
+define i64 @speculative_load_i64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_i64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr x0, [x0]
+; CHECK-NEXT: ret
+ %load = call i64 @llvm.speculative.load.i64.p0(ptr %ptr)
+ ret i64 %load
+}
+
+define float @speculative_load_f32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_f32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr s0, [x0]
+; CHECK-NEXT: ret
+ %load = call float @llvm.speculative.load.f32.p0(ptr %ptr)
+ ret float %load
+}
+
+define double @speculative_load_f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_f64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr d0, [x0]
+; CHECK-NEXT: ret
+ %load = call double @llvm.speculative.load.f64.p0(ptr align 8 %ptr)
+ ret double %load
+}
+
+declare i32 @llvm.speculative.load.i32.p0(ptr)
+declare i64 @llvm.speculative.load.i64.p0(ptr)
+declare float @llvm.speculative.load.f32.p0(ptr)
+declare double @llvm.speculative.load.f64.p0(ptr)
diff --git a/llvm/test/CodeGen/X86/can-load-speculatively.ll b/llvm/test/CodeGen/X86/can-load-speculatively.ll
new file mode 100644
index 0000000000000..f51d3847e921d
--- /dev/null
+++ b/llvm/test/CodeGen/X86/can-load-speculatively.ll
@@ -0,0 +1,32 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -mtriple=x86_64-unknown-linux-gnu -passes=pre-isel-intrinsic-lowering -S < %s | FileCheck %s
+
+; Test that @llvm.can.load.speculatively returns false (default) on X86,
+; as X86 does not provide a target-specific expansion.
+
+define i1 @can_load_speculatively_16(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_16(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_32(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_32(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 32)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_8(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_8(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 8)
+ ret i1 %can_load
+}
+
+declare i1 @llvm.can.load.speculatively.p0(ptr, i64)
+
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
new file mode 100644
index 0000000000000..4dd8375e9c42a
--- /dev/null
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -0,0 +1,146 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+sse4.1 < %s | FileCheck %s --check-prefix=SSE
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2 < %s | FileCheck %s --check-prefix=AVX
+
+; Test that @llvm.speculative.load is lowered to a regular load
+; in SelectionDAG.
+
+define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
+; SSE-LABEL: speculative_load_v4i32:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v4i32:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: retq
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr)
+ ret <4 x i32> %load
+}
+
+define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
+; SSE-LABEL: speculative_load_v8i32:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: movaps 16(%rdi), %xmm1
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v8i32:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %ymm0
+; AVX-NEXT: retq
+ %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr)
+ ret <8 x i32> %load
+}
+
+define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
+; SSE-LABEL: speculative_load_v2i64:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v2i64:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: retq
+ %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
+ ret <2 x i64> %load
+}
+
+define <4 x float> @speculative_load_v4f32(ptr %ptr) {
+; SSE-LABEL: speculative_load_v4f32:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v4f32:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: retq
+ %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
+ ret <4 x float> %load
+}
+
+define <2 x double> @speculative_load_v2f64(ptr %ptr) {
+; SSE-LABEL: speculative_load_v2f64:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v2f64:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: retq
+ %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr)
+ ret <2 x double> %load
+}
+
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr)
+declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr)
+declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr)
+declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr)
+declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr)
+
+; Scalar type tests
+
+define i32 @speculative_load_i32(ptr %ptr) {
+; SSE-LABEL: speculative_load_i32:
+; SSE: # %bb.0:
+; SSE-NEXT: movl (%rdi), %eax
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_i32:
+; AVX: # %bb.0:
+; AVX-NEXT: movl (%rdi), %eax
+; AVX-NEXT: retq
+ %load = call i32 @llvm.speculative.load.i32.p0(ptr %ptr)
+ ret i32 %load
+}
+
+define i64 @speculative_load_i64(ptr %ptr) {
+; SSE-LABEL: speculative_load_i64:
+; SSE: # %bb.0:
+; SSE-NEXT: movq (%rdi), %rax
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_i64:
+; AVX: # %bb.0:
+; AVX-NEXT: movq (%rdi), %rax
+; AVX-NEXT: retq
+ %load = call i64 @llvm.speculative.load.i64.p0(ptr align 8 %ptr)
+ ret i64 %load
+}
+
+define float @speculative_load_f32(ptr %ptr) {
+; SSE-LABEL: speculative_load_f32:
+; SSE: # %bb.0:
+; SSE-NEXT: movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_f32:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX-NEXT: retq
+ %load = call float @llvm.speculative.load.f32.p0(ptr align 4 %ptr)
+ ret float %load
+}
+
+define double @speculative_load_f64(ptr %ptr) {
+; SSE-LABEL: speculative_load_f64:
+; SSE: # %bb.0:
+; SSE-NEXT: movsd {{.*#+}} xmm0 = mem[0],zero
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_f64:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX-NEXT: retq
+ %load = call double @llvm.speculative.load.f64.p0(ptr %ptr)
+ ret double %load
+}
+
+declare i32 @llvm.speculative.load.i32.p0(ptr)
+declare i64 @llvm.speculative.load.i64.p0(ptr)
+declare float @llvm.speculative.load.f32.p0(ptr)
+declare double @llvm.speculative.load.f64.p0(ptr)
diff --git a/llvm/test/Verifier/can-load-speculatively.ll b/llvm/test/Verifier/can-load-speculatively.ll
new file mode 100644
index 0000000000000..d2d69f70cfb60
--- /dev/null
+++ b/llvm/test/Verifier/can-load-speculatively.ll
@@ -0,0 +1,19 @@
+; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
+
+declare i1 @llvm.can.load.speculatively.p0(ptr, i64)
+
+; Test that constant size must be a positive power of 2
+
+define i1 @test_size_zero(ptr %ptr) {
+; CHECK: llvm.can.load.speculatively size must be a positive power of 2
+; CHECK-NEXT: %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 0)
+ %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 0)
+ ret i1 %res
+}
+
+define i1 @test_non_power_of_2(ptr %ptr) {
+; CHECK: llvm.can.load.speculatively size must be a positive power of 2
+; CHECK-NEXT: %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 3)
+ %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 3)
+ ret i1 %res
+}
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
new file mode 100644
index 0000000000000..def46d20799c5
--- /dev/null
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -0,0 +1,18 @@
+; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
+
+declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr)
+declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr)
+
+define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
+; CHECK: llvm.speculative.load type must have a power-of-2 size
+; CHECK-NEXT: %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr)
+ %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr)
+ ret <3 x i32> %res
+}
+
+define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
+; CHECK: llvm.speculative.load type must have a power-of-2 size
+; CHECK-NEXT: %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr)
+ %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr)
+ ret <vscale x 3 x i32> %res
+}
>From 49f4bb9f1386a642a8a9f0f9d0ac2d81642bc5d6 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Thu, 5 Feb 2026 19:40:18 +0000
Subject: [PATCH 2/9] !fixup address comments, thanks
---
llvm/docs/LangRef.rst | 16 ++++++++--------
llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp | 3 ---
llvm/lib/Target/AArch64/AArch64ISelLowering.cpp | 13 +++++++------
.../CodeGen/AArch64/can-load-speculatively.ll | 5 +----
.../CodeGen/X86/speculative-load-intrinsic.ll | 8 ++++----
5 files changed, 20 insertions(+), 25 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 981034d5b43eb..5f0704e2fbceb 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28211,10 +28211,10 @@ Overview:
"""""""""
The '``llvm.speculative.load``' intrinsic loads a value from memory. Unlike a
-regular load, the memory access may
-extend beyond the bounds of the allocated object, provided the pointer has been
-verified by :ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to
-ensure the access cannot fault.
+regular load, the memory access may extend beyond the bounds of the allocated
+object, provided the pointer has been verified by
+:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure the
+access cannot fault.
Arguments:
""""""""""
@@ -28232,12 +28232,12 @@ the access cannot fault.
For bytes that are within the bounds of the allocated object, the intrinsic
returns the stored value. For bytes that are beyond the bounds of the
-allocated object, the intrinsic returns ``undef`` for those bytes. At least the
+allocated object, the intrinsic returns ``poison`` for those bytes. At least the
first accessed byte must be within the bounds of an allocated object the pointer is
based on.
The behavior is undefined if this intrinsic is used to load from a pointer
-for which ``llvm.can.load.speculatively`` returns false.
+for which ``llvm.can.load.speculatively`` would return false.
.. _int_can_load_speculatively:
@@ -28273,8 +28273,8 @@ Semantics:
""""""""""
This intrinsic has **target-dependent** semantics. It returns ``true`` if
-loading ``num_bytes`` bytes from ``ptr`` is guaranteed not to trap,
-even if the memory is beyond the bounds of an allocated object. It returns
+``num_bytes`` bytes starting at ``ptr`` can be loaded speculatively, even
+if the memory is beyond the bounds of an allocated object. It returns
``false`` otherwise.
The specific conditions under which this intrinsic returns ``true`` are
diff --git a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
index 06a65e7375f00..63c31cdb81970 100644
--- a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
+++ b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
@@ -141,9 +141,6 @@ static bool lowerLoadRelative(Function &F) {
/// TargetLowering::emitCanLoadSpeculatively.
/// The default expansion returns false (conservative).
static bool lowerCanLoadSpeculatively(Function &F, const TargetMachine *TM) {
- if (F.use_empty())
- return false;
-
bool Changed = false;
for (Use &U : llvm::make_early_inc_range(F.uses())) {
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
index f33f5ab242c04..fcbbf2da5d525 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
@@ -30952,6 +30952,10 @@ Value *AArch64TargetLowering::emitStoreConditional(IRBuilderBase &Builder,
Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
Value *Ptr,
Value *Size) const {
+ unsigned AS = cast<PointerType>(Ptr->getType())->getAddressSpace();
+ // Conservatively only allow speculation for address space 0.
+ if (AS != 0)
+ return nullptr;
// For power-of-2 sizes <= 16, emit alignment check: (ptr & (size - 1)) == 0.
// If the pointer is aligned to at least 'size' bytes, loading 'size' bytes
// cannot cross a page boundary, so it's safe to speculate.
@@ -30960,9 +30964,10 @@ Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
//
// The alignment check only works for power-of-2 sizes. For non-power-of-2
// sizes, we conservatively return false.
- const DataLayout &DL =
- Builder.GetInsertBlock()->getModule()->getDataLayout();
+ const DataLayout &DL = Builder.GetInsertBlock()->getModule()->getDataLayout();
+ unsigned PtrBits = DL.getPointerSizeInBits(AS);
+ Type *IntPtrTy = Builder.getIntNTy(PtrBits);
if (auto *CI = dyn_cast<ConstantInt>(Size)) {
uint64_t SizeVal = CI->getZExtValue();
assert(isPowerOf2_64(SizeVal) && "size must be power-of-two");
@@ -30971,8 +30976,6 @@ Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
return nullptr;
// Power-of-2 constant size <= 16: use fast alignment check.
- unsigned PtrBits = DL.getPointerSizeInBits();
- Type *IntPtrTy = Builder.getIntNTy(PtrBits);
Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
Value *Mask = ConstantInt::get(IntPtrTy, SizeVal - 1);
Value *Masked = Builder.CreateAnd(PtrInt, Mask);
@@ -30980,8 +30983,6 @@ Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
}
// Check power-of-2 size <= 16 and alignment.
- unsigned PtrBits = DL.getPointerSizeInBits();
- Type *IntPtrTy = Builder.getIntNTy(PtrBits);
Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
Value *SizeExt = Builder.CreateZExtOrTrunc(Size, IntPtrTy);
diff --git a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
index 7916f2e4d340f..e9af2eed8918a 100644
--- a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
+++ b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
@@ -38,10 +38,7 @@ define i1 @can_load_speculatively_64(ptr %ptr) {
; Test with address space
define i1 @can_load_speculatively_addrspace1(ptr addrspace(1) %ptr) {
; CHECK-LABEL: @can_load_speculatively_addrspace1(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr addrspace(1) [[PTR:%.*]] to i64
-; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
-; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
-; CHECK-NEXT: ret i1 [[TMP3]]
+; CHECK-NEXT: ret i1 false
;
%can_load = call i1 @llvm.can.load.speculatively.p1(ptr addrspace(1) %ptr, i64 16)
ret i1 %can_load
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index 4dd8375e9c42a..798ee2c03b1ab 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -37,12 +37,12 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
; SSE-LABEL: speculative_load_v2i64:
; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: movups (%rdi), %xmm0
; SSE-NEXT: retq
;
; AVX-LABEL: speculative_load_v2i64:
; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: vmovups (%rdi), %xmm0
; AVX-NEXT: retq
%load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
ret <2 x i64> %load
@@ -51,12 +51,12 @@ define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
define <4 x float> @speculative_load_v4f32(ptr %ptr) {
; SSE-LABEL: speculative_load_v4f32:
; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: movups (%rdi), %xmm0
; SSE-NEXT: retq
;
; AVX-LABEL: speculative_load_v4f32:
; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: vmovups (%rdi), %xmm0
; AVX-NEXT: retq
%load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
ret <4 x float> %load
>From 6dafaf1f741dcb0d2dbe3af9ae5dd1269f7ce2be Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Mon, 9 Feb 2026 18:42:04 +0000
Subject: [PATCH 3/9] !fixup add tests with aligned pointers, clarify deref
guarantees.
---
llvm/docs/LangRef.rst | 9 ++--
.../CodeGen/AArch64/can-load-speculatively.ll | 45 +++++++++++++++++++
2 files changed, 50 insertions(+), 4 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 5f0704e2fbceb..10e437201c945 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28273,13 +28273,14 @@ Semantics:
""""""""""
This intrinsic has **target-dependent** semantics. It returns ``true`` if
-``num_bytes`` bytes starting at ``ptr`` can be loaded speculatively, even
-if the memory is beyond the bounds of an allocated object. It returns
-``false`` otherwise.
+``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for any non-negative
+integer ``I`` where the computed address does not wrap around the address
+space, can be loaded speculatively, even if the memory is beyond the bounds of
+an allocated object. It returns ``false`` otherwise.
The specific conditions under which this intrinsic returns ``true`` are
determined by the target. For example, a target may check whether the pointer
-alignment guarantees the load cannot cross a page boundary.
+alignment guarantees all such loads cannot cross a page boundary.
.. code-block:: llvm
diff --git a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
index e9af2eed8918a..b6679f22b0989 100644
--- a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
+++ b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
@@ -17,6 +17,51 @@ define i1 @can_load_speculatively_16(ptr %ptr) {
ret i1 %can_load
}
+
+define i1 @can_load_speculatively_8_ptr_aligned(ptr align 8 %ptr) {
+; CHECK-LABEL: @can_load_speculatively_8_ptr_aligned(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_16_ptr_aligned(ptr align 16 %ptr) {
+; CHECK-LABEL: @can_load_speculatively_16_ptr_aligned(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_16_ptr_aligned2(ptr align 16 %ptr) {
+; CHECK-LABEL: @can_load_speculatively_16_ptr_aligned2(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr align 16 %ptr, i64 16)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_32_ptr_aligned(ptr align 32 %ptr) {
+; CHECK-LABEL: @can_load_speculatively_32_ptr_aligned(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
; Size > 16 - returns false (may cross MTE tag granule boundary)
define i1 @can_load_speculatively_32(ptr %ptr) {
; CHECK-LABEL: @can_load_speculatively_32(
>From 077d47913f47ad1656aeafe397fd4d1b869f4b4e Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Wed, 18 Feb 2026 09:09:09 +0000
Subject: [PATCH 4/9] !fixup clarify noalias interaction
---
llvm/docs/LangRef.rst | 27 +++++++++++++++++----------
1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 10e437201c945..6e1e9ea0e3956 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28226,15 +28226,22 @@ Semantics:
""""""""""
The '``llvm.speculative.load``' intrinsic performs a load that may access
-memory beyond the allocated object. It must be used in combination with
-:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure
-the access cannot fault.
+memory beyond what is accessible through the pointer. It must be used in
+combination with :ref:`llvm.can.load.speculatively <int_can_load_speculatively>`
+to ensure the access can be performed speculatively.
-For bytes that are within the bounds of the allocated object, the intrinsic
-returns the stored value. For bytes that are beyond the bounds of the
-allocated object, the intrinsic returns ``poison`` for those bytes. At least the
-first accessed byte must be within the bounds of an allocated object the pointer is
-based on.
+A byte at ``ptr + i`` is *accessible through* ``ptr`` if both of the following
+hold:
+
+1. The byte lies within the bounds of an allocated object that ``ptr`` is
+ :ref:`based <pointeraliasing>` on.
+2. Accessing the byte through ``ptr`` does not violate any ``noalias``
+ constraints.
+
+For accessible bytes, the intrinsic returns the stored value. For inaccessible
+bytes, the intrinsic returns ``poison`` and the bytes are not considered accessed
+for the purpose of data races or ``noalias`` constraints. At least the first
+byte must be accessible; otherwise the behavior is undefined.
The behavior is undefined if this intrinsic is used to load from a pointer
for which ``llvm.can.load.speculatively`` would return false.
@@ -28273,8 +28280,8 @@ Semantics:
""""""""""
This intrinsic has **target-dependent** semantics. It returns ``true`` if
-``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for any non-negative
-integer ``I`` where the computed address does not wrap around the address
+``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for all non-negative
+integers ``I`` where the computed address does not wrap around the address
space, can be loaded speculatively, even if the memory is beyond the bounds of
an allocated object. It returns ``false`` otherwise.
>From 203014f50f395d0e7cc6057527696279fa8ca486 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Mon, 23 Feb 2026 09:43:35 +0000
Subject: [PATCH 5/9] !fixup limit to executions that do not depend on
may-accessible bytes.
---
llvm/docs/LangRef.rst | 3 +++
1 file changed, 3 insertions(+)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 6e1e9ea0e3956..a1634dd09eeab 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28243,6 +28243,9 @@ bytes, the intrinsic returns ``poison`` and the bytes are not considered accesse
for the purpose of data races or ``noalias`` constraints. At least the first
byte must be accessible; otherwise the behavior is undefined.
+The behavior is undefined if program execution depends on any byte in the
+result that may not be accessible.
+
The behavior is undefined if this intrinsic is used to load from a pointer
for which ``llvm.can.load.speculatively`` would return false.
>From c4e555d60169ac191960c637bfd3e5388d3160a0 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Wed, 25 Feb 2026 17:23:24 +0000
Subject: [PATCH 6/9] !fixup add implicit freeze
---
llvm/docs/LangRef.rst | 11 ++++-------
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index a1634dd09eeab..6f853ef8defc9 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28238,13 +28238,10 @@ hold:
2. Accessing the byte through ``ptr`` does not violate any ``noalias``
constraints.
-For accessible bytes, the intrinsic returns the stored value. For inaccessible
-bytes, the intrinsic returns ``poison`` and the bytes are not considered accessed
-for the purpose of data races or ``noalias`` constraints. At least the first
-byte must be accessible; otherwise the behavior is undefined.
-
-The behavior is undefined if program execution depends on any byte in the
-result that may not be accessible.
+For each byte of the result, if the corresponding byte is accessible through
+``ptr``, the result contains the value stored in memory at that byte. Otherwise,
+the corresponding byte of the result is ``freeze poison`` and the byte is not
+considered accessed for the purpose of data races or ``noalias`` constraints.
The behavior is undefined if this intrinsic is used to load from a pointer
for which ``llvm.can.load.speculatively`` would return false.
>From d574368719c0c3634b37820812f893c5ce82f2ce Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Fri, 27 Feb 2026 13:52:11 +0000
Subject: [PATCH 7/9] !fixup add read bytes argument
---
llvm/docs/LangRef.rst | 35 ++++++++--------
llvm/include/llvm/IR/Intrinsics.td | 7 ++--
.../SelectionDAG/SelectionDAGBuilder.cpp | 2 +
.../AArch64/speculative-load-intrinsic-sve.ll | 24 +++++------
.../AArch64/speculative-load-intrinsic.ll | 40 +++++++++----------
.../CodeGen/X86/speculative-load-intrinsic.ll | 36 ++++++++---------
llvm/test/Verifier/speculative-load.ll | 12 +++---
7 files changed, 78 insertions(+), 78 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 6f853ef8defc9..0c682d9cc90a0 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28203,9 +28203,9 @@ This is an overloaded intrinsic.
::
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>)
- declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>)
- declare i64 @llvm.speculative.load.i64.p0(ptr <ptr>)
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i64 <num_read_bytes>)
+ declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i64 <num_read_bytes>)
+ declare i64 @llvm.speculative.load.i64.p0(ptr <ptr>, i64 <num_read_bytes>)
Overview:
"""""""""
@@ -28219,32 +28219,29 @@ access cannot fault.
Arguments:
""""""""""
-The argument is a pointer to the memory location to load from. The return type
-must have a power-of-2 size in bytes.
+The first argument is a pointer to the memory location to load from. The second
+argument is an ``i64`` specifying the number of bytes starting from ``ptr``
+that the intrinsic will read. The return type must have a power-of-2 size in
+bytes.
Semantics:
""""""""""
The '``llvm.speculative.load``' intrinsic performs a load that may access
-memory beyond what is accessible through the pointer. It must be used in
+memory beyond what is readable through the pointer. It must be used in
combination with :ref:`llvm.can.load.speculatively <int_can_load_speculatively>`
to ensure the access can be performed speculatively.
-A byte at ``ptr + i`` is *accessible through* ``ptr`` if both of the following
-hold:
+The intrinsic reads ``num_read_bytes`` bytes starting from ``ptr`` and returns
+the stored values for those bytes. The read bytes must lie within the bounds
+of an allocated object that ``ptr`` is :ref:`based <pointeraliasing>` on.
-1. The byte lies within the bounds of an allocated object that ``ptr`` is
- :ref:`based <pointeraliasing>` on.
-2. Accessing the byte through ``ptr`` does not violate any ``noalias``
- constraints.
-
-For each byte of the result, if the corresponding byte is accessible through
-``ptr``, the result contains the value stored in memory at that byte. Otherwise,
-the corresponding byte of the result is ``freeze poison`` and the byte is not
-considered accessed for the purpose of data races or ``noalias`` constraints.
+Bytes at offsets ``>= num_read_bytes`` (up to the size of the return type)
+are ``poison`` and are not considered accessed for the purpose of data races
+or ``noalias`` constraints.
The behavior is undefined if this intrinsic is used to load from a pointer
-for which ``llvm.can.load.speculatively`` would return false.
+for which ``llvm.can.load.speculatively`` returns false.
.. _int_can_load_speculatively:
@@ -28297,7 +28294,7 @@ alignment guarantees all such loads cannot cross a page boundary.
speculative_path:
; Safe to speculatively load from %ptr
- %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr)
+ %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 16)
...
safe_path:
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 49cc434809360..37a2414c441f6 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2653,11 +2653,12 @@ def int_masked_srem:
[IntrNoMem]>;
// Speculatively load a value from memory; lowers to a regular aligned load.
-// The loaded type must have a power-of-2 size.
+// The loaded type must have a power-of-2 size. The second argument specifies
+// the number of bytes the intrinsic will read from memory.
def int_speculative_load:
DefaultAttrsIntrinsic<[llvm_any_ty],
- [llvm_anyptr_ty],
- [IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
+ [llvm_anyptr_ty, llvm_i64_ty],
+ [IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
// The size can be a runtime value to support scalable vectors.
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 91ddc8d53357e..b6db6bd85971e 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5148,6 +5148,8 @@ void SelectionDAGBuilder::visitMaskedLoad(const CallInst &I, bool IsExpanding) {
void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
SDLoc sdl = getCurSDLoc();
Value *PtrOperand = I.getArgOperand(0);
+ // The second argument (num_read_bytes) is IR-level
+ // semantics only; it is not needed at codegen.
SDValue Ptr = getValue(PtrOperand);
const TargetLowering &TLI = DAG.getTargetLoweringInfo();
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
index 78a56f3539d11..4ae56cbca7ae9 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
@@ -9,7 +9,7 @@ define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr)
+ %load = call <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i64 0)
ret <vscale x 4 x i32> %load
}
@@ -18,7 +18,7 @@ define <vscale x 2 x i64> @speculative_load_nxv2i64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr %ptr)
+ %load = call <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr %ptr, i64 0)
ret <vscale x 2 x i64> %load
}
@@ -27,7 +27,7 @@ define <vscale x 8 x i16> @speculative_load_nxv8i16(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr align 8 %ptr)
+ %load = call <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr align 8 %ptr, i64 0)
ret <vscale x 8 x i16> %load
}
@@ -36,7 +36,7 @@ define <vscale x 16 x i8> @speculative_load_nxv16i8(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr %ptr)
+ %load = call <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr %ptr, i64 0)
ret <vscale x 16 x i8> %load
}
@@ -45,7 +45,7 @@ define <vscale x 4 x float> @speculative_load_nxv4f32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr align 4 %ptr)
+ %load = call <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr align 4 %ptr, i64 0)
ret <vscale x 4 x float> %load
}
@@ -54,13 +54,13 @@ define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr)
+ %load = call <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i64 0)
ret <vscale x 2 x double> %load
}
-declare <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr)
-declare <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr)
-declare <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr)
-declare <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr)
-declare <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr)
-declare <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr)
+declare <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr, i64)
+declare <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr, i64)
+declare <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr, i64)
+declare <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr, i64)
+declare <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr, i64)
+declare <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr, i64)
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
index 8f9a17414ff05..1c6b9d267ad46 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -9,7 +9,7 @@ define <4 x i32> @speculative_load_v4i32_align16(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr)
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 0)
ret <4 x i32> %load
}
@@ -18,7 +18,7 @@ define <4 x i32> @speculative_load_v4i32_align4(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 4 %ptr)
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 4 %ptr, i64 0)
ret <4 x i32> %load
}
@@ -27,7 +27,7 @@ define <4 x i32> @speculative_load_v4i32_noalign(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr)
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 0)
ret <4 x i32> %load
}
@@ -36,7 +36,7 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldp q0, q1, [x0]
; CHECK-NEXT: ret
- %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr)
+ %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 0)
ret <8 x i32> %load
}
@@ -45,7 +45,7 @@ define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
+ %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr, i64 0)
ret <2 x i64> %load
}
@@ -54,7 +54,7 @@ define <4 x float> @speculative_load_v4f32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
+ %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr, i64 0)
ret <4 x float> %load
}
@@ -63,15 +63,15 @@ define <2 x double> @speculative_load_v2f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr)
+ %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 0)
ret <2 x double> %load
}
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr)
-declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr)
-declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr)
-declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr)
-declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr)
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i64)
+declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr, i64)
+declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr, i64)
+declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr, i64)
+declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr, i64)
; Scalar type tests
@@ -80,7 +80,7 @@ define i32 @speculative_load_i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr w0, [x0]
; CHECK-NEXT: ret
- %load = call i32 @llvm.speculative.load.i32.p0(ptr align 4 %ptr)
+ %load = call i32 @llvm.speculative.load.i32.p0(ptr align 4 %ptr, i64 0)
ret i32 %load
}
@@ -89,7 +89,7 @@ define i64 @speculative_load_i64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr x0, [x0]
; CHECK-NEXT: ret
- %load = call i64 @llvm.speculative.load.i64.p0(ptr %ptr)
+ %load = call i64 @llvm.speculative.load.i64.p0(ptr %ptr, i64 0)
ret i64 %load
}
@@ -98,7 +98,7 @@ define float @speculative_load_f32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr s0, [x0]
; CHECK-NEXT: ret
- %load = call float @llvm.speculative.load.f32.p0(ptr %ptr)
+ %load = call float @llvm.speculative.load.f32.p0(ptr %ptr, i64 0)
ret float %load
}
@@ -107,11 +107,11 @@ define double @speculative_load_f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr d0, [x0]
; CHECK-NEXT: ret
- %load = call double @llvm.speculative.load.f64.p0(ptr align 8 %ptr)
+ %load = call double @llvm.speculative.load.f64.p0(ptr align 8 %ptr, i64 0)
ret double %load
}
-declare i32 @llvm.speculative.load.i32.p0(ptr)
-declare i64 @llvm.speculative.load.i64.p0(ptr)
-declare float @llvm.speculative.load.f32.p0(ptr)
-declare double @llvm.speculative.load.f64.p0(ptr)
+declare i32 @llvm.speculative.load.i32.p0(ptr, i64)
+declare i64 @llvm.speculative.load.i64.p0(ptr, i64)
+declare float @llvm.speculative.load.f32.p0(ptr, i64)
+declare double @llvm.speculative.load.f64.p0(ptr, i64)
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index 798ee2c03b1ab..5e12593f2d5a0 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -15,7 +15,7 @@ define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovaps (%rdi), %xmm0
; AVX-NEXT: retq
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr)
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 0)
ret <4 x i32> %load
}
@@ -30,7 +30,7 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovaps (%rdi), %ymm0
; AVX-NEXT: retq
- %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr)
+ %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 0)
ret <8 x i32> %load
}
@@ -44,7 +44,7 @@ define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovups (%rdi), %xmm0
; AVX-NEXT: retq
- %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
+ %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr, i64 0)
ret <2 x i64> %load
}
@@ -58,7 +58,7 @@ define <4 x float> @speculative_load_v4f32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovups (%rdi), %xmm0
; AVX-NEXT: retq
- %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
+ %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr, i64 0)
ret <4 x float> %load
}
@@ -72,15 +72,15 @@ define <2 x double> @speculative_load_v2f64(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovaps (%rdi), %xmm0
; AVX-NEXT: retq
- %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr)
+ %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 0)
ret <2 x double> %load
}
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr)
-declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr)
-declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr)
-declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr)
-declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr)
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i64)
+declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr, i64)
+declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr, i64)
+declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr, i64)
+declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr, i64)
; Scalar type tests
@@ -94,7 +94,7 @@ define i32 @speculative_load_i32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: movl (%rdi), %eax
; AVX-NEXT: retq
- %load = call i32 @llvm.speculative.load.i32.p0(ptr %ptr)
+ %load = call i32 @llvm.speculative.load.i32.p0(ptr %ptr, i64 0)
ret i32 %load
}
@@ -108,7 +108,7 @@ define i64 @speculative_load_i64(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: movq (%rdi), %rax
; AVX-NEXT: retq
- %load = call i64 @llvm.speculative.load.i64.p0(ptr align 8 %ptr)
+ %load = call i64 @llvm.speculative.load.i64.p0(ptr align 8 %ptr, i64 0)
ret i64 %load
}
@@ -122,7 +122,7 @@ define float @speculative_load_f32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
; AVX-NEXT: retq
- %load = call float @llvm.speculative.load.f32.p0(ptr align 4 %ptr)
+ %load = call float @llvm.speculative.load.f32.p0(ptr align 4 %ptr, i64 0)
ret float %load
}
@@ -136,11 +136,11 @@ define double @speculative_load_f64(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovsd {{.*#+}} xmm0 = mem[0],zero
; AVX-NEXT: retq
- %load = call double @llvm.speculative.load.f64.p0(ptr %ptr)
+ %load = call double @llvm.speculative.load.f64.p0(ptr %ptr, i64 0)
ret double %load
}
-declare i32 @llvm.speculative.load.i32.p0(ptr)
-declare i64 @llvm.speculative.load.i64.p0(ptr)
-declare float @llvm.speculative.load.f32.p0(ptr)
-declare double @llvm.speculative.load.f64.p0(ptr)
+declare i32 @llvm.speculative.load.i32.p0(ptr, i64)
+declare i64 @llvm.speculative.load.i64.p0(ptr, i64)
+declare float @llvm.speculative.load.f32.p0(ptr, i64)
+declare double @llvm.speculative.load.f64.p0(ptr, i64)
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
index def46d20799c5..c06881a5f50af 100644
--- a/llvm/test/Verifier/speculative-load.ll
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -1,18 +1,18 @@
; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
-declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr)
-declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr)
+declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, i64)
+declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, i64)
define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr)
- %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr)
+; CHECK-NEXT: %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
+ %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
ret <3 x i32> %res
}
define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr)
- %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr)
+; CHECK-NEXT: %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
+ %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
ret <vscale x 3 x i32> %res
}
>From 5a485f57910197a93b0002f791e03f6218b8b282 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Fri, 20 Mar 2026 11:58:45 +0000
Subject: [PATCH 8/9] !fixup make oracle function part of the arguments
---
llvm/docs/LangRef.rst | 48 +++--
llvm/include/llvm/IR/Intrinsics.td | 6 +-
.../SelectionDAG/SelectionDAGBuilder.cpp | 4 +-
llvm/lib/IR/Verifier.cpp | 44 ++++
.../AArch64/speculative-load-intrinsic-sve.ll | 66 ------
.../AArch64/speculative-load-intrinsic.ll | 123 ++++++------
.../CodeGen/X86/speculative-load-intrinsic.ll | 190 +++++++-----------
llvm/test/Verifier/speculative-load.ll | 74 ++++++-
8 files changed, 272 insertions(+), 283 deletions(-)
delete mode 100644 llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 0c682d9cc90a0..ccf0408d8b809 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28190,7 +28190,7 @@ Speculative Load Intrinsics
LLVM provides intrinsics for speculatively loading memory that may be
out-of-bounds. These intrinsics enable optimizations like early-exit loop
vectorization where the vectorized loop may read beyond the end of an array,
-provided the access is guaranteed to not trap by target-specific checks.
+provided the access is guaranteed be valid by target-specific checks.
.. _int_speculative_load:
@@ -28203,9 +28203,12 @@ This is an overloaded intrinsic.
::
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i64 <num_read_bytes>)
- declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i64 <num_read_bytes>)
- declare i64 @llvm.speculative.load.i64.p0(ptr <ptr>, i64 <num_read_bytes>)
+ ; Direct form: number of accessible bytes given as i64
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i64 <num_accessible_bytes>)
+ declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i64 <num_accessible_bytes>)
+
+ ; Oracle form: accessible bytes computed by calling oracle_fn(args...)
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, ptr <oracle_fn>, ...)
Overview:
"""""""""
@@ -28214,34 +28217,35 @@ The '``llvm.speculative.load``' intrinsic loads a value from memory. Unlike a
regular load, the memory access may extend beyond the bounds of the allocated
object, provided the pointer has been verified by
:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure the
-access cannot fault.
+access is valid.
Arguments:
""""""""""
-The first argument is a pointer to the memory location to load from. The second
-argument is an ``i64`` specifying the number of bytes starting from ``ptr``
-that the intrinsic will read. The return type must have a power-of-2 size in
-bytes.
+The first argument is a pointer to the memory location to load from. The return
+type must be a vector type with a power-of-2 size in bytes. The remaining
+arguments determine the *number of accessible bytes* starting from ``ptr``,
+denoted ``N`` below.
+
+In the **direct form**, the second argument is an ``i64`` specifying ``N``
+directly. In the **oracle form**, the second argument must be a direct
+reference to a function returning ``i64`` that may only read memory through its
+arguments (indirect function pointers are not permitted); the remaining
+arguments are forwarded to it, and the return value is ``N``.
Semantics:
""""""""""
-The '``llvm.speculative.load``' intrinsic performs a load that may access
-memory beyond what is readable through the pointer. It must be used in
-combination with :ref:`llvm.can.load.speculatively <int_can_load_speculatively>`
-to ensure the access can be performed speculatively.
-
-The intrinsic reads ``num_read_bytes`` bytes starting from ``ptr`` and returns
-the stored values for those bytes. The read bytes must lie within the bounds
-of an allocated object that ``ptr`` is :ref:`based <pointeraliasing>` on.
+The intrinsic reads ``N`` bytes starting from ``ptr`` and returns the stored
+values for those bytes. These bytes must lie within the bounds of an allocated
+object that ``ptr`` is :ref:`based <pointeraliasing>` on.
-Bytes at offsets ``>= num_read_bytes`` (up to the size of the return type)
-are ``poison`` and are not considered accessed for the purpose of data races
-or ``noalias`` constraints.
+Bytes at offsets in ``[N, size of the return type)`` are ``poison`` and are not
+considered accessed for the purposes of data races or ``noalias`` constraints.
-The behavior is undefined if this intrinsic is used to load from a pointer
-for which ``llvm.can.load.speculatively`` returns false.
+The behavior is undefined if the speculative load accesses memory that would
+fault (i.e., the oracle or ``llvm.can.load.speculatively`` would indicate the
+access is not safe).
.. _int_can_load_speculatively:
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 37a2414c441f6..27c95134808cd 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2653,11 +2653,9 @@ def int_masked_srem:
[IntrNoMem]>;
// Speculatively load a value from memory; lowers to a regular aligned load.
-// The loaded type must have a power-of-2 size. The second argument specifies
-// the number of bytes the intrinsic will read from memory.
def int_speculative_load:
- DefaultAttrsIntrinsic<[llvm_any_ty],
- [llvm_anyptr_ty, llvm_i64_ty],
+ DefaultAttrsIntrinsic<[llvm_anyvector_ty],
+ [llvm_anyptr_ty, llvm_vararg_ty],
[IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index b6db6bd85971e..0e360b97eeddd 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5148,8 +5148,8 @@ void SelectionDAGBuilder::visitMaskedLoad(const CallInst &I, bool IsExpanding) {
void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
SDLoc sdl = getCurSDLoc();
Value *PtrOperand = I.getArgOperand(0);
- // The second argument (num_read_bytes) is IR-level
- // semantics only; it is not needed at codegen.
+ // The remaining arguments (num_accessible_bytes or oracle function + args)
+ // are IR-level semantics only; they are not needed at codegen.
SDValue Ptr = getValue(PtrOperand);
const TargetLowering &TLI = DAG.getTargetLoweringInfo();
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 1c24473f91481..3dc5c32e647ae 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -6860,6 +6860,50 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
// For scalable vectors, check the known minimum size is a power of 2.
Check(Size.getKnownMinValue() > 0 && isPowerOf2_64(Size.getKnownMinValue()),
"llvm.speculative.load type must have a power-of-2 size", &Call);
+
+ unsigned NumArgs = Call.arg_size();
+ Check(NumArgs >= 2, "llvm.speculative.load requires at least 2 arguments",
+ &Call);
+
+ Value *SecondArg = Call.getArgOperand(1);
+ if (SecondArg->getType()->isIntegerTy(64)) {
+ // Direct form: (ptr, i64 num_accessible_bytes)
+ Check(NumArgs == 2,
+ "llvm.speculative.load direct form requires exactly 2 arguments",
+ &Call);
+ if (auto *CI = dyn_cast<ConstantInt>(SecondArg)) {
+ Check(Size.isScalable() || CI->getZExtValue() <= Size.getFixedValue(),
+ "llvm.speculative.load num_accessible_bytes must not exceed "
+ "the result size in bytes",
+ &Call);
+ }
+ } else {
+ // Oracle form: (ptr, oracle_fn_ptr, args...)
+ auto *OracleFn = dyn_cast<Function>(SecondArg);
+ Check(OracleFn,
+ "llvm.speculative.load second argument must be i64 or a direct "
+ "reference to an oracle function",
+ &Call);
+
+ Check(OracleFn->onlyReadsMemory() && OracleFn->onlyAccessesArgMemory(),
+ "llvm.speculative.load oracle function must not have side effects "
+ "and may only read memory through its arguments",
+ &Call);
+
+ FunctionType *FTy = OracleFn->getFunctionType();
+ Check(FTy->getReturnType()->isIntegerTy(64),
+ "llvm.speculative.load oracle function must return i64", &Call);
+ unsigned NumOracleArgs = NumArgs - 2;
+ Check(FTy->isVarArg() ? NumOracleArgs >= FTy->getNumParams()
+ : NumOracleArgs == FTy->getNumParams(),
+ "llvm.speculative.load oracle function argument count mismatch",
+ &Call);
+ for (unsigned I = 0, E = FTy->getNumParams(); I < E; ++I) {
+ Check(FTy->getParamType(I) == Call.getArgOperand(I + 2)->getType(),
+ "llvm.speculative.load oracle function argument type mismatch",
+ &Call);
+ }
+ }
break;
}
case Intrinsic::can_load_speculatively: {
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
deleted file mode 100644
index 4ae56cbca7ae9..0000000000000
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
+++ /dev/null
@@ -1,66 +0,0 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64-unknown-linux-gnu -mattr=+sve < %s | FileCheck %s
-
-; Test that @llvm.speculative.load with scalable vectors is lowered to a
-; regular load in SelectionDAG.
-
-define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv4i32:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i64 0)
- ret <vscale x 4 x i32> %load
-}
-
-define <vscale x 2 x i64> @speculative_load_nxv2i64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv2i64:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr %ptr, i64 0)
- ret <vscale x 2 x i64> %load
-}
-
-define <vscale x 8 x i16> @speculative_load_nxv8i16(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv8i16:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr align 8 %ptr, i64 0)
- ret <vscale x 8 x i16> %load
-}
-
-define <vscale x 16 x i8> @speculative_load_nxv16i8(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv16i8:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr %ptr, i64 0)
- ret <vscale x 16 x i8> %load
-}
-
-define <vscale x 4 x float> @speculative_load_nxv4f32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv4f32:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr align 4 %ptr, i64 0)
- ret <vscale x 4 x float> %load
-}
-
-define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv2f64:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i64 0)
- ret <vscale x 2 x double> %load
-}
-
-declare <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr, i64)
-declare <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr, i64)
-declare <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr, i64)
-declare <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr, i64)
-declare <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr, i64)
-declare <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr, i64)
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
index 1c6b9d267ad46..2746b9ea87daa 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -1,117 +1,112 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64-unknown-linux-gnu < %s | FileCheck %s
+; RUN: llc -mtriple=aarch64-unknown-linux-gnu -mattr=+sve < %s | FileCheck %s
; Test that @llvm.speculative.load is lowered to a regular load
-; in SelectionDAG, respecting the alignment attribute.
+; in SelectionDAG for fixed vectors, scalable vectors, and scalars.
-define <4 x i32> @speculative_load_v4i32_align16(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v4i32_align16:
+; Fixed-width vector tests
+
+define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 0)
+ %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 12)
ret <4 x i32> %load
}
-define <4 x i32> @speculative_load_v4i32_align4(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v4i32_align4:
+define <2 x double> @speculative_load_v2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v2f64:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 4 %ptr, i64 0)
- ret <4 x i32> %load
+ %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 8)
+ ret <2 x double> %load
}
-define <4 x i32> @speculative_load_v4i32_noalign(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v4i32_noalign:
+; Scalable vector tests
+
+define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv4i32:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 0)
- ret <4 x i32> %load
+ %load = call <vscale x 4 x i32> (ptr, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i64 8)
+ ret <vscale x 4 x i32> %load
}
-define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v8i32:
+define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv2f64:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldp q0, q1, [x0]
+; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 0)
- ret <8 x i32> %load
+ %load = call <vscale x 2 x double> (ptr, ...) @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i64 4)
+ ret <vscale x 2 x double> %load
}
-define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v2i64:
+; Oracle form tests
+
+declare i64 @oracle(ptr, i64) memory(argmem: read)
+
+define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr, i64 0)
- ret <2 x i64> %load
+ %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
}
-define <4 x float> @speculative_load_v4f32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v4f32:
+define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v2i64_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr, i64 0)
- ret <4 x float> %load
+ %load = call <2 x i64> (ptr, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <2 x i64> %load
}
-define <2 x double> @speculative_load_v2f64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v2f64:
+define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i16_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 0)
- ret <2 x double> %load
+ %load = call <8 x i16> (ptr, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i16> %load
}
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i64)
-declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr, i64)
-declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr, i64)
-declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr, i64)
-declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr, i64)
-
-; Scalar type tests
-
-define i32 @speculative_load_i32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_i32:
+define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v16i8_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr w0, [x0]
+; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call i32 @llvm.speculative.load.i32.p0(ptr align 4 %ptr, i64 0)
- ret i32 %load
+ %load = call <16 x i8> (ptr, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <16 x i8> %load
}
-define i64 @speculative_load_i64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_i64:
+define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4f32_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr x0, [x0]
+; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call i64 @llvm.speculative.load.i64.p0(ptr %ptr, i64 0)
- ret i64 %load
+ %load = call <4 x float> (ptr, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x float> %load
}
-define float @speculative_load_f32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_f32:
+define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v2f64_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr s0, [x0]
+; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call float @llvm.speculative.load.f32.p0(ptr %ptr, i64 0)
- ret float %load
+ %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <2 x double> %load
}
-define double @speculative_load_f64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_f64:
+define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_nxv4i32_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr d0, [x0]
+; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call double @llvm.speculative.load.f64.p0(ptr align 8 %ptr, i64 0)
- ret double %load
+ %load = call <vscale x 4 x i32> (ptr, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <vscale x 4 x i32> %load
}
-
-declare i32 @llvm.speculative.load.i32.p0(ptr, i64)
-declare i64 @llvm.speculative.load.i64.p0(ptr, i64)
-declare float @llvm.speculative.load.f32.p0(ptr, i64)
-declare double @llvm.speculative.load.f64.p0(ptr, i64)
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index 5e12593f2d5a0..e387a17990012 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -1,146 +1,98 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+sse4.1 < %s | FileCheck %s --check-prefix=SSE
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2 < %s | FileCheck %s --check-prefix=AVX
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2 < %s | FileCheck %s
; Test that @llvm.speculative.load is lowered to a regular load
; in SelectionDAG.
define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
-; SSE-LABEL: speculative_load_v4i32:
-; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v4i32:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %xmm0
-; AVX-NEXT: retq
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 0)
+; CHECK-LABEL: speculative_load_v4i32:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 8)
ret <4 x i32> %load
}
define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
-; SSE-LABEL: speculative_load_v8i32:
-; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
-; SSE-NEXT: movaps 16(%rdi), %xmm1
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v8i32:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %ymm0
-; AVX-NEXT: retq
- %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 0)
+; CHECK-LABEL: speculative_load_v8i32:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <8 x i32> (ptr, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 24)
ret <8 x i32> %load
}
-define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
-; SSE-LABEL: speculative_load_v2i64:
-; SSE: # %bb.0:
-; SSE-NEXT: movups (%rdi), %xmm0
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v2i64:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovups (%rdi), %xmm0
-; AVX-NEXT: retq
- %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr, i64 0)
- ret <2 x i64> %load
-}
+; Oracle form tests
+declare i64 @oracle(ptr, i64) memory(argmem: read)
-define <4 x float> @speculative_load_v4f32(ptr %ptr) {
-; SSE-LABEL: speculative_load_v4f32:
-; SSE: # %bb.0:
-; SSE-NEXT: movups (%rdi), %xmm0
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v4f32:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovups (%rdi), %xmm0
-; AVX-NEXT: retq
- %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr, i64 0)
- ret <4 x float> %load
+define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
}
-define <2 x double> @speculative_load_v2f64(ptr %ptr) {
-; SSE-LABEL: speculative_load_v2f64:
-; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v2f64:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %xmm0
-; AVX-NEXT: retq
- %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 0)
- ret <2 x double> %load
+define <8 x i32> @speculative_load_v8i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i32_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <8 x i32> (ptr, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i32> %load
}
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i64)
-declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr, i64)
-declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr, i64)
-declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr, i64)
-declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr, i64)
-
-; Scalar type tests
+define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v2i64_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <2 x i64> (ptr, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <2 x i64> %load
+}
-define i32 @speculative_load_i32(ptr %ptr) {
-; SSE-LABEL: speculative_load_i32:
-; SSE: # %bb.0:
-; SSE-NEXT: movl (%rdi), %eax
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_i32:
-; AVX: # %bb.0:
-; AVX-NEXT: movl (%rdi), %eax
-; AVX-NEXT: retq
- %load = call i32 @llvm.speculative.load.i32.p0(ptr %ptr, i64 0)
- ret i32 %load
+define <4 x i64> @speculative_load_v4i64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i64_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <4 x i64> (ptr, ...) @llvm.speculative.load.v4i64.p0(ptr align 32 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i64> %load
}
-define i64 @speculative_load_i64(ptr %ptr) {
-; SSE-LABEL: speculative_load_i64:
-; SSE: # %bb.0:
-; SSE-NEXT: movq (%rdi), %rax
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_i64:
-; AVX: # %bb.0:
-; AVX-NEXT: movq (%rdi), %rax
-; AVX-NEXT: retq
- %load = call i64 @llvm.speculative.load.i64.p0(ptr align 8 %ptr, i64 0)
- ret i64 %load
+define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4f32_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x float> (ptr, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x float> %load
}
-define float @speculative_load_f32(ptr %ptr) {
-; SSE-LABEL: speculative_load_f32:
-; SSE: # %bb.0:
-; SSE-NEXT: movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_f32:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; AVX-NEXT: retq
- %load = call float @llvm.speculative.load.f32.p0(ptr align 4 %ptr, i64 0)
- ret float %load
+define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v2f64_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <2 x double> %load
}
-define double @speculative_load_f64(ptr %ptr) {
-; SSE-LABEL: speculative_load_f64:
-; SSE: # %bb.0:
-; SSE-NEXT: movsd {{.*#+}} xmm0 = mem[0],zero
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_f64:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovsd {{.*#+}} xmm0 = mem[0],zero
-; AVX-NEXT: retq
- %load = call double @llvm.speculative.load.f64.p0(ptr %ptr, i64 0)
- ret double %load
+define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i16_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <8 x i16> (ptr, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i16> %load
}
-declare i32 @llvm.speculative.load.i32.p0(ptr, i64)
-declare i64 @llvm.speculative.load.i64.p0(ptr, i64)
-declare float @llvm.speculative.load.f32.p0(ptr, i64)
-declare double @llvm.speculative.load.f64.p0(ptr, i64)
+define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v16i8_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <16 x i8> (ptr, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <16 x i8> %load
+}
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
index c06881a5f50af..34629849025dc 100644
--- a/llvm/test/Verifier/speculative-load.ll
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -1,18 +1,80 @@
; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
-declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, i64)
-declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, i64)
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, ...)
+declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, ...)
+declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, ...)
+
+declare i32 @bad_oracle_ret(ptr, i64) memory(argmem: read)
+declare i64 @good_oracle(ptr, i64) memory(argmem: read)
+declare i64 @oracle_i32_param(i32) memory(argmem: read)
+declare i64 @side_effecting_oracle(ptr, i64)
define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
- %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
+; CHECK-NEXT: %res = call <3 x i32> (ptr, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
+ %res = call <3 x i32> (ptr, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
ret <3 x i32> %res
}
define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
- %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
+; CHECK-NEXT: %res = call <vscale x 3 x i32> (ptr, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
+ %res = call <vscale x 3 x i32> (ptr, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
ret <vscale x 3 x i32> %res
}
+
+define <4 x i32> @test_oracle_wrong_return_type(ptr %ptr, i64 %n) {
+; CHECK: llvm.speculative.load oracle function must return i64
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_oracle_too_few_args(ptr %ptr) {
+; CHECK: llvm.speculative.load oracle function argument count mismatch
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_oracle_too_many_args(ptr %ptr, i64 %n) {
+; CHECK: llvm.speculative.load oracle function argument count mismatch
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_oracle_arg_type_mismatch(ptr %ptr) {
+; CHECK: llvm.speculative.load oracle function argument type mismatch
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @oracle_i32_param, i64 42)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @oracle_i32_param, i64 42)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_non_function_oracle(ptr %ptr, ptr %not_fn) {
+; CHECK: llvm.speculative.load second argument must be i64 or a direct reference to an oracle function
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr %not_fn)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr %not_fn)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_direct_form_extra_args(ptr %ptr) {
+; CHECK: llvm.speculative.load direct form requires exactly 2 arguments
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 8, i64 4)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 8, i64 4)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_oracle_side_effects(ptr %ptr, i64 %n) {
+; CHECK: llvm.speculative.load oracle function must not have side effects and may only read memory through its arguments
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_num_accessible_bytes_exceeds_size(ptr %ptr) {
+; CHECK: llvm.speculative.load num_accessible_bytes must not exceed the result size in bytes
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 32)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 32)
+ ret <4 x i32> %res
+}
>From 425fd745af1266327e92b0aa589e87e45e260bc2 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Mon, 30 Mar 2026 16:30:28 +0100
Subject: [PATCH 9/9] !fixup add from_end argument
---
llvm/docs/LangRef.rst | 41 +++++++-----
llvm/include/llvm/IR/Intrinsics.td | 5 +-
.../SelectionDAG/SelectionDAGBuilder.cpp | 2 +-
llvm/lib/IR/Verifier.cpp | 28 ++++----
.../AArch64/speculative-load-intrinsic.ll | 60 +++++++++++++----
.../CodeGen/X86/speculative-load-intrinsic.ll | 58 ++++++++++++++---
llvm/test/Verifier/speculative-load.ll | 65 ++++++++++---------
7 files changed, 178 insertions(+), 81 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index ccf0408d8b809..a01a2a160a536 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28190,7 +28190,7 @@ Speculative Load Intrinsics
LLVM provides intrinsics for speculatively loading memory that may be
out-of-bounds. These intrinsics enable optimizations like early-exit loop
vectorization where the vectorized loop may read beyond the end of an array,
-provided the access is guaranteed be valid by target-specific checks.
+provided the access is guaranteed to be valid by target-specific checks.
.. _int_speculative_load:
@@ -28204,11 +28204,11 @@ This is an overloaded intrinsic.
::
; Direct form: number of accessible bytes given as i64
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i64 <num_accessible_bytes>)
- declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i64 <num_accessible_bytes>)
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
+ declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
; Oracle form: accessible bytes computed by calling oracle_fn(args...)
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, ptr <oracle_fn>, ...)
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i1 <from_end>, ptr <oracle_fn>, ...)
Overview:
"""""""""
@@ -28223,25 +28223,36 @@ Arguments:
""""""""""
The first argument is a pointer to the memory location to load from. The return
-type must be a vector type with a power-of-2 size in bytes. The remaining
-arguments determine the *number of accessible bytes* starting from ``ptr``,
+type must be a vector type with a power-of-2 size in bytes. The second argument
+is an ``i1`` constant flag ``from_end`` that specifies whether the ``N``
+accessible bytes are counted from the start or the end of the loaded values (see
+Semantics). The remaining arguments determine the *number of accessible bytes*,
denoted ``N`` below.
-In the **direct form**, the second argument is an ``i64`` specifying ``N``
-directly. In the **oracle form**, the second argument must be a direct
+In the **direct form**, the third argument is an ``i64`` specifying ``N``
+directly. In the **oracle form**, the third argument must be a direct
reference to a function returning ``i64`` that may only read memory through its
arguments (indirect function pointers are not permitted); the remaining
-arguments are forwarded to it, and the return value is ``N``.
+arguments are forwarded to it, and its return value is ``N``.
Semantics:
""""""""""
-The intrinsic reads ``N`` bytes starting from ``ptr`` and returns the stored
-values for those bytes. These bytes must lie within the bounds of an allocated
-object that ``ptr`` is :ref:`based <pointeraliasing>` on.
+Let ``S`` denote the size of the return type in bytes. The intrinsic performs
+a load of ``S`` bytes starting from ``ptr``.
-Bytes at offsets in ``[N, size of the return type)`` are ``poison`` and are not
-considered accessed for the purposes of data races or ``noalias`` constraints.
+When ``from_end`` is ``false``, the first ``N`` bytes (offsets ``[0, N)``)
+are the stored values read from memory. Bytes at offsets ``[N, S)`` are
+``poison``.
+
+When ``from_end`` is ``true``, the last ``N`` bytes (offsets ``[S - N, S)``)
+are the stored values read from memory. Bytes at offsets ``[0, S - N)`` are
+``poison``.
+
+In both cases, the ``N`` accessible bytes must lie within the bounds of an
+allocated object that ``ptr`` is :ref:`based <pointeraliasing>` on, and
+poison bytes are not considered accessed for the purposes of data races or
+``noalias`` constraints. The behavior is undefined if ``N`` exceeds ``S``.
The behavior is undefined if the speculative load accesses memory that would
fault (i.e., the oracle or ``llvm.can.load.speculatively`` would indicate the
@@ -28298,7 +28309,7 @@ alignment guarantees all such loads cannot cross a page boundary.
speculative_path:
; Safe to speculatively load from %ptr
- %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 16)
+ %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 16)
...
safe_path:
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 27c95134808cd..6580524ef3fbc 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2655,8 +2655,9 @@ def int_masked_srem:
// Speculatively load a value from memory; lowers to a regular aligned load.
def int_speculative_load:
DefaultAttrsIntrinsic<[llvm_anyvector_ty],
- [llvm_anyptr_ty, llvm_vararg_ty],
- [IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
+ [llvm_anyptr_ty, llvm_i1_ty, llvm_vararg_ty],
+ [IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>,
+ ImmArg<ArgIndex<1>>]>;
// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
// The size can be a runtime value to support scalable vectors.
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 0e360b97eeddd..bacabbe7cd3c0 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5165,7 +5165,7 @@ void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
MachineMemOperand::Flags MMOFlags = MachineMemOperand::MOLoad;
LocationSize LocSize = StoreSize.isScalable()
? LocationSize::beforeOrAfterPointer()
- : LocationSize::precise(StoreSize);
+ : LocationSize::upperBound(StoreSize);
MachineMemOperand *MMO = DAG.getMachineFunction().getMachineMemOperand(
MachinePointerInfo(PtrOperand), MMOFlags, LocSize, Alignment, AAInfo);
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 3dc5c32e647ae..e5f17ccf515b1 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -6862,26 +6862,25 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
"llvm.speculative.load type must have a power-of-2 size", &Call);
unsigned NumArgs = Call.arg_size();
- Check(NumArgs >= 2, "llvm.speculative.load requires at least 2 arguments",
+ Check(NumArgs >= 3, "llvm.speculative.load requires at least 3 arguments",
&Call);
- Value *SecondArg = Call.getArgOperand(1);
- if (SecondArg->getType()->isIntegerTy(64)) {
- // Direct form: (ptr, i64 num_accessible_bytes)
- Check(NumArgs == 2,
- "llvm.speculative.load direct form requires exactly 2 arguments",
- &Call);
- if (auto *CI = dyn_cast<ConstantInt>(SecondArg)) {
+ Value *PayloadArg = Call.getArgOperand(2);
+ if (PayloadArg->getType()->isIntegerTy(64)) {
+ // Direct form: (ptr, i1 from_end, i64 num_accessible_bytes)
+ Check(NumArgs == 3,
+ "llvm.speculative.load direct form has too many arguments", &Call);
+ if (auto *CI = dyn_cast<ConstantInt>(PayloadArg)) {
Check(Size.isScalable() || CI->getZExtValue() <= Size.getFixedValue(),
"llvm.speculative.load num_accessible_bytes must not exceed "
"the result size in bytes",
&Call);
}
} else {
- // Oracle form: (ptr, oracle_fn_ptr, args...)
- auto *OracleFn = dyn_cast<Function>(SecondArg);
+ // Oracle form: (ptr, i1 from_end, oracle_fn_ptr, args...)
+ auto *OracleFn = dyn_cast<Function>(PayloadArg);
Check(OracleFn,
- "llvm.speculative.load second argument must be i64 or a direct "
+ "llvm.speculative.load third argument must be i64 or a direct "
"reference to an oracle function",
&Call);
@@ -6893,13 +6892,16 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
FunctionType *FTy = OracleFn->getFunctionType();
Check(FTy->getReturnType()->isIntegerTy(64),
"llvm.speculative.load oracle function must return i64", &Call);
- unsigned NumOracleArgs = NumArgs - 2;
+
+ unsigned OracleArgsStart = 3;
+ unsigned NumOracleArgs = NumArgs - OracleArgsStart;
Check(FTy->isVarArg() ? NumOracleArgs >= FTy->getNumParams()
: NumOracleArgs == FTy->getNumParams(),
"llvm.speculative.load oracle function argument count mismatch",
&Call);
for (unsigned I = 0, E = FTy->getNumParams(); I < E; ++I) {
- Check(FTy->getParamType(I) == Call.getArgOperand(I + 2)->getType(),
+ Check(FTy->getParamType(I) ==
+ Call.getArgOperand(I + OracleArgsStart)->getType(),
"llvm.speculative.load oracle function argument type mismatch",
&Call);
}
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
index 2746b9ea87daa..a547cee576864 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -11,7 +11,7 @@ define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 12)
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, i64 12)
ret <4 x i32> %load
}
@@ -20,7 +20,7 @@ define <2 x double> @speculative_load_v2f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 8)
+ %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, i64 8)
ret <2 x double> %load
}
@@ -31,7 +31,7 @@ define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> (ptr, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i64 8)
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 false, i64 8)
ret <vscale x 4 x i32> %load
}
@@ -40,7 +40,7 @@ define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 2 x double> (ptr, ...) @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i64 4)
+ %load = call <vscale x 2 x double> (ptr, i1, ...) @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i1 false, i64 4)
ret <vscale x 2 x double> %load
}
@@ -53,7 +53,7 @@ define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x i32> %load
}
@@ -62,7 +62,7 @@ define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x i64> (ptr, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <2 x i64> (ptr, i1, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <2 x i64> %load
}
@@ -71,7 +71,7 @@ define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <8 x i16> (ptr, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <8 x i16> (ptr, i1, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <8 x i16> %load
}
@@ -80,7 +80,7 @@ define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <16 x i8> (ptr, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <16 x i8> (ptr, i1, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <16 x i8> %load
}
@@ -89,7 +89,7 @@ define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x float> (ptr, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x float> (ptr, i1, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x float> %load
}
@@ -98,15 +98,53 @@ define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <2 x double> %load
}
+; from_end tests
+
+define <4 x i32> @speculative_load_v4i32_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_from_end:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, i64 12)
+ ret <4 x i32> %load
+}
+
+define <vscale x 4 x i32> @speculative_load_nxv4i32_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv4i32_from_end:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 true, i64 8)
+ ret <vscale x 4 x i32> %load
+}
+
+define <4 x i32> @speculative_load_v4i32_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle_from_end:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
+}
+
+define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_nxv4i32_oracle_from_end:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret <vscale x 4 x i32> %load
+}
+
define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle(ptr %ptr, i64 %n) {
; CHECK-LABEL: speculative_load_nxv4i32_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> (ptr, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <vscale x 4 x i32> %load
}
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index e387a17990012..fa345408940d1 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -9,7 +9,7 @@ define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 8)
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, i64 8)
ret <4 x i32> %load
}
@@ -18,7 +18,7 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %ymm0
; CHECK-NEXT: retq
- %load = call <8 x i32> (ptr, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 24)
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 false, i64 24)
ret <8 x i32> %load
}
@@ -30,7 +30,7 @@ define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x i32> %load
}
@@ -39,7 +39,7 @@ define <8 x i32> @speculative_load_v8i32_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %ymm0
; CHECK-NEXT: retq
- %load = call <8 x i32> (ptr, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <8 x i32> %load
}
@@ -48,7 +48,7 @@ define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <2 x i64> (ptr, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <2 x i64> (ptr, i1, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <2 x i64> %load
}
@@ -57,7 +57,7 @@ define <4 x i64> @speculative_load_v4i64_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %ymm0
; CHECK-NEXT: retq
- %load = call <4 x i64> (ptr, ...) @llvm.speculative.load.v4i64.p0(ptr align 32 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x i64> (ptr, i1, ...) @llvm.speculative.load.v4i64.p0(ptr align 32 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x i64> %load
}
@@ -66,7 +66,7 @@ define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <4 x float> (ptr, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x float> (ptr, i1, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x float> %load
}
@@ -75,7 +75,7 @@ define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <2 x double> %load
}
@@ -84,7 +84,7 @@ define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <8 x i16> (ptr, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <8 x i16> (ptr, i1, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <8 x i16> %load
}
@@ -93,6 +93,44 @@ define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <16 x i8> (ptr, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <16 x i8> (ptr, i1, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <16 x i8> %load
}
+
+; from_end tests
+
+define <4 x i32> @speculative_load_v4i32_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_from_end:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, i64 8)
+ ret <4 x i32> %load
+}
+
+define <8 x i32> @speculative_load_v8i32_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v8i32_from_end:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 true, i64 24)
+ ret <8 x i32> %load
+}
+
+define <4 x i32> @speculative_load_v4i32_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle_from_end:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
+}
+
+define <8 x i32> @speculative_load_v8i32_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i32_oracle_from_end:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i32> %load
+}
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
index 34629849025dc..7fd2d340679aa 100644
--- a/llvm/test/Verifier/speculative-load.ll
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -1,8 +1,8 @@
; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, ...)
-declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, ...)
-declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, ...)
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i1, ...)
+declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, i1, ...)
+declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, i1, ...)
declare i32 @bad_oracle_ret(ptr, i64) memory(argmem: read)
declare i64 @good_oracle(ptr, i64) memory(argmem: read)
@@ -11,70 +11,77 @@ declare i64 @side_effecting_oracle(ptr, i64)
define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <3 x i32> (ptr, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
- %res = call <3 x i32> (ptr, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
+; CHECK-NEXT: %res = call <3 x i32> (ptr, i1, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i1 false, i64 0)
+ %res = call <3 x i32> (ptr, i1, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i1 false, i64 0)
ret <3 x i32> %res
}
define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <vscale x 3 x i32> (ptr, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
- %res = call <vscale x 3 x i32> (ptr, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
+; CHECK-NEXT: %res = call <vscale x 3 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i1 false, i64 0)
+ %res = call <vscale x 3 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i1 false, i64 0)
ret <vscale x 3 x i32> %res
}
+define <4 x i32> @test_too_few_args(ptr %ptr) {
+; CHECK: llvm.speculative.load requires at least 3 arguments
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 true)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 true)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_direct_form_extra_args(ptr %ptr) {
+; CHECK: llvm.speculative.load direct form has too many arguments
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 8, i64 4)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 8, i64 4)
+ ret <4 x i32> %res
+}
+
define <4 x i32> @test_oracle_wrong_return_type(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function must return i64
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
ret <4 x i32> %res
}
define <4 x i32> @test_oracle_too_few_args(ptr %ptr) {
; CHECK: llvm.speculative.load oracle function argument count mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle)
ret <4 x i32> %res
}
define <4 x i32> @test_oracle_too_many_args(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function argument count mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
ret <4 x i32> %res
}
define <4 x i32> @test_oracle_arg_type_mismatch(ptr %ptr) {
; CHECK: llvm.speculative.load oracle function argument type mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @oracle_i32_param, i64 42)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @oracle_i32_param, i64 42)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @oracle_i32_param, i64 42)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @oracle_i32_param, i64 42)
ret <4 x i32> %res
}
define <4 x i32> @test_non_function_oracle(ptr %ptr, ptr %not_fn) {
-; CHECK: llvm.speculative.load second argument must be i64 or a direct reference to an oracle function
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr %not_fn)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr %not_fn)
- ret <4 x i32> %res
-}
-
-define <4 x i32> @test_direct_form_extra_args(ptr %ptr) {
-; CHECK: llvm.speculative.load direct form requires exactly 2 arguments
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 8, i64 4)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 8, i64 4)
+; CHECK: llvm.speculative.load third argument must be i64 or a direct reference to an oracle function
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr %not_fn)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr %not_fn)
ret <4 x i32> %res
}
define <4 x i32> @test_oracle_side_effects(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function must not have side effects and may only read memory through its arguments
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
ret <4 x i32> %res
}
define <4 x i32> @test_num_accessible_bytes_exceeds_size(ptr %ptr) {
; CHECK: llvm.speculative.load num_accessible_bytes must not exceed the result size in bytes
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 32)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 32)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 32)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 32)
ret <4 x i32> %res
}
More information about the llvm-commits
mailing list