[llvm] Provide intrinsics for speculative loads (PR #179642)
Florian Hahn via llvm-commits
llvm-commits at lists.llvm.org
Mon Apr 13 03:52:56 PDT 2026
https://github.com/fhahn updated https://github.com/llvm/llvm-project/pull/179642
>From 6985acfa9ed21f305b1a18a9e5c5d3edd17ac002 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Wed, 4 Feb 2026 11:06:38 +0000
Subject: [PATCH 1/9] Add speculative load intrinsics
Introduce two new intrinsics to enable vectorization of loops with early
exits that have potentially faulting loads.
This has previously been discussed in
https://github.com/llvm/llvm-project/pull/120603 and is similar to
@nikic's https://hackmd.io/@nikic/S1O4QWYZkx, with the major difference
being that there is no `%defined_size` argument and instead the load
returns the stored values for the bytes within bounds and undef
otherwise. I don't think we can easily compute the defined size because
it may depend on the loaded values (i.e. at what lane the early exit has
been taken).
1. `@llvm.speculative.load` (name subject to change) - perform a load that
may access memory beyond the allocated object. It must be used in
combination with `@llvm.can.load.speculatively` to ensure the load is
guaranteed to not trap.
2. `@llvm.can.load.speculatively` - Returns true if it's safe to speculatively
load a given number of bytes from a pointer. The semantics are
target-dependent. On some targets, this may check that the access
does not cross page boundaries, or stricter checks for example on
AArch64 with MTE, which limits the access size to 16 bytes.
`@llvm.speculative.load` is lowered to a regular load in SelectionDAG
without MODereferenceable. I am not sure if we need to be more careful
than this, i.e. if we could still reason about SelectionDAG loads to
infer dereferencability for the pointer.
`@llvm.can.load.speculatively` is lowered to regular IR in PreISel
lowering, using a target-lowering hook. By default, it conservatively
expands to false.
These intrinsics should allow the loop vectorizer to vectorize early-exit
loops with potentially non-dereferenceable loads.
---
llvm/docs/LangRef.rst | 113 ++++++++++++++
llvm/include/llvm/CodeGen/TargetLowering.h | 13 ++
llvm/include/llvm/IR/Intrinsics.td | 14 ++
llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp | 39 +++++
.../SelectionDAG/SelectionDAGBuilder.cpp | 30 ++++
.../SelectionDAG/SelectionDAGBuilder.h | 1 +
llvm/lib/IR/Verifier.cpp | 18 +++
.../Target/AArch64/AArch64ISelLowering.cpp | 49 ++++++
llvm/lib/Target/AArch64/AArch64ISelLowering.h | 2 +
.../CodeGen/AArch64/can-load-speculatively.ll | 78 ++++++++++
.../AArch64/speculative-load-intrinsic-sve.ll | 66 ++++++++
.../AArch64/speculative-load-intrinsic.ll | 117 ++++++++++++++
.../CodeGen/X86/can-load-speculatively.ll | 32 ++++
.../CodeGen/X86/speculative-load-intrinsic.ll | 146 ++++++++++++++++++
llvm/test/Verifier/can-load-speculatively.ll | 19 +++
llvm/test/Verifier/speculative-load.ll | 18 +++
16 files changed, 755 insertions(+)
create mode 100644 llvm/test/CodeGen/AArch64/can-load-speculatively.ll
create mode 100644 llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
create mode 100644 llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
create mode 100644 llvm/test/CodeGen/X86/can-load-speculatively.ll
create mode 100644 llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
create mode 100644 llvm/test/Verifier/can-load-speculatively.ll
create mode 100644 llvm/test/Verifier/speculative-load.ll
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index b82d3e3ffbeb8..ffd1c3e1f8912 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28025,6 +28025,119 @@ Semantics:
Follows the same semantics as :ref:`srem <i_srem>` with the exception that disabled lanes cannot produce undefined behaviour and always result in poison.
+Speculative Load Intrinsics
+---------------------------
+
+LLVM provides intrinsics for speculatively loading memory that may be
+out-of-bounds. These intrinsics enable optimizations like early-exit loop
+vectorization where the vectorized loop may read beyond the end of an array,
+provided the access is guaranteed to not trap by target-specific checks.
+
+.. _int_speculative_load:
+
+'``llvm.speculative.load``' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic.
+
+::
+
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>)
+ declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>)
+ declare i64 @llvm.speculative.load.i64.p0(ptr <ptr>)
+
+Overview:
+"""""""""
+
+The '``llvm.speculative.load``' intrinsic loads a value from memory. Unlike a
+regular load, the memory access may
+extend beyond the bounds of the allocated object, provided the pointer has been
+verified by :ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to
+ensure the access cannot fault.
+
+Arguments:
+""""""""""
+
+The argument is a pointer to the memory location to load from. The return type
+must have a power-of-2 size in bytes.
+
+Semantics:
+""""""""""
+
+The '``llvm.speculative.load``' intrinsic performs a load that may access
+memory beyond the allocated object. It must be used in combination with
+:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure
+the access cannot fault.
+
+For bytes that are within the bounds of the allocated object, the intrinsic
+returns the stored value. For bytes that are beyond the bounds of the
+allocated object, the intrinsic returns ``undef`` for those bytes. At least the
+first accessed byte must be within the bounds of an allocated object the pointer is
+based on.
+
+The behavior is undefined if this intrinsic is used to load from a pointer
+for which ``llvm.can.load.speculatively`` returns false.
+
+.. _int_can_load_speculatively:
+
+'``llvm.can.load.speculatively``' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic.
+
+::
+
+ declare i1 @llvm.can.load.speculatively.p0(ptr <ptr>, i64 <num_bytes>)
+ declare i1 @llvm.can.load.speculatively.p1(ptr addrspace(1) <ptr>, i64 <num_bytes>)
+
+Overview:
+"""""""""
+
+The '``llvm.can.load.speculatively``' intrinsic returns true if it is safe
+to speculatively load ``num_bytes`` bytes starting from ``ptr``,
+even if the memory may be beyond the bounds of an allocated object.
+
+Arguments:
+""""""""""
+
+The first argument is a pointer to the memory location.
+
+The second argument is an i64 specifying the size in bytes of the load.
+The size must be a positive power of 2. If the size is not a power-of-2, the
+result is ``poison``.
+
+Semantics:
+""""""""""
+
+This intrinsic has **target-dependent** semantics. It returns ``true`` if
+loading ``num_bytes`` bytes from ``ptr`` is guaranteed not to trap,
+even if the memory is beyond the bounds of an allocated object. It returns
+``false`` otherwise.
+
+The specific conditions under which this intrinsic returns ``true`` are
+determined by the target. For example, a target may check whether the pointer
+alignment guarantees the load cannot cross a page boundary.
+
+.. code-block:: llvm
+
+ ; Check if we can safely load 16 bytes from %ptr
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ br i1 %can_load, label %speculative_path, label %safe_path
+
+ speculative_path:
+ ; Safe to speculatively load from %ptr
+ %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr)
+ ...
+
+ safe_path:
+ ; Fall back to masked load or scalar operations
+ ...
+
+
Memory Use Markers
------------------
diff --git a/llvm/include/llvm/CodeGen/TargetLowering.h b/llvm/include/llvm/CodeGen/TargetLowering.h
index 59a0f2d2e0c2a..9872113db6bbd 100644
--- a/llvm/include/llvm/CodeGen/TargetLowering.h
+++ b/llvm/include/llvm/CodeGen/TargetLowering.h
@@ -2316,6 +2316,19 @@ class LLVM_ABI TargetLoweringBase {
llvm_unreachable("Store conditional unimplemented on this target");
}
+ /// Emit code to check if a speculative load of the given size from Ptr is
+ /// safe. Returns a Value* representing the check result (i1), or nullptr
+ /// to use the default lowering (which returns false). Targets can override
+ /// to provide their own safety check (e.g., alignment-based page boundary
+ /// check).
+ /// \param Builder IRBuilder positioned at the intrinsic call site
+ /// \param Ptr the pointer operand
+ /// \param Size the size in bytes (constant or runtime value for scalable)
+ virtual Value *emitCanLoadSpeculatively(IRBuilderBase &Builder, Value *Ptr,
+ Value *Size) const {
+ return nullptr;
+ }
+
/// Perform a masked atomicrmw using a target-specific intrinsic. This
/// represents the core LL/SC loop which will be lowered at a late stage by
/// the backend. The target-specific intrinsic returns the loaded value and
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 80ef58f51e7bb..39397dc72b4ab 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2669,6 +2669,20 @@ def int_masked_srem:
LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>],
[IntrNoMem]>;
+// Speculatively load a value from memory; lowers to a regular aligned load.
+// The loaded type must have a power-of-2 size.
+def int_speculative_load:
+ DefaultAttrsIntrinsic<[llvm_any_ty],
+ [llvm_anyptr_ty],
+ [IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
+
+// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
+// The size can be a runtime value to support scalable vectors.
+def int_can_load_speculatively:
+ DefaultAttrsIntrinsic<[llvm_i1_ty],
+ [llvm_anyptr_ty, llvm_i64_ty],
+ [IntrNoMem, IntrSpeculatable, IntrWillReturn]>;
+
// Test whether a pointer is associated with a type metadata identifier.
def int_type_test : DefaultAttrsIntrinsic<[llvm_i1_ty], [llvm_ptr_ty, llvm_metadata_ty],
[IntrNoMem, IntrSpeculatable]>;
diff --git a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
index 51f7843543a3e..be705e48eb08d 100644
--- a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
+++ b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
@@ -136,6 +136,42 @@ static bool lowerLoadRelative(Function &F) {
return Changed;
}
+/// Lower @llvm.can.load.speculatively using target-specific expansion.
+/// Each target provides its own expansion via
+/// TargetLowering::emitCanLoadSpeculatively.
+/// The default expansion returns false (conservative).
+static bool lowerCanLoadSpeculatively(Function &F, const TargetMachine *TM) {
+ if (F.use_empty())
+ return false;
+
+ bool Changed = false;
+
+ for (Use &U : llvm::make_early_inc_range(F.uses())) {
+ auto *CI = dyn_cast<CallInst>(U.getUser());
+ if (!CI || CI->getCalledOperand() != &F)
+ continue;
+
+ Function *ParentFunc = CI->getFunction();
+ const TargetLowering *TLI =
+ TM->getSubtargetImpl(*ParentFunc)->getTargetLowering();
+
+ IRBuilder<> Builder(CI);
+ Value *Ptr = CI->getArgOperand(0);
+ Value *Size = CI->getArgOperand(1);
+
+ // Ask target for expansion; nullptr means use default (return false)
+ Value *Result = TLI->emitCanLoadSpeculatively(Builder, Ptr, Size);
+ if (!Result)
+ Result = Builder.getFalse();
+
+ CI->replaceAllUsesWith(Result);
+ CI->eraseFromParent();
+ Changed = true;
+ }
+
+ return Changed;
+}
+
// ObjCARC has knowledge about whether an obj-c runtime function needs to be
// always tail-called or never tail-called.
static CallInst::TailCallKind getOverridingTailCallKind(const Function &F) {
@@ -674,6 +710,9 @@ bool PreISelIntrinsicLowering::lowerIntrinsics(Module &M) const {
case Intrinsic::load_relative:
Changed |= lowerLoadRelative(F);
break;
+ case Intrinsic::can_load_speculatively:
+ Changed |= lowerCanLoadSpeculatively(F, TM);
+ break;
case Intrinsic::is_constant:
case Intrinsic::objectsize:
Changed |= forEachCall(F, [&](CallInst *CI) {
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 1f3b099c9c577..a3ffdb0ddf737 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5143,6 +5143,33 @@ void SelectionDAGBuilder::visitMaskedLoad(const CallInst &I, bool IsExpanding) {
setValue(&I, Res);
}
+void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
+ SDLoc sdl = getCurSDLoc();
+ Value *PtrOperand = I.getArgOperand(0);
+ SDValue Ptr = getValue(PtrOperand);
+
+ const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+ EVT VT = TLI.getValueType(DAG.getDataLayout(), I.getType());
+ Align Alignment = I.getParamAlign(0).valueOrOne();
+ AAMDNodes AAInfo = I.getAAMetadata();
+ TypeSize StoreSize = VT.getStoreSize();
+
+ SDValue InChain = DAG.getRoot();
+
+ // Use MOLoad but NOT MODereferenceable - the memory may not be
+ // fully dereferenceable.
+ MachineMemOperand::Flags MMOFlags = MachineMemOperand::MOLoad;
+ LocationSize LocSize = StoreSize.isScalable()
+ ? LocationSize::beforeOrAfterPointer()
+ : LocationSize::precise(StoreSize);
+ MachineMemOperand *MMO = DAG.getMachineFunction().getMachineMemOperand(
+ MachinePointerInfo(PtrOperand), MMOFlags, LocSize, Alignment, AAInfo);
+
+ SDValue Load = DAG.getLoad(VT, sdl, InChain, Ptr, MMO);
+ PendingLoads.push_back(Load.getValue(1));
+ setValue(&I, Load);
+}
+
void SelectionDAGBuilder::visitMaskedGather(const CallInst &I) {
SDLoc sdl = getCurSDLoc();
@@ -6904,6 +6931,9 @@ void SelectionDAGBuilder::visitIntrinsicCall(const CallInst &I,
case Intrinsic::masked_compressstore:
visitMaskedStore(I, true /* IsCompressing */);
return;
+ case Intrinsic::speculative_load:
+ visitSpeculativeLoad(I);
+ return;
case Intrinsic::powi:
setValue(&I, ExpandPowI(sdl, getValue(I.getArgOperand(0)),
getValue(I.getArgOperand(1)), DAG));
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
index bab0509dd138f..419e6a5fba6eb 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
@@ -620,6 +620,7 @@ class SelectionDAGBuilder {
void visitStore(const StoreInst &I);
void visitMaskedLoad(const CallInst &I, bool IsExpanding = false);
void visitMaskedStore(const CallInst &I, bool IsCompressing = false);
+ void visitSpeculativeLoad(const CallInst &I);
void visitMaskedGather(const CallInst &I);
void visitMaskedScatter(const CallInst &I);
void visitAtomicCmpXchg(const AtomicCmpXchgInst &I);
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index d4ade9c7ce534..bae181e098c49 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -6814,6 +6814,24 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
&Call);
break;
}
+ case Intrinsic::speculative_load: {
+ Type *LoadTy = Call.getType();
+ TypeSize Size = DL.getTypeStoreSize(LoadTy);
+ // For scalable vectors, check the known minimum size is a power of 2.
+ Check(Size.getKnownMinValue() > 0 && isPowerOf2_64(Size.getKnownMinValue()),
+ "llvm.speculative.load type must have a power-of-2 size", &Call);
+ break;
+ }
+ case Intrinsic::can_load_speculatively: {
+ // If size is a constant, verify it's a positive power of 2.
+ if (auto *SizeCI = dyn_cast<ConstantInt>(Call.getArgOperand(1))) {
+ uint64_t Size = SizeCI->getZExtValue();
+ Check(Size > 0 && isPowerOf2_64(Size),
+ "llvm.can.load.speculatively size must be a positive power of 2",
+ &Call);
+ }
+ break;
+ }
case Intrinsic::vector_insert: {
Value *Vec = Call.getArgOperand(0);
Value *SubVec = Call.getArgOperand(1);
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
index ece24767bdbb9..0a3a90f1df949 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
@@ -30745,6 +30745,55 @@ Value *AArch64TargetLowering::emitStoreConditional(IRBuilderBase &Builder,
return CI;
}
+Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
+ Value *Ptr,
+ Value *Size) const {
+ // For power-of-2 sizes <= 16, emit alignment check: (ptr & (size - 1)) == 0.
+ // If the pointer is aligned to at least 'size' bytes, loading 'size' bytes
+ // cannot cross a page boundary, so it's safe to speculate.
+ // The 16-byte limit ensures correctness with MTE (memory tagging), since
+ // MTE uses 16-byte tag granules.
+ //
+ // The alignment check only works for power-of-2 sizes. For non-power-of-2
+ // sizes, we conservatively return false.
+ const DataLayout &DL =
+ Builder.GetInsertBlock()->getModule()->getDataLayout();
+
+ if (auto *CI = dyn_cast<ConstantInt>(Size)) {
+ uint64_t SizeVal = CI->getZExtValue();
+ assert(isPowerOf2_64(SizeVal) && "size must be power-of-two");
+ // For constant sizes > 16, return nullptr (default false).
+ if (SizeVal > 16)
+ return nullptr;
+
+ // Power-of-2 constant size <= 16: use fast alignment check.
+ unsigned PtrBits = DL.getPointerSizeInBits();
+ Type *IntPtrTy = Builder.getIntNTy(PtrBits);
+ Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
+ Value *Mask = ConstantInt::get(IntPtrTy, SizeVal - 1);
+ Value *Masked = Builder.CreateAnd(PtrInt, Mask);
+ return Builder.CreateICmpEQ(Masked, ConstantInt::get(IntPtrTy, 0));
+ }
+
+ // Check power-of-2 size <= 16 and alignment.
+ unsigned PtrBits = DL.getPointerSizeInBits();
+ Type *IntPtrTy = Builder.getIntNTy(PtrBits);
+ Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
+ Value *SizeExt = Builder.CreateZExtOrTrunc(Size, IntPtrTy);
+
+ Value *SizeLE16 =
+ Builder.CreateICmpULE(SizeExt, ConstantInt::get(IntPtrTy, 16));
+
+ // alignment check: (ptr & (size - 1)) == 0
+ Value *SizeMinusOne =
+ Builder.CreateSub(SizeExt, ConstantInt::get(IntPtrTy, 1));
+ Value *Masked = Builder.CreateAnd(PtrInt, SizeMinusOne);
+ Value *AlignCheck =
+ Builder.CreateICmpEQ(Masked, ConstantInt::get(IntPtrTy, 0));
+
+ return Builder.CreateAnd(SizeLE16, AlignCheck);
+}
+
bool AArch64TargetLowering::functionArgumentNeedsConsecutiveRegisters(
Type *Ty, CallingConv::ID CallConv, bool isVarArg,
const DataLayout &DL) const {
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.h b/llvm/lib/Target/AArch64/AArch64ISelLowering.h
index 58efdd3e18fc0..2c5d32d135f11 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.h
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.h
@@ -340,6 +340,8 @@ class AArch64TargetLowering : public TargetLowering {
AtomicOrdering Ord) const override;
Value *emitStoreConditional(IRBuilderBase &Builder, Value *Val, Value *Addr,
AtomicOrdering Ord) const override;
+ Value *emitCanLoadSpeculatively(IRBuilderBase &Builder, Value *Ptr,
+ Value *Size) const override;
void emitAtomicCmpXchgNoStoreLLBalance(IRBuilderBase &Builder) const override;
diff --git a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
new file mode 100644
index 0000000000000..7916f2e4d340f
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
@@ -0,0 +1,78 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -mtriple=aarch64-unknown-linux-gnu -passes=pre-isel-intrinsic-lowering -S < %s | FileCheck %s
+
+; Test that @llvm.can.load.speculatively is lowered to an alignment check
+; for power-of-2 sizes <= 16 bytes on AArch64, and returns false for larger sizes.
+; The 16-byte limit ensures correctness with MTE (memory tagging).
+; Note: non-power-of-2 constant sizes are rejected by the verifier.
+
+define i1 @can_load_speculatively_16(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_16(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
+; Size > 16 - returns false (may cross MTE tag granule boundary)
+define i1 @can_load_speculatively_32(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_32(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 32)
+ ret i1 %can_load
+}
+
+; Size > 16 - returns false (may cross MTE tag granule boundary)
+define i1 @can_load_speculatively_64(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_64(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 64)
+ ret i1 %can_load
+}
+
+; Test with address space
+define i1 @can_load_speculatively_addrspace1(ptr addrspace(1) %ptr) {
+; CHECK-LABEL: @can_load_speculatively_addrspace1(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr addrspace(1) [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p1(ptr addrspace(1) %ptr, i64 16)
+ ret i1 %can_load
+}
+
+; Test size 8 (within limit, power-of-2)
+define i1 @can_load_speculatively_8(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_8(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 7
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 8)
+ ret i1 %can_load
+}
+
+; Test with runtime size - checks size <= 16 and alignment
+define i1 @can_load_speculatively_runtime(ptr %ptr, i64 %size) {
+; CHECK-LABEL: @can_load_speculatively_runtime(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = icmp ule i64 [[SIZE:%.*]], 16
+; CHECK-NEXT: [[TMP3:%.*]] = sub i64 [[SIZE]], 1
+; CHECK-NEXT: [[TMP4:%.*]] = and i64 [[TMP1]], [[TMP3]]
+; CHECK-NEXT: [[TMP5:%.*]] = icmp eq i64 [[TMP4]], 0
+; CHECK-NEXT: [[TMP6:%.*]] = and i1 [[TMP2]], [[TMP5]]
+; CHECK-NEXT: ret i1 [[TMP6]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 %size)
+ ret i1 %can_load
+}
+
+declare i1 @llvm.can.load.speculatively.p0(ptr, i64)
+declare i1 @llvm.can.load.speculatively.p1(ptr addrspace(1), i64)
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
new file mode 100644
index 0000000000000..78a56f3539d11
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
@@ -0,0 +1,66 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-unknown-linux-gnu -mattr=+sve < %s | FileCheck %s
+
+; Test that @llvm.speculative.load with scalable vectors is lowered to a
+; regular load in SelectionDAG.
+
+define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv4i32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr)
+ ret <vscale x 4 x i32> %load
+}
+
+define <vscale x 2 x i64> @speculative_load_nxv2i64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv2i64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr %ptr)
+ ret <vscale x 2 x i64> %load
+}
+
+define <vscale x 8 x i16> @speculative_load_nxv8i16(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv8i16:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr align 8 %ptr)
+ ret <vscale x 8 x i16> %load
+}
+
+define <vscale x 16 x i8> @speculative_load_nxv16i8(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv16i8:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr %ptr)
+ ret <vscale x 16 x i8> %load
+}
+
+define <vscale x 4 x float> @speculative_load_nxv4f32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv4f32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr align 4 %ptr)
+ ret <vscale x 4 x float> %load
+}
+
+define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv2f64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr)
+ ret <vscale x 2 x double> %load
+}
+
+declare <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr)
+declare <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr)
+declare <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr)
+declare <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr)
+declare <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr)
+declare <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr)
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
new file mode 100644
index 0000000000000..8f9a17414ff05
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -0,0 +1,117 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-unknown-linux-gnu < %s | FileCheck %s
+
+; Test that @llvm.speculative.load is lowered to a regular load
+; in SelectionDAG, respecting the alignment attribute.
+
+define <4 x i32> @speculative_load_v4i32_align16(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_align16:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr)
+ ret <4 x i32> %load
+}
+
+define <4 x i32> @speculative_load_v4i32_align4(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_align4:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 4 %ptr)
+ ret <4 x i32> %load
+}
+
+define <4 x i32> @speculative_load_v4i32_noalign(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_noalign:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr)
+ ret <4 x i32> %load
+}
+
+define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v8i32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldp q0, q1, [x0]
+; CHECK-NEXT: ret
+ %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr)
+ ret <8 x i32> %load
+}
+
+define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v2i64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
+ ret <2 x i64> %load
+}
+
+define <4 x float> @speculative_load_v4f32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4f32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
+ ret <4 x float> %load
+}
+
+define <2 x double> @speculative_load_v2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v2f64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr)
+ ret <2 x double> %load
+}
+
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr)
+declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr)
+declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr)
+declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr)
+declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr)
+
+; Scalar type tests
+
+define i32 @speculative_load_i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_i32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr w0, [x0]
+; CHECK-NEXT: ret
+ %load = call i32 @llvm.speculative.load.i32.p0(ptr align 4 %ptr)
+ ret i32 %load
+}
+
+define i64 @speculative_load_i64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_i64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr x0, [x0]
+; CHECK-NEXT: ret
+ %load = call i64 @llvm.speculative.load.i64.p0(ptr %ptr)
+ ret i64 %load
+}
+
+define float @speculative_load_f32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_f32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr s0, [x0]
+; CHECK-NEXT: ret
+ %load = call float @llvm.speculative.load.f32.p0(ptr %ptr)
+ ret float %load
+}
+
+define double @speculative_load_f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_f64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr d0, [x0]
+; CHECK-NEXT: ret
+ %load = call double @llvm.speculative.load.f64.p0(ptr align 8 %ptr)
+ ret double %load
+}
+
+declare i32 @llvm.speculative.load.i32.p0(ptr)
+declare i64 @llvm.speculative.load.i64.p0(ptr)
+declare float @llvm.speculative.load.f32.p0(ptr)
+declare double @llvm.speculative.load.f64.p0(ptr)
diff --git a/llvm/test/CodeGen/X86/can-load-speculatively.ll b/llvm/test/CodeGen/X86/can-load-speculatively.ll
new file mode 100644
index 0000000000000..f51d3847e921d
--- /dev/null
+++ b/llvm/test/CodeGen/X86/can-load-speculatively.ll
@@ -0,0 +1,32 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -mtriple=x86_64-unknown-linux-gnu -passes=pre-isel-intrinsic-lowering -S < %s | FileCheck %s
+
+; Test that @llvm.can.load.speculatively returns false (default) on X86,
+; as X86 does not provide a target-specific expansion.
+
+define i1 @can_load_speculatively_16(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_16(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_32(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_32(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 32)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_8(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_8(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 8)
+ ret i1 %can_load
+}
+
+declare i1 @llvm.can.load.speculatively.p0(ptr, i64)
+
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
new file mode 100644
index 0000000000000..4dd8375e9c42a
--- /dev/null
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -0,0 +1,146 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+sse4.1 < %s | FileCheck %s --check-prefix=SSE
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2 < %s | FileCheck %s --check-prefix=AVX
+
+; Test that @llvm.speculative.load is lowered to a regular load
+; in SelectionDAG.
+
+define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
+; SSE-LABEL: speculative_load_v4i32:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v4i32:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: retq
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr)
+ ret <4 x i32> %load
+}
+
+define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
+; SSE-LABEL: speculative_load_v8i32:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: movaps 16(%rdi), %xmm1
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v8i32:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %ymm0
+; AVX-NEXT: retq
+ %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr)
+ ret <8 x i32> %load
+}
+
+define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
+; SSE-LABEL: speculative_load_v2i64:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v2i64:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: retq
+ %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
+ ret <2 x i64> %load
+}
+
+define <4 x float> @speculative_load_v4f32(ptr %ptr) {
+; SSE-LABEL: speculative_load_v4f32:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v4f32:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: retq
+ %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
+ ret <4 x float> %load
+}
+
+define <2 x double> @speculative_load_v2f64(ptr %ptr) {
+; SSE-LABEL: speculative_load_v2f64:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v2f64:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: retq
+ %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr)
+ ret <2 x double> %load
+}
+
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr)
+declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr)
+declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr)
+declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr)
+declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr)
+
+; Scalar type tests
+
+define i32 @speculative_load_i32(ptr %ptr) {
+; SSE-LABEL: speculative_load_i32:
+; SSE: # %bb.0:
+; SSE-NEXT: movl (%rdi), %eax
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_i32:
+; AVX: # %bb.0:
+; AVX-NEXT: movl (%rdi), %eax
+; AVX-NEXT: retq
+ %load = call i32 @llvm.speculative.load.i32.p0(ptr %ptr)
+ ret i32 %load
+}
+
+define i64 @speculative_load_i64(ptr %ptr) {
+; SSE-LABEL: speculative_load_i64:
+; SSE: # %bb.0:
+; SSE-NEXT: movq (%rdi), %rax
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_i64:
+; AVX: # %bb.0:
+; AVX-NEXT: movq (%rdi), %rax
+; AVX-NEXT: retq
+ %load = call i64 @llvm.speculative.load.i64.p0(ptr align 8 %ptr)
+ ret i64 %load
+}
+
+define float @speculative_load_f32(ptr %ptr) {
+; SSE-LABEL: speculative_load_f32:
+; SSE: # %bb.0:
+; SSE-NEXT: movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_f32:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX-NEXT: retq
+ %load = call float @llvm.speculative.load.f32.p0(ptr align 4 %ptr)
+ ret float %load
+}
+
+define double @speculative_load_f64(ptr %ptr) {
+; SSE-LABEL: speculative_load_f64:
+; SSE: # %bb.0:
+; SSE-NEXT: movsd {{.*#+}} xmm0 = mem[0],zero
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_f64:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX-NEXT: retq
+ %load = call double @llvm.speculative.load.f64.p0(ptr %ptr)
+ ret double %load
+}
+
+declare i32 @llvm.speculative.load.i32.p0(ptr)
+declare i64 @llvm.speculative.load.i64.p0(ptr)
+declare float @llvm.speculative.load.f32.p0(ptr)
+declare double @llvm.speculative.load.f64.p0(ptr)
diff --git a/llvm/test/Verifier/can-load-speculatively.ll b/llvm/test/Verifier/can-load-speculatively.ll
new file mode 100644
index 0000000000000..d2d69f70cfb60
--- /dev/null
+++ b/llvm/test/Verifier/can-load-speculatively.ll
@@ -0,0 +1,19 @@
+; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
+
+declare i1 @llvm.can.load.speculatively.p0(ptr, i64)
+
+; Test that constant size must be a positive power of 2
+
+define i1 @test_size_zero(ptr %ptr) {
+; CHECK: llvm.can.load.speculatively size must be a positive power of 2
+; CHECK-NEXT: %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 0)
+ %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 0)
+ ret i1 %res
+}
+
+define i1 @test_non_power_of_2(ptr %ptr) {
+; CHECK: llvm.can.load.speculatively size must be a positive power of 2
+; CHECK-NEXT: %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 3)
+ %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 3)
+ ret i1 %res
+}
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
new file mode 100644
index 0000000000000..def46d20799c5
--- /dev/null
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -0,0 +1,18 @@
+; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
+
+declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr)
+declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr)
+
+define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
+; CHECK: llvm.speculative.load type must have a power-of-2 size
+; CHECK-NEXT: %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr)
+ %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr)
+ ret <3 x i32> %res
+}
+
+define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
+; CHECK: llvm.speculative.load type must have a power-of-2 size
+; CHECK-NEXT: %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr)
+ %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr)
+ ret <vscale x 3 x i32> %res
+}
>From 0250eb419cbc7873eafa17ccbcbf1465d63d0cea Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Thu, 5 Feb 2026 19:40:18 +0000
Subject: [PATCH 2/9] !fixup address comments, thanks
---
llvm/docs/LangRef.rst | 16 ++++++++--------
llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp | 3 ---
llvm/lib/Target/AArch64/AArch64ISelLowering.cpp | 13 +++++++------
.../CodeGen/AArch64/can-load-speculatively.ll | 5 +----
.../CodeGen/X86/speculative-load-intrinsic.ll | 8 ++++----
5 files changed, 20 insertions(+), 25 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index ffd1c3e1f8912..152ba408c590b 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28052,10 +28052,10 @@ Overview:
"""""""""
The '``llvm.speculative.load``' intrinsic loads a value from memory. Unlike a
-regular load, the memory access may
-extend beyond the bounds of the allocated object, provided the pointer has been
-verified by :ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to
-ensure the access cannot fault.
+regular load, the memory access may extend beyond the bounds of the allocated
+object, provided the pointer has been verified by
+:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure the
+access cannot fault.
Arguments:
""""""""""
@@ -28073,12 +28073,12 @@ the access cannot fault.
For bytes that are within the bounds of the allocated object, the intrinsic
returns the stored value. For bytes that are beyond the bounds of the
-allocated object, the intrinsic returns ``undef`` for those bytes. At least the
+allocated object, the intrinsic returns ``poison`` for those bytes. At least the
first accessed byte must be within the bounds of an allocated object the pointer is
based on.
The behavior is undefined if this intrinsic is used to load from a pointer
-for which ``llvm.can.load.speculatively`` returns false.
+for which ``llvm.can.load.speculatively`` would return false.
.. _int_can_load_speculatively:
@@ -28114,8 +28114,8 @@ Semantics:
""""""""""
This intrinsic has **target-dependent** semantics. It returns ``true`` if
-loading ``num_bytes`` bytes from ``ptr`` is guaranteed not to trap,
-even if the memory is beyond the bounds of an allocated object. It returns
+``num_bytes`` bytes starting at ``ptr`` can be loaded speculatively, even
+if the memory is beyond the bounds of an allocated object. It returns
``false`` otherwise.
The specific conditions under which this intrinsic returns ``true`` are
diff --git a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
index be705e48eb08d..a9e6da96aef40 100644
--- a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
+++ b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
@@ -141,9 +141,6 @@ static bool lowerLoadRelative(Function &F) {
/// TargetLowering::emitCanLoadSpeculatively.
/// The default expansion returns false (conservative).
static bool lowerCanLoadSpeculatively(Function &F, const TargetMachine *TM) {
- if (F.use_empty())
- return false;
-
bool Changed = false;
for (Use &U : llvm::make_early_inc_range(F.uses())) {
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
index 0a3a90f1df949..d7a27f8b7926e 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
@@ -30748,6 +30748,10 @@ Value *AArch64TargetLowering::emitStoreConditional(IRBuilderBase &Builder,
Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
Value *Ptr,
Value *Size) const {
+ unsigned AS = cast<PointerType>(Ptr->getType())->getAddressSpace();
+ // Conservatively only allow speculation for address space 0.
+ if (AS != 0)
+ return nullptr;
// For power-of-2 sizes <= 16, emit alignment check: (ptr & (size - 1)) == 0.
// If the pointer is aligned to at least 'size' bytes, loading 'size' bytes
// cannot cross a page boundary, so it's safe to speculate.
@@ -30756,9 +30760,10 @@ Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
//
// The alignment check only works for power-of-2 sizes. For non-power-of-2
// sizes, we conservatively return false.
- const DataLayout &DL =
- Builder.GetInsertBlock()->getModule()->getDataLayout();
+ const DataLayout &DL = Builder.GetInsertBlock()->getModule()->getDataLayout();
+ unsigned PtrBits = DL.getPointerSizeInBits(AS);
+ Type *IntPtrTy = Builder.getIntNTy(PtrBits);
if (auto *CI = dyn_cast<ConstantInt>(Size)) {
uint64_t SizeVal = CI->getZExtValue();
assert(isPowerOf2_64(SizeVal) && "size must be power-of-two");
@@ -30767,8 +30772,6 @@ Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
return nullptr;
// Power-of-2 constant size <= 16: use fast alignment check.
- unsigned PtrBits = DL.getPointerSizeInBits();
- Type *IntPtrTy = Builder.getIntNTy(PtrBits);
Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
Value *Mask = ConstantInt::get(IntPtrTy, SizeVal - 1);
Value *Masked = Builder.CreateAnd(PtrInt, Mask);
@@ -30776,8 +30779,6 @@ Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
}
// Check power-of-2 size <= 16 and alignment.
- unsigned PtrBits = DL.getPointerSizeInBits();
- Type *IntPtrTy = Builder.getIntNTy(PtrBits);
Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
Value *SizeExt = Builder.CreateZExtOrTrunc(Size, IntPtrTy);
diff --git a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
index 7916f2e4d340f..e9af2eed8918a 100644
--- a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
+++ b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
@@ -38,10 +38,7 @@ define i1 @can_load_speculatively_64(ptr %ptr) {
; Test with address space
define i1 @can_load_speculatively_addrspace1(ptr addrspace(1) %ptr) {
; CHECK-LABEL: @can_load_speculatively_addrspace1(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr addrspace(1) [[PTR:%.*]] to i64
-; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
-; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
-; CHECK-NEXT: ret i1 [[TMP3]]
+; CHECK-NEXT: ret i1 false
;
%can_load = call i1 @llvm.can.load.speculatively.p1(ptr addrspace(1) %ptr, i64 16)
ret i1 %can_load
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index 4dd8375e9c42a..798ee2c03b1ab 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -37,12 +37,12 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
; SSE-LABEL: speculative_load_v2i64:
; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: movups (%rdi), %xmm0
; SSE-NEXT: retq
;
; AVX-LABEL: speculative_load_v2i64:
; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: vmovups (%rdi), %xmm0
; AVX-NEXT: retq
%load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
ret <2 x i64> %load
@@ -51,12 +51,12 @@ define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
define <4 x float> @speculative_load_v4f32(ptr %ptr) {
; SSE-LABEL: speculative_load_v4f32:
; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: movups (%rdi), %xmm0
; SSE-NEXT: retq
;
; AVX-LABEL: speculative_load_v4f32:
; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: vmovups (%rdi), %xmm0
; AVX-NEXT: retq
%load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
ret <4 x float> %load
>From daa1ab1c491a812b8119dadc63d42448a49605c8 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Mon, 9 Feb 2026 18:42:04 +0000
Subject: [PATCH 3/9] !fixup add tests with aligned pointers, clarify deref
guarantees.
---
llvm/docs/LangRef.rst | 9 ++--
.../CodeGen/AArch64/can-load-speculatively.ll | 45 +++++++++++++++++++
2 files changed, 50 insertions(+), 4 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 152ba408c590b..9b453ae9dc4d7 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28114,13 +28114,14 @@ Semantics:
""""""""""
This intrinsic has **target-dependent** semantics. It returns ``true`` if
-``num_bytes`` bytes starting at ``ptr`` can be loaded speculatively, even
-if the memory is beyond the bounds of an allocated object. It returns
-``false`` otherwise.
+``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for any non-negative
+integer ``I`` where the computed address does not wrap around the address
+space, can be loaded speculatively, even if the memory is beyond the bounds of
+an allocated object. It returns ``false`` otherwise.
The specific conditions under which this intrinsic returns ``true`` are
determined by the target. For example, a target may check whether the pointer
-alignment guarantees the load cannot cross a page boundary.
+alignment guarantees all such loads cannot cross a page boundary.
.. code-block:: llvm
diff --git a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
index e9af2eed8918a..b6679f22b0989 100644
--- a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
+++ b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
@@ -17,6 +17,51 @@ define i1 @can_load_speculatively_16(ptr %ptr) {
ret i1 %can_load
}
+
+define i1 @can_load_speculatively_8_ptr_aligned(ptr align 8 %ptr) {
+; CHECK-LABEL: @can_load_speculatively_8_ptr_aligned(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_16_ptr_aligned(ptr align 16 %ptr) {
+; CHECK-LABEL: @can_load_speculatively_16_ptr_aligned(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_16_ptr_aligned2(ptr align 16 %ptr) {
+; CHECK-LABEL: @can_load_speculatively_16_ptr_aligned2(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr align 16 %ptr, i64 16)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_32_ptr_aligned(ptr align 32 %ptr) {
+; CHECK-LABEL: @can_load_speculatively_32_ptr_aligned(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
; Size > 16 - returns false (may cross MTE tag granule boundary)
define i1 @can_load_speculatively_32(ptr %ptr) {
; CHECK-LABEL: @can_load_speculatively_32(
>From 7e8caceaa8fda67fdfc585003d86eeb1b92595ab Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Wed, 18 Feb 2026 09:09:09 +0000
Subject: [PATCH 4/9] !fixup clarify noalias interaction
---
llvm/docs/LangRef.rst | 27 +++++++++++++++++----------
1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 9b453ae9dc4d7..178029454d700 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28067,15 +28067,22 @@ Semantics:
""""""""""
The '``llvm.speculative.load``' intrinsic performs a load that may access
-memory beyond the allocated object. It must be used in combination with
-:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure
-the access cannot fault.
+memory beyond what is accessible through the pointer. It must be used in
+combination with :ref:`llvm.can.load.speculatively <int_can_load_speculatively>`
+to ensure the access can be performed speculatively.
-For bytes that are within the bounds of the allocated object, the intrinsic
-returns the stored value. For bytes that are beyond the bounds of the
-allocated object, the intrinsic returns ``poison`` for those bytes. At least the
-first accessed byte must be within the bounds of an allocated object the pointer is
-based on.
+A byte at ``ptr + i`` is *accessible through* ``ptr`` if both of the following
+hold:
+
+1. The byte lies within the bounds of an allocated object that ``ptr`` is
+ :ref:`based <pointeraliasing>` on.
+2. Accessing the byte through ``ptr`` does not violate any ``noalias``
+ constraints.
+
+For accessible bytes, the intrinsic returns the stored value. For inaccessible
+bytes, the intrinsic returns ``poison`` and the bytes are not considered accessed
+for the purpose of data races or ``noalias`` constraints. At least the first
+byte must be accessible; otherwise the behavior is undefined.
The behavior is undefined if this intrinsic is used to load from a pointer
for which ``llvm.can.load.speculatively`` would return false.
@@ -28114,8 +28121,8 @@ Semantics:
""""""""""
This intrinsic has **target-dependent** semantics. It returns ``true`` if
-``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for any non-negative
-integer ``I`` where the computed address does not wrap around the address
+``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for all non-negative
+integers ``I`` where the computed address does not wrap around the address
space, can be loaded speculatively, even if the memory is beyond the bounds of
an allocated object. It returns ``false`` otherwise.
>From d208544262bff40f8acfd5b9d1c6be0cd2fa93e0 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Mon, 23 Feb 2026 09:43:35 +0000
Subject: [PATCH 5/9] !fixup limit to executions that do not depend on
may-accessible bytes.
---
llvm/docs/LangRef.rst | 3 +++
1 file changed, 3 insertions(+)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 178029454d700..7d7ba26f5c224 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28084,6 +28084,9 @@ bytes, the intrinsic returns ``poison`` and the bytes are not considered accesse
for the purpose of data races or ``noalias`` constraints. At least the first
byte must be accessible; otherwise the behavior is undefined.
+The behavior is undefined if program execution depends on any byte in the
+result that may not be accessible.
+
The behavior is undefined if this intrinsic is used to load from a pointer
for which ``llvm.can.load.speculatively`` would return false.
>From d53e77e939fb7db894a201a5526056a45ed7f35f Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Wed, 25 Feb 2026 17:23:24 +0000
Subject: [PATCH 6/9] !fixup add implicit freeze
---
llvm/docs/LangRef.rst | 11 ++++-------
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 7d7ba26f5c224..fc99e19f31cdc 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28079,13 +28079,10 @@ hold:
2. Accessing the byte through ``ptr`` does not violate any ``noalias``
constraints.
-For accessible bytes, the intrinsic returns the stored value. For inaccessible
-bytes, the intrinsic returns ``poison`` and the bytes are not considered accessed
-for the purpose of data races or ``noalias`` constraints. At least the first
-byte must be accessible; otherwise the behavior is undefined.
-
-The behavior is undefined if program execution depends on any byte in the
-result that may not be accessible.
+For each byte of the result, if the corresponding byte is accessible through
+``ptr``, the result contains the value stored in memory at that byte. Otherwise,
+the corresponding byte of the result is ``freeze poison`` and the byte is not
+considered accessed for the purpose of data races or ``noalias`` constraints.
The behavior is undefined if this intrinsic is used to load from a pointer
for which ``llvm.can.load.speculatively`` would return false.
>From 72ad14372f08bc42b780191776d9c053084e86bd Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Fri, 27 Feb 2026 13:52:11 +0000
Subject: [PATCH 7/9] !fixup add read bytes argument
---
llvm/docs/LangRef.rst | 35 ++++++++--------
llvm/include/llvm/IR/Intrinsics.td | 7 ++--
.../SelectionDAG/SelectionDAGBuilder.cpp | 2 +
.../AArch64/speculative-load-intrinsic-sve.ll | 24 +++++------
.../AArch64/speculative-load-intrinsic.ll | 40 +++++++++----------
.../CodeGen/X86/speculative-load-intrinsic.ll | 36 ++++++++---------
llvm/test/Verifier/speculative-load.ll | 12 +++---
7 files changed, 78 insertions(+), 78 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index fc99e19f31cdc..9329f25c2e2f4 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28044,9 +28044,9 @@ This is an overloaded intrinsic.
::
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>)
- declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>)
- declare i64 @llvm.speculative.load.i64.p0(ptr <ptr>)
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i64 <num_read_bytes>)
+ declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i64 <num_read_bytes>)
+ declare i64 @llvm.speculative.load.i64.p0(ptr <ptr>, i64 <num_read_bytes>)
Overview:
"""""""""
@@ -28060,32 +28060,29 @@ access cannot fault.
Arguments:
""""""""""
-The argument is a pointer to the memory location to load from. The return type
-must have a power-of-2 size in bytes.
+The first argument is a pointer to the memory location to load from. The second
+argument is an ``i64`` specifying the number of bytes starting from ``ptr``
+that the intrinsic will read. The return type must have a power-of-2 size in
+bytes.
Semantics:
""""""""""
The '``llvm.speculative.load``' intrinsic performs a load that may access
-memory beyond what is accessible through the pointer. It must be used in
+memory beyond what is readable through the pointer. It must be used in
combination with :ref:`llvm.can.load.speculatively <int_can_load_speculatively>`
to ensure the access can be performed speculatively.
-A byte at ``ptr + i`` is *accessible through* ``ptr`` if both of the following
-hold:
+The intrinsic reads ``num_read_bytes`` bytes starting from ``ptr`` and returns
+the stored values for those bytes. The read bytes must lie within the bounds
+of an allocated object that ``ptr`` is :ref:`based <pointeraliasing>` on.
-1. The byte lies within the bounds of an allocated object that ``ptr`` is
- :ref:`based <pointeraliasing>` on.
-2. Accessing the byte through ``ptr`` does not violate any ``noalias``
- constraints.
-
-For each byte of the result, if the corresponding byte is accessible through
-``ptr``, the result contains the value stored in memory at that byte. Otherwise,
-the corresponding byte of the result is ``freeze poison`` and the byte is not
-considered accessed for the purpose of data races or ``noalias`` constraints.
+Bytes at offsets ``>= num_read_bytes`` (up to the size of the return type)
+are ``poison`` and are not considered accessed for the purpose of data races
+or ``noalias`` constraints.
The behavior is undefined if this intrinsic is used to load from a pointer
-for which ``llvm.can.load.speculatively`` would return false.
+for which ``llvm.can.load.speculatively`` returns false.
.. _int_can_load_speculatively:
@@ -28138,7 +28135,7 @@ alignment guarantees all such loads cannot cross a page boundary.
speculative_path:
; Safe to speculatively load from %ptr
- %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr)
+ %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 16)
...
safe_path:
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 39397dc72b4ab..d037e8521208f 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2670,11 +2670,12 @@ def int_masked_srem:
[IntrNoMem]>;
// Speculatively load a value from memory; lowers to a regular aligned load.
-// The loaded type must have a power-of-2 size.
+// The loaded type must have a power-of-2 size. The second argument specifies
+// the number of bytes the intrinsic will read from memory.
def int_speculative_load:
DefaultAttrsIntrinsic<[llvm_any_ty],
- [llvm_anyptr_ty],
- [IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
+ [llvm_anyptr_ty, llvm_i64_ty],
+ [IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
// The size can be a runtime value to support scalable vectors.
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index a3ffdb0ddf737..b1c225f4de07e 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5146,6 +5146,8 @@ void SelectionDAGBuilder::visitMaskedLoad(const CallInst &I, bool IsExpanding) {
void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
SDLoc sdl = getCurSDLoc();
Value *PtrOperand = I.getArgOperand(0);
+ // The second argument (num_read_bytes) is IR-level
+ // semantics only; it is not needed at codegen.
SDValue Ptr = getValue(PtrOperand);
const TargetLowering &TLI = DAG.getTargetLoweringInfo();
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
index 78a56f3539d11..4ae56cbca7ae9 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
@@ -9,7 +9,7 @@ define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr)
+ %load = call <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i64 0)
ret <vscale x 4 x i32> %load
}
@@ -18,7 +18,7 @@ define <vscale x 2 x i64> @speculative_load_nxv2i64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr %ptr)
+ %load = call <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr %ptr, i64 0)
ret <vscale x 2 x i64> %load
}
@@ -27,7 +27,7 @@ define <vscale x 8 x i16> @speculative_load_nxv8i16(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr align 8 %ptr)
+ %load = call <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr align 8 %ptr, i64 0)
ret <vscale x 8 x i16> %load
}
@@ -36,7 +36,7 @@ define <vscale x 16 x i8> @speculative_load_nxv16i8(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr %ptr)
+ %load = call <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr %ptr, i64 0)
ret <vscale x 16 x i8> %load
}
@@ -45,7 +45,7 @@ define <vscale x 4 x float> @speculative_load_nxv4f32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr align 4 %ptr)
+ %load = call <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr align 4 %ptr, i64 0)
ret <vscale x 4 x float> %load
}
@@ -54,13 +54,13 @@ define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr)
+ %load = call <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i64 0)
ret <vscale x 2 x double> %load
}
-declare <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr)
-declare <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr)
-declare <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr)
-declare <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr)
-declare <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr)
-declare <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr)
+declare <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr, i64)
+declare <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr, i64)
+declare <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr, i64)
+declare <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr, i64)
+declare <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr, i64)
+declare <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr, i64)
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
index 8f9a17414ff05..1c6b9d267ad46 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -9,7 +9,7 @@ define <4 x i32> @speculative_load_v4i32_align16(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr)
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 0)
ret <4 x i32> %load
}
@@ -18,7 +18,7 @@ define <4 x i32> @speculative_load_v4i32_align4(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 4 %ptr)
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 4 %ptr, i64 0)
ret <4 x i32> %load
}
@@ -27,7 +27,7 @@ define <4 x i32> @speculative_load_v4i32_noalign(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr)
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 0)
ret <4 x i32> %load
}
@@ -36,7 +36,7 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldp q0, q1, [x0]
; CHECK-NEXT: ret
- %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr)
+ %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 0)
ret <8 x i32> %load
}
@@ -45,7 +45,7 @@ define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
+ %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr, i64 0)
ret <2 x i64> %load
}
@@ -54,7 +54,7 @@ define <4 x float> @speculative_load_v4f32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
+ %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr, i64 0)
ret <4 x float> %load
}
@@ -63,15 +63,15 @@ define <2 x double> @speculative_load_v2f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr)
+ %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 0)
ret <2 x double> %load
}
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr)
-declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr)
-declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr)
-declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr)
-declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr)
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i64)
+declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr, i64)
+declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr, i64)
+declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr, i64)
+declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr, i64)
; Scalar type tests
@@ -80,7 +80,7 @@ define i32 @speculative_load_i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr w0, [x0]
; CHECK-NEXT: ret
- %load = call i32 @llvm.speculative.load.i32.p0(ptr align 4 %ptr)
+ %load = call i32 @llvm.speculative.load.i32.p0(ptr align 4 %ptr, i64 0)
ret i32 %load
}
@@ -89,7 +89,7 @@ define i64 @speculative_load_i64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr x0, [x0]
; CHECK-NEXT: ret
- %load = call i64 @llvm.speculative.load.i64.p0(ptr %ptr)
+ %load = call i64 @llvm.speculative.load.i64.p0(ptr %ptr, i64 0)
ret i64 %load
}
@@ -98,7 +98,7 @@ define float @speculative_load_f32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr s0, [x0]
; CHECK-NEXT: ret
- %load = call float @llvm.speculative.load.f32.p0(ptr %ptr)
+ %load = call float @llvm.speculative.load.f32.p0(ptr %ptr, i64 0)
ret float %load
}
@@ -107,11 +107,11 @@ define double @speculative_load_f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr d0, [x0]
; CHECK-NEXT: ret
- %load = call double @llvm.speculative.load.f64.p0(ptr align 8 %ptr)
+ %load = call double @llvm.speculative.load.f64.p0(ptr align 8 %ptr, i64 0)
ret double %load
}
-declare i32 @llvm.speculative.load.i32.p0(ptr)
-declare i64 @llvm.speculative.load.i64.p0(ptr)
-declare float @llvm.speculative.load.f32.p0(ptr)
-declare double @llvm.speculative.load.f64.p0(ptr)
+declare i32 @llvm.speculative.load.i32.p0(ptr, i64)
+declare i64 @llvm.speculative.load.i64.p0(ptr, i64)
+declare float @llvm.speculative.load.f32.p0(ptr, i64)
+declare double @llvm.speculative.load.f64.p0(ptr, i64)
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index 798ee2c03b1ab..5e12593f2d5a0 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -15,7 +15,7 @@ define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovaps (%rdi), %xmm0
; AVX-NEXT: retq
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr)
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 0)
ret <4 x i32> %load
}
@@ -30,7 +30,7 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovaps (%rdi), %ymm0
; AVX-NEXT: retq
- %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr)
+ %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 0)
ret <8 x i32> %load
}
@@ -44,7 +44,7 @@ define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovups (%rdi), %xmm0
; AVX-NEXT: retq
- %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
+ %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr, i64 0)
ret <2 x i64> %load
}
@@ -58,7 +58,7 @@ define <4 x float> @speculative_load_v4f32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovups (%rdi), %xmm0
; AVX-NEXT: retq
- %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
+ %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr, i64 0)
ret <4 x float> %load
}
@@ -72,15 +72,15 @@ define <2 x double> @speculative_load_v2f64(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovaps (%rdi), %xmm0
; AVX-NEXT: retq
- %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr)
+ %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 0)
ret <2 x double> %load
}
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr)
-declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr)
-declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr)
-declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr)
-declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr)
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i64)
+declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr, i64)
+declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr, i64)
+declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr, i64)
+declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr, i64)
; Scalar type tests
@@ -94,7 +94,7 @@ define i32 @speculative_load_i32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: movl (%rdi), %eax
; AVX-NEXT: retq
- %load = call i32 @llvm.speculative.load.i32.p0(ptr %ptr)
+ %load = call i32 @llvm.speculative.load.i32.p0(ptr %ptr, i64 0)
ret i32 %load
}
@@ -108,7 +108,7 @@ define i64 @speculative_load_i64(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: movq (%rdi), %rax
; AVX-NEXT: retq
- %load = call i64 @llvm.speculative.load.i64.p0(ptr align 8 %ptr)
+ %load = call i64 @llvm.speculative.load.i64.p0(ptr align 8 %ptr, i64 0)
ret i64 %load
}
@@ -122,7 +122,7 @@ define float @speculative_load_f32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
; AVX-NEXT: retq
- %load = call float @llvm.speculative.load.f32.p0(ptr align 4 %ptr)
+ %load = call float @llvm.speculative.load.f32.p0(ptr align 4 %ptr, i64 0)
ret float %load
}
@@ -136,11 +136,11 @@ define double @speculative_load_f64(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovsd {{.*#+}} xmm0 = mem[0],zero
; AVX-NEXT: retq
- %load = call double @llvm.speculative.load.f64.p0(ptr %ptr)
+ %load = call double @llvm.speculative.load.f64.p0(ptr %ptr, i64 0)
ret double %load
}
-declare i32 @llvm.speculative.load.i32.p0(ptr)
-declare i64 @llvm.speculative.load.i64.p0(ptr)
-declare float @llvm.speculative.load.f32.p0(ptr)
-declare double @llvm.speculative.load.f64.p0(ptr)
+declare i32 @llvm.speculative.load.i32.p0(ptr, i64)
+declare i64 @llvm.speculative.load.i64.p0(ptr, i64)
+declare float @llvm.speculative.load.f32.p0(ptr, i64)
+declare double @llvm.speculative.load.f64.p0(ptr, i64)
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
index def46d20799c5..c06881a5f50af 100644
--- a/llvm/test/Verifier/speculative-load.ll
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -1,18 +1,18 @@
; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
-declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr)
-declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr)
+declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, i64)
+declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, i64)
define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr)
- %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr)
+; CHECK-NEXT: %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
+ %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
ret <3 x i32> %res
}
define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr)
- %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr)
+; CHECK-NEXT: %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
+ %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
ret <vscale x 3 x i32> %res
}
>From b33cc2b7686f320aaf6b2b1342802676d93b765d Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Fri, 20 Mar 2026 11:58:45 +0000
Subject: [PATCH 8/9] !fixup make oracle function part of the arguments
---
llvm/docs/LangRef.rst | 48 +++--
llvm/include/llvm/IR/Intrinsics.td | 6 +-
.../SelectionDAG/SelectionDAGBuilder.cpp | 4 +-
llvm/lib/IR/Verifier.cpp | 44 ++++
.../AArch64/speculative-load-intrinsic-sve.ll | 66 ------
.../AArch64/speculative-load-intrinsic.ll | 123 ++++++------
.../CodeGen/X86/speculative-load-intrinsic.ll | 190 +++++++-----------
llvm/test/Verifier/speculative-load.ll | 74 ++++++-
8 files changed, 272 insertions(+), 283 deletions(-)
delete mode 100644 llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 9329f25c2e2f4..75881dad9194d 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28031,7 +28031,7 @@ Speculative Load Intrinsics
LLVM provides intrinsics for speculatively loading memory that may be
out-of-bounds. These intrinsics enable optimizations like early-exit loop
vectorization where the vectorized loop may read beyond the end of an array,
-provided the access is guaranteed to not trap by target-specific checks.
+provided the access is guaranteed be valid by target-specific checks.
.. _int_speculative_load:
@@ -28044,9 +28044,12 @@ This is an overloaded intrinsic.
::
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i64 <num_read_bytes>)
- declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i64 <num_read_bytes>)
- declare i64 @llvm.speculative.load.i64.p0(ptr <ptr>, i64 <num_read_bytes>)
+ ; Direct form: number of accessible bytes given as i64
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i64 <num_accessible_bytes>)
+ declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i64 <num_accessible_bytes>)
+
+ ; Oracle form: accessible bytes computed by calling oracle_fn(args...)
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, ptr <oracle_fn>, ...)
Overview:
"""""""""
@@ -28055,34 +28058,35 @@ The '``llvm.speculative.load``' intrinsic loads a value from memory. Unlike a
regular load, the memory access may extend beyond the bounds of the allocated
object, provided the pointer has been verified by
:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure the
-access cannot fault.
+access is valid.
Arguments:
""""""""""
-The first argument is a pointer to the memory location to load from. The second
-argument is an ``i64`` specifying the number of bytes starting from ``ptr``
-that the intrinsic will read. The return type must have a power-of-2 size in
-bytes.
+The first argument is a pointer to the memory location to load from. The return
+type must be a vector type with a power-of-2 size in bytes. The remaining
+arguments determine the *number of accessible bytes* starting from ``ptr``,
+denoted ``N`` below.
+
+In the **direct form**, the second argument is an ``i64`` specifying ``N``
+directly. In the **oracle form**, the second argument must be a direct
+reference to a function returning ``i64`` that may only read memory through its
+arguments (indirect function pointers are not permitted); the remaining
+arguments are forwarded to it, and the return value is ``N``.
Semantics:
""""""""""
-The '``llvm.speculative.load``' intrinsic performs a load that may access
-memory beyond what is readable through the pointer. It must be used in
-combination with :ref:`llvm.can.load.speculatively <int_can_load_speculatively>`
-to ensure the access can be performed speculatively.
-
-The intrinsic reads ``num_read_bytes`` bytes starting from ``ptr`` and returns
-the stored values for those bytes. The read bytes must lie within the bounds
-of an allocated object that ``ptr`` is :ref:`based <pointeraliasing>` on.
+The intrinsic reads ``N`` bytes starting from ``ptr`` and returns the stored
+values for those bytes. These bytes must lie within the bounds of an allocated
+object that ``ptr`` is :ref:`based <pointeraliasing>` on.
-Bytes at offsets ``>= num_read_bytes`` (up to the size of the return type)
-are ``poison`` and are not considered accessed for the purpose of data races
-or ``noalias`` constraints.
+Bytes at offsets in ``[N, size of the return type)`` are ``poison`` and are not
+considered accessed for the purposes of data races or ``noalias`` constraints.
-The behavior is undefined if this intrinsic is used to load from a pointer
-for which ``llvm.can.load.speculatively`` returns false.
+The behavior is undefined if the speculative load accesses memory that would
+fault (i.e., the oracle or ``llvm.can.load.speculatively`` would indicate the
+access is not safe).
.. _int_can_load_speculatively:
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index d037e8521208f..5de9078f1836c 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2670,11 +2670,9 @@ def int_masked_srem:
[IntrNoMem]>;
// Speculatively load a value from memory; lowers to a regular aligned load.
-// The loaded type must have a power-of-2 size. The second argument specifies
-// the number of bytes the intrinsic will read from memory.
def int_speculative_load:
- DefaultAttrsIntrinsic<[llvm_any_ty],
- [llvm_anyptr_ty, llvm_i64_ty],
+ DefaultAttrsIntrinsic<[llvm_anyvector_ty],
+ [llvm_anyptr_ty, llvm_vararg_ty],
[IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index b1c225f4de07e..c1c82f5548183 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5146,8 +5146,8 @@ void SelectionDAGBuilder::visitMaskedLoad(const CallInst &I, bool IsExpanding) {
void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
SDLoc sdl = getCurSDLoc();
Value *PtrOperand = I.getArgOperand(0);
- // The second argument (num_read_bytes) is IR-level
- // semantics only; it is not needed at codegen.
+ // The remaining arguments (num_accessible_bytes or oracle function + args)
+ // are IR-level semantics only; they are not needed at codegen.
SDValue Ptr = getValue(PtrOperand);
const TargetLowering &TLI = DAG.getTargetLoweringInfo();
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index bae181e098c49..08cc1a7353ca4 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -6820,6 +6820,50 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
// For scalable vectors, check the known minimum size is a power of 2.
Check(Size.getKnownMinValue() > 0 && isPowerOf2_64(Size.getKnownMinValue()),
"llvm.speculative.load type must have a power-of-2 size", &Call);
+
+ unsigned NumArgs = Call.arg_size();
+ Check(NumArgs >= 2, "llvm.speculative.load requires at least 2 arguments",
+ &Call);
+
+ Value *SecondArg = Call.getArgOperand(1);
+ if (SecondArg->getType()->isIntegerTy(64)) {
+ // Direct form: (ptr, i64 num_accessible_bytes)
+ Check(NumArgs == 2,
+ "llvm.speculative.load direct form requires exactly 2 arguments",
+ &Call);
+ if (auto *CI = dyn_cast<ConstantInt>(SecondArg)) {
+ Check(Size.isScalable() || CI->getZExtValue() <= Size.getFixedValue(),
+ "llvm.speculative.load num_accessible_bytes must not exceed "
+ "the result size in bytes",
+ &Call);
+ }
+ } else {
+ // Oracle form: (ptr, oracle_fn_ptr, args...)
+ auto *OracleFn = dyn_cast<Function>(SecondArg);
+ Check(OracleFn,
+ "llvm.speculative.load second argument must be i64 or a direct "
+ "reference to an oracle function",
+ &Call);
+
+ Check(OracleFn->onlyReadsMemory() && OracleFn->onlyAccessesArgMemory(),
+ "llvm.speculative.load oracle function must not have side effects "
+ "and may only read memory through its arguments",
+ &Call);
+
+ FunctionType *FTy = OracleFn->getFunctionType();
+ Check(FTy->getReturnType()->isIntegerTy(64),
+ "llvm.speculative.load oracle function must return i64", &Call);
+ unsigned NumOracleArgs = NumArgs - 2;
+ Check(FTy->isVarArg() ? NumOracleArgs >= FTy->getNumParams()
+ : NumOracleArgs == FTy->getNumParams(),
+ "llvm.speculative.load oracle function argument count mismatch",
+ &Call);
+ for (unsigned I = 0, E = FTy->getNumParams(); I < E; ++I) {
+ Check(FTy->getParamType(I) == Call.getArgOperand(I + 2)->getType(),
+ "llvm.speculative.load oracle function argument type mismatch",
+ &Call);
+ }
+ }
break;
}
case Intrinsic::can_load_speculatively: {
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
deleted file mode 100644
index 4ae56cbca7ae9..0000000000000
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
+++ /dev/null
@@ -1,66 +0,0 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64-unknown-linux-gnu -mattr=+sve < %s | FileCheck %s
-
-; Test that @llvm.speculative.load with scalable vectors is lowered to a
-; regular load in SelectionDAG.
-
-define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv4i32:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i64 0)
- ret <vscale x 4 x i32> %load
-}
-
-define <vscale x 2 x i64> @speculative_load_nxv2i64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv2i64:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr %ptr, i64 0)
- ret <vscale x 2 x i64> %load
-}
-
-define <vscale x 8 x i16> @speculative_load_nxv8i16(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv8i16:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr align 8 %ptr, i64 0)
- ret <vscale x 8 x i16> %load
-}
-
-define <vscale x 16 x i8> @speculative_load_nxv16i8(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv16i8:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr %ptr, i64 0)
- ret <vscale x 16 x i8> %load
-}
-
-define <vscale x 4 x float> @speculative_load_nxv4f32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv4f32:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr align 4 %ptr, i64 0)
- ret <vscale x 4 x float> %load
-}
-
-define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv2f64:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i64 0)
- ret <vscale x 2 x double> %load
-}
-
-declare <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr, i64)
-declare <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr, i64)
-declare <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr, i64)
-declare <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr, i64)
-declare <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr, i64)
-declare <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr, i64)
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
index 1c6b9d267ad46..2746b9ea87daa 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -1,117 +1,112 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64-unknown-linux-gnu < %s | FileCheck %s
+; RUN: llc -mtriple=aarch64-unknown-linux-gnu -mattr=+sve < %s | FileCheck %s
; Test that @llvm.speculative.load is lowered to a regular load
-; in SelectionDAG, respecting the alignment attribute.
+; in SelectionDAG for fixed vectors, scalable vectors, and scalars.
-define <4 x i32> @speculative_load_v4i32_align16(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v4i32_align16:
+; Fixed-width vector tests
+
+define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 0)
+ %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 12)
ret <4 x i32> %load
}
-define <4 x i32> @speculative_load_v4i32_align4(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v4i32_align4:
+define <2 x double> @speculative_load_v2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v2f64:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 4 %ptr, i64 0)
- ret <4 x i32> %load
+ %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 8)
+ ret <2 x double> %load
}
-define <4 x i32> @speculative_load_v4i32_noalign(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v4i32_noalign:
+; Scalable vector tests
+
+define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv4i32:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 0)
- ret <4 x i32> %load
+ %load = call <vscale x 4 x i32> (ptr, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i64 8)
+ ret <vscale x 4 x i32> %load
}
-define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v8i32:
+define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv2f64:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldp q0, q1, [x0]
+; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 0)
- ret <8 x i32> %load
+ %load = call <vscale x 2 x double> (ptr, ...) @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i64 4)
+ ret <vscale x 2 x double> %load
}
-define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v2i64:
+; Oracle form tests
+
+declare i64 @oracle(ptr, i64) memory(argmem: read)
+
+define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr, i64 0)
- ret <2 x i64> %load
+ %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
}
-define <4 x float> @speculative_load_v4f32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v4f32:
+define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v2i64_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr, i64 0)
- ret <4 x float> %load
+ %load = call <2 x i64> (ptr, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <2 x i64> %load
}
-define <2 x double> @speculative_load_v2f64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v2f64:
+define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i16_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 0)
- ret <2 x double> %load
+ %load = call <8 x i16> (ptr, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i16> %load
}
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i64)
-declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr, i64)
-declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr, i64)
-declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr, i64)
-declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr, i64)
-
-; Scalar type tests
-
-define i32 @speculative_load_i32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_i32:
+define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v16i8_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr w0, [x0]
+; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call i32 @llvm.speculative.load.i32.p0(ptr align 4 %ptr, i64 0)
- ret i32 %load
+ %load = call <16 x i8> (ptr, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <16 x i8> %load
}
-define i64 @speculative_load_i64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_i64:
+define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4f32_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr x0, [x0]
+; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call i64 @llvm.speculative.load.i64.p0(ptr %ptr, i64 0)
- ret i64 %load
+ %load = call <4 x float> (ptr, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x float> %load
}
-define float @speculative_load_f32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_f32:
+define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v2f64_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr s0, [x0]
+; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call float @llvm.speculative.load.f32.p0(ptr %ptr, i64 0)
- ret float %load
+ %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <2 x double> %load
}
-define double @speculative_load_f64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_f64:
+define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_nxv4i32_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr d0, [x0]
+; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call double @llvm.speculative.load.f64.p0(ptr align 8 %ptr, i64 0)
- ret double %load
+ %load = call <vscale x 4 x i32> (ptr, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <vscale x 4 x i32> %load
}
-
-declare i32 @llvm.speculative.load.i32.p0(ptr, i64)
-declare i64 @llvm.speculative.load.i64.p0(ptr, i64)
-declare float @llvm.speculative.load.f32.p0(ptr, i64)
-declare double @llvm.speculative.load.f64.p0(ptr, i64)
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index 5e12593f2d5a0..e387a17990012 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -1,146 +1,98 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+sse4.1 < %s | FileCheck %s --check-prefix=SSE
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2 < %s | FileCheck %s --check-prefix=AVX
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2 < %s | FileCheck %s
; Test that @llvm.speculative.load is lowered to a regular load
; in SelectionDAG.
define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
-; SSE-LABEL: speculative_load_v4i32:
-; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v4i32:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %xmm0
-; AVX-NEXT: retq
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 0)
+; CHECK-LABEL: speculative_load_v4i32:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 8)
ret <4 x i32> %load
}
define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
-; SSE-LABEL: speculative_load_v8i32:
-; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
-; SSE-NEXT: movaps 16(%rdi), %xmm1
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v8i32:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %ymm0
-; AVX-NEXT: retq
- %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 0)
+; CHECK-LABEL: speculative_load_v8i32:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <8 x i32> (ptr, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 24)
ret <8 x i32> %load
}
-define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
-; SSE-LABEL: speculative_load_v2i64:
-; SSE: # %bb.0:
-; SSE-NEXT: movups (%rdi), %xmm0
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v2i64:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovups (%rdi), %xmm0
-; AVX-NEXT: retq
- %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr, i64 0)
- ret <2 x i64> %load
-}
+; Oracle form tests
+declare i64 @oracle(ptr, i64) memory(argmem: read)
-define <4 x float> @speculative_load_v4f32(ptr %ptr) {
-; SSE-LABEL: speculative_load_v4f32:
-; SSE: # %bb.0:
-; SSE-NEXT: movups (%rdi), %xmm0
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v4f32:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovups (%rdi), %xmm0
-; AVX-NEXT: retq
- %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr, i64 0)
- ret <4 x float> %load
+define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
}
-define <2 x double> @speculative_load_v2f64(ptr %ptr) {
-; SSE-LABEL: speculative_load_v2f64:
-; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v2f64:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %xmm0
-; AVX-NEXT: retq
- %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 0)
- ret <2 x double> %load
+define <8 x i32> @speculative_load_v8i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i32_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <8 x i32> (ptr, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i32> %load
}
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i64)
-declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr, i64)
-declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr, i64)
-declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr, i64)
-declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr, i64)
-
-; Scalar type tests
+define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v2i64_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <2 x i64> (ptr, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <2 x i64> %load
+}
-define i32 @speculative_load_i32(ptr %ptr) {
-; SSE-LABEL: speculative_load_i32:
-; SSE: # %bb.0:
-; SSE-NEXT: movl (%rdi), %eax
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_i32:
-; AVX: # %bb.0:
-; AVX-NEXT: movl (%rdi), %eax
-; AVX-NEXT: retq
- %load = call i32 @llvm.speculative.load.i32.p0(ptr %ptr, i64 0)
- ret i32 %load
+define <4 x i64> @speculative_load_v4i64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i64_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <4 x i64> (ptr, ...) @llvm.speculative.load.v4i64.p0(ptr align 32 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i64> %load
}
-define i64 @speculative_load_i64(ptr %ptr) {
-; SSE-LABEL: speculative_load_i64:
-; SSE: # %bb.0:
-; SSE-NEXT: movq (%rdi), %rax
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_i64:
-; AVX: # %bb.0:
-; AVX-NEXT: movq (%rdi), %rax
-; AVX-NEXT: retq
- %load = call i64 @llvm.speculative.load.i64.p0(ptr align 8 %ptr, i64 0)
- ret i64 %load
+define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4f32_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x float> (ptr, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x float> %load
}
-define float @speculative_load_f32(ptr %ptr) {
-; SSE-LABEL: speculative_load_f32:
-; SSE: # %bb.0:
-; SSE-NEXT: movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_f32:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; AVX-NEXT: retq
- %load = call float @llvm.speculative.load.f32.p0(ptr align 4 %ptr, i64 0)
- ret float %load
+define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v2f64_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <2 x double> %load
}
-define double @speculative_load_f64(ptr %ptr) {
-; SSE-LABEL: speculative_load_f64:
-; SSE: # %bb.0:
-; SSE-NEXT: movsd {{.*#+}} xmm0 = mem[0],zero
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_f64:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovsd {{.*#+}} xmm0 = mem[0],zero
-; AVX-NEXT: retq
- %load = call double @llvm.speculative.load.f64.p0(ptr %ptr, i64 0)
- ret double %load
+define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i16_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <8 x i16> (ptr, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i16> %load
}
-declare i32 @llvm.speculative.load.i32.p0(ptr, i64)
-declare i64 @llvm.speculative.load.i64.p0(ptr, i64)
-declare float @llvm.speculative.load.f32.p0(ptr, i64)
-declare double @llvm.speculative.load.f64.p0(ptr, i64)
+define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v16i8_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <16 x i8> (ptr, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <16 x i8> %load
+}
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
index c06881a5f50af..34629849025dc 100644
--- a/llvm/test/Verifier/speculative-load.ll
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -1,18 +1,80 @@
; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
-declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, i64)
-declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, i64)
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, ...)
+declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, ...)
+declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, ...)
+
+declare i32 @bad_oracle_ret(ptr, i64) memory(argmem: read)
+declare i64 @good_oracle(ptr, i64) memory(argmem: read)
+declare i64 @oracle_i32_param(i32) memory(argmem: read)
+declare i64 @side_effecting_oracle(ptr, i64)
define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
- %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
+; CHECK-NEXT: %res = call <3 x i32> (ptr, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
+ %res = call <3 x i32> (ptr, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
ret <3 x i32> %res
}
define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
- %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
+; CHECK-NEXT: %res = call <vscale x 3 x i32> (ptr, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
+ %res = call <vscale x 3 x i32> (ptr, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
ret <vscale x 3 x i32> %res
}
+
+define <4 x i32> @test_oracle_wrong_return_type(ptr %ptr, i64 %n) {
+; CHECK: llvm.speculative.load oracle function must return i64
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_oracle_too_few_args(ptr %ptr) {
+; CHECK: llvm.speculative.load oracle function argument count mismatch
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_oracle_too_many_args(ptr %ptr, i64 %n) {
+; CHECK: llvm.speculative.load oracle function argument count mismatch
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_oracle_arg_type_mismatch(ptr %ptr) {
+; CHECK: llvm.speculative.load oracle function argument type mismatch
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @oracle_i32_param, i64 42)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @oracle_i32_param, i64 42)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_non_function_oracle(ptr %ptr, ptr %not_fn) {
+; CHECK: llvm.speculative.load second argument must be i64 or a direct reference to an oracle function
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr %not_fn)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr %not_fn)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_direct_form_extra_args(ptr %ptr) {
+; CHECK: llvm.speculative.load direct form requires exactly 2 arguments
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 8, i64 4)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 8, i64 4)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_oracle_side_effects(ptr %ptr, i64 %n) {
+; CHECK: llvm.speculative.load oracle function must not have side effects and may only read memory through its arguments
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_num_accessible_bytes_exceeds_size(ptr %ptr) {
+; CHECK: llvm.speculative.load num_accessible_bytes must not exceed the result size in bytes
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 32)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 32)
+ ret <4 x i32> %res
+}
>From 5cce6a408d75660e09533a3d6cb73d5948df28dc Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Mon, 30 Mar 2026 16:30:28 +0100
Subject: [PATCH 9/9] !fixup add from_end argument
---
llvm/docs/LangRef.rst | 41 +++++++-----
llvm/include/llvm/IR/Intrinsics.td | 5 +-
.../SelectionDAG/SelectionDAGBuilder.cpp | 2 +-
llvm/lib/IR/Verifier.cpp | 28 ++++----
.../AArch64/speculative-load-intrinsic.ll | 60 +++++++++++++----
.../CodeGen/X86/speculative-load-intrinsic.ll | 58 ++++++++++++++---
llvm/test/Verifier/speculative-load.ll | 65 ++++++++++---------
7 files changed, 178 insertions(+), 81 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 75881dad9194d..d577e8f587226 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28031,7 +28031,7 @@ Speculative Load Intrinsics
LLVM provides intrinsics for speculatively loading memory that may be
out-of-bounds. These intrinsics enable optimizations like early-exit loop
vectorization where the vectorized loop may read beyond the end of an array,
-provided the access is guaranteed be valid by target-specific checks.
+provided the access is guaranteed to be valid by target-specific checks.
.. _int_speculative_load:
@@ -28045,11 +28045,11 @@ This is an overloaded intrinsic.
::
; Direct form: number of accessible bytes given as i64
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i64 <num_accessible_bytes>)
- declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i64 <num_accessible_bytes>)
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
+ declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
; Oracle form: accessible bytes computed by calling oracle_fn(args...)
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, ptr <oracle_fn>, ...)
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i1 <from_end>, ptr <oracle_fn>, ...)
Overview:
"""""""""
@@ -28064,25 +28064,36 @@ Arguments:
""""""""""
The first argument is a pointer to the memory location to load from. The return
-type must be a vector type with a power-of-2 size in bytes. The remaining
-arguments determine the *number of accessible bytes* starting from ``ptr``,
+type must be a vector type with a power-of-2 size in bytes. The second argument
+is an ``i1`` constant flag ``from_end`` that specifies whether the ``N``
+accessible bytes are counted from the start or the end of the loaded values (see
+Semantics). The remaining arguments determine the *number of accessible bytes*,
denoted ``N`` below.
-In the **direct form**, the second argument is an ``i64`` specifying ``N``
-directly. In the **oracle form**, the second argument must be a direct
+In the **direct form**, the third argument is an ``i64`` specifying ``N``
+directly. In the **oracle form**, the third argument must be a direct
reference to a function returning ``i64`` that may only read memory through its
arguments (indirect function pointers are not permitted); the remaining
-arguments are forwarded to it, and the return value is ``N``.
+arguments are forwarded to it, and its return value is ``N``.
Semantics:
""""""""""
-The intrinsic reads ``N`` bytes starting from ``ptr`` and returns the stored
-values for those bytes. These bytes must lie within the bounds of an allocated
-object that ``ptr`` is :ref:`based <pointeraliasing>` on.
+Let ``S`` denote the size of the return type in bytes. The intrinsic performs
+a load of ``S`` bytes starting from ``ptr``.
-Bytes at offsets in ``[N, size of the return type)`` are ``poison`` and are not
-considered accessed for the purposes of data races or ``noalias`` constraints.
+When ``from_end`` is ``false``, the first ``N`` bytes (offsets ``[0, N)``)
+are the stored values read from memory. Bytes at offsets ``[N, S)`` are
+``poison``.
+
+When ``from_end`` is ``true``, the last ``N`` bytes (offsets ``[S - N, S)``)
+are the stored values read from memory. Bytes at offsets ``[0, S - N)`` are
+``poison``.
+
+In both cases, the ``N`` accessible bytes must lie within the bounds of an
+allocated object that ``ptr`` is :ref:`based <pointeraliasing>` on, and
+poison bytes are not considered accessed for the purposes of data races or
+``noalias`` constraints. The behavior is undefined if ``N`` exceeds ``S``.
The behavior is undefined if the speculative load accesses memory that would
fault (i.e., the oracle or ``llvm.can.load.speculatively`` would indicate the
@@ -28139,7 +28150,7 @@ alignment guarantees all such loads cannot cross a page boundary.
speculative_path:
; Safe to speculatively load from %ptr
- %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 16)
+ %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 16)
...
safe_path:
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 5de9078f1836c..6436edff25070 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2672,8 +2672,9 @@ def int_masked_srem:
// Speculatively load a value from memory; lowers to a regular aligned load.
def int_speculative_load:
DefaultAttrsIntrinsic<[llvm_anyvector_ty],
- [llvm_anyptr_ty, llvm_vararg_ty],
- [IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
+ [llvm_anyptr_ty, llvm_i1_ty, llvm_vararg_ty],
+ [IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>,
+ ImmArg<ArgIndex<1>>]>;
// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
// The size can be a runtime value to support scalable vectors.
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index c1c82f5548183..507f4e58ff041 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5163,7 +5163,7 @@ void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
MachineMemOperand::Flags MMOFlags = MachineMemOperand::MOLoad;
LocationSize LocSize = StoreSize.isScalable()
? LocationSize::beforeOrAfterPointer()
- : LocationSize::precise(StoreSize);
+ : LocationSize::upperBound(StoreSize);
MachineMemOperand *MMO = DAG.getMachineFunction().getMachineMemOperand(
MachinePointerInfo(PtrOperand), MMOFlags, LocSize, Alignment, AAInfo);
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 08cc1a7353ca4..37ec8eb9cf06b 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -6822,26 +6822,25 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
"llvm.speculative.load type must have a power-of-2 size", &Call);
unsigned NumArgs = Call.arg_size();
- Check(NumArgs >= 2, "llvm.speculative.load requires at least 2 arguments",
+ Check(NumArgs >= 3, "llvm.speculative.load requires at least 3 arguments",
&Call);
- Value *SecondArg = Call.getArgOperand(1);
- if (SecondArg->getType()->isIntegerTy(64)) {
- // Direct form: (ptr, i64 num_accessible_bytes)
- Check(NumArgs == 2,
- "llvm.speculative.load direct form requires exactly 2 arguments",
- &Call);
- if (auto *CI = dyn_cast<ConstantInt>(SecondArg)) {
+ Value *PayloadArg = Call.getArgOperand(2);
+ if (PayloadArg->getType()->isIntegerTy(64)) {
+ // Direct form: (ptr, i1 from_end, i64 num_accessible_bytes)
+ Check(NumArgs == 3,
+ "llvm.speculative.load direct form has too many arguments", &Call);
+ if (auto *CI = dyn_cast<ConstantInt>(PayloadArg)) {
Check(Size.isScalable() || CI->getZExtValue() <= Size.getFixedValue(),
"llvm.speculative.load num_accessible_bytes must not exceed "
"the result size in bytes",
&Call);
}
} else {
- // Oracle form: (ptr, oracle_fn_ptr, args...)
- auto *OracleFn = dyn_cast<Function>(SecondArg);
+ // Oracle form: (ptr, i1 from_end, oracle_fn_ptr, args...)
+ auto *OracleFn = dyn_cast<Function>(PayloadArg);
Check(OracleFn,
- "llvm.speculative.load second argument must be i64 or a direct "
+ "llvm.speculative.load third argument must be i64 or a direct "
"reference to an oracle function",
&Call);
@@ -6853,13 +6852,16 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
FunctionType *FTy = OracleFn->getFunctionType();
Check(FTy->getReturnType()->isIntegerTy(64),
"llvm.speculative.load oracle function must return i64", &Call);
- unsigned NumOracleArgs = NumArgs - 2;
+
+ unsigned OracleArgsStart = 3;
+ unsigned NumOracleArgs = NumArgs - OracleArgsStart;
Check(FTy->isVarArg() ? NumOracleArgs >= FTy->getNumParams()
: NumOracleArgs == FTy->getNumParams(),
"llvm.speculative.load oracle function argument count mismatch",
&Call);
for (unsigned I = 0, E = FTy->getNumParams(); I < E; ++I) {
- Check(FTy->getParamType(I) == Call.getArgOperand(I + 2)->getType(),
+ Check(FTy->getParamType(I) ==
+ Call.getArgOperand(I + OracleArgsStart)->getType(),
"llvm.speculative.load oracle function argument type mismatch",
&Call);
}
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
index 2746b9ea87daa..a547cee576864 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -11,7 +11,7 @@ define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 12)
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, i64 12)
ret <4 x i32> %load
}
@@ -20,7 +20,7 @@ define <2 x double> @speculative_load_v2f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 8)
+ %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, i64 8)
ret <2 x double> %load
}
@@ -31,7 +31,7 @@ define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> (ptr, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i64 8)
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 false, i64 8)
ret <vscale x 4 x i32> %load
}
@@ -40,7 +40,7 @@ define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 2 x double> (ptr, ...) @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i64 4)
+ %load = call <vscale x 2 x double> (ptr, i1, ...) @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i1 false, i64 4)
ret <vscale x 2 x double> %load
}
@@ -53,7 +53,7 @@ define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x i32> %load
}
@@ -62,7 +62,7 @@ define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x i64> (ptr, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <2 x i64> (ptr, i1, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <2 x i64> %load
}
@@ -71,7 +71,7 @@ define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <8 x i16> (ptr, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <8 x i16> (ptr, i1, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <8 x i16> %load
}
@@ -80,7 +80,7 @@ define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <16 x i8> (ptr, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <16 x i8> (ptr, i1, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <16 x i8> %load
}
@@ -89,7 +89,7 @@ define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x float> (ptr, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x float> (ptr, i1, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x float> %load
}
@@ -98,15 +98,53 @@ define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <2 x double> %load
}
+; from_end tests
+
+define <4 x i32> @speculative_load_v4i32_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_from_end:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, i64 12)
+ ret <4 x i32> %load
+}
+
+define <vscale x 4 x i32> @speculative_load_nxv4i32_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv4i32_from_end:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 true, i64 8)
+ ret <vscale x 4 x i32> %load
+}
+
+define <4 x i32> @speculative_load_v4i32_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle_from_end:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
+}
+
+define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_nxv4i32_oracle_from_end:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret <vscale x 4 x i32> %load
+}
+
define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle(ptr %ptr, i64 %n) {
; CHECK-LABEL: speculative_load_nxv4i32_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> (ptr, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <vscale x 4 x i32> %load
}
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index e387a17990012..fa345408940d1 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -9,7 +9,7 @@ define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 8)
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, i64 8)
ret <4 x i32> %load
}
@@ -18,7 +18,7 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %ymm0
; CHECK-NEXT: retq
- %load = call <8 x i32> (ptr, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 24)
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 false, i64 24)
ret <8 x i32> %load
}
@@ -30,7 +30,7 @@ define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x i32> %load
}
@@ -39,7 +39,7 @@ define <8 x i32> @speculative_load_v8i32_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %ymm0
; CHECK-NEXT: retq
- %load = call <8 x i32> (ptr, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <8 x i32> %load
}
@@ -48,7 +48,7 @@ define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <2 x i64> (ptr, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <2 x i64> (ptr, i1, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <2 x i64> %load
}
@@ -57,7 +57,7 @@ define <4 x i64> @speculative_load_v4i64_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %ymm0
; CHECK-NEXT: retq
- %load = call <4 x i64> (ptr, ...) @llvm.speculative.load.v4i64.p0(ptr align 32 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x i64> (ptr, i1, ...) @llvm.speculative.load.v4i64.p0(ptr align 32 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x i64> %load
}
@@ -66,7 +66,7 @@ define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <4 x float> (ptr, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x float> (ptr, i1, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x float> %load
}
@@ -75,7 +75,7 @@ define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <2 x double> %load
}
@@ -84,7 +84,7 @@ define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <8 x i16> (ptr, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <8 x i16> (ptr, i1, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <8 x i16> %load
}
@@ -93,6 +93,44 @@ define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <16 x i8> (ptr, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <16 x i8> (ptr, i1, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <16 x i8> %load
}
+
+; from_end tests
+
+define <4 x i32> @speculative_load_v4i32_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_from_end:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, i64 8)
+ ret <4 x i32> %load
+}
+
+define <8 x i32> @speculative_load_v8i32_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v8i32_from_end:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 true, i64 24)
+ ret <8 x i32> %load
+}
+
+define <4 x i32> @speculative_load_v4i32_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle_from_end:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
+}
+
+define <8 x i32> @speculative_load_v8i32_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i32_oracle_from_end:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i32> %load
+}
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
index 34629849025dc..7fd2d340679aa 100644
--- a/llvm/test/Verifier/speculative-load.ll
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -1,8 +1,8 @@
; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, ...)
-declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, ...)
-declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, ...)
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i1, ...)
+declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, i1, ...)
+declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, i1, ...)
declare i32 @bad_oracle_ret(ptr, i64) memory(argmem: read)
declare i64 @good_oracle(ptr, i64) memory(argmem: read)
@@ -11,70 +11,77 @@ declare i64 @side_effecting_oracle(ptr, i64)
define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <3 x i32> (ptr, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
- %res = call <3 x i32> (ptr, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
+; CHECK-NEXT: %res = call <3 x i32> (ptr, i1, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i1 false, i64 0)
+ %res = call <3 x i32> (ptr, i1, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i1 false, i64 0)
ret <3 x i32> %res
}
define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <vscale x 3 x i32> (ptr, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
- %res = call <vscale x 3 x i32> (ptr, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
+; CHECK-NEXT: %res = call <vscale x 3 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i1 false, i64 0)
+ %res = call <vscale x 3 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i1 false, i64 0)
ret <vscale x 3 x i32> %res
}
+define <4 x i32> @test_too_few_args(ptr %ptr) {
+; CHECK: llvm.speculative.load requires at least 3 arguments
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 true)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 true)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_direct_form_extra_args(ptr %ptr) {
+; CHECK: llvm.speculative.load direct form has too many arguments
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 8, i64 4)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 8, i64 4)
+ ret <4 x i32> %res
+}
+
define <4 x i32> @test_oracle_wrong_return_type(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function must return i64
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
ret <4 x i32> %res
}
define <4 x i32> @test_oracle_too_few_args(ptr %ptr) {
; CHECK: llvm.speculative.load oracle function argument count mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle)
ret <4 x i32> %res
}
define <4 x i32> @test_oracle_too_many_args(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function argument count mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
ret <4 x i32> %res
}
define <4 x i32> @test_oracle_arg_type_mismatch(ptr %ptr) {
; CHECK: llvm.speculative.load oracle function argument type mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @oracle_i32_param, i64 42)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @oracle_i32_param, i64 42)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @oracle_i32_param, i64 42)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @oracle_i32_param, i64 42)
ret <4 x i32> %res
}
define <4 x i32> @test_non_function_oracle(ptr %ptr, ptr %not_fn) {
-; CHECK: llvm.speculative.load second argument must be i64 or a direct reference to an oracle function
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr %not_fn)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr %not_fn)
- ret <4 x i32> %res
-}
-
-define <4 x i32> @test_direct_form_extra_args(ptr %ptr) {
-; CHECK: llvm.speculative.load direct form requires exactly 2 arguments
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 8, i64 4)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 8, i64 4)
+; CHECK: llvm.speculative.load third argument must be i64 or a direct reference to an oracle function
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr %not_fn)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr %not_fn)
ret <4 x i32> %res
}
define <4 x i32> @test_oracle_side_effects(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function must not have side effects and may only read memory through its arguments
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
ret <4 x i32> %res
}
define <4 x i32> @test_num_accessible_bytes_exceeds_size(ptr %ptr) {
; CHECK: llvm.speculative.load num_accessible_bytes must not exceed the result size in bytes
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 32)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 32)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 32)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 32)
ret <4 x i32> %res
}
More information about the llvm-commits
mailing list