[llvm] Provide intrinsics for speculative loads (PR #179642)
Florian Hahn via llvm-commits
llvm-commits at lists.llvm.org
Mon Jun 1 04:16:07 PDT 2026
https://github.com/fhahn updated https://github.com/llvm/llvm-project/pull/179642
>From 18e5ef0e7ed20b53ed8974c73ad14ba2d26ced77 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Wed, 4 Feb 2026 11:06:38 +0000
Subject: [PATCH 01/14] Add speculative load intrinsics
Introduce two new intrinsics to enable vectorization of loops with early
exits that have potentially faulting loads.
This has previously been discussed in
https://github.com/llvm/llvm-project/pull/120603 and is similar to
@nikic's https://hackmd.io/@nikic/S1O4QWYZkx, with the major difference
being that there is no `%defined_size` argument and instead the load
returns the stored values for the bytes within bounds and undef
otherwise. I don't think we can easily compute the defined size because
it may depend on the loaded values (i.e. at what lane the early exit has
been taken).
1. `@llvm.speculative.load` (name subject to change) - perform a load that
may access memory beyond the allocated object. It must be used in
combination with `@llvm.can.load.speculatively` to ensure the load is
guaranteed to not trap.
2. `@llvm.can.load.speculatively` - Returns true if it's safe to speculatively
load a given number of bytes from a pointer. The semantics are
target-dependent. On some targets, this may check that the access
does not cross page boundaries, or stricter checks for example on
AArch64 with MTE, which limits the access size to 16 bytes.
`@llvm.speculative.load` is lowered to a regular load in SelectionDAG
without MODereferenceable. I am not sure if we need to be more careful
than this, i.e. if we could still reason about SelectionDAG loads to
infer dereferencability for the pointer.
`@llvm.can.load.speculatively` is lowered to regular IR in PreISel
lowering, using a target-lowering hook. By default, it conservatively
expands to false.
These intrinsics should allow the loop vectorizer to vectorize early-exit
loops with potentially non-dereferenceable loads.
---
llvm/docs/LangRef.rst | 113 ++++++++++++++
llvm/include/llvm/CodeGen/TargetLowering.h | 13 ++
llvm/include/llvm/IR/Intrinsics.td | 14 ++
llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp | 39 +++++
.../SelectionDAG/SelectionDAGBuilder.cpp | 30 ++++
.../SelectionDAG/SelectionDAGBuilder.h | 1 +
llvm/lib/IR/Verifier.cpp | 18 +++
.../Target/AArch64/AArch64ISelLowering.cpp | 49 ++++++
llvm/lib/Target/AArch64/AArch64ISelLowering.h | 2 +
.../CodeGen/AArch64/can-load-speculatively.ll | 78 ++++++++++
.../AArch64/speculative-load-intrinsic-sve.ll | 66 ++++++++
.../AArch64/speculative-load-intrinsic.ll | 117 ++++++++++++++
.../CodeGen/X86/can-load-speculatively.ll | 32 ++++
.../CodeGen/X86/speculative-load-intrinsic.ll | 146 ++++++++++++++++++
llvm/test/Verifier/can-load-speculatively.ll | 19 +++
llvm/test/Verifier/speculative-load.ll | 18 +++
16 files changed, 755 insertions(+)
create mode 100644 llvm/test/CodeGen/AArch64/can-load-speculatively.ll
create mode 100644 llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
create mode 100644 llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
create mode 100644 llvm/test/CodeGen/X86/can-load-speculatively.ll
create mode 100644 llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
create mode 100644 llvm/test/Verifier/can-load-speculatively.ll
create mode 100644 llvm/test/Verifier/speculative-load.ll
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 00cfcb20da74c..56f62cac1b6df 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28328,6 +28328,119 @@ Semantics:
Follows the same semantics as :ref:`srem <i_srem>` with the exception that disabled lanes cannot produce undefined behaviour and always result in poison.
+Speculative Load Intrinsics
+---------------------------
+
+LLVM provides intrinsics for speculatively loading memory that may be
+out-of-bounds. These intrinsics enable optimizations like early-exit loop
+vectorization where the vectorized loop may read beyond the end of an array,
+provided the access is guaranteed to not trap by target-specific checks.
+
+.. _int_speculative_load:
+
+'``llvm.speculative.load``' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic.
+
+::
+
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>)
+ declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>)
+ declare i64 @llvm.speculative.load.i64.p0(ptr <ptr>)
+
+Overview:
+"""""""""
+
+The '``llvm.speculative.load``' intrinsic loads a value from memory. Unlike a
+regular load, the memory access may
+extend beyond the bounds of the allocated object, provided the pointer has been
+verified by :ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to
+ensure the access cannot fault.
+
+Arguments:
+""""""""""
+
+The argument is a pointer to the memory location to load from. The return type
+must have a power-of-2 size in bytes.
+
+Semantics:
+""""""""""
+
+The '``llvm.speculative.load``' intrinsic performs a load that may access
+memory beyond the allocated object. It must be used in combination with
+:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure
+the access cannot fault.
+
+For bytes that are within the bounds of the allocated object, the intrinsic
+returns the stored value. For bytes that are beyond the bounds of the
+allocated object, the intrinsic returns ``undef`` for those bytes. At least the
+first accessed byte must be within the bounds of an allocated object the pointer is
+based on.
+
+The behavior is undefined if this intrinsic is used to load from a pointer
+for which ``llvm.can.load.speculatively`` returns false.
+
+.. _int_can_load_speculatively:
+
+'``llvm.can.load.speculatively``' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic.
+
+::
+
+ declare i1 @llvm.can.load.speculatively.p0(ptr <ptr>, i64 <num_bytes>)
+ declare i1 @llvm.can.load.speculatively.p1(ptr addrspace(1) <ptr>, i64 <num_bytes>)
+
+Overview:
+"""""""""
+
+The '``llvm.can.load.speculatively``' intrinsic returns true if it is safe
+to speculatively load ``num_bytes`` bytes starting from ``ptr``,
+even if the memory may be beyond the bounds of an allocated object.
+
+Arguments:
+""""""""""
+
+The first argument is a pointer to the memory location.
+
+The second argument is an i64 specifying the size in bytes of the load.
+The size must be a positive power of 2. If the size is not a power-of-2, the
+result is ``poison``.
+
+Semantics:
+""""""""""
+
+This intrinsic has **target-dependent** semantics. It returns ``true`` if
+loading ``num_bytes`` bytes from ``ptr`` is guaranteed not to trap,
+even if the memory is beyond the bounds of an allocated object. It returns
+``false`` otherwise.
+
+The specific conditions under which this intrinsic returns ``true`` are
+determined by the target. For example, a target may check whether the pointer
+alignment guarantees the load cannot cross a page boundary.
+
+.. code-block:: llvm
+
+ ; Check if we can safely load 16 bytes from %ptr
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ br i1 %can_load, label %speculative_path, label %safe_path
+
+ speculative_path:
+ ; Safe to speculatively load from %ptr
+ %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr)
+ ...
+
+ safe_path:
+ ; Fall back to masked load or scalar operations
+ ...
+
+
Memory Use Markers
------------------
diff --git a/llvm/include/llvm/CodeGen/TargetLowering.h b/llvm/include/llvm/CodeGen/TargetLowering.h
index 82c47cce0f522..d6a4f454d4b64 100644
--- a/llvm/include/llvm/CodeGen/TargetLowering.h
+++ b/llvm/include/llvm/CodeGen/TargetLowering.h
@@ -2325,6 +2325,19 @@ class LLVM_ABI TargetLoweringBase {
llvm_unreachable("Store conditional unimplemented on this target");
}
+ /// Emit code to check if a speculative load of the given size from Ptr is
+ /// safe. Returns a Value* representing the check result (i1), or nullptr
+ /// to use the default lowering (which returns false). Targets can override
+ /// to provide their own safety check (e.g., alignment-based page boundary
+ /// check).
+ /// \param Builder IRBuilder positioned at the intrinsic call site
+ /// \param Ptr the pointer operand
+ /// \param Size the size in bytes (constant or runtime value for scalable)
+ virtual Value *emitCanLoadSpeculatively(IRBuilderBase &Builder, Value *Ptr,
+ Value *Size) const {
+ return nullptr;
+ }
+
/// Perform a masked atomicrmw using a target-specific intrinsic. This
/// represents the core LL/SC loop which will be lowered at a late stage by
/// the backend. The target-specific intrinsic returns the loaded value and
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 5de1d6592a301..5855edc8117f2 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2714,6 +2714,20 @@ def int_masked_srem:
LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>],
[IntrNoMem]>;
+// Speculatively load a value from memory; lowers to a regular aligned load.
+// The loaded type must have a power-of-2 size.
+def int_speculative_load:
+ DefaultAttrsIntrinsic<[llvm_any_ty],
+ [llvm_anyptr_ty],
+ [IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
+
+// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
+// The size can be a runtime value to support scalable vectors.
+def int_can_load_speculatively:
+ DefaultAttrsIntrinsic<[llvm_i1_ty],
+ [llvm_anyptr_ty, llvm_i64_ty],
+ [IntrNoMem, IntrSpeculatable, IntrWillReturn]>;
+
// Test whether a pointer is associated with a type metadata identifier.
def int_type_test : DefaultAttrsIntrinsic<[llvm_i1_ty],
[llvm_ptr_ty, llvm_metadata_ty],
diff --git a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
index 2fa92b61bb297..06a65e7375f00 100644
--- a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
+++ b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
@@ -136,6 +136,42 @@ static bool lowerLoadRelative(Function &F) {
return Changed;
}
+/// Lower @llvm.can.load.speculatively using target-specific expansion.
+/// Each target provides its own expansion via
+/// TargetLowering::emitCanLoadSpeculatively.
+/// The default expansion returns false (conservative).
+static bool lowerCanLoadSpeculatively(Function &F, const TargetMachine *TM) {
+ if (F.use_empty())
+ return false;
+
+ bool Changed = false;
+
+ for (Use &U : llvm::make_early_inc_range(F.uses())) {
+ auto *CI = dyn_cast<CallInst>(U.getUser());
+ if (!CI || CI->getCalledOperand() != &F)
+ continue;
+
+ Function *ParentFunc = CI->getFunction();
+ const TargetLowering *TLI =
+ TM->getSubtargetImpl(*ParentFunc)->getTargetLowering();
+
+ IRBuilder<> Builder(CI);
+ Value *Ptr = CI->getArgOperand(0);
+ Value *Size = CI->getArgOperand(1);
+
+ // Ask target for expansion; nullptr means use default (return false)
+ Value *Result = TLI->emitCanLoadSpeculatively(Builder, Ptr, Size);
+ if (!Result)
+ Result = Builder.getFalse();
+
+ CI->replaceAllUsesWith(Result);
+ CI->eraseFromParent();
+ Changed = true;
+ }
+
+ return Changed;
+}
+
// ObjCARC has knowledge about whether an obj-c runtime function needs to be
// always tail-called or never tail-called.
static CallInst::TailCallKind getOverridingTailCallKind(const Function &F) {
@@ -674,6 +710,9 @@ bool PreISelIntrinsicLowering::lowerIntrinsics(Module &M) const {
case Intrinsic::load_relative:
Changed |= lowerLoadRelative(F);
break;
+ case Intrinsic::can_load_speculatively:
+ Changed |= lowerCanLoadSpeculatively(F, TM);
+ break;
case Intrinsic::is_constant:
case Intrinsic::objectsize:
Changed |= forEachCall(F, [&](CallInst *CI) {
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index eca5bb1598ae0..8879e77084f5c 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5144,6 +5144,33 @@ void SelectionDAGBuilder::visitMaskedLoad(const CallInst &I, bool IsExpanding) {
setValue(&I, Res);
}
+void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
+ SDLoc sdl = getCurSDLoc();
+ Value *PtrOperand = I.getArgOperand(0);
+ SDValue Ptr = getValue(PtrOperand);
+
+ const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+ EVT VT = TLI.getValueType(DAG.getDataLayout(), I.getType());
+ Align Alignment = I.getParamAlign(0).valueOrOne();
+ AAMDNodes AAInfo = I.getAAMetadata();
+ TypeSize StoreSize = VT.getStoreSize();
+
+ SDValue InChain = DAG.getRoot();
+
+ // Use MOLoad but NOT MODereferenceable - the memory may not be
+ // fully dereferenceable.
+ MachineMemOperand::Flags MMOFlags = MachineMemOperand::MOLoad;
+ LocationSize LocSize = StoreSize.isScalable()
+ ? LocationSize::beforeOrAfterPointer()
+ : LocationSize::precise(StoreSize);
+ MachineMemOperand *MMO = DAG.getMachineFunction().getMachineMemOperand(
+ MachinePointerInfo(PtrOperand), MMOFlags, LocSize, Alignment, AAInfo);
+
+ SDValue Load = DAG.getLoad(VT, sdl, InChain, Ptr, MMO);
+ PendingLoads.push_back(Load.getValue(1));
+ setValue(&I, Load);
+}
+
void SelectionDAGBuilder::visitMaskedGather(const CallInst &I) {
SDLoc sdl = getCurSDLoc();
@@ -6934,6 +6961,9 @@ void SelectionDAGBuilder::visitIntrinsicCall(const CallInst &I,
case Intrinsic::masked_compressstore:
visitMaskedStore(I, true /* IsCompressing */);
return;
+ case Intrinsic::speculative_load:
+ visitSpeculativeLoad(I);
+ return;
case Intrinsic::powi:
setValue(&I, ExpandPowI(sdl, getValue(I.getArgOperand(0)),
getValue(I.getArgOperand(1)), DAG));
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
index 21aac333a73cd..1f5405acd7045 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
@@ -620,6 +620,7 @@ class SelectionDAGBuilder {
void visitStore(const StoreInst &I);
void visitMaskedLoad(const CallInst &I, bool IsExpanding = false);
void visitMaskedStore(const CallInst &I, bool IsCompressing = false);
+ void visitSpeculativeLoad(const CallInst &I);
void visitMaskedGather(const CallInst &I);
void visitMaskedScatter(const CallInst &I);
void visitAtomicCmpXchg(const AtomicCmpXchgInst &I);
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index e58209a2499b4..8df3d8277fbaf 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -6941,6 +6941,24 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
&Call);
break;
}
+ case Intrinsic::speculative_load: {
+ Type *LoadTy = Call.getType();
+ TypeSize Size = DL.getTypeStoreSize(LoadTy);
+ // For scalable vectors, check the known minimum size is a power of 2.
+ Check(Size.getKnownMinValue() > 0 && isPowerOf2_64(Size.getKnownMinValue()),
+ "llvm.speculative.load type must have a power-of-2 size", &Call);
+ break;
+ }
+ case Intrinsic::can_load_speculatively: {
+ // If size is a constant, verify it's a positive power of 2.
+ if (auto *SizeCI = dyn_cast<ConstantInt>(Call.getArgOperand(1))) {
+ uint64_t Size = SizeCI->getZExtValue();
+ Check(Size > 0 && isPowerOf2_64(Size),
+ "llvm.can.load.speculatively size must be a positive power of 2",
+ &Call);
+ }
+ break;
+ }
case Intrinsic::vector_insert: {
Value *Vec = Call.getArgOperand(0);
Value *SubVec = Call.getArgOperand(1);
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
index 0992f329e1d6c..29b7a24c2d481 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
@@ -31386,6 +31386,55 @@ Value *AArch64TargetLowering::emitStoreConditional(IRBuilderBase &Builder,
return CI;
}
+Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
+ Value *Ptr,
+ Value *Size) const {
+ // For power-of-2 sizes <= 16, emit alignment check: (ptr & (size - 1)) == 0.
+ // If the pointer is aligned to at least 'size' bytes, loading 'size' bytes
+ // cannot cross a page boundary, so it's safe to speculate.
+ // The 16-byte limit ensures correctness with MTE (memory tagging), since
+ // MTE uses 16-byte tag granules.
+ //
+ // The alignment check only works for power-of-2 sizes. For non-power-of-2
+ // sizes, we conservatively return false.
+ const DataLayout &DL =
+ Builder.GetInsertBlock()->getModule()->getDataLayout();
+
+ if (auto *CI = dyn_cast<ConstantInt>(Size)) {
+ uint64_t SizeVal = CI->getZExtValue();
+ assert(isPowerOf2_64(SizeVal) && "size must be power-of-two");
+ // For constant sizes > 16, return nullptr (default false).
+ if (SizeVal > 16)
+ return nullptr;
+
+ // Power-of-2 constant size <= 16: use fast alignment check.
+ unsigned PtrBits = DL.getPointerSizeInBits();
+ Type *IntPtrTy = Builder.getIntNTy(PtrBits);
+ Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
+ Value *Mask = ConstantInt::get(IntPtrTy, SizeVal - 1);
+ Value *Masked = Builder.CreateAnd(PtrInt, Mask);
+ return Builder.CreateICmpEQ(Masked, ConstantInt::get(IntPtrTy, 0));
+ }
+
+ // Check power-of-2 size <= 16 and alignment.
+ unsigned PtrBits = DL.getPointerSizeInBits();
+ Type *IntPtrTy = Builder.getIntNTy(PtrBits);
+ Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
+ Value *SizeExt = Builder.CreateZExtOrTrunc(Size, IntPtrTy);
+
+ Value *SizeLE16 =
+ Builder.CreateICmpULE(SizeExt, ConstantInt::get(IntPtrTy, 16));
+
+ // alignment check: (ptr & (size - 1)) == 0
+ Value *SizeMinusOne =
+ Builder.CreateSub(SizeExt, ConstantInt::get(IntPtrTy, 1));
+ Value *Masked = Builder.CreateAnd(PtrInt, SizeMinusOne);
+ Value *AlignCheck =
+ Builder.CreateICmpEQ(Masked, ConstantInt::get(IntPtrTy, 0));
+
+ return Builder.CreateAnd(SizeLE16, AlignCheck);
+}
+
bool AArch64TargetLowering::functionArgumentNeedsConsecutiveRegisters(
Type *Ty, CallingConv::ID CallConv, bool isVarArg,
const DataLayout &DL) const {
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.h b/llvm/lib/Target/AArch64/AArch64ISelLowering.h
index 3a93d9a3c0d1e..ced4cba15ba90 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.h
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.h
@@ -347,6 +347,8 @@ class AArch64TargetLowering : public TargetLowering {
AtomicOrdering Ord) const override;
Value *emitStoreConditional(IRBuilderBase &Builder, Value *Val, Value *Addr,
AtomicOrdering Ord) const override;
+ Value *emitCanLoadSpeculatively(IRBuilderBase &Builder, Value *Ptr,
+ Value *Size) const override;
void emitAtomicCmpXchgNoStoreLLBalance(IRBuilderBase &Builder) const override;
diff --git a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
new file mode 100644
index 0000000000000..7916f2e4d340f
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
@@ -0,0 +1,78 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -mtriple=aarch64-unknown-linux-gnu -passes=pre-isel-intrinsic-lowering -S < %s | FileCheck %s
+
+; Test that @llvm.can.load.speculatively is lowered to an alignment check
+; for power-of-2 sizes <= 16 bytes on AArch64, and returns false for larger sizes.
+; The 16-byte limit ensures correctness with MTE (memory tagging).
+; Note: non-power-of-2 constant sizes are rejected by the verifier.
+
+define i1 @can_load_speculatively_16(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_16(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
+; Size > 16 - returns false (may cross MTE tag granule boundary)
+define i1 @can_load_speculatively_32(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_32(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 32)
+ ret i1 %can_load
+}
+
+; Size > 16 - returns false (may cross MTE tag granule boundary)
+define i1 @can_load_speculatively_64(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_64(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 64)
+ ret i1 %can_load
+}
+
+; Test with address space
+define i1 @can_load_speculatively_addrspace1(ptr addrspace(1) %ptr) {
+; CHECK-LABEL: @can_load_speculatively_addrspace1(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr addrspace(1) [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p1(ptr addrspace(1) %ptr, i64 16)
+ ret i1 %can_load
+}
+
+; Test size 8 (within limit, power-of-2)
+define i1 @can_load_speculatively_8(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_8(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 7
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 8)
+ ret i1 %can_load
+}
+
+; Test with runtime size - checks size <= 16 and alignment
+define i1 @can_load_speculatively_runtime(ptr %ptr, i64 %size) {
+; CHECK-LABEL: @can_load_speculatively_runtime(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = icmp ule i64 [[SIZE:%.*]], 16
+; CHECK-NEXT: [[TMP3:%.*]] = sub i64 [[SIZE]], 1
+; CHECK-NEXT: [[TMP4:%.*]] = and i64 [[TMP1]], [[TMP3]]
+; CHECK-NEXT: [[TMP5:%.*]] = icmp eq i64 [[TMP4]], 0
+; CHECK-NEXT: [[TMP6:%.*]] = and i1 [[TMP2]], [[TMP5]]
+; CHECK-NEXT: ret i1 [[TMP6]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 %size)
+ ret i1 %can_load
+}
+
+declare i1 @llvm.can.load.speculatively.p0(ptr, i64)
+declare i1 @llvm.can.load.speculatively.p1(ptr addrspace(1), i64)
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
new file mode 100644
index 0000000000000..78a56f3539d11
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
@@ -0,0 +1,66 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-unknown-linux-gnu -mattr=+sve < %s | FileCheck %s
+
+; Test that @llvm.speculative.load with scalable vectors is lowered to a
+; regular load in SelectionDAG.
+
+define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv4i32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr)
+ ret <vscale x 4 x i32> %load
+}
+
+define <vscale x 2 x i64> @speculative_load_nxv2i64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv2i64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr %ptr)
+ ret <vscale x 2 x i64> %load
+}
+
+define <vscale x 8 x i16> @speculative_load_nxv8i16(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv8i16:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr align 8 %ptr)
+ ret <vscale x 8 x i16> %load
+}
+
+define <vscale x 16 x i8> @speculative_load_nxv16i8(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv16i8:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr %ptr)
+ ret <vscale x 16 x i8> %load
+}
+
+define <vscale x 4 x float> @speculative_load_nxv4f32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv4f32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr align 4 %ptr)
+ ret <vscale x 4 x float> %load
+}
+
+define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv2f64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr)
+ ret <vscale x 2 x double> %load
+}
+
+declare <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr)
+declare <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr)
+declare <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr)
+declare <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr)
+declare <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr)
+declare <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr)
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
new file mode 100644
index 0000000000000..8f9a17414ff05
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -0,0 +1,117 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-unknown-linux-gnu < %s | FileCheck %s
+
+; Test that @llvm.speculative.load is lowered to a regular load
+; in SelectionDAG, respecting the alignment attribute.
+
+define <4 x i32> @speculative_load_v4i32_align16(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_align16:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr)
+ ret <4 x i32> %load
+}
+
+define <4 x i32> @speculative_load_v4i32_align4(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_align4:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 4 %ptr)
+ ret <4 x i32> %load
+}
+
+define <4 x i32> @speculative_load_v4i32_noalign(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_noalign:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr)
+ ret <4 x i32> %load
+}
+
+define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v8i32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldp q0, q1, [x0]
+; CHECK-NEXT: ret
+ %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr)
+ ret <8 x i32> %load
+}
+
+define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v2i64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
+ ret <2 x i64> %load
+}
+
+define <4 x float> @speculative_load_v4f32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4f32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
+ ret <4 x float> %load
+}
+
+define <2 x double> @speculative_load_v2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v2f64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr)
+ ret <2 x double> %load
+}
+
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr)
+declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr)
+declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr)
+declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr)
+declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr)
+
+; Scalar type tests
+
+define i32 @speculative_load_i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_i32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr w0, [x0]
+; CHECK-NEXT: ret
+ %load = call i32 @llvm.speculative.load.i32.p0(ptr align 4 %ptr)
+ ret i32 %load
+}
+
+define i64 @speculative_load_i64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_i64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr x0, [x0]
+; CHECK-NEXT: ret
+ %load = call i64 @llvm.speculative.load.i64.p0(ptr %ptr)
+ ret i64 %load
+}
+
+define float @speculative_load_f32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_f32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr s0, [x0]
+; CHECK-NEXT: ret
+ %load = call float @llvm.speculative.load.f32.p0(ptr %ptr)
+ ret float %load
+}
+
+define double @speculative_load_f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_f64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr d0, [x0]
+; CHECK-NEXT: ret
+ %load = call double @llvm.speculative.load.f64.p0(ptr align 8 %ptr)
+ ret double %load
+}
+
+declare i32 @llvm.speculative.load.i32.p0(ptr)
+declare i64 @llvm.speculative.load.i64.p0(ptr)
+declare float @llvm.speculative.load.f32.p0(ptr)
+declare double @llvm.speculative.load.f64.p0(ptr)
diff --git a/llvm/test/CodeGen/X86/can-load-speculatively.ll b/llvm/test/CodeGen/X86/can-load-speculatively.ll
new file mode 100644
index 0000000000000..f51d3847e921d
--- /dev/null
+++ b/llvm/test/CodeGen/X86/can-load-speculatively.ll
@@ -0,0 +1,32 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -mtriple=x86_64-unknown-linux-gnu -passes=pre-isel-intrinsic-lowering -S < %s | FileCheck %s
+
+; Test that @llvm.can.load.speculatively returns false (default) on X86,
+; as X86 does not provide a target-specific expansion.
+
+define i1 @can_load_speculatively_16(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_16(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_32(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_32(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 32)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_8(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_8(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 8)
+ ret i1 %can_load
+}
+
+declare i1 @llvm.can.load.speculatively.p0(ptr, i64)
+
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
new file mode 100644
index 0000000000000..4dd8375e9c42a
--- /dev/null
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -0,0 +1,146 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+sse4.1 < %s | FileCheck %s --check-prefix=SSE
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2 < %s | FileCheck %s --check-prefix=AVX
+
+; Test that @llvm.speculative.load is lowered to a regular load
+; in SelectionDAG.
+
+define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
+; SSE-LABEL: speculative_load_v4i32:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v4i32:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: retq
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr)
+ ret <4 x i32> %load
+}
+
+define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
+; SSE-LABEL: speculative_load_v8i32:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: movaps 16(%rdi), %xmm1
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v8i32:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %ymm0
+; AVX-NEXT: retq
+ %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr)
+ ret <8 x i32> %load
+}
+
+define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
+; SSE-LABEL: speculative_load_v2i64:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v2i64:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: retq
+ %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
+ ret <2 x i64> %load
+}
+
+define <4 x float> @speculative_load_v4f32(ptr %ptr) {
+; SSE-LABEL: speculative_load_v4f32:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v4f32:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: retq
+ %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
+ ret <4 x float> %load
+}
+
+define <2 x double> @speculative_load_v2f64(ptr %ptr) {
+; SSE-LABEL: speculative_load_v2f64:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v2f64:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: retq
+ %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr)
+ ret <2 x double> %load
+}
+
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr)
+declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr)
+declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr)
+declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr)
+declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr)
+
+; Scalar type tests
+
+define i32 @speculative_load_i32(ptr %ptr) {
+; SSE-LABEL: speculative_load_i32:
+; SSE: # %bb.0:
+; SSE-NEXT: movl (%rdi), %eax
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_i32:
+; AVX: # %bb.0:
+; AVX-NEXT: movl (%rdi), %eax
+; AVX-NEXT: retq
+ %load = call i32 @llvm.speculative.load.i32.p0(ptr %ptr)
+ ret i32 %load
+}
+
+define i64 @speculative_load_i64(ptr %ptr) {
+; SSE-LABEL: speculative_load_i64:
+; SSE: # %bb.0:
+; SSE-NEXT: movq (%rdi), %rax
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_i64:
+; AVX: # %bb.0:
+; AVX-NEXT: movq (%rdi), %rax
+; AVX-NEXT: retq
+ %load = call i64 @llvm.speculative.load.i64.p0(ptr align 8 %ptr)
+ ret i64 %load
+}
+
+define float @speculative_load_f32(ptr %ptr) {
+; SSE-LABEL: speculative_load_f32:
+; SSE: # %bb.0:
+; SSE-NEXT: movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_f32:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX-NEXT: retq
+ %load = call float @llvm.speculative.load.f32.p0(ptr align 4 %ptr)
+ ret float %load
+}
+
+define double @speculative_load_f64(ptr %ptr) {
+; SSE-LABEL: speculative_load_f64:
+; SSE: # %bb.0:
+; SSE-NEXT: movsd {{.*#+}} xmm0 = mem[0],zero
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_f64:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX-NEXT: retq
+ %load = call double @llvm.speculative.load.f64.p0(ptr %ptr)
+ ret double %load
+}
+
+declare i32 @llvm.speculative.load.i32.p0(ptr)
+declare i64 @llvm.speculative.load.i64.p0(ptr)
+declare float @llvm.speculative.load.f32.p0(ptr)
+declare double @llvm.speculative.load.f64.p0(ptr)
diff --git a/llvm/test/Verifier/can-load-speculatively.ll b/llvm/test/Verifier/can-load-speculatively.ll
new file mode 100644
index 0000000000000..d2d69f70cfb60
--- /dev/null
+++ b/llvm/test/Verifier/can-load-speculatively.ll
@@ -0,0 +1,19 @@
+; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
+
+declare i1 @llvm.can.load.speculatively.p0(ptr, i64)
+
+; Test that constant size must be a positive power of 2
+
+define i1 @test_size_zero(ptr %ptr) {
+; CHECK: llvm.can.load.speculatively size must be a positive power of 2
+; CHECK-NEXT: %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 0)
+ %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 0)
+ ret i1 %res
+}
+
+define i1 @test_non_power_of_2(ptr %ptr) {
+; CHECK: llvm.can.load.speculatively size must be a positive power of 2
+; CHECK-NEXT: %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 3)
+ %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 3)
+ ret i1 %res
+}
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
new file mode 100644
index 0000000000000..def46d20799c5
--- /dev/null
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -0,0 +1,18 @@
+; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
+
+declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr)
+declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr)
+
+define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
+; CHECK: llvm.speculative.load type must have a power-of-2 size
+; CHECK-NEXT: %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr)
+ %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr)
+ ret <3 x i32> %res
+}
+
+define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
+; CHECK: llvm.speculative.load type must have a power-of-2 size
+; CHECK-NEXT: %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr)
+ %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr)
+ ret <vscale x 3 x i32> %res
+}
>From 35fd4ae1c363cb78ea81474fcbbe19f13684fa30 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Thu, 5 Feb 2026 19:40:18 +0000
Subject: [PATCH 02/14] !fixup address comments, thanks
---
llvm/docs/LangRef.rst | 16 ++++++++--------
llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp | 3 ---
llvm/lib/Target/AArch64/AArch64ISelLowering.cpp | 13 +++++++------
.../CodeGen/AArch64/can-load-speculatively.ll | 5 +----
.../CodeGen/X86/speculative-load-intrinsic.ll | 8 ++++----
5 files changed, 20 insertions(+), 25 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 56f62cac1b6df..adcf4361dd23b 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28355,10 +28355,10 @@ Overview:
"""""""""
The '``llvm.speculative.load``' intrinsic loads a value from memory. Unlike a
-regular load, the memory access may
-extend beyond the bounds of the allocated object, provided the pointer has been
-verified by :ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to
-ensure the access cannot fault.
+regular load, the memory access may extend beyond the bounds of the allocated
+object, provided the pointer has been verified by
+:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure the
+access cannot fault.
Arguments:
""""""""""
@@ -28376,12 +28376,12 @@ the access cannot fault.
For bytes that are within the bounds of the allocated object, the intrinsic
returns the stored value. For bytes that are beyond the bounds of the
-allocated object, the intrinsic returns ``undef`` for those bytes. At least the
+allocated object, the intrinsic returns ``poison`` for those bytes. At least the
first accessed byte must be within the bounds of an allocated object the pointer is
based on.
The behavior is undefined if this intrinsic is used to load from a pointer
-for which ``llvm.can.load.speculatively`` returns false.
+for which ``llvm.can.load.speculatively`` would return false.
.. _int_can_load_speculatively:
@@ -28417,8 +28417,8 @@ Semantics:
""""""""""
This intrinsic has **target-dependent** semantics. It returns ``true`` if
-loading ``num_bytes`` bytes from ``ptr`` is guaranteed not to trap,
-even if the memory is beyond the bounds of an allocated object. It returns
+``num_bytes`` bytes starting at ``ptr`` can be loaded speculatively, even
+if the memory is beyond the bounds of an allocated object. It returns
``false`` otherwise.
The specific conditions under which this intrinsic returns ``true`` are
diff --git a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
index 06a65e7375f00..63c31cdb81970 100644
--- a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
+++ b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
@@ -141,9 +141,6 @@ static bool lowerLoadRelative(Function &F) {
/// TargetLowering::emitCanLoadSpeculatively.
/// The default expansion returns false (conservative).
static bool lowerCanLoadSpeculatively(Function &F, const TargetMachine *TM) {
- if (F.use_empty())
- return false;
-
bool Changed = false;
for (Use &U : llvm::make_early_inc_range(F.uses())) {
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
index 29b7a24c2d481..5f673b83fdb01 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
@@ -31389,6 +31389,10 @@ Value *AArch64TargetLowering::emitStoreConditional(IRBuilderBase &Builder,
Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
Value *Ptr,
Value *Size) const {
+ unsigned AS = cast<PointerType>(Ptr->getType())->getAddressSpace();
+ // Conservatively only allow speculation for address space 0.
+ if (AS != 0)
+ return nullptr;
// For power-of-2 sizes <= 16, emit alignment check: (ptr & (size - 1)) == 0.
// If the pointer is aligned to at least 'size' bytes, loading 'size' bytes
// cannot cross a page boundary, so it's safe to speculate.
@@ -31397,9 +31401,10 @@ Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
//
// The alignment check only works for power-of-2 sizes. For non-power-of-2
// sizes, we conservatively return false.
- const DataLayout &DL =
- Builder.GetInsertBlock()->getModule()->getDataLayout();
+ const DataLayout &DL = Builder.GetInsertBlock()->getModule()->getDataLayout();
+ unsigned PtrBits = DL.getPointerSizeInBits(AS);
+ Type *IntPtrTy = Builder.getIntNTy(PtrBits);
if (auto *CI = dyn_cast<ConstantInt>(Size)) {
uint64_t SizeVal = CI->getZExtValue();
assert(isPowerOf2_64(SizeVal) && "size must be power-of-two");
@@ -31408,8 +31413,6 @@ Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
return nullptr;
// Power-of-2 constant size <= 16: use fast alignment check.
- unsigned PtrBits = DL.getPointerSizeInBits();
- Type *IntPtrTy = Builder.getIntNTy(PtrBits);
Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
Value *Mask = ConstantInt::get(IntPtrTy, SizeVal - 1);
Value *Masked = Builder.CreateAnd(PtrInt, Mask);
@@ -31417,8 +31420,6 @@ Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
}
// Check power-of-2 size <= 16 and alignment.
- unsigned PtrBits = DL.getPointerSizeInBits();
- Type *IntPtrTy = Builder.getIntNTy(PtrBits);
Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
Value *SizeExt = Builder.CreateZExtOrTrunc(Size, IntPtrTy);
diff --git a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
index 7916f2e4d340f..e9af2eed8918a 100644
--- a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
+++ b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
@@ -38,10 +38,7 @@ define i1 @can_load_speculatively_64(ptr %ptr) {
; Test with address space
define i1 @can_load_speculatively_addrspace1(ptr addrspace(1) %ptr) {
; CHECK-LABEL: @can_load_speculatively_addrspace1(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr addrspace(1) [[PTR:%.*]] to i64
-; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
-; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
-; CHECK-NEXT: ret i1 [[TMP3]]
+; CHECK-NEXT: ret i1 false
;
%can_load = call i1 @llvm.can.load.speculatively.p1(ptr addrspace(1) %ptr, i64 16)
ret i1 %can_load
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index 4dd8375e9c42a..798ee2c03b1ab 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -37,12 +37,12 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
; SSE-LABEL: speculative_load_v2i64:
; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: movups (%rdi), %xmm0
; SSE-NEXT: retq
;
; AVX-LABEL: speculative_load_v2i64:
; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: vmovups (%rdi), %xmm0
; AVX-NEXT: retq
%load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
ret <2 x i64> %load
@@ -51,12 +51,12 @@ define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
define <4 x float> @speculative_load_v4f32(ptr %ptr) {
; SSE-LABEL: speculative_load_v4f32:
; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: movups (%rdi), %xmm0
; SSE-NEXT: retq
;
; AVX-LABEL: speculative_load_v4f32:
; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: vmovups (%rdi), %xmm0
; AVX-NEXT: retq
%load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
ret <4 x float> %load
>From 9cf7e00004a30ddcce1503c902e2ac4d95221fe5 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Mon, 9 Feb 2026 18:42:04 +0000
Subject: [PATCH 03/14] !fixup add tests with aligned pointers, clarify deref
guarantees.
---
llvm/docs/LangRef.rst | 9 ++--
.../CodeGen/AArch64/can-load-speculatively.ll | 45 +++++++++++++++++++
2 files changed, 50 insertions(+), 4 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index adcf4361dd23b..891dac04bc453 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28417,13 +28417,14 @@ Semantics:
""""""""""
This intrinsic has **target-dependent** semantics. It returns ``true`` if
-``num_bytes`` bytes starting at ``ptr`` can be loaded speculatively, even
-if the memory is beyond the bounds of an allocated object. It returns
-``false`` otherwise.
+``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for any non-negative
+integer ``I`` where the computed address does not wrap around the address
+space, can be loaded speculatively, even if the memory is beyond the bounds of
+an allocated object. It returns ``false`` otherwise.
The specific conditions under which this intrinsic returns ``true`` are
determined by the target. For example, a target may check whether the pointer
-alignment guarantees the load cannot cross a page boundary.
+alignment guarantees all such loads cannot cross a page boundary.
.. code-block:: llvm
diff --git a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
index e9af2eed8918a..b6679f22b0989 100644
--- a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
+++ b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
@@ -17,6 +17,51 @@ define i1 @can_load_speculatively_16(ptr %ptr) {
ret i1 %can_load
}
+
+define i1 @can_load_speculatively_8_ptr_aligned(ptr align 8 %ptr) {
+; CHECK-LABEL: @can_load_speculatively_8_ptr_aligned(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_16_ptr_aligned(ptr align 16 %ptr) {
+; CHECK-LABEL: @can_load_speculatively_16_ptr_aligned(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_16_ptr_aligned2(ptr align 16 %ptr) {
+; CHECK-LABEL: @can_load_speculatively_16_ptr_aligned2(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr align 16 %ptr, i64 16)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_32_ptr_aligned(ptr align 32 %ptr) {
+; CHECK-LABEL: @can_load_speculatively_32_ptr_aligned(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
; Size > 16 - returns false (may cross MTE tag granule boundary)
define i1 @can_load_speculatively_32(ptr %ptr) {
; CHECK-LABEL: @can_load_speculatively_32(
>From ef0c325520a3ece8380e6044194f1865d3c6af86 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Wed, 18 Feb 2026 09:09:09 +0000
Subject: [PATCH 04/14] !fixup clarify noalias interaction
---
llvm/docs/LangRef.rst | 27 +++++++++++++++++----------
1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 891dac04bc453..02ddf66f5bf4a 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28370,15 +28370,22 @@ Semantics:
""""""""""
The '``llvm.speculative.load``' intrinsic performs a load that may access
-memory beyond the allocated object. It must be used in combination with
-:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure
-the access cannot fault.
+memory beyond what is accessible through the pointer. It must be used in
+combination with :ref:`llvm.can.load.speculatively <int_can_load_speculatively>`
+to ensure the access can be performed speculatively.
-For bytes that are within the bounds of the allocated object, the intrinsic
-returns the stored value. For bytes that are beyond the bounds of the
-allocated object, the intrinsic returns ``poison`` for those bytes. At least the
-first accessed byte must be within the bounds of an allocated object the pointer is
-based on.
+A byte at ``ptr + i`` is *accessible through* ``ptr`` if both of the following
+hold:
+
+1. The byte lies within the bounds of an allocated object that ``ptr`` is
+ :ref:`based <pointeraliasing>` on.
+2. Accessing the byte through ``ptr`` does not violate any ``noalias``
+ constraints.
+
+For accessible bytes, the intrinsic returns the stored value. For inaccessible
+bytes, the intrinsic returns ``poison`` and the bytes are not considered accessed
+for the purpose of data races or ``noalias`` constraints. At least the first
+byte must be accessible; otherwise the behavior is undefined.
The behavior is undefined if this intrinsic is used to load from a pointer
for which ``llvm.can.load.speculatively`` would return false.
@@ -28417,8 +28424,8 @@ Semantics:
""""""""""
This intrinsic has **target-dependent** semantics. It returns ``true`` if
-``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for any non-negative
-integer ``I`` where the computed address does not wrap around the address
+``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for all non-negative
+integers ``I`` where the computed address does not wrap around the address
space, can be loaded speculatively, even if the memory is beyond the bounds of
an allocated object. It returns ``false`` otherwise.
>From 4b68c43dde97529d1b871f3e7426e3dbd157554c Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Mon, 23 Feb 2026 09:43:35 +0000
Subject: [PATCH 05/14] !fixup limit to executions that do not depend on
may-accessible bytes.
---
llvm/docs/LangRef.rst | 3 +++
1 file changed, 3 insertions(+)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 02ddf66f5bf4a..14ebf571dd5f8 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28387,6 +28387,9 @@ bytes, the intrinsic returns ``poison`` and the bytes are not considered accesse
for the purpose of data races or ``noalias`` constraints. At least the first
byte must be accessible; otherwise the behavior is undefined.
+The behavior is undefined if program execution depends on any byte in the
+result that may not be accessible.
+
The behavior is undefined if this intrinsic is used to load from a pointer
for which ``llvm.can.load.speculatively`` would return false.
>From 2509bddf1291d6139c6dd495429be295dcea6a41 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Wed, 25 Feb 2026 17:23:24 +0000
Subject: [PATCH 06/14] !fixup add implicit freeze
---
llvm/docs/LangRef.rst | 11 ++++-------
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 14ebf571dd5f8..61e1fd8c5f970 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28382,13 +28382,10 @@ hold:
2. Accessing the byte through ``ptr`` does not violate any ``noalias``
constraints.
-For accessible bytes, the intrinsic returns the stored value. For inaccessible
-bytes, the intrinsic returns ``poison`` and the bytes are not considered accessed
-for the purpose of data races or ``noalias`` constraints. At least the first
-byte must be accessible; otherwise the behavior is undefined.
-
-The behavior is undefined if program execution depends on any byte in the
-result that may not be accessible.
+For each byte of the result, if the corresponding byte is accessible through
+``ptr``, the result contains the value stored in memory at that byte. Otherwise,
+the corresponding byte of the result is ``freeze poison`` and the byte is not
+considered accessed for the purpose of data races or ``noalias`` constraints.
The behavior is undefined if this intrinsic is used to load from a pointer
for which ``llvm.can.load.speculatively`` would return false.
>From b31e72dce77f1f37544c12ac2b877814d6be5669 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Fri, 27 Feb 2026 13:52:11 +0000
Subject: [PATCH 07/14] !fixup add read bytes argument
---
llvm/docs/LangRef.rst | 35 ++++++++--------
llvm/include/llvm/IR/Intrinsics.td | 7 ++--
.../SelectionDAG/SelectionDAGBuilder.cpp | 2 +
.../AArch64/speculative-load-intrinsic-sve.ll | 24 +++++------
.../AArch64/speculative-load-intrinsic.ll | 40 +++++++++----------
.../CodeGen/X86/speculative-load-intrinsic.ll | 36 ++++++++---------
llvm/test/Verifier/speculative-load.ll | 12 +++---
7 files changed, 78 insertions(+), 78 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 61e1fd8c5f970..a186c7d062baa 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28347,9 +28347,9 @@ This is an overloaded intrinsic.
::
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>)
- declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>)
- declare i64 @llvm.speculative.load.i64.p0(ptr <ptr>)
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i64 <num_read_bytes>)
+ declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i64 <num_read_bytes>)
+ declare i64 @llvm.speculative.load.i64.p0(ptr <ptr>, i64 <num_read_bytes>)
Overview:
"""""""""
@@ -28363,32 +28363,29 @@ access cannot fault.
Arguments:
""""""""""
-The argument is a pointer to the memory location to load from. The return type
-must have a power-of-2 size in bytes.
+The first argument is a pointer to the memory location to load from. The second
+argument is an ``i64`` specifying the number of bytes starting from ``ptr``
+that the intrinsic will read. The return type must have a power-of-2 size in
+bytes.
Semantics:
""""""""""
The '``llvm.speculative.load``' intrinsic performs a load that may access
-memory beyond what is accessible through the pointer. It must be used in
+memory beyond what is readable through the pointer. It must be used in
combination with :ref:`llvm.can.load.speculatively <int_can_load_speculatively>`
to ensure the access can be performed speculatively.
-A byte at ``ptr + i`` is *accessible through* ``ptr`` if both of the following
-hold:
+The intrinsic reads ``num_read_bytes`` bytes starting from ``ptr`` and returns
+the stored values for those bytes. The read bytes must lie within the bounds
+of an allocated object that ``ptr`` is :ref:`based <pointeraliasing>` on.
-1. The byte lies within the bounds of an allocated object that ``ptr`` is
- :ref:`based <pointeraliasing>` on.
-2. Accessing the byte through ``ptr`` does not violate any ``noalias``
- constraints.
-
-For each byte of the result, if the corresponding byte is accessible through
-``ptr``, the result contains the value stored in memory at that byte. Otherwise,
-the corresponding byte of the result is ``freeze poison`` and the byte is not
-considered accessed for the purpose of data races or ``noalias`` constraints.
+Bytes at offsets ``>= num_read_bytes`` (up to the size of the return type)
+are ``poison`` and are not considered accessed for the purpose of data races
+or ``noalias`` constraints.
The behavior is undefined if this intrinsic is used to load from a pointer
-for which ``llvm.can.load.speculatively`` would return false.
+for which ``llvm.can.load.speculatively`` returns false.
.. _int_can_load_speculatively:
@@ -28441,7 +28438,7 @@ alignment guarantees all such loads cannot cross a page boundary.
speculative_path:
; Safe to speculatively load from %ptr
- %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr)
+ %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 16)
...
safe_path:
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 5855edc8117f2..410962dcbff6a 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2715,11 +2715,12 @@ def int_masked_srem:
[IntrNoMem]>;
// Speculatively load a value from memory; lowers to a regular aligned load.
-// The loaded type must have a power-of-2 size.
+// The loaded type must have a power-of-2 size. The second argument specifies
+// the number of bytes the intrinsic will read from memory.
def int_speculative_load:
DefaultAttrsIntrinsic<[llvm_any_ty],
- [llvm_anyptr_ty],
- [IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
+ [llvm_anyptr_ty, llvm_i64_ty],
+ [IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
// The size can be a runtime value to support scalable vectors.
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 8879e77084f5c..67fa11cdc25cd 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5147,6 +5147,8 @@ void SelectionDAGBuilder::visitMaskedLoad(const CallInst &I, bool IsExpanding) {
void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
SDLoc sdl = getCurSDLoc();
Value *PtrOperand = I.getArgOperand(0);
+ // The second argument (num_read_bytes) is IR-level
+ // semantics only; it is not needed at codegen.
SDValue Ptr = getValue(PtrOperand);
const TargetLowering &TLI = DAG.getTargetLoweringInfo();
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
index 78a56f3539d11..4ae56cbca7ae9 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
@@ -9,7 +9,7 @@ define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr)
+ %load = call <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i64 0)
ret <vscale x 4 x i32> %load
}
@@ -18,7 +18,7 @@ define <vscale x 2 x i64> @speculative_load_nxv2i64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr %ptr)
+ %load = call <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr %ptr, i64 0)
ret <vscale x 2 x i64> %load
}
@@ -27,7 +27,7 @@ define <vscale x 8 x i16> @speculative_load_nxv8i16(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr align 8 %ptr)
+ %load = call <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr align 8 %ptr, i64 0)
ret <vscale x 8 x i16> %load
}
@@ -36,7 +36,7 @@ define <vscale x 16 x i8> @speculative_load_nxv16i8(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr %ptr)
+ %load = call <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr %ptr, i64 0)
ret <vscale x 16 x i8> %load
}
@@ -45,7 +45,7 @@ define <vscale x 4 x float> @speculative_load_nxv4f32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr align 4 %ptr)
+ %load = call <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr align 4 %ptr, i64 0)
ret <vscale x 4 x float> %load
}
@@ -54,13 +54,13 @@ define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr)
+ %load = call <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i64 0)
ret <vscale x 2 x double> %load
}
-declare <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr)
-declare <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr)
-declare <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr)
-declare <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr)
-declare <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr)
-declare <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr)
+declare <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr, i64)
+declare <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr, i64)
+declare <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr, i64)
+declare <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr, i64)
+declare <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr, i64)
+declare <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr, i64)
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
index 8f9a17414ff05..1c6b9d267ad46 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -9,7 +9,7 @@ define <4 x i32> @speculative_load_v4i32_align16(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr)
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 0)
ret <4 x i32> %load
}
@@ -18,7 +18,7 @@ define <4 x i32> @speculative_load_v4i32_align4(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 4 %ptr)
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 4 %ptr, i64 0)
ret <4 x i32> %load
}
@@ -27,7 +27,7 @@ define <4 x i32> @speculative_load_v4i32_noalign(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr)
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 0)
ret <4 x i32> %load
}
@@ -36,7 +36,7 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldp q0, q1, [x0]
; CHECK-NEXT: ret
- %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr)
+ %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 0)
ret <8 x i32> %load
}
@@ -45,7 +45,7 @@ define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
+ %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr, i64 0)
ret <2 x i64> %load
}
@@ -54,7 +54,7 @@ define <4 x float> @speculative_load_v4f32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
+ %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr, i64 0)
ret <4 x float> %load
}
@@ -63,15 +63,15 @@ define <2 x double> @speculative_load_v2f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr)
+ %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 0)
ret <2 x double> %load
}
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr)
-declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr)
-declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr)
-declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr)
-declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr)
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i64)
+declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr, i64)
+declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr, i64)
+declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr, i64)
+declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr, i64)
; Scalar type tests
@@ -80,7 +80,7 @@ define i32 @speculative_load_i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr w0, [x0]
; CHECK-NEXT: ret
- %load = call i32 @llvm.speculative.load.i32.p0(ptr align 4 %ptr)
+ %load = call i32 @llvm.speculative.load.i32.p0(ptr align 4 %ptr, i64 0)
ret i32 %load
}
@@ -89,7 +89,7 @@ define i64 @speculative_load_i64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr x0, [x0]
; CHECK-NEXT: ret
- %load = call i64 @llvm.speculative.load.i64.p0(ptr %ptr)
+ %load = call i64 @llvm.speculative.load.i64.p0(ptr %ptr, i64 0)
ret i64 %load
}
@@ -98,7 +98,7 @@ define float @speculative_load_f32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr s0, [x0]
; CHECK-NEXT: ret
- %load = call float @llvm.speculative.load.f32.p0(ptr %ptr)
+ %load = call float @llvm.speculative.load.f32.p0(ptr %ptr, i64 0)
ret float %load
}
@@ -107,11 +107,11 @@ define double @speculative_load_f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr d0, [x0]
; CHECK-NEXT: ret
- %load = call double @llvm.speculative.load.f64.p0(ptr align 8 %ptr)
+ %load = call double @llvm.speculative.load.f64.p0(ptr align 8 %ptr, i64 0)
ret double %load
}
-declare i32 @llvm.speculative.load.i32.p0(ptr)
-declare i64 @llvm.speculative.load.i64.p0(ptr)
-declare float @llvm.speculative.load.f32.p0(ptr)
-declare double @llvm.speculative.load.f64.p0(ptr)
+declare i32 @llvm.speculative.load.i32.p0(ptr, i64)
+declare i64 @llvm.speculative.load.i64.p0(ptr, i64)
+declare float @llvm.speculative.load.f32.p0(ptr, i64)
+declare double @llvm.speculative.load.f64.p0(ptr, i64)
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index 798ee2c03b1ab..5e12593f2d5a0 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -15,7 +15,7 @@ define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovaps (%rdi), %xmm0
; AVX-NEXT: retq
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr)
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 0)
ret <4 x i32> %load
}
@@ -30,7 +30,7 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovaps (%rdi), %ymm0
; AVX-NEXT: retq
- %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr)
+ %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 0)
ret <8 x i32> %load
}
@@ -44,7 +44,7 @@ define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovups (%rdi), %xmm0
; AVX-NEXT: retq
- %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
+ %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr, i64 0)
ret <2 x i64> %load
}
@@ -58,7 +58,7 @@ define <4 x float> @speculative_load_v4f32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovups (%rdi), %xmm0
; AVX-NEXT: retq
- %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
+ %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr, i64 0)
ret <4 x float> %load
}
@@ -72,15 +72,15 @@ define <2 x double> @speculative_load_v2f64(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovaps (%rdi), %xmm0
; AVX-NEXT: retq
- %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr)
+ %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 0)
ret <2 x double> %load
}
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr)
-declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr)
-declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr)
-declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr)
-declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr)
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i64)
+declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr, i64)
+declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr, i64)
+declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr, i64)
+declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr, i64)
; Scalar type tests
@@ -94,7 +94,7 @@ define i32 @speculative_load_i32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: movl (%rdi), %eax
; AVX-NEXT: retq
- %load = call i32 @llvm.speculative.load.i32.p0(ptr %ptr)
+ %load = call i32 @llvm.speculative.load.i32.p0(ptr %ptr, i64 0)
ret i32 %load
}
@@ -108,7 +108,7 @@ define i64 @speculative_load_i64(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: movq (%rdi), %rax
; AVX-NEXT: retq
- %load = call i64 @llvm.speculative.load.i64.p0(ptr align 8 %ptr)
+ %load = call i64 @llvm.speculative.load.i64.p0(ptr align 8 %ptr, i64 0)
ret i64 %load
}
@@ -122,7 +122,7 @@ define float @speculative_load_f32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
; AVX-NEXT: retq
- %load = call float @llvm.speculative.load.f32.p0(ptr align 4 %ptr)
+ %load = call float @llvm.speculative.load.f32.p0(ptr align 4 %ptr, i64 0)
ret float %load
}
@@ -136,11 +136,11 @@ define double @speculative_load_f64(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovsd {{.*#+}} xmm0 = mem[0],zero
; AVX-NEXT: retq
- %load = call double @llvm.speculative.load.f64.p0(ptr %ptr)
+ %load = call double @llvm.speculative.load.f64.p0(ptr %ptr, i64 0)
ret double %load
}
-declare i32 @llvm.speculative.load.i32.p0(ptr)
-declare i64 @llvm.speculative.load.i64.p0(ptr)
-declare float @llvm.speculative.load.f32.p0(ptr)
-declare double @llvm.speculative.load.f64.p0(ptr)
+declare i32 @llvm.speculative.load.i32.p0(ptr, i64)
+declare i64 @llvm.speculative.load.i64.p0(ptr, i64)
+declare float @llvm.speculative.load.f32.p0(ptr, i64)
+declare double @llvm.speculative.load.f64.p0(ptr, i64)
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
index def46d20799c5..c06881a5f50af 100644
--- a/llvm/test/Verifier/speculative-load.ll
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -1,18 +1,18 @@
; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
-declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr)
-declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr)
+declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, i64)
+declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, i64)
define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr)
- %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr)
+; CHECK-NEXT: %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
+ %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
ret <3 x i32> %res
}
define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr)
- %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr)
+; CHECK-NEXT: %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
+ %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
ret <vscale x 3 x i32> %res
}
>From f7e34082f56d6cbc92bb34ee005071ed8bc9be86 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Fri, 20 Mar 2026 11:58:45 +0000
Subject: [PATCH 08/14] !fixup make oracle function part of the arguments
---
llvm/docs/LangRef.rst | 48 +++--
llvm/include/llvm/IR/Intrinsics.td | 6 +-
.../SelectionDAG/SelectionDAGBuilder.cpp | 4 +-
llvm/lib/IR/Verifier.cpp | 44 ++++
.../AArch64/speculative-load-intrinsic-sve.ll | 66 ------
.../AArch64/speculative-load-intrinsic.ll | 123 ++++++------
.../CodeGen/X86/speculative-load-intrinsic.ll | 190 +++++++-----------
llvm/test/Verifier/speculative-load.ll | 74 ++++++-
8 files changed, 272 insertions(+), 283 deletions(-)
delete mode 100644 llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index a186c7d062baa..1575ea6b06b2e 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28334,7 +28334,7 @@ Speculative Load Intrinsics
LLVM provides intrinsics for speculatively loading memory that may be
out-of-bounds. These intrinsics enable optimizations like early-exit loop
vectorization where the vectorized loop may read beyond the end of an array,
-provided the access is guaranteed to not trap by target-specific checks.
+provided the access is guaranteed be valid by target-specific checks.
.. _int_speculative_load:
@@ -28347,9 +28347,12 @@ This is an overloaded intrinsic.
::
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i64 <num_read_bytes>)
- declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i64 <num_read_bytes>)
- declare i64 @llvm.speculative.load.i64.p0(ptr <ptr>, i64 <num_read_bytes>)
+ ; Direct form: number of accessible bytes given as i64
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i64 <num_accessible_bytes>)
+ declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i64 <num_accessible_bytes>)
+
+ ; Oracle form: accessible bytes computed by calling oracle_fn(args...)
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, ptr <oracle_fn>, ...)
Overview:
"""""""""
@@ -28358,34 +28361,35 @@ The '``llvm.speculative.load``' intrinsic loads a value from memory. Unlike a
regular load, the memory access may extend beyond the bounds of the allocated
object, provided the pointer has been verified by
:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure the
-access cannot fault.
+access is valid.
Arguments:
""""""""""
-The first argument is a pointer to the memory location to load from. The second
-argument is an ``i64`` specifying the number of bytes starting from ``ptr``
-that the intrinsic will read. The return type must have a power-of-2 size in
-bytes.
+The first argument is a pointer to the memory location to load from. The return
+type must be a vector type with a power-of-2 size in bytes. The remaining
+arguments determine the *number of accessible bytes* starting from ``ptr``,
+denoted ``N`` below.
+
+In the **direct form**, the second argument is an ``i64`` specifying ``N``
+directly. In the **oracle form**, the second argument must be a direct
+reference to a function returning ``i64`` that may only read memory through its
+arguments (indirect function pointers are not permitted); the remaining
+arguments are forwarded to it, and the return value is ``N``.
Semantics:
""""""""""
-The '``llvm.speculative.load``' intrinsic performs a load that may access
-memory beyond what is readable through the pointer. It must be used in
-combination with :ref:`llvm.can.load.speculatively <int_can_load_speculatively>`
-to ensure the access can be performed speculatively.
-
-The intrinsic reads ``num_read_bytes`` bytes starting from ``ptr`` and returns
-the stored values for those bytes. The read bytes must lie within the bounds
-of an allocated object that ``ptr`` is :ref:`based <pointeraliasing>` on.
+The intrinsic reads ``N`` bytes starting from ``ptr`` and returns the stored
+values for those bytes. These bytes must lie within the bounds of an allocated
+object that ``ptr`` is :ref:`based <pointeraliasing>` on.
-Bytes at offsets ``>= num_read_bytes`` (up to the size of the return type)
-are ``poison`` and are not considered accessed for the purpose of data races
-or ``noalias`` constraints.
+Bytes at offsets in ``[N, size of the return type)`` are ``poison`` and are not
+considered accessed for the purposes of data races or ``noalias`` constraints.
-The behavior is undefined if this intrinsic is used to load from a pointer
-for which ``llvm.can.load.speculatively`` returns false.
+The behavior is undefined if the speculative load accesses memory that would
+fault (i.e., the oracle or ``llvm.can.load.speculatively`` would indicate the
+access is not safe).
.. _int_can_load_speculatively:
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 410962dcbff6a..d960ebcb918ca 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2715,11 +2715,9 @@ def int_masked_srem:
[IntrNoMem]>;
// Speculatively load a value from memory; lowers to a regular aligned load.
-// The loaded type must have a power-of-2 size. The second argument specifies
-// the number of bytes the intrinsic will read from memory.
def int_speculative_load:
- DefaultAttrsIntrinsic<[llvm_any_ty],
- [llvm_anyptr_ty, llvm_i64_ty],
+ DefaultAttrsIntrinsic<[llvm_anyvector_ty],
+ [llvm_anyptr_ty, llvm_vararg_ty],
[IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 67fa11cdc25cd..0d122aacbd695 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5147,8 +5147,8 @@ void SelectionDAGBuilder::visitMaskedLoad(const CallInst &I, bool IsExpanding) {
void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
SDLoc sdl = getCurSDLoc();
Value *PtrOperand = I.getArgOperand(0);
- // The second argument (num_read_bytes) is IR-level
- // semantics only; it is not needed at codegen.
+ // The remaining arguments (num_accessible_bytes or oracle function + args)
+ // are IR-level semantics only; they are not needed at codegen.
SDValue Ptr = getValue(PtrOperand);
const TargetLowering &TLI = DAG.getTargetLoweringInfo();
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 8df3d8277fbaf..bae3b2486b9ba 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -6947,6 +6947,50 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
// For scalable vectors, check the known minimum size is a power of 2.
Check(Size.getKnownMinValue() > 0 && isPowerOf2_64(Size.getKnownMinValue()),
"llvm.speculative.load type must have a power-of-2 size", &Call);
+
+ unsigned NumArgs = Call.arg_size();
+ Check(NumArgs >= 2, "llvm.speculative.load requires at least 2 arguments",
+ &Call);
+
+ Value *SecondArg = Call.getArgOperand(1);
+ if (SecondArg->getType()->isIntegerTy(64)) {
+ // Direct form: (ptr, i64 num_accessible_bytes)
+ Check(NumArgs == 2,
+ "llvm.speculative.load direct form requires exactly 2 arguments",
+ &Call);
+ if (auto *CI = dyn_cast<ConstantInt>(SecondArg)) {
+ Check(Size.isScalable() || CI->getZExtValue() <= Size.getFixedValue(),
+ "llvm.speculative.load num_accessible_bytes must not exceed "
+ "the result size in bytes",
+ &Call);
+ }
+ } else {
+ // Oracle form: (ptr, oracle_fn_ptr, args...)
+ auto *OracleFn = dyn_cast<Function>(SecondArg);
+ Check(OracleFn,
+ "llvm.speculative.load second argument must be i64 or a direct "
+ "reference to an oracle function",
+ &Call);
+
+ Check(OracleFn->onlyReadsMemory() && OracleFn->onlyAccessesArgMemory(),
+ "llvm.speculative.load oracle function must not have side effects "
+ "and may only read memory through its arguments",
+ &Call);
+
+ FunctionType *FTy = OracleFn->getFunctionType();
+ Check(FTy->getReturnType()->isIntegerTy(64),
+ "llvm.speculative.load oracle function must return i64", &Call);
+ unsigned NumOracleArgs = NumArgs - 2;
+ Check(FTy->isVarArg() ? NumOracleArgs >= FTy->getNumParams()
+ : NumOracleArgs == FTy->getNumParams(),
+ "llvm.speculative.load oracle function argument count mismatch",
+ &Call);
+ for (unsigned I = 0, E = FTy->getNumParams(); I < E; ++I) {
+ Check(FTy->getParamType(I) == Call.getArgOperand(I + 2)->getType(),
+ "llvm.speculative.load oracle function argument type mismatch",
+ &Call);
+ }
+ }
break;
}
case Intrinsic::can_load_speculatively: {
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
deleted file mode 100644
index 4ae56cbca7ae9..0000000000000
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
+++ /dev/null
@@ -1,66 +0,0 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64-unknown-linux-gnu -mattr=+sve < %s | FileCheck %s
-
-; Test that @llvm.speculative.load with scalable vectors is lowered to a
-; regular load in SelectionDAG.
-
-define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv4i32:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i64 0)
- ret <vscale x 4 x i32> %load
-}
-
-define <vscale x 2 x i64> @speculative_load_nxv2i64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv2i64:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr %ptr, i64 0)
- ret <vscale x 2 x i64> %load
-}
-
-define <vscale x 8 x i16> @speculative_load_nxv8i16(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv8i16:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr align 8 %ptr, i64 0)
- ret <vscale x 8 x i16> %load
-}
-
-define <vscale x 16 x i8> @speculative_load_nxv16i8(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv16i8:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr %ptr, i64 0)
- ret <vscale x 16 x i8> %load
-}
-
-define <vscale x 4 x float> @speculative_load_nxv4f32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv4f32:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr align 4 %ptr, i64 0)
- ret <vscale x 4 x float> %load
-}
-
-define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv2f64:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i64 0)
- ret <vscale x 2 x double> %load
-}
-
-declare <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr, i64)
-declare <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr, i64)
-declare <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr, i64)
-declare <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr, i64)
-declare <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr, i64)
-declare <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr, i64)
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
index 1c6b9d267ad46..2746b9ea87daa 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -1,117 +1,112 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64-unknown-linux-gnu < %s | FileCheck %s
+; RUN: llc -mtriple=aarch64-unknown-linux-gnu -mattr=+sve < %s | FileCheck %s
; Test that @llvm.speculative.load is lowered to a regular load
-; in SelectionDAG, respecting the alignment attribute.
+; in SelectionDAG for fixed vectors, scalable vectors, and scalars.
-define <4 x i32> @speculative_load_v4i32_align16(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v4i32_align16:
+; Fixed-width vector tests
+
+define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 0)
+ %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 12)
ret <4 x i32> %load
}
-define <4 x i32> @speculative_load_v4i32_align4(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v4i32_align4:
+define <2 x double> @speculative_load_v2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v2f64:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 4 %ptr, i64 0)
- ret <4 x i32> %load
+ %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 8)
+ ret <2 x double> %load
}
-define <4 x i32> @speculative_load_v4i32_noalign(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v4i32_noalign:
+; Scalable vector tests
+
+define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv4i32:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 0)
- ret <4 x i32> %load
+ %load = call <vscale x 4 x i32> (ptr, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i64 8)
+ ret <vscale x 4 x i32> %load
}
-define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v8i32:
+define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv2f64:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldp q0, q1, [x0]
+; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 0)
- ret <8 x i32> %load
+ %load = call <vscale x 2 x double> (ptr, ...) @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i64 4)
+ ret <vscale x 2 x double> %load
}
-define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v2i64:
+; Oracle form tests
+
+declare i64 @oracle(ptr, i64) memory(argmem: read)
+
+define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr, i64 0)
- ret <2 x i64> %load
+ %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
}
-define <4 x float> @speculative_load_v4f32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v4f32:
+define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v2i64_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr, i64 0)
- ret <4 x float> %load
+ %load = call <2 x i64> (ptr, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <2 x i64> %load
}
-define <2 x double> @speculative_load_v2f64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v2f64:
+define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i16_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 0)
- ret <2 x double> %load
+ %load = call <8 x i16> (ptr, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i16> %load
}
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i64)
-declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr, i64)
-declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr, i64)
-declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr, i64)
-declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr, i64)
-
-; Scalar type tests
-
-define i32 @speculative_load_i32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_i32:
+define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v16i8_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr w0, [x0]
+; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call i32 @llvm.speculative.load.i32.p0(ptr align 4 %ptr, i64 0)
- ret i32 %load
+ %load = call <16 x i8> (ptr, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <16 x i8> %load
}
-define i64 @speculative_load_i64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_i64:
+define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4f32_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr x0, [x0]
+; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call i64 @llvm.speculative.load.i64.p0(ptr %ptr, i64 0)
- ret i64 %load
+ %load = call <4 x float> (ptr, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x float> %load
}
-define float @speculative_load_f32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_f32:
+define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v2f64_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr s0, [x0]
+; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call float @llvm.speculative.load.f32.p0(ptr %ptr, i64 0)
- ret float %load
+ %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <2 x double> %load
}
-define double @speculative_load_f64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_f64:
+define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_nxv4i32_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr d0, [x0]
+; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call double @llvm.speculative.load.f64.p0(ptr align 8 %ptr, i64 0)
- ret double %load
+ %load = call <vscale x 4 x i32> (ptr, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <vscale x 4 x i32> %load
}
-
-declare i32 @llvm.speculative.load.i32.p0(ptr, i64)
-declare i64 @llvm.speculative.load.i64.p0(ptr, i64)
-declare float @llvm.speculative.load.f32.p0(ptr, i64)
-declare double @llvm.speculative.load.f64.p0(ptr, i64)
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index 5e12593f2d5a0..e387a17990012 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -1,146 +1,98 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+sse4.1 < %s | FileCheck %s --check-prefix=SSE
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2 < %s | FileCheck %s --check-prefix=AVX
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2 < %s | FileCheck %s
; Test that @llvm.speculative.load is lowered to a regular load
; in SelectionDAG.
define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
-; SSE-LABEL: speculative_load_v4i32:
-; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v4i32:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %xmm0
-; AVX-NEXT: retq
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 0)
+; CHECK-LABEL: speculative_load_v4i32:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 8)
ret <4 x i32> %load
}
define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
-; SSE-LABEL: speculative_load_v8i32:
-; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
-; SSE-NEXT: movaps 16(%rdi), %xmm1
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v8i32:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %ymm0
-; AVX-NEXT: retq
- %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 0)
+; CHECK-LABEL: speculative_load_v8i32:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <8 x i32> (ptr, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 24)
ret <8 x i32> %load
}
-define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
-; SSE-LABEL: speculative_load_v2i64:
-; SSE: # %bb.0:
-; SSE-NEXT: movups (%rdi), %xmm0
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v2i64:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovups (%rdi), %xmm0
-; AVX-NEXT: retq
- %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr, i64 0)
- ret <2 x i64> %load
-}
+; Oracle form tests
+declare i64 @oracle(ptr, i64) memory(argmem: read)
-define <4 x float> @speculative_load_v4f32(ptr %ptr) {
-; SSE-LABEL: speculative_load_v4f32:
-; SSE: # %bb.0:
-; SSE-NEXT: movups (%rdi), %xmm0
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v4f32:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovups (%rdi), %xmm0
-; AVX-NEXT: retq
- %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr, i64 0)
- ret <4 x float> %load
+define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
}
-define <2 x double> @speculative_load_v2f64(ptr %ptr) {
-; SSE-LABEL: speculative_load_v2f64:
-; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v2f64:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %xmm0
-; AVX-NEXT: retq
- %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 0)
- ret <2 x double> %load
+define <8 x i32> @speculative_load_v8i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i32_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <8 x i32> (ptr, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i32> %load
}
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i64)
-declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr, i64)
-declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr, i64)
-declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr, i64)
-declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr, i64)
-
-; Scalar type tests
+define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v2i64_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <2 x i64> (ptr, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <2 x i64> %load
+}
-define i32 @speculative_load_i32(ptr %ptr) {
-; SSE-LABEL: speculative_load_i32:
-; SSE: # %bb.0:
-; SSE-NEXT: movl (%rdi), %eax
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_i32:
-; AVX: # %bb.0:
-; AVX-NEXT: movl (%rdi), %eax
-; AVX-NEXT: retq
- %load = call i32 @llvm.speculative.load.i32.p0(ptr %ptr, i64 0)
- ret i32 %load
+define <4 x i64> @speculative_load_v4i64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i64_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <4 x i64> (ptr, ...) @llvm.speculative.load.v4i64.p0(ptr align 32 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i64> %load
}
-define i64 @speculative_load_i64(ptr %ptr) {
-; SSE-LABEL: speculative_load_i64:
-; SSE: # %bb.0:
-; SSE-NEXT: movq (%rdi), %rax
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_i64:
-; AVX: # %bb.0:
-; AVX-NEXT: movq (%rdi), %rax
-; AVX-NEXT: retq
- %load = call i64 @llvm.speculative.load.i64.p0(ptr align 8 %ptr, i64 0)
- ret i64 %load
+define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4f32_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x float> (ptr, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x float> %load
}
-define float @speculative_load_f32(ptr %ptr) {
-; SSE-LABEL: speculative_load_f32:
-; SSE: # %bb.0:
-; SSE-NEXT: movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_f32:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; AVX-NEXT: retq
- %load = call float @llvm.speculative.load.f32.p0(ptr align 4 %ptr, i64 0)
- ret float %load
+define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v2f64_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <2 x double> %load
}
-define double @speculative_load_f64(ptr %ptr) {
-; SSE-LABEL: speculative_load_f64:
-; SSE: # %bb.0:
-; SSE-NEXT: movsd {{.*#+}} xmm0 = mem[0],zero
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_f64:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovsd {{.*#+}} xmm0 = mem[0],zero
-; AVX-NEXT: retq
- %load = call double @llvm.speculative.load.f64.p0(ptr %ptr, i64 0)
- ret double %load
+define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i16_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <8 x i16> (ptr, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i16> %load
}
-declare i32 @llvm.speculative.load.i32.p0(ptr, i64)
-declare i64 @llvm.speculative.load.i64.p0(ptr, i64)
-declare float @llvm.speculative.load.f32.p0(ptr, i64)
-declare double @llvm.speculative.load.f64.p0(ptr, i64)
+define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v16i8_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <16 x i8> (ptr, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <16 x i8> %load
+}
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
index c06881a5f50af..34629849025dc 100644
--- a/llvm/test/Verifier/speculative-load.ll
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -1,18 +1,80 @@
; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
-declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, i64)
-declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, i64)
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, ...)
+declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, ...)
+declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, ...)
+
+declare i32 @bad_oracle_ret(ptr, i64) memory(argmem: read)
+declare i64 @good_oracle(ptr, i64) memory(argmem: read)
+declare i64 @oracle_i32_param(i32) memory(argmem: read)
+declare i64 @side_effecting_oracle(ptr, i64)
define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
- %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
+; CHECK-NEXT: %res = call <3 x i32> (ptr, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
+ %res = call <3 x i32> (ptr, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
ret <3 x i32> %res
}
define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
- %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
+; CHECK-NEXT: %res = call <vscale x 3 x i32> (ptr, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
+ %res = call <vscale x 3 x i32> (ptr, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
ret <vscale x 3 x i32> %res
}
+
+define <4 x i32> @test_oracle_wrong_return_type(ptr %ptr, i64 %n) {
+; CHECK: llvm.speculative.load oracle function must return i64
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_oracle_too_few_args(ptr %ptr) {
+; CHECK: llvm.speculative.load oracle function argument count mismatch
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_oracle_too_many_args(ptr %ptr, i64 %n) {
+; CHECK: llvm.speculative.load oracle function argument count mismatch
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_oracle_arg_type_mismatch(ptr %ptr) {
+; CHECK: llvm.speculative.load oracle function argument type mismatch
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @oracle_i32_param, i64 42)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @oracle_i32_param, i64 42)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_non_function_oracle(ptr %ptr, ptr %not_fn) {
+; CHECK: llvm.speculative.load second argument must be i64 or a direct reference to an oracle function
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr %not_fn)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr %not_fn)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_direct_form_extra_args(ptr %ptr) {
+; CHECK: llvm.speculative.load direct form requires exactly 2 arguments
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 8, i64 4)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 8, i64 4)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_oracle_side_effects(ptr %ptr, i64 %n) {
+; CHECK: llvm.speculative.load oracle function must not have side effects and may only read memory through its arguments
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_num_accessible_bytes_exceeds_size(ptr %ptr) {
+; CHECK: llvm.speculative.load num_accessible_bytes must not exceed the result size in bytes
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 32)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 32)
+ ret <4 x i32> %res
+}
>From 4dde84fe7fa2aff72c8c780902c85dd7c5e81898 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Mon, 30 Mar 2026 16:30:28 +0100
Subject: [PATCH 09/14] !fixup add from_end argument
---
llvm/docs/LangRef.rst | 41 +++++++-----
llvm/include/llvm/IR/Intrinsics.td | 5 +-
.../SelectionDAG/SelectionDAGBuilder.cpp | 2 +-
llvm/lib/IR/Verifier.cpp | 28 ++++----
.../AArch64/speculative-load-intrinsic.ll | 60 +++++++++++++----
.../CodeGen/X86/speculative-load-intrinsic.ll | 58 ++++++++++++++---
llvm/test/Verifier/speculative-load.ll | 65 ++++++++++---------
7 files changed, 178 insertions(+), 81 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 1575ea6b06b2e..ad5d2dadf8092 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28334,7 +28334,7 @@ Speculative Load Intrinsics
LLVM provides intrinsics for speculatively loading memory that may be
out-of-bounds. These intrinsics enable optimizations like early-exit loop
vectorization where the vectorized loop may read beyond the end of an array,
-provided the access is guaranteed be valid by target-specific checks.
+provided the access is guaranteed to be valid by target-specific checks.
.. _int_speculative_load:
@@ -28348,11 +28348,11 @@ This is an overloaded intrinsic.
::
; Direct form: number of accessible bytes given as i64
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i64 <num_accessible_bytes>)
- declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i64 <num_accessible_bytes>)
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
+ declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
; Oracle form: accessible bytes computed by calling oracle_fn(args...)
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, ptr <oracle_fn>, ...)
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i1 <from_end>, ptr <oracle_fn>, ...)
Overview:
"""""""""
@@ -28367,25 +28367,36 @@ Arguments:
""""""""""
The first argument is a pointer to the memory location to load from. The return
-type must be a vector type with a power-of-2 size in bytes. The remaining
-arguments determine the *number of accessible bytes* starting from ``ptr``,
+type must be a vector type with a power-of-2 size in bytes. The second argument
+is an ``i1`` constant flag ``from_end`` that specifies whether the ``N``
+accessible bytes are counted from the start or the end of the loaded values (see
+Semantics). The remaining arguments determine the *number of accessible bytes*,
denoted ``N`` below.
-In the **direct form**, the second argument is an ``i64`` specifying ``N``
-directly. In the **oracle form**, the second argument must be a direct
+In the **direct form**, the third argument is an ``i64`` specifying ``N``
+directly. In the **oracle form**, the third argument must be a direct
reference to a function returning ``i64`` that may only read memory through its
arguments (indirect function pointers are not permitted); the remaining
-arguments are forwarded to it, and the return value is ``N``.
+arguments are forwarded to it, and its return value is ``N``.
Semantics:
""""""""""
-The intrinsic reads ``N`` bytes starting from ``ptr`` and returns the stored
-values for those bytes. These bytes must lie within the bounds of an allocated
-object that ``ptr`` is :ref:`based <pointeraliasing>` on.
+Let ``S`` denote the size of the return type in bytes. The intrinsic performs
+a load of ``S`` bytes starting from ``ptr``.
-Bytes at offsets in ``[N, size of the return type)`` are ``poison`` and are not
-considered accessed for the purposes of data races or ``noalias`` constraints.
+When ``from_end`` is ``false``, the first ``N`` bytes (offsets ``[0, N)``)
+are the stored values read from memory. Bytes at offsets ``[N, S)`` are
+``poison``.
+
+When ``from_end`` is ``true``, the last ``N`` bytes (offsets ``[S - N, S)``)
+are the stored values read from memory. Bytes at offsets ``[0, S - N)`` are
+``poison``.
+
+In both cases, the ``N`` accessible bytes must lie within the bounds of an
+allocated object that ``ptr`` is :ref:`based <pointeraliasing>` on, and
+poison bytes are not considered accessed for the purposes of data races or
+``noalias`` constraints. The behavior is undefined if ``N`` exceeds ``S``.
The behavior is undefined if the speculative load accesses memory that would
fault (i.e., the oracle or ``llvm.can.load.speculatively`` would indicate the
@@ -28442,7 +28453,7 @@ alignment guarantees all such loads cannot cross a page boundary.
speculative_path:
; Safe to speculatively load from %ptr
- %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 16)
+ %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 16)
...
safe_path:
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index d960ebcb918ca..fe2a4b81c1f22 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2717,8 +2717,9 @@ def int_masked_srem:
// Speculatively load a value from memory; lowers to a regular aligned load.
def int_speculative_load:
DefaultAttrsIntrinsic<[llvm_anyvector_ty],
- [llvm_anyptr_ty, llvm_vararg_ty],
- [IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
+ [llvm_anyptr_ty, llvm_i1_ty, llvm_vararg_ty],
+ [IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>,
+ ImmArg<ArgIndex<1>>]>;
// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
// The size can be a runtime value to support scalable vectors.
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 0d122aacbd695..4f702fd27d6f0 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5164,7 +5164,7 @@ void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
MachineMemOperand::Flags MMOFlags = MachineMemOperand::MOLoad;
LocationSize LocSize = StoreSize.isScalable()
? LocationSize::beforeOrAfterPointer()
- : LocationSize::precise(StoreSize);
+ : LocationSize::upperBound(StoreSize);
MachineMemOperand *MMO = DAG.getMachineFunction().getMachineMemOperand(
MachinePointerInfo(PtrOperand), MMOFlags, LocSize, Alignment, AAInfo);
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index bae3b2486b9ba..81b61187fc8f9 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -6949,26 +6949,25 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
"llvm.speculative.load type must have a power-of-2 size", &Call);
unsigned NumArgs = Call.arg_size();
- Check(NumArgs >= 2, "llvm.speculative.load requires at least 2 arguments",
+ Check(NumArgs >= 3, "llvm.speculative.load requires at least 3 arguments",
&Call);
- Value *SecondArg = Call.getArgOperand(1);
- if (SecondArg->getType()->isIntegerTy(64)) {
- // Direct form: (ptr, i64 num_accessible_bytes)
- Check(NumArgs == 2,
- "llvm.speculative.load direct form requires exactly 2 arguments",
- &Call);
- if (auto *CI = dyn_cast<ConstantInt>(SecondArg)) {
+ Value *PayloadArg = Call.getArgOperand(2);
+ if (PayloadArg->getType()->isIntegerTy(64)) {
+ // Direct form: (ptr, i1 from_end, i64 num_accessible_bytes)
+ Check(NumArgs == 3,
+ "llvm.speculative.load direct form has too many arguments", &Call);
+ if (auto *CI = dyn_cast<ConstantInt>(PayloadArg)) {
Check(Size.isScalable() || CI->getZExtValue() <= Size.getFixedValue(),
"llvm.speculative.load num_accessible_bytes must not exceed "
"the result size in bytes",
&Call);
}
} else {
- // Oracle form: (ptr, oracle_fn_ptr, args...)
- auto *OracleFn = dyn_cast<Function>(SecondArg);
+ // Oracle form: (ptr, i1 from_end, oracle_fn_ptr, args...)
+ auto *OracleFn = dyn_cast<Function>(PayloadArg);
Check(OracleFn,
- "llvm.speculative.load second argument must be i64 or a direct "
+ "llvm.speculative.load third argument must be i64 or a direct "
"reference to an oracle function",
&Call);
@@ -6980,13 +6979,16 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
FunctionType *FTy = OracleFn->getFunctionType();
Check(FTy->getReturnType()->isIntegerTy(64),
"llvm.speculative.load oracle function must return i64", &Call);
- unsigned NumOracleArgs = NumArgs - 2;
+
+ unsigned OracleArgsStart = 3;
+ unsigned NumOracleArgs = NumArgs - OracleArgsStart;
Check(FTy->isVarArg() ? NumOracleArgs >= FTy->getNumParams()
: NumOracleArgs == FTy->getNumParams(),
"llvm.speculative.load oracle function argument count mismatch",
&Call);
for (unsigned I = 0, E = FTy->getNumParams(); I < E; ++I) {
- Check(FTy->getParamType(I) == Call.getArgOperand(I + 2)->getType(),
+ Check(FTy->getParamType(I) ==
+ Call.getArgOperand(I + OracleArgsStart)->getType(),
"llvm.speculative.load oracle function argument type mismatch",
&Call);
}
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
index 2746b9ea87daa..a547cee576864 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -11,7 +11,7 @@ define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 12)
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, i64 12)
ret <4 x i32> %load
}
@@ -20,7 +20,7 @@ define <2 x double> @speculative_load_v2f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 8)
+ %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, i64 8)
ret <2 x double> %load
}
@@ -31,7 +31,7 @@ define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> (ptr, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i64 8)
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 false, i64 8)
ret <vscale x 4 x i32> %load
}
@@ -40,7 +40,7 @@ define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 2 x double> (ptr, ...) @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i64 4)
+ %load = call <vscale x 2 x double> (ptr, i1, ...) @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i1 false, i64 4)
ret <vscale x 2 x double> %load
}
@@ -53,7 +53,7 @@ define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x i32> %load
}
@@ -62,7 +62,7 @@ define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x i64> (ptr, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <2 x i64> (ptr, i1, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <2 x i64> %load
}
@@ -71,7 +71,7 @@ define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <8 x i16> (ptr, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <8 x i16> (ptr, i1, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <8 x i16> %load
}
@@ -80,7 +80,7 @@ define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <16 x i8> (ptr, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <16 x i8> (ptr, i1, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <16 x i8> %load
}
@@ -89,7 +89,7 @@ define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x float> (ptr, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x float> (ptr, i1, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x float> %load
}
@@ -98,15 +98,53 @@ define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <2 x double> %load
}
+; from_end tests
+
+define <4 x i32> @speculative_load_v4i32_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_from_end:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, i64 12)
+ ret <4 x i32> %load
+}
+
+define <vscale x 4 x i32> @speculative_load_nxv4i32_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv4i32_from_end:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 true, i64 8)
+ ret <vscale x 4 x i32> %load
+}
+
+define <4 x i32> @speculative_load_v4i32_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle_from_end:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
+}
+
+define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_nxv4i32_oracle_from_end:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret <vscale x 4 x i32> %load
+}
+
define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle(ptr %ptr, i64 %n) {
; CHECK-LABEL: speculative_load_nxv4i32_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> (ptr, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <vscale x 4 x i32> %load
}
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index e387a17990012..fa345408940d1 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -9,7 +9,7 @@ define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 8)
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, i64 8)
ret <4 x i32> %load
}
@@ -18,7 +18,7 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %ymm0
; CHECK-NEXT: retq
- %load = call <8 x i32> (ptr, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 24)
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 false, i64 24)
ret <8 x i32> %load
}
@@ -30,7 +30,7 @@ define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x i32> %load
}
@@ -39,7 +39,7 @@ define <8 x i32> @speculative_load_v8i32_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %ymm0
; CHECK-NEXT: retq
- %load = call <8 x i32> (ptr, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <8 x i32> %load
}
@@ -48,7 +48,7 @@ define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <2 x i64> (ptr, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <2 x i64> (ptr, i1, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <2 x i64> %load
}
@@ -57,7 +57,7 @@ define <4 x i64> @speculative_load_v4i64_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %ymm0
; CHECK-NEXT: retq
- %load = call <4 x i64> (ptr, ...) @llvm.speculative.load.v4i64.p0(ptr align 32 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x i64> (ptr, i1, ...) @llvm.speculative.load.v4i64.p0(ptr align 32 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x i64> %load
}
@@ -66,7 +66,7 @@ define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <4 x float> (ptr, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x float> (ptr, i1, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x float> %load
}
@@ -75,7 +75,7 @@ define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <2 x double> %load
}
@@ -84,7 +84,7 @@ define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <8 x i16> (ptr, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <8 x i16> (ptr, i1, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <8 x i16> %load
}
@@ -93,6 +93,44 @@ define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <16 x i8> (ptr, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <16 x i8> (ptr, i1, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <16 x i8> %load
}
+
+; from_end tests
+
+define <4 x i32> @speculative_load_v4i32_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_from_end:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, i64 8)
+ ret <4 x i32> %load
+}
+
+define <8 x i32> @speculative_load_v8i32_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v8i32_from_end:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 true, i64 24)
+ ret <8 x i32> %load
+}
+
+define <4 x i32> @speculative_load_v4i32_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle_from_end:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
+}
+
+define <8 x i32> @speculative_load_v8i32_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i32_oracle_from_end:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i32> %load
+}
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
index 34629849025dc..7fd2d340679aa 100644
--- a/llvm/test/Verifier/speculative-load.ll
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -1,8 +1,8 @@
; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, ...)
-declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, ...)
-declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, ...)
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i1, ...)
+declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, i1, ...)
+declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, i1, ...)
declare i32 @bad_oracle_ret(ptr, i64) memory(argmem: read)
declare i64 @good_oracle(ptr, i64) memory(argmem: read)
@@ -11,70 +11,77 @@ declare i64 @side_effecting_oracle(ptr, i64)
define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <3 x i32> (ptr, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
- %res = call <3 x i32> (ptr, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
+; CHECK-NEXT: %res = call <3 x i32> (ptr, i1, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i1 false, i64 0)
+ %res = call <3 x i32> (ptr, i1, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i1 false, i64 0)
ret <3 x i32> %res
}
define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <vscale x 3 x i32> (ptr, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
- %res = call <vscale x 3 x i32> (ptr, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
+; CHECK-NEXT: %res = call <vscale x 3 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i1 false, i64 0)
+ %res = call <vscale x 3 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i1 false, i64 0)
ret <vscale x 3 x i32> %res
}
+define <4 x i32> @test_too_few_args(ptr %ptr) {
+; CHECK: llvm.speculative.load requires at least 3 arguments
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 true)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 true)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_direct_form_extra_args(ptr %ptr) {
+; CHECK: llvm.speculative.load direct form has too many arguments
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 8, i64 4)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 8, i64 4)
+ ret <4 x i32> %res
+}
+
define <4 x i32> @test_oracle_wrong_return_type(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function must return i64
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
ret <4 x i32> %res
}
define <4 x i32> @test_oracle_too_few_args(ptr %ptr) {
; CHECK: llvm.speculative.load oracle function argument count mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle)
ret <4 x i32> %res
}
define <4 x i32> @test_oracle_too_many_args(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function argument count mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
ret <4 x i32> %res
}
define <4 x i32> @test_oracle_arg_type_mismatch(ptr %ptr) {
; CHECK: llvm.speculative.load oracle function argument type mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @oracle_i32_param, i64 42)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @oracle_i32_param, i64 42)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @oracle_i32_param, i64 42)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @oracle_i32_param, i64 42)
ret <4 x i32> %res
}
define <4 x i32> @test_non_function_oracle(ptr %ptr, ptr %not_fn) {
-; CHECK: llvm.speculative.load second argument must be i64 or a direct reference to an oracle function
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr %not_fn)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr %not_fn)
- ret <4 x i32> %res
-}
-
-define <4 x i32> @test_direct_form_extra_args(ptr %ptr) {
-; CHECK: llvm.speculative.load direct form requires exactly 2 arguments
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 8, i64 4)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 8, i64 4)
+; CHECK: llvm.speculative.load third argument must be i64 or a direct reference to an oracle function
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr %not_fn)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr %not_fn)
ret <4 x i32> %res
}
define <4 x i32> @test_oracle_side_effects(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function must not have side effects and may only read memory through its arguments
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
ret <4 x i32> %res
}
define <4 x i32> @test_num_accessible_bytes_exceeds_size(ptr %ptr) {
; CHECK: llvm.speculative.load num_accessible_bytes must not exceed the result size in bytes
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 32)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 32)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 32)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 32)
ret <4 x i32> %res
}
>From 258096887b2f16bccaff4530f3dfb43d9e7e1f90 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Thu, 14 May 2026 18:25:02 +0100
Subject: [PATCH 10/14] !fixup address comments, thanks!
---
.../SelectionDAG/SelectionDAGBuilder.cpp | 5 +---
llvm/lib/IR/Verifier.cpp | 16 ----------
.../Target/AArch64/AArch64ISelLowering.cpp | 29 +++++++++----------
.../CodeGen/AArch64/can-load-speculatively.ll | 14 ++++-----
llvm/test/Verifier/can-load-speculatively.ll | 19 ------------
llvm/test/Verifier/speculative-load.ll | 2 --
6 files changed, 21 insertions(+), 64 deletions(-)
delete mode 100644 llvm/test/Verifier/can-load-speculatively.ll
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 4f702fd27d6f0..743bb4c1d5eb1 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5155,16 +5155,13 @@ void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
EVT VT = TLI.getValueType(DAG.getDataLayout(), I.getType());
Align Alignment = I.getParamAlign(0).valueOrOne();
AAMDNodes AAInfo = I.getAAMetadata();
- TypeSize StoreSize = VT.getStoreSize();
SDValue InChain = DAG.getRoot();
// Use MOLoad but NOT MODereferenceable - the memory may not be
// fully dereferenceable.
MachineMemOperand::Flags MMOFlags = MachineMemOperand::MOLoad;
- LocationSize LocSize = StoreSize.isScalable()
- ? LocationSize::beforeOrAfterPointer()
- : LocationSize::upperBound(StoreSize);
+ LocationSize LocSize = LocationSize::upperBound(VT.getStoreSize());
MachineMemOperand *MMO = DAG.getMachineFunction().getMachineMemOperand(
MachinePointerInfo(PtrOperand), MMOFlags, LocSize, Alignment, AAInfo);
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 81b61187fc8f9..8b51914a058b8 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -6957,12 +6957,6 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
// Direct form: (ptr, i1 from_end, i64 num_accessible_bytes)
Check(NumArgs == 3,
"llvm.speculative.load direct form has too many arguments", &Call);
- if (auto *CI = dyn_cast<ConstantInt>(PayloadArg)) {
- Check(Size.isScalable() || CI->getZExtValue() <= Size.getFixedValue(),
- "llvm.speculative.load num_accessible_bytes must not exceed "
- "the result size in bytes",
- &Call);
- }
} else {
// Oracle form: (ptr, i1 from_end, oracle_fn_ptr, args...)
auto *OracleFn = dyn_cast<Function>(PayloadArg);
@@ -6995,16 +6989,6 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
}
break;
}
- case Intrinsic::can_load_speculatively: {
- // If size is a constant, verify it's a positive power of 2.
- if (auto *SizeCI = dyn_cast<ConstantInt>(Call.getArgOperand(1))) {
- uint64_t Size = SizeCI->getZExtValue();
- Check(Size > 0 && isPowerOf2_64(Size),
- "llvm.can.load.speculatively size must be a positive power of 2",
- &Call);
- }
- break;
- }
case Intrinsic::vector_insert: {
Value *Vec = Call.getArgOperand(0);
Value *SubVec = Call.getArgOperand(1);
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
index 5f673b83fdb01..dd2d7d45e66b4 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
@@ -31389,9 +31389,8 @@ Value *AArch64TargetLowering::emitStoreConditional(IRBuilderBase &Builder,
Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
Value *Ptr,
Value *Size) const {
- unsigned AS = cast<PointerType>(Ptr->getType())->getAddressSpace();
// Conservatively only allow speculation for address space 0.
- if (AS != 0)
+ if (cast<PointerType>(Ptr->getType())->getAddressSpace() != 0)
return nullptr;
// For power-of-2 sizes <= 16, emit alignment check: (ptr & (size - 1)) == 0.
// If the pointer is aligned to at least 'size' bytes, loading 'size' bytes
@@ -31402,9 +31401,8 @@ Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
// The alignment check only works for power-of-2 sizes. For non-power-of-2
// sizes, we conservatively return false.
const DataLayout &DL = Builder.GetInsertBlock()->getModule()->getDataLayout();
+ Type *AddrTy = DL.getAddressType(Ptr->getType());
- unsigned PtrBits = DL.getPointerSizeInBits(AS);
- Type *IntPtrTy = Builder.getIntNTy(PtrBits);
if (auto *CI = dyn_cast<ConstantInt>(Size)) {
uint64_t SizeVal = CI->getZExtValue();
assert(isPowerOf2_64(SizeVal) && "size must be power-of-two");
@@ -31413,25 +31411,24 @@ Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
return nullptr;
// Power-of-2 constant size <= 16: use fast alignment check.
- Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
- Value *Mask = ConstantInt::get(IntPtrTy, SizeVal - 1);
- Value *Masked = Builder.CreateAnd(PtrInt, Mask);
- return Builder.CreateICmpEQ(Masked, ConstantInt::get(IntPtrTy, 0));
+ Value *PtrAddr = Builder.CreatePtrToAddr(Ptr);
+
+ Value *Mask = ConstantInt::get(AddrTy, SizeVal - 1);
+ Value *Masked = Builder.CreateAnd(PtrAddr, Mask);
+ return Builder.CreateICmpEQ(Masked, ConstantInt::get(AddrTy, 0));
}
// Check power-of-2 size <= 16 and alignment.
- Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
- Value *SizeExt = Builder.CreateZExtOrTrunc(Size, IntPtrTy);
+ Value *PtrAddr = Builder.CreatePtrToAddr(Ptr);
+ Value *SizeExt = Builder.CreateZExtOrTrunc(Size, AddrTy);
Value *SizeLE16 =
- Builder.CreateICmpULE(SizeExt, ConstantInt::get(IntPtrTy, 16));
+ Builder.CreateICmpULE(SizeExt, ConstantInt::get(AddrTy, 16));
// alignment check: (ptr & (size - 1)) == 0
- Value *SizeMinusOne =
- Builder.CreateSub(SizeExt, ConstantInt::get(IntPtrTy, 1));
- Value *Masked = Builder.CreateAnd(PtrInt, SizeMinusOne);
- Value *AlignCheck =
- Builder.CreateICmpEQ(Masked, ConstantInt::get(IntPtrTy, 0));
+ Value *SizeMinusOne = Builder.CreateSub(SizeExt, ConstantInt::get(AddrTy, 1));
+ Value *Masked = Builder.CreateAnd(PtrAddr, SizeMinusOne);
+ Value *AlignCheck = Builder.CreateICmpEQ(Masked, ConstantInt::get(AddrTy, 0));
return Builder.CreateAnd(SizeLE16, AlignCheck);
}
diff --git a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
index b6679f22b0989..c9d1bd10c1a10 100644
--- a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
+++ b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
@@ -8,7 +8,7 @@
define i1 @can_load_speculatively_16(ptr %ptr) {
; CHECK-LABEL: @can_load_speculatively_16(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[PTR:%.*]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
; CHECK-NEXT: ret i1 [[TMP3]]
@@ -20,7 +20,7 @@ define i1 @can_load_speculatively_16(ptr %ptr) {
define i1 @can_load_speculatively_8_ptr_aligned(ptr align 8 %ptr) {
; CHECK-LABEL: @can_load_speculatively_8_ptr_aligned(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[PTR:%.*]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
; CHECK-NEXT: ret i1 [[TMP3]]
@@ -31,7 +31,7 @@ define i1 @can_load_speculatively_8_ptr_aligned(ptr align 8 %ptr) {
define i1 @can_load_speculatively_16_ptr_aligned(ptr align 16 %ptr) {
; CHECK-LABEL: @can_load_speculatively_16_ptr_aligned(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[PTR:%.*]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
; CHECK-NEXT: ret i1 [[TMP3]]
@@ -42,7 +42,7 @@ define i1 @can_load_speculatively_16_ptr_aligned(ptr align 16 %ptr) {
define i1 @can_load_speculatively_16_ptr_aligned2(ptr align 16 %ptr) {
; CHECK-LABEL: @can_load_speculatively_16_ptr_aligned2(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[PTR:%.*]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
; CHECK-NEXT: ret i1 [[TMP3]]
@@ -53,7 +53,7 @@ define i1 @can_load_speculatively_16_ptr_aligned2(ptr align 16 %ptr) {
define i1 @can_load_speculatively_32_ptr_aligned(ptr align 32 %ptr) {
; CHECK-LABEL: @can_load_speculatively_32_ptr_aligned(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[PTR:%.*]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
; CHECK-NEXT: ret i1 [[TMP3]]
@@ -92,7 +92,7 @@ define i1 @can_load_speculatively_addrspace1(ptr addrspace(1) %ptr) {
; Test size 8 (within limit, power-of-2)
define i1 @can_load_speculatively_8(ptr %ptr) {
; CHECK-LABEL: @can_load_speculatively_8(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[PTR:%.*]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 7
; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
; CHECK-NEXT: ret i1 [[TMP3]]
@@ -104,7 +104,7 @@ define i1 @can_load_speculatively_8(ptr %ptr) {
; Test with runtime size - checks size <= 16 and alignment
define i1 @can_load_speculatively_runtime(ptr %ptr, i64 %size) {
; CHECK-LABEL: @can_load_speculatively_runtime(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[PTR:%.*]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = icmp ule i64 [[SIZE:%.*]], 16
; CHECK-NEXT: [[TMP3:%.*]] = sub i64 [[SIZE]], 1
; CHECK-NEXT: [[TMP4:%.*]] = and i64 [[TMP1]], [[TMP3]]
diff --git a/llvm/test/Verifier/can-load-speculatively.ll b/llvm/test/Verifier/can-load-speculatively.ll
deleted file mode 100644
index d2d69f70cfb60..0000000000000
--- a/llvm/test/Verifier/can-load-speculatively.ll
+++ /dev/null
@@ -1,19 +0,0 @@
-; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
-
-declare i1 @llvm.can.load.speculatively.p0(ptr, i64)
-
-; Test that constant size must be a positive power of 2
-
-define i1 @test_size_zero(ptr %ptr) {
-; CHECK: llvm.can.load.speculatively size must be a positive power of 2
-; CHECK-NEXT: %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 0)
- %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 0)
- ret i1 %res
-}
-
-define i1 @test_non_power_of_2(ptr %ptr) {
-; CHECK: llvm.can.load.speculatively size must be a positive power of 2
-; CHECK-NEXT: %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 3)
- %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 3)
- ret i1 %res
-}
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
index 7fd2d340679aa..f5e3f484d77c0 100644
--- a/llvm/test/Verifier/speculative-load.ll
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -80,8 +80,6 @@ define <4 x i32> @test_oracle_side_effects(ptr %ptr, i64 %n) {
}
define <4 x i32> @test_num_accessible_bytes_exceeds_size(ptr %ptr) {
-; CHECK: llvm.speculative.load num_accessible_bytes must not exceed the result size in bytes
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 32)
%res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 32)
ret <4 x i32> %res
}
>From 99057b8b939e972d108a9818f1d296b84f1369dc Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Thu, 14 May 2026 20:50:57 +0100
Subject: [PATCH 11/14] !fixup refine LanfRef
---
llvm/docs/LangRef.rst | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index ad5d2dadf8092..50c761756b8e2 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28359,9 +28359,9 @@ Overview:
The '``llvm.speculative.load``' intrinsic loads a value from memory. Unlike a
regular load, the memory access may extend beyond the bounds of the allocated
-object, provided the pointer has been verified by
-:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure the
-access is valid.
+object, provided the memory safely accessed on the underlying hardware.
+:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` can be used to
+check if the access is.
Arguments:
""""""""""
@@ -28439,7 +28439,9 @@ This intrinsic has **target-dependent** semantics. It returns ``true`` if
``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for all non-negative
integers ``I`` where the computed address does not wrap around the address
space, can be loaded speculatively, even if the memory is beyond the bounds of
-an allocated object. It returns ``false`` otherwise.
+an allocated object. It returns ``false`` otherwise. The first byte of each
+access must be part of the same underlying object as ``ptr`` in order to
+speculatively load the whole range.
The specific conditions under which this intrinsic returns ``true`` are
determined by the target. For example, a target may check whether the pointer
>From 58c735d95ee5d33bdc8bf199f0ebaed75f9cc13f Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Fri, 15 May 2026 13:52:46 +0100
Subject: [PATCH 12/14] !fixup first or last byte deref
---
llvm/docs/LangRef.rst | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 50c761756b8e2..eec8af472f16f 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28439,8 +28439,8 @@ This intrinsic has **target-dependent** semantics. It returns ``true`` if
``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for all non-negative
integers ``I`` where the computed address does not wrap around the address
space, can be loaded speculatively, even if the memory is beyond the bounds of
-an allocated object. It returns ``false`` otherwise. The first byte of each
-access must be part of the same underlying object as ``ptr`` in order to
+an allocated object. It returns ``false`` otherwise. The first or last byte of
+each access must be part of the same underlying object as ``ptr`` in order to
speculatively load the whole range.
The specific conditions under which this intrinsic returns ``true`` are
>From 98a0c8157300ffd70d255a908a1db8702c1e60bd Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Fri, 15 May 2026 15:10:07 +0100
Subject: [PATCH 13/14] !fixup support byte type in addition to vector type
---
llvm/docs/LangRef.rst | 36 +++--
llvm/include/llvm/IR/Intrinsics.td | 5 +-
llvm/lib/IR/Verifier.cpp | 22 ++-
.../CodeGen/AArch64/can-load-speculatively.ll | 13 +-
.../AArch64/speculative-load-intrinsic.ll | 105 ++++++------
.../CodeGen/X86/speculative-load-intrinsic.ll | 148 ++++++++++-------
llvm/test/Verifier/speculative-load.ll | 153 ++++++++++++------
7 files changed, 289 insertions(+), 193 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index eec8af472f16f..50935998a1760 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28348,30 +28348,32 @@ This is an overloaded intrinsic.
::
; Direct form: number of accessible bytes given as i64
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
- declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
+ declare b128 @llvm.speculative.load.b128.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
+ declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
+ declare <vscale x 4 x i32>
+ @llvm.speculative.load.nxv4i32.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
; Oracle form: accessible bytes computed by calling oracle_fn(args...)
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i1 <from_end>, ptr <oracle_fn>, ...)
+ declare b128 @llvm.speculative.load.b128.p0(ptr <ptr>, i1 <from_end>, ptr <oracle_fn>, ...)
Overview:
"""""""""
The '``llvm.speculative.load``' intrinsic loads a value from memory. Unlike a
regular load, the memory access may extend beyond the bounds of the allocated
-object, provided the memory safely accessed on the underlying hardware.
+object, provided the memory can be safely accessed on the underlying hardware.
:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` can be used to
-check if the access is.
+check if the access is safe.
Arguments:
""""""""""
The first argument is a pointer to the memory location to load from. The return
-type must be a vector type with a power-of-2 size in bytes. The second argument
-is an ``i1`` constant flag ``from_end`` that specifies whether the ``N``
-accessible bytes are counted from the start or the end of the loaded values (see
-Semantics). The remaining arguments determine the *number of accessible bytes*,
-denoted ``N`` below.
+type must be a byte type or a vector type, and its size in bytes must be a
+positive power of 2. The second argument is an ``i1`` constant flag ``from_end``
+selecting whether the ``N`` accessible bytes are counted from the start or end
+of the loaded value (see Semantics). The remaining arguments determine the
+*number of accessible bytes*, denoted ``N`` below.
In the **direct form**, the third argument is an ``i64`` specifying ``N``
directly. In the **oracle form**, the third argument must be a direct
@@ -28382,8 +28384,7 @@ arguments are forwarded to it, and its return value is ``N``.
Semantics:
""""""""""
-Let ``S`` denote the size of the return type in bytes. The intrinsic performs
-a load of ``S`` bytes starting from ``ptr``.
+Let ``S`` denote the size of the return type in bytes.
When ``from_end`` is ``false``, the first ``N`` bytes (offsets ``[0, N)``)
are the stored values read from memory. Bytes at offsets ``[N, S)`` are
@@ -28398,9 +28399,10 @@ allocated object that ``ptr`` is :ref:`based <pointeraliasing>` on, and
poison bytes are not considered accessed for the purposes of data races or
``noalias`` constraints. The behavior is undefined if ``N`` exceeds ``S``.
-The behavior is undefined if the speculative load accesses memory that would
-fault (i.e., the oracle or ``llvm.can.load.speculatively`` would indicate the
-access is not safe).
+The behavior is undefined if any byte the underlying load actually reads is
+not safe to speculatively access.
+:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` can be used
+to check safety.
.. _int_can_load_speculatively:
@@ -28428,8 +28430,8 @@ Arguments:
The first argument is a pointer to the memory location.
-The second argument is an i64 specifying the size in bytes of the load.
-The size must be a positive power of 2. If the size is not a power-of-2, the
+The second argument is an i64 specifying the number of accessed bytes,
+and must be a positive power of 2. If the size is not a power-of-2, the
result is ``poison``.
Semantics:
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index fe2a4b81c1f22..38c4913e8f392 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2714,15 +2714,14 @@ def int_masked_srem:
LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>],
[IntrNoMem]>;
-// Speculatively load a value from memory; lowers to a regular aligned load.
+// Speculatively load a value from memory.
def int_speculative_load:
- DefaultAttrsIntrinsic<[llvm_anyvector_ty],
+ DefaultAttrsIntrinsic<[llvm_any_ty],
[llvm_anyptr_ty, llvm_i1_ty, llvm_vararg_ty],
[IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>,
ImmArg<ArgIndex<1>>]>;
// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
-// The size can be a runtime value to support scalable vectors.
def int_can_load_speculatively:
DefaultAttrsIntrinsic<[llvm_i1_ty],
[llvm_anyptr_ty, llvm_i64_ty],
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 8b51914a058b8..ce1f02c490a35 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -6943,10 +6943,24 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
}
case Intrinsic::speculative_load: {
Type *LoadTy = Call.getType();
- TypeSize Size = DL.getTypeStoreSize(LoadTy);
- // For scalable vectors, check the known minimum size is a power of 2.
- Check(Size.getKnownMinValue() > 0 && isPowerOf2_64(Size.getKnownMinValue()),
- "llvm.speculative.load type must have a power-of-2 size", &Call);
+ Check(LoadTy->isByteTy() || LoadTy->isVectorTy(),
+ "llvm.speculative.load return type must be a byte type or a "
+ "vector type",
+ &Call);
+ if (LoadTy->isByteOrByteVectorTy()) {
+ unsigned BitWidth = LoadTy->getScalarType()->getByteBitWidth();
+ Check((BitWidth % 8) == 0,
+ "llvm.speculative.load byte type must have a bit width that is "
+ "a multiple of 8",
+ &Call);
+ }
+
+ TypeSize SizeInBits = LoadTy->getPrimitiveSizeInBits();
+ uint64_t MinBits = SizeInBits.getKnownMinValue();
+ Check(MinBits != 0 && (MinBits % 8) == 0 && isPowerOf2_64(MinBits / 8),
+ "llvm.speculative.load return type size in bytes must be a "
+ "positive power of 2",
+ &Call);
unsigned NumArgs = Call.arg_size();
Check(NumArgs >= 3, "llvm.speculative.load requires at least 3 arguments",
diff --git a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
index c9d1bd10c1a10..0b1225a4bd0e3 100644
--- a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
+++ b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
@@ -2,9 +2,8 @@
; RUN: opt -mtriple=aarch64-unknown-linux-gnu -passes=pre-isel-intrinsic-lowering -S < %s | FileCheck %s
; Test that @llvm.can.load.speculatively is lowered to an alignment check
-; for power-of-2 sizes <= 16 bytes on AArch64, and returns false for larger sizes.
+; for power-of-2 sizes <= 16 bytes on AArch64, and returns false otherwise.
; The 16-byte limit ensures correctness with MTE (memory tagging).
-; Note: non-power-of-2 constant sizes are rejected by the verifier.
define i1 @can_load_speculatively_16(ptr %ptr) {
; CHECK-LABEL: @can_load_speculatively_16(
@@ -101,16 +100,16 @@ define i1 @can_load_speculatively_8(ptr %ptr) {
ret i1 %can_load
}
-; Test with runtime size - checks size <= 16 and alignment
+; Test with runtime size - checks size <= 16, power-of-2, and alignment
define i1 @can_load_speculatively_runtime(ptr %ptr, i64 %size) {
; CHECK-LABEL: @can_load_speculatively_runtime(
; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[PTR:%.*]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = icmp ule i64 [[SIZE:%.*]], 16
; CHECK-NEXT: [[TMP3:%.*]] = sub i64 [[SIZE]], 1
-; CHECK-NEXT: [[TMP4:%.*]] = and i64 [[TMP1]], [[TMP3]]
-; CHECK-NEXT: [[TMP5:%.*]] = icmp eq i64 [[TMP4]], 0
-; CHECK-NEXT: [[TMP6:%.*]] = and i1 [[TMP2]], [[TMP5]]
-; CHECK-NEXT: ret i1 [[TMP6]]
+; CHECK-NEXT: [[TMP8:%.*]] = and i64 [[TMP1]], [[TMP3]]
+; CHECK-NEXT: [[TMP9:%.*]] = icmp eq i64 [[TMP8]], 0
+; CHECK-NEXT: [[TMP11:%.*]] = and i1 [[TMP2]], [[TMP9]]
+; CHECK-NEXT: ret i1 [[TMP11]]
;
%can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 %size)
ret i1 %can_load
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
index a547cee576864..e407fbc4adae2 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -2,9 +2,30 @@
; RUN: llc -mtriple=aarch64-unknown-linux-gnu -mattr=+sve < %s | FileCheck %s
; Test that @llvm.speculative.load is lowered to a regular load
-; in SelectionDAG for fixed vectors, scalable vectors, and scalars.
+; in SelectionDAG for fixed vectors, scalable vectors, and bytes.
-; Fixed-width vector tests
+; Byte-type tests
+
+define b128 @speculative_load_b128(ptr %ptr) {
+; CHECK-LABEL: speculative_load_b128:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldp x8, x1, [x0]
+; CHECK-NEXT: mov x0, x8
+; CHECK-NEXT: ret
+ %load = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr align 16 %ptr, i1 false, i64 12)
+ ret b128 %load
+}
+
+define b64 @speculative_load_b64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_b64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr x0, [x0]
+; CHECK-NEXT: ret
+ %load = call b64 (ptr, i1, ...) @llvm.speculative.load.b64.p0(ptr align 8 %ptr, i1 false, i64 8)
+ ret b64 %load
+}
+
+; Fixed-vector tests
define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
; CHECK-LABEL: speculative_load_v4i32:
@@ -48,62 +69,46 @@ define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
declare i64 @oracle(ptr, i64) memory(argmem: read)
-define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v4i32_oracle:
+define b128 @speculative_load_b128_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_b128_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ldp x8, x1, [x0]
+; CHECK-NEXT: mov x0, x8
; CHECK-NEXT: ret
- %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <4 x i32> %load
-}
-
-define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v2i64_oracle:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr q0, [x0]
-; CHECK-NEXT: ret
- %load = call <2 x i64> (ptr, i1, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <2 x i64> %load
+ %load = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
+ ret b128 %load
}
-define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v8i16_oracle:
+define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <8 x i16> (ptr, i1, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <8 x i16> %load
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
}
-define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v16i8_oracle:
+define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_nxv4i32_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <16 x i8> (ptr, i1, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <16 x i8> %load
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
+ ret <vscale x 4 x i32> %load
}
-define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v4f32_oracle:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr q0, [x0]
-; CHECK-NEXT: ret
- %load = call <4 x float> (ptr, i1, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <4 x float> %load
-}
+; from_end tests
-define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v2f64_oracle:
+define b128 @speculative_load_b128_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_b128_from_end:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ldp x8, x1, [x0]
+; CHECK-NEXT: mov x0, x8
; CHECK-NEXT: ret
- %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <2 x double> %load
+ %load = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr align 16 %ptr, i1 true, i64 12)
+ ret b128 %load
}
-; from_end tests
-
define <4 x i32> @speculative_load_v4i32_from_end(ptr %ptr) {
; CHECK-LABEL: speculative_load_v4i32_from_end:
; CHECK: // %bb.0:
@@ -122,13 +127,14 @@ define <vscale x 4 x i32> @speculative_load_nxv4i32_from_end(ptr %ptr) {
ret <vscale x 4 x i32> %load
}
-define <4 x i32> @speculative_load_v4i32_oracle_from_end(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v4i32_oracle_from_end:
+define b128 @speculative_load_b128_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_b128_oracle_from_end:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ldp x8, x1, [x0]
+; CHECK-NEXT: mov x0, x8
; CHECK-NEXT: ret
- %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
- ret <4 x i32> %load
+ %load = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret b128 %load
}
define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle_from_end(ptr %ptr, i64 %n) {
@@ -139,12 +145,3 @@ define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle_from_end(ptr %ptr, i6
%load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
ret <vscale x 4 x i32> %load
}
-
-define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_nxv4i32_oracle:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <vscale x 4 x i32> %load
-}
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index fa345408940d1..60ad671321777 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -2,7 +2,42 @@
; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2 < %s | FileCheck %s
; Test that @llvm.speculative.load is lowered to a regular load
-; in SelectionDAG.
+; in SelectionDAG for fixed vectors and bytes.
+
+; Byte-type tests
+
+define b128 @speculative_load_b128(ptr %ptr) {
+; CHECK-LABEL: speculative_load_b128:
+; CHECK: # %bb.0:
+; CHECK-NEXT: movq (%rdi), %rax
+; CHECK-NEXT: movq 8(%rdi), %rdx
+; CHECK-NEXT: retq
+ %load = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr align 16 %ptr, i1 false, i64 8)
+ ret b128 %load
+}
+
+define b256 @speculative_load_b256(ptr %ptr) {
+; CHECK-LABEL: speculative_load_b256:
+; CHECK: # %bb.0:
+; CHECK-NEXT: movq %rdi, %rax
+; CHECK-NEXT: vmovaps (%rsi), %ymm0
+; CHECK-NEXT: vmovups %ymm0, (%rdi)
+; CHECK-NEXT: vzeroupper
+; CHECK-NEXT: retq
+ %load = call b256 (ptr, i1, ...) @llvm.speculative.load.b256.p0(ptr align 32 %ptr, i1 false, i64 24)
+ ret b256 %load
+}
+
+define b64 @speculative_load_b64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_b64:
+; CHECK: # %bb.0:
+; CHECK-NEXT: movq (%rdi), %rax
+; CHECK-NEXT: retq
+ %load = call b64 (ptr, i1, ...) @llvm.speculative.load.b64.p0(ptr align 8 %ptr, i1 false, i64 8)
+ ret b64 %load
+}
+
+; Fixed-vector tests
define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
; CHECK-LABEL: speculative_load_v4i32:
@@ -22,83 +57,82 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
ret <8 x i32> %load
}
-; Oracle form tests
-declare i64 @oracle(ptr, i64) memory(argmem: read)
-
-define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v4i32_oracle:
+define <2 x double> @speculative_load_v2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v2f64:
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <4 x i32> %load
+ %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, i64 8)
+ ret <2 x double> %load
}
-define <8 x i32> @speculative_load_v8i32_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v8i32_oracle:
-; CHECK: # %bb.0:
-; CHECK-NEXT: vmovaps (%rdi), %ymm0
-; CHECK-NEXT: retq
- %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <8 x i32> %load
-}
+; Oracle form tests
+declare i64 @oracle(ptr, i64) memory(argmem: read)
-define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v2i64_oracle:
+define b128 @speculative_load_b128_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_b128_oracle:
; CHECK: # %bb.0:
-; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: movq (%rdi), %rax
+; CHECK-NEXT: movq 8(%rdi), %rdx
; CHECK-NEXT: retq
- %load = call <2 x i64> (ptr, i1, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <2 x i64> %load
+ %load = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
+ ret b128 %load
}
-define <4 x i64> @speculative_load_v4i64_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v4i64_oracle:
+define b256 @speculative_load_b256_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_b256_oracle:
; CHECK: # %bb.0:
-; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: movq %rdi, %rax
+; CHECK-NEXT: vmovaps (%rsi), %ymm0
+; CHECK-NEXT: vmovups %ymm0, (%rdi)
+; CHECK-NEXT: vzeroupper
; CHECK-NEXT: retq
- %load = call <4 x i64> (ptr, i1, ...) @llvm.speculative.load.v4i64.p0(ptr align 32 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <4 x i64> %load
+ %load = call b256 (ptr, i1, ...) @llvm.speculative.load.b256.p0(ptr align 32 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
+ ret b256 %load
}
-define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v4f32_oracle:
+define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle:
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <4 x float> (ptr, i1, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <4 x float> %load
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
}
-define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v2f64_oracle:
+define <8 x i32> @speculative_load_v8i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i32_oracle:
; CHECK: # %bb.0:
-; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
; CHECK-NEXT: retq
- %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <2 x double> %load
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i32> %load
}
-define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v8i16_oracle:
+; from_end tests
+
+define b128 @speculative_load_b128_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_b128_from_end:
; CHECK: # %bb.0:
-; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: movq (%rdi), %rax
+; CHECK-NEXT: movq 8(%rdi), %rdx
; CHECK-NEXT: retq
- %load = call <8 x i16> (ptr, i1, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <8 x i16> %load
+ %load = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr align 16 %ptr, i1 true, i64 8)
+ ret b128 %load
}
-define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v16i8_oracle:
+define b256 @speculative_load_b256_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_b256_from_end:
; CHECK: # %bb.0:
-; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: movq %rdi, %rax
+; CHECK-NEXT: vmovaps (%rsi), %ymm0
+; CHECK-NEXT: vmovups %ymm0, (%rdi)
+; CHECK-NEXT: vzeroupper
; CHECK-NEXT: retq
- %load = call <16 x i8> (ptr, i1, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <16 x i8> %load
+ %load = call b256 (ptr, i1, ...) @llvm.speculative.load.b256.p0(ptr align 32 %ptr, i1 true, i64 24)
+ ret b256 %load
}
-; from_end tests
-
define <4 x i32> @speculative_load_v4i32_from_end(ptr %ptr) {
; CHECK-LABEL: speculative_load_v4i32_from_end:
; CHECK: # %bb.0:
@@ -108,13 +142,14 @@ define <4 x i32> @speculative_load_v4i32_from_end(ptr %ptr) {
ret <4 x i32> %load
}
-define <8 x i32> @speculative_load_v8i32_from_end(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v8i32_from_end:
+define b128 @speculative_load_b128_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_b128_oracle_from_end:
; CHECK: # %bb.0:
-; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: movq (%rdi), %rax
+; CHECK-NEXT: movq 8(%rdi), %rdx
; CHECK-NEXT: retq
- %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 true, i64 24)
- ret <8 x i32> %load
+ %load = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret b128 %load
}
define <4 x i32> @speculative_load_v4i32_oracle_from_end(ptr %ptr, i64 %n) {
@@ -125,12 +160,3 @@ define <4 x i32> @speculative_load_v4i32_oracle_from_end(ptr %ptr, i64 %n) {
%load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x i32> %load
}
-
-define <8 x i32> @speculative_load_v8i32_oracle_from_end(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v8i32_oracle_from_end:
-; CHECK: # %bb.0:
-; CHECK-NEXT: vmovaps (%rdi), %ymm0
-; CHECK-NEXT: retq
- %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
- ret <8 x i32> %load
-}
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
index f5e3f484d77c0..fb9d06a10f7eb 100644
--- a/llvm/test/Verifier/speculative-load.ll
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -1,85 +1,144 @@
; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i1, ...)
+declare b128 @llvm.speculative.load.b128.p0(ptr, i1, ...)
+declare i32 @llvm.speculative.load.i32.p0(ptr, i1, ...)
+declare float @llvm.speculative.load.f32.p0(ptr, i1, ...)
+declare b12 @llvm.speculative.load.b12.p0(ptr, i1, ...)
+declare b1 @llvm.speculative.load.b1.p0(ptr, i1, ...)
+declare b9 @llvm.speculative.load.b9.p0(ptr, i1, ...)
+declare b24 @llvm.speculative.load.b24.p0(ptr, i1, ...)
declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, i1, ...)
-declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, i1, ...)
+declare {i32, i32} @llvm.speculative.load.sl_i32i32s.p0(ptr, i1, ...)
+declare [4 x i32] @llvm.speculative.load.a4i32.p0(ptr, i1, ...)
+declare <4 x b3> @llvm.speculative.load.v4b3.p0(ptr, i1, ...)
declare i32 @bad_oracle_ret(ptr, i64) memory(argmem: read)
declare i64 @good_oracle(ptr, i64) memory(argmem: read)
declare i64 @oracle_i32_param(i32) memory(argmem: read)
declare i64 @side_effecting_oracle(ptr, i64)
-define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
-; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <3 x i32> (ptr, i1, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i1 false, i64 0)
- %res = call <3 x i32> (ptr, i1, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i1 false, i64 0)
- ret <3 x i32> %res
+define i32 @test_non_byte_non_vector_int(ptr %ptr) {
+; CHECK: llvm.speculative.load return type must be a byte type or a vector type
+; CHECK-NEXT: %res = call i32 (ptr, i1, ...) @llvm.speculative.load.i32.p0(ptr %ptr, i1 false, i64 0)
+ %res = call i32 (ptr, i1, ...) @llvm.speculative.load.i32.p0(ptr %ptr, i1 false, i64 0)
+ ret i32 %res
+}
+
+define float @test_non_byte_non_vector_float(ptr %ptr) {
+; CHECK: llvm.speculative.load return type must be a byte type or a vector type
+; CHECK-NEXT: %res = call float (ptr, i1, ...) @llvm.speculative.load.f32.p0(ptr %ptr, i1 false, i64 0)
+ %res = call float (ptr, i1, ...) @llvm.speculative.load.f32.p0(ptr %ptr, i1 false, i64 0)
+ ret float %res
+}
+
+define {i32, i32} @test_struct(ptr %ptr) {
+; CHECK: llvm.speculative.load return type must be a byte type or a vector type
+; CHECK-NEXT: %res = call { i32, i32 } (ptr, i1, ...) @llvm.speculative.load.sl_i32i32s.p0(ptr %ptr, i1 false, i64 0)
+ %res = call {i32, i32} (ptr, i1, ...) @llvm.speculative.load.sl_i32i32s.p0(ptr %ptr, i1 false, i64 0)
+ ret {i32, i32} %res
+}
+
+define [4 x i32] @test_array(ptr %ptr) {
+; CHECK: llvm.speculative.load return type must be a byte type or a vector type
+; CHECK-NEXT: %res = call [4 x i32] (ptr, i1, ...) @llvm.speculative.load.a4i32.p0(ptr %ptr, i1 false, i64 0)
+ %res = call [4 x i32] (ptr, i1, ...) @llvm.speculative.load.a4i32.p0(ptr %ptr, i1 false, i64 0)
+ ret [4 x i32] %res
+}
+
+define b1 @test_byte_b1(ptr %ptr) {
+; CHECK: llvm.speculative.load byte type must have a bit width that is a multiple of 8
+; CHECK-NEXT: %res = call b1 (ptr, i1, ...) @llvm.speculative.load.b1.p0(ptr %ptr, i1 false, i64 0)
+ %res = call b1 (ptr, i1, ...) @llvm.speculative.load.b1.p0(ptr %ptr, i1 false, i64 0)
+ ret b1 %res
+}
+
+define b9 @test_byte_b9(ptr %ptr) {
+; CHECK: llvm.speculative.load byte type must have a bit width that is a multiple of 8
+; CHECK-NEXT: %res = call b9 (ptr, i1, ...) @llvm.speculative.load.b9.p0(ptr %ptr, i1 false, i64 0)
+ %res = call b9 (ptr, i1, ...) @llvm.speculative.load.b9.p0(ptr %ptr, i1 false, i64 0)
+ ret b9 %res
+}
+
+define b12 @test_byte_not_multiple_of_8(ptr %ptr) {
+; CHECK: llvm.speculative.load byte type must have a bit width that is a multiple of 8
+; CHECK-NEXT: %res = call b12 (ptr, i1, ...) @llvm.speculative.load.b12.p0(ptr %ptr, i1 false, i64 0)
+ %res = call b12 (ptr, i1, ...) @llvm.speculative.load.b12.p0(ptr %ptr, i1 false, i64 0)
+ ret b12 %res
}
-define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
-; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <vscale x 3 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i1 false, i64 0)
- %res = call <vscale x 3 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i1 false, i64 0)
- ret <vscale x 3 x i32> %res
+define <4 x b3> @test_byte_vector_not_multiple_of_8(ptr %ptr) {
+; CHECK: llvm.speculative.load byte type must have a bit width that is a multiple of 8
+; CHECK-NEXT: %res = call <4 x b3> (ptr, i1, ...) @llvm.speculative.load.v4b3.p0(ptr %ptr, i1 false, i64 0)
+ %res = call <4 x b3> (ptr, i1, ...) @llvm.speculative.load.v4b3.p0(ptr %ptr, i1 false, i64 0)
+ ret <4 x b3> %res
}
-define <4 x i32> @test_too_few_args(ptr %ptr) {
+define b128 @test_too_few_args(ptr %ptr) {
; CHECK: llvm.speculative.load requires at least 3 arguments
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 true)
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 true)
- ret <4 x i32> %res
+; CHECK-NEXT: call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 true)
+ %res = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 true)
+ ret b128 %res
}
-define <4 x i32> @test_direct_form_extra_args(ptr %ptr) {
+define b128 @test_direct_form_extra_args(ptr %ptr) {
; CHECK: llvm.speculative.load direct form has too many arguments
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 8, i64 4)
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 8, i64 4)
- ret <4 x i32> %res
+; CHECK-NEXT: call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, i64 8, i64 4)
+ %res = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, i64 8, i64 4)
+ ret b128 %res
}
-define <4 x i32> @test_oracle_wrong_return_type(ptr %ptr, i64 %n) {
+define b128 @test_oracle_wrong_return_type(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function must return i64
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
- ret <4 x i32> %res
+; CHECK-NEXT: call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+ %res = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+ ret b128 %res
}
-define <4 x i32> @test_oracle_too_few_args(ptr %ptr) {
+define b128 @test_oracle_too_few_args(ptr %ptr) {
; CHECK: llvm.speculative.load oracle function argument count mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle)
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle)
- ret <4 x i32> %res
+; CHECK-NEXT: call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @good_oracle)
+ %res = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @good_oracle)
+ ret b128 %res
}
-define <4 x i32> @test_oracle_too_many_args(ptr %ptr, i64 %n) {
+define b128 @test_oracle_too_many_args(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function argument count mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
- ret <4 x i32> %res
+; CHECK-NEXT: call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+ %res = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+ ret b128 %res
}
-define <4 x i32> @test_oracle_arg_type_mismatch(ptr %ptr) {
+define b128 @test_oracle_arg_type_mismatch(ptr %ptr) {
; CHECK: llvm.speculative.load oracle function argument type mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @oracle_i32_param, i64 42)
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @oracle_i32_param, i64 42)
- ret <4 x i32> %res
+; CHECK-NEXT: call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @oracle_i32_param, i64 42)
+ %res = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @oracle_i32_param, i64 42)
+ ret b128 %res
}
-define <4 x i32> @test_non_function_oracle(ptr %ptr, ptr %not_fn) {
+define b128 @test_non_function_oracle(ptr %ptr, ptr %not_fn) {
; CHECK: llvm.speculative.load third argument must be i64 or a direct reference to an oracle function
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr %not_fn)
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr %not_fn)
- ret <4 x i32> %res
+; CHECK-NEXT: call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr %not_fn)
+ %res = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr %not_fn)
+ ret b128 %res
}
-define <4 x i32> @test_oracle_side_effects(ptr %ptr, i64 %n) {
+define b128 @test_oracle_side_effects(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function must not have side effects and may only read memory through its arguments
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
- ret <4 x i32> %res
+; CHECK-NEXT: call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+ %res = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+ ret b128 %res
}
-define <4 x i32> @test_num_accessible_bytes_exceeds_size(ptr %ptr) {
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 32)
- ret <4 x i32> %res
+define b24 @test_byte_size_not_pow2(ptr %ptr) {
+; CHECK: llvm.speculative.load return type size in bytes must be a positive power of 2
+; CHECK-NEXT: %res = call b24 (ptr, i1, ...) @llvm.speculative.load.b24.p0(ptr %ptr, i1 false, i64 0)
+ %res = call b24 (ptr, i1, ...) @llvm.speculative.load.b24.p0(ptr %ptr, i1 false, i64 0)
+ ret b24 %res
+}
+
+define <3 x i32> @test_vector_byte_size_not_pow2(ptr %ptr) {
+; CHECK: llvm.speculative.load return type size in bytes must be a positive power of 2
+; CHECK-NEXT: %res = call <3 x i32> (ptr, i1, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i1 false, i64 0)
+ %res = call <3 x i32> (ptr, i1, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i1 false, i64 0)
+ ret <3 x i32> %res
}
>From 1a9c3a6f5bb6150de62847ac979ba61e69df9dc8 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Wed, 27 May 2026 10:50:42 +0100
Subject: [PATCH 14/14] !fixup adjust wording for can_load_speculatively.
---
llvm/docs/LangRef.rst | 10 ++++------
1 file changed, 4 insertions(+), 6 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 50935998a1760..6a3ae5de71e11 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28438,12 +28438,10 @@ Semantics:
""""""""""
This intrinsic has **target-dependent** semantics. It returns ``true`` if
-``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for all non-negative
-integers ``I`` where the computed address does not wrap around the address
-space, can be loaded speculatively, even if the memory is beyond the bounds of
-an allocated object. It returns ``false`` otherwise. The first or last byte of
-each access must be part of the same underlying object as ``ptr`` in order to
-speculatively load the whole range.
+``num_bytes`` bytes starting at ``ptr + I * num_bytes`` can be loaded
+speculatively, for all non-negative integers ``I`` where the computed address
+does not wrap around the address space and its first or last byte is part of
+the same underlying object as ``ptr``.
The specific conditions under which this intrinsic returns ``true`` are
determined by the target. For example, a target may check whether the pointer
More information about the llvm-commits
mailing list