[llvm] Provide intrinsics for speculative loads (PR #179642)
Florian Hahn via llvm-commits
llvm-commits at lists.llvm.org
Tue Jun 9 12:52:11 PDT 2026
https://github.com/fhahn updated https://github.com/llvm/llvm-project/pull/179642
>From 1576373ff56d3f764069dca2f199d35ca27b1a2e Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Wed, 4 Feb 2026 11:06:38 +0000
Subject: [PATCH 01/14] Add speculative load intrinsics
Introduce two new intrinsics to enable vectorization of loops with early
exits that have potentially faulting loads.
This has previously been discussed in
https://github.com/llvm/llvm-project/pull/120603 and is similar to
@nikic's https://hackmd.io/@nikic/S1O4QWYZkx, with the major difference
being that there is no `%defined_size` argument and instead the load
returns the stored values for the bytes within bounds and undef
otherwise. I don't think we can easily compute the defined size because
it may depend on the loaded values (i.e. at what lane the early exit has
been taken).
1. `@llvm.speculative.load` (name subject to change) - perform a load that
may access memory beyond the allocated object. It must be used in
combination with `@llvm.can.load.speculatively` to ensure the load is
guaranteed to not trap.
2. `@llvm.can.load.speculatively` - Returns true if it's safe to speculatively
load a given number of bytes from a pointer. The semantics are
target-dependent. On some targets, this may check that the access
does not cross page boundaries, or stricter checks for example on
AArch64 with MTE, which limits the access size to 16 bytes.
`@llvm.speculative.load` is lowered to a regular load in SelectionDAG
without MODereferenceable. I am not sure if we need to be more careful
than this, i.e. if we could still reason about SelectionDAG loads to
infer dereferencability for the pointer.
`@llvm.can.load.speculatively` is lowered to regular IR in PreISel
lowering, using a target-lowering hook. By default, it conservatively
expands to false.
These intrinsics should allow the loop vectorizer to vectorize early-exit
loops with potentially non-dereferenceable loads.
---
llvm/docs/LangRef.rst | 113 ++++++++++++++
llvm/include/llvm/CodeGen/TargetLowering.h | 13 ++
llvm/include/llvm/IR/Intrinsics.td | 14 ++
llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp | 39 +++++
.../SelectionDAG/SelectionDAGBuilder.cpp | 30 ++++
.../SelectionDAG/SelectionDAGBuilder.h | 1 +
llvm/lib/IR/Verifier.cpp | 18 +++
.../Target/AArch64/AArch64ISelLowering.cpp | 49 ++++++
llvm/lib/Target/AArch64/AArch64ISelLowering.h | 2 +
.../CodeGen/AArch64/can-load-speculatively.ll | 78 ++++++++++
.../AArch64/speculative-load-intrinsic-sve.ll | 66 ++++++++
.../AArch64/speculative-load-intrinsic.ll | 117 ++++++++++++++
.../CodeGen/X86/can-load-speculatively.ll | 32 ++++
.../CodeGen/X86/speculative-load-intrinsic.ll | 146 ++++++++++++++++++
llvm/test/Verifier/can-load-speculatively.ll | 19 +++
llvm/test/Verifier/speculative-load.ll | 18 +++
16 files changed, 755 insertions(+)
create mode 100644 llvm/test/CodeGen/AArch64/can-load-speculatively.ll
create mode 100644 llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
create mode 100644 llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
create mode 100644 llvm/test/CodeGen/X86/can-load-speculatively.ll
create mode 100644 llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
create mode 100644 llvm/test/Verifier/can-load-speculatively.ll
create mode 100644 llvm/test/Verifier/speculative-load.ll
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index cf052513c5ef8..b92a98e56fe2e 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28339,6 +28339,119 @@ Semantics:
Follows the same semantics as :ref:`srem <i_srem>` with the exception that disabled lanes cannot produce undefined behaviour and always result in poison.
+Speculative Load Intrinsics
+---------------------------
+
+LLVM provides intrinsics for speculatively loading memory that may be
+out-of-bounds. These intrinsics enable optimizations like early-exit loop
+vectorization where the vectorized loop may read beyond the end of an array,
+provided the access is guaranteed to not trap by target-specific checks.
+
+.. _int_speculative_load:
+
+'``llvm.speculative.load``' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic.
+
+::
+
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>)
+ declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>)
+ declare i64 @llvm.speculative.load.i64.p0(ptr <ptr>)
+
+Overview:
+"""""""""
+
+The '``llvm.speculative.load``' intrinsic loads a value from memory. Unlike a
+regular load, the memory access may
+extend beyond the bounds of the allocated object, provided the pointer has been
+verified by :ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to
+ensure the access cannot fault.
+
+Arguments:
+""""""""""
+
+The argument is a pointer to the memory location to load from. The return type
+must have a power-of-2 size in bytes.
+
+Semantics:
+""""""""""
+
+The '``llvm.speculative.load``' intrinsic performs a load that may access
+memory beyond the allocated object. It must be used in combination with
+:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure
+the access cannot fault.
+
+For bytes that are within the bounds of the allocated object, the intrinsic
+returns the stored value. For bytes that are beyond the bounds of the
+allocated object, the intrinsic returns ``undef`` for those bytes. At least the
+first accessed byte must be within the bounds of an allocated object the pointer is
+based on.
+
+The behavior is undefined if this intrinsic is used to load from a pointer
+for which ``llvm.can.load.speculatively`` returns false.
+
+.. _int_can_load_speculatively:
+
+'``llvm.can.load.speculatively``' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic.
+
+::
+
+ declare i1 @llvm.can.load.speculatively.p0(ptr <ptr>, i64 <num_bytes>)
+ declare i1 @llvm.can.load.speculatively.p1(ptr addrspace(1) <ptr>, i64 <num_bytes>)
+
+Overview:
+"""""""""
+
+The '``llvm.can.load.speculatively``' intrinsic returns true if it is safe
+to speculatively load ``num_bytes`` bytes starting from ``ptr``,
+even if the memory may be beyond the bounds of an allocated object.
+
+Arguments:
+""""""""""
+
+The first argument is a pointer to the memory location.
+
+The second argument is an i64 specifying the size in bytes of the load.
+The size must be a positive power of 2. If the size is not a power-of-2, the
+result is ``poison``.
+
+Semantics:
+""""""""""
+
+This intrinsic has **target-dependent** semantics. It returns ``true`` if
+loading ``num_bytes`` bytes from ``ptr`` is guaranteed not to trap,
+even if the memory is beyond the bounds of an allocated object. It returns
+``false`` otherwise.
+
+The specific conditions under which this intrinsic returns ``true`` are
+determined by the target. For example, a target may check whether the pointer
+alignment guarantees the load cannot cross a page boundary.
+
+.. code-block:: llvm
+
+ ; Check if we can safely load 16 bytes from %ptr
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ br i1 %can_load, label %speculative_path, label %safe_path
+
+ speculative_path:
+ ; Safe to speculatively load from %ptr
+ %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr)
+ ...
+
+ safe_path:
+ ; Fall back to masked load or scalar operations
+ ...
+
+
Memory Use Markers
------------------
diff --git a/llvm/include/llvm/CodeGen/TargetLowering.h b/llvm/include/llvm/CodeGen/TargetLowering.h
index 82c47cce0f522..d6a4f454d4b64 100644
--- a/llvm/include/llvm/CodeGen/TargetLowering.h
+++ b/llvm/include/llvm/CodeGen/TargetLowering.h
@@ -2325,6 +2325,19 @@ class LLVM_ABI TargetLoweringBase {
llvm_unreachable("Store conditional unimplemented on this target");
}
+ /// Emit code to check if a speculative load of the given size from Ptr is
+ /// safe. Returns a Value* representing the check result (i1), or nullptr
+ /// to use the default lowering (which returns false). Targets can override
+ /// to provide their own safety check (e.g., alignment-based page boundary
+ /// check).
+ /// \param Builder IRBuilder positioned at the intrinsic call site
+ /// \param Ptr the pointer operand
+ /// \param Size the size in bytes (constant or runtime value for scalable)
+ virtual Value *emitCanLoadSpeculatively(IRBuilderBase &Builder, Value *Ptr,
+ Value *Size) const {
+ return nullptr;
+ }
+
/// Perform a masked atomicrmw using a target-specific intrinsic. This
/// represents the core LL/SC loop which will be lowered at a late stage by
/// the backend. The target-specific intrinsic returns the loaded value and
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index b1b2bb2a72c65..ad5e864bd3864 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2713,6 +2713,20 @@ def int_masked_srem:
LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>],
[IntrNoMem]>;
+// Speculatively load a value from memory; lowers to a regular aligned load.
+// The loaded type must have a power-of-2 size.
+def int_speculative_load:
+ DefaultAttrsIntrinsic<[llvm_any_ty],
+ [llvm_anyptr_ty],
+ [IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
+
+// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
+// The size can be a runtime value to support scalable vectors.
+def int_can_load_speculatively:
+ DefaultAttrsIntrinsic<[llvm_i1_ty],
+ [llvm_anyptr_ty, llvm_i64_ty],
+ [IntrNoMem, IntrSpeculatable, IntrWillReturn]>;
+
// Test whether a pointer is associated with a type metadata identifier.
def int_type_test : DefaultAttrsIntrinsic<[llvm_i1_ty],
[llvm_ptr_ty, llvm_metadata_ty],
diff --git a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
index 2fa92b61bb297..06a65e7375f00 100644
--- a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
+++ b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
@@ -136,6 +136,42 @@ static bool lowerLoadRelative(Function &F) {
return Changed;
}
+/// Lower @llvm.can.load.speculatively using target-specific expansion.
+/// Each target provides its own expansion via
+/// TargetLowering::emitCanLoadSpeculatively.
+/// The default expansion returns false (conservative).
+static bool lowerCanLoadSpeculatively(Function &F, const TargetMachine *TM) {
+ if (F.use_empty())
+ return false;
+
+ bool Changed = false;
+
+ for (Use &U : llvm::make_early_inc_range(F.uses())) {
+ auto *CI = dyn_cast<CallInst>(U.getUser());
+ if (!CI || CI->getCalledOperand() != &F)
+ continue;
+
+ Function *ParentFunc = CI->getFunction();
+ const TargetLowering *TLI =
+ TM->getSubtargetImpl(*ParentFunc)->getTargetLowering();
+
+ IRBuilder<> Builder(CI);
+ Value *Ptr = CI->getArgOperand(0);
+ Value *Size = CI->getArgOperand(1);
+
+ // Ask target for expansion; nullptr means use default (return false)
+ Value *Result = TLI->emitCanLoadSpeculatively(Builder, Ptr, Size);
+ if (!Result)
+ Result = Builder.getFalse();
+
+ CI->replaceAllUsesWith(Result);
+ CI->eraseFromParent();
+ Changed = true;
+ }
+
+ return Changed;
+}
+
// ObjCARC has knowledge about whether an obj-c runtime function needs to be
// always tail-called or never tail-called.
static CallInst::TailCallKind getOverridingTailCallKind(const Function &F) {
@@ -674,6 +710,9 @@ bool PreISelIntrinsicLowering::lowerIntrinsics(Module &M) const {
case Intrinsic::load_relative:
Changed |= lowerLoadRelative(F);
break;
+ case Intrinsic::can_load_speculatively:
+ Changed |= lowerCanLoadSpeculatively(F, TM);
+ break;
case Intrinsic::is_constant:
case Intrinsic::objectsize:
Changed |= forEachCall(F, [&](CallInst *CI) {
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 52af36014c5b3..faa6130566de5 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5147,6 +5147,33 @@ void SelectionDAGBuilder::visitMaskedLoad(const CallInst &I, bool IsExpanding) {
setValue(&I, Res);
}
+void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
+ SDLoc sdl = getCurSDLoc();
+ Value *PtrOperand = I.getArgOperand(0);
+ SDValue Ptr = getValue(PtrOperand);
+
+ const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+ EVT VT = TLI.getValueType(DAG.getDataLayout(), I.getType());
+ Align Alignment = I.getParamAlign(0).valueOrOne();
+ AAMDNodes AAInfo = I.getAAMetadata();
+ TypeSize StoreSize = VT.getStoreSize();
+
+ SDValue InChain = DAG.getRoot();
+
+ // Use MOLoad but NOT MODereferenceable - the memory may not be
+ // fully dereferenceable.
+ MachineMemOperand::Flags MMOFlags = MachineMemOperand::MOLoad;
+ LocationSize LocSize = StoreSize.isScalable()
+ ? LocationSize::beforeOrAfterPointer()
+ : LocationSize::precise(StoreSize);
+ MachineMemOperand *MMO = DAG.getMachineFunction().getMachineMemOperand(
+ MachinePointerInfo(PtrOperand), MMOFlags, LocSize, Alignment, AAInfo);
+
+ SDValue Load = DAG.getLoad(VT, sdl, InChain, Ptr, MMO);
+ PendingLoads.push_back(Load.getValue(1));
+ setValue(&I, Load);
+}
+
void SelectionDAGBuilder::visitMaskedGather(const CallInst &I) {
SDLoc sdl = getCurSDLoc();
@@ -6931,6 +6958,9 @@ void SelectionDAGBuilder::visitIntrinsicCall(const CallInst &I,
case Intrinsic::masked_compressstore:
visitMaskedStore(I, true /* IsCompressing */);
return;
+ case Intrinsic::speculative_load:
+ visitSpeculativeLoad(I);
+ return;
case Intrinsic::powi:
setValue(&I, ExpandPowI(sdl, getValue(I.getArgOperand(0)),
getValue(I.getArgOperand(1)), DAG));
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
index 21aac333a73cd..1f5405acd7045 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h
@@ -620,6 +620,7 @@ class SelectionDAGBuilder {
void visitStore(const StoreInst &I);
void visitMaskedLoad(const CallInst &I, bool IsExpanding = false);
void visitMaskedStore(const CallInst &I, bool IsCompressing = false);
+ void visitSpeculativeLoad(const CallInst &I);
void visitMaskedGather(const CallInst &I);
void visitMaskedScatter(const CallInst &I);
void visitAtomicCmpXchg(const AtomicCmpXchgInst &I);
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index f0363b6553440..a10073953d3b1 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -6947,6 +6947,24 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
&Call);
break;
}
+ case Intrinsic::speculative_load: {
+ Type *LoadTy = Call.getType();
+ TypeSize Size = DL.getTypeStoreSize(LoadTy);
+ // For scalable vectors, check the known minimum size is a power of 2.
+ Check(Size.getKnownMinValue() > 0 && isPowerOf2_64(Size.getKnownMinValue()),
+ "llvm.speculative.load type must have a power-of-2 size", &Call);
+ break;
+ }
+ case Intrinsic::can_load_speculatively: {
+ // If size is a constant, verify it's a positive power of 2.
+ if (auto *SizeCI = dyn_cast<ConstantInt>(Call.getArgOperand(1))) {
+ uint64_t Size = SizeCI->getZExtValue();
+ Check(Size > 0 && isPowerOf2_64(Size),
+ "llvm.can.load.speculatively size must be a positive power of 2",
+ &Call);
+ }
+ break;
+ }
case Intrinsic::vector_insert: {
Value *Vec = Call.getArgOperand(0);
Value *SubVec = Call.getArgOperand(1);
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
index bb3d7b560b534..1649ea66f0058 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
@@ -31407,6 +31407,55 @@ Value *AArch64TargetLowering::emitStoreConditional(IRBuilderBase &Builder,
return CI;
}
+Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
+ Value *Ptr,
+ Value *Size) const {
+ // For power-of-2 sizes <= 16, emit alignment check: (ptr & (size - 1)) == 0.
+ // If the pointer is aligned to at least 'size' bytes, loading 'size' bytes
+ // cannot cross a page boundary, so it's safe to speculate.
+ // The 16-byte limit ensures correctness with MTE (memory tagging), since
+ // MTE uses 16-byte tag granules.
+ //
+ // The alignment check only works for power-of-2 sizes. For non-power-of-2
+ // sizes, we conservatively return false.
+ const DataLayout &DL =
+ Builder.GetInsertBlock()->getModule()->getDataLayout();
+
+ if (auto *CI = dyn_cast<ConstantInt>(Size)) {
+ uint64_t SizeVal = CI->getZExtValue();
+ assert(isPowerOf2_64(SizeVal) && "size must be power-of-two");
+ // For constant sizes > 16, return nullptr (default false).
+ if (SizeVal > 16)
+ return nullptr;
+
+ // Power-of-2 constant size <= 16: use fast alignment check.
+ unsigned PtrBits = DL.getPointerSizeInBits();
+ Type *IntPtrTy = Builder.getIntNTy(PtrBits);
+ Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
+ Value *Mask = ConstantInt::get(IntPtrTy, SizeVal - 1);
+ Value *Masked = Builder.CreateAnd(PtrInt, Mask);
+ return Builder.CreateICmpEQ(Masked, ConstantInt::get(IntPtrTy, 0));
+ }
+
+ // Check power-of-2 size <= 16 and alignment.
+ unsigned PtrBits = DL.getPointerSizeInBits();
+ Type *IntPtrTy = Builder.getIntNTy(PtrBits);
+ Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
+ Value *SizeExt = Builder.CreateZExtOrTrunc(Size, IntPtrTy);
+
+ Value *SizeLE16 =
+ Builder.CreateICmpULE(SizeExt, ConstantInt::get(IntPtrTy, 16));
+
+ // alignment check: (ptr & (size - 1)) == 0
+ Value *SizeMinusOne =
+ Builder.CreateSub(SizeExt, ConstantInt::get(IntPtrTy, 1));
+ Value *Masked = Builder.CreateAnd(PtrInt, SizeMinusOne);
+ Value *AlignCheck =
+ Builder.CreateICmpEQ(Masked, ConstantInt::get(IntPtrTy, 0));
+
+ return Builder.CreateAnd(SizeLE16, AlignCheck);
+}
+
bool AArch64TargetLowering::functionArgumentNeedsConsecutiveRegisters(
Type *Ty, CallingConv::ID CallConv, bool isVarArg,
const DataLayout &DL) const {
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.h b/llvm/lib/Target/AArch64/AArch64ISelLowering.h
index 7e4c4e1ba25ff..199eb36547741 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.h
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.h
@@ -347,6 +347,8 @@ class AArch64TargetLowering : public TargetLowering {
AtomicOrdering Ord) const override;
Value *emitStoreConditional(IRBuilderBase &Builder, Value *Val, Value *Addr,
AtomicOrdering Ord) const override;
+ Value *emitCanLoadSpeculatively(IRBuilderBase &Builder, Value *Ptr,
+ Value *Size) const override;
void emitAtomicCmpXchgNoStoreLLBalance(IRBuilderBase &Builder) const override;
diff --git a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
new file mode 100644
index 0000000000000..7916f2e4d340f
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
@@ -0,0 +1,78 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -mtriple=aarch64-unknown-linux-gnu -passes=pre-isel-intrinsic-lowering -S < %s | FileCheck %s
+
+; Test that @llvm.can.load.speculatively is lowered to an alignment check
+; for power-of-2 sizes <= 16 bytes on AArch64, and returns false for larger sizes.
+; The 16-byte limit ensures correctness with MTE (memory tagging).
+; Note: non-power-of-2 constant sizes are rejected by the verifier.
+
+define i1 @can_load_speculatively_16(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_16(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
+; Size > 16 - returns false (may cross MTE tag granule boundary)
+define i1 @can_load_speculatively_32(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_32(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 32)
+ ret i1 %can_load
+}
+
+; Size > 16 - returns false (may cross MTE tag granule boundary)
+define i1 @can_load_speculatively_64(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_64(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 64)
+ ret i1 %can_load
+}
+
+; Test with address space
+define i1 @can_load_speculatively_addrspace1(ptr addrspace(1) %ptr) {
+; CHECK-LABEL: @can_load_speculatively_addrspace1(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr addrspace(1) [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p1(ptr addrspace(1) %ptr, i64 16)
+ ret i1 %can_load
+}
+
+; Test size 8 (within limit, power-of-2)
+define i1 @can_load_speculatively_8(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_8(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 7
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 8)
+ ret i1 %can_load
+}
+
+; Test with runtime size - checks size <= 16 and alignment
+define i1 @can_load_speculatively_runtime(ptr %ptr, i64 %size) {
+; CHECK-LABEL: @can_load_speculatively_runtime(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = icmp ule i64 [[SIZE:%.*]], 16
+; CHECK-NEXT: [[TMP3:%.*]] = sub i64 [[SIZE]], 1
+; CHECK-NEXT: [[TMP4:%.*]] = and i64 [[TMP1]], [[TMP3]]
+; CHECK-NEXT: [[TMP5:%.*]] = icmp eq i64 [[TMP4]], 0
+; CHECK-NEXT: [[TMP6:%.*]] = and i1 [[TMP2]], [[TMP5]]
+; CHECK-NEXT: ret i1 [[TMP6]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 %size)
+ ret i1 %can_load
+}
+
+declare i1 @llvm.can.load.speculatively.p0(ptr, i64)
+declare i1 @llvm.can.load.speculatively.p1(ptr addrspace(1), i64)
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
new file mode 100644
index 0000000000000..78a56f3539d11
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
@@ -0,0 +1,66 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-unknown-linux-gnu -mattr=+sve < %s | FileCheck %s
+
+; Test that @llvm.speculative.load with scalable vectors is lowered to a
+; regular load in SelectionDAG.
+
+define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv4i32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr)
+ ret <vscale x 4 x i32> %load
+}
+
+define <vscale x 2 x i64> @speculative_load_nxv2i64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv2i64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr %ptr)
+ ret <vscale x 2 x i64> %load
+}
+
+define <vscale x 8 x i16> @speculative_load_nxv8i16(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv8i16:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr align 8 %ptr)
+ ret <vscale x 8 x i16> %load
+}
+
+define <vscale x 16 x i8> @speculative_load_nxv16i8(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv16i8:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr %ptr)
+ ret <vscale x 16 x i8> %load
+}
+
+define <vscale x 4 x float> @speculative_load_nxv4f32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv4f32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr align 4 %ptr)
+ ret <vscale x 4 x float> %load
+}
+
+define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv2f64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr)
+ ret <vscale x 2 x double> %load
+}
+
+declare <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr)
+declare <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr)
+declare <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr)
+declare <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr)
+declare <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr)
+declare <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr)
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
new file mode 100644
index 0000000000000..8f9a17414ff05
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -0,0 +1,117 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-unknown-linux-gnu < %s | FileCheck %s
+
+; Test that @llvm.speculative.load is lowered to a regular load
+; in SelectionDAG, respecting the alignment attribute.
+
+define <4 x i32> @speculative_load_v4i32_align16(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_align16:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr)
+ ret <4 x i32> %load
+}
+
+define <4 x i32> @speculative_load_v4i32_align4(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_align4:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 4 %ptr)
+ ret <4 x i32> %load
+}
+
+define <4 x i32> @speculative_load_v4i32_noalign(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_noalign:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr)
+ ret <4 x i32> %load
+}
+
+define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v8i32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldp q0, q1, [x0]
+; CHECK-NEXT: ret
+ %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr)
+ ret <8 x i32> %load
+}
+
+define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v2i64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
+ ret <2 x i64> %load
+}
+
+define <4 x float> @speculative_load_v4f32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4f32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
+ ret <4 x float> %load
+}
+
+define <2 x double> @speculative_load_v2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v2f64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr)
+ ret <2 x double> %load
+}
+
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr)
+declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr)
+declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr)
+declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr)
+declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr)
+
+; Scalar type tests
+
+define i32 @speculative_load_i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_i32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr w0, [x0]
+; CHECK-NEXT: ret
+ %load = call i32 @llvm.speculative.load.i32.p0(ptr align 4 %ptr)
+ ret i32 %load
+}
+
+define i64 @speculative_load_i64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_i64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr x0, [x0]
+; CHECK-NEXT: ret
+ %load = call i64 @llvm.speculative.load.i64.p0(ptr %ptr)
+ ret i64 %load
+}
+
+define float @speculative_load_f32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_f32:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr s0, [x0]
+; CHECK-NEXT: ret
+ %load = call float @llvm.speculative.load.f32.p0(ptr %ptr)
+ ret float %load
+}
+
+define double @speculative_load_f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_f64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr d0, [x0]
+; CHECK-NEXT: ret
+ %load = call double @llvm.speculative.load.f64.p0(ptr align 8 %ptr)
+ ret double %load
+}
+
+declare i32 @llvm.speculative.load.i32.p0(ptr)
+declare i64 @llvm.speculative.load.i64.p0(ptr)
+declare float @llvm.speculative.load.f32.p0(ptr)
+declare double @llvm.speculative.load.f64.p0(ptr)
diff --git a/llvm/test/CodeGen/X86/can-load-speculatively.ll b/llvm/test/CodeGen/X86/can-load-speculatively.ll
new file mode 100644
index 0000000000000..f51d3847e921d
--- /dev/null
+++ b/llvm/test/CodeGen/X86/can-load-speculatively.ll
@@ -0,0 +1,32 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -mtriple=x86_64-unknown-linux-gnu -passes=pre-isel-intrinsic-lowering -S < %s | FileCheck %s
+
+; Test that @llvm.can.load.speculatively returns false (default) on X86,
+; as X86 does not provide a target-specific expansion.
+
+define i1 @can_load_speculatively_16(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_16(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_32(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_32(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 32)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_8(ptr %ptr) {
+; CHECK-LABEL: @can_load_speculatively_8(
+; CHECK-NEXT: ret i1 false
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 8)
+ ret i1 %can_load
+}
+
+declare i1 @llvm.can.load.speculatively.p0(ptr, i64)
+
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
new file mode 100644
index 0000000000000..4dd8375e9c42a
--- /dev/null
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -0,0 +1,146 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+sse4.1 < %s | FileCheck %s --check-prefix=SSE
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2 < %s | FileCheck %s --check-prefix=AVX
+
+; Test that @llvm.speculative.load is lowered to a regular load
+; in SelectionDAG.
+
+define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
+; SSE-LABEL: speculative_load_v4i32:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v4i32:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: retq
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr)
+ ret <4 x i32> %load
+}
+
+define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
+; SSE-LABEL: speculative_load_v8i32:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: movaps 16(%rdi), %xmm1
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v8i32:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %ymm0
+; AVX-NEXT: retq
+ %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr)
+ ret <8 x i32> %load
+}
+
+define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
+; SSE-LABEL: speculative_load_v2i64:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v2i64:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: retq
+ %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
+ ret <2 x i64> %load
+}
+
+define <4 x float> @speculative_load_v4f32(ptr %ptr) {
+; SSE-LABEL: speculative_load_v4f32:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v4f32:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: retq
+ %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
+ ret <4 x float> %load
+}
+
+define <2 x double> @speculative_load_v2f64(ptr %ptr) {
+; SSE-LABEL: speculative_load_v2f64:
+; SSE: # %bb.0:
+; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_v2f64:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: retq
+ %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr)
+ ret <2 x double> %load
+}
+
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr)
+declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr)
+declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr)
+declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr)
+declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr)
+
+; Scalar type tests
+
+define i32 @speculative_load_i32(ptr %ptr) {
+; SSE-LABEL: speculative_load_i32:
+; SSE: # %bb.0:
+; SSE-NEXT: movl (%rdi), %eax
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_i32:
+; AVX: # %bb.0:
+; AVX-NEXT: movl (%rdi), %eax
+; AVX-NEXT: retq
+ %load = call i32 @llvm.speculative.load.i32.p0(ptr %ptr)
+ ret i32 %load
+}
+
+define i64 @speculative_load_i64(ptr %ptr) {
+; SSE-LABEL: speculative_load_i64:
+; SSE: # %bb.0:
+; SSE-NEXT: movq (%rdi), %rax
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_i64:
+; AVX: # %bb.0:
+; AVX-NEXT: movq (%rdi), %rax
+; AVX-NEXT: retq
+ %load = call i64 @llvm.speculative.load.i64.p0(ptr align 8 %ptr)
+ ret i64 %load
+}
+
+define float @speculative_load_f32(ptr %ptr) {
+; SSE-LABEL: speculative_load_f32:
+; SSE: # %bb.0:
+; SSE-NEXT: movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_f32:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX-NEXT: retq
+ %load = call float @llvm.speculative.load.f32.p0(ptr align 4 %ptr)
+ ret float %load
+}
+
+define double @speculative_load_f64(ptr %ptr) {
+; SSE-LABEL: speculative_load_f64:
+; SSE: # %bb.0:
+; SSE-NEXT: movsd {{.*#+}} xmm0 = mem[0],zero
+; SSE-NEXT: retq
+;
+; AVX-LABEL: speculative_load_f64:
+; AVX: # %bb.0:
+; AVX-NEXT: vmovsd {{.*#+}} xmm0 = mem[0],zero
+; AVX-NEXT: retq
+ %load = call double @llvm.speculative.load.f64.p0(ptr %ptr)
+ ret double %load
+}
+
+declare i32 @llvm.speculative.load.i32.p0(ptr)
+declare i64 @llvm.speculative.load.i64.p0(ptr)
+declare float @llvm.speculative.load.f32.p0(ptr)
+declare double @llvm.speculative.load.f64.p0(ptr)
diff --git a/llvm/test/Verifier/can-load-speculatively.ll b/llvm/test/Verifier/can-load-speculatively.ll
new file mode 100644
index 0000000000000..d2d69f70cfb60
--- /dev/null
+++ b/llvm/test/Verifier/can-load-speculatively.ll
@@ -0,0 +1,19 @@
+; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
+
+declare i1 @llvm.can.load.speculatively.p0(ptr, i64)
+
+; Test that constant size must be a positive power of 2
+
+define i1 @test_size_zero(ptr %ptr) {
+; CHECK: llvm.can.load.speculatively size must be a positive power of 2
+; CHECK-NEXT: %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 0)
+ %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 0)
+ ret i1 %res
+}
+
+define i1 @test_non_power_of_2(ptr %ptr) {
+; CHECK: llvm.can.load.speculatively size must be a positive power of 2
+; CHECK-NEXT: %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 3)
+ %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 3)
+ ret i1 %res
+}
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
new file mode 100644
index 0000000000000..def46d20799c5
--- /dev/null
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -0,0 +1,18 @@
+; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
+
+declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr)
+declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr)
+
+define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
+; CHECK: llvm.speculative.load type must have a power-of-2 size
+; CHECK-NEXT: %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr)
+ %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr)
+ ret <3 x i32> %res
+}
+
+define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
+; CHECK: llvm.speculative.load type must have a power-of-2 size
+; CHECK-NEXT: %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr)
+ %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr)
+ ret <vscale x 3 x i32> %res
+}
>From 70951d005f2f728d64cb011e2e5ef86b23311fa9 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Thu, 5 Feb 2026 19:40:18 +0000
Subject: [PATCH 02/14] !fixup address comments, thanks
---
llvm/docs/LangRef.rst | 16 ++++++++--------
llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp | 3 ---
llvm/lib/Target/AArch64/AArch64ISelLowering.cpp | 13 +++++++------
.../CodeGen/AArch64/can-load-speculatively.ll | 5 +----
.../CodeGen/X86/speculative-load-intrinsic.ll | 8 ++++----
5 files changed, 20 insertions(+), 25 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index b92a98e56fe2e..93565410b7104 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28366,10 +28366,10 @@ Overview:
"""""""""
The '``llvm.speculative.load``' intrinsic loads a value from memory. Unlike a
-regular load, the memory access may
-extend beyond the bounds of the allocated object, provided the pointer has been
-verified by :ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to
-ensure the access cannot fault.
+regular load, the memory access may extend beyond the bounds of the allocated
+object, provided the pointer has been verified by
+:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure the
+access cannot fault.
Arguments:
""""""""""
@@ -28387,12 +28387,12 @@ the access cannot fault.
For bytes that are within the bounds of the allocated object, the intrinsic
returns the stored value. For bytes that are beyond the bounds of the
-allocated object, the intrinsic returns ``undef`` for those bytes. At least the
+allocated object, the intrinsic returns ``poison`` for those bytes. At least the
first accessed byte must be within the bounds of an allocated object the pointer is
based on.
The behavior is undefined if this intrinsic is used to load from a pointer
-for which ``llvm.can.load.speculatively`` returns false.
+for which ``llvm.can.load.speculatively`` would return false.
.. _int_can_load_speculatively:
@@ -28428,8 +28428,8 @@ Semantics:
""""""""""
This intrinsic has **target-dependent** semantics. It returns ``true`` if
-loading ``num_bytes`` bytes from ``ptr`` is guaranteed not to trap,
-even if the memory is beyond the bounds of an allocated object. It returns
+``num_bytes`` bytes starting at ``ptr`` can be loaded speculatively, even
+if the memory is beyond the bounds of an allocated object. It returns
``false`` otherwise.
The specific conditions under which this intrinsic returns ``true`` are
diff --git a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
index 06a65e7375f00..63c31cdb81970 100644
--- a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
+++ b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
@@ -141,9 +141,6 @@ static bool lowerLoadRelative(Function &F) {
/// TargetLowering::emitCanLoadSpeculatively.
/// The default expansion returns false (conservative).
static bool lowerCanLoadSpeculatively(Function &F, const TargetMachine *TM) {
- if (F.use_empty())
- return false;
-
bool Changed = false;
for (Use &U : llvm::make_early_inc_range(F.uses())) {
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
index 1649ea66f0058..7df6cf8394d33 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
@@ -31410,6 +31410,10 @@ Value *AArch64TargetLowering::emitStoreConditional(IRBuilderBase &Builder,
Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
Value *Ptr,
Value *Size) const {
+ unsigned AS = cast<PointerType>(Ptr->getType())->getAddressSpace();
+ // Conservatively only allow speculation for address space 0.
+ if (AS != 0)
+ return nullptr;
// For power-of-2 sizes <= 16, emit alignment check: (ptr & (size - 1)) == 0.
// If the pointer is aligned to at least 'size' bytes, loading 'size' bytes
// cannot cross a page boundary, so it's safe to speculate.
@@ -31418,9 +31422,10 @@ Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
//
// The alignment check only works for power-of-2 sizes. For non-power-of-2
// sizes, we conservatively return false.
- const DataLayout &DL =
- Builder.GetInsertBlock()->getModule()->getDataLayout();
+ const DataLayout &DL = Builder.GetInsertBlock()->getModule()->getDataLayout();
+ unsigned PtrBits = DL.getPointerSizeInBits(AS);
+ Type *IntPtrTy = Builder.getIntNTy(PtrBits);
if (auto *CI = dyn_cast<ConstantInt>(Size)) {
uint64_t SizeVal = CI->getZExtValue();
assert(isPowerOf2_64(SizeVal) && "size must be power-of-two");
@@ -31429,8 +31434,6 @@ Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
return nullptr;
// Power-of-2 constant size <= 16: use fast alignment check.
- unsigned PtrBits = DL.getPointerSizeInBits();
- Type *IntPtrTy = Builder.getIntNTy(PtrBits);
Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
Value *Mask = ConstantInt::get(IntPtrTy, SizeVal - 1);
Value *Masked = Builder.CreateAnd(PtrInt, Mask);
@@ -31438,8 +31441,6 @@ Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
}
// Check power-of-2 size <= 16 and alignment.
- unsigned PtrBits = DL.getPointerSizeInBits();
- Type *IntPtrTy = Builder.getIntNTy(PtrBits);
Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
Value *SizeExt = Builder.CreateZExtOrTrunc(Size, IntPtrTy);
diff --git a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
index 7916f2e4d340f..e9af2eed8918a 100644
--- a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
+++ b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
@@ -38,10 +38,7 @@ define i1 @can_load_speculatively_64(ptr %ptr) {
; Test with address space
define i1 @can_load_speculatively_addrspace1(ptr addrspace(1) %ptr) {
; CHECK-LABEL: @can_load_speculatively_addrspace1(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr addrspace(1) [[PTR:%.*]] to i64
-; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
-; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
-; CHECK-NEXT: ret i1 [[TMP3]]
+; CHECK-NEXT: ret i1 false
;
%can_load = call i1 @llvm.can.load.speculatively.p1(ptr addrspace(1) %ptr, i64 16)
ret i1 %can_load
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index 4dd8375e9c42a..798ee2c03b1ab 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -37,12 +37,12 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
; SSE-LABEL: speculative_load_v2i64:
; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: movups (%rdi), %xmm0
; SSE-NEXT: retq
;
; AVX-LABEL: speculative_load_v2i64:
; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: vmovups (%rdi), %xmm0
; AVX-NEXT: retq
%load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
ret <2 x i64> %load
@@ -51,12 +51,12 @@ define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
define <4 x float> @speculative_load_v4f32(ptr %ptr) {
; SSE-LABEL: speculative_load_v4f32:
; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
+; SSE-NEXT: movups (%rdi), %xmm0
; SSE-NEXT: retq
;
; AVX-LABEL: speculative_load_v4f32:
; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %xmm0
+; AVX-NEXT: vmovups (%rdi), %xmm0
; AVX-NEXT: retq
%load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
ret <4 x float> %load
>From 76caf723a34b7ed43dd313c72da8f21da716fc9f Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Mon, 9 Feb 2026 18:42:04 +0000
Subject: [PATCH 03/14] !fixup add tests with aligned pointers, clarify deref
guarantees.
---
llvm/docs/LangRef.rst | 9 ++--
.../CodeGen/AArch64/can-load-speculatively.ll | 45 +++++++++++++++++++
2 files changed, 50 insertions(+), 4 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 93565410b7104..68afca332fe94 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28428,13 +28428,14 @@ Semantics:
""""""""""
This intrinsic has **target-dependent** semantics. It returns ``true`` if
-``num_bytes`` bytes starting at ``ptr`` can be loaded speculatively, even
-if the memory is beyond the bounds of an allocated object. It returns
-``false`` otherwise.
+``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for any non-negative
+integer ``I`` where the computed address does not wrap around the address
+space, can be loaded speculatively, even if the memory is beyond the bounds of
+an allocated object. It returns ``false`` otherwise.
The specific conditions under which this intrinsic returns ``true`` are
determined by the target. For example, a target may check whether the pointer
-alignment guarantees the load cannot cross a page boundary.
+alignment guarantees all such loads cannot cross a page boundary.
.. code-block:: llvm
diff --git a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
index e9af2eed8918a..b6679f22b0989 100644
--- a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
+++ b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
@@ -17,6 +17,51 @@ define i1 @can_load_speculatively_16(ptr %ptr) {
ret i1 %can_load
}
+
+define i1 @can_load_speculatively_8_ptr_aligned(ptr align 8 %ptr) {
+; CHECK-LABEL: @can_load_speculatively_8_ptr_aligned(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_16_ptr_aligned(ptr align 16 %ptr) {
+; CHECK-LABEL: @can_load_speculatively_16_ptr_aligned(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_16_ptr_aligned2(ptr align 16 %ptr) {
+; CHECK-LABEL: @can_load_speculatively_16_ptr_aligned2(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr align 16 %ptr, i64 16)
+ ret i1 %can_load
+}
+
+define i1 @can_load_speculatively_32_ptr_aligned(ptr align 32 %ptr) {
+; CHECK-LABEL: @can_load_speculatively_32_ptr_aligned(
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
+; CHECK-NEXT: ret i1 [[TMP3]]
+;
+ %can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 16)
+ ret i1 %can_load
+}
+
; Size > 16 - returns false (may cross MTE tag granule boundary)
define i1 @can_load_speculatively_32(ptr %ptr) {
; CHECK-LABEL: @can_load_speculatively_32(
>From 3a3755a6b8d87373f53fbe69cb2725c7c30b207e Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Wed, 18 Feb 2026 09:09:09 +0000
Subject: [PATCH 04/14] !fixup clarify noalias interaction
---
llvm/docs/LangRef.rst | 27 +++++++++++++++++----------
1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 68afca332fe94..062a517320470 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28381,15 +28381,22 @@ Semantics:
""""""""""
The '``llvm.speculative.load``' intrinsic performs a load that may access
-memory beyond the allocated object. It must be used in combination with
-:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure
-the access cannot fault.
+memory beyond what is accessible through the pointer. It must be used in
+combination with :ref:`llvm.can.load.speculatively <int_can_load_speculatively>`
+to ensure the access can be performed speculatively.
-For bytes that are within the bounds of the allocated object, the intrinsic
-returns the stored value. For bytes that are beyond the bounds of the
-allocated object, the intrinsic returns ``poison`` for those bytes. At least the
-first accessed byte must be within the bounds of an allocated object the pointer is
-based on.
+A byte at ``ptr + i`` is *accessible through* ``ptr`` if both of the following
+hold:
+
+1. The byte lies within the bounds of an allocated object that ``ptr`` is
+ :ref:`based <pointeraliasing>` on.
+2. Accessing the byte through ``ptr`` does not violate any ``noalias``
+ constraints.
+
+For accessible bytes, the intrinsic returns the stored value. For inaccessible
+bytes, the intrinsic returns ``poison`` and the bytes are not considered accessed
+for the purpose of data races or ``noalias`` constraints. At least the first
+byte must be accessible; otherwise the behavior is undefined.
The behavior is undefined if this intrinsic is used to load from a pointer
for which ``llvm.can.load.speculatively`` would return false.
@@ -28428,8 +28435,8 @@ Semantics:
""""""""""
This intrinsic has **target-dependent** semantics. It returns ``true`` if
-``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for any non-negative
-integer ``I`` where the computed address does not wrap around the address
+``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for all non-negative
+integers ``I`` where the computed address does not wrap around the address
space, can be loaded speculatively, even if the memory is beyond the bounds of
an allocated object. It returns ``false`` otherwise.
>From f9a0fb937e538daf8e17d7a3759520267ce7a4b8 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Mon, 23 Feb 2026 09:43:35 +0000
Subject: [PATCH 05/14] !fixup limit to executions that do not depend on
may-accessible bytes.
---
llvm/docs/LangRef.rst | 3 +++
1 file changed, 3 insertions(+)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 062a517320470..800e9792744eb 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28398,6 +28398,9 @@ bytes, the intrinsic returns ``poison`` and the bytes are not considered accesse
for the purpose of data races or ``noalias`` constraints. At least the first
byte must be accessible; otherwise the behavior is undefined.
+The behavior is undefined if program execution depends on any byte in the
+result that may not be accessible.
+
The behavior is undefined if this intrinsic is used to load from a pointer
for which ``llvm.can.load.speculatively`` would return false.
>From 6df6e6c57a975ff67017cf5cad42a9e025fff3e0 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Wed, 25 Feb 2026 17:23:24 +0000
Subject: [PATCH 06/14] !fixup add implicit freeze
---
llvm/docs/LangRef.rst | 11 ++++-------
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 800e9792744eb..d95da6674d8f6 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28393,13 +28393,10 @@ hold:
2. Accessing the byte through ``ptr`` does not violate any ``noalias``
constraints.
-For accessible bytes, the intrinsic returns the stored value. For inaccessible
-bytes, the intrinsic returns ``poison`` and the bytes are not considered accessed
-for the purpose of data races or ``noalias`` constraints. At least the first
-byte must be accessible; otherwise the behavior is undefined.
-
-The behavior is undefined if program execution depends on any byte in the
-result that may not be accessible.
+For each byte of the result, if the corresponding byte is accessible through
+``ptr``, the result contains the value stored in memory at that byte. Otherwise,
+the corresponding byte of the result is ``freeze poison`` and the byte is not
+considered accessed for the purpose of data races or ``noalias`` constraints.
The behavior is undefined if this intrinsic is used to load from a pointer
for which ``llvm.can.load.speculatively`` would return false.
>From 407c0077e52463d8a8361e1d73c1e0e209321e3d Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Fri, 27 Feb 2026 13:52:11 +0000
Subject: [PATCH 07/14] !fixup add read bytes argument
---
llvm/docs/LangRef.rst | 35 ++++++++--------
llvm/include/llvm/IR/Intrinsics.td | 7 ++--
.../SelectionDAG/SelectionDAGBuilder.cpp | 2 +
.../AArch64/speculative-load-intrinsic-sve.ll | 24 +++++------
.../AArch64/speculative-load-intrinsic.ll | 40 +++++++++----------
.../CodeGen/X86/speculative-load-intrinsic.ll | 36 ++++++++---------
llvm/test/Verifier/speculative-load.ll | 12 +++---
7 files changed, 78 insertions(+), 78 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index d95da6674d8f6..3e8165fcef106 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28358,9 +28358,9 @@ This is an overloaded intrinsic.
::
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>)
- declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>)
- declare i64 @llvm.speculative.load.i64.p0(ptr <ptr>)
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i64 <num_read_bytes>)
+ declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i64 <num_read_bytes>)
+ declare i64 @llvm.speculative.load.i64.p0(ptr <ptr>, i64 <num_read_bytes>)
Overview:
"""""""""
@@ -28374,32 +28374,29 @@ access cannot fault.
Arguments:
""""""""""
-The argument is a pointer to the memory location to load from. The return type
-must have a power-of-2 size in bytes.
+The first argument is a pointer to the memory location to load from. The second
+argument is an ``i64`` specifying the number of bytes starting from ``ptr``
+that the intrinsic will read. The return type must have a power-of-2 size in
+bytes.
Semantics:
""""""""""
The '``llvm.speculative.load``' intrinsic performs a load that may access
-memory beyond what is accessible through the pointer. It must be used in
+memory beyond what is readable through the pointer. It must be used in
combination with :ref:`llvm.can.load.speculatively <int_can_load_speculatively>`
to ensure the access can be performed speculatively.
-A byte at ``ptr + i`` is *accessible through* ``ptr`` if both of the following
-hold:
+The intrinsic reads ``num_read_bytes`` bytes starting from ``ptr`` and returns
+the stored values for those bytes. The read bytes must lie within the bounds
+of an allocated object that ``ptr`` is :ref:`based <pointeraliasing>` on.
-1. The byte lies within the bounds of an allocated object that ``ptr`` is
- :ref:`based <pointeraliasing>` on.
-2. Accessing the byte through ``ptr`` does not violate any ``noalias``
- constraints.
-
-For each byte of the result, if the corresponding byte is accessible through
-``ptr``, the result contains the value stored in memory at that byte. Otherwise,
-the corresponding byte of the result is ``freeze poison`` and the byte is not
-considered accessed for the purpose of data races or ``noalias`` constraints.
+Bytes at offsets ``>= num_read_bytes`` (up to the size of the return type)
+are ``poison`` and are not considered accessed for the purpose of data races
+or ``noalias`` constraints.
The behavior is undefined if this intrinsic is used to load from a pointer
-for which ``llvm.can.load.speculatively`` would return false.
+for which ``llvm.can.load.speculatively`` returns false.
.. _int_can_load_speculatively:
@@ -28452,7 +28449,7 @@ alignment guarantees all such loads cannot cross a page boundary.
speculative_path:
; Safe to speculatively load from %ptr
- %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr)
+ %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 16)
...
safe_path:
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index ad5e864bd3864..ddd5c1e857ae1 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2714,11 +2714,12 @@ def int_masked_srem:
[IntrNoMem]>;
// Speculatively load a value from memory; lowers to a regular aligned load.
-// The loaded type must have a power-of-2 size.
+// The loaded type must have a power-of-2 size. The second argument specifies
+// the number of bytes the intrinsic will read from memory.
def int_speculative_load:
DefaultAttrsIntrinsic<[llvm_any_ty],
- [llvm_anyptr_ty],
- [IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
+ [llvm_anyptr_ty, llvm_i64_ty],
+ [IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
// The size can be a runtime value to support scalable vectors.
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index faa6130566de5..acf0aeb7818b9 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5150,6 +5150,8 @@ void SelectionDAGBuilder::visitMaskedLoad(const CallInst &I, bool IsExpanding) {
void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
SDLoc sdl = getCurSDLoc();
Value *PtrOperand = I.getArgOperand(0);
+ // The second argument (num_read_bytes) is IR-level
+ // semantics only; it is not needed at codegen.
SDValue Ptr = getValue(PtrOperand);
const TargetLowering &TLI = DAG.getTargetLoweringInfo();
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
index 78a56f3539d11..4ae56cbca7ae9 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
@@ -9,7 +9,7 @@ define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr)
+ %load = call <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i64 0)
ret <vscale x 4 x i32> %load
}
@@ -18,7 +18,7 @@ define <vscale x 2 x i64> @speculative_load_nxv2i64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr %ptr)
+ %load = call <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr %ptr, i64 0)
ret <vscale x 2 x i64> %load
}
@@ -27,7 +27,7 @@ define <vscale x 8 x i16> @speculative_load_nxv8i16(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr align 8 %ptr)
+ %load = call <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr align 8 %ptr, i64 0)
ret <vscale x 8 x i16> %load
}
@@ -36,7 +36,7 @@ define <vscale x 16 x i8> @speculative_load_nxv16i8(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr %ptr)
+ %load = call <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr %ptr, i64 0)
ret <vscale x 16 x i8> %load
}
@@ -45,7 +45,7 @@ define <vscale x 4 x float> @speculative_load_nxv4f32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr align 4 %ptr)
+ %load = call <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr align 4 %ptr, i64 0)
ret <vscale x 4 x float> %load
}
@@ -54,13 +54,13 @@ define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr)
+ %load = call <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i64 0)
ret <vscale x 2 x double> %load
}
-declare <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr)
-declare <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr)
-declare <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr)
-declare <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr)
-declare <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr)
-declare <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr)
+declare <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr, i64)
+declare <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr, i64)
+declare <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr, i64)
+declare <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr, i64)
+declare <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr, i64)
+declare <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr, i64)
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
index 8f9a17414ff05..1c6b9d267ad46 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -9,7 +9,7 @@ define <4 x i32> @speculative_load_v4i32_align16(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr)
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 0)
ret <4 x i32> %load
}
@@ -18,7 +18,7 @@ define <4 x i32> @speculative_load_v4i32_align4(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 4 %ptr)
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 4 %ptr, i64 0)
ret <4 x i32> %load
}
@@ -27,7 +27,7 @@ define <4 x i32> @speculative_load_v4i32_noalign(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr)
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 0)
ret <4 x i32> %load
}
@@ -36,7 +36,7 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldp q0, q1, [x0]
; CHECK-NEXT: ret
- %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr)
+ %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 0)
ret <8 x i32> %load
}
@@ -45,7 +45,7 @@ define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
+ %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr, i64 0)
ret <2 x i64> %load
}
@@ -54,7 +54,7 @@ define <4 x float> @speculative_load_v4f32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
+ %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr, i64 0)
ret <4 x float> %load
}
@@ -63,15 +63,15 @@ define <2 x double> @speculative_load_v2f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr)
+ %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 0)
ret <2 x double> %load
}
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr)
-declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr)
-declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr)
-declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr)
-declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr)
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i64)
+declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr, i64)
+declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr, i64)
+declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr, i64)
+declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr, i64)
; Scalar type tests
@@ -80,7 +80,7 @@ define i32 @speculative_load_i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr w0, [x0]
; CHECK-NEXT: ret
- %load = call i32 @llvm.speculative.load.i32.p0(ptr align 4 %ptr)
+ %load = call i32 @llvm.speculative.load.i32.p0(ptr align 4 %ptr, i64 0)
ret i32 %load
}
@@ -89,7 +89,7 @@ define i64 @speculative_load_i64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr x0, [x0]
; CHECK-NEXT: ret
- %load = call i64 @llvm.speculative.load.i64.p0(ptr %ptr)
+ %load = call i64 @llvm.speculative.load.i64.p0(ptr %ptr, i64 0)
ret i64 %load
}
@@ -98,7 +98,7 @@ define float @speculative_load_f32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr s0, [x0]
; CHECK-NEXT: ret
- %load = call float @llvm.speculative.load.f32.p0(ptr %ptr)
+ %load = call float @llvm.speculative.load.f32.p0(ptr %ptr, i64 0)
ret float %load
}
@@ -107,11 +107,11 @@ define double @speculative_load_f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr d0, [x0]
; CHECK-NEXT: ret
- %load = call double @llvm.speculative.load.f64.p0(ptr align 8 %ptr)
+ %load = call double @llvm.speculative.load.f64.p0(ptr align 8 %ptr, i64 0)
ret double %load
}
-declare i32 @llvm.speculative.load.i32.p0(ptr)
-declare i64 @llvm.speculative.load.i64.p0(ptr)
-declare float @llvm.speculative.load.f32.p0(ptr)
-declare double @llvm.speculative.load.f64.p0(ptr)
+declare i32 @llvm.speculative.load.i32.p0(ptr, i64)
+declare i64 @llvm.speculative.load.i64.p0(ptr, i64)
+declare float @llvm.speculative.load.f32.p0(ptr, i64)
+declare double @llvm.speculative.load.f64.p0(ptr, i64)
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index 798ee2c03b1ab..5e12593f2d5a0 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -15,7 +15,7 @@ define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovaps (%rdi), %xmm0
; AVX-NEXT: retq
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr)
+ %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 0)
ret <4 x i32> %load
}
@@ -30,7 +30,7 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovaps (%rdi), %ymm0
; AVX-NEXT: retq
- %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr)
+ %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 0)
ret <8 x i32> %load
}
@@ -44,7 +44,7 @@ define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovups (%rdi), %xmm0
; AVX-NEXT: retq
- %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr)
+ %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr, i64 0)
ret <2 x i64> %load
}
@@ -58,7 +58,7 @@ define <4 x float> @speculative_load_v4f32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovups (%rdi), %xmm0
; AVX-NEXT: retq
- %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr)
+ %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr, i64 0)
ret <4 x float> %load
}
@@ -72,15 +72,15 @@ define <2 x double> @speculative_load_v2f64(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovaps (%rdi), %xmm0
; AVX-NEXT: retq
- %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr)
+ %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 0)
ret <2 x double> %load
}
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr)
-declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr)
-declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr)
-declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr)
-declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr)
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i64)
+declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr, i64)
+declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr, i64)
+declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr, i64)
+declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr, i64)
; Scalar type tests
@@ -94,7 +94,7 @@ define i32 @speculative_load_i32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: movl (%rdi), %eax
; AVX-NEXT: retq
- %load = call i32 @llvm.speculative.load.i32.p0(ptr %ptr)
+ %load = call i32 @llvm.speculative.load.i32.p0(ptr %ptr, i64 0)
ret i32 %load
}
@@ -108,7 +108,7 @@ define i64 @speculative_load_i64(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: movq (%rdi), %rax
; AVX-NEXT: retq
- %load = call i64 @llvm.speculative.load.i64.p0(ptr align 8 %ptr)
+ %load = call i64 @llvm.speculative.load.i64.p0(ptr align 8 %ptr, i64 0)
ret i64 %load
}
@@ -122,7 +122,7 @@ define float @speculative_load_f32(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
; AVX-NEXT: retq
- %load = call float @llvm.speculative.load.f32.p0(ptr align 4 %ptr)
+ %load = call float @llvm.speculative.load.f32.p0(ptr align 4 %ptr, i64 0)
ret float %load
}
@@ -136,11 +136,11 @@ define double @speculative_load_f64(ptr %ptr) {
; AVX: # %bb.0:
; AVX-NEXT: vmovsd {{.*#+}} xmm0 = mem[0],zero
; AVX-NEXT: retq
- %load = call double @llvm.speculative.load.f64.p0(ptr %ptr)
+ %load = call double @llvm.speculative.load.f64.p0(ptr %ptr, i64 0)
ret double %load
}
-declare i32 @llvm.speculative.load.i32.p0(ptr)
-declare i64 @llvm.speculative.load.i64.p0(ptr)
-declare float @llvm.speculative.load.f32.p0(ptr)
-declare double @llvm.speculative.load.f64.p0(ptr)
+declare i32 @llvm.speculative.load.i32.p0(ptr, i64)
+declare i64 @llvm.speculative.load.i64.p0(ptr, i64)
+declare float @llvm.speculative.load.f32.p0(ptr, i64)
+declare double @llvm.speculative.load.f64.p0(ptr, i64)
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
index def46d20799c5..c06881a5f50af 100644
--- a/llvm/test/Verifier/speculative-load.ll
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -1,18 +1,18 @@
; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
-declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr)
-declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr)
+declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, i64)
+declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, i64)
define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr)
- %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr)
+; CHECK-NEXT: %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
+ %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
ret <3 x i32> %res
}
define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr)
- %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr)
+; CHECK-NEXT: %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
+ %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
ret <vscale x 3 x i32> %res
}
>From a9ba130685242ec57d8877172bda426aad0fffbd Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Fri, 20 Mar 2026 11:58:45 +0000
Subject: [PATCH 08/14] !fixup make oracle function part of the arguments
---
llvm/docs/LangRef.rst | 48 +++--
llvm/include/llvm/IR/Intrinsics.td | 6 +-
.../SelectionDAG/SelectionDAGBuilder.cpp | 4 +-
llvm/lib/IR/Verifier.cpp | 44 ++++
.../AArch64/speculative-load-intrinsic-sve.ll | 66 ------
.../AArch64/speculative-load-intrinsic.ll | 123 ++++++------
.../CodeGen/X86/speculative-load-intrinsic.ll | 190 +++++++-----------
llvm/test/Verifier/speculative-load.ll | 74 ++++++-
8 files changed, 272 insertions(+), 283 deletions(-)
delete mode 100644 llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 3e8165fcef106..a3878965cefea 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28345,7 +28345,7 @@ Speculative Load Intrinsics
LLVM provides intrinsics for speculatively loading memory that may be
out-of-bounds. These intrinsics enable optimizations like early-exit loop
vectorization where the vectorized loop may read beyond the end of an array,
-provided the access is guaranteed to not trap by target-specific checks.
+provided the access is guaranteed be valid by target-specific checks.
.. _int_speculative_load:
@@ -28358,9 +28358,12 @@ This is an overloaded intrinsic.
::
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i64 <num_read_bytes>)
- declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i64 <num_read_bytes>)
- declare i64 @llvm.speculative.load.i64.p0(ptr <ptr>, i64 <num_read_bytes>)
+ ; Direct form: number of accessible bytes given as i64
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i64 <num_accessible_bytes>)
+ declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i64 <num_accessible_bytes>)
+
+ ; Oracle form: accessible bytes computed by calling oracle_fn(args...)
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, ptr <oracle_fn>, ...)
Overview:
"""""""""
@@ -28369,34 +28372,35 @@ The '``llvm.speculative.load``' intrinsic loads a value from memory. Unlike a
regular load, the memory access may extend beyond the bounds of the allocated
object, provided the pointer has been verified by
:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure the
-access cannot fault.
+access is valid.
Arguments:
""""""""""
-The first argument is a pointer to the memory location to load from. The second
-argument is an ``i64`` specifying the number of bytes starting from ``ptr``
-that the intrinsic will read. The return type must have a power-of-2 size in
-bytes.
+The first argument is a pointer to the memory location to load from. The return
+type must be a vector type with a power-of-2 size in bytes. The remaining
+arguments determine the *number of accessible bytes* starting from ``ptr``,
+denoted ``N`` below.
+
+In the **direct form**, the second argument is an ``i64`` specifying ``N``
+directly. In the **oracle form**, the second argument must be a direct
+reference to a function returning ``i64`` that may only read memory through its
+arguments (indirect function pointers are not permitted); the remaining
+arguments are forwarded to it, and the return value is ``N``.
Semantics:
""""""""""
-The '``llvm.speculative.load``' intrinsic performs a load that may access
-memory beyond what is readable through the pointer. It must be used in
-combination with :ref:`llvm.can.load.speculatively <int_can_load_speculatively>`
-to ensure the access can be performed speculatively.
-
-The intrinsic reads ``num_read_bytes`` bytes starting from ``ptr`` and returns
-the stored values for those bytes. The read bytes must lie within the bounds
-of an allocated object that ``ptr`` is :ref:`based <pointeraliasing>` on.
+The intrinsic reads ``N`` bytes starting from ``ptr`` and returns the stored
+values for those bytes. These bytes must lie within the bounds of an allocated
+object that ``ptr`` is :ref:`based <pointeraliasing>` on.
-Bytes at offsets ``>= num_read_bytes`` (up to the size of the return type)
-are ``poison`` and are not considered accessed for the purpose of data races
-or ``noalias`` constraints.
+Bytes at offsets in ``[N, size of the return type)`` are ``poison`` and are not
+considered accessed for the purposes of data races or ``noalias`` constraints.
-The behavior is undefined if this intrinsic is used to load from a pointer
-for which ``llvm.can.load.speculatively`` returns false.
+The behavior is undefined if the speculative load accesses memory that would
+fault (i.e., the oracle or ``llvm.can.load.speculatively`` would indicate the
+access is not safe).
.. _int_can_load_speculatively:
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index ddd5c1e857ae1..4680ea52f62dc 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2714,11 +2714,9 @@ def int_masked_srem:
[IntrNoMem]>;
// Speculatively load a value from memory; lowers to a regular aligned load.
-// The loaded type must have a power-of-2 size. The second argument specifies
-// the number of bytes the intrinsic will read from memory.
def int_speculative_load:
- DefaultAttrsIntrinsic<[llvm_any_ty],
- [llvm_anyptr_ty, llvm_i64_ty],
+ DefaultAttrsIntrinsic<[llvm_anyvector_ty],
+ [llvm_anyptr_ty, llvm_vararg_ty],
[IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index acf0aeb7818b9..74edd5fe49c86 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5150,8 +5150,8 @@ void SelectionDAGBuilder::visitMaskedLoad(const CallInst &I, bool IsExpanding) {
void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
SDLoc sdl = getCurSDLoc();
Value *PtrOperand = I.getArgOperand(0);
- // The second argument (num_read_bytes) is IR-level
- // semantics only; it is not needed at codegen.
+ // The remaining arguments (num_accessible_bytes or oracle function + args)
+ // are IR-level semantics only; they are not needed at codegen.
SDValue Ptr = getValue(PtrOperand);
const TargetLowering &TLI = DAG.getTargetLoweringInfo();
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index a10073953d3b1..8b8e3f093b07b 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -6953,6 +6953,50 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
// For scalable vectors, check the known minimum size is a power of 2.
Check(Size.getKnownMinValue() > 0 && isPowerOf2_64(Size.getKnownMinValue()),
"llvm.speculative.load type must have a power-of-2 size", &Call);
+
+ unsigned NumArgs = Call.arg_size();
+ Check(NumArgs >= 2, "llvm.speculative.load requires at least 2 arguments",
+ &Call);
+
+ Value *SecondArg = Call.getArgOperand(1);
+ if (SecondArg->getType()->isIntegerTy(64)) {
+ // Direct form: (ptr, i64 num_accessible_bytes)
+ Check(NumArgs == 2,
+ "llvm.speculative.load direct form requires exactly 2 arguments",
+ &Call);
+ if (auto *CI = dyn_cast<ConstantInt>(SecondArg)) {
+ Check(Size.isScalable() || CI->getZExtValue() <= Size.getFixedValue(),
+ "llvm.speculative.load num_accessible_bytes must not exceed "
+ "the result size in bytes",
+ &Call);
+ }
+ } else {
+ // Oracle form: (ptr, oracle_fn_ptr, args...)
+ auto *OracleFn = dyn_cast<Function>(SecondArg);
+ Check(OracleFn,
+ "llvm.speculative.load second argument must be i64 or a direct "
+ "reference to an oracle function",
+ &Call);
+
+ Check(OracleFn->onlyReadsMemory() && OracleFn->onlyAccessesArgMemory(),
+ "llvm.speculative.load oracle function must not have side effects "
+ "and may only read memory through its arguments",
+ &Call);
+
+ FunctionType *FTy = OracleFn->getFunctionType();
+ Check(FTy->getReturnType()->isIntegerTy(64),
+ "llvm.speculative.load oracle function must return i64", &Call);
+ unsigned NumOracleArgs = NumArgs - 2;
+ Check(FTy->isVarArg() ? NumOracleArgs >= FTy->getNumParams()
+ : NumOracleArgs == FTy->getNumParams(),
+ "llvm.speculative.load oracle function argument count mismatch",
+ &Call);
+ for (unsigned I = 0, E = FTy->getNumParams(); I < E; ++I) {
+ Check(FTy->getParamType(I) == Call.getArgOperand(I + 2)->getType(),
+ "llvm.speculative.load oracle function argument type mismatch",
+ &Call);
+ }
+ }
break;
}
case Intrinsic::can_load_speculatively: {
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
deleted file mode 100644
index 4ae56cbca7ae9..0000000000000
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic-sve.ll
+++ /dev/null
@@ -1,66 +0,0 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64-unknown-linux-gnu -mattr=+sve < %s | FileCheck %s
-
-; Test that @llvm.speculative.load with scalable vectors is lowered to a
-; regular load in SelectionDAG.
-
-define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv4i32:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i64 0)
- ret <vscale x 4 x i32> %load
-}
-
-define <vscale x 2 x i64> @speculative_load_nxv2i64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv2i64:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr %ptr, i64 0)
- ret <vscale x 2 x i64> %load
-}
-
-define <vscale x 8 x i16> @speculative_load_nxv8i16(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv8i16:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr align 8 %ptr, i64 0)
- ret <vscale x 8 x i16> %load
-}
-
-define <vscale x 16 x i8> @speculative_load_nxv16i8(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv16i8:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr %ptr, i64 0)
- ret <vscale x 16 x i8> %load
-}
-
-define <vscale x 4 x float> @speculative_load_nxv4f32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv4f32:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr align 4 %ptr, i64 0)
- ret <vscale x 4 x float> %load
-}
-
-define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_nxv2f64:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i64 0)
- ret <vscale x 2 x double> %load
-}
-
-declare <vscale x 4 x i32> @llvm.speculative.load.nxv4i32.p0(ptr, i64)
-declare <vscale x 2 x i64> @llvm.speculative.load.nxv2i64.p0(ptr, i64)
-declare <vscale x 8 x i16> @llvm.speculative.load.nxv8i16.p0(ptr, i64)
-declare <vscale x 16 x i8> @llvm.speculative.load.nxv16i8.p0(ptr, i64)
-declare <vscale x 4 x float> @llvm.speculative.load.nxv4f32.p0(ptr, i64)
-declare <vscale x 2 x double> @llvm.speculative.load.nxv2f64.p0(ptr, i64)
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
index 1c6b9d267ad46..2746b9ea87daa 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -1,117 +1,112 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64-unknown-linux-gnu < %s | FileCheck %s
+; RUN: llc -mtriple=aarch64-unknown-linux-gnu -mattr=+sve < %s | FileCheck %s
; Test that @llvm.speculative.load is lowered to a regular load
-; in SelectionDAG, respecting the alignment attribute.
+; in SelectionDAG for fixed vectors, scalable vectors, and scalars.
-define <4 x i32> @speculative_load_v4i32_align16(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v4i32_align16:
+; Fixed-width vector tests
+
+define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 0)
+ %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 12)
ret <4 x i32> %load
}
-define <4 x i32> @speculative_load_v4i32_align4(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v4i32_align4:
+define <2 x double> @speculative_load_v2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v2f64:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 4 %ptr, i64 0)
- ret <4 x i32> %load
+ %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 8)
+ ret <2 x double> %load
}
-define <4 x i32> @speculative_load_v4i32_noalign(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v4i32_noalign:
+; Scalable vector tests
+
+define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv4i32:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 0)
- ret <4 x i32> %load
+ %load = call <vscale x 4 x i32> (ptr, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i64 8)
+ ret <vscale x 4 x i32> %load
}
-define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v8i32:
+define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv2f64:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldp q0, q1, [x0]
+; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 0)
- ret <8 x i32> %load
+ %load = call <vscale x 2 x double> (ptr, ...) @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i64 4)
+ ret <vscale x 2 x double> %load
}
-define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v2i64:
+; Oracle form tests
+
+declare i64 @oracle(ptr, i64) memory(argmem: read)
+
+define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr, i64 0)
- ret <2 x i64> %load
+ %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
}
-define <4 x float> @speculative_load_v4f32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v4f32:
+define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v2i64_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr, i64 0)
- ret <4 x float> %load
+ %load = call <2 x i64> (ptr, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <2 x i64> %load
}
-define <2 x double> @speculative_load_v2f64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v2f64:
+define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i16_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 0)
- ret <2 x double> %load
+ %load = call <8 x i16> (ptr, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i16> %load
}
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i64)
-declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr, i64)
-declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr, i64)
-declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr, i64)
-declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr, i64)
-
-; Scalar type tests
-
-define i32 @speculative_load_i32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_i32:
+define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v16i8_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr w0, [x0]
+; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call i32 @llvm.speculative.load.i32.p0(ptr align 4 %ptr, i64 0)
- ret i32 %load
+ %load = call <16 x i8> (ptr, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <16 x i8> %load
}
-define i64 @speculative_load_i64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_i64:
+define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4f32_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr x0, [x0]
+; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call i64 @llvm.speculative.load.i64.p0(ptr %ptr, i64 0)
- ret i64 %load
+ %load = call <4 x float> (ptr, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x float> %load
}
-define float @speculative_load_f32(ptr %ptr) {
-; CHECK-LABEL: speculative_load_f32:
+define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v2f64_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr s0, [x0]
+; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call float @llvm.speculative.load.f32.p0(ptr %ptr, i64 0)
- ret float %load
+ %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <2 x double> %load
}
-define double @speculative_load_f64(ptr %ptr) {
-; CHECK-LABEL: speculative_load_f64:
+define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_nxv4i32_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr d0, [x0]
+; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call double @llvm.speculative.load.f64.p0(ptr align 8 %ptr, i64 0)
- ret double %load
+ %load = call <vscale x 4 x i32> (ptr, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <vscale x 4 x i32> %load
}
-
-declare i32 @llvm.speculative.load.i32.p0(ptr, i64)
-declare i64 @llvm.speculative.load.i64.p0(ptr, i64)
-declare float @llvm.speculative.load.f32.p0(ptr, i64)
-declare double @llvm.speculative.load.f64.p0(ptr, i64)
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index 5e12593f2d5a0..e387a17990012 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -1,146 +1,98 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+sse4.1 < %s | FileCheck %s --check-prefix=SSE
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2 < %s | FileCheck %s --check-prefix=AVX
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2 < %s | FileCheck %s
; Test that @llvm.speculative.load is lowered to a regular load
; in SelectionDAG.
define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
-; SSE-LABEL: speculative_load_v4i32:
-; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v4i32:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %xmm0
-; AVX-NEXT: retq
- %load = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 0)
+; CHECK-LABEL: speculative_load_v4i32:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 8)
ret <4 x i32> %load
}
define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
-; SSE-LABEL: speculative_load_v8i32:
-; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
-; SSE-NEXT: movaps 16(%rdi), %xmm1
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v8i32:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %ymm0
-; AVX-NEXT: retq
- %load = call <8 x i32> @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 0)
+; CHECK-LABEL: speculative_load_v8i32:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <8 x i32> (ptr, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 24)
ret <8 x i32> %load
}
-define <2 x i64> @speculative_load_v2i64(ptr %ptr) {
-; SSE-LABEL: speculative_load_v2i64:
-; SSE: # %bb.0:
-; SSE-NEXT: movups (%rdi), %xmm0
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v2i64:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovups (%rdi), %xmm0
-; AVX-NEXT: retq
- %load = call <2 x i64> @llvm.speculative.load.v2i64.p0(ptr %ptr, i64 0)
- ret <2 x i64> %load
-}
+; Oracle form tests
+declare i64 @oracle(ptr, i64) memory(argmem: read)
-define <4 x float> @speculative_load_v4f32(ptr %ptr) {
-; SSE-LABEL: speculative_load_v4f32:
-; SSE: # %bb.0:
-; SSE-NEXT: movups (%rdi), %xmm0
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v4f32:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovups (%rdi), %xmm0
-; AVX-NEXT: retq
- %load = call <4 x float> @llvm.speculative.load.v4f32.p0(ptr align 8 %ptr, i64 0)
- ret <4 x float> %load
+define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
}
-define <2 x double> @speculative_load_v2f64(ptr %ptr) {
-; SSE-LABEL: speculative_load_v2f64:
-; SSE: # %bb.0:
-; SSE-NEXT: movaps (%rdi), %xmm0
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_v2f64:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovaps (%rdi), %xmm0
-; AVX-NEXT: retq
- %load = call <2 x double> @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 0)
- ret <2 x double> %load
+define <8 x i32> @speculative_load_v8i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i32_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <8 x i32> (ptr, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i32> %load
}
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i64)
-declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr, i64)
-declare <2 x i64> @llvm.speculative.load.v2i64.p0(ptr, i64)
-declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr, i64)
-declare <2 x double> @llvm.speculative.load.v2f64.p0(ptr, i64)
-
-; Scalar type tests
+define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v2i64_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <2 x i64> (ptr, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <2 x i64> %load
+}
-define i32 @speculative_load_i32(ptr %ptr) {
-; SSE-LABEL: speculative_load_i32:
-; SSE: # %bb.0:
-; SSE-NEXT: movl (%rdi), %eax
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_i32:
-; AVX: # %bb.0:
-; AVX-NEXT: movl (%rdi), %eax
-; AVX-NEXT: retq
- %load = call i32 @llvm.speculative.load.i32.p0(ptr %ptr, i64 0)
- ret i32 %load
+define <4 x i64> @speculative_load_v4i64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i64_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <4 x i64> (ptr, ...) @llvm.speculative.load.v4i64.p0(ptr align 32 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i64> %load
}
-define i64 @speculative_load_i64(ptr %ptr) {
-; SSE-LABEL: speculative_load_i64:
-; SSE: # %bb.0:
-; SSE-NEXT: movq (%rdi), %rax
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_i64:
-; AVX: # %bb.0:
-; AVX-NEXT: movq (%rdi), %rax
-; AVX-NEXT: retq
- %load = call i64 @llvm.speculative.load.i64.p0(ptr align 8 %ptr, i64 0)
- ret i64 %load
+define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4f32_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x float> (ptr, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x float> %load
}
-define float @speculative_load_f32(ptr %ptr) {
-; SSE-LABEL: speculative_load_f32:
-; SSE: # %bb.0:
-; SSE-NEXT: movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_f32:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; AVX-NEXT: retq
- %load = call float @llvm.speculative.load.f32.p0(ptr align 4 %ptr, i64 0)
- ret float %load
+define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v2f64_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <2 x double> %load
}
-define double @speculative_load_f64(ptr %ptr) {
-; SSE-LABEL: speculative_load_f64:
-; SSE: # %bb.0:
-; SSE-NEXT: movsd {{.*#+}} xmm0 = mem[0],zero
-; SSE-NEXT: retq
-;
-; AVX-LABEL: speculative_load_f64:
-; AVX: # %bb.0:
-; AVX-NEXT: vmovsd {{.*#+}} xmm0 = mem[0],zero
-; AVX-NEXT: retq
- %load = call double @llvm.speculative.load.f64.p0(ptr %ptr, i64 0)
- ret double %load
+define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i16_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <8 x i16> (ptr, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i16> %load
}
-declare i32 @llvm.speculative.load.i32.p0(ptr, i64)
-declare i64 @llvm.speculative.load.i64.p0(ptr, i64)
-declare float @llvm.speculative.load.f32.p0(ptr, i64)
-declare double @llvm.speculative.load.f64.p0(ptr, i64)
+define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v16i8_oracle:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <16 x i8> (ptr, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ ret <16 x i8> %load
+}
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
index c06881a5f50af..34629849025dc 100644
--- a/llvm/test/Verifier/speculative-load.ll
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -1,18 +1,80 @@
; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
-declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, i64)
-declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, i64)
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, ...)
+declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, ...)
+declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, ...)
+
+declare i32 @bad_oracle_ret(ptr, i64) memory(argmem: read)
+declare i64 @good_oracle(ptr, i64) memory(argmem: read)
+declare i64 @oracle_i32_param(i32) memory(argmem: read)
+declare i64 @side_effecting_oracle(ptr, i64)
define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
- %res = call <3 x i32> @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
+; CHECK-NEXT: %res = call <3 x i32> (ptr, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
+ %res = call <3 x i32> (ptr, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
ret <3 x i32> %res
}
define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
- %res = call <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
+; CHECK-NEXT: %res = call <vscale x 3 x i32> (ptr, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
+ %res = call <vscale x 3 x i32> (ptr, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
ret <vscale x 3 x i32> %res
}
+
+define <4 x i32> @test_oracle_wrong_return_type(ptr %ptr, i64 %n) {
+; CHECK: llvm.speculative.load oracle function must return i64
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_oracle_too_few_args(ptr %ptr) {
+; CHECK: llvm.speculative.load oracle function argument count mismatch
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_oracle_too_many_args(ptr %ptr, i64 %n) {
+; CHECK: llvm.speculative.load oracle function argument count mismatch
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_oracle_arg_type_mismatch(ptr %ptr) {
+; CHECK: llvm.speculative.load oracle function argument type mismatch
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @oracle_i32_param, i64 42)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @oracle_i32_param, i64 42)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_non_function_oracle(ptr %ptr, ptr %not_fn) {
+; CHECK: llvm.speculative.load second argument must be i64 or a direct reference to an oracle function
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr %not_fn)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr %not_fn)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_direct_form_extra_args(ptr %ptr) {
+; CHECK: llvm.speculative.load direct form requires exactly 2 arguments
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 8, i64 4)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 8, i64 4)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_oracle_side_effects(ptr %ptr, i64 %n) {
+; CHECK: llvm.speculative.load oracle function must not have side effects and may only read memory through its arguments
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_num_accessible_bytes_exceeds_size(ptr %ptr) {
+; CHECK: llvm.speculative.load num_accessible_bytes must not exceed the result size in bytes
+; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 32)
+ %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 32)
+ ret <4 x i32> %res
+}
>From d65e3068a58fe31c0a9fde41e4876927278e8c60 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Mon, 30 Mar 2026 16:30:28 +0100
Subject: [PATCH 09/14] !fixup add from_end argument
---
llvm/docs/LangRef.rst | 41 +++++++-----
llvm/include/llvm/IR/Intrinsics.td | 5 +-
.../SelectionDAG/SelectionDAGBuilder.cpp | 2 +-
llvm/lib/IR/Verifier.cpp | 28 ++++----
.../AArch64/speculative-load-intrinsic.ll | 60 +++++++++++++----
.../CodeGen/X86/speculative-load-intrinsic.ll | 58 ++++++++++++++---
llvm/test/Verifier/speculative-load.ll | 65 ++++++++++---------
7 files changed, 178 insertions(+), 81 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index a3878965cefea..40a3b277cc9d9 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28345,7 +28345,7 @@ Speculative Load Intrinsics
LLVM provides intrinsics for speculatively loading memory that may be
out-of-bounds. These intrinsics enable optimizations like early-exit loop
vectorization where the vectorized loop may read beyond the end of an array,
-provided the access is guaranteed be valid by target-specific checks.
+provided the access is guaranteed to be valid by target-specific checks.
.. _int_speculative_load:
@@ -28359,11 +28359,11 @@ This is an overloaded intrinsic.
::
; Direct form: number of accessible bytes given as i64
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i64 <num_accessible_bytes>)
- declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i64 <num_accessible_bytes>)
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
+ declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
; Oracle form: accessible bytes computed by calling oracle_fn(args...)
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, ptr <oracle_fn>, ...)
+ declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i1 <from_end>, ptr <oracle_fn>, ...)
Overview:
"""""""""
@@ -28378,25 +28378,36 @@ Arguments:
""""""""""
The first argument is a pointer to the memory location to load from. The return
-type must be a vector type with a power-of-2 size in bytes. The remaining
-arguments determine the *number of accessible bytes* starting from ``ptr``,
+type must be a vector type with a power-of-2 size in bytes. The second argument
+is an ``i1`` constant flag ``from_end`` that specifies whether the ``N``
+accessible bytes are counted from the start or the end of the loaded values (see
+Semantics). The remaining arguments determine the *number of accessible bytes*,
denoted ``N`` below.
-In the **direct form**, the second argument is an ``i64`` specifying ``N``
-directly. In the **oracle form**, the second argument must be a direct
+In the **direct form**, the third argument is an ``i64`` specifying ``N``
+directly. In the **oracle form**, the third argument must be a direct
reference to a function returning ``i64`` that may only read memory through its
arguments (indirect function pointers are not permitted); the remaining
-arguments are forwarded to it, and the return value is ``N``.
+arguments are forwarded to it, and its return value is ``N``.
Semantics:
""""""""""
-The intrinsic reads ``N`` bytes starting from ``ptr`` and returns the stored
-values for those bytes. These bytes must lie within the bounds of an allocated
-object that ``ptr`` is :ref:`based <pointeraliasing>` on.
+Let ``S`` denote the size of the return type in bytes. The intrinsic performs
+a load of ``S`` bytes starting from ``ptr``.
-Bytes at offsets in ``[N, size of the return type)`` are ``poison`` and are not
-considered accessed for the purposes of data races or ``noalias`` constraints.
+When ``from_end`` is ``false``, the first ``N`` bytes (offsets ``[0, N)``)
+are the stored values read from memory. Bytes at offsets ``[N, S)`` are
+``poison``.
+
+When ``from_end`` is ``true``, the last ``N`` bytes (offsets ``[S - N, S)``)
+are the stored values read from memory. Bytes at offsets ``[0, S - N)`` are
+``poison``.
+
+In both cases, the ``N`` accessible bytes must lie within the bounds of an
+allocated object that ``ptr`` is :ref:`based <pointeraliasing>` on, and
+poison bytes are not considered accessed for the purposes of data races or
+``noalias`` constraints. The behavior is undefined if ``N`` exceeds ``S``.
The behavior is undefined if the speculative load accesses memory that would
fault (i.e., the oracle or ``llvm.can.load.speculatively`` would indicate the
@@ -28453,7 +28464,7 @@ alignment guarantees all such loads cannot cross a page boundary.
speculative_path:
; Safe to speculatively load from %ptr
- %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 16)
+ %vec = call <4 x i32> @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 16)
...
safe_path:
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 4680ea52f62dc..c89b4b24a433e 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2716,8 +2716,9 @@ def int_masked_srem:
// Speculatively load a value from memory; lowers to a regular aligned load.
def int_speculative_load:
DefaultAttrsIntrinsic<[llvm_anyvector_ty],
- [llvm_anyptr_ty, llvm_vararg_ty],
- [IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>]>;
+ [llvm_anyptr_ty, llvm_i1_ty, llvm_vararg_ty],
+ [IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>,
+ ImmArg<ArgIndex<1>>]>;
// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
// The size can be a runtime value to support scalable vectors.
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 74edd5fe49c86..a6bb010d69558 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5167,7 +5167,7 @@ void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
MachineMemOperand::Flags MMOFlags = MachineMemOperand::MOLoad;
LocationSize LocSize = StoreSize.isScalable()
? LocationSize::beforeOrAfterPointer()
- : LocationSize::precise(StoreSize);
+ : LocationSize::upperBound(StoreSize);
MachineMemOperand *MMO = DAG.getMachineFunction().getMachineMemOperand(
MachinePointerInfo(PtrOperand), MMOFlags, LocSize, Alignment, AAInfo);
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 8b8e3f093b07b..8efe756918030 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -6955,26 +6955,25 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
"llvm.speculative.load type must have a power-of-2 size", &Call);
unsigned NumArgs = Call.arg_size();
- Check(NumArgs >= 2, "llvm.speculative.load requires at least 2 arguments",
+ Check(NumArgs >= 3, "llvm.speculative.load requires at least 3 arguments",
&Call);
- Value *SecondArg = Call.getArgOperand(1);
- if (SecondArg->getType()->isIntegerTy(64)) {
- // Direct form: (ptr, i64 num_accessible_bytes)
- Check(NumArgs == 2,
- "llvm.speculative.load direct form requires exactly 2 arguments",
- &Call);
- if (auto *CI = dyn_cast<ConstantInt>(SecondArg)) {
+ Value *PayloadArg = Call.getArgOperand(2);
+ if (PayloadArg->getType()->isIntegerTy(64)) {
+ // Direct form: (ptr, i1 from_end, i64 num_accessible_bytes)
+ Check(NumArgs == 3,
+ "llvm.speculative.load direct form has too many arguments", &Call);
+ if (auto *CI = dyn_cast<ConstantInt>(PayloadArg)) {
Check(Size.isScalable() || CI->getZExtValue() <= Size.getFixedValue(),
"llvm.speculative.load num_accessible_bytes must not exceed "
"the result size in bytes",
&Call);
}
} else {
- // Oracle form: (ptr, oracle_fn_ptr, args...)
- auto *OracleFn = dyn_cast<Function>(SecondArg);
+ // Oracle form: (ptr, i1 from_end, oracle_fn_ptr, args...)
+ auto *OracleFn = dyn_cast<Function>(PayloadArg);
Check(OracleFn,
- "llvm.speculative.load second argument must be i64 or a direct "
+ "llvm.speculative.load third argument must be i64 or a direct "
"reference to an oracle function",
&Call);
@@ -6986,13 +6985,16 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
FunctionType *FTy = OracleFn->getFunctionType();
Check(FTy->getReturnType()->isIntegerTy(64),
"llvm.speculative.load oracle function must return i64", &Call);
- unsigned NumOracleArgs = NumArgs - 2;
+
+ unsigned OracleArgsStart = 3;
+ unsigned NumOracleArgs = NumArgs - OracleArgsStart;
Check(FTy->isVarArg() ? NumOracleArgs >= FTy->getNumParams()
: NumOracleArgs == FTy->getNumParams(),
"llvm.speculative.load oracle function argument count mismatch",
&Call);
for (unsigned I = 0, E = FTy->getNumParams(); I < E; ++I) {
- Check(FTy->getParamType(I) == Call.getArgOperand(I + 2)->getType(),
+ Check(FTy->getParamType(I) ==
+ Call.getArgOperand(I + OracleArgsStart)->getType(),
"llvm.speculative.load oracle function argument type mismatch",
&Call);
}
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
index 2746b9ea87daa..a547cee576864 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -11,7 +11,7 @@ define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 12)
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, i64 12)
ret <4 x i32> %load
}
@@ -20,7 +20,7 @@ define <2 x double> @speculative_load_v2f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i64 8)
+ %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, i64 8)
ret <2 x double> %load
}
@@ -31,7 +31,7 @@ define <vscale x 4 x i32> @speculative_load_nxv4i32(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> (ptr, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i64 8)
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 false, i64 8)
ret <vscale x 4 x i32> %load
}
@@ -40,7 +40,7 @@ define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 2 x double> (ptr, ...) @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i64 4)
+ %load = call <vscale x 2 x double> (ptr, i1, ...) @llvm.speculative.load.nxv2f64.p0(ptr align 16 %ptr, i1 false, i64 4)
ret <vscale x 2 x double> %load
}
@@ -53,7 +53,7 @@ define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x i32> %load
}
@@ -62,7 +62,7 @@ define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x i64> (ptr, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <2 x i64> (ptr, i1, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <2 x i64> %load
}
@@ -71,7 +71,7 @@ define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <8 x i16> (ptr, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <8 x i16> (ptr, i1, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <8 x i16> %load
}
@@ -80,7 +80,7 @@ define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <16 x i8> (ptr, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <16 x i8> (ptr, i1, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <16 x i8> %load
}
@@ -89,7 +89,7 @@ define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <4 x float> (ptr, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x float> (ptr, i1, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x float> %load
}
@@ -98,15 +98,53 @@ define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <2 x double> %load
}
+; from_end tests
+
+define <4 x i32> @speculative_load_v4i32_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_from_end:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, i64 12)
+ ret <4 x i32> %load
+}
+
+define <vscale x 4 x i32> @speculative_load_nxv4i32_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_nxv4i32_from_end:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 true, i64 8)
+ ret <vscale x 4 x i32> %load
+}
+
+define <4 x i32> @speculative_load_v4i32_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle_from_end:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ret
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
+}
+
+define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_nxv4i32_oracle_from_end:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr z0, [x0]
+; CHECK-NEXT: ret
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret <vscale x 4 x i32> %load
+}
+
define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle(ptr %ptr, i64 %n) {
; CHECK-LABEL: speculative_load_nxv4i32_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> (ptr, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <vscale x 4 x i32> %load
}
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index e387a17990012..fa345408940d1 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -9,7 +9,7 @@ define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i64 8)
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, i64 8)
ret <4 x i32> %load
}
@@ -18,7 +18,7 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %ymm0
; CHECK-NEXT: retq
- %load = call <8 x i32> (ptr, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i64 24)
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 false, i64 24)
ret <8 x i32> %load
}
@@ -30,7 +30,7 @@ define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x i32> %load
}
@@ -39,7 +39,7 @@ define <8 x i32> @speculative_load_v8i32_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %ymm0
; CHECK-NEXT: retq
- %load = call <8 x i32> (ptr, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <8 x i32> %load
}
@@ -48,7 +48,7 @@ define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <2 x i64> (ptr, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <2 x i64> (ptr, i1, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <2 x i64> %load
}
@@ -57,7 +57,7 @@ define <4 x i64> @speculative_load_v4i64_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %ymm0
; CHECK-NEXT: retq
- %load = call <4 x i64> (ptr, ...) @llvm.speculative.load.v4i64.p0(ptr align 32 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x i64> (ptr, i1, ...) @llvm.speculative.load.v4i64.p0(ptr align 32 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x i64> %load
}
@@ -66,7 +66,7 @@ define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <4 x float> (ptr, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <4 x float> (ptr, i1, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x float> %load
}
@@ -75,7 +75,7 @@ define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <2 x double> (ptr, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <2 x double> %load
}
@@ -84,7 +84,7 @@ define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <8 x i16> (ptr, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <8 x i16> (ptr, i1, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <8 x i16> %load
}
@@ -93,6 +93,44 @@ define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <16 x i8> (ptr, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, ptr @oracle, ptr %ptr, i64 %n)
+ %load = call <16 x i8> (ptr, i1, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
ret <16 x i8> %load
}
+
+; from_end tests
+
+define <4 x i32> @speculative_load_v4i32_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v4i32_from_end:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, i64 8)
+ ret <4 x i32> %load
+}
+
+define <8 x i32> @speculative_load_v8i32_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v8i32_from_end:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 true, i64 24)
+ ret <8 x i32> %load
+}
+
+define <4 x i32> @speculative_load_v4i32_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle_from_end:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: retq
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
+}
+
+define <8 x i32> @speculative_load_v8i32_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i32_oracle_from_end:
+; CHECK: # %bb.0:
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: retq
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i32> %load
+}
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
index 34629849025dc..7fd2d340679aa 100644
--- a/llvm/test/Verifier/speculative-load.ll
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -1,8 +1,8 @@
; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, ...)
-declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, ...)
-declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, ...)
+declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i1, ...)
+declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, i1, ...)
+declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, i1, ...)
declare i32 @bad_oracle_ret(ptr, i64) memory(argmem: read)
declare i64 @good_oracle(ptr, i64) memory(argmem: read)
@@ -11,70 +11,77 @@ declare i64 @side_effecting_oracle(ptr, i64)
define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <3 x i32> (ptr, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
- %res = call <3 x i32> (ptr, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i64 0)
+; CHECK-NEXT: %res = call <3 x i32> (ptr, i1, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i1 false, i64 0)
+ %res = call <3 x i32> (ptr, i1, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i1 false, i64 0)
ret <3 x i32> %res
}
define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <vscale x 3 x i32> (ptr, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
- %res = call <vscale x 3 x i32> (ptr, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i64 0)
+; CHECK-NEXT: %res = call <vscale x 3 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i1 false, i64 0)
+ %res = call <vscale x 3 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i1 false, i64 0)
ret <vscale x 3 x i32> %res
}
+define <4 x i32> @test_too_few_args(ptr %ptr) {
+; CHECK: llvm.speculative.load requires at least 3 arguments
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 true)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 true)
+ ret <4 x i32> %res
+}
+
+define <4 x i32> @test_direct_form_extra_args(ptr %ptr) {
+; CHECK: llvm.speculative.load direct form has too many arguments
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 8, i64 4)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 8, i64 4)
+ ret <4 x i32> %res
+}
+
define <4 x i32> @test_oracle_wrong_return_type(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function must return i64
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
ret <4 x i32> %res
}
define <4 x i32> @test_oracle_too_few_args(ptr %ptr) {
; CHECK: llvm.speculative.load oracle function argument count mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle)
ret <4 x i32> %res
}
define <4 x i32> @test_oracle_too_many_args(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function argument count mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
ret <4 x i32> %res
}
define <4 x i32> @test_oracle_arg_type_mismatch(ptr %ptr) {
; CHECK: llvm.speculative.load oracle function argument type mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @oracle_i32_param, i64 42)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @oracle_i32_param, i64 42)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @oracle_i32_param, i64 42)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @oracle_i32_param, i64 42)
ret <4 x i32> %res
}
define <4 x i32> @test_non_function_oracle(ptr %ptr, ptr %not_fn) {
-; CHECK: llvm.speculative.load second argument must be i64 or a direct reference to an oracle function
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr %not_fn)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr %not_fn)
- ret <4 x i32> %res
-}
-
-define <4 x i32> @test_direct_form_extra_args(ptr %ptr) {
-; CHECK: llvm.speculative.load direct form requires exactly 2 arguments
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 8, i64 4)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 8, i64 4)
+; CHECK: llvm.speculative.load third argument must be i64 or a direct reference to an oracle function
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr %not_fn)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr %not_fn)
ret <4 x i32> %res
}
define <4 x i32> @test_oracle_side_effects(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function must not have side effects and may only read memory through its arguments
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
ret <4 x i32> %res
}
define <4 x i32> @test_num_accessible_bytes_exceeds_size(ptr %ptr) {
; CHECK: llvm.speculative.load num_accessible_bytes must not exceed the result size in bytes
-; CHECK-NEXT: call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 32)
- %res = call <4 x i32> (ptr, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i64 32)
+; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 32)
+ %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 32)
ret <4 x i32> %res
}
>From aed6d4a5e0ea74fa65fe5fd3fa81b1450ec136ee Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Thu, 14 May 2026 18:25:02 +0100
Subject: [PATCH 10/14] !fixup address comments, thanks!
---
.../SelectionDAG/SelectionDAGBuilder.cpp | 5 +---
llvm/lib/IR/Verifier.cpp | 16 ----------
.../Target/AArch64/AArch64ISelLowering.cpp | 29 +++++++++----------
.../CodeGen/AArch64/can-load-speculatively.ll | 14 ++++-----
llvm/test/Verifier/can-load-speculatively.ll | 19 ------------
llvm/test/Verifier/speculative-load.ll | 2 --
6 files changed, 21 insertions(+), 64 deletions(-)
delete mode 100644 llvm/test/Verifier/can-load-speculatively.ll
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index a6bb010d69558..6d938ca8231be 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5158,16 +5158,13 @@ void SelectionDAGBuilder::visitSpeculativeLoad(const CallInst &I) {
EVT VT = TLI.getValueType(DAG.getDataLayout(), I.getType());
Align Alignment = I.getParamAlign(0).valueOrOne();
AAMDNodes AAInfo = I.getAAMetadata();
- TypeSize StoreSize = VT.getStoreSize();
SDValue InChain = DAG.getRoot();
// Use MOLoad but NOT MODereferenceable - the memory may not be
// fully dereferenceable.
MachineMemOperand::Flags MMOFlags = MachineMemOperand::MOLoad;
- LocationSize LocSize = StoreSize.isScalable()
- ? LocationSize::beforeOrAfterPointer()
- : LocationSize::upperBound(StoreSize);
+ LocationSize LocSize = LocationSize::upperBound(VT.getStoreSize());
MachineMemOperand *MMO = DAG.getMachineFunction().getMachineMemOperand(
MachinePointerInfo(PtrOperand), MMOFlags, LocSize, Alignment, AAInfo);
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 8efe756918030..b7ec928614e48 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -6963,12 +6963,6 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
// Direct form: (ptr, i1 from_end, i64 num_accessible_bytes)
Check(NumArgs == 3,
"llvm.speculative.load direct form has too many arguments", &Call);
- if (auto *CI = dyn_cast<ConstantInt>(PayloadArg)) {
- Check(Size.isScalable() || CI->getZExtValue() <= Size.getFixedValue(),
- "llvm.speculative.load num_accessible_bytes must not exceed "
- "the result size in bytes",
- &Call);
- }
} else {
// Oracle form: (ptr, i1 from_end, oracle_fn_ptr, args...)
auto *OracleFn = dyn_cast<Function>(PayloadArg);
@@ -7001,16 +6995,6 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
}
break;
}
- case Intrinsic::can_load_speculatively: {
- // If size is a constant, verify it's a positive power of 2.
- if (auto *SizeCI = dyn_cast<ConstantInt>(Call.getArgOperand(1))) {
- uint64_t Size = SizeCI->getZExtValue();
- Check(Size > 0 && isPowerOf2_64(Size),
- "llvm.can.load.speculatively size must be a positive power of 2",
- &Call);
- }
- break;
- }
case Intrinsic::vector_insert: {
Value *Vec = Call.getArgOperand(0);
Value *SubVec = Call.getArgOperand(1);
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
index 7df6cf8394d33..a31a616fd90a1 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
@@ -31410,9 +31410,8 @@ Value *AArch64TargetLowering::emitStoreConditional(IRBuilderBase &Builder,
Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
Value *Ptr,
Value *Size) const {
- unsigned AS = cast<PointerType>(Ptr->getType())->getAddressSpace();
// Conservatively only allow speculation for address space 0.
- if (AS != 0)
+ if (cast<PointerType>(Ptr->getType())->getAddressSpace() != 0)
return nullptr;
// For power-of-2 sizes <= 16, emit alignment check: (ptr & (size - 1)) == 0.
// If the pointer is aligned to at least 'size' bytes, loading 'size' bytes
@@ -31423,9 +31422,8 @@ Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
// The alignment check only works for power-of-2 sizes. For non-power-of-2
// sizes, we conservatively return false.
const DataLayout &DL = Builder.GetInsertBlock()->getModule()->getDataLayout();
+ Type *AddrTy = DL.getAddressType(Ptr->getType());
- unsigned PtrBits = DL.getPointerSizeInBits(AS);
- Type *IntPtrTy = Builder.getIntNTy(PtrBits);
if (auto *CI = dyn_cast<ConstantInt>(Size)) {
uint64_t SizeVal = CI->getZExtValue();
assert(isPowerOf2_64(SizeVal) && "size must be power-of-two");
@@ -31434,25 +31432,24 @@ Value *AArch64TargetLowering::emitCanLoadSpeculatively(IRBuilderBase &Builder,
return nullptr;
// Power-of-2 constant size <= 16: use fast alignment check.
- Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
- Value *Mask = ConstantInt::get(IntPtrTy, SizeVal - 1);
- Value *Masked = Builder.CreateAnd(PtrInt, Mask);
- return Builder.CreateICmpEQ(Masked, ConstantInt::get(IntPtrTy, 0));
+ Value *PtrAddr = Builder.CreatePtrToAddr(Ptr);
+
+ Value *Mask = ConstantInt::get(AddrTy, SizeVal - 1);
+ Value *Masked = Builder.CreateAnd(PtrAddr, Mask);
+ return Builder.CreateICmpEQ(Masked, ConstantInt::get(AddrTy, 0));
}
// Check power-of-2 size <= 16 and alignment.
- Value *PtrInt = Builder.CreatePtrToInt(Ptr, IntPtrTy);
- Value *SizeExt = Builder.CreateZExtOrTrunc(Size, IntPtrTy);
+ Value *PtrAddr = Builder.CreatePtrToAddr(Ptr);
+ Value *SizeExt = Builder.CreateZExtOrTrunc(Size, AddrTy);
Value *SizeLE16 =
- Builder.CreateICmpULE(SizeExt, ConstantInt::get(IntPtrTy, 16));
+ Builder.CreateICmpULE(SizeExt, ConstantInt::get(AddrTy, 16));
// alignment check: (ptr & (size - 1)) == 0
- Value *SizeMinusOne =
- Builder.CreateSub(SizeExt, ConstantInt::get(IntPtrTy, 1));
- Value *Masked = Builder.CreateAnd(PtrInt, SizeMinusOne);
- Value *AlignCheck =
- Builder.CreateICmpEQ(Masked, ConstantInt::get(IntPtrTy, 0));
+ Value *SizeMinusOne = Builder.CreateSub(SizeExt, ConstantInt::get(AddrTy, 1));
+ Value *Masked = Builder.CreateAnd(PtrAddr, SizeMinusOne);
+ Value *AlignCheck = Builder.CreateICmpEQ(Masked, ConstantInt::get(AddrTy, 0));
return Builder.CreateAnd(SizeLE16, AlignCheck);
}
diff --git a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
index b6679f22b0989..c9d1bd10c1a10 100644
--- a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
+++ b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
@@ -8,7 +8,7 @@
define i1 @can_load_speculatively_16(ptr %ptr) {
; CHECK-LABEL: @can_load_speculatively_16(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[PTR:%.*]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
; CHECK-NEXT: ret i1 [[TMP3]]
@@ -20,7 +20,7 @@ define i1 @can_load_speculatively_16(ptr %ptr) {
define i1 @can_load_speculatively_8_ptr_aligned(ptr align 8 %ptr) {
; CHECK-LABEL: @can_load_speculatively_8_ptr_aligned(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[PTR:%.*]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
; CHECK-NEXT: ret i1 [[TMP3]]
@@ -31,7 +31,7 @@ define i1 @can_load_speculatively_8_ptr_aligned(ptr align 8 %ptr) {
define i1 @can_load_speculatively_16_ptr_aligned(ptr align 16 %ptr) {
; CHECK-LABEL: @can_load_speculatively_16_ptr_aligned(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[PTR:%.*]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
; CHECK-NEXT: ret i1 [[TMP3]]
@@ -42,7 +42,7 @@ define i1 @can_load_speculatively_16_ptr_aligned(ptr align 16 %ptr) {
define i1 @can_load_speculatively_16_ptr_aligned2(ptr align 16 %ptr) {
; CHECK-LABEL: @can_load_speculatively_16_ptr_aligned2(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[PTR:%.*]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
; CHECK-NEXT: ret i1 [[TMP3]]
@@ -53,7 +53,7 @@ define i1 @can_load_speculatively_16_ptr_aligned2(ptr align 16 %ptr) {
define i1 @can_load_speculatively_32_ptr_aligned(ptr align 32 %ptr) {
; CHECK-LABEL: @can_load_speculatively_32_ptr_aligned(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[PTR:%.*]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 15
; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
; CHECK-NEXT: ret i1 [[TMP3]]
@@ -92,7 +92,7 @@ define i1 @can_load_speculatively_addrspace1(ptr addrspace(1) %ptr) {
; Test size 8 (within limit, power-of-2)
define i1 @can_load_speculatively_8(ptr %ptr) {
; CHECK-LABEL: @can_load_speculatively_8(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[PTR:%.*]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = and i64 [[TMP1]], 7
; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[TMP2]], 0
; CHECK-NEXT: ret i1 [[TMP3]]
@@ -104,7 +104,7 @@ define i1 @can_load_speculatively_8(ptr %ptr) {
; Test with runtime size - checks size <= 16 and alignment
define i1 @can_load_speculatively_runtime(ptr %ptr, i64 %size) {
; CHECK-LABEL: @can_load_speculatively_runtime(
-; CHECK-NEXT: [[TMP1:%.*]] = ptrtoint ptr [[PTR:%.*]] to i64
+; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[PTR:%.*]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = icmp ule i64 [[SIZE:%.*]], 16
; CHECK-NEXT: [[TMP3:%.*]] = sub i64 [[SIZE]], 1
; CHECK-NEXT: [[TMP4:%.*]] = and i64 [[TMP1]], [[TMP3]]
diff --git a/llvm/test/Verifier/can-load-speculatively.ll b/llvm/test/Verifier/can-load-speculatively.ll
deleted file mode 100644
index d2d69f70cfb60..0000000000000
--- a/llvm/test/Verifier/can-load-speculatively.ll
+++ /dev/null
@@ -1,19 +0,0 @@
-; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
-
-declare i1 @llvm.can.load.speculatively.p0(ptr, i64)
-
-; Test that constant size must be a positive power of 2
-
-define i1 @test_size_zero(ptr %ptr) {
-; CHECK: llvm.can.load.speculatively size must be a positive power of 2
-; CHECK-NEXT: %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 0)
- %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 0)
- ret i1 %res
-}
-
-define i1 @test_non_power_of_2(ptr %ptr) {
-; CHECK: llvm.can.load.speculatively size must be a positive power of 2
-; CHECK-NEXT: %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 3)
- %res = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 3)
- ret i1 %res
-}
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
index 7fd2d340679aa..f5e3f484d77c0 100644
--- a/llvm/test/Verifier/speculative-load.ll
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -80,8 +80,6 @@ define <4 x i32> @test_oracle_side_effects(ptr %ptr, i64 %n) {
}
define <4 x i32> @test_num_accessible_bytes_exceeds_size(ptr %ptr) {
-; CHECK: llvm.speculative.load num_accessible_bytes must not exceed the result size in bytes
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 32)
%res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 32)
ret <4 x i32> %res
}
>From 73990010f125972be8e2e0c2625e612a30075d43 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Thu, 14 May 2026 20:50:57 +0100
Subject: [PATCH 11/14] !fixup refine LanfRef
---
llvm/docs/LangRef.rst | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 40a3b277cc9d9..b6163ddd9efe5 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28370,9 +28370,9 @@ Overview:
The '``llvm.speculative.load``' intrinsic loads a value from memory. Unlike a
regular load, the memory access may extend beyond the bounds of the allocated
-object, provided the pointer has been verified by
-:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` to ensure the
-access is valid.
+object, provided the memory safely accessed on the underlying hardware.
+:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` can be used to
+check if the access is.
Arguments:
""""""""""
@@ -28450,7 +28450,9 @@ This intrinsic has **target-dependent** semantics. It returns ``true`` if
``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for all non-negative
integers ``I`` where the computed address does not wrap around the address
space, can be loaded speculatively, even if the memory is beyond the bounds of
-an allocated object. It returns ``false`` otherwise.
+an allocated object. It returns ``false`` otherwise. The first byte of each
+access must be part of the same underlying object as ``ptr`` in order to
+speculatively load the whole range.
The specific conditions under which this intrinsic returns ``true`` are
determined by the target. For example, a target may check whether the pointer
>From 133f8794a54ce1ffecf32b5be6191f72c2e169e6 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Fri, 15 May 2026 13:52:46 +0100
Subject: [PATCH 12/14] !fixup first or last byte deref
---
llvm/docs/LangRef.rst | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index b6163ddd9efe5..9a9828638f5e4 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28450,8 +28450,8 @@ This intrinsic has **target-dependent** semantics. It returns ``true`` if
``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for all non-negative
integers ``I`` where the computed address does not wrap around the address
space, can be loaded speculatively, even if the memory is beyond the bounds of
-an allocated object. It returns ``false`` otherwise. The first byte of each
-access must be part of the same underlying object as ``ptr`` in order to
+an allocated object. It returns ``false`` otherwise. The first or last byte of
+each access must be part of the same underlying object as ``ptr`` in order to
speculatively load the whole range.
The specific conditions under which this intrinsic returns ``true`` are
>From b070938753d724e950036f8bc8a4265fa633191e Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Fri, 15 May 2026 15:10:07 +0100
Subject: [PATCH 13/14] !fixup support byte type in addition to vector type
---
llvm/docs/LangRef.rst | 36 +++--
llvm/include/llvm/IR/Intrinsics.td | 5 +-
llvm/lib/IR/Verifier.cpp | 22 ++-
.../CodeGen/AArch64/can-load-speculatively.ll | 13 +-
.../AArch64/speculative-load-intrinsic.ll | 105 ++++++------
.../CodeGen/X86/speculative-load-intrinsic.ll | 148 ++++++++++-------
llvm/test/Verifier/speculative-load.ll | 153 ++++++++++++------
7 files changed, 289 insertions(+), 193 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 9a9828638f5e4..df2257f772351 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28359,30 +28359,32 @@ This is an overloaded intrinsic.
::
; Direct form: number of accessible bytes given as i64
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
- declare <8 x i32> @llvm.speculative.load.v8i32.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
+ declare b128 @llvm.speculative.load.b128.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
+ declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
+ declare <vscale x 4 x i32>
+ @llvm.speculative.load.nxv4i32.p0(ptr <ptr>, i1 <from_end>, i64 <num_accessible_bytes>)
; Oracle form: accessible bytes computed by calling oracle_fn(args...)
- declare <4 x float> @llvm.speculative.load.v4f32.p0(ptr <ptr>, i1 <from_end>, ptr <oracle_fn>, ...)
+ declare b128 @llvm.speculative.load.b128.p0(ptr <ptr>, i1 <from_end>, ptr <oracle_fn>, ...)
Overview:
"""""""""
The '``llvm.speculative.load``' intrinsic loads a value from memory. Unlike a
regular load, the memory access may extend beyond the bounds of the allocated
-object, provided the memory safely accessed on the underlying hardware.
+object, provided the memory can be safely accessed on the underlying hardware.
:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` can be used to
-check if the access is.
+check if the access is safe.
Arguments:
""""""""""
The first argument is a pointer to the memory location to load from. The return
-type must be a vector type with a power-of-2 size in bytes. The second argument
-is an ``i1`` constant flag ``from_end`` that specifies whether the ``N``
-accessible bytes are counted from the start or the end of the loaded values (see
-Semantics). The remaining arguments determine the *number of accessible bytes*,
-denoted ``N`` below.
+type must be a byte type or a vector type, and its size in bytes must be a
+positive power of 2. The second argument is an ``i1`` constant flag ``from_end``
+selecting whether the ``N`` accessible bytes are counted from the start or end
+of the loaded value (see Semantics). The remaining arguments determine the
+*number of accessible bytes*, denoted ``N`` below.
In the **direct form**, the third argument is an ``i64`` specifying ``N``
directly. In the **oracle form**, the third argument must be a direct
@@ -28393,8 +28395,7 @@ arguments are forwarded to it, and its return value is ``N``.
Semantics:
""""""""""
-Let ``S`` denote the size of the return type in bytes. The intrinsic performs
-a load of ``S`` bytes starting from ``ptr``.
+Let ``S`` denote the size of the return type in bytes.
When ``from_end`` is ``false``, the first ``N`` bytes (offsets ``[0, N)``)
are the stored values read from memory. Bytes at offsets ``[N, S)`` are
@@ -28409,9 +28410,10 @@ allocated object that ``ptr`` is :ref:`based <pointeraliasing>` on, and
poison bytes are not considered accessed for the purposes of data races or
``noalias`` constraints. The behavior is undefined if ``N`` exceeds ``S``.
-The behavior is undefined if the speculative load accesses memory that would
-fault (i.e., the oracle or ``llvm.can.load.speculatively`` would indicate the
-access is not safe).
+The behavior is undefined if any byte the underlying load actually reads is
+not safe to speculatively access.
+:ref:`llvm.can.load.speculatively <int_can_load_speculatively>` can be used
+to check safety.
.. _int_can_load_speculatively:
@@ -28439,8 +28441,8 @@ Arguments:
The first argument is a pointer to the memory location.
-The second argument is an i64 specifying the size in bytes of the load.
-The size must be a positive power of 2. If the size is not a power-of-2, the
+The second argument is an i64 specifying the number of accessed bytes,
+and must be a positive power of 2. If the size is not a power-of-2, the
result is ``poison``.
Semantics:
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index c89b4b24a433e..3d4993754f7d2 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2713,15 +2713,14 @@ def int_masked_srem:
LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>],
[IntrNoMem]>;
-// Speculatively load a value from memory; lowers to a regular aligned load.
+// Speculatively load a value from memory.
def int_speculative_load:
- DefaultAttrsIntrinsic<[llvm_anyvector_ty],
+ DefaultAttrsIntrinsic<[llvm_any_ty],
[llvm_anyptr_ty, llvm_i1_ty, llvm_vararg_ty],
[IntrReadMem, IntrArgMemOnly, IntrWillReturn, NoCapture<ArgIndex<0>>,
ImmArg<ArgIndex<1>>]>;
// Returns true if it's safe to speculatively load 'num_bytes' from 'ptr'.
-// The size can be a runtime value to support scalable vectors.
def int_can_load_speculatively:
DefaultAttrsIntrinsic<[llvm_i1_ty],
[llvm_anyptr_ty, llvm_i64_ty],
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index b7ec928614e48..4d0fec64ab700 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -6949,10 +6949,24 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
}
case Intrinsic::speculative_load: {
Type *LoadTy = Call.getType();
- TypeSize Size = DL.getTypeStoreSize(LoadTy);
- // For scalable vectors, check the known minimum size is a power of 2.
- Check(Size.getKnownMinValue() > 0 && isPowerOf2_64(Size.getKnownMinValue()),
- "llvm.speculative.load type must have a power-of-2 size", &Call);
+ Check(LoadTy->isByteTy() || LoadTy->isVectorTy(),
+ "llvm.speculative.load return type must be a byte type or a "
+ "vector type",
+ &Call);
+ if (LoadTy->isByteOrByteVectorTy()) {
+ unsigned BitWidth = LoadTy->getScalarType()->getByteBitWidth();
+ Check((BitWidth % 8) == 0,
+ "llvm.speculative.load byte type must have a bit width that is "
+ "a multiple of 8",
+ &Call);
+ }
+
+ TypeSize SizeInBits = LoadTy->getPrimitiveSizeInBits();
+ uint64_t MinBits = SizeInBits.getKnownMinValue();
+ Check(MinBits != 0 && (MinBits % 8) == 0 && isPowerOf2_64(MinBits / 8),
+ "llvm.speculative.load return type size in bytes must be a "
+ "positive power of 2",
+ &Call);
unsigned NumArgs = Call.arg_size();
Check(NumArgs >= 3, "llvm.speculative.load requires at least 3 arguments",
diff --git a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
index c9d1bd10c1a10..0b1225a4bd0e3 100644
--- a/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
+++ b/llvm/test/CodeGen/AArch64/can-load-speculatively.ll
@@ -2,9 +2,8 @@
; RUN: opt -mtriple=aarch64-unknown-linux-gnu -passes=pre-isel-intrinsic-lowering -S < %s | FileCheck %s
; Test that @llvm.can.load.speculatively is lowered to an alignment check
-; for power-of-2 sizes <= 16 bytes on AArch64, and returns false for larger sizes.
+; for power-of-2 sizes <= 16 bytes on AArch64, and returns false otherwise.
; The 16-byte limit ensures correctness with MTE (memory tagging).
-; Note: non-power-of-2 constant sizes are rejected by the verifier.
define i1 @can_load_speculatively_16(ptr %ptr) {
; CHECK-LABEL: @can_load_speculatively_16(
@@ -101,16 +100,16 @@ define i1 @can_load_speculatively_8(ptr %ptr) {
ret i1 %can_load
}
-; Test with runtime size - checks size <= 16 and alignment
+; Test with runtime size - checks size <= 16, power-of-2, and alignment
define i1 @can_load_speculatively_runtime(ptr %ptr, i64 %size) {
; CHECK-LABEL: @can_load_speculatively_runtime(
; CHECK-NEXT: [[TMP1:%.*]] = ptrtoaddr ptr [[PTR:%.*]] to i64
; CHECK-NEXT: [[TMP2:%.*]] = icmp ule i64 [[SIZE:%.*]], 16
; CHECK-NEXT: [[TMP3:%.*]] = sub i64 [[SIZE]], 1
-; CHECK-NEXT: [[TMP4:%.*]] = and i64 [[TMP1]], [[TMP3]]
-; CHECK-NEXT: [[TMP5:%.*]] = icmp eq i64 [[TMP4]], 0
-; CHECK-NEXT: [[TMP6:%.*]] = and i1 [[TMP2]], [[TMP5]]
-; CHECK-NEXT: ret i1 [[TMP6]]
+; CHECK-NEXT: [[TMP8:%.*]] = and i64 [[TMP1]], [[TMP3]]
+; CHECK-NEXT: [[TMP9:%.*]] = icmp eq i64 [[TMP8]], 0
+; CHECK-NEXT: [[TMP11:%.*]] = and i1 [[TMP2]], [[TMP9]]
+; CHECK-NEXT: ret i1 [[TMP11]]
;
%can_load = call i1 @llvm.can.load.speculatively.p0(ptr %ptr, i64 %size)
ret i1 %can_load
diff --git a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
index a547cee576864..e407fbc4adae2 100644
--- a/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/AArch64/speculative-load-intrinsic.ll
@@ -2,9 +2,30 @@
; RUN: llc -mtriple=aarch64-unknown-linux-gnu -mattr=+sve < %s | FileCheck %s
; Test that @llvm.speculative.load is lowered to a regular load
-; in SelectionDAG for fixed vectors, scalable vectors, and scalars.
+; in SelectionDAG for fixed vectors, scalable vectors, and bytes.
-; Fixed-width vector tests
+; Byte-type tests
+
+define b128 @speculative_load_b128(ptr %ptr) {
+; CHECK-LABEL: speculative_load_b128:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldp x8, x1, [x0]
+; CHECK-NEXT: mov x0, x8
+; CHECK-NEXT: ret
+ %load = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr align 16 %ptr, i1 false, i64 12)
+ ret b128 %load
+}
+
+define b64 @speculative_load_b64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_b64:
+; CHECK: // %bb.0:
+; CHECK-NEXT: ldr x0, [x0]
+; CHECK-NEXT: ret
+ %load = call b64 (ptr, i1, ...) @llvm.speculative.load.b64.p0(ptr align 8 %ptr, i1 false, i64 8)
+ ret b64 %load
+}
+
+; Fixed-vector tests
define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
; CHECK-LABEL: speculative_load_v4i32:
@@ -48,62 +69,46 @@ define <vscale x 2 x double> @speculative_load_nxv2f64(ptr %ptr) {
declare i64 @oracle(ptr, i64) memory(argmem: read)
-define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v4i32_oracle:
+define b128 @speculative_load_b128_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_b128_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ldp x8, x1, [x0]
+; CHECK-NEXT: mov x0, x8
; CHECK-NEXT: ret
- %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <4 x i32> %load
-}
-
-define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v2i64_oracle:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr q0, [x0]
-; CHECK-NEXT: ret
- %load = call <2 x i64> (ptr, i1, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <2 x i64> %load
+ %load = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
+ ret b128 %load
}
-define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v8i16_oracle:
+define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle:
; CHECK: // %bb.0:
; CHECK-NEXT: ldr q0, [x0]
; CHECK-NEXT: ret
- %load = call <8 x i16> (ptr, i1, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <8 x i16> %load
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
}
-define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v16i8_oracle:
+define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_nxv4i32_oracle:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ldr z0, [x0]
; CHECK-NEXT: ret
- %load = call <16 x i8> (ptr, i1, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <16 x i8> %load
+ %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
+ ret <vscale x 4 x i32> %load
}
-define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v4f32_oracle:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr q0, [x0]
-; CHECK-NEXT: ret
- %load = call <4 x float> (ptr, i1, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <4 x float> %load
-}
+; from_end tests
-define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v2f64_oracle:
+define b128 @speculative_load_b128_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_b128_from_end:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ldp x8, x1, [x0]
+; CHECK-NEXT: mov x0, x8
; CHECK-NEXT: ret
- %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <2 x double> %load
+ %load = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr align 16 %ptr, i1 true, i64 12)
+ ret b128 %load
}
-; from_end tests
-
define <4 x i32> @speculative_load_v4i32_from_end(ptr %ptr) {
; CHECK-LABEL: speculative_load_v4i32_from_end:
; CHECK: // %bb.0:
@@ -122,13 +127,14 @@ define <vscale x 4 x i32> @speculative_load_nxv4i32_from_end(ptr %ptr) {
ret <vscale x 4 x i32> %load
}
-define <4 x i32> @speculative_load_v4i32_oracle_from_end(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v4i32_oracle_from_end:
+define b128 @speculative_load_b128_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_b128_oracle_from_end:
; CHECK: // %bb.0:
-; CHECK-NEXT: ldr q0, [x0]
+; CHECK-NEXT: ldp x8, x1, [x0]
+; CHECK-NEXT: mov x0, x8
; CHECK-NEXT: ret
- %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
- ret <4 x i32> %load
+ %load = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret b128 %load
}
define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle_from_end(ptr %ptr, i64 %n) {
@@ -139,12 +145,3 @@ define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle_from_end(ptr %ptr, i6
%load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
ret <vscale x 4 x i32> %load
}
-
-define <vscale x 4 x i32> @speculative_load_nxv4i32_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_nxv4i32_oracle:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ldr z0, [x0]
-; CHECK-NEXT: ret
- %load = call <vscale x 4 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <vscale x 4 x i32> %load
-}
diff --git a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
index fa345408940d1..60ad671321777 100644
--- a/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
+++ b/llvm/test/CodeGen/X86/speculative-load-intrinsic.ll
@@ -2,7 +2,42 @@
; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2 < %s | FileCheck %s
; Test that @llvm.speculative.load is lowered to a regular load
-; in SelectionDAG.
+; in SelectionDAG for fixed vectors and bytes.
+
+; Byte-type tests
+
+define b128 @speculative_load_b128(ptr %ptr) {
+; CHECK-LABEL: speculative_load_b128:
+; CHECK: # %bb.0:
+; CHECK-NEXT: movq (%rdi), %rax
+; CHECK-NEXT: movq 8(%rdi), %rdx
+; CHECK-NEXT: retq
+ %load = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr align 16 %ptr, i1 false, i64 8)
+ ret b128 %load
+}
+
+define b256 @speculative_load_b256(ptr %ptr) {
+; CHECK-LABEL: speculative_load_b256:
+; CHECK: # %bb.0:
+; CHECK-NEXT: movq %rdi, %rax
+; CHECK-NEXT: vmovaps (%rsi), %ymm0
+; CHECK-NEXT: vmovups %ymm0, (%rdi)
+; CHECK-NEXT: vzeroupper
+; CHECK-NEXT: retq
+ %load = call b256 (ptr, i1, ...) @llvm.speculative.load.b256.p0(ptr align 32 %ptr, i1 false, i64 24)
+ ret b256 %load
+}
+
+define b64 @speculative_load_b64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_b64:
+; CHECK: # %bb.0:
+; CHECK-NEXT: movq (%rdi), %rax
+; CHECK-NEXT: retq
+ %load = call b64 (ptr, i1, ...) @llvm.speculative.load.b64.p0(ptr align 8 %ptr, i1 false, i64 8)
+ ret b64 %load
+}
+
+; Fixed-vector tests
define <4 x i32> @speculative_load_v4i32(ptr %ptr) {
; CHECK-LABEL: speculative_load_v4i32:
@@ -22,83 +57,82 @@ define <8 x i32> @speculative_load_v8i32(ptr %ptr) {
ret <8 x i32> %load
}
-; Oracle form tests
-declare i64 @oracle(ptr, i64) memory(argmem: read)
-
-define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v4i32_oracle:
+define <2 x double> @speculative_load_v2f64(ptr %ptr) {
+; CHECK-LABEL: speculative_load_v2f64:
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <4 x i32> %load
+ %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, i64 8)
+ ret <2 x double> %load
}
-define <8 x i32> @speculative_load_v8i32_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v8i32_oracle:
-; CHECK: # %bb.0:
-; CHECK-NEXT: vmovaps (%rdi), %ymm0
-; CHECK-NEXT: retq
- %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <8 x i32> %load
-}
+; Oracle form tests
+declare i64 @oracle(ptr, i64) memory(argmem: read)
-define <2 x i64> @speculative_load_v2i64_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v2i64_oracle:
+define b128 @speculative_load_b128_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_b128_oracle:
; CHECK: # %bb.0:
-; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: movq (%rdi), %rax
+; CHECK-NEXT: movq 8(%rdi), %rdx
; CHECK-NEXT: retq
- %load = call <2 x i64> (ptr, i1, ...) @llvm.speculative.load.v2i64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <2 x i64> %load
+ %load = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
+ ret b128 %load
}
-define <4 x i64> @speculative_load_v4i64_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v4i64_oracle:
+define b256 @speculative_load_b256_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_b256_oracle:
; CHECK: # %bb.0:
-; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: movq %rdi, %rax
+; CHECK-NEXT: vmovaps (%rsi), %ymm0
+; CHECK-NEXT: vmovups %ymm0, (%rdi)
+; CHECK-NEXT: vzeroupper
; CHECK-NEXT: retq
- %load = call <4 x i64> (ptr, i1, ...) @llvm.speculative.load.v4i64.p0(ptr align 32 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <4 x i64> %load
+ %load = call b256 (ptr, i1, ...) @llvm.speculative.load.b256.p0(ptr align 32 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
+ ret b256 %load
}
-define <4 x float> @speculative_load_v4f32_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v4f32_oracle:
+define <4 x i32> @speculative_load_v4i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v4i32_oracle:
; CHECK: # %bb.0:
; CHECK-NEXT: vmovaps (%rdi), %xmm0
; CHECK-NEXT: retq
- %load = call <4 x float> (ptr, i1, ...) @llvm.speculative.load.v4f32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <4 x float> %load
+ %load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
+ ret <4 x i32> %load
}
-define <2 x double> @speculative_load_v2f64_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v2f64_oracle:
+define <8 x i32> @speculative_load_v8i32_oracle(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_v8i32_oracle:
; CHECK: # %bb.0:
-; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: vmovaps (%rdi), %ymm0
; CHECK-NEXT: retq
- %load = call <2 x double> (ptr, i1, ...) @llvm.speculative.load.v2f64.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <2 x double> %load
+ %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
+ ret <8 x i32> %load
}
-define <8 x i16> @speculative_load_v8i16_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v8i16_oracle:
+; from_end tests
+
+define b128 @speculative_load_b128_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_b128_from_end:
; CHECK: # %bb.0:
-; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: movq (%rdi), %rax
+; CHECK-NEXT: movq 8(%rdi), %rdx
; CHECK-NEXT: retq
- %load = call <8 x i16> (ptr, i1, ...) @llvm.speculative.load.v8i16.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <8 x i16> %load
+ %load = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr align 16 %ptr, i1 true, i64 8)
+ ret b128 %load
}
-define <16 x i8> @speculative_load_v16i8_oracle(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v16i8_oracle:
+define b256 @speculative_load_b256_from_end(ptr %ptr) {
+; CHECK-LABEL: speculative_load_b256_from_end:
; CHECK: # %bb.0:
-; CHECK-NEXT: vmovaps (%rdi), %xmm0
+; CHECK-NEXT: movq %rdi, %rax
+; CHECK-NEXT: vmovaps (%rsi), %ymm0
+; CHECK-NEXT: vmovups %ymm0, (%rdi)
+; CHECK-NEXT: vzeroupper
; CHECK-NEXT: retq
- %load = call <16 x i8> (ptr, i1, ...) @llvm.speculative.load.v16i8.p0(ptr align 16 %ptr, i1 false, ptr @oracle, ptr %ptr, i64 %n)
- ret <16 x i8> %load
+ %load = call b256 (ptr, i1, ...) @llvm.speculative.load.b256.p0(ptr align 32 %ptr, i1 true, i64 24)
+ ret b256 %load
}
-; from_end tests
-
define <4 x i32> @speculative_load_v4i32_from_end(ptr %ptr) {
; CHECK-LABEL: speculative_load_v4i32_from_end:
; CHECK: # %bb.0:
@@ -108,13 +142,14 @@ define <4 x i32> @speculative_load_v4i32_from_end(ptr %ptr) {
ret <4 x i32> %load
}
-define <8 x i32> @speculative_load_v8i32_from_end(ptr %ptr) {
-; CHECK-LABEL: speculative_load_v8i32_from_end:
+define b128 @speculative_load_b128_oracle_from_end(ptr %ptr, i64 %n) {
+; CHECK-LABEL: speculative_load_b128_oracle_from_end:
; CHECK: # %bb.0:
-; CHECK-NEXT: vmovaps (%rdi), %ymm0
+; CHECK-NEXT: movq (%rdi), %rax
+; CHECK-NEXT: movq 8(%rdi), %rdx
; CHECK-NEXT: retq
- %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 true, i64 24)
- ret <8 x i32> %load
+ %load = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
+ ret b128 %load
}
define <4 x i32> @speculative_load_v4i32_oracle_from_end(ptr %ptr, i64 %n) {
@@ -125,12 +160,3 @@ define <4 x i32> @speculative_load_v4i32_oracle_from_end(ptr %ptr, i64 %n) {
%load = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr align 16 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
ret <4 x i32> %load
}
-
-define <8 x i32> @speculative_load_v8i32_oracle_from_end(ptr %ptr, i64 %n) {
-; CHECK-LABEL: speculative_load_v8i32_oracle_from_end:
-; CHECK: # %bb.0:
-; CHECK-NEXT: vmovaps (%rdi), %ymm0
-; CHECK-NEXT: retq
- %load = call <8 x i32> (ptr, i1, ...) @llvm.speculative.load.v8i32.p0(ptr align 32 %ptr, i1 true, ptr @oracle, ptr %ptr, i64 %n)
- ret <8 x i32> %load
-}
diff --git a/llvm/test/Verifier/speculative-load.ll b/llvm/test/Verifier/speculative-load.ll
index f5e3f484d77c0..fb9d06a10f7eb 100644
--- a/llvm/test/Verifier/speculative-load.ll
+++ b/llvm/test/Verifier/speculative-load.ll
@@ -1,85 +1,144 @@
; RUN: not llvm-as < %s -o /dev/null 2>&1 | FileCheck %s
-declare <4 x i32> @llvm.speculative.load.v4i32.p0(ptr, i1, ...)
+declare b128 @llvm.speculative.load.b128.p0(ptr, i1, ...)
+declare i32 @llvm.speculative.load.i32.p0(ptr, i1, ...)
+declare float @llvm.speculative.load.f32.p0(ptr, i1, ...)
+declare b12 @llvm.speculative.load.b12.p0(ptr, i1, ...)
+declare b1 @llvm.speculative.load.b1.p0(ptr, i1, ...)
+declare b9 @llvm.speculative.load.b9.p0(ptr, i1, ...)
+declare b24 @llvm.speculative.load.b24.p0(ptr, i1, ...)
declare <3 x i32> @llvm.speculative.load.v3i32.p0(ptr, i1, ...)
-declare <vscale x 3 x i32> @llvm.speculative.load.nxv3i32.p0(ptr, i1, ...)
+declare {i32, i32} @llvm.speculative.load.sl_i32i32s.p0(ptr, i1, ...)
+declare [4 x i32] @llvm.speculative.load.a4i32.p0(ptr, i1, ...)
+declare <4 x b3> @llvm.speculative.load.v4b3.p0(ptr, i1, ...)
declare i32 @bad_oracle_ret(ptr, i64) memory(argmem: read)
declare i64 @good_oracle(ptr, i64) memory(argmem: read)
declare i64 @oracle_i32_param(i32) memory(argmem: read)
declare i64 @side_effecting_oracle(ptr, i64)
-define <3 x i32> @test_non_power_of_2_fixed(ptr %ptr) {
-; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <3 x i32> (ptr, i1, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i1 false, i64 0)
- %res = call <3 x i32> (ptr, i1, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i1 false, i64 0)
- ret <3 x i32> %res
+define i32 @test_non_byte_non_vector_int(ptr %ptr) {
+; CHECK: llvm.speculative.load return type must be a byte type or a vector type
+; CHECK-NEXT: %res = call i32 (ptr, i1, ...) @llvm.speculative.load.i32.p0(ptr %ptr, i1 false, i64 0)
+ %res = call i32 (ptr, i1, ...) @llvm.speculative.load.i32.p0(ptr %ptr, i1 false, i64 0)
+ ret i32 %res
+}
+
+define float @test_non_byte_non_vector_float(ptr %ptr) {
+; CHECK: llvm.speculative.load return type must be a byte type or a vector type
+; CHECK-NEXT: %res = call float (ptr, i1, ...) @llvm.speculative.load.f32.p0(ptr %ptr, i1 false, i64 0)
+ %res = call float (ptr, i1, ...) @llvm.speculative.load.f32.p0(ptr %ptr, i1 false, i64 0)
+ ret float %res
+}
+
+define {i32, i32} @test_struct(ptr %ptr) {
+; CHECK: llvm.speculative.load return type must be a byte type or a vector type
+; CHECK-NEXT: %res = call { i32, i32 } (ptr, i1, ...) @llvm.speculative.load.sl_i32i32s.p0(ptr %ptr, i1 false, i64 0)
+ %res = call {i32, i32} (ptr, i1, ...) @llvm.speculative.load.sl_i32i32s.p0(ptr %ptr, i1 false, i64 0)
+ ret {i32, i32} %res
+}
+
+define [4 x i32] @test_array(ptr %ptr) {
+; CHECK: llvm.speculative.load return type must be a byte type or a vector type
+; CHECK-NEXT: %res = call [4 x i32] (ptr, i1, ...) @llvm.speculative.load.a4i32.p0(ptr %ptr, i1 false, i64 0)
+ %res = call [4 x i32] (ptr, i1, ...) @llvm.speculative.load.a4i32.p0(ptr %ptr, i1 false, i64 0)
+ ret [4 x i32] %res
+}
+
+define b1 @test_byte_b1(ptr %ptr) {
+; CHECK: llvm.speculative.load byte type must have a bit width that is a multiple of 8
+; CHECK-NEXT: %res = call b1 (ptr, i1, ...) @llvm.speculative.load.b1.p0(ptr %ptr, i1 false, i64 0)
+ %res = call b1 (ptr, i1, ...) @llvm.speculative.load.b1.p0(ptr %ptr, i1 false, i64 0)
+ ret b1 %res
+}
+
+define b9 @test_byte_b9(ptr %ptr) {
+; CHECK: llvm.speculative.load byte type must have a bit width that is a multiple of 8
+; CHECK-NEXT: %res = call b9 (ptr, i1, ...) @llvm.speculative.load.b9.p0(ptr %ptr, i1 false, i64 0)
+ %res = call b9 (ptr, i1, ...) @llvm.speculative.load.b9.p0(ptr %ptr, i1 false, i64 0)
+ ret b9 %res
+}
+
+define b12 @test_byte_not_multiple_of_8(ptr %ptr) {
+; CHECK: llvm.speculative.load byte type must have a bit width that is a multiple of 8
+; CHECK-NEXT: %res = call b12 (ptr, i1, ...) @llvm.speculative.load.b12.p0(ptr %ptr, i1 false, i64 0)
+ %res = call b12 (ptr, i1, ...) @llvm.speculative.load.b12.p0(ptr %ptr, i1 false, i64 0)
+ ret b12 %res
}
-define <vscale x 3 x i32> @test_non_power_of_2_scalable(ptr %ptr) {
-; CHECK: llvm.speculative.load type must have a power-of-2 size
-; CHECK-NEXT: %res = call <vscale x 3 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i1 false, i64 0)
- %res = call <vscale x 3 x i32> (ptr, i1, ...) @llvm.speculative.load.nxv3i32.p0(ptr %ptr, i1 false, i64 0)
- ret <vscale x 3 x i32> %res
+define <4 x b3> @test_byte_vector_not_multiple_of_8(ptr %ptr) {
+; CHECK: llvm.speculative.load byte type must have a bit width that is a multiple of 8
+; CHECK-NEXT: %res = call <4 x b3> (ptr, i1, ...) @llvm.speculative.load.v4b3.p0(ptr %ptr, i1 false, i64 0)
+ %res = call <4 x b3> (ptr, i1, ...) @llvm.speculative.load.v4b3.p0(ptr %ptr, i1 false, i64 0)
+ ret <4 x b3> %res
}
-define <4 x i32> @test_too_few_args(ptr %ptr) {
+define b128 @test_too_few_args(ptr %ptr) {
; CHECK: llvm.speculative.load requires at least 3 arguments
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 true)
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 true)
- ret <4 x i32> %res
+; CHECK-NEXT: call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 true)
+ %res = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 true)
+ ret b128 %res
}
-define <4 x i32> @test_direct_form_extra_args(ptr %ptr) {
+define b128 @test_direct_form_extra_args(ptr %ptr) {
; CHECK: llvm.speculative.load direct form has too many arguments
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 8, i64 4)
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 8, i64 4)
- ret <4 x i32> %res
+; CHECK-NEXT: call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, i64 8, i64 4)
+ %res = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, i64 8, i64 4)
+ ret b128 %res
}
-define <4 x i32> @test_oracle_wrong_return_type(ptr %ptr, i64 %n) {
+define b128 @test_oracle_wrong_return_type(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function must return i64
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
- ret <4 x i32> %res
+; CHECK-NEXT: call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+ %res = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @bad_oracle_ret, ptr %ptr, i64 %n)
+ ret b128 %res
}
-define <4 x i32> @test_oracle_too_few_args(ptr %ptr) {
+define b128 @test_oracle_too_few_args(ptr %ptr) {
; CHECK: llvm.speculative.load oracle function argument count mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle)
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle)
- ret <4 x i32> %res
+; CHECK-NEXT: call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @good_oracle)
+ %res = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @good_oracle)
+ ret b128 %res
}
-define <4 x i32> @test_oracle_too_many_args(ptr %ptr, i64 %n) {
+define b128 @test_oracle_too_many_args(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function argument count mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
- ret <4 x i32> %res
+; CHECK-NEXT: call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+ %res = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @good_oracle, ptr %ptr, i64 %n, i64 %n)
+ ret b128 %res
}
-define <4 x i32> @test_oracle_arg_type_mismatch(ptr %ptr) {
+define b128 @test_oracle_arg_type_mismatch(ptr %ptr) {
; CHECK: llvm.speculative.load oracle function argument type mismatch
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @oracle_i32_param, i64 42)
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @oracle_i32_param, i64 42)
- ret <4 x i32> %res
+; CHECK-NEXT: call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @oracle_i32_param, i64 42)
+ %res = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @oracle_i32_param, i64 42)
+ ret b128 %res
}
-define <4 x i32> @test_non_function_oracle(ptr %ptr, ptr %not_fn) {
+define b128 @test_non_function_oracle(ptr %ptr, ptr %not_fn) {
; CHECK: llvm.speculative.load third argument must be i64 or a direct reference to an oracle function
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr %not_fn)
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr %not_fn)
- ret <4 x i32> %res
+; CHECK-NEXT: call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr %not_fn)
+ %res = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr %not_fn)
+ ret b128 %res
}
-define <4 x i32> @test_oracle_side_effects(ptr %ptr, i64 %n) {
+define b128 @test_oracle_side_effects(ptr %ptr, i64 %n) {
; CHECK: llvm.speculative.load oracle function must not have side effects and may only read memory through its arguments
-; CHECK-NEXT: call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
- ret <4 x i32> %res
+; CHECK-NEXT: call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+ %res = call b128 (ptr, i1, ...) @llvm.speculative.load.b128.p0(ptr %ptr, i1 false, ptr @side_effecting_oracle, ptr %ptr, i64 %n)
+ ret b128 %res
}
-define <4 x i32> @test_num_accessible_bytes_exceeds_size(ptr %ptr) {
- %res = call <4 x i32> (ptr, i1, ...) @llvm.speculative.load.v4i32.p0(ptr %ptr, i1 false, i64 32)
- ret <4 x i32> %res
+define b24 @test_byte_size_not_pow2(ptr %ptr) {
+; CHECK: llvm.speculative.load return type size in bytes must be a positive power of 2
+; CHECK-NEXT: %res = call b24 (ptr, i1, ...) @llvm.speculative.load.b24.p0(ptr %ptr, i1 false, i64 0)
+ %res = call b24 (ptr, i1, ...) @llvm.speculative.load.b24.p0(ptr %ptr, i1 false, i64 0)
+ ret b24 %res
+}
+
+define <3 x i32> @test_vector_byte_size_not_pow2(ptr %ptr) {
+; CHECK: llvm.speculative.load return type size in bytes must be a positive power of 2
+; CHECK-NEXT: %res = call <3 x i32> (ptr, i1, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i1 false, i64 0)
+ %res = call <3 x i32> (ptr, i1, ...) @llvm.speculative.load.v3i32.p0(ptr %ptr, i1 false, i64 0)
+ ret <3 x i32> %res
}
>From 4a1e10a4b4d5c1f76d64be147c90019504635b04 Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo at fhahn.com>
Date: Wed, 27 May 2026 10:50:42 +0100
Subject: [PATCH 14/14] !fixup adjust wording for can_load_speculatively.
---
llvm/docs/LangRef.rst | 10 ++++------
1 file changed, 4 insertions(+), 6 deletions(-)
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index df2257f772351..0bcb7cf86ac10 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -28449,12 +28449,10 @@ Semantics:
""""""""""
This intrinsic has **target-dependent** semantics. It returns ``true`` if
-``num_bytes`` bytes starting at ``ptr + I * num_bytes``, for all non-negative
-integers ``I`` where the computed address does not wrap around the address
-space, can be loaded speculatively, even if the memory is beyond the bounds of
-an allocated object. It returns ``false`` otherwise. The first or last byte of
-each access must be part of the same underlying object as ``ptr`` in order to
-speculatively load the whole range.
+``num_bytes`` bytes starting at ``ptr + I * num_bytes`` can be loaded
+speculatively, for all non-negative integers ``I`` where the computed address
+does not wrap around the address space and its first or last byte is part of
+the same underlying object as ``ptr``.
The specific conditions under which this intrinsic returns ``true`` are
determined by the target. For example, a target may check whether the pointer
More information about the llvm-commits
mailing list