[Mlir-commits] [mlir] 586b892 - [mlir][VectorToLLVM] add opt-in `enable-gep-inbounds-nuw` pass flag for `vector.load/store` (#202118)
llvmlistbot at llvm.org
llvmlistbot at llvm.org
Thu Jun 18 06:49:49 PDT 2026
Author: Federico Bruzzone
Date: 2026-06-18T13:49:43Z
New Revision: 586b892672b478d0aa739e49eb3a5ef7f960530f
URL: https://github.com/llvm/llvm-project/commit/586b892672b478d0aa739e49eb3a5ef7f960530f
DIFF: https://github.com/llvm/llvm-project/commit/586b892672b478d0aa739e49eb3a5ef7f960530f.diff
LOG: [mlir][VectorToLLVM] add opt-in `enable-gep-inbounds-nuw` pass flag for `vector.load/store` (#202118)
> This patch follows up on #201180 and the refactoring #202766 (which
made `affine-super-vectorize` emit `in_bounds = [true]` on
`vector.transfer_read`/`write` when accesses are statically provable to
be within bounds). With that in place, the `VectorToLLVM` lowering was
still emitting `llvm.getelementptr` without `inbounds`/`nuw`, so LLVM
could not exploit the no-wrap guarantee: SCEV could not prove the index
arithmetic monotone (loop vectorizer bailed out) and BasicAliasAnalysis
fell back to conservative aliasing.
Without `inbounds`/`nuw` on the GEP that `vector.load`/`vector.store`
lower to, LLVM cannot exploit no-wrap guarantees: SCEV fails to prove
loop-index monotonicity (loop vectorizer bails), and BasicAliasAnalysis
falls back to conservative aliasing.
### Why opt-in
Unlike `memref.load`, `vector.load`/`vector.store` intentionally allow
out-of-bounds indices on several targets (SPIR-V robust buffer access,
AMDGPU buffer fat pointers read zeros / drop writes). Emitting
`inbounds` unconditionally would silently miscompile those targets. A
pass flag lets a caller assert the preconditions hold for a given
pipeline.
### What this patch does
Adds `enable-gep-inbounds-nuw` (default: off) to the
`ConvertVectorToLLVM` pass.
When set:
- `inbounds` (which implies `nusw`, hence `nsw` on intermediate
arithmetic) is always emitted.
- `nuw` is emitted only when all memref strides are statically
non-negative. `getStridedElementPtr` propagates `GEPNoWrapFlags::nuw` to
every intermediate `llvm.mul` and `llvm.add`; with a negative stride
(represented as 2^64−1 unsigned), `mul nuw idx, (2^64−1)` wraps unsigned
→ poison, even for an in-bounds index. Gating on non-negative strides
prevents this.
Masked variants (`vector.maskedload`/`vector.maskedstore`) are excluded:
they are designed for near-boundary access and must never receive these
flags.
An `assert` is added in the lowering to make the unit-trailing-stride
invariant explicit (the verifier already guarantees it).
**Thanks to @banach-space, @krzysz00 and @fabianmcg for the detailed
semantics discussion that shaped this design.**
Depends on: #201180 and #202766
AI Disclaimer: I used AI for the tests.
---------
Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Added:
mlir/test/Conversion/VectorToLLVM/vector-load-store-to-llvm.mlir
Modified:
mlir/include/mlir/Conversion/Passes.td
mlir/include/mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h
mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp
mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir
mlir/test/Dialect/Affine/SuperVectorize/vectorize_2d_inbounds.mlir
Removed:
################################################################################
diff --git a/mlir/include/mlir/Conversion/Passes.td b/mlir/include/mlir/Conversion/Passes.td
index 4f8a66957d954..ae93769a66762 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -1628,6 +1628,12 @@ def ConvertVectorToLLVMPass : Pass<"convert-vector-to-llvm"> {
"vector access are naturally aligned. If operations have an "
"alignment attribute set, the alignment attribute takes priority "
"over this option ">,
+ Option<"enableGEPInboundsNuw", "enable-gep-inbounds-nuw",
+ "bool", /*default=*/"false",
+ "Emit inbounds|nuw flags on GEPs generated by vector.load/store "
+ "lowering. Assumes 0 <= idx < dim_size and non-negative strides for "
+ "all dimensions. Setting this when those conditions do not hold is "
+ "undefined behavior under the LLVM lowering.">,
Option<"armNeon", "enable-arm-neon",
"bool", /*default=*/"false",
"Enables the use of ArmNeon dialect while lowering the vector "
diff --git a/mlir/include/mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h b/mlir/include/mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h
index cfb6cc313bc63..021a644ddd8a6 100644
--- a/mlir/include/mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h
+++ b/mlir/include/mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h
@@ -17,7 +17,7 @@ class LLVMTypeConverter;
void populateVectorToLLVMConversionPatterns(
const LLVMTypeConverter &converter, RewritePatternSet &patterns,
bool reassociateFPReductions = false, bool force32BitVectorIndices = false,
- bool useVectorAlignment = false);
+ bool useVectorAlignment = false, bool enableGEPInboundsNuw = false);
namespace vector {
void registerConvertVectorToLLVMInterface(DialectRegistry ®istry);
diff --git a/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp b/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
index 43e0824fef6cd..8e9d37648841a 100644
--- a/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
+++ b/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
@@ -223,16 +223,27 @@ static void replaceLoadOrStoreOp(vector::MaskedStoreOp storeOp,
storeOp, adaptor.getValueToStore(), ptr, adaptor.getMask(), align);
}
+/// Returns true if all strides of `memRefTy` are static and non-negative. A
+/// negative (or dynamic, hence unknown-sign) stride would make `mul nuw` on the
+/// index arithmetic wrap, so `nuw` must not be emitted in that case.
+static bool hasNonNegativeStrides(MemRefType memRefTy) {
+ auto [strides, offset] = memRefTy.getStridesAndOffset();
+ return llvm::all_of(strides, [](int64_t stride) {
+ return !ShapedType::isDynamic(stride) && stride >= 0;
+ });
+}
+
/// Conversion pattern for a vector.load, vector.store, vector.maskedload, and
/// vector.maskedstore.
template <class LoadOrStoreOp>
class VectorLoadStoreConversion : public ConvertOpToLLVMPattern<LoadOrStoreOp> {
public:
explicit VectorLoadStoreConversion(const LLVMTypeConverter &typeConv,
- bool useVectorAlign)
+ bool useVectorAlign,
+ bool enableGEPInboundsNuw)
: ConvertOpToLLVMPattern<LoadOrStoreOp>(typeConv),
- useVectorAlignment(useVectorAlign) {}
- using ConvertOpToLLVMPattern<LoadOrStoreOp>::ConvertOpToLLVMPattern;
+ useVectorAlignment(useVectorAlign),
+ enableGEPInboundsNuw(enableGEPInboundsNuw) {}
LogicalResult
matchAndRewrite(LoadOrStoreOp loadOrStoreOp,
@@ -256,10 +267,33 @@ class VectorLoadStoreConversion : public ConvertOpToLLVMPattern<LoadOrStoreOp> {
"could not resolve alignment");
// Resolve address.
+ // When --enable-gep-inbounds-nuw is set, emit inbounds|nuw on the GEP so
+ // LLVM can apply no-wrap optimizations on the index arithmetic. This
+ // assumes 0 <= idx < dim_size and non-negative strides; the caller is
+ // responsible for ensuring those conditions hold. Masked variants are
+ // designed for near-boundary access and never receive these flags.
+ LLVM::GEPNoWrapFlags noWrapFlags = LLVM::GEPNoWrapFlags::none;
+ if constexpr (std::is_same_v<LoadOrStoreOp, vector::LoadOp> ||
+ std::is_same_v<LoadOrStoreOp, vector::StoreOp>) {
+ // The verifier (verifyLoadStoreMemRefLayout) guarantees that the
+ // trailing (most minor) stride of the memref is 1. Assert to make
+ // the invariant explicit in the lowering code.
+ auto [strides, offset] = memRefTy.getStridesAndOffset();
+ assert((strides.empty() || strides.back() == 1) &&
+ "vector.load/store requires unit trailing memref stride");
+ if (enableGEPInboundsNuw) {
+ noWrapFlags = noWrapFlags | LLVM::GEPNoWrapFlags::inbounds;
+ // `nuw` additionally requires non-negative strides; skip it when the
+ // memref has dynamic or negative strides to avoid emitting poison.
+ if (hasNonNegativeStrides(memRefTy))
+ noWrapFlags = noWrapFlags | LLVM::GEPNoWrapFlags::nuw;
+ }
+ }
auto vtype = cast<VectorType>(
this->typeConverter->convertType(loadOrStoreOp.getVectorType()));
- Value dataPtr = this->getStridedElementPtr(
- rewriter, loc, memRefTy, adaptor.getBase(), adaptor.getIndices());
+ Value dataPtr =
+ this->getStridedElementPtr(rewriter, loc, memRefTy, adaptor.getBase(),
+ adaptor.getIndices(), noWrapFlags);
replaceLoadOrStoreOp(loadOrStoreOp, adaptor, vtype, dataPtr, align,
rewriter);
return success();
@@ -271,6 +305,7 @@ class VectorLoadStoreConversion : public ConvertOpToLLVMPattern<LoadOrStoreOp> {
// of the memref. This flag is intended for use with hardware
// backends that require alignment of vector operations.
const bool useVectorAlignment;
+ const bool enableGEPInboundsNuw;
};
/// Conversion pattern for a vector.gather.
@@ -2218,7 +2253,7 @@ void mlir::vector::populateVectorTransposeToFlatTranspose(
void mlir::populateVectorToLLVMConversionPatterns(
const LLVMTypeConverter &converter, RewritePatternSet &patterns,
bool reassociateFPReductions, bool force32BitVectorIndices,
- bool useVectorAlignment) {
+ bool useVectorAlignment, bool enableGEPInboundsNuw) {
// This function populates only ConversionPatterns, not RewritePatterns.
MLIRContext *ctx = converter.getDialect()->getContext();
patterns.add<VectorReductionOpConversion>(converter, reassociateFPReductions);
@@ -2226,8 +2261,9 @@ void mlir::populateVectorToLLVMConversionPatterns(
patterns.add<VectorLoadStoreConversion<vector::LoadOp>,
VectorLoadStoreConversion<vector::MaskedLoadOp>,
VectorLoadStoreConversion<vector::StoreOp>,
- VectorLoadStoreConversion<vector::MaskedStoreOp>,
- VectorGatherOpConversion, VectorScatterOpConversion>(
+ VectorLoadStoreConversion<vector::MaskedStoreOp>>(
+ converter, useVectorAlignment, enableGEPInboundsNuw);
+ patterns.add<VectorGatherOpConversion, VectorScatterOpConversion>(
converter, useVectorAlignment);
patterns.add<VectorBitCastOpConversion, VectorShuffleOpConversion,
VectorExtractOpConversion, VectorFMAOp1DConversion,
diff --git a/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp b/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp
index 4cc5704353382..2b358d312dcfe 100644
--- a/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp
+++ b/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp
@@ -114,7 +114,7 @@ void ConvertVectorToLLVMPass::runOnOperation() {
populateVectorTransferLoweringPatterns(patterns);
populateVectorToLLVMConversionPatterns(
converter, patterns, reassociateFPReductions, force32BitVectorIndices,
- useVectorAlignment);
+ useVectorAlignment, enableGEPInboundsNuw);
// Architecture specific augmentations.
LLVMConversionTarget target(getContext());
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-load-store-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-load-store-to-llvm.mlir
new file mode 100644
index 0000000000000..cab462efee2b5
--- /dev/null
+++ b/mlir/test/Conversion/VectorToLLVM/vector-load-store-to-llvm.mlir
@@ -0,0 +1,238 @@
+// RUN: mlir-opt %s -convert-vector-to-llvm -split-input-file | FileCheck %s --check-prefixes=ALL,DEFAULT
+// RUN: mlir-opt %s -convert-vector-to-llvm='enable-gep-inbounds-nuw=1' -split-input-file | FileCheck %s --check-prefixes=ALL,INBOUNDS
+
+//===----------------------------------------------------------------------===//
+// vector.load
+//===----------------------------------------------------------------------===//
+
+func.func @load(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<8xf32> {
+ %0 = vector.load %memref[%i, %j] : memref<200x100xf32>, vector<8xf32>
+ return %0 : vector<8xf32>
+}
+
+// ALL-LABEL: func @load
+// ALL: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
+// ALL: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]]
+// ALL: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}}
+// DEFAULT: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// INBOUNDS: %[[GEP:.*]] = llvm.getelementptr inbounds|nuw %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// ALL: llvm.load %[[GEP]] {alignment = 4 : i64} : !llvm.ptr -> vector<8xf32>
+
+// -----
+
+func.func @load_scalable(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<[8]xf32> {
+ %0 = vector.load %memref[%i, %j] : memref<200x100xf32>, vector<[8]xf32>
+ return %0 : vector<[8]xf32>
+}
+
+// ALL-LABEL: func @load_scalable
+// ALL: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
+// ALL: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]]
+// ALL: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}}
+// DEFAULT: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// INBOUNDS: %[[GEP:.*]] = llvm.getelementptr inbounds|nuw %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// ALL: llvm.load %[[GEP]] {alignment = 4 : i64} : !llvm.ptr -> vector<[8]xf32>
+
+// -----
+
+func.func @load_nontemporal(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<8xf32> {
+ %0 = vector.load %memref[%i, %j] {nontemporal = true} : memref<200x100xf32>, vector<8xf32>
+ return %0 : vector<8xf32>
+}
+
+// ALL-LABEL: func @load_nontemporal
+// ALL: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
+// ALL: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]]
+// ALL: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}}
+// DEFAULT: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// INBOUNDS: %[[GEP:.*]] = llvm.getelementptr inbounds|nuw %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// ALL: llvm.load %[[GEP]] {alignment = 4 : i64, nontemporal} : !llvm.ptr -> vector<8xf32>
+
+// -----
+
+func.func @load_nontemporal_scalable(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<[8]xf32> {
+ %0 = vector.load %memref[%i, %j] {nontemporal = true} : memref<200x100xf32>, vector<[8]xf32>
+ return %0 : vector<[8]xf32>
+}
+
+// ALL-LABEL: func @load_nontemporal_scalable
+// ALL: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
+// ALL: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]]
+// ALL: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}}
+// DEFAULT: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// INBOUNDS: %[[GEP:.*]] = llvm.getelementptr inbounds|nuw %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// ALL: llvm.load %[[GEP]] {alignment = 4 : i64, nontemporal} : !llvm.ptr -> vector<[8]xf32>
+
+// -----
+
+func.func @load_index(%memref : memref<200x100xindex>, %i : index, %j : index) -> vector<8xindex> {
+ %0 = vector.load %memref[%i, %j] : memref<200x100xindex>, vector<8xindex>
+ return %0 : vector<8xindex>
+}
+// ALL-LABEL: func @load_index
+// ALL: %[[T0:.*]] = llvm.load %{{.*}} {alignment = 8 : i64} : !llvm.ptr -> vector<8xi64>
+// ALL: %[[T1:.*]] = builtin.unrealized_conversion_cast %[[T0]] : vector<8xi64> to vector<8xindex>
+// ALL: return %[[T1]] : vector<8xindex>
+
+// -----
+
+func.func @load_index_scalable(%memref : memref<200x100xindex>, %i : index, %j : index) -> vector<[8]xindex> {
+ %0 = vector.load %memref[%i, %j] : memref<200x100xindex>, vector<[8]xindex>
+ return %0 : vector<[8]xindex>
+}
+// ALL-LABEL: func @load_index_scalable
+// ALL: %[[T0:.*]] = llvm.load %{{.*}} {alignment = 8 : i64} : !llvm.ptr -> vector<[8]xi64>
+// ALL: %[[T1:.*]] = builtin.unrealized_conversion_cast %[[T0]] : vector<[8]xi64> to vector<[8]xindex>
+// ALL: return %[[T1]] : vector<[8]xindex>
+
+// -----
+
+func.func @load_0d(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<f32> {
+ %0 = vector.load %memref[%i, %j] : memref<200x100xf32>, vector<f32>
+ return %0 : vector<f32>
+}
+
+// ALL-LABEL: func @load_0d
+// ALL: %[[J:.*]] = builtin.unrealized_conversion_cast %{{.*}} : index to i64
+// ALL: %[[I:.*]] = builtin.unrealized_conversion_cast %{{.*}} : index to i64
+// ALL: %[[CAST_MEMREF:.*]] = builtin.unrealized_conversion_cast %{{.*}} : memref<200x100xf32> to !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)>
+// ALL: %[[REF:.*]] = llvm.extractvalue %[[CAST_MEMREF]][1] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)>
+// ALL: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
+// ALL: %[[MUL:.*]] = llvm.mul %[[I]], %[[C100]]
+// ALL: %[[ADD:.*]] = llvm.add %[[MUL]], %[[J]]
+// DEFAULT: %[[ADDR:.*]] = llvm.getelementptr %[[REF]][%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// INBOUNDS: %[[ADDR:.*]] = llvm.getelementptr inbounds|nuw %[[REF]][%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// ALL: %[[LOAD:.*]] = llvm.load %[[ADDR]] {alignment = 4 : i64} : !llvm.ptr -> vector<1xf32>
+// ALL: %[[RES:.*]] = builtin.unrealized_conversion_cast %[[LOAD]] : vector<1xf32> to vector<f32>
+// ALL: return %[[RES]] : vector<f32>
+
+// -----
+
+func.func @load_with_alignment(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<8xf32> {
+ %0 = vector.load %memref[%i, %j] { alignment = 8 } : memref<200x100xf32>, vector<8xf32>
+ return %0 : vector<8xf32>
+}
+
+// ALL-LABEL: func @load_with_alignment
+// ALL: llvm.load {{.*}} {alignment = 8 : i64} : !llvm.ptr -> vector<8xf32>
+
+// -----
+
+//===----------------------------------------------------------------------===//
+// vector.store
+//===----------------------------------------------------------------------===//
+
+func.func @store(%memref : memref<200x100xf32>, %i : index, %j : index) {
+ %val = arith.constant dense<11.0> : vector<4xf32>
+ vector.store %val, %memref[%i, %j] : memref<200x100xf32>, vector<4xf32>
+ return
+}
+
+// ALL-LABEL: func @store
+// ALL: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
+// ALL: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]]
+// ALL: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}}
+// DEFAULT: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// INBOUNDS: %[[GEP:.*]] = llvm.getelementptr inbounds|nuw %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// ALL: llvm.store %{{.*}}, %[[GEP]] {alignment = 4 : i64} : vector<4xf32>, !llvm.ptr
+
+// -----
+
+func.func @store_scalable(%memref : memref<200x100xf32>, %i : index, %j : index) {
+ %val = arith.constant dense<11.0> : vector<[4]xf32>
+ vector.store %val, %memref[%i, %j] : memref<200x100xf32>, vector<[4]xf32>
+ return
+}
+
+// ALL-LABEL: func @store_scalable
+// ALL: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
+// ALL: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]]
+// ALL: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}}
+// DEFAULT: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// INBOUNDS: %[[GEP:.*]] = llvm.getelementptr inbounds|nuw %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// ALL: llvm.store %{{.*}}, %[[GEP]] {alignment = 4 : i64} : vector<[4]xf32>, !llvm.ptr
+
+// -----
+
+func.func @store_nontemporal(%memref : memref<200x100xf32>, %i : index, %j : index) {
+ %val = arith.constant dense<11.0> : vector<4xf32>
+ vector.store %val, %memref[%i, %j] {nontemporal = true} : memref<200x100xf32>, vector<4xf32>
+ return
+}
+
+// ALL-LABEL: func @store_nontemporal
+// ALL: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
+// ALL: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]]
+// ALL: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}}
+// DEFAULT: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// INBOUNDS: %[[GEP:.*]] = llvm.getelementptr inbounds|nuw %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// ALL: llvm.store %{{.*}}, %[[GEP]] {alignment = 4 : i64, nontemporal} : vector<4xf32>, !llvm.ptr
+
+// -----
+
+func.func @store_nontemporal_scalable(%memref : memref<200x100xf32>, %i : index, %j : index) {
+ %val = arith.constant dense<11.0> : vector<[4]xf32>
+ vector.store %val, %memref[%i, %j] {nontemporal = true} : memref<200x100xf32>, vector<[4]xf32>
+ return
+}
+
+// ALL-LABEL: func @store_nontemporal_scalable
+// ALL: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
+// ALL: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]]
+// ALL: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}}
+// DEFAULT: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// INBOUNDS: %[[GEP:.*]] = llvm.getelementptr inbounds|nuw %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// ALL: llvm.store %{{.*}}, %[[GEP]] {alignment = 4 : i64, nontemporal} : vector<[4]xf32>, !llvm.ptr
+
+// -----
+
+func.func @store_index(%memref : memref<200x100xindex>, %i : index, %j : index) {
+ %val = arith.constant dense<11> : vector<4xindex>
+ vector.store %val, %memref[%i, %j] : memref<200x100xindex>, vector<4xindex>
+ return
+}
+// ALL-LABEL: func @store_index
+// ALL: llvm.store %{{.*}}, %{{.*}} {alignment = 8 : i64} : vector<4xi64>, !llvm.ptr
+
+// -----
+
+func.func @store_index_scalable(%memref : memref<200x100xindex>, %i : index, %j : index) {
+ %val = arith.constant dense<11> : vector<[4]xindex>
+ vector.store %val, %memref[%i, %j] : memref<200x100xindex>, vector<[4]xindex>
+ return
+}
+// ALL-LABEL: func @store_index_scalable
+// ALL: llvm.store %{{.*}}, %{{.*}} {alignment = 8 : i64} : vector<[4]xi64>, !llvm.ptr
+
+// -----
+
+func.func @store_0d(%memref : memref<200x100xf32>, %i : index, %j : index) {
+ %val = arith.constant dense<11.0> : vector<f32>
+ vector.store %val, %memref[%i, %j] : memref<200x100xf32>, vector<f32>
+ return
+}
+
+// ALL-LABEL: func @store_0d
+// ALL: %[[J:.*]] = builtin.unrealized_conversion_cast %{{.*}} : index to i64
+// ALL: %[[I:.*]] = builtin.unrealized_conversion_cast %{{.*}} : index to i64
+// ALL: %[[CAST_MEMREF:.*]] = builtin.unrealized_conversion_cast %{{.*}} : memref<200x100xf32> to !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)>
+// ALL: %[[CST:.*]] = arith.constant dense<1.100000e+01> : vector<f32>
+// ALL: %[[VAL:.*]] = builtin.unrealized_conversion_cast %[[CST]] : vector<f32> to vector<1xf32>
+// ALL: %[[REF:.*]] = llvm.extractvalue %[[CAST_MEMREF]][1] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)>
+// ALL: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
+// ALL: %[[MUL:.*]] = llvm.mul %[[I]], %[[C100]]
+// ALL: %[[ADD:.*]] = llvm.add %[[MUL]], %[[J]]
+// DEFAULT: %[[ADDR:.*]] = llvm.getelementptr %[[REF]][%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// INBOUNDS: %[[ADDR:.*]] = llvm.getelementptr inbounds|nuw %[[REF]][%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// ALL: llvm.store %[[VAL]], %[[ADDR]] {alignment = 4 : i64} : vector<1xf32>, !llvm.ptr
+// ALL: return
+
+// -----
+
+func.func @store_with_alignment(%memref : memref<200x100xf32>, %i : index, %j : index, %val : vector<4xf32>) {
+ vector.store %val, %memref[%i, %j] {alignment = 8} : memref<200x100xf32>, vector<4xf32>
+ return
+}
+
+// ALL-LABEL: func @store_with_alignment
+// ALL: llvm.store %{{.*}} {alignment = 8 : i64} : vector<4xf32>, !llvm.ptr
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir b/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir
index d570d46e11b4a..e82f75d93066a 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir
@@ -1577,234 +1577,6 @@ func.func @transpose_0d(%arg0: vector<f32>) -> vector<f32> {
// -----
-//===----------------------------------------------------------------------===//
-// vector.load
-//===----------------------------------------------------------------------===//
-
-func.func @load(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<8xf32> {
- %0 = vector.load %memref[%i, %j] : memref<200x100xf32>, vector<8xf32>
- return %0 : vector<8xf32>
-}
-
-// CHECK-LABEL: func @load
-// CHECK: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
-// CHECK: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]] : i64
-// CHECK: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}} : i64
-// CHECK: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
-// CHECK: llvm.load %[[GEP]] {alignment = 4 : i64} : !llvm.ptr -> vector<8xf32>
-
-// -----
-
-func.func @load_scalable(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<[8]xf32> {
- %0 = vector.load %memref[%i, %j] : memref<200x100xf32>, vector<[8]xf32>
- return %0 : vector<[8]xf32>
-}
-
-// CHECK-LABEL: func @load_scalable
-// CHECK: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
-// CHECK: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]] : i64
-// CHECK: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}} : i64
-// CHECK: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
-// CHECK: llvm.load %[[GEP]] {alignment = 4 : i64} : !llvm.ptr -> vector<[8]xf32>
-
-// -----
-
-func.func @load_nontemporal(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<8xf32> {
- %0 = vector.load %memref[%i, %j] {nontemporal = true} : memref<200x100xf32>, vector<8xf32>
- return %0 : vector<8xf32>
-}
-
-// CHECK-LABEL: func @load_nontemporal
-// CHECK: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
-// CHECK: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]] : i64
-// CHECK: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}} : i64
-// CHECK: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
-// CHECK: llvm.load %[[GEP]] {alignment = 4 : i64, nontemporal} : !llvm.ptr -> vector<8xf32>
-
-// -----
-
-func.func @load_nontemporal_scalable(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<[8]xf32> {
- %0 = vector.load %memref[%i, %j] {nontemporal = true} : memref<200x100xf32>, vector<[8]xf32>
- return %0 : vector<[8]xf32>
-}
-
-// CHECK-LABEL: func @load_nontemporal_scalable
-// CHECK: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
-// CHECK: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]] : i64
-// CHECK: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}} : i64
-// CHECK: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
-// CHECK: llvm.load %[[GEP]] {alignment = 4 : i64, nontemporal} : !llvm.ptr -> vector<[8]xf32>
-
-// -----
-
-func.func @load_index(%memref : memref<200x100xindex>, %i : index, %j : index) -> vector<8xindex> {
- %0 = vector.load %memref[%i, %j] : memref<200x100xindex>, vector<8xindex>
- return %0 : vector<8xindex>
-}
-// CHECK-LABEL: func @load_index
-// CHECK: %[[T0:.*]] = llvm.load %{{.*}} {alignment = 8 : i64} : !llvm.ptr -> vector<8xi64>
-// CHECK: %[[T1:.*]] = builtin.unrealized_conversion_cast %[[T0]] : vector<8xi64> to vector<8xindex>
-// CHECK: return %[[T1]] : vector<8xindex>
-
-// -----
-
-func.func @load_index_scalable(%memref : memref<200x100xindex>, %i : index, %j : index) -> vector<[8]xindex> {
- %0 = vector.load %memref[%i, %j] : memref<200x100xindex>, vector<[8]xindex>
- return %0 : vector<[8]xindex>
-}
-// CHECK-LABEL: func @load_index_scalable
-// CHECK: %[[T0:.*]] = llvm.load %{{.*}} {alignment = 8 : i64} : !llvm.ptr -> vector<[8]xi64>
-// CHECK: %[[T1:.*]] = builtin.unrealized_conversion_cast %[[T0]] : vector<[8]xi64> to vector<[8]xindex>
-// CHECK: return %[[T1]] : vector<[8]xindex>
-
-// -----
-
-func.func @load_0d(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<f32> {
- %0 = vector.load %memref[%i, %j] : memref<200x100xf32>, vector<f32>
- return %0 : vector<f32>
-}
-
-// CHECK-LABEL: func @load_0d
-// CHECK: %[[J:.*]] = builtin.unrealized_conversion_cast %{{.*}} : index to i64
-// CHECK: %[[I:.*]] = builtin.unrealized_conversion_cast %{{.*}} : index to i64
-// CHECK: %[[CAST_MEMREF:.*]] = builtin.unrealized_conversion_cast %{{.*}} : memref<200x100xf32> to !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)>
-// CHECK: %[[REF:.*]] = llvm.extractvalue %[[CAST_MEMREF]][1] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)>
-// CHECK: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
-// CHECK: %[[MUL:.*]] = llvm.mul %[[I]], %[[C100]] : i64
-// CHECK: %[[ADD:.*]] = llvm.add %[[MUL]], %[[J]] : i64
-// CHECK: %[[ADDR:.*]] = llvm.getelementptr %[[REF]][%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
-// CHECK: %[[LOAD:.*]] = llvm.load %[[ADDR]] {alignment = 4 : i64} : !llvm.ptr -> vector<1xf32>
-// CHECK: %[[RES:.*]] = builtin.unrealized_conversion_cast %[[LOAD]] : vector<1xf32> to vector<f32>
-// CHECK: return %[[RES]] : vector<f32>
-
-// -----
-
-func.func @load_with_alignment(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<8xf32> {
- %0 = vector.load %memref[%i, %j] { alignment = 8 } : memref<200x100xf32>, vector<8xf32>
- return %0 : vector<8xf32>
-}
-
-// CHECK-LABEL: func @load_with_alignment
-// CHECK: llvm.load {{.*}} {alignment = 8 : i64} : !llvm.ptr -> vector<8xf32>
-
-// -----
-
-//===----------------------------------------------------------------------===//
-// vector.store
-//===----------------------------------------------------------------------===//
-
-func.func @store(%memref : memref<200x100xf32>, %i : index, %j : index) {
- %val = arith.constant dense<11.0> : vector<4xf32>
- vector.store %val, %memref[%i, %j] : memref<200x100xf32>, vector<4xf32>
- return
-}
-
-// CHECK-LABEL: func @store
-// CHECK: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
-// CHECK: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]] : i64
-// CHECK: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}} : i64
-// CHECK: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
-// CHECK: llvm.store %{{.*}}, %[[GEP]] {alignment = 4 : i64} : vector<4xf32>, !llvm.ptr
-
-// -----
-
-func.func @store_scalable(%memref : memref<200x100xf32>, %i : index, %j : index) {
- %val = arith.constant dense<11.0> : vector<[4]xf32>
- vector.store %val, %memref[%i, %j] : memref<200x100xf32>, vector<[4]xf32>
- return
-}
-
-// CHECK-LABEL: func @store_scalable
-// CHECK: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
-// CHECK: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]] : i64
-// CHECK: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}} : i64
-// CHECK: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
-// CHECK: llvm.store %{{.*}}, %[[GEP]] {alignment = 4 : i64} : vector<[4]xf32>, !llvm.ptr
-
-// -----
-
-func.func @store_nontemporal(%memref : memref<200x100xf32>, %i : index, %j : index) {
- %val = arith.constant dense<11.0> : vector<4xf32>
- vector.store %val, %memref[%i, %j] {nontemporal = true} : memref<200x100xf32>, vector<4xf32>
- return
-}
-
-// CHECK-LABEL: func @store_nontemporal
-// CHECK: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
-// CHECK: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]] : i64
-// CHECK: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}} : i64
-// CHECK: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
-// CHECK: llvm.store %{{.*}}, %[[GEP]] {alignment = 4 : i64, nontemporal} : vector<4xf32>, !llvm.ptr
-
-// -----
-
-func.func @store_nontemporal_scalable(%memref : memref<200x100xf32>, %i : index, %j : index) {
- %val = arith.constant dense<11.0> : vector<[4]xf32>
- vector.store %val, %memref[%i, %j] {nontemporal = true} : memref<200x100xf32>, vector<[4]xf32>
- return
-}
-
-// CHECK-LABEL: func @store_nontemporal_scalable
-// CHECK: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
-// CHECK: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]] : i64
-// CHECK: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}} : i64
-// CHECK: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
-// CHECK: llvm.store %{{.*}}, %[[GEP]] {alignment = 4 : i64, nontemporal} : vector<[4]xf32>, !llvm.ptr
-
-// -----
-
-func.func @store_index(%memref : memref<200x100xindex>, %i : index, %j : index) {
- %val = arith.constant dense<11> : vector<4xindex>
- vector.store %val, %memref[%i, %j] : memref<200x100xindex>, vector<4xindex>
- return
-}
-// CHECK-LABEL: func @store_index
-// CHECK: llvm.store %{{.*}}, %{{.*}} {alignment = 8 : i64} : vector<4xi64>, !llvm.ptr
-
-// -----
-
-func.func @store_index_scalable(%memref : memref<200x100xindex>, %i : index, %j : index) {
- %val = arith.constant dense<11> : vector<[4]xindex>
- vector.store %val, %memref[%i, %j] : memref<200x100xindex>, vector<[4]xindex>
- return
-}
-// CHECK-LABEL: func @store_index_scalable
-// CHECK: llvm.store %{{.*}}, %{{.*}} {alignment = 8 : i64} : vector<[4]xi64>, !llvm.ptr
-
-// -----
-
-func.func @store_0d(%memref : memref<200x100xf32>, %i : index, %j : index) {
- %val = arith.constant dense<11.0> : vector<f32>
- vector.store %val, %memref[%i, %j] : memref<200x100xf32>, vector<f32>
- return
-}
-
-// CHECK-LABEL: func @store_0d
-// CHECK: %[[J:.*]] = builtin.unrealized_conversion_cast %{{.*}} : index to i64
-// CHECK: %[[I:.*]] = builtin.unrealized_conversion_cast %{{.*}} : index to i64
-// CHECK: %[[CAST_MEMREF:.*]] = builtin.unrealized_conversion_cast %{{.*}} : memref<200x100xf32> to !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)>
-// CHECK: %[[CST:.*]] = arith.constant dense<1.100000e+01> : vector<f32>
-// CHECK: %[[VAL:.*]] = builtin.unrealized_conversion_cast %[[CST]] : vector<f32> to vector<1xf32>
-// CHECK: %[[REF:.*]] = llvm.extractvalue %[[CAST_MEMREF]][1] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)>
-// CHECK: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i64
-// CHECK: %[[MUL:.*]] = llvm.mul %[[I]], %[[C100]] : i64
-// CHECK: %[[ADD:.*]] = llvm.add %[[MUL]], %[[J]] : i64
-// CHECK: %[[ADDR:.*]] = llvm.getelementptr %[[REF]][%[[ADD]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
-// CHECK: llvm.store %[[VAL]], %[[ADDR]] {alignment = 4 : i64} : vector<1xf32>, !llvm.ptr
-// CHECK: return
-
-// -----
-
-func.func @store_with_alignment(%memref : memref<200x100xf32>, %i : index, %j : index, %val : vector<4xf32>) {
- vector.store %val, %memref[%i, %j] {alignment = 8} : memref<200x100xf32>, vector<4xf32>
- return
-}
-
-// CHECK-LABEL: func @store_with_alignment
-// CHECK: llvm.store %{{.*}} {alignment = 8 : i64} : vector<4xf32>, !llvm.ptr
-
-// -----
-
//===----------------------------------------------------------------------===//
// vector.maskedload
//===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_2d_inbounds.mlir b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_2d_inbounds.mlir
index d95c5bdee79d7..49ba1a0c31ee1 100644
--- a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_2d_inbounds.mlir
+++ b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_2d_inbounds.mlir
@@ -8,7 +8,7 @@
// RUN: | FileCheck %s --check-prefix=MATMUL
// RUN: mlir-opt %s \
// RUN: --affine-super-vectorize="virtual-vector-size=4" \
-// RUN: --convert-vector-to-llvm \
+// RUN: --convert-vector-to-llvm='enable-gep-inbounds-nuw=1' \
// RUN: --finalize-memref-to-llvm \
// RUN: --convert-func-to-llvm \
// RUN: | FileCheck %s --check-prefix=LLVM
@@ -22,8 +22,11 @@
// CHECK-NOT: vector.transfer_write
// LLVM-LABEL: llvm.func @copy
-// Verify that in_bounds lowers to plain llvm.load/store, not masked intrinsics.
+// Verify that in_bounds lowers to plain llvm.load/store with inbounds|nuw GEP,
+// not masked intrinsics, enabling LLVM to apply no-wrap optimizations.
+// LLVM: llvm.getelementptr inbounds|nuw {{.*}} : (!llvm.ptr, i64) -> !llvm.ptr, f32
// LLVM: llvm.load {{.*}} : !llvm.ptr -> vector<4xf32>
+// LLVM: llvm.getelementptr inbounds|nuw {{.*}} : (!llvm.ptr, i64) -> !llvm.ptr, f32
// LLVM: llvm.store {{.*}} : vector<4xf32>, !llvm.ptr
// LLVM-NOT: llvm.intr.masked.load
// LLVM-NOT: llvm.intr.masked.store
More information about the Mlir-commits
mailing list