[Mlir-commits] [mlir] [mlir][memref] Promote buffers accessed via memref.copy (PR #232078)
Jianhui Li
llvmlistbot at llvm.org
Thu Oct 15 23:10:20 PDT 2026
https://github.com/Jianhui-Li updated https://github.com/llvm/llvm-project/pull/232078
>From 4b4692643edf8467c244f5008caea0a6252770ab Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Wed, 14 Oct 2026 19:08:27 +0000
Subject: [PATCH 1/2] [mlir][mem2reg] Promote buffers accessed via memref.copy
Add a `PromotableMemOpInterface` model for `memref.copy` so that buffers
accessed by copies can be promoted to a vector SSA value:
* A copy into the slot becomes a `vector.transfer_read` of the source.
* A copy out of the slot becomes a `vector.transfer_write` to the target.
* A copy between two slots is rewritten one slot at a time, in either
order.
For a dynamic-subview slot, the copy is restricted to the subview's extent:
copies into the slot are masked by
`PromotableAliaserInterface::projectAliasValueToSlotValue`, and copies out
of it are masked by this model via `vector.create_mask`.
Copies of scalable slots and self-copies are not modelled and prevent
promotion.
Also reorganise the Mem2Reg tests. Since this logic spans the MemRef and
Vector dialects, there is no obvious dialect sub-directory for the tests,
so they move to a new `mlir/test/Transforms/mem2reg/` directory:
Transforms/mem2reg.mlir -> mem2reg/core.mlir
Dialect/Vector/mem2reg-dynamic-subview.mlir -> mem2reg/dynamic-subview.mlir
Dialect/Vector/mem2reg.mlir (subview part) -> mem2reg/static-subview.mlir
`Dialect/Vector/mem2reg.mlir` keeps the cases that only exercise
`memref.alloca` plus vector transfer ops, which do have a natural home in
the Vector dialect tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply at anthropic.com>
---
.../Transforms/MemorySlotOpInterfaceImpl.cpp | 140 +++++-
.../Transforms/MemorySlotOpInterfaceImpl.cpp | 4 +-
mlir/test/Dialect/Vector/mem2reg.mlir | 468 ------------------
.../{mem2reg.mlir => mem2reg/core.mlir} | 0
.../mem2reg/dynamic-subview.mlir} | 0
mlir/test/Transforms/mem2reg/memref-copy.mlir | 173 +++++++
.../Transforms/mem2reg/static-subview.mlir | 467 +++++++++++++++++
7 files changed, 776 insertions(+), 476 deletions(-)
rename mlir/test/Transforms/{mem2reg.mlir => mem2reg/core.mlir} (100%)
rename mlir/test/{Dialect/Vector/mem2reg-dynamic-subview.mlir => Transforms/mem2reg/dynamic-subview.mlir} (100%)
create mode 100644 mlir/test/Transforms/mem2reg/memref-copy.mlir
create mode 100644 mlir/test/Transforms/mem2reg/static-subview.mlir
diff --git a/mlir/lib/Dialect/MemRef/Transforms/MemorySlotOpInterfaceImpl.cpp b/mlir/lib/Dialect/MemRef/Transforms/MemorySlotOpInterfaceImpl.cpp
index c562c67433f470..2c87125ef92253 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/MemorySlotOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/MemorySlotOpInterfaceImpl.cpp
@@ -7,10 +7,10 @@
//===----------------------------------------------------------------------===//
//
// This file implements memory slot interface models used by Mem2Reg to promote
-// a memref to a single vector SSA value. `PromotableAliaserInterface` exposes
-// `memref.subview` aliases, and `PromotableOpInterface` removes the subviews
-// after promotion. Vector transfer ops are handled by the models in the
-// Vector dialect.
+// a memref to a single vector SSA value. `PromotableMemOpInterface` handles
+// `memref.copy`. `PromotableAliaserInterface` exposes `memref.subview` aliases,
+// and `PromotableOpInterface` removes the subviews after promotion. Vector
+// transfer ops are handled by the models in the Vector dialect.
//
// A memory slot pairs a memref with the vector type used to represent its
// contents. Promotion replaces accesses to the slot with uses and definitions
@@ -19,6 +19,10 @@
//
// The accesses are rewritten as follows:
//
+// * A `memref.copy` into the slot becomes a `vector.transfer_read` of the
+// source. A copy out of the slot becomes a `vector.transfer_write` to the
+// target. A copy between two slots is rewritten one slot at a time.
+//
// * A same-rank, unit-stride `memref.subview` gets an alias slot that is
// promoted together with the parent:
//
@@ -31,9 +35,9 @@
// accesses to the subview's extent. The subview must start at the
// parent's origin.
//
-// Subviews require a statically shaped parent buffer. A dynamically
+// Copies and subviews require a statically shaped parent buffer. A dynamically
// shaped buffer can instead promote to a 1-D scalable vector through
-// whole-buffer transfers, but subviews are not supported.
+// whole-buffer transfers, but copies and subviews are not supported.
//
// Unsupported accesses, such as dynamic subview offsets, rank reduction,
// non-unit subview strides, or non-zero transfer indices, prevent promotion.
@@ -162,7 +166,130 @@ Value mlir::memref::buildDynamicSubViewMask(OpBuilder &builder, Location loc,
return buildSubViewMask(builder, loc, subView);
}
+/// Produces the value that a `memref.copy` stores into a slot: reads `mem` at
+/// the origin into `vecType` with an identity map, marking a dimension
+/// in-bounds only when the memref extent is statically at least the vector
+/// extent.
+///
+/// The padding value does not affect the copy: both operands must have the same
+/// runtime shape. For a whole-buffer slot, the source covers the whole vector.
+/// For a dynamic-subview slot,
+/// `PromotableAliaserInterface::projectAliasValueToSlotValue` discards lanes
+/// outside the subview's extent with a masked select.
+static Value readMemRefAsVector(OpBuilder &builder, Location loc, Value mem,
+ VectorType vecType) {
+ assert(!vecType.isScalable() && "expected a fixed-size vector");
+ auto memType = cast<MemRefType>(mem.getType());
+ int64_t rank = vecType.getRank();
+ Value zero = arith::ConstantIndexOp::create(builder, loc, 0);
+ SmallVector<Value> indices(rank, zero);
+ Value padding = arith::ConstantOp::create(
+ builder, loc, builder.getZeroAttr(vecType.getElementType()));
+ SmallVector<bool> inBounds(rank);
+ for (int64_t d = 0; d < rank; ++d)
+ inBounds[d] = !memType.isDynamicDim(d) &&
+ memType.getDimSize(d) >= vecType.getDimSize(d);
+ return vector::TransferReadOp::create(
+ builder, loc, vecType, mem, indices,
+ AffineMapAttr::get(builder.getMultiDimIdentityMap(rank)), padding,
+ /*mask=*/Value(), builder.getBoolArrayAttr(inBounds));
+}
+
+/// Stores a slot's value to the target of a `memref.copy`: writes `vec` into
+/// `mem` at the origin, with the identity map and in-bounds rule of
+/// `readMemRefAsVector`. `mask`, when set, restricts the write to the lanes it
+/// selects.
+static void writeVectorToMemRef(OpBuilder &builder, Location loc, Value vec,
+ Value mem, Value mask = {}) {
+ auto memType = cast<MemRefType>(mem.getType());
+ auto vecType = cast<VectorType>(vec.getType());
+ assert(!vecType.isScalable() && "expected a fixed-size vector");
+ int64_t rank = vecType.getRank();
+ Value zero = arith::ConstantIndexOp::create(builder, loc, 0);
+ SmallVector<Value> indices(rank, zero);
+ SmallVector<bool> inBounds(rank);
+ for (int64_t d = 0; d < rank; ++d)
+ inBounds[d] = !memType.isDynamicDim(d) &&
+ memType.getDimSize(d) >= vecType.getDimSize(d);
+ vector::TransferWriteOp::create(
+ builder, loc, vec, mem, indices,
+ AffineMapAttr::get(builder.getMultiDimIdentityMap(rank)), mask,
+ builder.getBoolArrayAttr(inBounds));
+}
+
+//===----------------------------------------------------------------------===//
+// PromotableMemOpInterface model for memref.copy
+//===----------------------------------------------------------------------===//
namespace {
+/// `PromotableMemOpInterface` model for `memref.copy`.
+///
+/// A copy into the slot becomes a `vector.transfer_read` of the source in
+/// `getStored`. A copy out of the slot becomes a `vector.transfer_write` of the
+/// reaching definition in `removeBlockingUses`.
+///
+/// A copy between two slots is rewritten one slot at a time, in either order.
+/// For dynamic subviews,
+/// `PromotableAliaserInterface::projectAliasValueToSlotValue` masks writes into
+/// the slot, and this model masks copies out of it.
+struct CopyOpMemOpModel
+ : public PromotableMemOpInterface::ExternalModel<CopyOpMemOpModel,
+ memref::CopyOp> {
+ bool loadsFrom(Operation *op, const MemorySlot &slot) const {
+ return cast<memref::CopyOp>(op).getSource() == slot.ptr;
+ }
+
+ bool storesTo(Operation *op, const MemorySlot &slot) const {
+ return cast<memref::CopyOp>(op).getTarget() == slot.ptr;
+ }
+
+ // Promotion replaces the slot with a vector value, so a copy into the slot
+ // becomes the code producing that value: a read of the copy's source.
+ Value getStored(Operation *op, const MemorySlot &slot, OpBuilder &builder,
+ Value reachingDef, const DataLayout &dataLayout) const {
+ auto copyOp = cast<memref::CopyOp>(op);
+ return readMemRefAsVector(builder, op->getLoc(), copyOp.getSource(),
+ cast<VectorType>(slot.valueType));
+ }
+
+ bool canUsesBeRemoved(Operation *op, const MemorySlot &slot,
+ const SmallPtrSetImpl<OpOperand *> &blockingUses,
+ SmallVectorImpl<OpOperand *> &newBlockingUses,
+ const DataLayout &dataLayout) const {
+ auto copyOp = cast<memref::CopyOp>(op);
+ auto vecType = dyn_cast<VectorType>(slot.valueType);
+ if (!vecType || vecType.isScalable())
+ return false;
+ bool srcIsSlot = copyOp.getSource() == slot.ptr;
+ bool dstIsSlot = copyOp.getTarget() == slot.ptr;
+ // Exactly one side must be this slot. A self-copy (both sides the slot) is
+ // not modeled here.
+ if (srcIsSlot == dstIsSlot)
+ return false;
+ // No further check: `memref.copy` verifies that both operands have the same
+ // shape and element type.
+ return true;
+ }
+
+ // A copy out of the slot is a use of the slot's vector value: it becomes a
+ // `vector.transfer_write` of that value into the copy's target.
+ DeletionKind
+ removeBlockingUses(Operation *op, const MemorySlot &slot,
+ const SmallPtrSetImpl<OpOperand *> &blockingUses,
+ OpBuilder &builder, Value reachingDefinition,
+ const DataLayout &dataLayout) const {
+ auto copyOp = cast<memref::CopyOp>(op);
+ if (copyOp.getSource() == slot.ptr) {
+ Location loc = op->getLoc();
+ Value mask;
+ if (memref::SubViewOp subView = getDynamicSubView(slot.ptr))
+ mask = buildSubViewMask(builder, loc, subView);
+ writeVectorToMemRef(builder, loc, reachingDefinition, copyOp.getTarget(),
+ mask);
+ }
+ return DeletionKind::Delete;
+ }
+};
+
//===----------------------------------------------------------------------===//
// PromotableOpInterface model for memref.subview
//===----------------------------------------------------------------------===//
@@ -295,5 +422,6 @@ void mlir::memref::registerMemorySlotOpInterfaceExternalModels(
registry.addExtension(+[](MLIRContext *ctx, memref::MemRefDialect *dialect) {
memref::SubViewOp::attachInterface<SubViewOpAliasModel>(*ctx);
memref::SubViewOp::attachInterface<SubViewOpPromotableModel>(*ctx);
+ memref::CopyOp::attachInterface<CopyOpMemOpModel>(*ctx);
});
}
diff --git a/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp b/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp
index c8c17f2039b025..995ed648eb8fa5 100644
--- a/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp
@@ -9,8 +9,8 @@
// This file implements memory slot interface models for
// `vector.transfer_read` and `vector.transfer_write`. These models implement
// `PromotableMemOpInterface` so Mem2Reg can replace vector transfers with uses
-// and definitions of a vector SSA value. The models for `memref.subview`
-// live in the MemRef dialect.
+// and definitions of a vector SSA value. The models for `memref.copy` and
+// `memref.subview` live in the MemRef dialect.
//
// A read uses the slot's current value; a write defines its next value.
// Static subviews use a vector matching the subview's shape: reads extract that
diff --git a/mlir/test/Dialect/Vector/mem2reg.mlir b/mlir/test/Dialect/Vector/mem2reg.mlir
index 62a35c90d324e3..e68918c4a9d13d 100644
--- a/mlir/test/Dialect/Vector/mem2reg.mlir
+++ b/mlir/test/Dialect/Vector/mem2reg.mlir
@@ -222,471 +222,3 @@ func.func @negative_dynamic_shape(%pad: f32, %d: index) -> vector<4xf32> {
%r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<?xf32>, vector<4xf32>
return %r : vector<4xf32>
}
-
-// -----
-
-// A static, same-rank memref.subview is a promotable sub-slice alias: a write
-// into the subview becomes vector.insert_strided_slice into the buffer's value,
-// and the whole-buffer read returns that composed value.
-
-// CHECK-LABEL: func.func @subview_static_write
-// CHECK-SAME: (%[[V:.*]]: vector<4xf32>, %[[INIT:.*]]: vector<8xf32>, %[[PAD:.*]]: f32)
-// CHECK-NOT: memref.alloca
-// CHECK-NOT: memref.subview
-// CHECK-NOT: vector.transfer_write
-// CHECK-NOT: vector.transfer_read
-// CHECK: %[[INS:.*]] = vector.insert_strided_slice %[[V]], %[[INIT]] offsets = [2], strides = [1]
-// CHECK: return %[[INS]] : vector<8xf32>
-func.func @subview_static_write(%v: vector<4xf32>, %init: vector<8xf32>, %pad: f32) -> vector<8xf32> {
- %c0 = arith.constant 0 : index
- %a = memref.alloca() : memref<8xf32>
- vector.transfer_write %init, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
- // Write via a subview.
- %sv = memref.subview %a[2] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: 2>>
- vector.transfer_write %v, %sv[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1], offset: 2>>
- %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<8xf32>, vector<8xf32>
- return %r : vector<8xf32>
-}
-
-// -----
-
-// A read of a static subview becomes vector.extract_strided_slice of the value.
-
-// CHECK-LABEL: func.func @subview_static_read
-// CHECK-NOT: memref.alloca
-// CHECK-NOT: memref.subview
-// CHECK: %[[EXT:.*]] = vector.extract_strided_slice %{{.*}} offsets = [2], sizes = [4], strides = [1]
-// CHECK: return %[[EXT]] : vector<4xf32>
-func.func @subview_static_read(%init: vector<8xf32>, %pad: f32) -> vector<4xf32> {
- %c0 = arith.constant 0 : index
- %a = memref.alloca() : memref<8xf32>
- vector.transfer_write %init, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
- // Read via a subview.
- %sv = memref.subview %a[2] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: 2>>
- %r = vector.transfer_read %sv[%c0], %pad {in_bounds = [true]} : memref<4xf32, strided<[1], offset: 2>>, vector<4xf32>
- return %r : vector<4xf32>
-}
-
-// -----
-
-// Each subview in a chain aliases its own parent, so a write through the inner
-// one is inserted into the outer slice, which is inserted back into the
-// buffer's value. The write lands at the sum of the two offsets.
-
-// CHECK-LABEL: func.func @subview_of_subview_write
-// CHECK-SAME: (%[[V:.*]]: vector<2xf32>, %[[INIT:.*]]: vector<8xf32>, %[[PAD:.*]]: f32)
-// CHECK-NOT: memref.alloca
-// CHECK-NOT: memref.subview
-// CHECK-NOT: vector.transfer_write
-// CHECK-NOT: vector.transfer_read
-// CHECK: vector.extract_strided_slice %[[INIT]] offsets = [2], sizes = [4], strides = [1]
-// CHECK: %[[INNER:.*]] = vector.insert_strided_slice %[[V]], %{{.*}} offsets = [1], strides = [1] : vector<2xf32> into vector<4xf32>
-// CHECK: %[[OUTER:.*]] = vector.insert_strided_slice %[[INNER]], %[[INIT]] offsets = [2], strides = [1] : vector<4xf32> into vector<8xf32>
-// CHECK: return %[[OUTER]] : vector<8xf32>
-func.func @subview_of_subview_write(%v: vector<2xf32>, %init: vector<8xf32>, %pad: f32) -> vector<8xf32> {
- %c0 = arith.constant 0 : index
- %a = memref.alloca() : memref<8xf32>
- vector.transfer_write %init, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
- // Write via a subview of a subview: element 3 of the buffer.
- %s1 = memref.subview %a[2] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: 2>>
- %s2 = memref.subview %s1[1] [2] [1] : memref<4xf32, strided<[1], offset: 2>> to memref<2xf32, strided<[1], offset: 3>>
- vector.transfer_write %v, %s2[%c0] {in_bounds = [true]} : vector<2xf32>, memref<2xf32, strided<[1], offset: 3>>
- %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<8xf32>, vector<8xf32>
- return %r : vector<8xf32>
-}
-
-// -----
-
-// A buffer accessed *only* through subviews, with no whole-buffer transfer to
-// seed the promoted value: the reaching definition is the allocator's default
-// value (ub.poison), and the written slice is inserted into it and read back
-// out.
-
-// CHECK-LABEL: func.func @subview_only_write_read
-// CHECK-SAME: (%[[V:.*]]: vector<4xf32>, %[[PAD:.*]]: f32)
-// CHECK-NOT: memref.alloca
-// CHECK-NOT: memref.subview
-// CHECK-NOT: vector.transfer_write
-// CHECK-NOT: vector.transfer_read
-// CHECK: %[[POISON:.*]] = ub.poison : vector<8xf32>
-// CHECK: vector.insert_strided_slice %[[V]], %[[POISON]] offsets = [0], strides = [1]
-func.func @subview_only_write_read(%v: vector<4xf32>, %pad: f32) -> vector<4xf32> {
- %c0 = arith.constant 0 : index
- %a = memref.alloca() : memref<8xf32>
- %svW = memref.subview %a[0] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1]>>
- vector.transfer_write %v, %svW[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1]>>
- %svR = memref.subview %a[0] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1]>>
- %r = vector.transfer_read %svR[%c0], %pad {in_bounds = [true]} : memref<4xf32, strided<[1]>>, vector<4xf32>
- return %r : vector<4xf32>
-}
-
-// -----
-
-// Two disjoint subview writes that together cover the buffer, followed by a
-// subview read spanning the boundary between them. The read composes both
-// writes through the parent value: it returns the low half of the first write
-// and the high half of the second (i.e. insert both slices, then extract).
-
-// CHECK-LABEL: func.func @subview_disjoint_writes_boundary_read
-// CHECK-SAME: (%[[VA:.*]]: vector<4xf32>, %[[VB:.*]]: vector<4xf32>, %[[PAD:.*]]: f32)
-// CHECK-NOT: memref.alloca
-// CHECK-NOT: memref.subview
-// CHECK: %[[D0:.*]] = vector.insert_strided_slice %[[VA]], %{{.*}} offsets = [0], strides = [1]
-// CHECK: %[[D1:.*]] = vector.insert_strided_slice %[[VB]], %[[D0]] offsets = [4], strides = [1]
-// CHECK: %[[R:.*]] = vector.extract_strided_slice %[[D1]] offsets = [2], sizes = [4], strides = [1]
-// CHECK: return %[[R]] : vector<4xf32>
-func.func @subview_disjoint_writes_boundary_read(%vA: vector<4xf32>, %vB: vector<4xf32>, %pad: f32) -> vector<4xf32> {
- %c0 = arith.constant 0 : index
- %a = memref.alloca() : memref<8xf32>
- %s0 = memref.subview %a[0] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1]>>
- %s4 = memref.subview %a[4] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: 4>>
- %s2 = memref.subview %a[2] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: 2>>
- vector.transfer_write %vA, %s0[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1]>>
- vector.transfer_write %vB, %s4[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1], offset: 4>>
- %r = vector.transfer_read %s2[%c0], %pad {in_bounds = [true]} : memref<4xf32, strided<[1], offset: 2>>, vector<4xf32>
- return %r : vector<4xf32>
-}
-
-// -----
-
-// Two overlapping subview writes: %vA covers [0, 4) and %vB covers [2, 6),
-// overlapping on [2, 4). Each write composes into the parent value in program
-// order, so the buffer holds %vA at [0, 2) and %vB at [2, 6). A subview read
-// spanning [1, 5) then extracts across the overlap, reading one lane of %vA
-// and three lanes of %vB from the composed value.
-
-// CHECK-LABEL: func.func @subview_overlapping_writes_overlap_read
-// CHECK-SAME: (%[[VA:.*]]: vector<4xf32>, %[[VB:.*]]: vector<4xf32>, %[[PAD:.*]]: f32)
-// CHECK-NOT: memref.alloca
-// CHECK-NOT: memref.subview
-// CHECK: %[[D0:.*]] = vector.insert_strided_slice %[[VA]], %{{.*}} offsets = [0], strides = [1]
-// CHECK: %[[D1:.*]] = vector.insert_strided_slice %[[VB]], %[[D0]] offsets = [2], strides = [1]
-// CHECK: %[[R:.*]] = vector.extract_strided_slice %[[D1]] offsets = [1], sizes = [4], strides = [1]
-// CHECK: return %[[R]] : vector<4xf32>
-func.func @subview_overlapping_writes_overlap_read(%vA: vector<4xf32>, %vB: vector<4xf32>, %pad: f32) -> vector<4xf32> {
- %c0 = arith.constant 0 : index
- %a = memref.alloca() : memref<8xf32>
- %s0 = memref.subview %a[0] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1]>>
- %s2 = memref.subview %a[2] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: 2>>
- %s1 = memref.subview %a[1] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: 1>>
- vector.transfer_write %vA, %s0[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1]>>
- vector.transfer_write %vB, %s2[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1], offset: 2>>
- %r = vector.transfer_read %s1[%c0], %pad {in_bounds = [true]} : memref<4xf32, strided<[1], offset: 1>>, vector<4xf32>
- return %r : vector<4xf32>
-}
-
-// -----
-
-// A buffer allocated before an scf.for and accessed inside the loop body only
-// through a subview, with a cross-iteration dependence (each iteration reads the
-// value the previous iteration wrote). The subview aliaser composes with region
-// promotion: the whole buffer is carried across iterations as a vector iter_arg,
-// the in-body read/write become extract/insert_strided_slice on that value.
-
-// CHECK-LABEL: func.func @subview_in_scf_for_cross_iter
-// CHECK-NOT: memref.alloca
-// CHECK-NOT: memref.subview
-// CHECK-NOT: vector.transfer_write
-// CHECK-NOT: vector.transfer_read
-// CHECK: %[[R:.*]] = scf.for %{{.*}} iter_args(%[[IT:.*]] = %{{.*}}) -> (vector<8xf32>)
-// CHECK: %[[V:.*]] = vector.extract_strided_slice %[[IT]] offsets = [0], sizes = [4], strides = [1]
-// CHECK: %[[N:.*]] = arith.addf %[[V]], %[[V]]
-// CHECK: %[[INS:.*]] = vector.insert_strided_slice %[[N]], %[[IT]] offsets = [0], strides = [1]
-// CHECK: scf.yield %[[INS]] : vector<8xf32>
-// CHECK: vector.extract_strided_slice %[[R]] offsets = [0], sizes = [4], strides = [1]
-func.func @subview_in_scf_for_cross_iter(%lb: index, %ub: index, %step: index, %init: vector<8xf32>, %pad: f32) -> vector<4xf32> {
- %c0 = arith.constant 0 : index
- %a = memref.alloca() : memref<8xf32>
- vector.transfer_write %init, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
- scf.for %i = %lb to %ub step %step {
- %sv = memref.subview %a[0] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1]>>
- %v = vector.transfer_read %sv[%c0], %pad {in_bounds = [true]} : memref<4xf32, strided<[1]>>, vector<4xf32>
- %n = arith.addf %v, %v : vector<4xf32>
- vector.transfer_write %n, %sv[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1]>>
- }
- %svr = memref.subview %a[0] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1]>>
- %r = vector.transfer_read %svr[%c0], %pad {in_bounds = [true]} : memref<4xf32, strided<[1]>>, vector<4xf32>
- return %r : vector<4xf32>
-}
-
-// -----
-
-// Reads and writes may occur in any order: a read of the subview before any
-// write returns the slot's default value (ub.poison), matching the semantics of
-// reading an uninitialized alloca. A later read observes the written value.
-
-// CHECK-LABEL: func.func @subview_read_before_write
-// CHECK-SAME: (%[[V:.*]]: vector<4xf32>, %[[PAD:.*]]: f32)
-// CHECK-NOT: memref.alloca
-// CHECK-NOT: memref.subview
-// CHECK: %[[POISON:.*]] = ub.poison : vector<8xf32>
-// CHECK: %[[R0:.*]] = vector.extract_strided_slice %[[POISON]] offsets = [0], sizes = [4], strides = [1]
-// CHECK: %[[INS:.*]] = vector.insert_strided_slice %[[V]], %[[POISON]] offsets = [0], strides = [1]
-// CHECK: %[[R1:.*]] = vector.extract_strided_slice %[[INS]] offsets = [0], sizes = [4], strides = [1]
-// CHECK: return %[[R0]], %[[R1]] : vector<4xf32>, vector<4xf32>
-func.func @subview_read_before_write(%v: vector<4xf32>, %pad: f32) -> (vector<4xf32>, vector<4xf32>) {
- %c0 = arith.constant 0 : index
- %a = memref.alloca() : memref<8xf32>
- %s = memref.subview %a[0] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1]>>
- %r0 = vector.transfer_read %s[%c0], %pad {in_bounds = [true]} : memref<4xf32, strided<[1]>>, vector<4xf32>
- vector.transfer_write %v, %s[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1]>>
- %r1 = vector.transfer_read %s[%c0], %pad {in_bounds = [true]} : memref<4xf32, strided<[1]>>, vector<4xf32>
- return %r0, %r1 : vector<4xf32>, vector<4xf32>
-}
-
-// -----
-
-// A dynamic subview offset cannot be expressed as a static strided slice: the
-// buffer is left untouched.
-
-// CHECK-LABEL: func.func @negative_subview_dynamic_offset
-// CHECK: memref.alloca
-// CHECK: memref.subview
-func.func @negative_subview_dynamic_offset(%v: vector<4xf32>, %init: vector<8xf32>, %pad: f32, %off: index) -> vector<8xf32> {
- %c0 = arith.constant 0 : index
- %a = memref.alloca() : memref<8xf32>
- vector.transfer_write %init, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
- %sv = memref.subview %a[%off] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: ?>>
- vector.transfer_write %v, %sv[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1], offset: ?>>
- %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<8xf32>, vector<8xf32>
- return %r : vector<8xf32>
-}
-
-// -----
-
-// A rank-reducing subview (2d -> 1d) is not promotable: strided-slice projection
-// requires equal rank.
-
-// CHECK-LABEL: func.func @negative_subview_rank_reducing
-// CHECK: memref.alloca
-// CHECK: memref.subview
-func.func @negative_subview_rank_reducing(%v: vector<4xf32>, %init: vector<2x4xf32>, %pad: f32) -> vector<2x4xf32> {
- %c0 = arith.constant 0 : index
- %a = memref.alloca() : memref<2x4xf32>
- vector.transfer_write %init, %a[%c0, %c0] {in_bounds = [true, true]} : vector<2x4xf32>, memref<2x4xf32>
- %sv = memref.subview %a[1, 0] [1, 4] [1, 1] : memref<2x4xf32> to memref<4xf32, strided<[1], offset: 4>>
- vector.transfer_write %v, %sv[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1], offset: 4>>
- %r = vector.transfer_read %a[%c0, %c0], %pad {in_bounds = [true, true]} : memref<2x4xf32>, vector<2x4xf32>
- return %r : vector<2x4xf32>
-}
-
-
-// A prefetching software-pipelined loop. The stage buffer double-buffers the
-// prefetched tile: the prefetch of the next tile is skipped on the last iteration.
-
-// CHECK-LABEL: func.func @pipelined_prefetch
-// CHECK-NOT: memref.alloca
-// CHECK: %[[P0:.*]] = vector.transfer_read %{{.*}} : memref<64xf32>, vector<8xf32>
-// CHECK-NOT: vector.transfer_write
-// CHECK: scf.for %[[I:.*]] = {{.*}} iter_args(%[[ACC:.*]] = %{{.*}}, %[[STAGE:.*]] = %[[P0]]) -> (vector<8xf32>, vector<8xf32>)
-// CHECK: %[[ACCN:.*]] = arith.addf %[[ACC]], %[[STAGE]] : vector<8xf32>
-// CHECK: %[[INEXT:.*]] = arith.addi %[[I]], %{{.*}}
-// CHECK: %[[G:.*]] = arith.cmpi slt, %[[INEXT]], %{{.*}}
-// CHECK: %[[NEXT:.*]] = scf.if %[[G]] -> (vector<8xf32>) {
-// CHECK: %[[CUR:.*]] = vector.transfer_read %{{.*}}[%[[INEXT]]]
-// CHECK: scf.yield %[[CUR]] : vector<8xf32>
-// CHECK: } else {
-// CHECK: scf.yield %[[STAGE]] : vector<8xf32>
-// CHECK: }
-// CHECK: scf.yield %[[ACCN]], %[[NEXT]] : vector<8xf32>, vector<8xf32>
-func.func @pipelined_prefetch(%lb: index, %ub: index, %step: index, %in: memref<64xf32>, %pad: f32) -> vector<8xf32> {
- %c0 = arith.constant 0 : index
- %c1 = arith.constant 1 : index
- %cst = arith.constant dense<0.0> : vector<8xf32>
- %stage = memref.alloca() : memref<8xf32>
- // Prologue: prefetch tile[0] into the stage buffer.
- %p0 = vector.transfer_read %in[%c0], %pad {in_bounds = [true]} : memref<64xf32>, vector<8xf32>
- vector.transfer_write %p0, %stage[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
- %r = scf.for %i = %lb to %ub step %step iter_args(%acc = %cst) -> (vector<8xf32>) {
- // Consume the tile prefetched by the previous iteration (unconditional).
- %tile = vector.transfer_read %stage[%c0], %pad {in_bounds = [true]} : memref<8xf32>, vector<8xf32>
- %accn = arith.addf %acc, %tile : vector<8xf32>
- // Prefetch the next tile into the stage buffer (skip on the last iteration).
- %inext = arith.addi %i, %c1 : index
- %g = arith.cmpi slt, %inext, %ub : index
- scf.if %g {
- %next = vector.transfer_read %in[%inext], %pad {in_bounds = [true]} : memref<64xf32>, vector<8xf32>
- vector.transfer_write %next, %stage[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
- }
- scf.yield %accn : vector<8xf32>
- }
- return %r : vector<8xf32>
-}
-
-// -----
-
-// A whole-buffer accumulator updated conditionally inside an scf.if (a masked
-// running reduction).
-
-// CHECK-LABEL: func.func @cond_accumulate
-// CHECK-NOT: memref.alloca
-// CHECK-NOT: vector.transfer_write
-// CHECK-NOT: vector.transfer_read
-// CHECK: %[[R:.*]] = scf.for {{.*}} iter_args(%[[ACC:.*]] = %{{.*}}) -> (vector<4xf32>)
-// CHECK: %[[M:.*]] = memref.load
-// CHECK: %[[NEW:.*]] = scf.if %[[M]] -> (vector<4xf32>) {
-// CHECK: %[[D:.*]] = arith.addf %[[ACC]], %[[ACC]] : vector<4xf32>
-// CHECK: scf.yield %[[D]] : vector<4xf32>
-// CHECK: } else {
-// CHECK: scf.yield %[[ACC]] : vector<4xf32>
-// CHECK: }
-// CHECK: scf.yield %[[NEW]] : vector<4xf32>
-// CHECK: return %[[R]] : vector<4xf32>
-func.func @cond_accumulate(%lb: index, %ub: index, %step: index, %init: vector<4xf32>, %pad: f32, %mask: memref<?xi1>) -> vector<4xf32> {
- %c0 = arith.constant 0 : index
- %a = memref.alloca() : memref<4xf32>
- vector.transfer_write %init, %a[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32>
- scf.for %i = %lb to %ub step %step {
- %m = memref.load %mask[%i] : memref<?xi1>
- %v = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<4xf32>, vector<4xf32>
- scf.if %m {
- %d = arith.addf %v, %v : vector<4xf32>
- vector.transfer_write %d, %a[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32>
- }
- scf.yield
- }
- %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<4xf32>, vector<4xf32>
- return %r : vector<4xf32>
-}
-
-// -----
-
-// A tiled GEMM with a dynamic K early-exit. A local accumulator tile D sums the
-// K-tiles via vector.contract only while k < dyn_k.
-
-// CHECK-LABEL: func.func @gemm_k_early_exit
-// CHECK-SAME: (%[[A:.*]]: memref<4x16xf32>, %[[B:.*]]: memref<16x4xf32>, %[[C:.*]]: memref<4x4xf32>, %[[DYNK:.*]]: index, %[[PAD:.*]]: f32)
-// CHECK-NOT: memref.alloca
-// CHECK: %[[R:.*]] = scf.for %[[K:.*]] = {{.*}} iter_args(%[[ACC:.*]] = %{{.*}}) -> (vector<4x4xf32>)
-// CHECK: %[[INRANGE:.*]] = arith.cmpi slt, %[[K]], %[[DYNK]]
-// CHECK: %[[NEW:.*]] = scf.if %[[INRANGE]] -> (vector<4x4xf32>) {
-// CHECK: %[[ATILE:.*]] = vector.transfer_read %[[A]]
-// CHECK: %[[BTILE:.*]] = vector.transfer_read %[[B]]
-// CHECK: %[[MM:.*]] = vector.contract {{.*}} %[[ATILE]], %[[BTILE]], %[[ACC]]
-// CHECK: scf.yield %[[MM]] : vector<4x4xf32>
-// CHECK: } else {
-// CHECK: scf.yield %[[ACC]] : vector<4x4xf32>
-// CHECK: }
-// CHECK: scf.yield %[[NEW]] : vector<4x4xf32>
-// CHECK: %[[CVAL:.*]] = vector.transfer_read %[[C]]
-// CHECK: %[[SUM:.*]] = arith.addf %[[CVAL]], %[[R]] : vector<4x4xf32>
-// CHECK: vector.transfer_write %[[SUM]], %[[C]]
-func.func @gemm_k_early_exit(%A: memref<4x16xf32>, %B: memref<16x4xf32>,
- %C: memref<4x4xf32>, %dyn_k: index, %pad: f32) {
- %c0 = arith.constant 0 : index
- %c4 = arith.constant 4 : index
- %c16 = arith.constant 16 : index
- %cst = arith.constant dense<0.0> : vector<4x4xf32>
- // Local accumulator tile D, zero-initialized.
- %d = memref.alloca() : memref<4x4xf32>
- vector.transfer_write %cst, %d[%c0, %c0] {in_bounds = [true, true]} : vector<4x4xf32>, memref<4x4xf32>
- scf.for %k = %c0 to %c16 step %c4 {
- // Early exit: only accumulate K tiles whose offset is below dyn_k.
- %inrange = arith.cmpi slt, %k, %dyn_k : index
- scf.if %inrange {
- %atile = vector.transfer_read %A[%c0, %k], %pad {in_bounds = [true, true]} : memref<4x16xf32>, vector<4x4xf32>
- %btile = vector.transfer_read %B[%k, %c0], %pad {in_bounds = [true, true]} : memref<16x4xf32>, vector<4x4xf32>
- %acc = vector.transfer_read %d[%c0, %c0], %pad {in_bounds = [true, true]} : memref<4x4xf32>, vector<4x4xf32>
- %mm = vector.contract {indexing_maps = [affine_map<(m, n, k) -> (m, k)>, affine_map<(m, n, k) -> (k, n)>, affine_map<(m, n, k) -> (m, n)>], iterator_types = ["parallel", "parallel", "reduction"], kind = #vector.kind<add>} %atile, %btile, %acc : vector<4x4xf32>, vector<4x4xf32> into vector<4x4xf32>
- vector.transfer_write %mm, %d[%c0, %c0] {in_bounds = [true, true]} : vector<4x4xf32>, memref<4x4xf32>
- }
- scf.yield
- }
- // Accumulate the local tile D into the C parameter: C = C + D.
- %cval = vector.transfer_read %C[%c0, %c0], %pad {in_bounds = [true, true]} : memref<4x4xf32>, vector<4x4xf32>
- %dval = vector.transfer_read %d[%c0, %c0], %pad {in_bounds = [true, true]} : memref<4x4xf32>, vector<4x4xf32>
- %sum = arith.addf %cval, %dval : vector<4x4xf32>
- vector.transfer_write %sum, %C[%c0, %c0] {in_bounds = [true, true]} : vector<4x4xf32>, memref<4x4xf32>
- return
-}
-
-// -----
-
-// A whole-buffer read with a mask promotes to select(mask, reachingDef, padding).
-// CHECK-LABEL: func.func @read_with_mask(
-// CHECK-SAME: %[[V:.*]]: vector<8xf32>, %[[M:.*]]: vector<8xi1>, %[[PAD:.*]]: f32
-// CHECK-NOT: memref.alloca
-// CHECK: %[[PS:.*]] = vector.broadcast %[[PAD]] : f32 to vector<8xf32>
-// CHECK: %[[SEL:.*]] = arith.select %[[M]], %[[V]], %[[PS]]
-// CHECK: return %[[SEL]]
-func.func @read_with_mask(%v: vector<8xf32>, %m: vector<8xi1>, %pad: f32) -> vector<8xf32> {
- %c0 = arith.constant 0 : index
- %a = memref.alloca() : memref<8xf32>
- vector.transfer_write %v, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
- %r = vector.transfer_read %a[%c0], %pad, %m {in_bounds = [true]} : memref<8xf32>, vector<8xf32>
- return %r : vector<8xf32>
-}
-
-// -----
-
-// A whole-buffer write with a mask selects the stored value for active lanes
-// and preserves the previous value for the rest.
-// CHECK-LABEL: func.func @write_with_mask(
-// CHECK-SAME: %[[V:.*]]: vector<8xf32>, %[[W:.*]]: vector<8xf32>, %[[M:.*]]: vector<8xi1>
-// CHECK-NOT: memref.alloca
-// CHECK: %[[SEL:.*]] = arith.select %[[M]], %[[W]], %[[V]]
-// CHECK: return %[[SEL]]
-func.func @write_with_mask(%v: vector<8xf32>, %w: vector<8xf32>, %m: vector<8xi1>, %pad: f32) -> vector<8xf32> {
- %c0 = arith.constant 0 : index
- %a = memref.alloca() : memref<8xf32>
- vector.transfer_write %v, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
- vector.transfer_write %w, %a[%c0], %m {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
- %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<8xf32>, vector<8xf32>
- return %r : vector<8xf32>
-}
-
-// -----
-
-// A write with a mask into a static subview composes through the subview aliaser:
-// the region is projected out (extract_strided_slice), the mask selects the
-// stored value, and the result is inserted back (insert_strided_slice).
-// CHECK-LABEL: func.func @write_with_mask_static_subview(
-// CHECK-SAME: %[[V:.*]]: vector<4xf32>, %[[INIT:.*]]: vector<8xf32>, %{{.*}}: f32, %[[M:.*]]: vector<4xi1>
-// CHECK-NOT: memref.alloca
-// CHECK: %[[EX:.*]] = vector.extract_strided_slice %[[INIT]] offsets = [2], sizes = [4]
-// CHECK: %[[SEL:.*]] = arith.select %[[M]], %[[V]], %[[EX]]
-// CHECK: %[[INS:.*]] = vector.insert_strided_slice %[[SEL]], %[[INIT]] offsets = [2]
-// CHECK: return %[[INS]]
-func.func @write_with_mask_static_subview(%v: vector<4xf32>, %init: vector<8xf32>, %pad: f32, %m: vector<4xi1>) -> vector<8xf32> {
- %c0 = arith.constant 0 : index
- %a = memref.alloca() : memref<8xf32>
- vector.transfer_write %init, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
- %sv = memref.subview %a[2] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: 2>>
- vector.transfer_write %v, %sv[%c0], %m {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1], offset: 2>>
- %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<8xf32>, vector<8xf32>
- return %r : vector<8xf32>
-}
-
-// -----
-
-// NEGATIVE: the masked form (`vector.mask` wrapping the transfer) is not
-// promoted. The transfer's use of the buffer is inside a region, and Mem2Reg
-// only promotes through regions whose parent op implements
-// `PromotableRegionOpInterface`, which `vector.mask` does not.
-
-// CHECK-LABEL: func.func @negative_masked_read(
-// CHECK: memref.alloca
-// CHECK: vector.mask
-func.func @negative_masked_read(%v: vector<8xf32>, %m: vector<8xi1>, %pad: f32) -> vector<8xf32> {
- %c0 = arith.constant 0 : index
- %a = memref.alloca() : memref<8xf32>
- vector.transfer_write %v, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
- %r = vector.mask %m { vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<8xf32>, vector<8xf32> } : vector<8xi1> -> vector<8xf32>
- return %r : vector<8xf32>
-}
-
-// -----
-
-// NEGATIVE: a write wrapped in vector.mask also prevents buffer promotion.
-
-// CHECK-LABEL: func.func @negative_masked_write(
-// CHECK: memref.alloca
-// CHECK: vector.mask
-func.func @negative_masked_write(%v: vector<8xf32>, %w: vector<8xf32>, %m: vector<8xi1>, %pad: f32) -> vector<8xf32> {
- %c0 = arith.constant 0 : index
- %a = memref.alloca() : memref<8xf32>
- vector.transfer_write %v, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
- vector.mask %m { vector.transfer_write %w, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32> } : vector<8xi1>
- %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<8xf32>, vector<8xf32>
- return %r : vector<8xf32>
-}
diff --git a/mlir/test/Transforms/mem2reg.mlir b/mlir/test/Transforms/mem2reg/core.mlir
similarity index 100%
rename from mlir/test/Transforms/mem2reg.mlir
rename to mlir/test/Transforms/mem2reg/core.mlir
diff --git a/mlir/test/Dialect/Vector/mem2reg-dynamic-subview.mlir b/mlir/test/Transforms/mem2reg/dynamic-subview.mlir
similarity index 100%
rename from mlir/test/Dialect/Vector/mem2reg-dynamic-subview.mlir
rename to mlir/test/Transforms/mem2reg/dynamic-subview.mlir
diff --git a/mlir/test/Transforms/mem2reg/memref-copy.mlir b/mlir/test/Transforms/mem2reg/memref-copy.mlir
new file mode 100644
index 00000000000000..27bd5a7e6100d7
--- /dev/null
+++ b/mlir/test/Transforms/mem2reg/memref-copy.mlir
@@ -0,0 +1,173 @@
+// RUN: mlir-opt %s -mem2reg -split-input-file | FileCheck %s
+
+
+// `memref.copy` participates in promotion as a transfer of the buffer's value.
+// Copying into the whole slot and reading it becomes a read of the source.
+
+// CHECK-LABEL: func.func @copy_in_whole(
+// CHECK-SAME: %[[SRC:.*]]: memref<8x16xf32>, %[[PAD:.*]]: f32
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: memref.copy
+// CHECK: %[[R:.*]] = vector.transfer_read %[[SRC]]{{.*}} : memref<8x16xf32>, vector<8x16xf32>
+// CHECK: return %[[R]]
+func.func @copy_in_whole(%src: memref<8x16xf32>, %pad: f32) -> vector<8x16xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8x16xf32>
+ memref.copy %src, %a : memref<8x16xf32> to memref<8x16xf32>
+ %r = vector.transfer_read %a[%c0, %c0], %pad {in_bounds = [true, true]} : memref<8x16xf32>, vector<8x16xf32>
+ return %r : vector<8x16xf32>
+}
+
+// -----
+
+// Copying out of the slot becomes a transfer_write of its value to the target.
+
+// CHECK-LABEL: func.func @copy_out(
+// CHECK-SAME: %[[V:.*]]: vector<8x16xf32>, %[[DST:.*]]: memref<8x16xf32>
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: memref.copy
+// CHECK: vector.transfer_write %[[V]], %[[DST]]
+func.func @copy_out(%v: vector<8x16xf32>, %dst: memref<8x16xf32>) {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8x16xf32>
+ vector.transfer_write %v, %a[%c0, %c0] {in_bounds = [true, true]} : vector<8x16xf32>, memref<8x16xf32>
+ memref.copy %a, %dst : memref<8x16xf32> to memref<8x16xf32>
+ return
+}
+
+// -----
+
+// Copying between two promotable slots eliminates both buffers.
+
+// CHECK-LABEL: func.func @copy_between_slots(
+// CHECK-SAME: %[[V:.*]]: vector<8x16xf32>
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: memref.copy
+// CHECK: return %[[V]]
+func.func @copy_between_slots(%v: vector<8x16xf32>, %pad: f32) -> vector<8x16xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8x16xf32>
+ %b = memref.alloca() : memref<8x16xf32>
+ vector.transfer_write %v, %a[%c0, %c0] {in_bounds = [true, true]} : vector<8x16xf32>, memref<8x16xf32>
+ memref.copy %a, %b : memref<8x16xf32> to memref<8x16xf32>
+ %r = vector.transfer_read %b[%c0, %c0], %pad {in_bounds = [true, true]} : memref<8x16xf32>, vector<8x16xf32>
+ return %r : vector<8x16xf32>
+}
+
+// -----
+
+// NEGATIVE: a self-copy references the slot on both sides and is not modeled, so
+// the buffer is not promoted.
+
+// CHECK-LABEL: func.func @negative_self_copy(
+// CHECK: memref.alloca
+// CHECK: memref.copy
+// CHECK: vector.transfer_read
+func.func @negative_self_copy(%v: vector<8x16xf32>, %pad: f32) -> vector<8x16xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8x16xf32>
+ vector.transfer_write %v, %a[%c0, %c0] {in_bounds = [true, true]} : vector<8x16xf32>, memref<8x16xf32>
+ memref.copy %a, %a : memref<8x16xf32> to memref<8x16xf32>
+ %r = vector.transfer_read %a[%c0, %c0], %pad {in_bounds = [true, true]} : memref<8x16xf32>, vector<8x16xf32>
+ return %r : vector<8x16xf32>
+}
+
+// -----
+
+// NEGATIVE: copies of scalable slots are not supported. Such buffers can still
+// be promoted through transfers (see @scalable_whole_buffer_in_loop).
+
+// CHECK-LABEL: func.func @negative_scalable_copy(
+// CHECK: memref.alloca
+// CHECK: memref.copy
+// CHECK: vector.transfer_read
+func.func @negative_scalable_copy(%src: memref<8xf32>, %pad: f32) -> vector<[4]xf32> {
+ %c0 = arith.constant 0 : index
+ %c4 = arith.constant 4 : index
+ %vs = vector.vscale
+ %sz = arith.muli %vs, %c4 : index
+ %a = memref.alloca(%sz) : memref<?xf32>
+ memref.copy %src, %a : memref<8xf32> to memref<?xf32>
+ %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<?xf32>, vector<[4]xf32>
+ return %r : vector<[4]xf32>
+}
+
+// -----
+
+// A `memref.copy` into a dynamic subview, read back afterwards: the copy
+// becomes a read of the source, kept only where it falls inside the extent.
+
+// CHECK-LABEL: func.func @copy_into_dyn_subview(
+// CHECK-SAME: %[[SRC:.*]]: memref<?x?xf32>, %[[N:.*]]: index
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: memref.copy
+// CHECK: %[[Z:.*]] = vector.broadcast %{{.*}} : f32 to vector<8x16xf32>
+// CHECK: %[[RD:.*]] = vector.transfer_read %[[SRC]]{{.*}} : memref<?x?xf32>, vector<8x16xf32>
+// CHECK: %[[M:.*]] = vector.create_mask %[[N]], %[[N]] : vector<8x16xi1>
+// CHECK: %[[SEL:.*]] = arith.select %[[M]], %[[RD]], %[[Z]]
+// CHECK: return %[[SEL]]
+func.func @copy_into_dyn_subview(%src: memref<?x?xf32>, %n: index, %pad: f32) -> vector<8x16xf32> {
+ %c0 = arith.constant 0 : index
+ %z = arith.constant 0.000000e+00 : f32
+ %zv = vector.broadcast %z : f32 to vector<8x16xf32>
+ %a = memref.alloca() : memref<8x16xf32>
+ vector.transfer_write %zv, %a[%c0, %c0] {in_bounds = [true, true]} : vector<8x16xf32>, memref<8x16xf32>
+ %sv = memref.subview %a[0, 0] [%n, %n] [1, 1] : memref<8x16xf32> to memref<?x?xf32, strided<[16, 1]>>
+ memref.copy %src, %sv : memref<?x?xf32> to memref<?x?xf32, strided<[16, 1]>>
+ %r = vector.transfer_read %a[%c0, %c0], %pad {in_bounds = [true, true]} : memref<8x16xf32>, vector<8x16xf32>
+ return %r : vector<8x16xf32>
+}
+
+// -----
+
+// Copy from one dynamic subview into another: the source subview is read, and
+// only the elements inside the extent reach the buffer's value.
+
+// CHECK-LABEL: func.func @copy_dynsub_to_dynsub(
+// CHECK-SAME: %[[V:.*]]: vector<8x16xf32>, %[[SRC:.*]]: memref<8x16xf32>, %[[N:.*]]: index
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: memref.copy
+// CHECK: %[[SS:.*]] = memref.subview %[[SRC]][0, 0] [8, %[[N]]] [1, 1]
+// CHECK: %[[RD:.*]] = vector.transfer_read %[[SS]]{{.*}} {in_bounds = [true, false]}
+// CHECK: %[[M:.*]] = vector.create_mask %{{.*}}, %[[N]] : vector<8x16xi1>
+// CHECK: %[[SEL:.*]] = arith.select %[[M]], %[[RD]], %[[V]]
+// CHECK: return %[[SEL]]
+func.func @copy_dynsub_to_dynsub(%v: vector<8x16xf32>, %src: memref<8x16xf32>, %n: index, %pad: f32) -> vector<8x16xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8x16xf32>
+ vector.transfer_write %v, %a[%c0, %c0] {in_bounds = [true, true]} : vector<8x16xf32>, memref<8x16xf32>
+ %ssrc = memref.subview %src[0, 0] [8, %n] [1, 1] : memref<8x16xf32> to memref<8x?xf32, strided<[16, 1]>>
+ %sdst = memref.subview %a[0, 0] [8, %n] [1, 1] : memref<8x16xf32> to memref<8x?xf32, strided<[16, 1]>>
+ memref.copy %ssrc, %sdst : memref<8x?xf32, strided<[16, 1]>> to memref<8x?xf32, strided<[16, 1]>>
+ %r = vector.transfer_read %a[%c0, %c0], %pad {in_bounds = [true, true]} : memref<8x16xf32>, vector<8x16xf32>
+ return %r : vector<8x16xf32>
+}
+
+// -----
+
+// Copy from a global into a dynamic subview of a static staging buffer, in a
+// non-default memory space.
+memref.global "private" constant @tile : memref<128x64xf16> = dense<1.000000e+00>
+
+// CHECK-LABEL: func.func @copy_global_to_dyn_subview(
+// CHECK-SAME: %[[V:[a-z0-9_]+]]: vector<128x64xf16>, %[[N:[a-z0-9_]+]]: index
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: memref.copy
+// CHECK: %[[G:.*]] = memref.get_global @tile : memref<128x64xf16>
+// CHECK: %[[GSV:.*]] = memref.subview %[[G]][0, 0] [%[[N]], 64] [1, 1]
+// CHECK: %[[PAD:.*]] = arith.constant 0.000000e+00 : f16
+// CHECK: %[[RD:.*]] = vector.transfer_read %[[GSV]][%{{.*}}, %{{.*}}], %[[PAD]] {in_bounds = [false, true]} : memref<?x64xf16, strided<[64, 1]>>, vector<128x64xf16>
+// CHECK: %[[CM:.*]] = vector.create_mask %[[N]], %{{.*}} : vector<128x64xi1>
+// CHECK: %[[SEL:.*]] = arith.select %[[CM]], %[[RD]], %[[V]]
+// CHECK: return %[[SEL]]
+func.func @copy_global_to_dyn_subview(%v: vector<128x64xf16>, %n: index, %pad: f16) -> vector<128x64xf16> {
+ %c0 = arith.constant 0 : index
+ %g = memref.get_global @tile : memref<128x64xf16>
+ %gsv = memref.subview %g[0, 0] [%n, 64] [1, 1] : memref<128x64xf16> to memref<?x64xf16, strided<[64, 1]>>
+ %slm = memref.alloca() : memref<128x64xf16, 3>
+ vector.transfer_write %v, %slm[%c0, %c0] {in_bounds = [true, true]} : vector<128x64xf16>, memref<128x64xf16, 3>
+ %sv = memref.subview %slm[0, 0] [%n, 64] [1, 1] : memref<128x64xf16, 3> to memref<?x64xf16, strided<[64, 1]>, 3>
+ memref.copy %gsv, %sv : memref<?x64xf16, strided<[64, 1]>> to memref<?x64xf16, strided<[64, 1]>, 3>
+ %r = vector.transfer_read %slm[%c0, %c0], %pad {in_bounds = [true, true]} : memref<128x64xf16, 3>, vector<128x64xf16>
+ return %r : vector<128x64xf16>
+}
diff --git a/mlir/test/Transforms/mem2reg/static-subview.mlir b/mlir/test/Transforms/mem2reg/static-subview.mlir
new file mode 100644
index 00000000000000..73f2743295f058
--- /dev/null
+++ b/mlir/test/Transforms/mem2reg/static-subview.mlir
@@ -0,0 +1,467 @@
+// RUN: mlir-opt %s --mem2reg --split-input-file | FileCheck %s
+
+// A static, same-rank memref.subview is a promotable sub-slice alias: a write
+// into the subview becomes vector.insert_strided_slice into the buffer's value,
+// and the whole-buffer read returns that composed value.
+
+// CHECK-LABEL: func.func @subview_static_write
+// CHECK-SAME: (%[[V:.*]]: vector<4xf32>, %[[INIT:.*]]: vector<8xf32>, %[[PAD:.*]]: f32)
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: memref.subview
+// CHECK-NOT: vector.transfer_write
+// CHECK-NOT: vector.transfer_read
+// CHECK: %[[INS:.*]] = vector.insert_strided_slice %[[V]], %[[INIT]] offsets = [2], strides = [1]
+// CHECK: return %[[INS]] : vector<8xf32>
+func.func @subview_static_write(%v: vector<4xf32>, %init: vector<8xf32>, %pad: f32) -> vector<8xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8xf32>
+ vector.transfer_write %init, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
+ // Write via a subview.
+ %sv = memref.subview %a[2] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: 2>>
+ vector.transfer_write %v, %sv[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1], offset: 2>>
+ %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<8xf32>, vector<8xf32>
+ return %r : vector<8xf32>
+}
+
+// -----
+
+// A read of a static subview becomes vector.extract_strided_slice of the value.
+
+// CHECK-LABEL: func.func @subview_static_read
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: memref.subview
+// CHECK: %[[EXT:.*]] = vector.extract_strided_slice %{{.*}} offsets = [2], sizes = [4], strides = [1]
+// CHECK: return %[[EXT]] : vector<4xf32>
+func.func @subview_static_read(%init: vector<8xf32>, %pad: f32) -> vector<4xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8xf32>
+ vector.transfer_write %init, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
+ // Read via a subview.
+ %sv = memref.subview %a[2] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: 2>>
+ %r = vector.transfer_read %sv[%c0], %pad {in_bounds = [true]} : memref<4xf32, strided<[1], offset: 2>>, vector<4xf32>
+ return %r : vector<4xf32>
+}
+
+// -----
+
+// Each subview in a chain aliases its own parent, so a write through the inner
+// one is inserted into the outer slice, which is inserted back into the
+// buffer's value. The write lands at the sum of the two offsets.
+
+// CHECK-LABEL: func.func @subview_of_subview_write
+// CHECK-SAME: (%[[V:.*]]: vector<2xf32>, %[[INIT:.*]]: vector<8xf32>, %[[PAD:.*]]: f32)
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: memref.subview
+// CHECK-NOT: vector.transfer_write
+// CHECK-NOT: vector.transfer_read
+// CHECK: vector.extract_strided_slice %[[INIT]] offsets = [2], sizes = [4], strides = [1]
+// CHECK: %[[INNER:.*]] = vector.insert_strided_slice %[[V]], %{{.*}} offsets = [1], strides = [1] : vector<2xf32> into vector<4xf32>
+// CHECK: %[[OUTER:.*]] = vector.insert_strided_slice %[[INNER]], %[[INIT]] offsets = [2], strides = [1] : vector<4xf32> into vector<8xf32>
+// CHECK: return %[[OUTER]] : vector<8xf32>
+func.func @subview_of_subview_write(%v: vector<2xf32>, %init: vector<8xf32>, %pad: f32) -> vector<8xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8xf32>
+ vector.transfer_write %init, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
+ // Write via a subview of a subview: element 3 of the buffer.
+ %s1 = memref.subview %a[2] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: 2>>
+ %s2 = memref.subview %s1[1] [2] [1] : memref<4xf32, strided<[1], offset: 2>> to memref<2xf32, strided<[1], offset: 3>>
+ vector.transfer_write %v, %s2[%c0] {in_bounds = [true]} : vector<2xf32>, memref<2xf32, strided<[1], offset: 3>>
+ %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<8xf32>, vector<8xf32>
+ return %r : vector<8xf32>
+}
+
+// -----
+
+// A buffer accessed *only* through subviews, with no whole-buffer transfer to
+// seed the promoted value: the reaching definition is the allocator's default
+// value (ub.poison), and the written slice is inserted into it and read back
+// out.
+
+// CHECK-LABEL: func.func @subview_only_write_read
+// CHECK-SAME: (%[[V:.*]]: vector<4xf32>, %[[PAD:.*]]: f32)
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: memref.subview
+// CHECK-NOT: vector.transfer_write
+// CHECK-NOT: vector.transfer_read
+// CHECK: %[[POISON:.*]] = ub.poison : vector<8xf32>
+// CHECK: vector.insert_strided_slice %[[V]], %[[POISON]] offsets = [0], strides = [1]
+func.func @subview_only_write_read(%v: vector<4xf32>, %pad: f32) -> vector<4xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8xf32>
+ %svW = memref.subview %a[0] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1]>>
+ vector.transfer_write %v, %svW[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1]>>
+ %svR = memref.subview %a[0] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1]>>
+ %r = vector.transfer_read %svR[%c0], %pad {in_bounds = [true]} : memref<4xf32, strided<[1]>>, vector<4xf32>
+ return %r : vector<4xf32>
+}
+
+// -----
+
+// Two disjoint subview writes that together cover the buffer, followed by a
+// subview read spanning the boundary between them. The read composes both
+// writes through the parent value: it returns the low half of the first write
+// and the high half of the second (i.e. insert both slices, then extract).
+
+// CHECK-LABEL: func.func @subview_disjoint_writes_boundary_read
+// CHECK-SAME: (%[[VA:.*]]: vector<4xf32>, %[[VB:.*]]: vector<4xf32>, %[[PAD:.*]]: f32)
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: memref.subview
+// CHECK: %[[D0:.*]] = vector.insert_strided_slice %[[VA]], %{{.*}} offsets = [0], strides = [1]
+// CHECK: %[[D1:.*]] = vector.insert_strided_slice %[[VB]], %[[D0]] offsets = [4], strides = [1]
+// CHECK: %[[R:.*]] = vector.extract_strided_slice %[[D1]] offsets = [2], sizes = [4], strides = [1]
+// CHECK: return %[[R]] : vector<4xf32>
+func.func @subview_disjoint_writes_boundary_read(%vA: vector<4xf32>, %vB: vector<4xf32>, %pad: f32) -> vector<4xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8xf32>
+ %s0 = memref.subview %a[0] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1]>>
+ %s4 = memref.subview %a[4] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: 4>>
+ %s2 = memref.subview %a[2] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: 2>>
+ vector.transfer_write %vA, %s0[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1]>>
+ vector.transfer_write %vB, %s4[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1], offset: 4>>
+ %r = vector.transfer_read %s2[%c0], %pad {in_bounds = [true]} : memref<4xf32, strided<[1], offset: 2>>, vector<4xf32>
+ return %r : vector<4xf32>
+}
+
+// -----
+
+// Two overlapping subview writes: %vA covers [0, 4) and %vB covers [2, 6),
+// overlapping on [2, 4). Each write composes into the parent value in program
+// order, so the buffer holds %vA at [0, 2) and %vB at [2, 6). A subview read
+// spanning [1, 5) then extracts across the overlap, reading one lane of %vA
+// and three lanes of %vB from the composed value.
+
+// CHECK-LABEL: func.func @subview_overlapping_writes_overlap_read
+// CHECK-SAME: (%[[VA:.*]]: vector<4xf32>, %[[VB:.*]]: vector<4xf32>, %[[PAD:.*]]: f32)
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: memref.subview
+// CHECK: %[[D0:.*]] = vector.insert_strided_slice %[[VA]], %{{.*}} offsets = [0], strides = [1]
+// CHECK: %[[D1:.*]] = vector.insert_strided_slice %[[VB]], %[[D0]] offsets = [2], strides = [1]
+// CHECK: %[[R:.*]] = vector.extract_strided_slice %[[D1]] offsets = [1], sizes = [4], strides = [1]
+// CHECK: return %[[R]] : vector<4xf32>
+func.func @subview_overlapping_writes_overlap_read(%vA: vector<4xf32>, %vB: vector<4xf32>, %pad: f32) -> vector<4xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8xf32>
+ %s0 = memref.subview %a[0] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1]>>
+ %s2 = memref.subview %a[2] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: 2>>
+ %s1 = memref.subview %a[1] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: 1>>
+ vector.transfer_write %vA, %s0[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1]>>
+ vector.transfer_write %vB, %s2[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1], offset: 2>>
+ %r = vector.transfer_read %s1[%c0], %pad {in_bounds = [true]} : memref<4xf32, strided<[1], offset: 1>>, vector<4xf32>
+ return %r : vector<4xf32>
+}
+
+// -----
+
+// A buffer allocated before an scf.for and accessed inside the loop body only
+// through a subview, with a cross-iteration dependence (each iteration reads the
+// value the previous iteration wrote). The subview aliaser composes with region
+// promotion: the whole buffer is carried across iterations as a vector iter_arg,
+// the in-body read/write become extract/insert_strided_slice on that value.
+
+// CHECK-LABEL: func.func @subview_in_scf_for_cross_iter
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: memref.subview
+// CHECK-NOT: vector.transfer_write
+// CHECK-NOT: vector.transfer_read
+// CHECK: %[[R:.*]] = scf.for %{{.*}} iter_args(%[[IT:.*]] = %{{.*}}) -> (vector<8xf32>)
+// CHECK: %[[V:.*]] = vector.extract_strided_slice %[[IT]] offsets = [0], sizes = [4], strides = [1]
+// CHECK: %[[N:.*]] = arith.addf %[[V]], %[[V]]
+// CHECK: %[[INS:.*]] = vector.insert_strided_slice %[[N]], %[[IT]] offsets = [0], strides = [1]
+// CHECK: scf.yield %[[INS]] : vector<8xf32>
+// CHECK: vector.extract_strided_slice %[[R]] offsets = [0], sizes = [4], strides = [1]
+func.func @subview_in_scf_for_cross_iter(%lb: index, %ub: index, %step: index, %init: vector<8xf32>, %pad: f32) -> vector<4xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8xf32>
+ vector.transfer_write %init, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
+ scf.for %i = %lb to %ub step %step {
+ %sv = memref.subview %a[0] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1]>>
+ %v = vector.transfer_read %sv[%c0], %pad {in_bounds = [true]} : memref<4xf32, strided<[1]>>, vector<4xf32>
+ %n = arith.addf %v, %v : vector<4xf32>
+ vector.transfer_write %n, %sv[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1]>>
+ }
+ %svr = memref.subview %a[0] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1]>>
+ %r = vector.transfer_read %svr[%c0], %pad {in_bounds = [true]} : memref<4xf32, strided<[1]>>, vector<4xf32>
+ return %r : vector<4xf32>
+}
+
+// -----
+
+// Reads and writes may occur in any order: a read of the subview before any
+// write returns the slot's default value (ub.poison), matching the semantics of
+// reading an uninitialized alloca. A later read observes the written value.
+
+// CHECK-LABEL: func.func @subview_read_before_write
+// CHECK-SAME: (%[[V:.*]]: vector<4xf32>, %[[PAD:.*]]: f32)
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: memref.subview
+// CHECK: %[[POISON:.*]] = ub.poison : vector<8xf32>
+// CHECK: %[[R0:.*]] = vector.extract_strided_slice %[[POISON]] offsets = [0], sizes = [4], strides = [1]
+// CHECK: %[[INS:.*]] = vector.insert_strided_slice %[[V]], %[[POISON]] offsets = [0], strides = [1]
+// CHECK: %[[R1:.*]] = vector.extract_strided_slice %[[INS]] offsets = [0], sizes = [4], strides = [1]
+// CHECK: return %[[R0]], %[[R1]] : vector<4xf32>, vector<4xf32>
+func.func @subview_read_before_write(%v: vector<4xf32>, %pad: f32) -> (vector<4xf32>, vector<4xf32>) {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8xf32>
+ %s = memref.subview %a[0] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1]>>
+ %r0 = vector.transfer_read %s[%c0], %pad {in_bounds = [true]} : memref<4xf32, strided<[1]>>, vector<4xf32>
+ vector.transfer_write %v, %s[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1]>>
+ %r1 = vector.transfer_read %s[%c0], %pad {in_bounds = [true]} : memref<4xf32, strided<[1]>>, vector<4xf32>
+ return %r0, %r1 : vector<4xf32>, vector<4xf32>
+}
+
+// -----
+
+// A dynamic subview offset cannot be expressed as a static strided slice: the
+// buffer is left untouched.
+
+// CHECK-LABEL: func.func @negative_subview_dynamic_offset
+// CHECK: memref.alloca
+// CHECK: memref.subview
+func.func @negative_subview_dynamic_offset(%v: vector<4xf32>, %init: vector<8xf32>, %pad: f32, %off: index) -> vector<8xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8xf32>
+ vector.transfer_write %init, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
+ %sv = memref.subview %a[%off] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: ?>>
+ vector.transfer_write %v, %sv[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1], offset: ?>>
+ %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<8xf32>, vector<8xf32>
+ return %r : vector<8xf32>
+}
+
+// -----
+
+// A rank-reducing subview (2d -> 1d) is not promotable: strided-slice projection
+// requires equal rank.
+
+// CHECK-LABEL: func.func @negative_subview_rank_reducing
+// CHECK: memref.alloca
+// CHECK: memref.subview
+func.func @negative_subview_rank_reducing(%v: vector<4xf32>, %init: vector<2x4xf32>, %pad: f32) -> vector<2x4xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<2x4xf32>
+ vector.transfer_write %init, %a[%c0, %c0] {in_bounds = [true, true]} : vector<2x4xf32>, memref<2x4xf32>
+ %sv = memref.subview %a[1, 0] [1, 4] [1, 1] : memref<2x4xf32> to memref<4xf32, strided<[1], offset: 4>>
+ vector.transfer_write %v, %sv[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1], offset: 4>>
+ %r = vector.transfer_read %a[%c0, %c0], %pad {in_bounds = [true, true]} : memref<2x4xf32>, vector<2x4xf32>
+ return %r : vector<2x4xf32>
+}
+
+
+// A prefetching software-pipelined loop. The stage buffer double-buffers the
+// prefetched tile: the prefetch of the next tile is skipped on the last iteration.
+
+// CHECK-LABEL: func.func @pipelined_prefetch
+// CHECK-NOT: memref.alloca
+// CHECK: %[[P0:.*]] = vector.transfer_read %{{.*}} : memref<64xf32>, vector<8xf32>
+// CHECK-NOT: vector.transfer_write
+// CHECK: scf.for %[[I:.*]] = {{.*}} iter_args(%[[ACC:.*]] = %{{.*}}, %[[STAGE:.*]] = %[[P0]]) -> (vector<8xf32>, vector<8xf32>)
+// CHECK: %[[ACCN:.*]] = arith.addf %[[ACC]], %[[STAGE]] : vector<8xf32>
+// CHECK: %[[INEXT:.*]] = arith.addi %[[I]], %{{.*}}
+// CHECK: %[[G:.*]] = arith.cmpi slt, %[[INEXT]], %{{.*}}
+// CHECK: %[[NEXT:.*]] = scf.if %[[G]] -> (vector<8xf32>) {
+// CHECK: %[[CUR:.*]] = vector.transfer_read %{{.*}}[%[[INEXT]]]
+// CHECK: scf.yield %[[CUR]] : vector<8xf32>
+// CHECK: } else {
+// CHECK: scf.yield %[[STAGE]] : vector<8xf32>
+// CHECK: }
+// CHECK: scf.yield %[[ACCN]], %[[NEXT]] : vector<8xf32>, vector<8xf32>
+func.func @pipelined_prefetch(%lb: index, %ub: index, %step: index, %in: memref<64xf32>, %pad: f32) -> vector<8xf32> {
+ %c0 = arith.constant 0 : index
+ %c1 = arith.constant 1 : index
+ %cst = arith.constant dense<0.0> : vector<8xf32>
+ %stage = memref.alloca() : memref<8xf32>
+ // Prologue: prefetch tile[0] into the stage buffer.
+ %p0 = vector.transfer_read %in[%c0], %pad {in_bounds = [true]} : memref<64xf32>, vector<8xf32>
+ vector.transfer_write %p0, %stage[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
+ %r = scf.for %i = %lb to %ub step %step iter_args(%acc = %cst) -> (vector<8xf32>) {
+ // Consume the tile prefetched by the previous iteration (unconditional).
+ %tile = vector.transfer_read %stage[%c0], %pad {in_bounds = [true]} : memref<8xf32>, vector<8xf32>
+ %accn = arith.addf %acc, %tile : vector<8xf32>
+ // Prefetch the next tile into the stage buffer (skip on the last iteration).
+ %inext = arith.addi %i, %c1 : index
+ %g = arith.cmpi slt, %inext, %ub : index
+ scf.if %g {
+ %next = vector.transfer_read %in[%inext], %pad {in_bounds = [true]} : memref<64xf32>, vector<8xf32>
+ vector.transfer_write %next, %stage[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
+ }
+ scf.yield %accn : vector<8xf32>
+ }
+ return %r : vector<8xf32>
+}
+
+// -----
+
+// A whole-buffer accumulator updated conditionally inside an scf.if (a masked
+// running reduction).
+
+// CHECK-LABEL: func.func @cond_accumulate
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: vector.transfer_write
+// CHECK-NOT: vector.transfer_read
+// CHECK: %[[R:.*]] = scf.for {{.*}} iter_args(%[[ACC:.*]] = %{{.*}}) -> (vector<4xf32>)
+// CHECK: %[[M:.*]] = memref.load
+// CHECK: %[[NEW:.*]] = scf.if %[[M]] -> (vector<4xf32>) {
+// CHECK: %[[D:.*]] = arith.addf %[[ACC]], %[[ACC]] : vector<4xf32>
+// CHECK: scf.yield %[[D]] : vector<4xf32>
+// CHECK: } else {
+// CHECK: scf.yield %[[ACC]] : vector<4xf32>
+// CHECK: }
+// CHECK: scf.yield %[[NEW]] : vector<4xf32>
+// CHECK: return %[[R]] : vector<4xf32>
+func.func @cond_accumulate(%lb: index, %ub: index, %step: index, %init: vector<4xf32>, %pad: f32, %mask: memref<?xi1>) -> vector<4xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<4xf32>
+ vector.transfer_write %init, %a[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32>
+ scf.for %i = %lb to %ub step %step {
+ %m = memref.load %mask[%i] : memref<?xi1>
+ %v = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<4xf32>, vector<4xf32>
+ scf.if %m {
+ %d = arith.addf %v, %v : vector<4xf32>
+ vector.transfer_write %d, %a[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32>
+ }
+ scf.yield
+ }
+ %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<4xf32>, vector<4xf32>
+ return %r : vector<4xf32>
+}
+
+// -----
+
+// A tiled GEMM with a dynamic K early-exit. A local accumulator tile D sums the
+// K-tiles via vector.contract only while k < dyn_k.
+
+// CHECK-LABEL: func.func @gemm_k_early_exit
+// CHECK-SAME: (%[[A:.*]]: memref<4x16xf32>, %[[B:.*]]: memref<16x4xf32>, %[[C:.*]]: memref<4x4xf32>, %[[DYNK:.*]]: index, %[[PAD:.*]]: f32)
+// CHECK-NOT: memref.alloca
+// CHECK: %[[R:.*]] = scf.for %[[K:.*]] = {{.*}} iter_args(%[[ACC:.*]] = %{{.*}}) -> (vector<4x4xf32>)
+// CHECK: %[[INRANGE:.*]] = arith.cmpi slt, %[[K]], %[[DYNK]]
+// CHECK: %[[NEW:.*]] = scf.if %[[INRANGE]] -> (vector<4x4xf32>) {
+// CHECK: %[[ATILE:.*]] = vector.transfer_read %[[A]]
+// CHECK: %[[BTILE:.*]] = vector.transfer_read %[[B]]
+// CHECK: %[[MM:.*]] = vector.contract {{.*}} %[[ATILE]], %[[BTILE]], %[[ACC]]
+// CHECK: scf.yield %[[MM]] : vector<4x4xf32>
+// CHECK: } else {
+// CHECK: scf.yield %[[ACC]] : vector<4x4xf32>
+// CHECK: }
+// CHECK: scf.yield %[[NEW]] : vector<4x4xf32>
+// CHECK: %[[CVAL:.*]] = vector.transfer_read %[[C]]
+// CHECK: %[[SUM:.*]] = arith.addf %[[CVAL]], %[[R]] : vector<4x4xf32>
+// CHECK: vector.transfer_write %[[SUM]], %[[C]]
+func.func @gemm_k_early_exit(%A: memref<4x16xf32>, %B: memref<16x4xf32>,
+ %C: memref<4x4xf32>, %dyn_k: index, %pad: f32) {
+ %c0 = arith.constant 0 : index
+ %c4 = arith.constant 4 : index
+ %c16 = arith.constant 16 : index
+ %cst = arith.constant dense<0.0> : vector<4x4xf32>
+ // Local accumulator tile D, zero-initialized.
+ %d = memref.alloca() : memref<4x4xf32>
+ vector.transfer_write %cst, %d[%c0, %c0] {in_bounds = [true, true]} : vector<4x4xf32>, memref<4x4xf32>
+ scf.for %k = %c0 to %c16 step %c4 {
+ // Early exit: only accumulate K tiles whose offset is below dyn_k.
+ %inrange = arith.cmpi slt, %k, %dyn_k : index
+ scf.if %inrange {
+ %atile = vector.transfer_read %A[%c0, %k], %pad {in_bounds = [true, true]} : memref<4x16xf32>, vector<4x4xf32>
+ %btile = vector.transfer_read %B[%k, %c0], %pad {in_bounds = [true, true]} : memref<16x4xf32>, vector<4x4xf32>
+ %acc = vector.transfer_read %d[%c0, %c0], %pad {in_bounds = [true, true]} : memref<4x4xf32>, vector<4x4xf32>
+ %mm = vector.contract {indexing_maps = [affine_map<(m, n, k) -> (m, k)>, affine_map<(m, n, k) -> (k, n)>, affine_map<(m, n, k) -> (m, n)>], iterator_types = ["parallel", "parallel", "reduction"], kind = #vector.kind<add>} %atile, %btile, %acc : vector<4x4xf32>, vector<4x4xf32> into vector<4x4xf32>
+ vector.transfer_write %mm, %d[%c0, %c0] {in_bounds = [true, true]} : vector<4x4xf32>, memref<4x4xf32>
+ }
+ scf.yield
+ }
+ // Accumulate the local tile D into the C parameter: C = C + D.
+ %cval = vector.transfer_read %C[%c0, %c0], %pad {in_bounds = [true, true]} : memref<4x4xf32>, vector<4x4xf32>
+ %dval = vector.transfer_read %d[%c0, %c0], %pad {in_bounds = [true, true]} : memref<4x4xf32>, vector<4x4xf32>
+ %sum = arith.addf %cval, %dval : vector<4x4xf32>
+ vector.transfer_write %sum, %C[%c0, %c0] {in_bounds = [true, true]} : vector<4x4xf32>, memref<4x4xf32>
+ return
+}
+
+// -----
+
+// A whole-buffer read with a mask promotes to select(mask, reachingDef, padding).
+// CHECK-LABEL: func.func @read_with_mask(
+// CHECK-SAME: %[[V:.*]]: vector<8xf32>, %[[M:.*]]: vector<8xi1>, %[[PAD:.*]]: f32
+// CHECK-NOT: memref.alloca
+// CHECK: %[[PS:.*]] = vector.broadcast %[[PAD]] : f32 to vector<8xf32>
+// CHECK: %[[SEL:.*]] = arith.select %[[M]], %[[V]], %[[PS]]
+// CHECK: return %[[SEL]]
+func.func @read_with_mask(%v: vector<8xf32>, %m: vector<8xi1>, %pad: f32) -> vector<8xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8xf32>
+ vector.transfer_write %v, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
+ %r = vector.transfer_read %a[%c0], %pad, %m {in_bounds = [true]} : memref<8xf32>, vector<8xf32>
+ return %r : vector<8xf32>
+}
+
+// -----
+
+// A whole-buffer write with a mask selects the stored value for active lanes
+// and preserves the previous value for the rest.
+// CHECK-LABEL: func.func @write_with_mask(
+// CHECK-SAME: %[[V:.*]]: vector<8xf32>, %[[W:.*]]: vector<8xf32>, %[[M:.*]]: vector<8xi1>
+// CHECK-NOT: memref.alloca
+// CHECK: %[[SEL:.*]] = arith.select %[[M]], %[[W]], %[[V]]
+// CHECK: return %[[SEL]]
+func.func @write_with_mask(%v: vector<8xf32>, %w: vector<8xf32>, %m: vector<8xi1>, %pad: f32) -> vector<8xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8xf32>
+ vector.transfer_write %v, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
+ vector.transfer_write %w, %a[%c0], %m {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
+ %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<8xf32>, vector<8xf32>
+ return %r : vector<8xf32>
+}
+
+// -----
+
+// A write with a mask into a static subview composes through the subview aliaser:
+// the region is projected out (extract_strided_slice), the mask selects the
+// stored value, and the result is inserted back (insert_strided_slice).
+// CHECK-LABEL: func.func @write_with_mask_static_subview(
+// CHECK-SAME: %[[V:.*]]: vector<4xf32>, %[[INIT:.*]]: vector<8xf32>, %{{.*}}: f32, %[[M:.*]]: vector<4xi1>
+// CHECK-NOT: memref.alloca
+// CHECK: %[[EX:.*]] = vector.extract_strided_slice %[[INIT]] offsets = [2], sizes = [4]
+// CHECK: %[[SEL:.*]] = arith.select %[[M]], %[[V]], %[[EX]]
+// CHECK: %[[INS:.*]] = vector.insert_strided_slice %[[SEL]], %[[INIT]] offsets = [2]
+// CHECK: return %[[INS]]
+func.func @write_with_mask_static_subview(%v: vector<4xf32>, %init: vector<8xf32>, %pad: f32, %m: vector<4xi1>) -> vector<8xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8xf32>
+ vector.transfer_write %init, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
+ %sv = memref.subview %a[2] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: 2>>
+ vector.transfer_write %v, %sv[%c0], %m {in_bounds = [true]} : vector<4xf32>, memref<4xf32, strided<[1], offset: 2>>
+ %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<8xf32>, vector<8xf32>
+ return %r : vector<8xf32>
+}
+
+// -----
+
+// NEGATIVE: the masked form (`vector.mask` wrapping the transfer) is not
+// promoted. The transfer's use of the buffer is inside a region, and Mem2Reg
+// only promotes through regions whose parent op implements
+// `PromotableRegionOpInterface`, which `vector.mask` does not.
+
+// CHECK-LABEL: func.func @negative_masked_read(
+// CHECK: memref.alloca
+// CHECK: vector.mask
+func.func @negative_masked_read(%v: vector<8xf32>, %m: vector<8xi1>, %pad: f32) -> vector<8xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8xf32>
+ vector.transfer_write %v, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
+ %r = vector.mask %m { vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<8xf32>, vector<8xf32> } : vector<8xi1> -> vector<8xf32>
+ return %r : vector<8xf32>
+}
+
+// -----
+
+// NEGATIVE: a write wrapped in vector.mask also prevents buffer promotion.
+
+// CHECK-LABEL: func.func @negative_masked_write(
+// CHECK: memref.alloca
+// CHECK: vector.mask
+func.func @negative_masked_write(%v: vector<8xf32>, %w: vector<8xf32>, %m: vector<8xi1>, %pad: f32) -> vector<8xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8xf32>
+ vector.transfer_write %v, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
+ vector.mask %m { vector.transfer_write %w, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32> } : vector<8xi1>
+ %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<8xf32>, vector<8xf32>
+ return %r : vector<8xf32>
+}
>From 0305cef696e0696db22a599495fc9fc88262b5fe Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Fri, 16 Oct 2026 05:57:44 +0000
Subject: [PATCH 2/2] [mlir][mem2reg] Drop the mask on copies out of a dynamic
subview
A copy out of a dynamic-subview slot does not need a mask: `memref.copy`
requires the target to have the subview's runtime shape, and the
`vector.transfer_write` it becomes is out-of-bounds on every dimension
where the target may be shorter than the vector, so lanes outside the
extent are not written.
Add tests for a copy out of a dynamic subview, a copy between dynamic
subviews of two slots, and negative tests for copies between two subviews
of the same buffer.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply at anthropic.com>
---
.../Transforms/MemorySlotOpInterfaceImpl.cpp | 24 ++---
mlir/test/Transforms/mem2reg/memref-copy.mlir | 87 +++++++++++++++++++
2 files changed, 96 insertions(+), 15 deletions(-)
diff --git a/mlir/lib/Dialect/MemRef/Transforms/MemorySlotOpInterfaceImpl.cpp b/mlir/lib/Dialect/MemRef/Transforms/MemorySlotOpInterfaceImpl.cpp
index 2c87125ef92253..f3b4a383cd60c6 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/MemorySlotOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/MemorySlotOpInterfaceImpl.cpp
@@ -197,10 +197,9 @@ static Value readMemRefAsVector(OpBuilder &builder, Location loc, Value mem,
/// Stores a slot's value to the target of a `memref.copy`: writes `vec` into
/// `mem` at the origin, with the identity map and in-bounds rule of
-/// `readMemRefAsVector`. `mask`, when set, restricts the write to the lanes it
-/// selects.
+/// `readMemRefAsVector`.
static void writeVectorToMemRef(OpBuilder &builder, Location loc, Value vec,
- Value mem, Value mask = {}) {
+ Value mem) {
auto memType = cast<MemRefType>(mem.getType());
auto vecType = cast<VectorType>(vec.getType());
assert(!vecType.isScalable() && "expected a fixed-size vector");
@@ -213,8 +212,8 @@ static void writeVectorToMemRef(OpBuilder &builder, Location loc, Value vec,
memType.getDimSize(d) >= vecType.getDimSize(d);
vector::TransferWriteOp::create(
builder, loc, vec, mem, indices,
- AffineMapAttr::get(builder.getMultiDimIdentityMap(rank)), mask,
- builder.getBoolArrayAttr(inBounds));
+ AffineMapAttr::get(builder.getMultiDimIdentityMap(rank)),
+ /*mask=*/Value(), builder.getBoolArrayAttr(inBounds));
}
//===----------------------------------------------------------------------===//
@@ -230,7 +229,7 @@ namespace {
/// A copy between two slots is rewritten one slot at a time, in either order.
/// For dynamic subviews,
/// `PromotableAliaserInterface::projectAliasValueToSlotValue` masks writes into
-/// the slot, and this model masks copies out of it.
+/// the slot.
struct CopyOpMemOpModel
: public PromotableMemOpInterface::ExternalModel<CopyOpMemOpModel,
memref::CopyOp> {
@@ -265,7 +264,7 @@ struct CopyOpMemOpModel
// not modeled here.
if (srcIsSlot == dstIsSlot)
return false;
- // No further check: `memref.copy` verifies that both operands have the same
+ // No further check: `memref.copy` requires both operands to have the same
// shape and element type.
return true;
}
@@ -278,14 +277,9 @@ struct CopyOpMemOpModel
OpBuilder &builder, Value reachingDefinition,
const DataLayout &dataLayout) const {
auto copyOp = cast<memref::CopyOp>(op);
- if (copyOp.getSource() == slot.ptr) {
- Location loc = op->getLoc();
- Value mask;
- if (memref::SubViewOp subView = getDynamicSubView(slot.ptr))
- mask = buildSubViewMask(builder, loc, subView);
- writeVectorToMemRef(builder, loc, reachingDefinition, copyOp.getTarget(),
- mask);
- }
+ if (copyOp.getSource() == slot.ptr)
+ writeVectorToMemRef(builder, op->getLoc(), reachingDefinition,
+ copyOp.getTarget());
return DeletionKind::Delete;
}
};
diff --git a/mlir/test/Transforms/mem2reg/memref-copy.mlir b/mlir/test/Transforms/mem2reg/memref-copy.mlir
index 27bd5a7e6100d7..144178ad8efb9e 100644
--- a/mlir/test/Transforms/mem2reg/memref-copy.mlir
+++ b/mlir/test/Transforms/mem2reg/memref-copy.mlir
@@ -171,3 +171,90 @@ func.func @copy_global_to_dyn_subview(%v: vector<128x64xf16>, %n: index, %pad: f
%r = vector.transfer_read %slm[%c0, %c0], %pad {in_bounds = [true, true]} : memref<128x64xf16, 3>, vector<128x64xf16>
return %r : vector<128x64xf16>
}
+
+// -----
+
+// A copy out of a dynamic subview becomes an unmasked transfer_write: the
+// target has the subview's runtime shape, so the write's out-of-bounds lanes
+// fall outside it.
+
+// CHECK-LABEL: func.func @copy_out_of_dyn_subview(
+// CHECK-SAME: %[[V:.*]]: vector<8x16xf32>, %[[DST:.*]]: memref<?x?xf32>
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: memref.copy
+// CHECK-NOT: vector.create_mask
+// CHECK: vector.transfer_write %[[V]], %[[DST]][%{{.*}}, %{{.*}}] : vector<8x16xf32>, memref<?x?xf32>
+// CHECK-NEXT: return
+func.func @copy_out_of_dyn_subview(%v: vector<8x16xf32>, %dst: memref<?x?xf32>, %n: index) {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8x16xf32>
+ vector.transfer_write %v, %a[%c0, %c0] {in_bounds = [true, true]} : vector<8x16xf32>, memref<8x16xf32>
+ %sv = memref.subview %a[0, 0] [%n, %n] [1, 1] : memref<8x16xf32> to memref<?x?xf32, strided<[16, 1]>>
+ memref.copy %sv, %dst : memref<?x?xf32, strided<[16, 1]>> to memref<?x?xf32>
+ return
+}
+
+// -----
+
+// Copying between dynamic subviews of two promotable slots eliminates both
+// buffers: only the elements inside the extent reach the target's value.
+
+// CHECK-LABEL: func.func @copy_dyn_subview_between_slots(
+// CHECK-SAME: %[[VA:.*]]: vector<8x16xf32>, %[[VB:.*]]: vector<8x16xf32>, %[[N:.*]]: index
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: memref.copy
+// CHECK: %[[M:.*]] = vector.create_mask %{{.*}}, %[[N]] : vector<8x16xi1>
+// CHECK: %[[SEL:.*]] = arith.select %[[M]], %[[VA]], %[[VB]]
+// CHECK: return %[[SEL]]
+func.func @copy_dyn_subview_between_slots(%va: vector<8x16xf32>, %vb: vector<8x16xf32>, %n: index, %pad: f32) -> vector<8x16xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8x16xf32>
+ %b = memref.alloca() : memref<8x16xf32>
+ vector.transfer_write %va, %a[%c0, %c0] {in_bounds = [true, true]} : vector<8x16xf32>, memref<8x16xf32>
+ vector.transfer_write %vb, %b[%c0, %c0] {in_bounds = [true, true]} : vector<8x16xf32>, memref<8x16xf32>
+ %sa = memref.subview %a[0, 0] [8, %n] [1, 1] : memref<8x16xf32> to memref<8x?xf32, strided<[16, 1]>>
+ %sb = memref.subview %b[0, 0] [8, %n] [1, 1] : memref<8x16xf32> to memref<8x?xf32, strided<[16, 1]>>
+ memref.copy %sa, %sb : memref<8x?xf32, strided<[16, 1]>> to memref<8x?xf32, strided<[16, 1]>>
+ %r = vector.transfer_read %b[%c0, %c0], %pad {in_bounds = [true, true]} : memref<8x16xf32>, vector<8x16xf32>
+ return %r : vector<8x16xf32>
+}
+
+// -----
+
+// NEGATIVE: a copy between two dynamic subviews of the same buffer is not
+// promoted.
+
+// CHECK-LABEL: func.func @negative_copy_dyn_subviews_same_buffer(
+// CHECK: memref.alloca
+// CHECK: memref.copy
+// CHECK: vector.transfer_read
+func.func @negative_copy_dyn_subviews_same_buffer(%v: vector<8x16xf32>, %n: index, %m: index, %pad: f32) -> vector<8x16xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8x16xf32>
+ vector.transfer_write %v, %a[%c0, %c0] {in_bounds = [true, true]} : vector<8x16xf32>, memref<8x16xf32>
+ %s1 = memref.subview %a[0, 0] [8, %n] [1, 1] : memref<8x16xf32> to memref<8x?xf32, strided<[16, 1]>>
+ %s2 = memref.subview %a[0, 0] [8, %m] [1, 1] : memref<8x16xf32> to memref<8x?xf32, strided<[16, 1]>>
+ memref.copy %s1, %s2 : memref<8x?xf32, strided<[16, 1]>> to memref<8x?xf32, strided<[16, 1]>>
+ %r = vector.transfer_read %a[%c0, %c0], %pad {in_bounds = [true, true]} : memref<8x16xf32>, vector<8x16xf32>
+ return %r : vector<8x16xf32>
+}
+
+// -----
+
+// NEGATIVE: a copy between two static subviews of the same buffer, shifting
+// the low half into the high half, is not promoted.
+
+// CHECK-LABEL: func.func @negative_copy_static_subviews_same_buffer(
+// CHECK: memref.alloca
+// CHECK: memref.copy
+// CHECK: vector.transfer_read
+func.func @negative_copy_static_subviews_same_buffer(%v: vector<8xf32>, %pad: f32) -> vector<8xf32> {
+ %c0 = arith.constant 0 : index
+ %a = memref.alloca() : memref<8xf32>
+ vector.transfer_write %v, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
+ %lo = memref.subview %a[0] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1]>>
+ %hi = memref.subview %a[4] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1], offset: 4>>
+ memref.copy %lo, %hi : memref<4xf32, strided<[1]>> to memref<4xf32, strided<[1], offset: 4>>
+ %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<8xf32>, vector<8xf32>
+ return %r : vector<8xf32>
+}
More information about the Mlir-commits
mailing list